On demand pull request benchmark (#5286)

* Enable on-demand benchmarking of a pull request.

* Better benchmark debug printing.

* PR benchmarking instructions.

* Specify conda_channels in asv.conf.json .

* Use GitHub markdown for benchmark_this label.

* What's New entry.

Author: trexfeathers

.github/workflows/benchmark.yml

@@ -12,10 +12,15 @@ on:
         description: "Argument to be passed to the overnight benchmark script."
         required: false
         type: string
+  pull_request:
+    types: [labeled]
 
 jobs:
   benchmark:
-    if: "github.repository == 'SciTools/iris'"
+    if: >
+      github.repository == 'SciTools/iris' && 
+      (github.event_name != 'pull_request' || 
+      github.event.label.name == 'benchmark_this')
     runs-on: ubuntu-latest
 
     env:
@@ -67,7 +72,14 @@ jobs:
         run: |
           echo "OVERRIDE_TEST_DATA_REPOSITORY=${GITHUB_WORKSPACE}/${IRIS_TEST_DATA_PATH}/test_data" >> $GITHUB_ENV
 
+      - name: Benchmark this pull request
+        if: ${{ github.event.label.name == 'benchmark_this' }}
+        run: |
+          git checkout ${{ github.head_ref }}
+          python benchmarks/bm_runner.py branch origin/${{ github.base_ref }}
+
       - name: Run overnight benchmarks
+        if: ${{ github.event_name != 'pull_request' }}
         run: |
           first_commit=${{ inputs.first_commit }}
           if [ "$first_commit" == "" ]
@@ -81,6 +93,7 @@ jobs:
           fi
 
       - name: Create issues for performance shifts
+        if: ${{ github.event_name != 'pull_request' }}
         env:
           GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
         run: |

benchmarks/README.md

@@ -11,6 +11,15 @@ shifts in performance being flagged in a new GitHub issue.
 
 ## Running benchmarks
 
+On GitHub: a Pull Request can be benchmarked by adding the 
+https://github.com/SciTools/iris/labels/benchmark_this 
+label to the PR (to run a second time: just remove and re-add the label).
+Note that a benchmark run could take an hour or more to complete.
+This runs a comparison between the PR branch's ``HEAD`` and its merge-base with
+the PR's base branch, thus showing performance differences introduced
+by the PR. (This run is managed by 
+[the aforementioned GitHub Action](../.github/workflows/benchmark.yml)).
+
 `asv ...` commands must be run from this directory. You will need to have ASV
 installed, as well as Nox (see
 [Benchmark environments](#benchmark-environments)).

benchmarks/asv.conf.json

@@ -4,6 +4,7 @@
     "project_url": "https://github.com/SciTools/iris",
     "repo": "..",
     "environment_type": "conda-delegated",
+    "conda_channels": ["conda-forge", "defaults"],
     "show_commit_url": "http://github.com/scitools/iris/commit/",
     "branches": ["upstream/main"],
 

benchmarks/bm_runner.py

@@ -33,7 +33,9 @@
 
 
 def _subprocess_run_print(args, **kwargs):
-    print(f"BM_RUNNER DEBUG: {' '.join(args)}")
+    # Use subprocess for printing to reduce chance of printing out of sequence
+    #  with the subsequent calls.
+    subprocess.run(["echo", f"BM_RUNNER DEBUG: {' '.join(args)}"])
     return subprocess.run(args, **kwargs)
 
 

docs/src/developers_guide/contributing_benchmarks.rst

@@ -22,10 +22,11 @@ requests, instead it is **run overnight against each the commits of the
 previous day** to check if any commit has introduced performance shifts.
 Detected shifts are reported in a new Iris GitHub issue.
 
+.. _on_demand_pr_benchmark:
+
 If a pull request author/reviewer suspects their changes may cause performance
-shifts, a convenience script is available to replicate the
-overnight benchmark run but comparing the current ``HEAD`` with a requested
-branch (e.g. ``upstream/main``). Read more in `benchmarks/README.md`_.
+shifts, they can manually order their pull request to be benchmarked by adding
+the ``benchmark_this`` label to the PR. Read more in `benchmarks/README.md`_.
 
 Other Uses
 ----------

docs/src/whatsnew/latest.rst

@@ -119,6 +119,9 @@ This document explains the changes made to Iris for this release
 #. `@bjlittle`_ adopted `pypa/build`_ recommended best practice to build a
    binary ``wheel`` from the ``sdist``. (:pull:`5266`)
 
+#. `@trexfeathers`_ enabled on-demand benchmarking of Pull Requests; see
+   :ref:`here <on_demand_pr_benchmark>`. (:pull:`5286`)
+
 
 .. comment
     Whatsnew author names (@github name) in alphabetical order. Note that,