(maybe) Add Tooling and Automation for Benchmarking/Profiling

[ericsnowcurrently]

One we have our benchmarking machine set up there are a number of things we should consider doing to make life easier for us. Most will involve working through the persistent Azure host we'll be using for tunneling anyway.

Here is the possible tooling and automation to think about:

• move the scripts from and to a repo (e.g. https://github.com/faster-cpython/tools)
• <bench>
  • add a job queue (for benchmarking & profiling)
    • CLI tool ("add", "list", "remove")
    • cron job or service/daemon to run jobs out of the queue
    • upload
  • add a cron job or service to keep master up-to-date
• <portal>
  • run the job queue (from above) here instead of on the benchmarking machine
  • artifact storage, including benchmark results
  • a web interface for benchmark results (see speed.python.org and https://github.com/tobami/codespeed)
  • a simple web interface for the job queue
  • set up a cron job or service to generate up-to-date benchmarking results
    • for a curated list of tags (plus any new ones) and for master (maybe once each day)
    • we'll use these for comparison
• <pyperformance>
  • a script to set up a "compile" config for a job
• <github>
  • create a github action for requesting a benchmarking run and checking the results
  • do benchmarking runs for PRs; options:
    • a PR check (fail if performance regresses)
    • a bot to listen for requests in PR comments
• <speed.python.org> keep those up-to-date results here instead
• ...

[gvanrossum]

How are we doing access control? Note that Mark (and other core devs who might help out) won’t have MS intranet access.

[ericsnowcurrently]

Good question. For shell access we'll use SSH keys. For any web interface we might introduce I suppose we can use OAuth through GitHub.

[gvanrossum]

Or maybe the web UI should be read-only? We're working on open source so that should be fine.