Benchmarks on CI run out of memory

[f0uriest]

Ugh. We might need to reduce the number of trials or overall number of benchmarks, or see if we can split the job in a sensible way.

Also it would be nice if we modified the action to always use the benchmark.py from the current branch, so that if we make changes to that we can test as part of a single PR. Right now we use the PR branch version of benchmark.py for running code on the PR branch, but then compare to running the master branch version of benchmark.py. Ideally we wouldn't be changing the file too often, but it makes debugging stuff a pain.

[dpanici]

I agree it is frustrating but I am not sure how to handle it as if we have added new benchmakrs in a PR, it makes no sense to run benchmark on it

That being said if we are not adding new files, it could be as simple as having the benchmark action checkout benchmark.py from the PR, or saving the benchmark.py as another file which is called when master is checked out.