Benchmarking Suite

[VeckoTheGecko]

Establishing and documenting a standard benchmark suite for Parcels would bring an active focus to performance within the Parcels project.

This benchmarking suite could include whole simulation tests, as well as tests that target specific parts of the codebase (e.g., particle file writing, as would be important in #1661). Note that tests relating to MPI should be realistic whole simulation tests as the loading of fieldsets and locations of particles have a significant impact on performance.

Ideally:

Have a suite of tests that can run on CI (when requested via a change in PR label) testing various core parts of the codebase, saving and uploading a waterfall report for the execution time and memory use for each function (such as those generated by sciagraph as well as IO.

Known tools:

• The Python Profilers
• asv on Conda (used in numpy and scipy) (this is used by the xarray team)
• pytest-benchmark on Conda
• Sciagraph (commercial software)

The benchmarks should be run on a machine with consistent resources (large simulations can be run on Lorenz at IMAU which has significant resources and access to hydrodynamic forcing for realistic simulations).

Related:

• seaching the issue tracker for 'benchmark'

[VeckoTheGecko]

The more I look at projects in this domain, the more I see asv being used to create benchmarks and compare them across the Git history for a project. I propose that we use asv for creating benchmarks of notable functions and classes so that we can track their performance. Whole simulation benchmarks will likely need to be done in another way (perhaps using an offline approach).

[VeckoTheGecko]

Refs:

https://labs.quansight.org/blog/2021/08/github-actions-benchmarks
pydata/xarray#5796

I'll be opting for a similar approach used in scikit-image, xarray etc. where we can run them on demand using a label for the PR. I'll work on the infrastructure and we can decide the exact benchmarks to include at a later time.

Looking into asv it seems very versatile! With ability to measure timing, memory consumption, and give an insight into profiling

[VeckoTheGecko]

@fluidnumerics-joe Can you link here the profilers you were mentioning yesterday?

[fluidnumerics-joe]

DOE's HPC Toolkit with Hatchet can be quite useful for hotspot, trace, and hardware events profiling. It's open source, free, and maintained by groups in the Department of Energy's Exascale Computing Project

Tau from the University of Oregon is another free and open source profiler. They also have a python package, pytau that can be used for manually instrumenting python applications for profiling.

Commercial profilers like Arm Forge (which now looks like it's changed to "Linaro Forge") are quite capable and easy to use. Many HPC centers carry licenses for users.