Skip to content

Commit f7b173f

Browse files
jenshnielsenjenshnielsen
authored
Merge pull request #1202 from astafan8/setup-benchmarking-and-add-simple-dataset-benchmark
Setup benchmarking and add simple dataset benchmark
2 parents ded5447 + f97c5c0 commit f7b173f

File tree

7 files changed

+409
-7
lines changed

7 files changed

+409
-7
lines changed

‎.gitignore

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -89,4 +89,12 @@ docs/examples/data/*
8989
.DS_Store
9090

9191
# qcodes related
92-
experiments.db
92+
experiments.db
93+
94+
# Do not ignore asv configuration
95+
!benchmarking/asv.conf.json
96+
97+
# Ignore files generated by asv benchmarking
98+
benchmarking/env/
99+
benchmarking/html/
100+
benchmarking/results/

‎.travis.yml

Lines changed: 11 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -31,13 +31,18 @@ before_script: # configure a headless display to test plot generation
3131
- sleep 3 # give xvfb some time to start
3232

3333
script:
34-
- cd qcodes
35-
- py.test --cov=qcodes --cov-report xml --cov-config=.coveragerc
36-
# build docs with warnings as errors
3734
- |
38-
cd ..
39-
mypy qcodes
40-
35+
cd qcodes
36+
py.test --cov=qcodes --cov-report xml --cov-config=.coveragerc
37+
- cd ..
38+
- mypy qcodes
39+
# run benchmarks to make sure they are correct
40+
- |
41+
cd benchmarking
42+
asv machine --machine travis
43+
asv dev --machine travis
44+
- cd ..
45+
# build docs with warnings as errors
4146
- |
4247
cd docs
4348
make SPHINXOPTS="-W" html-api

‎benchmarking/README.rst

Lines changed: 121 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,121 @@
1+
Benchmarking QCoDeS
2+
===================
3+
4+
This directory contains benchmarks implemented for usage with airspeed
5+
velocity ``asv`` benchmarking package. Refer to `asv documentation`_ for
6+
information on how to use it.
7+
8+
.. _asv documentation: https://asv.readthedocs.io/en/stable/index.html
9+
10+
The ``asv.conf.json`` file in this directory configures ``asv`` to work
11+
correctly with QCoDeS.
12+
13+
Usage
14+
-----
15+
16+
Run benchmarking
17+
````````````````
18+
19+
If you already have a python environment set up for QCoDeS, then run the
20+
following command from this directory:
21+
22+
.. code:: bash
23+
24+
asv run python=same
25+
26+
If you do not have an environment set, then ``asv`` can set it up
27+
automatically. The benchmarks are executed in the same way:
28+
29+
.. code:: bash
30+
31+
asv run
32+
33+
Either of the commands above will execute benchmarking for the latest commit
34+
of the master branch.
35+
36+
If you want to run benchmarking for a particular commit, use the same syntax
37+
as there is used for ``git log`` (commit id with ``^!`` at the end; note that in
38+
some terminals you will need to type ``^`` two times like this ``^^!``):
39+
40+
.. code:: bash
41+
42+
asv run ed9b6fe8^!
43+
44+
Use the ``--bench`` option with a regular expression to tell ``asv`` which
45+
benchmarks you would like to execute. For example, use the following syntax
46+
to execute a benchmark called ``saving`` in ``data.py`` benchmark module:
47+
48+
.. code:: bash
49+
50+
asv run --bench data.saving
51+
52+
Refer to `asv documentation`_ for more information on the various ways the
53+
benchmarking can be executed (for example, how to run a particular
54+
benchmark, how to compare results between commits, etc).
55+
56+
Display benchmarking results
57+
````````````````````````````
58+
59+
In order to view the benchmarking results, execute the following command
60+
to generate a convenient website
61+
62+
.. code:: bash
63+
64+
asv publish
65+
66+
and the following command to start a simple server that could host the
67+
website locally (the generated website is not static, hence the server is
68+
needed)
69+
70+
.. code:: bash
71+
72+
asv preview -b
73+
74+
The ``-b`` option opens the website automatically in your default browser
75+
(the URL that it opens automatically is also printed to the terminal). In
76+
order to stop the server, press ``Ctrl+C`` in the terminal where you've
77+
started it.
78+
79+
Note that the benchmarking results are created locally on your machine, and
80+
they get accumulated.
81+
82+
In order to compare benchmarking results of two commits, use the following
83+
command (note that the benchmarking results for these two commits should
84+
already exist):
85+
86+
.. code:: bash
87+
88+
asv compare ed859c0a 8984aefb
89+
90+
91+
Profile during benchmarking
92+
```````````````````````````
93+
94+
If you would like to also profile while benchmarking in order to get more
95+
insights on the performance of the code, use either
96+
97+
.. code:: bash
98+
99+
asv run --profile
100+
101+
command or
102+
103+
.. code:: bash
104+
105+
asv profile
106+
107+
command.
108+
109+
In case you would like to use a visualization tool for the profile results,
110+
you can install one, for example, ``snakeviz``, and run benchmarking with
111+
profiling as follows:
112+
113+
.. code:: bash
114+
115+
asv profile --gui=snakeviz
116+
117+
118+
ToDo for QCoDeS/core
119+
--------------------
120+
121+
- host results and their html representation (GitHub pages?)

‎benchmarking/asv.conf.json

Lines changed: 141 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,141 @@
1+
{
2+
// The version of the config file format. Do not change, unless
3+
// you know what you are doing.
4+
"version": 1,
5+
6+
// The name of the project being benchmarked
7+
"project": "Qcodes",
8+
9+
// The project's homepage
10+
"project_url": "http://github.com/QCoDeS/Qcodes",
11+
12+
// The URL or local path of the source code repository for the
13+
// project being benchmarked
14+
"repo": "..",
15+
16+
// List of branches to benchmark. If not provided, defaults to "master"
17+
// (for git) or "default" (for mercurial).
18+
// "branches": ["master"], // for git
19+
// "branches": ["default"], // for mercurial
20+
21+
// The DVCS being used. If not set, it will be automatically
22+
// determined from "repo" by looking at the protocol in the URL
23+
// (if remote), or by looking for special directories, such as
24+
// ".git" (if local).
25+
"dvcs": "git",
26+
27+
// The tool to use to create environments. May be "conda",
28+
// "virtualenv" or other value depending on the plugins in use.
29+
// If missing or the empty string, the tool will be automatically
30+
// determined by looking for tools on the PATH environment
31+
// variable.
32+
"environment_type": "conda",
33+
34+
// timeout in seconds for installing any dependencies in environment
35+
// defaults to 10 min
36+
//"install_timeout": 600,
37+
38+
// the base URL to show a commit for the project.
39+
"show_commit_url": "https://github.com/QCoDeS/Qcodes/commit/",
40+
41+
// The Pythons you'd like to test against. If not provided, defaults
42+
// to the current version of Python used to run `asv`.
43+
// "pythons": ["2.7", "3.3"],
44+
45+
// The matrix of dependencies to test. Each key is the name of a
46+
// package (in PyPI) and the values are version numbers. An empty
47+
// list or empty string indicates to just test against the default
48+
// (latest) version. null indicates that the package is to not be
49+
// installed. If the package to be tested is only available from
50+
// PyPi, and the 'environment_type' is conda, then you can preface
51+
// the package name by 'pip+', and the package will be installed via
52+
// pip (with all the conda available packages installed first,
53+
// followed by the pip installed packages).
54+
//
55+
// "matrix": {
56+
// "numpy": ["1.6", "1.7"],
57+
// "six": ["", null], // test with and without six installed
58+
// "pip+emcee": [""], // emcee is only available for install with pip.
59+
// },
60+
61+
// Combinations of libraries/python versions can be excluded/included
62+
// from the set to test. Each entry is a dictionary containing additional
63+
// key-value pairs to include/exclude.
64+
//
65+
// An exclude entry excludes entries where all values match. The
66+
// values are regexps that should match the whole string.
67+
//
68+
// An include entry adds an environment. Only the packages listed
69+
// are installed. The 'python' key is required. The exclude rules
70+
// do not apply to includes.
71+
//
72+
// In addition to package names, the following keys are available:
73+
//
74+
// - python
75+
// Python version, as in the *pythons* variable above.
76+
// - environment_type
77+
// Environment type, as above.
78+
// - sys_platform
79+
// Platform, as in sys.platform. Possible values for the common
80+
// cases: 'linux2', 'win32', 'cygwin', 'darwin'.
81+
//
82+
// "exclude": [
83+
// {"python": "3.2", "sys_platform": "win32"}, // skip py3.2 on windows
84+
// {"environment_type": "conda", "six": null}, // don't run without six on conda
85+
// ],
86+
//
87+
// "include": [
88+
// // additional env for python2.7
89+
// {"python": "2.7", "numpy": "1.8"},
90+
// // additional env if run on windows+conda
91+
// {"platform": "win32", "environment_type": "conda", "python": "2.7", "libpython": ""},
92+
// ],
93+
94+
// The directory (relative to the current directory) that benchmarks are
95+
// stored in. If not provided, defaults to "benchmarks"
96+
"benchmark_dir": "benchmarks",
97+
98+
// The directory (relative to the current directory) to cache the Python
99+
// environments in. If not provided, defaults to "env"
100+
"env_dir": "env",
101+
102+
// The directory (relative to the current directory) that raw benchmark
103+
// results are stored in. If not provided, defaults to "results".
104+
"results_dir": "results",
105+
106+
// The directory (relative to the current directory) that the html tree
107+
// should be written to. If not provided, defaults to "html".
108+
"html_dir": "html",
109+
110+
// The number of characters to retain in the commit hashes.
111+
// "hash_length": 8,
112+
113+
// `asv` will cache wheels of the recent builds in each
114+
// environment, making them faster to install next time. This is
115+
// number of builds to keep, per environment.
116+
// "wheel_cache_size": 0
117+
118+
// The commits after which the regression search in `asv publish`
119+
// should start looking for regressions. Dictionary whose keys are
120+
// regexps matching to benchmark names, and values corresponding to
121+
// the commit (exclusive) after which to start looking for
122+
// regressions. The default is to start from the first commit
123+
// with results. If the commit is `null`, regression detection is
124+
// skipped for the matching benchmark.
125+
//
126+
// "regressions_first_commits": {
127+
// "some_benchmark": "352cdf", // Consider regressions only after this commit
128+
// "another_benchmark": null, // Skip regression detection altogether
129+
// }
130+
131+
// The thresholds for relative change in results, after which `asv
132+
// publish` starts reporting regressions. Dictionary of the same
133+
// form as in ``regressions_first_commits``, with values
134+
// indicating the thresholds. If multiple entries match, the
135+
// maximum is taken. If no entry matches, the default is 5%.
136+
//
137+
// "regressions_thresholds": {
138+
// "some_benchmark": 0.01, // Threshold of 1%
139+
// "another_benchmark": 0.5, // Threshold of 50%
140+
// }
141+
}

‎benchmarking/benchmarks/__init__.py

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
"""
2+
This module contains benchmarks for QCoDeS. These benchmarks shall be executed
3+
by the `asv` (airspeed velocity) package.
4+
"""

0 commit comments

Comments
 (0)