bpo-36044: Reduce number of unit tests run for PGO build. #14702
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This speeds up the PGO profile generation by a factor of about 15X.
Running the full unit test suite is slow. This may result in a
slightly less optimized build since not as many code branches will
be executed. If you are willing to wait for the slower build, the
old behavior can be enabled as follows:
./configure [...] --with-profile-task='-m test --pgo'
Add --pgo-extended option to regrtest.
vstinner
reviewed
Jul 16, 2019
| @@ -264,7 +264,9 @@ def _create_parser(): | |||
| help='only write the name of test cases that will be run' | |||
| ' , don\'t execute them') | |||
| group.add_argument('-P', '--pgo', dest='pgo', action='store_true', | |||
| help='enable Profile Guided Optimization training') | |||
| help='enable Profile Guided Optimization (PGO) training') | |||
| group.add_argument('--pgo-extended', action='store_true', | |||
There was a problem hiding this comment.
I don't understand what "extended" means. I suggest to rename the option to "--pgo-all-tests" and rewrite the help to explain that it runs all tests by default.
There was a problem hiding this comment.
Think list.extend(). I.e. run more tests. I didn't want to use --pgo-all because we don't actually run all the tests. Some are excluded using the support.PGO flag. However, I don't feel strongly so if you really like --pgo-all, I will change it.
Lib/test/libregrtest/main.py
Outdated
| # chosen based on the following criteria: either they exercise a commonly used | ||
| # C extension module or type, or they run some relatively typical Python code. | ||
| # Long running tests should be avoided because the PGO instrumented executable | ||
| # runs slowly. |
There was a problem hiding this comment.
I would prefer to keep main.py small. Would you mind to create libregrtest/pgo.py for this list please?
more detail and a performance disclaimer.
gpshead
approved these changes
Jul 17, 2019
| help='enable Profile Guided Optimization training') | ||
| help='enable Profile Guided Optimization (PGO) training') | ||
| group.add_argument('--pgo-extended', action='store_true', | ||
| help='enable extended PGO training (slower training)') |
There was a problem hiding this comment.
I'd add to the description of this that "User are encouraged to benchmark the interpreter resulting from a PROFILE_TASK with this flag to decide if it is meaningfully faster than those produced using --pgo for their use cases".
I don't care what the flag name is, --pgo-extended is fine by me. as is --pgo-very-long-training-run (a flag who's length matches the build time seems... appropriate). ;)
The configure flags --with-something are intended for enabling optional external software. Just use a variable instead.
methane
approved these changes
Jul 20, 2019
|
Thanks @nascheme for the PR 🌮🎉.. I'm working now to backport this PR to: 3.8. |
miss-islington
pushed a commit
to miss-islington/cpython
that referenced
this pull request
Jul 22, 2019
) Reduce the number of unit tests run for the PGO generation task. This speeds up the task by a factor of about 15x. Running the full unit test suite is slow. This change may result in a slightly less optimized build since not as many code branches will be executed. If you are willing to wait for the much slower build, the old behavior can be restored using './configure [..] PROFILE_TASK="-m test --pgo-extended"'. We make no guarantees as to which PGO task set produces a faster build. Users who care should run their own relevant benchmarks as results can depend on the environment, workload, and compiler tool chain. (cherry picked from commit 4e16a4a) Co-authored-by: Neil Schemenauer <[email protected]>
|
GH-14910 is a backport of this pull request to the 3.8 branch. |
miss-islington
added a commit
that referenced
this pull request
Jul 22, 2019
Reduce the number of unit tests run for the PGO generation task. This speeds up the task by a factor of about 15x. Running the full unit test suite is slow. This change may result in a slightly less optimized build since not as many code branches will be executed. If you are willing to wait for the much slower build, the old behavior can be restored using './configure [..] PROFILE_TASK="-m test --pgo-extended"'. We make no guarantees as to which PGO task set produces a faster build. Users who care should run their own relevant benchmarks as results can depend on the environment, workload, and compiler tool chain. (cherry picked from commit 4e16a4a) Co-authored-by: Neil Schemenauer <[email protected]>
|
@nascheme It seems that only 40 tests are run on Travis after this change (for example, https://travis-ci.org/python/cpython/jobs/562362678) instead of the entire test suite. Please take a look at this issue. |
lisroach
pushed a commit
to lisroach/cpython
that referenced
this pull request
Sep 10, 2019
) Reduce the number of unit tests run for the PGO generation task. This speeds up the task by a factor of about 15x. Running the full unit test suite is slow. This change may result in a slightly less optimized build since not as many code branches will be executed. If you are willing to wait for the much slower build, the old behavior can be restored using './configure [..] PROFILE_TASK="-m test --pgo-extended"'. We make no guarantees as to which PGO task set produces a faster build. Users who care should run their own relevant benchmarks as results can depend on the environment, workload, and compiler tool chain.
DinoV
pushed a commit
to DinoV/cpython
that referenced
this pull request
Jan 14, 2020
) Reduce the number of unit tests run for the PGO generation task. This speeds up the task by a factor of about 15x. Running the full unit test suite is slow. This change may result in a slightly less optimized build since not as many code branches will be executed. If you are willing to wait for the much slower build, the old behavior can be restored using './configure [..] PROFILE_TASK="-m test --pgo-extended"'. We make no guarantees as to which PGO task set produces a faster build. Users who care should run their own relevant benchmarks as results can depend on the environment, workload, and compiler tool chain.
websurfer5
pushed a commit
to websurfer5/cpython
that referenced
this pull request
Jul 20, 2020
) Reduce the number of unit tests run for the PGO generation task. This speeds up the task by a factor of about 15x. Running the full unit test suite is slow. This change may result in a slightly less optimized build since not as many code branches will be executed. If you are willing to wait for the much slower build, the old behavior can be restored using './configure [..] PROFILE_TASK="-m test --pgo-extended"'. We make no guarantees as to which PGO task set produces a faster build. Users who care should run their own relevant benchmarks as results can depend on the environment, workload, and compiler tool chain.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This speeds up the PGO profile generation by a factor of about 15X.
Running the full unit test suite is slow. This may result in a
slightly less optimized build since not as many code branches will
be executed. If you are willing to wait for the slower build, the
old behavior can be enabled as follows:
https://bugs.python.org/issue36044