bpo-46576: bpo-46524: Disable compiler optimization within test_peg_generator. #31015

gpshead · 2022-01-29T20:40:20Z

This speed up test_peg_generator by always disabling compiler
optimizations by using -O0 by default when the test is building its
own C extensions.

A build not using --with-pydebug in order to speed up test execution
winds up with this test taking a very long time as it would do
repeated compilation of parser C code using the same optimization
flags as CPython was built with.

This speeds the test up 6-8x on gps-raspbian.

Includes the win32 changes from #31017 by Kumar Aditya

https://bugs.python.org/issue46576

This speed up test_peg_generator by always disabling compiler optimizations by using -O0 by default when the test is building its own C extensions. A build not using --with-pydebug in order to speed up test execution winds up with this test taking a very long time as it would do repeated compilation of parser C code using the same optimization flags as CPython was built with. This speeds the test up 6-8x on gps-raspbian.

sweeneyde · 2022-01-30T05:54:13Z

Windows tests show:

cl : Command line warning D9025 : overriding '/DNDEBUG' with '/UNDEBUG'
cl : Command line warning D9002 : ignoring unknown option '-O0'

Is there a corresponding option for msvc?

kumaraditya303 · 2022-01-30T06:20:42Z

MSVC has /Od to speed up compilation, See https://docs.microsoft.com/en-us/cpp/build/reference/od-disable-debug?view=msvc-170

Tools/peg_generator/pegen/build.py

Co-authored-by: Christian Heimes <[email protected]>

bedevere-bot · 2022-01-31T06:27:38Z

🤖 New build scheduled with the buildbot fleet by @gpshead for commit c4457db 🤖

If you want to schedule another build, you need to add the ":hammer: test-with-buildbots" label again.

bedevere-bot · 2022-02-01T19:08:15Z

🤖 New build scheduled with the buildbot fleet by @pablogsal for commit c4457db 🤖

If you want to schedule another build, you need to add the ":hammer: test-with-buildbots" label again.

gpshead · 2022-02-01T22:25:30Z

I believe I need to rebase my PR branch before buildbottest is useful.

bedevere-bot · 2022-02-01T22:29:37Z

🤖 New build scheduled with the buildbot fleet by @gpshead for commit 5b99059 🤖

If you want to schedule another build, you need to add the ":hammer: test-with-buildbots" label again.

miss-islington · 2022-02-02T20:15:18Z

Thanks @gpshead for the PR 🌮🎉.. I'm working now to backport this PR to: 3.9, 3.10.
🐍🍒⛏🤖

bedevere-bot · 2022-02-02T20:15:19Z

@gpshead: Please replace # with GH- in the commit message next time. Thanks!

miss-islington · 2022-02-02T20:15:23Z

Sorry, @gpshead, I could not cleanly backport this to 3.10 due to a conflict.
Please backport using cherry_picker on command line.
cherry_picker 164a017e13ee96bd1ea1ae79f5ac9e25fe83994e 3.10

miss-islington · 2022-02-02T20:15:25Z

Sorry @gpshead, I had trouble checking out the 3.9 backport branch.
Please backport using cherry_picker on command line.
cherry_picker 164a017e13ee96bd1ea1ae79f5ac9e25fe83994e 3.9

…t_peg_generator. (pythonGH-31015) Disable compiler optimization within test_peg_generator. This speed up test_peg_generator by always disabling compiler optimizations by using -O0 or equivalent when the test is building its own C extensions. A build not using --with-pydebug in order to speed up test execution winds up with this test taking a very long time as it would do repeated compilation of parser C code using the same optimization flags as CPython was built with. This speeds the test up 6-8x on gps-raspbian. Also incorporate's pythonGH-31017's win32 conditional and flags. Co-authored-by: Kumar Aditya kumaraditya303. (cherry picked from commit 164a017) Co-authored-by: Gregory P. Smith <[email protected]>

vstinner · 2022-02-02T22:33:32Z

In the Windows (x64) job, test_peg_generator only took 3 min 40 sec: it's way better than 8 minutes, yeah!

…t_peg_generator. (GH-31015) (GH-31089) Disable compiler optimization within test_peg_generator. This speed up test_peg_generator by always disabling compiler optimizations by using -O0 or equivalent when the test is building its own C extensions. A build not using --with-pydebug in order to speed up test execution winds up with this test taking a very long time as it would do repeated compilation of parser C code using the same optimization flags as CPython was built with. This speeds the test up 6-8x on gps-raspbian. Also incorporate's GH-31017's win32 conditional and flags. Co-authored-by: Kumar Aditya kumaraditya303. (cherry picked from commit 164a017) Co-authored-by: Gregory P. Smith <[email protected]>

…t_peg_generator. (pythonGH-31015) (pythonGH-31089) Disable compiler optimization within test_peg_generator. This speed up test_peg_generator by always disabling compiler optimizations by using -O0 or equivalent when the test is building its own C extensions. A build not using --with-pydebug in order to speed up test execution winds up with this test taking a very long time as it would do repeated compilation of parser C code using the same optimization flags as CPython was built with. This speeds the test up 6-8x on gps-raspbian. Also incorporate's pythonGH-31017's win32 conditional and flags. Co-authored-by: Kumar Aditya kumaraditya303. (cherry picked from commit 164a017) Co-authored-by: Gregory P. Smith <[email protected]> (cherry picked from commit f5ebec4) Co-authored-by: Gregory P. Smith <[email protected]>

…in test_peg_generator. (GH-31015) (GH-31089) (GH-31093) Disable compiler optimization within test_peg_generator. This speed up test_peg_generator by always disabling compiler optimizations by using -O0 or equivalent when the test is building its own C extensions. A build not using --with-pydebug in order to speed up test execution winds up with this test taking a very long time as it would do repeated compilation of parser C code using the same optimization flags as CPython was built with. This speeds the test up 6-8x on gps-raspbian. Also incorporate's GH-31017's win32 conditional and flags. Co-authored-by: Kumar Aditya kumaraditya303. (cherry picked from commit 164a017) Co-authored-by: Gregory P. Smith <[email protected]> (cherry picked from commit f5ebec4) Co-authored-by: Gregory P. Smith <[email protected]> Automerge-Triggered-By: GH:gpshead

gpshead · 2022-02-03T08:38:13Z

In the Windows (x64) job, test_peg_generator only took 3 min 40 sec: it's way better than 8 minutes, yeah!

yeah. though i also saw it take both 16 minutes or 5 minutes in similar CI runs on other PRs prior to this... so I can't make any judgement call. it certainly should not be worse. I hope it trends better.

…in test_peg_generator. (pythonGH-31015) (pythonGH-31089) (pythonGH-31093) Disable compiler optimization within test_peg_generator. This speed up test_peg_generator by always disabling compiler optimizations by using -O0 or equivalent when the test is building its own C extensions. A build not using --with-pydebug in order to speed up test execution winds up with this test taking a very long time as it would do repeated compilation of parser C code using the same optimization flags as CPython was built with. This speeds the test up 6-8x on gps-raspbian. Also incorporate's pythonGH-31017's win32 conditional and flags. Co-authored-by: Kumar Aditya kumaraditya303. (cherry picked from commit 164a017) Co-authored-by: Gregory P. Smith <[email protected]> (cherry picked from commit f5ebec4) Co-authored-by: Gregory P. Smith <[email protected]> Automerge-Triggered-By: GH:gpshead

gpshead added tests Tests in the Lib/test dir needs backport to 3.9 only security fixes needs backport to 3.10 only security fixes labels Jan 29, 2022

gpshead requested a review from pablogsal January 29, 2022 20:40

gpshead requested a review from lysnikolaou as a code owner January 29, 2022 20:40

the-knights-who-say-ni added the CLA signed label Jan 29, 2022

bedevere-bot added the awaiting core review label Jan 29, 2022

gpshead assigned pablogsal Jan 29, 2022

pablogsal approved these changes Jan 29, 2022

View reviewed changes

bedevere-bot added awaiting merge and removed awaiting core review labels Jan 29, 2022

gpshead commented Jan 30, 2022

View reviewed changes

Tools/peg_generator/pegen/build.py Outdated Show resolved Hide resolved

gpshead assigned gpshead and unassigned gpshead and pablogsal Jan 30, 2022

gpshead added 2 commits January 30, 2022 00:40

incorporate python#31017's win32 conditional and flags

36044f3

reword news

42b989c

gpshead changed the title ~~bpo-46576: Disable compiler optimization within test_peg_generator.~~ bpo-46576: bpo-46524: Disable compiler optimization within test_peg_generator. Jan 30, 2022

tiran reviewed Jan 30, 2022

View reviewed changes

Tools/peg_generator/pegen/build.py Show resolved Hide resolved

Add -fno-lto

c4457db

Co-authored-by: Christian Heimes <[email protected]>

gpshead added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Jan 31, 2022

bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Jan 31, 2022

pablogsal added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Feb 1, 2022

bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Feb 1, 2022

Merge branch 'python:main' into faster_test_pegen

5b99059

gpshead added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Feb 1, 2022

bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Feb 1, 2022

gpshead merged commit 164a017 into python:main Feb 2, 2022

bedevere-bot removed the awaiting merge label Feb 2, 2022

gpshead deleted the faster_test_pegen branch February 2, 2022 20:15

gpshead mentioned this pull request Feb 2, 2022

bpo-46524: speed up compilation of peg_generator on windows #31017

Closed

gpshead mentioned this pull request Feb 2, 2022

[3.10] bpo-46576: bpo-46524: Disable compiler optimization within test_peg_generator #31089

Merged

gpshead removed needs backport to 3.9 only security fixes needs backport to 3.10 only security fixes labels Feb 3, 2022

Uh oh!

bpo-46576: bpo-46524: Disable compiler optimization within test_peg_generator. #31015

bpo-46576: bpo-46524: Disable compiler optimization within test_peg_generator. #31015

Uh oh!

Conversation

gpshead commented Jan 29, 2022 • edited by bedevere-bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sweeneyde commented Jan 30, 2022

Uh oh!

kumaraditya303 commented Jan 30, 2022

Uh oh!

Uh oh!

Uh oh!

bedevere-bot commented Jan 31, 2022

Uh oh!

bedevere-bot commented Feb 1, 2022

Uh oh!

gpshead commented Feb 1, 2022

Uh oh!

bedevere-bot commented Feb 1, 2022

Uh oh!

miss-islington commented Feb 2, 2022

Uh oh!

bedevere-bot commented Feb 2, 2022

Uh oh!

miss-islington commented Feb 2, 2022

Uh oh!

miss-islington commented Feb 2, 2022

Uh oh!

vstinner commented Feb 2, 2022

Uh oh!

gpshead commented Feb 3, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

gpshead commented Jan 29, 2022 •

edited by bedevere-bot

Loading