Freeze all modules imported during startup.

[ericsnowcurrently]

BPO	45020
Nosy	@malemburg, @gvanrossum, @warsaw, @brettcannon, @nascheme, @rhettinger, @terryjreedy, @gpshead, @ronaldoussoren, @ncoghlan, @vstinner, @larryhastings, @tiran, @tiran, @tiran, @methane, @markshannon, @ericsnowcurrently, @indygreg, @lysnikolaou, @pablogsal, @miss-islington, @brandtbucher, @isidentical, @shihai1991, @FFY00, @softsol solutions
PRs	bpo-45020: Freeze the modules imported during startup. #28107 bpo-45020: Add -X frozen_modules=[on|off] to explicitly control use of frozen modules. #28320 bpo-45020: Freeze some of the modules imported during startup. #28335 bpo-45020: Don't test IDLE with frozen module. #28344 [3.10] bpo-45020: Don't test IDLE with frozen module. (GH-28344) #28345 [3.9] bpo-45020: Don't test IDLE with frozen module. (GH-28344) #28346 bpo-45020: Drop the frozen .h files from the repo. #28375 bpo-45020: Revert "Drop the frozen .h files from the repo. (gh-28375)" #28380 bpo-45020: Drop the frozen .h files from the repo. #28392 bpo-45020: Freeze os, site, and codecs. #28398 bpo-45020: Fix build out of source tree #28410 bpo-45020: Fix some corner cases for frozen module generation. #28538 bpo-45020: Make "make all" output less noisy. #28554 bpo-45020: Use a common recipe for freezing modules. #28583 bpo-45020: Default to using frozen modules only on non-debug builds. #28590 bpo-45020: Do not freeze <pkg>/__init__.py twice. #28635 bpo-45020: Add more test cases for frozen modules. #28664 Also install __phello__, in addition to __hello__. #28665 bpo-45020: Identify which frozen modules are actually aliases. #28655 bpo-45020: Default to using frozen modules unless running from source tree. #28940 bpo-45020: Add tests for the -X "frozen_modules" option. #28997 bpo-45020: Fix strict-prototypes warning (GH-29755) #29755 bpo-45020: Fix strict-prototypes warning (GH-29755) #29755 bpo-45020: Fix strict-prototypes warning (GH-29755) #29755
Dependencies	bpo-45186: Marshal output isn't completely deterministic. bpo-45188: De-couple the Windows builds from freezing modules.

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/ericsnowcurrently'
closed_at = <Date 2021-10-28.19:52:39.815>
created_at = <Date 2021-08-26.19:23:58.261>
labels = ['interpreter-core', 'type-feature', '3.11']
title = 'Freeze all modules imported during startup.'
updated_at = <Date 2021-11-24.19:01:47.807>
user = 'https://github.com/ericsnowcurrently'

bugs.python.org fields:

activity = <Date 2021-11-24.19:01:47.807>
actor = 'christian.heimes'
assignee = 'eric.snow'
closed = True
closed_date = <Date 2021-10-28.19:52:39.815>
closer = 'eric.snow'
components = ['Parser']
creation = <Date 2021-08-26.19:23:58.261>
creator = 'eric.snow'
dependencies = ['45186', '45188']
files = []
hgrepos = []
issue_num = 45020
keywords = ['patch']
message_count = 96.0
messages = ['400370', '400371', '400372', '400373', '400374', '400381', '400386', '400394', '400421', '400422', '400427', '400447', '400449', '400450', '400454', '400455', '400459', '400460', '400461', '400469', '400507', '400508', '400510', '400629', '400636', '400638', '400639', '400660', '400664', '400665', '400666', '400667', '400675', '400766', '400769', '400808', '400855', '400856', '401027', '401040', '401734', '401740', '401805', '401811', '401813', '401814', '401848', '401865', '401870', '401905', '401911', '401917', '401918', '401919', '401921', '401954', '401966', '401988', '402017', '402020', '402070', '402080', '402100', '402101', '402103', '402113', '402116', '402118', '402119', '402151', '402356', '402463', '402464', '402476', '402587', '402609', '402629', '402633', '402634', '402898', '402993', '403024', '403025', '403255', '403256', '403323', '403324', '404111', '404164', '404257', '404344', '405002', '405203', '405244', '406143', '406948']
nosy_count = 25.0
nosy_names = ['lemburg', 'gvanrossum', 'barry', 'brett.cannon', 'nascheme', 'rhettinger', 'terry.reedy', 'gregory.p.smith', 'ronaldoussoren', 'ncoghlan', 'vstinner', 'larry', 'christian.heimes', 'christian.heimes', 'christian.heimes', 'methane', 'Mark.Shannon', 'eric.snow', 'indygreg', 'lys.nikolaou', 'pablogsal', 'miss-islington', 'brandtbucher', 'BTaskaya', 'shihai1991', 'FFY00', 'santhu_reddy12']
pr_nums = ['28107', '28320', '28335', '28344', '28345', '28346', '28375', '28380', '28392', '28398', '28410', '28538', '28554', '28583', '28590', '28635', '28664', '28665', '28655', '28940', '28997', '29755', '29755', '29755']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue45020'
versions = ['Python 3.11']

[ericsnowcurrently]

Currently we freeze the 3 main import-related modules into the python binary (along with one test module). This allows us to bootstrap the import machinery from Python modules. It also means we get better performance importing those modules.

If we freeze modules that are likely to be used during execution then we get even better startup times. I'll be putting up a PR that does so, freezing all the modules that are imported during startup. This could also be done for any stdlib modules that are commonly imported.

(also see bpo-45019 and faster-cpython/ideas#82)

[ericsnowcurrently]

FYI, with my branch I'm getting a 15% improvement to startup for "./python -c pass".

[ericsnowcurrently]

I'm aware of two potentially problematic consequences to this change:

• making changes to those source files for the modules will not be reflected during execution until after "make" is run
• tricks to inject hooks ASAP (e.g. coverage.py swaps the encodings module) may lose their entry point

For the former, I'm not sure there's a way around it. We may consider the inconvenience worth it in order to get the performance benefits.

For the latter, the obvious solution is to introduce a startup hook (e.g. on the CLI) like we've talked about doing for years. (I wasn't able to find previous discussions on that topic after a quick search.)

[malemburg]

Not sure whether you are aware, but the PyRun project I'm maintaining
does that and goes beyond this by freezing almost the complete stdlib
and statically linking most C extensions into a single binary:

https://www.egenix.com/products/python/PyRun/

Startup is indeed better, but not as much as you might think.
You do save stat calls and can share resources across processes.

The big time consumer is turning marshal'ed code objects back
into Python objects, though. If that could be made faster by
e.g. using a more efficient storage format such as one which is
platform dependent, it'd be a much bigger win than the freezing
approach.

[ericsnowcurrently]

The big time consumer is turning marshal'ed code objects back
into Python objects, though. If that could be made faster by
e.g. using a more efficient storage format such as one which is
platform dependent, it'd be a much bigger win than the freezing
approach.

That's something Guido has been exploring. :)

See: faster-cpython/ideas#84 (and others)

[ericsnowcurrently]

For the latter, the obvious solution is to introduce a startup hook

I'm not sure why I said "obvious". Sorry about that.

[gvanrossum]

I noticed nedbat un-nosied himself. Probably he didn't realize you were
calling him out because it's possible this would affect coverage.py?