Repository navigation
[Docker] Compile Python bytecode at image build time - #55422
Conversation
The runtime images install every Python package with uv, which does not compile bytecode unless asked, so vllm/vllm-openai ships 408 .pyc files for 27,753 .py files. A fresh container compiles torch, transformers, and vLLM bytecode on the first import and loses it with the overlay. Set UV_COMPILE_BYTECODE=1 in the runtime stage of each Dockerfile that installs with uv, so the shipped image carries the bytecode. Build-only stages are unchanged. Measured on the published image (n=3): the serve import takes 6.07 s in a fresh container and 2.15 s once bytecode exists; an image with bytecode precompiled takes 2.19 s on the first import. The bytecode adds about 156 MB compressed to an 8.6 GB image. Signed-off-by: Nils Matteson <nilsmatteson@icloud.com>
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Repository UI Review profile: CHILL Plan: Team Run ID: 📒 Files selected for processing (8)
Included review availability: Your plan provides up to 10 included reviews per hour; 9 remain after this review. WalkthroughThe Docker build stages now set ChangesDocker bytecode compilation
Estimated code review effort: 1 (Trivial) | ~5 minutes Merge Risk: ⚪ Minimal · up to This change moves Python bytecode compilation into image builds across supported Docker images, reducing startup work. No actionable current-head merge risk remains. 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
Can we isolate this without container as a test:
The experiment should be done to exactly figure out the effect of bytedance compile. Alternatively, if Python has a way to measure bytecode compilation time that could be helpful as well. |
|
I ran this without Docker, in a new virtual environment on my rig. For the cached/uncached comparison, I warmed the same 23,273 Python and shared-object files and checked 100% residency with Linux A separate serving-CLI import trace recorded zero source compilations with bytecode versus 5,118 without, with 8.20 s spent in compilation calls. That gives us evidence for a bytecode benefit with the installed source/native files already resident. I wouldn't attribute the entire ten seconds to compilation alone; cache I/O and reuse between processes are part of that result. This was Python 3.12.3/vLLM 0.28.0, four logical CPUs, cached Qwen3-0.6B, with normal compilation and CUDA graphs enabled. The fresh-install attempt hit a missing FlashInfer native prerequisite. After installing the matching official AOT package, another no-bytecode launch without explicit warming returned correct output at 59.44 s. Its cleanup check mistook |
|
✅ @matteso1, CI is now available for this PR.
|
Signed-off-by: Nils Matteson <nilsmatteson@icloud.com>
Head branch was pushed to by a user without write access
Signed-off-by: Nils Matteson <nilsmatteson@icloud.com>
…tage-20260923 Signed-off-by: Nils Matteson <nilsmatteson@icloud.com>
…tage-20260923 Signed-off-by: Nils Matteson <nilsmatteson@icloud.com>
Head branch was pushed to by a user without write access
|
/ci run |
|
✅ Triggered Buildkite CI #90765 for commit |
…p-20260923 Signed-off-by: Nils Matteson <nilsmatteson@icloud.com>
|
/ci run |
|
✅ Triggered Buildkite CI #90796 for commit |
Signed-off-by: Nils Matteson <nilsmatteson@icloud.com>
|
/ci run |
|
✅ Triggered Buildkite CI #90829 for commit |
Signed-off-by: Nils Matteson <nilsmatteson@icloud.com>
|
/ci run |
|
❌ This PR is 4 commits behind upstream |
…/bytecode-ft-20260924 Signed-off-by: Nils Matteson <nilsmatteson@icloud.com>
|
/ci run |
|
✅ Triggered Buildkite CI #91034 for commit |
|
/ci run |
|
❌ This PR is 7 commits behind upstream |
|
/ci run |
|
✅ Triggered Buildkite CI #92239 for commit |
Purpose
Fresh containers compile thousands of Python files during startup. This PR sets
UV_COMPILE_BYTECODE=1in eight Dockerfiles, moving that work into image construction through uv's existing option.Current CI (September 24)
CI 91034 is running on
da0a5f6ed074. The H100 fault-tolerance job now passes both tests with #58593 included. The nightly dependency check also passed its one retry, after SQLAlchemy's compatible wheel became available.The structured-output failure repeats with the same parameter and output on main 91027 and 91036. B200 Mooncake already has an automatic retry queued. AMD 13576 timed out after slow checkpoint loading; its later stall remains unexplained. That native job consumes a wheel built before the changed final-image stage. Other jobs are still running.
The diff remains eight ENV lines plus one blank line. Pre-commit and paired Linux CPU Dockerfile definition checks pass, with the same two existing warnings as main. The head is unchanged while this run finishes.
Results
Matched build, delivery, and serving
I built
470fe3942ecdb889b2f3c0b57ace4b09d61dd103with and without the CUDA Dockerfile ENV line (vllm-openai, CUDA sm_89, Python 3.12.3, Torch 2.13.0+cu130, uv 0.12.10). Source, interpreter, 244 Python distributions, 26,734 Python files, native payloads, and non-bytecode site-packages bytes matched. Each observation pulled from a loopback registry into an empty Docker store, then served local/offline Qwen3-0.6B through the first correct HTTP completion on one RTX 4090 Laptop with a four-CPU quota.The bytecode image adds 153,083,004 compressed bytes (+1.7985%). Its median paired saving is 15.3606 s, n=3 pairs, and all three pairs improved. Five independent next-token checks and four-request bursts passed per activation with matching choices and usage. Candidate import compiled zero source files; control import compiled 5,257. Bypassing candidate caches restored 5,478 compilations.
Native serving attribution
In a separate installed vLLM 0.28.0 experiment, the same 8.85 GB of Python and shared-object files were 100% page-resident before every start. First-correct-response medians were 57.0394 s without bytecode and 47.6911 s with it, with a 10.1300 s median paired saving, n=3 pairs; all pairs and output checks passed. The measured startup saving includes avoided compilation, cache-file work, and normal parent/child reuse. It is not entirely compiler CPU. Protocol, observations, and limits.
Direct compiler reproduction
The exact 5,118-file import corpus, 78.60 MB and about 2.16 million source lines, was read into memory and compiled without execution or bytecode writes. Native CPython 3.12.3 took 6.3033 s median, n=3 whole-corpus passes. A real uncached serving-CLI import took 12.2019 s with 6.9880 s inside 5,118 compiler calls; cached import took 5.0095 s and made zero compiler calls. Source hashes and 196 package versions match the resident-page study, but the Python binary build changed, so absolute times are not interchangeable. To reproduce without vLLM or a GPU, invoke
compile(source_bytes, filename, "exec", dont_inherit=True, optimize=0)over the recorded trace paths in a freshpython3.12 -I -S -Bprocess after preloading the sources.Inclusive import roots
Fresh-process medians below use fully resident source/native pages, n=3 paired runs per root. Rows include transitive imports and are not additive. Paired-saving medians need not equal differences of arm medians.
import torchimport transformersimport vllmimport vllm.entrypoints.cli.serveIn this vLLM 0.28.0 environment (Torch 2.13.0, Transformers 5.17.0), bare Torch did not load Dynamo, Inductor, or SymPy; importing vLLM loaded all three. Bare Transformers did not load Torch. One eager path is the Inductor patch in
vllm.env_override. Dependency-file ownership therefore does not tell us which entry point introduces the import cost.Bytecode portability
Pure-Python timestamp and checked-hash caches created on macOS ARM64, CPython 3.12.13 loaded unchanged on Linux x86-64, CPython 3.12.3 with source compilation blocked. Both used cache tag
cpython-312and magiccb0d0d0a; all nine reuse and validation cases matched their expected outcomes. Python documents compiled modules as architecture-independent, with compatible interpreter format still required. The image pins its Python/native platform; the PR compiles within that build path and retains source. This fixture is not a full cross-platform vLLM deployment.Known caveats
uv recommends bytecode compilation for production images, trading build work and image size for startup time. The upstream issues I found concern reproducible cache/image bytes and compilation hangs under affected QEMU-user setups. I reproduced the former on Mac CPython 3.12.13: identical source and checked-hash headers produced different serialized bytes depending on compilation order, but equal decoded code objects and correct execution. We should not claim reproducible image digests. The QEMU report concerns the build environment; native CUDA validation does not qualify emulated builds. No cache-format or concurrency workaround is added by this PR.
Secondary observations, reproduction details, and limits
Native serving pairs, without/with bytecode: 54.7488/44.6188 s, 57.0394/47.6911 s, and 60.6387/48.4054 s. Its separate September 9 CLI trace measured 8.1962 s in 5,118 source-compilation calls. The September 15 preloaded direct-compile passes were 6.2461, 6.3033 and 6.3194 s; their Python 3.12.3 build is newer, despite matching source hashes and package versions.
Separate diagnostic import-root traces measured compiler time/source files as Torch 1.3441 s/871, Transformers 0.6780 s/398, vLLM 4.7605 s/2,293 and serving CLI 7.5139 s/5,118. These hooked traces are excluded from the uninstrumented root medians above. The cached serving trace made zero compiler calls. Within the uncached CLI trace, exclusive compiler costs by source-file owner were Torch 2.852 s, SymPy 1.032 s, Transformers 0.498 s and vLLM 0.431 s, plus other dependencies. These are compiler timings, not inclusive package-import durations.
The earlier unchanged-runtime mechanism fixture used the published vLLM 0.28.0 image plus
python3 -m compileall -q -j 8 /usr/local/lib/python3.12/dist-packages. With image and model present, medians were 63.967 s control and 51.953 s bytecode, with a 12.014 s median paired saving, n=5. With the image absent and model present, loopback pull-through-response medians were 145.041 s and 134.109 s, with a 10.337 s median paired saving, n=3. Its added gzip layer was 153,159,438 bytes. This fixture isolates the mechanism but is not a matched full build of this PR.Exact-head paired pull-through-response observations were 149.8156/134.4550 s, 155.7265/144.5137 s, and 162.3850/146.2806 s for control/bytecode. Host source pages were warm or uncontrolled in that experiment. The same bytecode artifact exported with the separate opt-in zstd change was 17.7633% smaller than control and saved 37.7032 s median paired, but those combined numbers include both changes and are not this PR's isolated effect.
The full-build recipes shared one recorded host-network preparation because the rig lacks
docker0. The first control build failed in 1.32 s before installation; no timed cell was affected. Build durations, 688.83 s control and 629.16 s candidate, are not a matched build-time comparison. In the older delivery fixture, one post-timing mount cleanup failed after the clock and functional checks; that observation was retained without replacement and cleanup was resolved before continuing. The resident-page study retained two excluded preparation attempts and corrected a TCP cleanup probe before its six measured cells. The import-root study's first preparation reached 98.0101% residency and timed no imports; equal mmap page touches then produced 100% residency without relaxing the gate.These are one-host medians with local models, no Internet transfer, warm or controlled pages as stated, and no p95 or throughput claim. They do not validate missing-model startup, every accelerator image, published multi-platform size, slow links, fleets, pip installs, or the earlier A10/Qwen3-8B baseline. Normal source-backed loader validation has limits: a corrupt marshalled body can fail without source fallback, and timestamp caches can accept changed same-size/same-mtime source. uv initially compiled only the reinstalled vLLM distribution in the package study, so the ENV line alone does not prove complete cache coverage for every image variant.
Related to #48193. An open-PR title search for bytecode on September 15 returned only this PR. AI assistance was used for implementation, experiment tooling, and analysis.