Skip to content

[Docker] Compile Python bytecode at image build time - #55422

Merged
simon-mo merged 31 commits into
vllm-project:mainfrom
matteso1:image-bytecode
Oct 1, 2026
Merged

simon-mo merged 31 commits into
vllm-project:mainfrom
matteso1:image-bytecode

Conversation

@matteso1

@matteso1 matteso1 commented Sep 5, 2026 •

Copy link
Copy Markdown
Contributor

Purpose

Fresh containers compile thousands of Python files during startup. This PR sets UV_COMPILE_BYTECODE=1 in eight Dockerfiles, moving that work into image construction through uv's existing option.

Current CI (September 24)

CI 91034 is running on da0a5f6ed074. The H100 fault-tolerance job now passes both tests with #58593 included. The nightly dependency check also passed its one retry, after SQLAlchemy's compatible wheel became available.

The structured-output failure repeats with the same parameter and output on main 91027 and 91036. B200 Mooncake already has an automatic retry queued. AMD 13576 timed out after slow checkpoint loading; its later stall remains unexplained. That native job consumes a wheel built before the changed final-image stage. Other jobs are still running.

The diff remains eight ENV lines plus one blank line. Pre-commit and paired Linux CPU Dockerfile definition checks pass, with the same two existing warnings as main. The head is unchanged while this run finishes.

Results

Matched build, delivery, and serving

I built 470fe3942ecdb889b2f3c0b57ace4b09d61dd103 with and without the CUDA Dockerfile ENV line (vllm-openai, CUDA sm_89, Python 3.12.3, Torch 2.13.0+cu130, uv 0.12.10). Source, interpreter, 244 Python distributions, 26,734 Python files, native payloads, and non-bytecode site-packages bytes matched. Each observation pulled from a loopback registry into an empty Docker store, then served local/offline Qwen3-0.6B through the first correct HTTP completion on one RTX 4090 Laptop with a four-CPU quota.

Artifact Compressed runtime layers Median pull to correct response
Control, gzip 8,511,741,380 bytes 155.7265 s
Bytecode, gzip 8,664,824,384 bytes 144.5137 s

The bytecode image adds 153,083,004 compressed bytes (+1.7985%). Its median paired saving is 15.3606 s, n=3 pairs, and all three pairs improved. Five independent next-token checks and four-request bursts passed per activation with matching choices and usage. Candidate import compiled zero source files; control import compiled 5,257. Bypassing candidate caches restored 5,478 compilations.

Native serving attribution

In a separate installed vLLM 0.28.0 experiment, the same 8.85 GB of Python and shared-object files were 100% page-resident before every start. First-correct-response medians were 57.0394 s without bytecode and 47.6911 s with it, with a 10.1300 s median paired saving, n=3 pairs; all pairs and output checks passed. The measured startup saving includes avoided compilation, cache-file work, and normal parent/child reuse. It is not entirely compiler CPU. Protocol, observations, and limits.

Direct compiler reproduction

The exact 5,118-file import corpus, 78.60 MB and about 2.16 million source lines, was read into memory and compiled without execution or bytecode writes. Native CPython 3.12.3 took 6.3033 s median, n=3 whole-corpus passes. A real uncached serving-CLI import took 12.2019 s with 6.9880 s inside 5,118 compiler calls; cached import took 5.0095 s and made zero compiler calls. Source hashes and 196 package versions match the resident-page study, but the Python binary build changed, so absolute times are not interchangeable. To reproduce without vLLM or a GPU, invoke compile(source_bytes, filename, "exec", dont_inherit=True, optimize=0) over the recorded trace paths in a fresh python3.12 -I -S -B process after preloading the sources.

Inclusive import roots

Fresh-process medians below use fully resident source/native pages, n=3 paired runs per root. Rows include transitive imports and are not additive. Paired-saving medians need not equal differences of arm medians.

Entry point No bytecode Bytecode Median paired saving
import torch 2.1306 s 0.8025 s 1.3281 s
import transformers 1.2943 s 0.6760 s 0.6018 s
import vllm 6.7075 s 2.1068 s 4.6007 s
import vllm.entrypoints.cli.serve 12.2659 s 5.0198 s 7.2461 s

In this vLLM 0.28.0 environment (Torch 2.13.0, Transformers 5.17.0), bare Torch did not load Dynamo, Inductor, or SymPy; importing vLLM loaded all three. Bare Transformers did not load Torch. One eager path is the Inductor patch in vllm.env_override. Dependency-file ownership therefore does not tell us which entry point introduces the import cost.

Bytecode portability

Pure-Python timestamp and checked-hash caches created on macOS ARM64, CPython 3.12.13 loaded unchanged on Linux x86-64, CPython 3.12.3 with source compilation blocked. Both used cache tag cpython-312 and magic cb0d0d0a; all nine reuse and validation cases matched their expected outcomes. Python documents compiled modules as architecture-independent, with compatible interpreter format still required. The image pins its Python/native platform; the PR compiles within that build path and retains source. This fixture is not a full cross-platform vLLM deployment.

Known caveats

uv recommends bytecode compilation for production images, trading build work and image size for startup time. The upstream issues I found concern reproducible cache/image bytes and compilation hangs under affected QEMU-user setups. I reproduced the former on Mac CPython 3.12.13: identical source and checked-hash headers produced different serialized bytes depending on compilation order, but equal decoded code objects and correct execution. We should not claim reproducible image digests. The QEMU report concerns the build environment; native CUDA validation does not qualify emulated builds. No cache-format or concurrency workaround is added by this PR.

Secondary observations, reproduction details, and limits

Native serving pairs, without/with bytecode: 54.7488/44.6188 s, 57.0394/47.6911 s, and 60.6387/48.4054 s. Its separate September 9 CLI trace measured 8.1962 s in 5,118 source-compilation calls. The September 15 preloaded direct-compile passes were 6.2461, 6.3033 and 6.3194 s; their Python 3.12.3 build is newer, despite matching source hashes and package versions.

Separate diagnostic import-root traces measured compiler time/source files as Torch 1.3441 s/871, Transformers 0.6780 s/398, vLLM 4.7605 s/2,293 and serving CLI 7.5139 s/5,118. These hooked traces are excluded from the uninstrumented root medians above. The cached serving trace made zero compiler calls. Within the uncached CLI trace, exclusive compiler costs by source-file owner were Torch 2.852 s, SymPy 1.032 s, Transformers 0.498 s and vLLM 0.431 s, plus other dependencies. These are compiler timings, not inclusive package-import durations.

The earlier unchanged-runtime mechanism fixture used the published vLLM 0.28.0 image plus python3 -m compileall -q -j 8 /usr/local/lib/python3.12/dist-packages. With image and model present, medians were 63.967 s control and 51.953 s bytecode, with a 12.014 s median paired saving, n=5. With the image absent and model present, loopback pull-through-response medians were 145.041 s and 134.109 s, with a 10.337 s median paired saving, n=3. Its added gzip layer was 153,159,438 bytes. This fixture isolates the mechanism but is not a matched full build of this PR.

Exact-head paired pull-through-response observations were 149.8156/134.4550 s, 155.7265/144.5137 s, and 162.3850/146.2806 s for control/bytecode. Host source pages were warm or uncontrolled in that experiment. The same bytecode artifact exported with the separate opt-in zstd change was 17.7633% smaller than control and saved 37.7032 s median paired, but those combined numbers include both changes and are not this PR's isolated effect.

The full-build recipes shared one recorded host-network preparation because the rig lacks docker0. The first control build failed in 1.32 s before installation; no timed cell was affected. Build durations, 688.83 s control and 629.16 s candidate, are not a matched build-time comparison. In the older delivery fixture, one post-timing mount cleanup failed after the clock and functional checks; that observation was retained without replacement and cleanup was resolved before continuing. The resident-page study retained two excluded preparation attempts and corrected a TCP cleanup probe before its six measured cells. The import-root study's first preparation reached 98.0101% residency and timed no imports; equal mmap page touches then produced 100% residency without relaxing the gate.

These are one-host medians with local models, no Internet transfer, warm or controlled pages as stated, and no p95 or throughput claim. They do not validate missing-model startup, every accelerator image, published multi-platform size, slow links, fleets, pip installs, or the earlier A10/Qwen3-8B baseline. Normal source-backed loader validation has limits: a corrupt marshalled body can fail without source fallback, and timestamp caches can accept changed same-size/same-mtime source. uv initially compiled only the reinstalled vLLM distribution in the package study, so the ENV line alone does not prove complete cache coverage for every image variant.

Related to #48193. An open-PR title search for bytecode on September 15 returned only this PR. AI assistance was used for implementation, experiment tooling, and analysis.

The runtime images install every Python package with uv, which does not
compile bytecode unless asked, so vllm/vllm-openai ships 408 .pyc files for
27,753 .py files. A fresh container compiles torch, transformers, and vLLM
bytecode on the first import and loses it with the overlay. Set
UV_COMPILE_BYTECODE=1 in the runtime stage of each Dockerfile that installs
with uv, so the shipped image carries the bytecode. Build-only stages are
unchanged.

Measured on the published image (n=3): the serve import takes 6.07 s in a
fresh container and 2.15 s once bytecode exists; an image with bytecode
precompiled takes 2.19 s on the first import. The bytecode adds about
156 MB compressed to an 8.6 GB image.

Signed-off-by: Nils Matteson <nilsmatteson@icloud.com>

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

@mergify mergify Bot added ci/build rocm Related to AMD ROCm labels Sep 5, 2026
@mergify mergify Bot added the intel-gpu Related to Intel GPU label Sep 5, 2026
@github-project-automation github-project-automation Bot moved this to Todo in AMD Sep 5, 2026
@mergify mergify Bot added the cpu Related to CPU backends label Sep 5, 2026
@coderabbitai

coderabbitai Bot commented Sep 5, 2026 •

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Team

Run ID: 8292874f-1346-4ca3-afcd-77945da5617e

📥 Commits

Reviewing files that changed from the base of the PR and between 6cbb3c1 and 470fe39.

📒 Files selected for processing (8)
  • docker/Dockerfile
  • docker/Dockerfile.cpu
  • docker/Dockerfile.ppc64le
  • docker/Dockerfile.rock
  • docker/Dockerfile.rocm
  • docker/Dockerfile.rocm_gfx1250
  • docker/Dockerfile.s390x
  • docker/Dockerfile.xpu

Included review availability: Your plan provides up to 10 included reviews per hour; 9 remain after this review.


Walkthrough

The Docker build stages now set UV_COMPILE_BYTECODE=1. This enables uv to compile Python bytecode during package installation across the standard, CPU, accelerator, and architecture-specific images.

Changes

Docker bytecode compilation

Layer / File(s) Summary
Docker build environment
docker/Dockerfile, docker/Dockerfile.*
The relevant build and runtime stages set UV_COMPILE_BYTECODE=1 for uv package installations.

Estimated code review effort: 1 (Trivial) | ~5 minutes

Merge Risk: ⚪ Minimal · up to 470fe

This change moves Python bytecode compilation into image builds across supported Docker images, reducing startup work. No actionable current-head merge risk remains.

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 0…
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Title check ✅ Passed The title clearly summarizes the main change: compiling Python bytecode during Docker image construction.
Description check ✅ Passed The description directly explains the Dockerfile changes, expected startup benefits, image-size tradeoff, testing, and known limitations.

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@simon-mo

simon-mo commented Sep 9, 2026 •

Copy link
Copy Markdown
Collaborator

Can we isolate this without container as a test:

  • vllm serve with a new installation
  • vllm serve with bytecode cached
  • vllm serve without bytecode cached
    The reason to do the third experiment is to test the effect of the Python and so files in OS page cache, which can be a confounding factor.

The experiment should be done to exactly figure out the effect of bytedance compile. Alternatively, if Python has a way to measure bytecode compilation time that could be helpful as well.

@matteso1

matteso1 commented Sep 9, 2026

Copy link
Copy Markdown
Contributor Author

I ran this without Docker, in a new virtual environment on my rig. For the cached/uncached comparison, I warmed the same 23,273 Python and shared-object files and checked 100% residency with Linux mincore before every start. First-correct-response medians were 47.69 s with bytecode and 57.04 s without. The median paired saving was 10.13 s across three pairs; all three were positive and all six output/cleanup checks passed. Bytecode writes stayed enabled, so the engine children could reuse caches written by the parent.

A separate serving-CLI import trace recorded zero source compilations with bytecode versus 5,118 without, with 8.20 s spent in compilation calls. That gives us evidence for a bytecode benefit with the installed source/native files already resident. I wouldn't attribute the entire ten seconds to compilation alone; cache I/O and reuse between processes are part of that result. This was Python 3.12.3/vLLM 0.28.0, four logical CPUs, cached Qwen3-0.6B, with normal compilation and CUDA graphs enabled.

The fresh-install attempt hit a missing FlashInfer native prerequisite. After installing the matching official AOT package, another no-bytecode launch without explicit warming returned correct output at 59.44 s. Its cleanup check mistook TIME_WAIT for an active listener; I verified there was no listener and corrected that check before the paired runs. Both observations are retained, but neither is part of the paired comparison or evidence of an OS-cold baseline. The PR code and earlier image results are unchanged.

@simon-mo
simon-mo enabled auto-merge (squash) September 16, 2026 17:19
@simon-mo simon-mo added the ready ONLY add when PR is ready to merge/full CI is needed label Sep 16, 2026
@github-actions

Copy link
Copy Markdown

✅ @matteso1, CI is now available for this PR.

  • /ci run starts upstream CI; /amd-ci run starts AMD CI only.
  • Your branch must contain every commit currently on its upstream target branch. Merge or rebase onto the latest target branch, then rerun the command. Append --allow-stale to a run command to test an outdated branch at your own risk.
  • /ci retry retries failed jobs in the CI build for the current PR head. If the current head has no CI build, it starts a new CI build for the current head containing only jobs that failed in the latest earlier CI build for this PR.
  • /amd-ci retry retries failed jobs in AMD CI for the current PR head. Use /amd-ci run when the current head has no AMD CI build.
  • /ci cancel cancels scheduled or running CI builds for this PR branch; /amd-ci cancel does the same for AMD CI only.

Signed-off-by: Nils Matteson <nilsmatteson@icloud.com>
auto-merge was automatically disabled September 16, 2026 18:42

Head branch was pushed to by a user without write access

@simon-mo
simon-mo enabled auto-merge (squash) September 23, 2026 17:09
Signed-off-by: Nils Matteson <nilsmatteson@icloud.com>
…tage-20260923

Signed-off-by: Nils Matteson <nilsmatteson@icloud.com>
…tage-20260923

Signed-off-by: Nils Matteson <nilsmatteson@icloud.com>
auto-merge was automatically disabled September 23, 2026 20:37

Head branch was pushed to by a user without write access

@matteso1

Copy link
Copy Markdown
Contributor Author

/ci run

@github-actions

Copy link
Copy Markdown

✅ Triggered Buildkite CI #90765 for commit ccb781d8686d.

…p-20260923

Signed-off-by: Nils Matteson <nilsmatteson@icloud.com>
@matteso1

Copy link
Copy Markdown
Contributor Author

/ci run

@github-actions

Copy link
Copy Markdown

✅ Triggered Buildkite CI #90796 for commit 05873a0a18ea.

Signed-off-by: Nils Matteson <nilsmatteson@icloud.com>
@matteso1

Copy link
Copy Markdown
Contributor Author

/ci run

@github-actions

Copy link
Copy Markdown

✅ Triggered Buildkite CI #90829 for commit 603205687b50.

Signed-off-by: Nils Matteson <nilsmatteson@icloud.com>
@matteso1

Copy link
Copy Markdown
Contributor Author

/ci run

@github-actions

Copy link
Copy Markdown

❌ This PR is 4 commits behind upstream main. Your branch must contain every commit currently on upstream main. No new CI build was started. Merge or rebase onto the latest main, then rerun /ci run. To test this branch at your own risk, use /ci run --allow-stale.

…/bytecode-ft-20260924

Signed-off-by: Nils Matteson <nilsmatteson@icloud.com>
@matteso1

Copy link
Copy Markdown
Contributor Author

/ci run

@github-actions

Copy link
Copy Markdown

✅ Triggered Buildkite CI #91034 for commit da0a5f6ed074.

@simon-mo
simon-mo enabled auto-merge (squash) September 30, 2026 17:26
@matteso1

Copy link
Copy Markdown
Contributor Author

/ci run

@github-actions

Copy link
Copy Markdown

❌ This PR is 7 commits behind upstream main. Your branch must contain every commit currently on upstream main. No new CI build was started. Merge or rebase onto the latest main, then rerun /ci run. To test this branch at your own risk, use /ci run --allow-stale.

@matteso1

Copy link
Copy Markdown
Contributor Author

/ci run

@github-actions

Copy link
Copy Markdown

✅ Triggered Buildkite CI #92239 for commit 9bc65de60f5e.

@simon-mo
simon-mo merged commit 76d3929 into vllm-project:main Oct 1, 2026
448 of 449 checks passed
michael-gpu pushed a commit to practical-inference/vllm-ampere-extended that referenced this pull request Oct 20, 2026
)

Signed-off-by: Nils Matteson <nilsmatteson@icloud.com>
Co-authored-by: Simon Mo <simon.mo@hey.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci/build cpu Related to CPU backends intel-gpu Related to Intel GPU kv-connector ready ONLY add when PR is ready to merge/full CI is needed rocm Related to AMD ROCm

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

2 participants