Repository navigation
perf(engine): stop the collector walking a heap it never reclaims - #1991
Merged
valarLip merged 4 commits intoAug 23, 2026
Merged
Conversation
At steady state the garbage collector in these processes finds nothing.
Measured on DeepSeek-V4-Flash-DSpark tp1 with a `gc.callbacks` probe: over
1754 gen-0, 158 gen-1 and 50 gen-2 passes after startup, it reclaimed
**zero** objects. Everything the hot loop allocates is acyclic and reference
counting takes it. What each pass does cost, being stop-the-world and
proportional to the live heap:
EngineCore 268 ms of 819,923 tracked
TP0 worker 979 ms of 959,694
api_server 265 ms of 632,424
Almost all of that heap is startup state -- model, compiled graph, tokenizer,
KV block pool -- so `gc.freeze()` after warmup removes the work rather than
merely spacing it out, which is all raising thresholds (#1980) can do. With it
on, gen-2 stops firing entirely in all three processes for the rest of the
run, while gen-0/1 keep going at under 2 ms: this narrows the collector, it
does not disable it, so a cycle written by later code is still caught.
`unfreeze_gc_heap` on shutdown is not optional. A frozen object is invisible
to the collector, so an engine destroyed inside a live interpreter -- a test,
an RL loop that rebuilds it -- would leave its weights and KV cache
unreachable *and* uncollectable, which presents as a GPU memory leak with
nothing about GC in the symptom. vLLM hit this and handles it the same way.
Placement is one call per process at the boundary where startup ends:
- api_server, in the lifespan just before it yields
- EngineCore, at the end of `__init__` -- KV cache allocated, graphs
captured, BlockPool built (94,763 `Block` objects on a V4-Flash tp1)
- each ModelRunner worker, over an RPC the EngineCore sends, because only
the caller knows warmup is over: weights, compile and capture all arrive
as RPCs it issues, and `--enforce-eager` has no capture step to hook
Disaggregated decode is the exception and is left partly covered. Its block
pool is built in `_init_disagg`, after `super().__init__()` has already sent
READY, so a freeze there could make queued requests permanent. It keeps the
base freeze and forgoes the pool.
`tune_gc` moves into the new module and stays as the fallback for
`ATOM_GC_FREEZE=0`. `ATOM_GC_DEBUG` is added alongside: it logs every
collection's generation, duration and reclaimed count, which is the only way
to see these pauses -- a stall in the EngineCore idles the workers, and an
idle worker emits no trace event at all. It is expensive (~90s of extra
startup) and off by default.
Two gaps found next door and closed, since the fix needed a shared per-process
setup anyway: `PrefillEngineCore.run_engine` and `DecodeEngineCore.run_engine`
override the base without calling it, so a disaggregated deployment got
neither `enable_orphan_reaping` nor `tune_gc`, and neither ever set a process
title -- both showed as bare `python` in `ps`. `_setup_engine_process` now
owns identity, reaping and GC policy for all three, which also collapses the
base's three `set_process_title` branches into one and makes the process title
and the GC log lines agree by construction.
Throughput is unchanged at this scale (32098 vs 32122 tok/s, and 32098 with
freezing off), because gen-2 fires roughly once per seven minutes in steady
state here. This is a tail-latency change; it pays at the concurrency #1980
measured at, which is not reproduced here.
The worker side had the drift the EngineCore side just lost. Three places
produced its name and only one knew about data parallelism:
- `set_process_title` in `AsyncIOProc.__init__`, dp-aware (`DP0TP0`)
- the GC debug callback, `f"TP{rank}"`
- `ModelRunner.freeze_gc_heap`, `f"TP{rank}"`
So under dp>1 every rank's worker logged as `TP0`, which is exactly the case
worth telling apart -- and `ps` disagreed with both log lines.
`engine_process_name` and `worker_process_name` now live next to
`set_process_title`, since naming a process is what that function is for, and
all five call sites take their string from them. `AsyncIOProc` names itself
once, before anything logs, which also retires a `try/except Exception` that
existed only to repeat the fallback the helper now expresses directly.
`EngineCore._process_name_for` moves out to join them rather than staying a
method, so a reader asking "what are these processes called" finds one answer.
…this Enumerating which processes get the GC policy has now gone stale twice. #1980 reached the API server but not the disaggregated EngineCores, whose `run_engine` does not call the base one. The first version of this branch reached those but not atomesh, which builds its own engine in `launch_atom_standalone` and never runs the FastAPI lifespan the API server applies the policy from -- so that frontend had neither `tune_gc` (since #1980) nor the freeze, while the EngineCore and workers it spawns had both. It is the API server's counterpart, with the same profile: the tokenizer, the per-request accumulators, ~632k tracked objects and a 265 ms gen-2 pause. Rather than fix the third instance and wait for a fourth, the rule is now tested: `test_every_serving_frontend_applies_the_gc_policy` walks every module that builds an engine and serves, and fails naming the ones that do not apply the policy. Written the obvious way it was vacuous -- an unused import satisfies a search for the name -- so it requires the call. The prose enumerations in `tune_gc` and the env-var docs are replaced by the rule they kept failing to track.
An earlier commit on this branch removed `_freeze_after_startup` from `DecodeEngineCore` on the grounds that READY had already gone out, so freezing could catch requests the input thread had queued and make them permanent. That is not what the code does. `DecodeEngineCore.__init__` sets `_ready_deferred = True` before calling `super().__init__()`, precisely so the base's ready signal is suppressed; the real one is sent nine lines *after* the block pool is built, once the kvcache IPC import and graph capture are done. Nothing can have been admitted at that point, so there is no window and the freeze is safe -- and it is what covers decode's 94,763 `Block` objects. The comment left in its place asserted the false premise in so many words, which would have taught the next reader the same wrong thing.
valarLip
added a commit
that referenced
this pull request
Aug 23, 2026
… one (#1990) * fix(engine): a fourth consumer of token ids, and a guard for the next one `Sequence.token_ids` became an `array("i")` in #1989. That commit named three places that compare token ids and fixed each. Sweeping the rest of the chain turned up a fourth consumer of a different kind -- one that serializes rather than compares -- and it fails silently too. `publish_loaded_prefix` hands `_hash_block_tokens`' output straight into a `BlockStored` event, which is msgpack-encoded. msgspec has no encoding for `array.array`, and `KVEventPublisher.publish` counts encode failures rather than raising, so with KV events enabled the whole event stream goes dark after one log line. The other two BlockStored sites accumulate into a list via `extend` and were never affected. Rather than convert at each site, the two boundaries where the type is load-bearing now assert it: - `Block.update` takes an array and refuses a list. A list there costs twice and neither cost announces itself: it never compares equal to the arrays the other publish paths store, so every hit on that block reads as a collision; and it hands the collector one traversal slot per token. - `_make_block_stored` takes a list and refuses an array, for the encoder. The rest of the chain follows the same axis: everything that grows once per token is an array now. `Sequence.output_tokens` (the completion half of `token_ids`, kept in step with it by every writer), `Sequence.logprobs`, the API server's three per-request accumulators, the streaming detokenizer's buffer, and atomesh's two. `tokenizer.decode` takes an array as is, so only the JSON edges convert back -- and only `LLMEngine`'s, because the HTTP response builders read `text` and the counters and never `token_ids`. A test pins that, so a builder that starts forwarding the key fails there rather than in production. Four annotations said `list[int]` where an array now flows. Measured on a live V4-Flash-DSpark tp1 (94,681 blocks) with a `gc.callbacks` probe: the EngineCore's stop-the-world gen-2 pause is 288.6ms with list storage and 242.8ms with arrays, and with arrays it stops growing as the prefix cache fills. Throughput is unchanged at this scale -- gen-2 fires about once per seven minutes in steady state here, so the pause matters at the concurrency #1980 measured at, not at this one. * fix(engine): two more token-id annotations that said list `compute_hash` was annotated `list[int] | array.array` on the reasoning that one caller still passed a list. It does not. All six production call sites pass an `array("i")`: - `block_manager.py` x5, from `_hash_block_tokens` / `seq.block(i)`, both slices of `Sequence.token_ids` - `policy.py`, from `_chained_prefix_hashes(seq.token_ids, ...)` The union came from reading `_chained_prefix_hashes(token_ids: list[int], ...)` and taking the signature for the fact -- but that annotation is stale in the same way, since `connector.py` hands it `seq.token_ids`. Both are narrowed to what actually arrives. The int64 pin stays. Its reason changes rather than disappearing: nothing passes a list today, so it is no longer reconciling two live callers, but it is what stops a caller who does pass one from silently computing a different digest for the same tokens. Only tests pass lists now, and `test_compute_hash_does_not_depend_on_its_ argument_type` keeps asserting the two agree -- the annotation says what callers should pass, that test says the function is robust if they do not. * perf(engine): stop the collector walking a heap it never reclaims (#1991) * perf(engine): stop the collector walking a heap it never reclaims At steady state the garbage collector in these processes finds nothing. Measured on DeepSeek-V4-Flash-DSpark tp1 with a `gc.callbacks` probe: over 1754 gen-0, 158 gen-1 and 50 gen-2 passes after startup, it reclaimed **zero** objects. Everything the hot loop allocates is acyclic and reference counting takes it. What each pass does cost, being stop-the-world and proportional to the live heap: EngineCore 268 ms of 819,923 tracked TP0 worker 979 ms of 959,694 api_server 265 ms of 632,424 Almost all of that heap is startup state -- model, compiled graph, tokenizer, KV block pool -- so `gc.freeze()` after warmup removes the work rather than merely spacing it out, which is all raising thresholds (#1980) can do. With it on, gen-2 stops firing entirely in all three processes for the rest of the run, while gen-0/1 keep going at under 2 ms: this narrows the collector, it does not disable it, so a cycle written by later code is still caught. `unfreeze_gc_heap` on shutdown is not optional. A frozen object is invisible to the collector, so an engine destroyed inside a live interpreter -- a test, an RL loop that rebuilds it -- would leave its weights and KV cache unreachable *and* uncollectable, which presents as a GPU memory leak with nothing about GC in the symptom. vLLM hit this and handles it the same way. Placement is one call per process at the boundary where startup ends: - api_server, in the lifespan just before it yields - EngineCore, at the end of `__init__` -- KV cache allocated, graphs captured, BlockPool built (94,763 `Block` objects on a V4-Flash tp1) - each ModelRunner worker, over an RPC the EngineCore sends, because only the caller knows warmup is over: weights, compile and capture all arrive as RPCs it issues, and `--enforce-eager` has no capture step to hook Disaggregated decode is the exception and is left partly covered. Its block pool is built in `_init_disagg`, after `super().__init__()` has already sent READY, so a freeze there could make queued requests permanent. It keeps the base freeze and forgoes the pool. `tune_gc` moves into the new module and stays as the fallback for `ATOM_GC_FREEZE=0`. `ATOM_GC_DEBUG` is added alongside: it logs every collection's generation, duration and reclaimed count, which is the only way to see these pauses -- a stall in the EngineCore idles the workers, and an idle worker emits no trace event at all. It is expensive (~90s of extra startup) and off by default. Two gaps found next door and closed, since the fix needed a shared per-process setup anyway: `PrefillEngineCore.run_engine` and `DecodeEngineCore.run_engine` override the base without calling it, so a disaggregated deployment got neither `enable_orphan_reaping` nor `tune_gc`, and neither ever set a process title -- both showed as bare `python` in `ps`. `_setup_engine_process` now owns identity, reaping and GC policy for all three, which also collapses the base's three `set_process_title` branches into one and makes the process title and the GC log lines agree by construction. Throughput is unchanged at this scale (32098 vs 32122 tok/s, and 32098 with freezing off), because gen-2 fires roughly once per seven minutes in steady state here. This is a tail-latency change; it pays at the concurrency #1980 measured at, which is not reproduced here. * fix(engine): one name per worker process, not three The worker side had the drift the EngineCore side just lost. Three places produced its name and only one knew about data parallelism: - `set_process_title` in `AsyncIOProc.__init__`, dp-aware (`DP0TP0`) - the GC debug callback, `f"TP{rank}"` - `ModelRunner.freeze_gc_heap`, `f"TP{rank}"` So under dp>1 every rank's worker logged as `TP0`, which is exactly the case worth telling apart -- and `ps` disagreed with both log lines. `engine_process_name` and `worker_process_name` now live next to `set_process_title`, since naming a process is what that function is for, and all five call sites take their string from them. `AsyncIOProc` names itself once, before anything logs, which also retires a `try/except Exception` that existed only to repeat the fallback the helper now expresses directly. `EngineCore._process_name_for` moves out to join them rather than staying a method, so a reader asking "what are these processes called" finds one answer. * fix(entrypoints): the atomesh frontend was the third process to miss this Enumerating which processes get the GC policy has now gone stale twice. #1980 reached the API server but not the disaggregated EngineCores, whose `run_engine` does not call the base one. The first version of this branch reached those but not atomesh, which builds its own engine in `launch_atom_standalone` and never runs the FastAPI lifespan the API server applies the policy from -- so that frontend had neither `tune_gc` (since #1980) nor the freeze, while the EngineCore and workers it spawns had both. It is the API server's counterpart, with the same profile: the tokenizer, the per-request accumulators, ~632k tracked objects and a 265 ms gen-2 pause. Rather than fix the third instance and wait for a fourth, the rule is now tested: `test_every_serving_frontend_applies_the_gc_policy` walks every module that builds an engine and serves, and fails naming the ones that do not apply the policy. Written the obvious way it was vacuous -- an unused import satisfies a search for the name -- so it requires the call. The prose enumerations in `tune_gc` and the env-var docs are replaced by the rule they kept failing to track. * fix(engine): restore the decode freeze, dropped on a misreading An earlier commit on this branch removed `_freeze_after_startup` from `DecodeEngineCore` on the grounds that READY had already gone out, so freezing could catch requests the input thread had queued and make them permanent. That is not what the code does. `DecodeEngineCore.__init__` sets `_ready_deferred = True` before calling `super().__init__()`, precisely so the base's ready signal is suppressed; the real one is sent nine lines *after* the block pool is built, once the kvcache IPC import and graph capture are done. Nothing can have been admitted at that point, so there is no window and the freeze is safe -- and it is what covers decode's 94,763 `Block` objects. The comment left in its place asserted the false premise in so many words, which would have taught the next reader the same wrong thing. * style(entrypoints): make atomesh/server.py clean, not just no-worse CI runs ruff through reviewdog with `-filter-mode=diff_context`, so it reports anything landing on a changed line or in the context around it -- pre-existing or not. Adding an import inside this file's already-unsorted block pulled a long-standing `I001` into review and failed the check. Comparing whole-file findings against `main` is the wrong local check for that: identical counts still fail if one of them is near your edit. Cheaper to leave the file with none. Sorting the imports is the fix CI asked for. The other two go with it: `initialize_standalone_service` declared `global engine, tokenizer` but only reads them, which `global` is not for; and the blind except around the version banner gets the reason it always had. * fix(tests): the worker-RPC contract test needed no aiter, and no import CI's non-GPU runner has no aiter, and `atom.model_engine.model_runner` imports it. Importing `ModelRunner` inside the test to read `freeze_gc_heap` off it therefore failed there with `ImportError: cannot import name destroy_dist_env`. The import-without-aiter probe missed it because that probe models collection: it imports each test module and stops. An import inside a test body only runs when the test does. The contract is about what the source says -- an RPC target must return something or `call_func(..., wait_out=True)` hangs -- so it is now read with `ast` straight from the file. No import, no aiter, no GPU, and the whole module drops from 5.7s to 1.6s. Re-verified by running every test file this branch touches with `aiter` and `triton` blocked at the import hook: 142 passed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Freeze the startup heap with
gc.freeze()once warmup is done, in everyserving process. Adds
atom/utils/gc_utils.py;tune_gc(#1980) moves thereand stays as the fallback.
Why: the collector finds nothing
Measured on DeepSeek-V4-Flash-DSpark tp1 with a
gc.callbacksprobe. Afterstartup, across 1754 gen-0, 158 gen-1 and 50 gen-2 passes, the collector
reclaimed zero objects. Everything the hot loop allocates is acyclic and
reference counting takes it.
What a pass costs anyway, being stop-the-world and proportional to the live
heap:
Almost all of that heap is startup state — model, compiled graph, tokenizer,
KV block pool (94,763
Blockobjects). Raising thresholds (#1980) divides thefrequency; freezing removes the work.
Result
With freezing on, over a full 512-request workload:
The collector is narrowed, not disabled — a cycle written by later code is
still caught. A test pins exactly that distinction (asserting
gc.isenabled()plus that an automatic collection still fires), because
gc.collect()runseven when the collector is off and a naive test passes against a
freeze_gc_heapthat also calledgc.disable().unfreezeon shutdown is not optionalA frozen object is invisible to the collector, so an engine destroyed inside a
live interpreter — a test, an RL loop that rebuilds it — would leave its
weights and KV cache unreachable and uncollectable. That presents as a GPU
memory leak with nothing about GC in the symptom. vLLM hit this and handles it
the same way.
Placement
One call per process, at the boundary where startup ends:
__init__: KV cache allocated, graphs captured,BlockPoolbuiltthe caller knows warmup is over: weights, compile and capture all arrive as
RPCs it issues, and
--enforce-eagerhas no capture step to hookDisaggregated decode is deliberately only partly covered. Its block pool is
built in
_init_disagg, aftersuper().__init__()has already sent READY —so a freeze there could catch requests the input thread has queued and make
them permanent. It keeps the base freeze and forgoes the pool.
Two gaps closed next door
The fix needed a shared per-process setup, which surfaced these:
PrefillEngineCore.run_engineandDecodeEngineCore.run_engineoverride thebase without calling it, so a disaggregated deployment got neither
enable_orphan_reapingnortune_gcpythoninps_setup_engine_processnow owns identity, reaping and GC policy for all three.It also collapses the base's three
set_process_titlebranches into one andmakes the process title and the GC log lines agree by construction.
New environment variables
ATOM_GC_FREEZE0for the old behaviourATOM_GC_DEBUGATOM_GC_DEBUGis the only way to see these pauses — a stall in the EngineCoreidles the workers, and an idle worker emits no trace event at all. It is
expensive (~90 s of extra startup) and off by default. Upstream parallel:
VLLM_GC_DEBUG.What this does not buy, at this scale
Throughput is unchanged: 32122 tok/s with freezing, 32098 without. gen-2
fires roughly once per seven minutes in steady state at c=32/tp1, so there is
almost nothing to remove here. This is a tail-latency change and it pays at the
concurrency #1980 measured at (c=2048, dp=4) — which is not reproduced in
this PR.
Prior art
vLLM freezes in all three process kinds automatically and has no
gc.set_thresholdanywhere. SGLang exposes freezing asPOST /freeze_gcplus a warning that tells operators to call it. Both reached freezing as the
primary mechanism.
Test plan
tests/test_gc_utils.py, each with the failure shape it guardspermanent) and adding
gc.disable()(the safety net) are both caught —the second only after fixing a test that
gc.collect()had made vacuousATOM_GC_DEBUGmainper file; black cleanNot done, and worth a reviewer's attention:
gc.get_freeze_count()). Freezing's onereal risk is anything alive at freeze time that later becomes garbage;
unfreeze_gc_heapcloses the teardown path but not a long run.