Skip to content

perf(engine): stop the collector walking a heap it never reclaims - #1991

Merged
valarLip merged 4 commits into
valarLip/token-ids-fourth-consumerfrom
valarLip/gc-freeze-heap
Aug 23, 2026
Merged

valarLip merged 4 commits into
valarLip/token-ids-fourth-consumerfrom
valarLip/gc-freeze-heap

Conversation

@valarLip

Copy link
Copy Markdown
Collaborator

Stacked on #1990 — base is valarLip/token-ids-fourth-consumer, and this
diff is only the second commit. Retarget to main once #1990 lands.

What

Freeze the startup heap with gc.freeze() once warmup is done, in every
serving process. Adds atom/utils/gc_utils.py; tune_gc (#1980) moves there
and stays as the fallback.

Why: the collector finds nothing

Measured on DeepSeek-V4-Flash-DSpark tp1 with a gc.callbacks probe. After
startup, across 1754 gen-0, 158 gen-1 and 50 gen-2 passes, the collector
reclaimed zero objects. Everything the hot loop allocates is acyclic and
reference counting takes it.

What a pass costs anyway, being stop-the-world and proportional to the live
heap:

process pause tracked reclaimed
EngineCore 268 ms 819,923 0
TP0 worker 979 ms 959,694 0
api_server 265 ms 632,424 36

Almost all of that heap is startup state — model, compiled graph, tokenizer,
KV block pool (94,763 Block objects). Raising thresholds (#1980) divides the
frequency; freezing removes the work.

Result

With freezing on, over a full 512-request workload:

gen-2:  0 in all three processes
gen-0/1: still firing (TP0 123+11, api_server 34+3), each under 2 ms

The collector is narrowed, not disabled — a cycle written by later code is
still caught. A test pins exactly that distinction (asserting gc.isenabled()
plus that an automatic collection still fires), because gc.collect() runs
even when the collector is off and a naive test passes against a
freeze_gc_heap that also called gc.disable().

unfreeze on shutdown is not optional

A frozen object is invisible to the collector, so an engine destroyed inside a
live interpreter — a test, an RL loop that rebuilds it — would leave its
weights and KV cache unreachable and uncollectable. That presents as a GPU
memory leak with nothing about GC in the symptom. vLLM hit this and handles it
the same way.

Placement

One call per process, at the boundary where startup ends:

  • api_server — in the lifespan, just before it yields
  • EngineCore — end of __init__: KV cache allocated, graphs captured,
    BlockPool built
  • each ModelRunner worker — over an RPC the EngineCore sends, because only
    the caller knows warmup is over: weights, compile and capture all arrive as
    RPCs it issues, and --enforce-eager has no capture step to hook

Disaggregated decode is deliberately only partly covered. Its block pool is
built in _init_disagg, after super().__init__() has already sent READY —
so a freeze there could catch requests the input thread has queued and make
them permanent. It keeps the base freeze and forgoes the pool.

Two gaps closed next door

The fix needed a shared per-process setup, which surfaced these:

  • PrefillEngineCore.run_engine and DecodeEngineCore.run_engine override the
    base without calling it, so a disaggregated deployment got neither
    enable_orphan_reaping nor tune_gc
  • neither ever set a process title — both showed as bare python in ps

_setup_engine_process now owns identity, reaping and GC policy for all three.
It also collapses the base's three set_process_title branches into one and
makes the process title and the GC log lines agree by construction.

New environment variables

Variable Default
ATOM_GC_FREEZE 1 the above; set 0 for the old behaviour
ATOM_GC_DEBUG 0 log every collection's generation, duration and reclaimed count

ATOM_GC_DEBUG is the only way to see these pauses — a stall in the EngineCore
idles the workers, and an idle worker emits no trace event at all. It is
expensive (~90 s of extra startup) and off by default. Upstream parallel:
VLLM_GC_DEBUG.

What this does not buy, at this scale

Throughput is unchanged: 32122 tok/s with freezing, 32098 without. gen-2
fires roughly once per seven minutes in steady state at c=32/tp1, so there is
almost nothing to remove here. This is a tail-latency change and it pays at the
concurrency #1980 measured at (c=2048, dp=4) — which is not reproduced in
this PR.

Prior art

vLLM freezes in all three process kinds automatically and has no
gc.set_threshold anywhere
. SGLang exposes freezing as POST /freeze_gc
plus a warning that tells operators to call it. Both reached freezing as the
primary mechanism.

Test plan

  • 7 tests in tests/test_gc_utils.py, each with the failure shape it guards
  • Mutation-tested twice: removing the pre-collect (garbage would be made
    permanent) and adding gc.disable() (the safety net) are both caught —
    the second only after fixing a test that gc.collect() had made vacuous
  • Real server, all three processes freeze; verified via ATOM_GC_DEBUG
  • Full CI suite: 9 failed / 2436 passed — the 9 are pre-existing
  • ruff histogram identical to main per file; black clean

Not done, and worth a reviewer's attention:

  • 30-minute leak soak (RSS slope + gc.get_freeze_count()). Freezing's one
    real risk is anything alive at freeze time that later becomes garbage;
    unfreeze_gc_heap closes the teardown path but not a long run.
  • c=2048 + dp=4, the regime where this is expected to pay.

At steady state the garbage collector in these processes finds nothing.
Measured on DeepSeek-V4-Flash-DSpark tp1 with a `gc.callbacks` probe: over
1754 gen-0, 158 gen-1 and 50 gen-2 passes after startup, it reclaimed
**zero** objects. Everything the hot loop allocates is acyclic and reference
counting takes it. What each pass does cost, being stop-the-world and
proportional to the live heap:

    EngineCore   268 ms      of 819,923 tracked
    TP0 worker   979 ms      of 959,694
    api_server   265 ms      of 632,424

Almost all of that heap is startup state -- model, compiled graph, tokenizer,
KV block pool -- so `gc.freeze()` after warmup removes the work rather than
merely spacing it out, which is all raising thresholds (#1980) can do. With it
on, gen-2 stops firing entirely in all three processes for the rest of the
run, while gen-0/1 keep going at under 2 ms: this narrows the collector, it
does not disable it, so a cycle written by later code is still caught.

`unfreeze_gc_heap` on shutdown is not optional. A frozen object is invisible
to the collector, so an engine destroyed inside a live interpreter -- a test,
an RL loop that rebuilds it -- would leave its weights and KV cache
unreachable *and* uncollectable, which presents as a GPU memory leak with
nothing about GC in the symptom. vLLM hit this and handles it the same way.

Placement is one call per process at the boundary where startup ends:

  - api_server, in the lifespan just before it yields
  - EngineCore, at the end of `__init__` -- KV cache allocated, graphs
    captured, BlockPool built (94,763 `Block` objects on a V4-Flash tp1)
  - each ModelRunner worker, over an RPC the EngineCore sends, because only
    the caller knows warmup is over: weights, compile and capture all arrive
    as RPCs it issues, and `--enforce-eager` has no capture step to hook

Disaggregated decode is the exception and is left partly covered. Its block
pool is built in `_init_disagg`, after `super().__init__()` has already sent
READY, so a freeze there could make queued requests permanent. It keeps the
base freeze and forgoes the pool.

`tune_gc` moves into the new module and stays as the fallback for
`ATOM_GC_FREEZE=0`. `ATOM_GC_DEBUG` is added alongside: it logs every
collection's generation, duration and reclaimed count, which is the only way
to see these pauses -- a stall in the EngineCore idles the workers, and an
idle worker emits no trace event at all. It is expensive (~90s of extra
startup) and off by default.

Two gaps found next door and closed, since the fix needed a shared per-process
setup anyway: `PrefillEngineCore.run_engine` and `DecodeEngineCore.run_engine`
override the base without calling it, so a disaggregated deployment got
neither `enable_orphan_reaping` nor `tune_gc`, and neither ever set a process
title -- both showed as bare `python` in `ps`. `_setup_engine_process` now
owns identity, reaping and GC policy for all three, which also collapses the
base's three `set_process_title` branches into one and makes the process title
and the GC log lines agree by construction.

Throughput is unchanged at this scale (32098 vs 32122 tok/s, and 32098 with
freezing off), because gen-2 fires roughly once per seven minutes in steady
state here. This is a tail-latency change; it pays at the concurrency #1980
measured at, which is not reproduced here.
The worker side had the drift the EngineCore side just lost. Three places
produced its name and only one knew about data parallelism:

  - `set_process_title` in `AsyncIOProc.__init__`, dp-aware (`DP0TP0`)
  - the GC debug callback, `f"TP{rank}"`
  - `ModelRunner.freeze_gc_heap`, `f"TP{rank}"`

So under dp>1 every rank's worker logged as `TP0`, which is exactly the case
worth telling apart -- and `ps` disagreed with both log lines.

`engine_process_name` and `worker_process_name` now live next to
`set_process_title`, since naming a process is what that function is for, and
all five call sites take their string from them. `AsyncIOProc` names itself
once, before anything logs, which also retires a `try/except Exception` that
existed only to repeat the fallback the helper now expresses directly.

`EngineCore._process_name_for` moves out to join them rather than staying a
method, so a reader asking "what are these processes called" finds one answer.
…this

Enumerating which processes get the GC policy has now gone stale twice. #1980
reached the API server but not the disaggregated EngineCores, whose
`run_engine` does not call the base one. The first version of this branch
reached those but not atomesh, which builds its own engine in
`launch_atom_standalone` and never runs the FastAPI lifespan the API server
applies the policy from -- so that frontend had neither `tune_gc` (since
#1980) nor the freeze, while the EngineCore and workers it spawns had both.

It is the API server's counterpart, with the same profile: the tokenizer, the
per-request accumulators, ~632k tracked objects and a 265 ms gen-2 pause.

Rather than fix the third instance and wait for a fourth, the rule is now
tested: `test_every_serving_frontend_applies_the_gc_policy` walks every module
that builds an engine and serves, and fails naming the ones that do not apply
the policy. Written the obvious way it was vacuous -- an unused import
satisfies a search for the name -- so it requires the call.

The prose enumerations in `tune_gc` and the env-var docs are replaced by the
rule they kept failing to track.
An earlier commit on this branch removed `_freeze_after_startup` from
`DecodeEngineCore` on the grounds that READY had already gone out, so freezing
could catch requests the input thread had queued and make them permanent.

That is not what the code does. `DecodeEngineCore.__init__` sets
`_ready_deferred = True` before calling `super().__init__()`, precisely so the
base's ready signal is suppressed; the real one is sent nine lines *after* the
block pool is built, once the kvcache IPC import and graph capture are done.
Nothing can have been admitted at that point, so there is no window and the
freeze is safe -- and it is what covers decode's 94,763 `Block` objects.

The comment left in its place asserted the false premise in so many words,
which would have taught the next reader the same wrong thing.
@valarLip
valarLip merged commit 49c86ec into valarLip/token-ids-fourth-consumer Aug 23, 2026
1 check passed
@valarLip
valarLip deleted the valarLip/gc-freeze-heap branch August 23, 2026 11:24
valarLip added a commit that referenced this pull request Aug 23, 2026
… one (#1990)

* fix(engine): a fourth consumer of token ids, and a guard for the next one

`Sequence.token_ids` became an `array("i")` in #1989. That commit named
three places that compare token ids and fixed each. Sweeping the rest of
the chain turned up a fourth consumer of a different kind -- one that
serializes rather than compares -- and it fails silently too.

`publish_loaded_prefix` hands `_hash_block_tokens`' output straight into a
`BlockStored` event, which is msgpack-encoded. msgspec has no encoding for
`array.array`, and `KVEventPublisher.publish` counts encode failures rather
than raising, so with KV events enabled the whole event stream goes dark
after one log line. The other two BlockStored sites accumulate into a list
via `extend` and were never affected.

Rather than convert at each site, the two boundaries where the type is
load-bearing now assert it:

  - `Block.update` takes an array and refuses a list. A list there costs
    twice and neither cost announces itself: it never compares equal to the
    arrays the other publish paths store, so every hit on that block reads
    as a collision; and it hands the collector one traversal slot per token.
  - `_make_block_stored` takes a list and refuses an array, for the encoder.

The rest of the chain follows the same axis: everything that grows once per
token is an array now. `Sequence.output_tokens` (the completion half of
`token_ids`, kept in step with it by every writer), `Sequence.logprobs`, the
API server's three per-request accumulators, the streaming detokenizer's
buffer, and atomesh's two. `tokenizer.decode` takes an array as is, so only
the JSON edges convert back -- and only `LLMEngine`'s, because the HTTP
response builders read `text` and the counters and never `token_ids`. A test
pins that, so a builder that starts forwarding the key fails there rather
than in production.

Four annotations said `list[int]` where an array now flows.

Measured on a live V4-Flash-DSpark tp1 (94,681 blocks) with a `gc.callbacks`
probe: the EngineCore's stop-the-world gen-2 pause is 288.6ms with list
storage and 242.8ms with arrays, and with arrays it stops growing as the
prefix cache fills. Throughput is unchanged at this scale -- gen-2 fires
about once per seven minutes in steady state here, so the pause matters at
the concurrency #1980 measured at, not at this one.

* fix(engine): two more token-id annotations that said list

`compute_hash` was annotated `list[int] | array.array` on the reasoning that
one caller still passed a list. It does not. All six production call sites
pass an `array("i")`:

  - `block_manager.py` x5, from `_hash_block_tokens` / `seq.block(i)`, both
    slices of `Sequence.token_ids`
  - `policy.py`, from `_chained_prefix_hashes(seq.token_ids, ...)`

The union came from reading `_chained_prefix_hashes(token_ids: list[int], ...)`
and taking the signature for the fact -- but that annotation is stale in the
same way, since `connector.py` hands it `seq.token_ids`. Both are narrowed to
what actually arrives.

The int64 pin stays. Its reason changes rather than disappearing: nothing
passes a list today, so it is no longer reconciling two live callers, but it is
what stops a caller who does pass one from silently computing a different
digest for the same tokens.

Only tests pass lists now, and `test_compute_hash_does_not_depend_on_its_
argument_type` keeps asserting the two agree -- the annotation says what
callers should pass, that test says the function is robust if they do not.

* perf(engine): stop the collector walking a heap it never reclaims (#1991)

* perf(engine): stop the collector walking a heap it never reclaims

At steady state the garbage collector in these processes finds nothing.
Measured on DeepSeek-V4-Flash-DSpark tp1 with a `gc.callbacks` probe: over
1754 gen-0, 158 gen-1 and 50 gen-2 passes after startup, it reclaimed
**zero** objects. Everything the hot loop allocates is acyclic and reference
counting takes it. What each pass does cost, being stop-the-world and
proportional to the live heap:

    EngineCore   268 ms      of 819,923 tracked
    TP0 worker   979 ms      of 959,694
    api_server   265 ms      of 632,424

Almost all of that heap is startup state -- model, compiled graph, tokenizer,
KV block pool -- so `gc.freeze()` after warmup removes the work rather than
merely spacing it out, which is all raising thresholds (#1980) can do. With it
on, gen-2 stops firing entirely in all three processes for the rest of the
run, while gen-0/1 keep going at under 2 ms: this narrows the collector, it
does not disable it, so a cycle written by later code is still caught.

`unfreeze_gc_heap` on shutdown is not optional. A frozen object is invisible
to the collector, so an engine destroyed inside a live interpreter -- a test,
an RL loop that rebuilds it -- would leave its weights and KV cache
unreachable *and* uncollectable, which presents as a GPU memory leak with
nothing about GC in the symptom. vLLM hit this and handles it the same way.

Placement is one call per process at the boundary where startup ends:

  - api_server, in the lifespan just before it yields
  - EngineCore, at the end of `__init__` -- KV cache allocated, graphs
    captured, BlockPool built (94,763 `Block` objects on a V4-Flash tp1)
  - each ModelRunner worker, over an RPC the EngineCore sends, because only
    the caller knows warmup is over: weights, compile and capture all arrive
    as RPCs it issues, and `--enforce-eager` has no capture step to hook

Disaggregated decode is the exception and is left partly covered. Its block
pool is built in `_init_disagg`, after `super().__init__()` has already sent
READY, so a freeze there could make queued requests permanent. It keeps the
base freeze and forgoes the pool.

`tune_gc` moves into the new module and stays as the fallback for
`ATOM_GC_FREEZE=0`. `ATOM_GC_DEBUG` is added alongside: it logs every
collection's generation, duration and reclaimed count, which is the only way
to see these pauses -- a stall in the EngineCore idles the workers, and an
idle worker emits no trace event at all. It is expensive (~90s of extra
startup) and off by default.

Two gaps found next door and closed, since the fix needed a shared per-process
setup anyway: `PrefillEngineCore.run_engine` and `DecodeEngineCore.run_engine`
override the base without calling it, so a disaggregated deployment got
neither `enable_orphan_reaping` nor `tune_gc`, and neither ever set a process
title -- both showed as bare `python` in `ps`. `_setup_engine_process` now
owns identity, reaping and GC policy for all three, which also collapses the
base's three `set_process_title` branches into one and makes the process title
and the GC log lines agree by construction.

Throughput is unchanged at this scale (32098 vs 32122 tok/s, and 32098 with
freezing off), because gen-2 fires roughly once per seven minutes in steady
state here. This is a tail-latency change; it pays at the concurrency #1980
measured at, which is not reproduced here.

* fix(engine): one name per worker process, not three

The worker side had the drift the EngineCore side just lost. Three places
produced its name and only one knew about data parallelism:

  - `set_process_title` in `AsyncIOProc.__init__`, dp-aware (`DP0TP0`)
  - the GC debug callback, `f"TP{rank}"`
  - `ModelRunner.freeze_gc_heap`, `f"TP{rank}"`

So under dp>1 every rank's worker logged as `TP0`, which is exactly the case
worth telling apart -- and `ps` disagreed with both log lines.

`engine_process_name` and `worker_process_name` now live next to
`set_process_title`, since naming a process is what that function is for, and
all five call sites take their string from them. `AsyncIOProc` names itself
once, before anything logs, which also retires a `try/except Exception` that
existed only to repeat the fallback the helper now expresses directly.

`EngineCore._process_name_for` moves out to join them rather than staying a
method, so a reader asking "what are these processes called" finds one answer.

* fix(entrypoints): the atomesh frontend was the third process to miss this

Enumerating which processes get the GC policy has now gone stale twice. #1980
reached the API server but not the disaggregated EngineCores, whose
`run_engine` does not call the base one. The first version of this branch
reached those but not atomesh, which builds its own engine in
`launch_atom_standalone` and never runs the FastAPI lifespan the API server
applies the policy from -- so that frontend had neither `tune_gc` (since
#1980) nor the freeze, while the EngineCore and workers it spawns had both.

It is the API server's counterpart, with the same profile: the tokenizer, the
per-request accumulators, ~632k tracked objects and a 265 ms gen-2 pause.

Rather than fix the third instance and wait for a fourth, the rule is now
tested: `test_every_serving_frontend_applies_the_gc_policy` walks every module
that builds an engine and serves, and fails naming the ones that do not apply
the policy. Written the obvious way it was vacuous -- an unused import
satisfies a search for the name -- so it requires the call.

The prose enumerations in `tune_gc` and the env-var docs are replaced by the
rule they kept failing to track.

* fix(engine): restore the decode freeze, dropped on a misreading

An earlier commit on this branch removed `_freeze_after_startup` from
`DecodeEngineCore` on the grounds that READY had already gone out, so freezing
could catch requests the input thread had queued and make them permanent.

That is not what the code does. `DecodeEngineCore.__init__` sets
`_ready_deferred = True` before calling `super().__init__()`, precisely so the
base's ready signal is suppressed; the real one is sent nine lines *after* the
block pool is built, once the kvcache IPC import and graph capture are done.
Nothing can have been admitted at that point, so there is no window and the
freeze is safe -- and it is what covers decode's 94,763 `Block` objects.

The comment left in its place asserted the false premise in so many words,
which would have taught the next reader the same wrong thing.

* style(entrypoints): make atomesh/server.py clean, not just no-worse

CI runs ruff through reviewdog with `-filter-mode=diff_context`, so it reports
anything landing on a changed line or in the context around it -- pre-existing
or not. Adding an import inside this file's already-unsorted block pulled a
long-standing `I001` into review and failed the check.

Comparing whole-file findings against `main` is the wrong local check for
that: identical counts still fail if one of them is near your edit. Cheaper to
leave the file with none.

Sorting the imports is the fix CI asked for. The other two go with it:
`initialize_standalone_service` declared `global engine, tokenizer` but only
reads them, which `global` is not for; and the blind except around the version
banner gets the reason it always had.

* fix(tests): the worker-RPC contract test needed no aiter, and no import

CI's non-GPU runner has no aiter, and `atom.model_engine.model_runner` imports
it. Importing `ModelRunner` inside the test to read `freeze_gc_heap` off it
therefore failed there with `ImportError: cannot import name
destroy_dist_env`.

The import-without-aiter probe missed it because that probe models collection:
it imports each test module and stops. An import inside a test body only runs
when the test does.

The contract is about what the source says -- an RPC target must return
something or `call_func(..., wait_out=True)` hangs -- so it is now read with
`ast` straight from the file. No import, no aiter, no GPU, and the whole module
drops from 5.7s to 1.6s. Re-verified by running every test file this branch
touches with `aiter` and `triton` blocked at the import hook: 142 passed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant