Repository navigation
perf(v4): cut the host work between two forwards - #1989
Merged
Merged
Conversation
Every forward copies each sequence's block table into the int32
`block_tables` buffer. As a Python list that costs one CPython int
unboxing per block -- ~17k of them per step at 50 sequences by 100k
context, which is 0.30 ms of pure interpreter work on the host path
between two decode forwards.
Storing the ids in an `array("i")` makes the same copy a memcpy, and
hoisting the per-row `[i] = 0` into one span memset removes the second
half of the loop's cost: 0.3032 -> 0.0147 ms, 20.7x, measured against
the shipped callable with all variants checked byte-identical first.
`array("i")` is a list for append/pop/index/len/iterate but has no
`.clear()` or `.copy()`, so the six sites using those move to
`del bt[:]` and `list(bt)`. The disagg connectors keep handing out
plain lists: their ids cross a process boundary into protocol dicts
that are typed `list[int]` all the way down.
One site needed more than a rename. The TBO prefill snapshot took
`np.asarray(bt, dtype=np.int32)`, which over an `array("i")` is a
zero-copy view -- it pins a Py_buffer export, and the next
`BlockManager` append then raises BufferError a step later. Its only
reader wants `len(row)` and a slice assignment, both of which the row
serves directly, so the conversion is gone rather than repaired.
`_populate_block_tables` claimed in its docstring to mirror
`prepare_block_tables` and now delegates to it through a `limit`
argument, so the two cannot drift. Costs 0.32 us.
Gates are deterministic, not throughput: this is 0.29 ms inside a
32 ms step and no benchmark here can resolve it. Bit-exactness is
parametrised over bs in {0,1,2,50,64,256} against the pre-hoist loop,
with the destination poisoned so an uncleared cell and an unwritten
cell are distinguishable; five mutated marshals were confirmed to fail
it. The BufferError guard was confirmed to fail against the aliasing
version it replaces.
Decode metadata expands `block_tables[bs]` into one row per query token for the paged-MQA kernels. The host was doing it: zero the whole `T_pad x cols` buffer, build a boolean mask over the row -> seq map, run two fancy indexes, then ship the result. At bs=256 with full-width rows that is 0.205 ms of host time and 4 MB of H2D per step, to carry rows the device could have assembled from tensors it already holds; at bs=4096 it is 11.9 ms and 64 MB. Both inputs are on the device already. `block_tables` is staged before this runs, and in the per-token layout the row -> seq map *is* `batch_id_per_token`, staged a few lines earlier. So the expansion becomes one `index_select` and the transfer goes away. Host cost drops to 0.006 ms and, unlike the work it replaces, stays there: it is one launch, so it does not grow with the batch. The device wins too rather than absorbing the host's work -- 64 MB over PCIe is 1.18 ms where the gather is 0.012 ms, the source being small enough to stay in cache. The gather covers a prefix rather than clamping a sentinel, which is sound because the rows the old mask rejected are a contiguous tail under both layouts: per-token rows past `total_tokens`, and rect rows past `scheduled_bs * full_q`. That is the property the change rests on, so it is asserted directly rather than inferred -- if an invalid row were ever interior, only the padding rows would differ and the kernel skips those (their `n_committed_per_token` is 0), so nothing downstream would report it. The tail fill is guarded because an empty `zero_` still costs a dispatch, and unpadded batches make that slice empty. Two things follow from the buffer no longer having a host writer. It becomes a plain device tensor, which frees `T_dec * cols * 4` of pinned memory per buffer set that nothing writes; sharing it across PP ring slots is safe because the hazard `_clone_slot` guards is a host write landing while an earlier microbatch reads, and a kernel write is ordered by the stream. And its width has to match the `block_tables` it gathers from, which two call sites were computing independently and happening to agree on, so that expression is now named once as `block_table_cols` and shared by all five buffers that need it. `index_select` with an int32 index and a row-slice `out=` was the fastest of five formulations measured; `gather` looks better only if the expanded int64 index is built outside the timed region, which production cannot do. Verified bit-exact against the host expansion it replaces: on CPU tensors across five layouts including unpadded and heavily padded batches, and on real device buffers at four shapes, where the destination is also checked to be fully overwritten when dirty, free of stale `block_tables` rows, and unmoved in memory so a captured graph keeps its address.
Every decode step, each token's HCA slice needs one paged offset per
committed compressed row -- `ctx/128` of them, so ~700 per token at 100k
context. The host was computing all of them: a repeat, a cumsum, two
fancy indexes and a scatter over `T * ctx/128` int32, then shipping the
result. That was the largest single host cost between two forwards, and
the direct cause of the GPU bubble that started this work.
The kernel that writes each slice's SWA prefix already runs, already
knows the token, and the block tables it needs are already resident. So
it fills the compress section too, and the numpy build and its transfer
both go away: 2.1-2.7 ms and 1.33 MB per step at the traced shape,
15.8-16.6 ms and 6.8 MB at bs=256. What remains on the host is the
launch that was happening anyway -- 0.031 ms, flat in batch and context.
The device pays 1.0-1.2 us more for the fill, against the 25-131 us of
PCIe it replaces.
The section's length is not passed in. The slice was sized
`n_swa + n_committed_hca`, and `n_swa` is the prefix the kernel just
wrote, so the remainder is exactly the committed count. That identity is
what removes the need for a new per-seq device buffer, and with it the
question of which buffer set a TBO ubatch should read.
Filling it is opt-in via `hca_block_tables`. Four bridge call sites fill
this same section themselves from a different row formula, and a fifth
serves SWA only; unconditional writes would mean two writers and a race
decided by launch order.
Head and tail now tile each slice exactly, so the `-1` pre-fill is gone
-- and with it the last host writer of `v4_kv_indices_{swa,csa,hca}`.
All three become plain device tensors, freeing 45 MiB of pinned memory
at 131k context and 213 MiB at 1M. The capacity bound `_stage` used to
enforce is restored explicitly, since without it an over-long slice is a
silent out-of-bounds device write.
Gates. The kernel's output is bit-identical to the numpy oracle it
replaces across four shapes, including counts either side of the
two-rows-per-block boundary and sequences with nothing committed. Two
mutations were confirmed to fail those assertions: dropping the
row-within-block term (the shape of a bug this file already carries a
regression test for) and sizing the head one cell short. A separate
assertion holds that no cell of a live slice keeps its poison, which is
what the dropped pre-fill rests on -- a gap there would be a stale pool
index that nothing downstream reports.
End to end on V4-Flash-DSpark, tp1, fp8 KV, fp4 indexer, 5 speculative
tokens: GSM8K 0.9500 flexible / 0.9507 strict, acceptance 65.2% at 4.26
tokens per forward, no faults. That is the check that matters here: a
mis-wired block table would point the gather at another request's rows,
which collapses the score rather than shifting it.
Marshalling the block tables costs the same ~250 ns per row whether the
row holds eight blocks or five hundred: it is numpy's slice-assignment
dispatch, not the copy. Making the rows `array("i")` removed the
per-element cost but left that fixed one, so a batch of many short
sequences still pays it in full -- 4.4 ms at bs=16384 with 2k contexts,
of which the actual bytes moved are a rounding error.
A memoryview assignment is the same copy without the dispatch: ~50 ns
per row. That is 2.9x across the range, and it holds at both ends --
0.087 -> 0.030 ms at bs=256 with full-width rows, 4.41 -> 1.52 ms at
bs=16384 with short ones. What is left there is 16k iterations of Python
and the memset; the loop is the floor until block tables are stored
differently.
Flattening the destination is the part that needs care. `reshape(-1)`
returns a *copy* of a non-contiguous array, and every row would then be
written into a temporary and lost -- silently. `memoryview.cast`
refuses such a buffer instead, and unlike an assert it still refuses
under `python -O`.
The flat write also has no row edge. `block_tables[i, :n] = row` raised
when a row was wider than the destination; a flat write lands the
overflow in the next request's row, leaving it a block table pointing
at someone else's KV. Row width is bounded by construction, but that is
the class of assumption this file has been bitten by, so the bound is
checked. It costs 6-8%, the cheapest of three forms measured -- an
up-front `max(len)` is 12-14% and a per-row window slice is 27-35%.
Both failure modes were confirmed silent before being guarded, and both
guards were confirmed to fire. `zip(range(0, n*cols, cols), rows)` was
measured as an alternative to the running offset and is a wash, so the
simpler form stays.
End to end on V4-Flash-DSpark: GSM8K 0.9477 flexible, acceptance 65.25%
at 4.26 tokens per forward, no faults -- acceptance being the sharper
probe here, since a mis-written block table desynchronises draft from
target before it moves the score.
Two comments in this repo give opposite advice about the same call. `paged_state_copy` says a pageable `torch.from_numpy(x).to(dev)` synchronizes the current stream, so the host waits out everything already queued rather than the transfer's own bytes, and cites 2.9 ms against 0.1 ms staged. The V4 prefill metadata builder says the same call "avoids stream drain", and makes six of them per prefill. Both are right. The driver stages a pageable copy up to 512 KB and returns; past that the host waits. Measured here with 4 ms of work in flight, host time charged to the copy: 64 KB costs 0.018 ms, 512 KB 0.032, and 560 KB costs 3.047. Neither comment names the number that separates them, so a reader picks whichever they read first. `upload_numpy` picks instead. Under the limit it stays pageable, which is the cheapest route there because it allocates nothing; over it, it pins, and warns, because a caller with no buffer of its own can do no better. The prefill builder's six uploads go through it: at `max_num_batched_tokens` 16384 the indptrs are 64 KB and nothing changes, but the same arrays are 524 KB at 131072, one config knob from a 3 ms stall nobody would connect to the knob. A shared reusable pinned block was tried and is not a third option. Its copy goes out on the current stream, so the event that makes reuse safe sits behind whatever is queued, and waiting on it measured +84.8 us per queued matmul against pageable's +84.6 -- the whole cost, back again. Only a block nobody else touches escapes it, which is `CpuGpuBuffer`: flat against queue depth where a per-call `pin_memory()` is +3 to +36 depending on how fragmented the host allocator already is. The test asserts that mechanism rather than a ratio at one depth: a synchronizing copy inherits the queue, so its cost has a slope against queue depth and an async one's does not. Its gate on the helper is deliberately loose, because +36 against +84 is what pinning can deliver and asserting more would be asserting a number this route does not have. Also drops `v4_skip_prefix_len_csa`. Nothing has read or written it since the decode path started deriving skip inline; the comment saying it was kept for prefill was wrong, as prefill uploads its own tensor.
The scheduler copies a slice of `seq.token_ids` into the flat
`scheduled_tokens` buffer once per sequence per step. From a list that
is one CPython int unboxing per token, which on a prefill means the
whole chunk: 0.28 ms for 16k tokens, against 0.001 ms from an
`array("i")`. Counting the one-time conversion against it, a 100k-token
prompt goes 1.94 -> 0.90 ms across its seven chunks.
The larger number is memory. A list of 100k ids is 3.4 MiB of boxed
ints; the array is 0.38. At fifty concurrent requests of that length
that is 172 MiB against 19, and it grows with concurrency times context
rather than with either alone.
An `array("i")` behaves as a list for everything `Sequence` does to it,
including holding the -1 exit sentinel. What it does not do is compare
equal to a list, and three places compare token ids -- each of which
fails silently:
- the stop check compares a slice against `stop_token_sequences`, so
those are stored as arrays too. Left as lists, no generation ever
stops on a stop sequence and nothing says so.
- the prefix cache compares a fresh slice against the one published
into the block. Both now come from the same storage.
- `compute_hash` fed its argument to `np.array`, which infers int64
from a list and int32 from an array, so the digest depended on the
caller's Python type. That was already a hazard before this change
and is now pinned to int64 -- the value lists gave, so every hash
recorded before this commit still matches.
One consumer wants a real list back: the output dict is serialized, and
`json.dumps` refuses an array.
Each of the three is pinned by a test that also asserts the failure
shape it guards, since a happy-path check would pass just as well
against the broken versions. Both production fixes were reverted in
turn to confirm their test fails.
End to end on V4-Flash-DSpark with prefix caching on -- the first run
here to exercise the hash and block-compare paths -- GSM8K 0.9477
flexible, acceptance 65.27% at 4.26 tokens per forward, real cache hits
in the log, no faults.
`ScheduledBatch.__init__` marshalled two things a row at a time: the flat
token window each sequence contributes, and the dense draft rows a
speculative step left behind. A numpy slice-assign costs ~245ns of dispatch
whatever its length, and at decode both rows are a handful of ints, so both
loops were dispatch and nothing else.
The token windows now stage into one `array("i")` -- `extend` between two of
those is a memcpy -- and land in a single assign. That assign is whole-array
rather than `[: len(staged)]` on purpose: a sequence too short to fill its
window makes the run short, and only the exact-length form refuses it.
The draft rows are collected and concatenated once. Rows that are not full
width are padded here rather than dropping the batch back to the per-row
form, because at a large batch a sequence joins decode nearly every step and
one such sequence would take every other one down with it. `concatenate`
takes its dtype from its inputs, so it is pinned -- consumers index this into
int32 buffers.
The offset loop also read `num_rejected` out of an int32 array, building a
numpy scalar per sequence. The list `np.asarray` consumes is now kept rather
than discarded, which costs nothing.
Measured at bs=256, k=5 on MI355X:
token windows 0.073 -> 0.018 ms
draft rows 0.086 -> 0.023 ms
offset arithmetic 0.028 -> 0.012 ms
whole constructor 0.330 -> 0.166 ms
GSM8K 3-shot on V4-Flash-DSpark tp1 is 0.9538 against a 0.948-0.954 band,
acceptance 65.30% at 4.27 toks/fwd.
…ller
`prepare_block_tables` was not the only place padding ragged rows into a
rectangular int32 buffer a row at a time -- the TBO straddle rebuild does the
same thing, and was missed when the first one was fixed. Rather than fix the
second copy, both now call `pack_rows`, so the measurement that justifies the
shape lives in one place instead of being restated in comments.
The helper gains a guard the inlined version did not have: it refuses a
destination that is not int32. `.cast("i")` reinterprets bits rather than
converting them, so a float32 destination would have taken 1.0 as 1065353216
and said nothing.
The TBO site's four uploads go through `upload_numpy` as well. One of them is
[requests x blocks], which a long enough context pushes past the pageable
limit -- exactly the crossing the helper exists to notice.
Per-row cost measured on MI355X, flat in the row's length until the bytes
start to matter around 500 ints: ~245ns for the numpy form, ~46ns through the
memoryview. The memoryview setup is a fixed ~200ns, so there is no batch
small enough for it to lose.
Five chunk-metadata uploads spelled `pin_memory().to(device)` outright. With
`attn_prefill_chunk_size` at 16384 the arrays they carry are 1 KB to 64 KB --
always under the limit where pinning starts to pay -- so the pin bought
nothing and cost a fresh pinned allocation every call.
Idle host cost on MI355X:
1 KB pageable 0.0048 pin_memory 0.0078 upload_numpy 0.0053
128 KB pageable 0.0053 pin_memory 0.0090 upload_numpy 0.0049
512 KB pageable 0.0119 pin_memory 0.0275 upload_numpy 0.0122
A few microseconds a call, which is not the point. The point is that a call
site cannot know its own size -- it moves with a config knob -- and the
helper decides at the call.
The comment claimed this dict is serialized. It is not -- nothing in the tree
even reads the key; the server builds its own response from
`RequestOutput.output_tokens`. What is serialized is the `Sequence` itself, on
its way back from the engine core, and an `array("i")` pickles smaller there
than the list did.
The copy stays: it is what `generate()` hands a caller, and the storage type
underneath is ours to change again.
…urce
Staging the batch's token windows into an `array("i")` is what removes the
per-sequence numpy dispatch, but copying that array into a fresh `np.empty`
afterwards buys nothing: nothing downstream writes to `scheduled_tokens` --
it is read into `input_ids` or rebound wholesale -- so the staging buffer can
be the buffer. `np.frombuffer` wraps it in place.
That copy was not free at the shape a long prompt actually runs. Measured
per call on MI355X:
before staged wrapped
bs=256, 1 token/seq 0.0729 0.0181 0.0179
bs=8, 2048/seq 0.0044 0.0040 0.0033
bs=1, 16384/seq 0.0017 0.0024 0.0018
The third row is one sequence contributing a full `max_num_batched_tokens`
chunk -- every prefill step of a 100k-token prompt. With one row there is no
dispatch to amortize, so staging was pure added copy and the previous commit
made that shape slower than the loop it replaced. Wrapping takes it back.
The length check returns with it. The assign it replaces refused a short run
for free, because assigning into a whole array requires the exact length; an
array that wraps whatever was staged has no such opinion, and a sequence too
short to fill its window would leave the batch quietly short instead.
`pack_rows` refused an over-wide row with a message naming the row and the
destination, and refused an over-long batch with
ValueError: memoryview assignment: lvalue and rvalue have different
structures
which says nothing about what happened. Writing through one memoryview
removes the row edge in both directions, and only one of them was put back;
the numpy form this replaced raised on either.
Also drops a comment in `Sequence.__init__` that had been pasted twice. That
is the second comment-only defect on this branch, and both got through black,
ruff, the unit suite and a diff review -- comments are the one part of a
change with no gate on them.
Contributor
🏷️ CI GuideRuns automatically on every eligible PR before approval:
Heavy model tests:
|
`test_block_table_marshal.py` imported `CommonAttentionBuilder` directly while guarding the two other builders with `importorskip`. That module imports AITER at load, so on the non-GPU runner the import raised during collection -- and a collection error does not fail one file, it aborts the session. The whole suite reported "38 skipped, 1 error" and ran nothing. `pack_rows` itself needs no AITER, and it is the piece both marshals share, so its contract moves to `test_pack_rows.py` where CI can execute it: which rows it clears, which destinations it refuses, and both edges a flat write does not have. What stays behind is what genuinely needs a builder -- that each marshal writes what the per-row loop it replaced wrote -- now skipping whole rather than erroring. Also gives `mark_spliting_op` a `None` default instead of a shared `[]`. Nothing mutated it, but the formatter modernized the type hint one line above, which pulled the pre-existing B006 into the PR's diff context, where CI's reviewdog reports it. Nothing local reproduced either failure: this box has AITER, Triton and eight GPUs. `/app/logs_claude/check_tests_import_without_aiter.py` imports each test module with that set withheld, which is the gate that was missing.
aiter e4048607 replaced `inverse_rope_group_quant`'s `scale_shuffle` bool with
a `scale_layout` enum ("row" | "mfma_tile" | "n32k4"), so the wo_a mxscale path
raised
TypeError: inverse_rope_group_quant() got an unexpected keyword argument
'scale_shuffle'
`False` and `"row"` name the same row-major [S, G, Ks] scale. Said outright
rather than left to the default, because that buffer is allocated here as
[S, G, D // 128] and read back by `batched_gemm_a8w8_mxscale` as the same:
either other layout would fill those bytes in a shuffled order, with no error
and wrong numbers.
Sole call site in the tree; the `scale_shuffle_padding` arguments elsewhere
belong to ATOM's own MXFP4 activation-quant sizing and are unrelated.
Unverified at runtime here: the aiter checked out on this machine still has
the old signature, so this code needs the newer one to run at all.
5 tasks done
valarLip
added a commit
that referenced
this pull request
Aug 23, 2026
… one (#1990) * fix(engine): a fourth consumer of token ids, and a guard for the next one `Sequence.token_ids` became an `array("i")` in #1989. That commit named three places that compare token ids and fixed each. Sweeping the rest of the chain turned up a fourth consumer of a different kind -- one that serializes rather than compares -- and it fails silently too. `publish_loaded_prefix` hands `_hash_block_tokens`' output straight into a `BlockStored` event, which is msgpack-encoded. msgspec has no encoding for `array.array`, and `KVEventPublisher.publish` counts encode failures rather than raising, so with KV events enabled the whole event stream goes dark after one log line. The other two BlockStored sites accumulate into a list via `extend` and were never affected. Rather than convert at each site, the two boundaries where the type is load-bearing now assert it: - `Block.update` takes an array and refuses a list. A list there costs twice and neither cost announces itself: it never compares equal to the arrays the other publish paths store, so every hit on that block reads as a collision; and it hands the collector one traversal slot per token. - `_make_block_stored` takes a list and refuses an array, for the encoder. The rest of the chain follows the same axis: everything that grows once per token is an array now. `Sequence.output_tokens` (the completion half of `token_ids`, kept in step with it by every writer), `Sequence.logprobs`, the API server's three per-request accumulators, the streaming detokenizer's buffer, and atomesh's two. `tokenizer.decode` takes an array as is, so only the JSON edges convert back -- and only `LLMEngine`'s, because the HTTP response builders read `text` and the counters and never `token_ids`. A test pins that, so a builder that starts forwarding the key fails there rather than in production. Four annotations said `list[int]` where an array now flows. Measured on a live V4-Flash-DSpark tp1 (94,681 blocks) with a `gc.callbacks` probe: the EngineCore's stop-the-world gen-2 pause is 288.6ms with list storage and 242.8ms with arrays, and with arrays it stops growing as the prefix cache fills. Throughput is unchanged at this scale -- gen-2 fires about once per seven minutes in steady state here, so the pause matters at the concurrency #1980 measured at, not at this one. * fix(engine): two more token-id annotations that said list `compute_hash` was annotated `list[int] | array.array` on the reasoning that one caller still passed a list. It does not. All six production call sites pass an `array("i")`: - `block_manager.py` x5, from `_hash_block_tokens` / `seq.block(i)`, both slices of `Sequence.token_ids` - `policy.py`, from `_chained_prefix_hashes(seq.token_ids, ...)` The union came from reading `_chained_prefix_hashes(token_ids: list[int], ...)` and taking the signature for the fact -- but that annotation is stale in the same way, since `connector.py` hands it `seq.token_ids`. Both are narrowed to what actually arrives. The int64 pin stays. Its reason changes rather than disappearing: nothing passes a list today, so it is no longer reconciling two live callers, but it is what stops a caller who does pass one from silently computing a different digest for the same tokens. Only tests pass lists now, and `test_compute_hash_does_not_depend_on_its_ argument_type` keeps asserting the two agree -- the annotation says what callers should pass, that test says the function is robust if they do not. * perf(engine): stop the collector walking a heap it never reclaims (#1991) * perf(engine): stop the collector walking a heap it never reclaims At steady state the garbage collector in these processes finds nothing. Measured on DeepSeek-V4-Flash-DSpark tp1 with a `gc.callbacks` probe: over 1754 gen-0, 158 gen-1 and 50 gen-2 passes after startup, it reclaimed **zero** objects. Everything the hot loop allocates is acyclic and reference counting takes it. What each pass does cost, being stop-the-world and proportional to the live heap: EngineCore 268 ms of 819,923 tracked TP0 worker 979 ms of 959,694 api_server 265 ms of 632,424 Almost all of that heap is startup state -- model, compiled graph, tokenizer, KV block pool -- so `gc.freeze()` after warmup removes the work rather than merely spacing it out, which is all raising thresholds (#1980) can do. With it on, gen-2 stops firing entirely in all three processes for the rest of the run, while gen-0/1 keep going at under 2 ms: this narrows the collector, it does not disable it, so a cycle written by later code is still caught. `unfreeze_gc_heap` on shutdown is not optional. A frozen object is invisible to the collector, so an engine destroyed inside a live interpreter -- a test, an RL loop that rebuilds it -- would leave its weights and KV cache unreachable *and* uncollectable, which presents as a GPU memory leak with nothing about GC in the symptom. vLLM hit this and handles it the same way. Placement is one call per process at the boundary where startup ends: - api_server, in the lifespan just before it yields - EngineCore, at the end of `__init__` -- KV cache allocated, graphs captured, BlockPool built (94,763 `Block` objects on a V4-Flash tp1) - each ModelRunner worker, over an RPC the EngineCore sends, because only the caller knows warmup is over: weights, compile and capture all arrive as RPCs it issues, and `--enforce-eager` has no capture step to hook Disaggregated decode is the exception and is left partly covered. Its block pool is built in `_init_disagg`, after `super().__init__()` has already sent READY, so a freeze there could make queued requests permanent. It keeps the base freeze and forgoes the pool. `tune_gc` moves into the new module and stays as the fallback for `ATOM_GC_FREEZE=0`. `ATOM_GC_DEBUG` is added alongside: it logs every collection's generation, duration and reclaimed count, which is the only way to see these pauses -- a stall in the EngineCore idles the workers, and an idle worker emits no trace event at all. It is expensive (~90s of extra startup) and off by default. Two gaps found next door and closed, since the fix needed a shared per-process setup anyway: `PrefillEngineCore.run_engine` and `DecodeEngineCore.run_engine` override the base without calling it, so a disaggregated deployment got neither `enable_orphan_reaping` nor `tune_gc`, and neither ever set a process title -- both showed as bare `python` in `ps`. `_setup_engine_process` now owns identity, reaping and GC policy for all three, which also collapses the base's three `set_process_title` branches into one and makes the process title and the GC log lines agree by construction. Throughput is unchanged at this scale (32098 vs 32122 tok/s, and 32098 with freezing off), because gen-2 fires roughly once per seven minutes in steady state here. This is a tail-latency change; it pays at the concurrency #1980 measured at, which is not reproduced here. * fix(engine): one name per worker process, not three The worker side had the drift the EngineCore side just lost. Three places produced its name and only one knew about data parallelism: - `set_process_title` in `AsyncIOProc.__init__`, dp-aware (`DP0TP0`) - the GC debug callback, `f"TP{rank}"` - `ModelRunner.freeze_gc_heap`, `f"TP{rank}"` So under dp>1 every rank's worker logged as `TP0`, which is exactly the case worth telling apart -- and `ps` disagreed with both log lines. `engine_process_name` and `worker_process_name` now live next to `set_process_title`, since naming a process is what that function is for, and all five call sites take their string from them. `AsyncIOProc` names itself once, before anything logs, which also retires a `try/except Exception` that existed only to repeat the fallback the helper now expresses directly. `EngineCore._process_name_for` moves out to join them rather than staying a method, so a reader asking "what are these processes called" finds one answer. * fix(entrypoints): the atomesh frontend was the third process to miss this Enumerating which processes get the GC policy has now gone stale twice. #1980 reached the API server but not the disaggregated EngineCores, whose `run_engine` does not call the base one. The first version of this branch reached those but not atomesh, which builds its own engine in `launch_atom_standalone` and never runs the FastAPI lifespan the API server applies the policy from -- so that frontend had neither `tune_gc` (since #1980) nor the freeze, while the EngineCore and workers it spawns had both. It is the API server's counterpart, with the same profile: the tokenizer, the per-request accumulators, ~632k tracked objects and a 265 ms gen-2 pause. Rather than fix the third instance and wait for a fourth, the rule is now tested: `test_every_serving_frontend_applies_the_gc_policy` walks every module that builds an engine and serves, and fails naming the ones that do not apply the policy. Written the obvious way it was vacuous -- an unused import satisfies a search for the name -- so it requires the call. The prose enumerations in `tune_gc` and the env-var docs are replaced by the rule they kept failing to track. * fix(engine): restore the decode freeze, dropped on a misreading An earlier commit on this branch removed `_freeze_after_startup` from `DecodeEngineCore` on the grounds that READY had already gone out, so freezing could catch requests the input thread had queued and make them permanent. That is not what the code does. `DecodeEngineCore.__init__` sets `_ready_deferred = True` before calling `super().__init__()`, precisely so the base's ready signal is suppressed; the real one is sent nine lines *after* the block pool is built, once the kvcache IPC import and graph capture are done. Nothing can have been admitted at that point, so there is no window and the freeze is safe -- and it is what covers decode's 94,763 `Block` objects. The comment left in its place asserted the false premise in so many words, which would have taught the next reader the same wrong thing. * style(entrypoints): make atomesh/server.py clean, not just no-worse CI runs ruff through reviewdog with `-filter-mode=diff_context`, so it reports anything landing on a changed line or in the context around it -- pre-existing or not. Adding an import inside this file's already-unsorted block pulled a long-standing `I001` into review and failed the check. Comparing whole-file findings against `main` is the wrong local check for that: identical counts still fail if one of them is near your edit. Cheaper to leave the file with none. Sorting the imports is the fix CI asked for. The other two go with it: `initialize_standalone_service` declared `global engine, tokenizer` but only reads them, which `global` is not for; and the blind except around the version banner gets the reason it always had. * fix(tests): the worker-RPC contract test needed no aiter, and no import CI's non-GPU runner has no aiter, and `atom.model_engine.model_runner` imports it. Importing `ModelRunner` inside the test to read `freeze_gc_heap` off it therefore failed there with `ImportError: cannot import name destroy_dist_env`. The import-without-aiter probe missed it because that probe models collection: it imports each test module and stops. An import inside a test body only runs when the test does. The contract is about what the source says -- an RPC target must return something or `call_func(..., wait_out=True)` hangs -- so it is now read with `ast` straight from the file. No import, no aiter, no GPU, and the whole module drops from 5.7s to 1.6s. Re-verified by running every test file this branch touches with `aiter` and `triton` blocked at the import hook: 142 passed.
zejunchen-zejun
added a commit
that referenced
this pull request
Aug 26, 2026
Rebasing onto main brought #1989/#1990, which turned `Sequence.token_ids` and `Sequence.block_table` into `array("i")` for the GC pause it buys at long context. An `array("i")` never compares equal to a list, so the two assertions this branch added in `TestJointClaimReusesResidentBlocks` -- correct when written against a `list` block table -- started failing. Same fix main applied to its own test in #1990 (`assert list(seq.output_tokens) == [...]`): convert the array side. assert list(seq.block_table[:4]) == resident `seq.block_table = [0]` in `test_scheduler.py` is deliberately left alone: main has two of its own at lines 469 and 489 of that file, so a lone converted third would read as the odd one out, and it passes. The formatting half is `black .` over the nine source files and three tests this branch had left non-compliant; main is black-clean, so every one of them was ours. Three of the reformats join a wrapped expression onto one line and the rest are blank lines -- `git diff -w --ignore-blank-lines` is empty outside those three. Verified: `pytest tests/ --ignore=tests/plugin` -> 3780 passed, 192 skipped, 3 xfailed. The two remaining collection errors (msgpack, aiter) are on main as well. `tests/plugin` is excluded because `test_rtpllm_forward_context_semantics.py` and `test_rtpllm_glm5_sparse_backend_contract.py` replace `sys.modules["atom.utils.forward_context"]` without restoring it, which breaks every later test that builds a Scheduler -- 66 of main's own included. Pre-existing and not this branch's to fix. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
zejunchen-zejun
added a commit
that referenced
this pull request
Aug 27, 2026
Rebasing onto main brought #1989/#1990, which turned `Sequence.token_ids` and `Sequence.block_table` into `array("i")` for the GC pause it buys at long context. An `array("i")` never compares equal to a list, so the two assertions this branch added in `TestJointClaimReusesResidentBlocks` -- correct when written against a `list` block table -- started failing. Same fix main applied to its own test in #1990 (`assert list(seq.output_tokens) == [...]`): convert the array side. assert list(seq.block_table[:4]) == resident `seq.block_table = [0]` in `test_scheduler.py` is deliberately left alone: main has two of its own at lines 469 and 489 of that file, so a lone converted third would read as the odd one out, and it passes. The formatting half is `black .` over the nine source files and three tests this branch had left non-compliant; main is black-clean, so every one of them was ours. Three of the reformats join a wrapped expression onto one line and the rest are blank lines -- `git diff -w --ignore-blank-lines` is empty outside those three. Verified: `pytest tests/ --ignore=tests/plugin` -> 3780 passed, 192 skipped, 3 xfailed. The two remaining collection errors (msgpack, aiter) are on main as well. `tests/plugin` is excluded because `test_rtpllm_forward_context_semantics.py` and `test_rtpllm_glm5_sparse_backend_contract.py` replace `sys.modules["atom.utils.forward_context"]` without restoring it, which breaks every later test that builds a Scheduler -- 66 of main's own included. Pre-existing and not this branch's to fix. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
PerryZhang01
pushed a commit
that referenced
this pull request
Sep 1, 2026
`atom/models/deepseek_v4.py` calls `inverse_rope_group_quant(scale_layout=…)` since #1989. aiter main renamed that keyword from `scale_shuffle`, so on any image whose aiter predates the rename -- the released ATOM images freeze the aiter of their build day -- DeepSeek-V4-Flash fails to load with TypeError: inverse_rope_group_quant() got an unexpected keyword argument 'scale_layout' `scale_shuffle=False` and `scale_layout="row"` request the same row-major `[S, G, Ks]` scale, so a thin adapter keeps the call site working on both. It lives in a new `atom/model_ops/aiter_compat.py` so the model file only changes which module it imports from, leaving the traced graph untouched, and so the next aiter rename has somewhere obvious to go. Layouts the old aiter cannot express raise rather than silently emitting the wrong one. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
PerryZhang01
pushed a commit
that referenced
this pull request
Sep 8, 2026
`atom/models/deepseek_v4.py` calls `inverse_rope_group_quant(scale_layout=…)` since #1989. aiter main renamed that keyword from `scale_shuffle`, so on any image whose aiter predates the rename -- the released ATOM images freeze the aiter of their build day -- DeepSeek-V4-Flash fails to load with TypeError: inverse_rope_group_quant() got an unexpected keyword argument 'scale_layout' `scale_shuffle=False` and `scale_layout="row"` request the same row-major `[S, G, Ks]` scale, so a thin adapter keeps the call site working on both. It lives in a new `atom/model_ops/aiter_compat.py` so the model file only changes which module it imports from, leaving the traced graph untouched, and so the next aiter rename has somewhere obvious to go. Layouts the old aiter cannot express raise rather than silently emitting the wrong one. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
valarLip
pushed a commit
that referenced
this pull request
Sep 9, 2026
* feat(vllm): move the OOT plugin to the vLLM 0.28.0 release
Pin `atom/plugin/vllm/vllm-version.env` to the v0.28.0 tag
(`2cf0a6915ce544dc493a0990f2ea38d81601128a`) and refresh the PyTorch 2.10
stable-ABI patch against that source.
The patch's two CMake hunks and its `csrc/libtorch_stable/cuda_view.cu` hunk
still apply verbatim; only their line offsets moved. The
`vllm/utils/torch_utils.py` hunk stopped applying because upstream grew the
XPU branch of `get_accelerator_view_from_cpu_tensor()`, which is the context
the hunk matched on. The patch's own content is unchanged.
`docker/vllm_release.dockerfile` needs no change: the two upstream
requirement edits that reach the ROCm image are already satisfied by the base
image (`huggingface_hub >= 1.27.0`, which is exactly the pinned version, and
`fastsafetensors >= 0.3.3`, installed from source at 0.3.3).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(vllm): keep the worker rendezvous on a TCPStore for aiter custom AR
vLLM 0.28 switched the multiproc and uniproc executors to a `file://`
rendezvous and keeps the previous `tcp://` one only when
`vllm.utils.network_utils.aiter_requires_tcp_store()` is true. That helper
reads vLLM's own AITER switches (`VLLM_ROCM_USE_AITER` plus
`VLLM_ROCM_USE_AITER_CUSTOM_AR`), which the ATOM plugin never sets: ATOM
initialises aiter's distributed environment itself, and aiter's
`CustomAllreduce` asserts the default store is a `TCPStore` before exchanging
IPC metadata. Every TP>1 run therefore died during model load:
AssertionError: IPC metadata exchange requires a pure-TCP KV store
(torch.distributed.TCPStore), got FileStore.
Pin the helper to true from `register_platform()`, which restores the 0.27.1
rendezvous. The patch is a no-op on vLLM versions without the helper, and it
can be dropped once aiter's IPC exchange accepts a `FileStore`.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(vllm): give ATOM's attention backends vLLM 0.28's customize_spec
vLLM 0.28 added `AttentionBackend.customize_spec(spec)` and calls it while
building KV cache specs, from `v1/worker/gpu/attn_utils.py`,
`v1/worker/gpu_model_runner.py` and
`model_executor/layers/attention/attention.py`. ATOM's vLLM-facing backend
surfaces are duck-typed rather than derived from vLLM's `AttentionBackend`, so
they do not inherit the default and both MiniMax-M3 and Kimi-K3 aborted with
AttributeError: type object 'SparseMHAIndexerBackend' has no attribute
'customize_spec'
Add the identity implementation to the four standalone surfaces, which covers
every backend derived from them. The surfaces that already derive from a vLLM
class inherit it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(vllm): build the MLA DCP manager vLLM 0.28 expects on the layer
vLLM 0.28 moved MLA's DCP collectives behind a new `MLADCPManager`
(`vllm/v1/attention/ops/dcp_utils.py`; the symbol does not exist in 0.27.1).
vLLM's own `MLAAttention.__init__` builds one when `impl.dcp_world_size > 1`,
and `MLACommonMetadataBuilder.__init__` reads it back off the registered layer
and asserts the type.
`AttentionForVllmMLA` reports `dcp_world_size = -1` on purpose, to keep vLLM's
DCP paths out of ATOM's decode kernels. vLLM therefore never built a manager,
and every DCP>1 MLA run -- Kimi-K3 DSpark DCP8 in PR CI -- aborted on that
assert while building attention metadata, before a single token was served.
Build the manager for the registered layer instead, with the arguments vLLM
passes: ATOM never feeds MLA a quantized query, so the query keeps the layer
dtype. The metadata builder is its only consumer here -- it hands the manager
the chunked-prefill workspace for the context KV all-gather.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(vllm): keep vLLM's PAD slots when localizing DSpark draft KV under DCP
vLLM 0.28 taught `_prepare_dflash_inputs_kernel` to store `PAD_SLOT_ID` for
draft rows that must not receive KV: the rejected suffix of a speculative
context span, which `num_valid_ctx` now separates from `num_ctx`, and any row
whose block-table entry is the null block after a sliding-window eviction.
The DCP patch re-derives both slot mappings after the kernel runs, to express
them in rank-local terms. Under 0.28 that unconditional rewrite resurrected
exactly those rows into real slots, so the Kimi-K3 DSpark draft would write KV
over live context -- physical block 0 included.
Whether a row is writable at all is the kernel's decision, and the DCP rewrite
has no business overturning it. Keep every `PAD_SLOT_ID` it produced and
re-localize only the rows it addressed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(v4): accept both spellings of aiter's inverse-RoPE scale layout
`atom/models/deepseek_v4.py` calls `inverse_rope_group_quant(scale_layout=…)`
since #1989. aiter main renamed that keyword from `scale_shuffle`, so on any
image whose aiter predates the rename -- the released ATOM images freeze the
aiter of their build day -- DeepSeek-V4-Flash fails to load with
TypeError: inverse_rope_group_quant() got an unexpected keyword argument
'scale_layout'
`scale_shuffle=False` and `scale_layout="row"` request the same row-major
`[S, G, Ks]` scale, so a thin adapter keeps the call site working on both. It
lives in a new `atom/model_ops/aiter_compat.py` so the model file only changes
which module it imports from, leaving the traced graph untouched, and so the
next aiter rename has somewhere obvious to go. Layouts the old aiter cannot
express raise rather than silently emitting the wrong one.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(vllm): harden 0.28 compatibility adapters
Co-authored-by: Cursor <cursoragent@cursor.com>
* revert(aiter): drop the inverse-RoPE and MLA head-fold compat shims
aiter main (c6ef26a5) now carries both API changes the shims bridged:
`inverse_rope_group_quant` takes `scale_layout` and no longer accepts
`scale_shuffle`, and `aiter/mla.py` defines `_fold_seqlen_indptr` and uses
it for the DCP round-robin fold. Both adapters are dead code against it.
Drop `atom/model_ops/aiter_compat.py` and call aiter directly from
`deepseek_v4.py`; drop `_aiter_supports_mla_dcp_head_fold()` and restore
`mla_dcp_kernel_num_heads` to the width it computed before, so a persistent
DCP decode takes the gathered width as-is again. Both files are byte-identical
to main after this.
`test_dense_mla_decode_pads_gathered_dcp_heads` asserted a width of 128 that
main's own code never produces for its inputs -- 12 heads over DCP8 gathers to
96, a multiple of 16, which a persistent decode folds without padding. The
assertion now states 96 and a zero pad, which is what runs.
This raises ATOM's minimum aiter to one carrying both changes; images that
froze an older aiter need a rebuild.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: perzhang <perzhang@amd.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this is
Twelve commits against the host-side path between two forwards, found by
profiling DeepSeek-V4-Flash-DSpark at 100k-token prompts. Nothing here changes
what the engine computes -- every change is a different way of getting the same
bytes into the same buffers.
One shape accounts for most of it: a numpy slice-assign costs ~245ns of
dispatch whatever its length, so a marshal that writes one row per sequence
is dispatch and nothing else once the batch is large and the rows are short.
The repo had six of those. Four are fixed here behind one helper
(
atom.utils.pack_rows); the fifth is a documented reference path and thesixth sits behind a flag that is off by default.
Changes
Storage —
Sequence.token_idsandSequence.block_tablebecomearray("i"). Both are marshalled into int32 buffers every forward, where alist costs one CPython int unboxing per element. A 100k-token prompt also drops
from 3.4 MiB of boxed ints to 0.38.
Marshals —
pack_rowswrites ragged rows through onememoryviewof thedestination (~46ns per row against ~245ns).
ScheduledBatch.__init__stagesits flat token window into one array and wraps it with
np.frombuffer, anddensifies speculative draft rows with one
concatenate.Device work that was being done on the host — the per-token block-table
expansion becomes an
index_selecton the device, and the HCA compress indicesare built inside the existing
paged_decode_indicesTriton kernel from blocktables already resident there.
Host-to-device route —
upload_numpypicks pageable or pinned by the sizethat actually decides it (a 512 KB staging-buffer cliff, measured). Ten call
sites had the route hardcoded, in both directions.
Measured (MI355X)
ScheduledBatch.__init__, bs=256 k=5token_idsmarshal, 100k promptPinned host memory drops 45 MiB at 131k context and 213 MiB at 1M, from three
V4 metadata buffers that are written and read only by kernels and never needed
a host mirror.
Trace confirmation: per inter-forward interval, H2D goes 2,469,856 -> 102,400
bytes, and the two removed copies match their predicted sizes exactly
(1,048,576 and 1,318,880). Largest GPU gap in a decode step 2.360 -> 0.020 ms.
Correctness
An
array("i")never compares equal to a list, and three places compare tokenids -- a stop-sequence check, a prefix-cache block compare, and a hash. Each
failed silently rather than loudly, so each is pinned by a test carrying a
positive control.
BlockManager.compute_hashalso had a latent bug thissurfaced: it inferred int64 from a list and int32 from an array, so the digest
depended on the caller's Python type. It is pinned to int64, which is what
lists produced, so no recorded hash moves.
Test plan
bash .github/scripts/run_unit_tests.sh-- 2424 passed, 54 skipped.Nine failures are pre-existing on the base commit (
ffdf3bb5d): foureplb, three WoA dequant, two tracing.
routing, the storage change and the fused kernel. Every guard added here
was mutation-tested -- the check was removed and the test confirmed to
fail (9 mutations).
/mnt/DeepSeek-V4-Flash-DSparktp1, fp8 KV + fp4indexer + DSpark 5, seven runs: 0.9500 / 0.9477 / 0.9500 / 0.9484 /
0.9538 / 0.9492 / 0.9530. MTP acceptance 65.11-65.30% at 4.26 toks/fwd.
No faults. One run had prefix caching on, to exercise the hash and block
compare paths.
blackandruffclean on every touched file.TBO straddle path and the DCP branches in
aiter_mlaare covered by unittests only. Worth a look from someone who runs those.
Not included
prepare_prefillbuilds two Python lists the length of the whole chunk, whichmeasures at 0.69 ms per prefill step at mnbt=16384 and would go to ~0.09 with
numpy. It is left out because the same function has a DCP branch that cannot be
vectorized the same way, and that deserves its own change.