Skip to content

perf(v4): cut the host work between two forwards - #1989

Merged
valarLip merged 14 commits into
mainfrom
valarLip/v4-inter-forward-host-path
Aug 23, 2026
Merged

valarLip merged 14 commits into
mainfrom
valarLip/v4-inter-forward-host-path

Conversation

@valarLip

@valarLip valarLip commented Aug 23, 2026 •

Copy link
Copy Markdown
Collaborator

What this is

Twelve commits against the host-side path between two forwards, found by
profiling DeepSeek-V4-Flash-DSpark at 100k-token prompts. Nothing here changes
what the engine computes -- every change is a different way of getting the same
bytes into the same buffers.

One shape accounts for most of it: a numpy slice-assign costs ~245ns of
dispatch whatever its length
, so a marshal that writes one row per sequence
is dispatch and nothing else once the batch is large and the rows are short.
The repo had six of those. Four are fixed here behind one helper
(atom.utils.pack_rows); the fifth is a documented reference path and the
sixth sits behind a flag that is off by default.

Changes

Storage — Sequence.token_ids and Sequence.block_table become
array("i"). Both are marshalled into int32 buffers every forward, where a
list costs one CPython int unboxing per element. A 100k-token prompt also drops
from 3.4 MiB of boxed ints to 0.38.

Marshals — pack_rows writes ragged rows through one memoryview of the
destination (~46ns per row against ~245ns). ScheduledBatch.__init__ stages
its flat token window into one array and wraps it with np.frombuffer, and
densifies speculative draft rows with one concatenate.

Device work that was being done on the host — the per-token block-table
expansion becomes an index_select on the device, and the HCA compress indices
are built inside the existing paged_decode_indices Triton kernel from block
tables already resident there.

Host-to-device route — upload_numpy picks pageable or pinned by the size
that actually decides it (a 512 KB staging-buffer cliff, measured). Ten call
sites had the route hardcoded, in both directions.

Measured (MI355X)

before after
block-table marshal, bs=256 x 512 cols 2.235 ms 0.030
same, 1M context (4096 cols) 17.33 ms 0.128
same, 16384 short rows 8.55 ms 1.52
per-token block table, bs=256 0.205 ms + 4 MB H2D 0.006
HCA compress indices, bs=256 / 90k ctx 15.8 ms + 6.8 MB H2D 0.032
ScheduledBatch.__init__, bs=256 k=5 0.330 ms 0.138
token_ids marshal, 100k prompt 1.94 ms 0.90

Pinned host memory drops 45 MiB at 131k context and 213 MiB at 1M, from three
V4 metadata buffers that are written and read only by kernels and never needed
a host mirror.

Trace confirmation: per inter-forward interval, H2D goes 2,469,856 -> 102,400
bytes, and the two removed copies match their predicted sizes exactly
(1,048,576 and 1,318,880). Largest GPU gap in a decode step 2.360 -> 0.020 ms.

Correctness

An array("i") never compares equal to a list, and three places compare token
ids -- a stop-sequence check, a prefix-cache block compare, and a hash. Each
failed silently rather than loudly, so each is pinned by a test carrying a
positive control. BlockManager.compute_hash also had a latent bug this
surfaced: it inferred int64 from a list and int32 from an array, so the digest
depended on the caller's Python type. It is pinned to int64, which is what
lists produced, so no recorded hash moves.

Test plan

  • CI's non-GPU job: 1917 passed, 123 skipped, no errors.
  • Locally, where AITER and the GPUs are present:
    bash .github/scripts/run_unit_tests.sh -- 2424 passed, 54 skipped.
    Nine failures are pre-existing on the base commit (ffdf3bb5d): four
    eplb, three WoA dequant, two tracing.
  • 6 new test files / 103 new cases, covering the marshals, the upload
    routing, the storage change and the fused kernel. Every guard added here
    was mutation-tested -- the check was removed and the test confirmed to
    fail (9 mutations).
  • GSM8K 3-shot on /mnt/DeepSeek-V4-Flash-DSpark tp1, fp8 KV + fp4
    indexer + DSpark 5, seven runs: 0.9500 / 0.9477 / 0.9500 / 0.9484 /
    0.9538 / 0.9492 / 0.9530. MTP acceptance 65.11-65.30% at 4.26 toks/fwd.
    No faults. One run had prefix caching on, to exercise the hash and block
    compare paths.
  • black and ruff clean on every touched file.
  • Multi-GPU: all accuracy runs are tp1 and the trace work is tp8, but the
    TBO straddle path and the DCP branches in aiter_mla are covered by unit
    tests only. Worth a look from someone who runs those.

Not included

prepare_prefill builds two Python lists the length of the whole chunk, which
measures at 0.69 ms per prefill step at mnbt=16384 and would go to ~0.09 with
numpy. It is left out because the same function has a DCP branch that cannot be
vectorized the same way, and that deserves its own change.

Every forward copies each sequence's block table into the int32
`block_tables` buffer. As a Python list that costs one CPython int
unboxing per block -- ~17k of them per step at 50 sequences by 100k
context, which is 0.30 ms of pure interpreter work on the host path
between two decode forwards.

Storing the ids in an `array("i")` makes the same copy a memcpy, and
hoisting the per-row `[i] = 0` into one span memset removes the second
half of the loop's cost: 0.3032 -> 0.0147 ms, 20.7x, measured against
the shipped callable with all variants checked byte-identical first.

`array("i")` is a list for append/pop/index/len/iterate but has no
`.clear()` or `.copy()`, so the six sites using those move to
`del bt[:]` and `list(bt)`. The disagg connectors keep handing out
plain lists: their ids cross a process boundary into protocol dicts
that are typed `list[int]` all the way down.

One site needed more than a rename. The TBO prefill snapshot took
`np.asarray(bt, dtype=np.int32)`, which over an `array("i")` is a
zero-copy view -- it pins a Py_buffer export, and the next
`BlockManager` append then raises BufferError a step later. Its only
reader wants `len(row)` and a slice assignment, both of which the row
serves directly, so the conversion is gone rather than repaired.

`_populate_block_tables` claimed in its docstring to mirror
`prepare_block_tables` and now delegates to it through a `limit`
argument, so the two cannot drift. Costs 0.32 us.

Gates are deterministic, not throughput: this is 0.29 ms inside a
32 ms step and no benchmark here can resolve it. Bit-exactness is
parametrised over bs in {0,1,2,50,64,256} against the pre-hoist loop,
with the destination poisoned so an uncleared cell and an unwritten
cell are distinguishable; five mutated marshals were confirmed to fail
it. The BufferError guard was confirmed to fail against the aliasing
version it replaces.
Decode metadata expands `block_tables[bs]` into one row per query token
for the paged-MQA kernels. The host was doing it: zero the whole
`T_pad x cols` buffer, build a boolean mask over the row -> seq map,
run two fancy indexes, then ship the result. At bs=256 with full-width
rows that is 0.205 ms of host time and 4 MB of H2D per step, to carry
rows the device could have assembled from tensors it already holds; at
bs=4096 it is 11.9 ms and 64 MB.

Both inputs are on the device already. `block_tables` is staged before
this runs, and in the per-token layout the row -> seq map *is*
`batch_id_per_token`, staged a few lines earlier. So the expansion
becomes one `index_select` and the transfer goes away. Host cost drops
to 0.006 ms and, unlike the work it replaces, stays there: it is one
launch, so it does not grow with the batch. The device wins too rather
than absorbing the host's work -- 64 MB over PCIe is 1.18 ms where the
gather is 0.012 ms, the source being small enough to stay in cache.

The gather covers a prefix rather than clamping a sentinel, which is
sound because the rows the old mask rejected are a contiguous tail
under both layouts: per-token rows past `total_tokens`, and rect rows
past `scheduled_bs * full_q`. That is the property the change rests on,
so it is asserted directly rather than inferred -- if an invalid row
were ever interior, only the padding rows would differ and the kernel
skips those (their `n_committed_per_token` is 0), so nothing downstream
would report it. The tail fill is guarded because an empty `zero_` still
costs a dispatch, and unpadded batches make that slice empty.

Two things follow from the buffer no longer having a host writer. It
becomes a plain device tensor, which frees `T_dec * cols * 4` of pinned
memory per buffer set that nothing writes; sharing it across PP ring
slots is safe because the hazard `_clone_slot` guards is a host write
landing while an earlier microbatch reads, and a kernel write is
ordered by the stream. And its width has to match the `block_tables` it
gathers from, which two call sites were computing independently and
happening to agree on, so that expression is now named once as
`block_table_cols` and shared by all five buffers that need it.

`index_select` with an int32 index and a row-slice `out=` was the
fastest of five formulations measured; `gather` looks better only if
the expanded int64 index is built outside the timed region, which
production cannot do.

Verified bit-exact against the host expansion it replaces: on CPU
tensors across five layouts including unpadded and heavily padded
batches, and on real device buffers at four shapes, where the
destination is also checked to be fully overwritten when dirty, free of
stale `block_tables` rows, and unmoved in memory so a captured graph
keeps its address.
Every decode step, each token's HCA slice needs one paged offset per
committed compressed row -- `ctx/128` of them, so ~700 per token at 100k
context. The host was computing all of them: a repeat, a cumsum, two
fancy indexes and a scatter over `T * ctx/128` int32, then shipping the
result. That was the largest single host cost between two forwards, and
the direct cause of the GPU bubble that started this work.

The kernel that writes each slice's SWA prefix already runs, already
knows the token, and the block tables it needs are already resident. So
it fills the compress section too, and the numpy build and its transfer
both go away: 2.1-2.7 ms and 1.33 MB per step at the traced shape,
15.8-16.6 ms and 6.8 MB at bs=256. What remains on the host is the
launch that was happening anyway -- 0.031 ms, flat in batch and context.
The device pays 1.0-1.2 us more for the fill, against the 25-131 us of
PCIe it replaces.

The section's length is not passed in. The slice was sized
`n_swa + n_committed_hca`, and `n_swa` is the prefix the kernel just
wrote, so the remainder is exactly the committed count. That identity is
what removes the need for a new per-seq device buffer, and with it the
question of which buffer set a TBO ubatch should read.

Filling it is opt-in via `hca_block_tables`. Four bridge call sites fill
this same section themselves from a different row formula, and a fifth
serves SWA only; unconditional writes would mean two writers and a race
decided by launch order.

Head and tail now tile each slice exactly, so the `-1` pre-fill is gone
-- and with it the last host writer of `v4_kv_indices_{swa,csa,hca}`.
All three become plain device tensors, freeing 45 MiB of pinned memory
at 131k context and 213 MiB at 1M. The capacity bound `_stage` used to
enforce is restored explicitly, since without it an over-long slice is a
silent out-of-bounds device write.

Gates. The kernel's output is bit-identical to the numpy oracle it
replaces across four shapes, including counts either side of the
two-rows-per-block boundary and sequences with nothing committed. Two
mutations were confirmed to fail those assertions: dropping the
row-within-block term (the shape of a bug this file already carries a
regression test for) and sizing the head one cell short. A separate
assertion holds that no cell of a live slice keeps its poison, which is
what the dropped pre-fill rests on -- a gap there would be a stale pool
index that nothing downstream reports.

End to end on V4-Flash-DSpark, tp1, fp8 KV, fp4 indexer, 5 speculative
tokens: GSM8K 0.9500 flexible / 0.9507 strict, acceptance 65.2% at 4.26
tokens per forward, no faults. That is the check that matters here: a
mis-wired block table would point the gather at another request's rows,
which collapses the score rather than shifting it.
Marshalling the block tables costs the same ~250 ns per row whether the
row holds eight blocks or five hundred: it is numpy's slice-assignment
dispatch, not the copy. Making the rows `array("i")` removed the
per-element cost but left that fixed one, so a batch of many short
sequences still pays it in full -- 4.4 ms at bs=16384 with 2k contexts,
of which the actual bytes moved are a rounding error.

A memoryview assignment is the same copy without the dispatch: ~50 ns
per row. That is 2.9x across the range, and it holds at both ends --
0.087 -> 0.030 ms at bs=256 with full-width rows, 4.41 -> 1.52 ms at
bs=16384 with short ones. What is left there is 16k iterations of Python
and the memset; the loop is the floor until block tables are stored
differently.

Flattening the destination is the part that needs care. `reshape(-1)`
returns a *copy* of a non-contiguous array, and every row would then be
written into a temporary and lost -- silently. `memoryview.cast`
refuses such a buffer instead, and unlike an assert it still refuses
under `python -O`.

The flat write also has no row edge. `block_tables[i, :n] = row` raised
when a row was wider than the destination; a flat write lands the
overflow in the next request's row, leaving it a block table pointing
at someone else's KV. Row width is bounded by construction, but that is
the class of assumption this file has been bitten by, so the bound is
checked. It costs 6-8%, the cheapest of three forms measured -- an
up-front `max(len)` is 12-14% and a per-row window slice is 27-35%.

Both failure modes were confirmed silent before being guarded, and both
guards were confirmed to fire. `zip(range(0, n*cols, cols), rows)` was
measured as an alternative to the running offset and is a wash, so the
simpler form stays.

End to end on V4-Flash-DSpark: GSM8K 0.9477 flexible, acceptance 65.25%
at 4.26 tokens per forward, no faults -- acceptance being the sharper
probe here, since a mis-written block table desynchronises draft from
target before it moves the score.
Two comments in this repo give opposite advice about the same call.
`paged_state_copy` says a pageable `torch.from_numpy(x).to(dev)`
synchronizes the current stream, so the host waits out everything
already queued rather than the transfer's own bytes, and cites 2.9 ms
against 0.1 ms staged. The V4 prefill metadata builder says the same
call "avoids stream drain", and makes six of them per prefill.

Both are right. The driver stages a pageable copy up to 512 KB and
returns; past that the host waits. Measured here with 4 ms of work in
flight, host time charged to the copy: 64 KB costs 0.018 ms, 512 KB
0.032, and 560 KB costs 3.047. Neither comment names the number that
separates them, so a reader picks whichever they read first.

`upload_numpy` picks instead. Under the limit it stays pageable, which
is the cheapest route there because it allocates nothing; over it, it
pins, and warns, because a caller with no buffer of its own can do no
better. The prefill builder's six uploads go through it: at
`max_num_batched_tokens` 16384 the indptrs are 64 KB and nothing
changes, but the same arrays are 524 KB at 131072, one config knob from
a 3 ms stall nobody would connect to the knob.

A shared reusable pinned block was tried and is not a third option. Its
copy goes out on the current stream, so the event that makes reuse safe
sits behind whatever is queued, and waiting on it measured +84.8 us per
queued matmul against pageable's +84.6 -- the whole cost, back again.
Only a block nobody else touches escapes it, which is `CpuGpuBuffer`:
flat against queue depth where a per-call `pin_memory()` is +3 to +36
depending on how fragmented the host allocator already is.

The test asserts that mechanism rather than a ratio at one depth: a
synchronizing copy inherits the queue, so its cost has a slope against
queue depth and an async one's does not. Its gate on the helper is
deliberately loose, because +36 against +84 is what pinning can deliver
and asserting more would be asserting a number this route does not have.

Also drops `v4_skip_prefix_len_csa`. Nothing has read or written it
since the decode path started deriving skip inline; the comment saying
it was kept for prefill was wrong, as prefill uploads its own tensor.
The scheduler copies a slice of `seq.token_ids` into the flat
`scheduled_tokens` buffer once per sequence per step. From a list that
is one CPython int unboxing per token, which on a prefill means the
whole chunk: 0.28 ms for 16k tokens, against 0.001 ms from an
`array("i")`. Counting the one-time conversion against it, a 100k-token
prompt goes 1.94 -> 0.90 ms across its seven chunks.

The larger number is memory. A list of 100k ids is 3.4 MiB of boxed
ints; the array is 0.38. At fifty concurrent requests of that length
that is 172 MiB against 19, and it grows with concurrency times context
rather than with either alone.

An `array("i")` behaves as a list for everything `Sequence` does to it,
including holding the -1 exit sentinel. What it does not do is compare
equal to a list, and three places compare token ids -- each of which
fails silently:

  - the stop check compares a slice against `stop_token_sequences`, so
    those are stored as arrays too. Left as lists, no generation ever
    stops on a stop sequence and nothing says so.
  - the prefix cache compares a fresh slice against the one published
    into the block. Both now come from the same storage.
  - `compute_hash` fed its argument to `np.array`, which infers int64
    from a list and int32 from an array, so the digest depended on the
    caller's Python type. That was already a hazard before this change
    and is now pinned to int64 -- the value lists gave, so every hash
    recorded before this commit still matches.

One consumer wants a real list back: the output dict is serialized, and
`json.dumps` refuses an array.

Each of the three is pinned by a test that also asserts the failure
shape it guards, since a happy-path check would pass just as well
against the broken versions. Both production fixes were reverted in
turn to confirm their test fails.

End to end on V4-Flash-DSpark with prefix caching on -- the first run
here to exercise the hash and block-compare paths -- GSM8K 0.9477
flexible, acceptance 65.27% at 4.26 tokens per forward, real cache hits
in the log, no faults.
`ScheduledBatch.__init__` marshalled two things a row at a time: the flat
token window each sequence contributes, and the dense draft rows a
speculative step left behind. A numpy slice-assign costs ~245ns of dispatch
whatever its length, and at decode both rows are a handful of ints, so both
loops were dispatch and nothing else.

The token windows now stage into one `array("i")` -- `extend` between two of
those is a memcpy -- and land in a single assign. That assign is whole-array
rather than `[: len(staged)]` on purpose: a sequence too short to fill its
window makes the run short, and only the exact-length form refuses it.

The draft rows are collected and concatenated once. Rows that are not full
width are padded here rather than dropping the batch back to the per-row
form, because at a large batch a sequence joins decode nearly every step and
one such sequence would take every other one down with it. `concatenate`
takes its dtype from its inputs, so it is pinned -- consumers index this into
int32 buffers.

The offset loop also read `num_rejected` out of an int32 array, building a
numpy scalar per sequence. The list `np.asarray` consumes is now kept rather
than discarded, which costs nothing.

Measured at bs=256, k=5 on MI355X:

    token windows      0.073 -> 0.018 ms
    draft rows         0.086 -> 0.023 ms
    offset arithmetic  0.028 -> 0.012 ms
    whole constructor  0.330 -> 0.166 ms

GSM8K 3-shot on V4-Flash-DSpark tp1 is 0.9538 against a 0.948-0.954 band,
acceptance 65.30% at 4.27 toks/fwd.
…ller

`prepare_block_tables` was not the only place padding ragged rows into a
rectangular int32 buffer a row at a time -- the TBO straddle rebuild does the
same thing, and was missed when the first one was fixed. Rather than fix the
second copy, both now call `pack_rows`, so the measurement that justifies the
shape lives in one place instead of being restated in comments.

The helper gains a guard the inlined version did not have: it refuses a
destination that is not int32. `.cast("i")` reinterprets bits rather than
converting them, so a float32 destination would have taken 1.0 as 1065353216
and said nothing.

The TBO site's four uploads go through `upload_numpy` as well. One of them is
[requests x blocks], which a long enough context pushes past the pageable
limit -- exactly the crossing the helper exists to notice.

Per-row cost measured on MI355X, flat in the row's length until the bytes
start to matter around 500 ints: ~245ns for the numpy form, ~46ns through the
memoryview. The memoryview setup is a fixed ~200ns, so there is no batch
small enough for it to lose.
Five chunk-metadata uploads spelled `pin_memory().to(device)` outright. With
`attn_prefill_chunk_size` at 16384 the arrays they carry are 1 KB to 64 KB --
always under the limit where pinning starts to pay -- so the pin bought
nothing and cost a fresh pinned allocation every call.

Idle host cost on MI355X:

    1 KB     pageable 0.0048   pin_memory 0.0078   upload_numpy 0.0053
    128 KB   pageable 0.0053   pin_memory 0.0090   upload_numpy 0.0049
    512 KB   pageable 0.0119   pin_memory 0.0275   upload_numpy 0.0122

A few microseconds a call, which is not the point. The point is that a call
site cannot know its own size -- it moves with a config knob -- and the
helper decides at the call.
The comment claimed this dict is serialized. It is not -- nothing in the tree
even reads the key; the server builds its own response from
`RequestOutput.output_tokens`. What is serialized is the `Sequence` itself, on
its way back from the engine core, and an `array("i")` pickles smaller there
than the list did.

The copy stays: it is what `generate()` hands a caller, and the storage type
underneath is ours to change again.
…urce

Staging the batch's token windows into an `array("i")` is what removes the
per-sequence numpy dispatch, but copying that array into a fresh `np.empty`
afterwards buys nothing: nothing downstream writes to `scheduled_tokens` --
it is read into `input_ids` or rebound wholesale -- so the staging buffer can
be the buffer. `np.frombuffer` wraps it in place.

That copy was not free at the shape a long prompt actually runs. Measured
per call on MI355X:

                          before   staged   wrapped
    bs=256, 1 token/seq   0.0729   0.0181   0.0179
    bs=8, 2048/seq        0.0044   0.0040   0.0033
    bs=1, 16384/seq       0.0017   0.0024   0.0018

The third row is one sequence contributing a full `max_num_batched_tokens`
chunk -- every prefill step of a 100k-token prompt. With one row there is no
dispatch to amortize, so staging was pure added copy and the previous commit
made that shape slower than the loop it replaced. Wrapping takes it back.

The length check returns with it. The assign it replaces refused a short run
for free, because assigning into a whole array requires the exact length; an
array that wraps whatever was staged has no such opinion, and a sequence too
short to fill its window would leave the batch quietly short instead.
`pack_rows` refused an over-wide row with a message naming the row and the
destination, and refused an over-long batch with

    ValueError: memoryview assignment: lvalue and rvalue have different
    structures

which says nothing about what happened. Writing through one memoryview
removes the row edge in both directions, and only one of them was put back;
the numpy form this replaced raised on either.

Also drops a comment in `Sequence.__init__` that had been pasted twice. That
is the second comment-only defect on this branch, and both got through black,
ruff, the unit suite and a diff review -- comments are the one part of a
change with no gate on them.
@github-actions

Copy link
Copy Markdown
Contributor

🏷️ CI Guide

Runs automatically on every eligible PR before approval:

  • ✅ Pre Checkin: Black, Ruff, catalog schema validation, non-GPU unit tests

Heavy model tests:

  • ✅ Run after the PR is approved and Pre Checkin passes
  • ✅ Run immediately when an approval review is submitted
  • ✅ Can be requested before approval with labels
Label Tests
ci:full Run all heavy PR model tests: native ATOM, vLLM, and SGLang
ci:atom Run native ATOM model accuracy tests
ci:vllm Run ATOM vLLM OOT model accuracy tests
ci:sglang Run ATOM SGLang model accuracy tests

Heavy jobs are skipped when the PR is not approved and no matching ci:* label is present.
Add labels via the sidebar or gh pr edit 1989 --add-label <label>

`test_block_table_marshal.py` imported `CommonAttentionBuilder` directly while
guarding the two other builders with `importorskip`. That module imports AITER
at load, so on the non-GPU runner the import raised during collection -- and a
collection error does not fail one file, it aborts the session. The whole
suite reported "38 skipped, 1 error" and ran nothing.

`pack_rows` itself needs no AITER, and it is the piece both marshals share, so
its contract moves to `test_pack_rows.py` where CI can execute it: which rows
it clears, which destinations it refuses, and both edges a flat write does not
have. What stays behind is what genuinely needs a builder -- that each marshal
writes what the per-row loop it replaced wrote -- now skipping whole rather
than erroring.

Also gives `mark_spliting_op` a `None` default instead of a shared `[]`.
Nothing mutated it, but the formatter modernized the type hint one line above,
which pulled the pre-existing B006 into the PR's diff context, where CI's
reviewdog reports it.

Nothing local reproduced either failure: this box has AITER, Triton and eight
GPUs. `/app/logs_claude/check_tests_import_without_aiter.py` imports each test
module with that set withheld, which is the gate that was missing.
aiter e4048607 replaced `inverse_rope_group_quant`'s `scale_shuffle` bool with
a `scale_layout` enum ("row" | "mfma_tile" | "n32k4"), so the wo_a mxscale path
raised

    TypeError: inverse_rope_group_quant() got an unexpected keyword argument
    'scale_shuffle'

`False` and `"row"` name the same row-major [S, G, Ks] scale. Said outright
rather than left to the default, because that buffer is allocated here as
[S, G, D // 128] and read back by `batched_gemm_a8w8_mxscale` as the same:
either other layout would fill those bytes in a shuffled order, with no error
and wrong numbers.

Sole call site in the tree; the `scale_shuffle_padding` arguments elsewhere
belong to ATOM's own MXFP4 activation-quant sizing and are unrelated.

Unverified at runtime here: the aiter checked out on this machine still has
the old signature, so this code needs the newer one to run at all.
@valarLip
valarLip merged commit 971a752 into main Aug 23, 2026
36 of 41 checks passed
@valarLip
valarLip deleted the valarLip/v4-inter-forward-host-path branch August 23, 2026 09:31
valarLip added a commit that referenced this pull request Aug 23, 2026
… one (#1990)

* fix(engine): a fourth consumer of token ids, and a guard for the next one

`Sequence.token_ids` became an `array("i")` in #1989. That commit named
three places that compare token ids and fixed each. Sweeping the rest of
the chain turned up a fourth consumer of a different kind -- one that
serializes rather than compares -- and it fails silently too.

`publish_loaded_prefix` hands `_hash_block_tokens`' output straight into a
`BlockStored` event, which is msgpack-encoded. msgspec has no encoding for
`array.array`, and `KVEventPublisher.publish` counts encode failures rather
than raising, so with KV events enabled the whole event stream goes dark
after one log line. The other two BlockStored sites accumulate into a list
via `extend` and were never affected.

Rather than convert at each site, the two boundaries where the type is
load-bearing now assert it:

  - `Block.update` takes an array and refuses a list. A list there costs
    twice and neither cost announces itself: it never compares equal to the
    arrays the other publish paths store, so every hit on that block reads
    as a collision; and it hands the collector one traversal slot per token.
  - `_make_block_stored` takes a list and refuses an array, for the encoder.

The rest of the chain follows the same axis: everything that grows once per
token is an array now. `Sequence.output_tokens` (the completion half of
`token_ids`, kept in step with it by every writer), `Sequence.logprobs`, the
API server's three per-request accumulators, the streaming detokenizer's
buffer, and atomesh's two. `tokenizer.decode` takes an array as is, so only
the JSON edges convert back -- and only `LLMEngine`'s, because the HTTP
response builders read `text` and the counters and never `token_ids`. A test
pins that, so a builder that starts forwarding the key fails there rather
than in production.

Four annotations said `list[int]` where an array now flows.

Measured on a live V4-Flash-DSpark tp1 (94,681 blocks) with a `gc.callbacks`
probe: the EngineCore's stop-the-world gen-2 pause is 288.6ms with list
storage and 242.8ms with arrays, and with arrays it stops growing as the
prefix cache fills. Throughput is unchanged at this scale -- gen-2 fires
about once per seven minutes in steady state here, so the pause matters at
the concurrency #1980 measured at, not at this one.

* fix(engine): two more token-id annotations that said list

`compute_hash` was annotated `list[int] | array.array` on the reasoning that
one caller still passed a list. It does not. All six production call sites
pass an `array("i")`:

  - `block_manager.py` x5, from `_hash_block_tokens` / `seq.block(i)`, both
    slices of `Sequence.token_ids`
  - `policy.py`, from `_chained_prefix_hashes(seq.token_ids, ...)`

The union came from reading `_chained_prefix_hashes(token_ids: list[int], ...)`
and taking the signature for the fact -- but that annotation is stale in the
same way, since `connector.py` hands it `seq.token_ids`. Both are narrowed to
what actually arrives.

The int64 pin stays. Its reason changes rather than disappearing: nothing
passes a list today, so it is no longer reconciling two live callers, but it is
what stops a caller who does pass one from silently computing a different
digest for the same tokens.

Only tests pass lists now, and `test_compute_hash_does_not_depend_on_its_
argument_type` keeps asserting the two agree -- the annotation says what
callers should pass, that test says the function is robust if they do not.

* perf(engine): stop the collector walking a heap it never reclaims (#1991)

* perf(engine): stop the collector walking a heap it never reclaims

At steady state the garbage collector in these processes finds nothing.
Measured on DeepSeek-V4-Flash-DSpark tp1 with a `gc.callbacks` probe: over
1754 gen-0, 158 gen-1 and 50 gen-2 passes after startup, it reclaimed
**zero** objects. Everything the hot loop allocates is acyclic and reference
counting takes it. What each pass does cost, being stop-the-world and
proportional to the live heap:

    EngineCore   268 ms      of 819,923 tracked
    TP0 worker   979 ms      of 959,694
    api_server   265 ms      of 632,424

Almost all of that heap is startup state -- model, compiled graph, tokenizer,
KV block pool -- so `gc.freeze()` after warmup removes the work rather than
merely spacing it out, which is all raising thresholds (#1980) can do. With it
on, gen-2 stops firing entirely in all three processes for the rest of the
run, while gen-0/1 keep going at under 2 ms: this narrows the collector, it
does not disable it, so a cycle written by later code is still caught.

`unfreeze_gc_heap` on shutdown is not optional. A frozen object is invisible
to the collector, so an engine destroyed inside a live interpreter -- a test,
an RL loop that rebuilds it -- would leave its weights and KV cache
unreachable *and* uncollectable, which presents as a GPU memory leak with
nothing about GC in the symptom. vLLM hit this and handles it the same way.

Placement is one call per process at the boundary where startup ends:

  - api_server, in the lifespan just before it yields
  - EngineCore, at the end of `__init__` -- KV cache allocated, graphs
    captured, BlockPool built (94,763 `Block` objects on a V4-Flash tp1)
  - each ModelRunner worker, over an RPC the EngineCore sends, because only
    the caller knows warmup is over: weights, compile and capture all arrive
    as RPCs it issues, and `--enforce-eager` has no capture step to hook

Disaggregated decode is the exception and is left partly covered. Its block
pool is built in `_init_disagg`, after `super().__init__()` has already sent
READY, so a freeze there could make queued requests permanent. It keeps the
base freeze and forgoes the pool.

`tune_gc` moves into the new module and stays as the fallback for
`ATOM_GC_FREEZE=0`. `ATOM_GC_DEBUG` is added alongside: it logs every
collection's generation, duration and reclaimed count, which is the only way
to see these pauses -- a stall in the EngineCore idles the workers, and an
idle worker emits no trace event at all. It is expensive (~90s of extra
startup) and off by default.

Two gaps found next door and closed, since the fix needed a shared per-process
setup anyway: `PrefillEngineCore.run_engine` and `DecodeEngineCore.run_engine`
override the base without calling it, so a disaggregated deployment got
neither `enable_orphan_reaping` nor `tune_gc`, and neither ever set a process
title -- both showed as bare `python` in `ps`. `_setup_engine_process` now
owns identity, reaping and GC policy for all three, which also collapses the
base's three `set_process_title` branches into one and makes the process title
and the GC log lines agree by construction.

Throughput is unchanged at this scale (32098 vs 32122 tok/s, and 32098 with
freezing off), because gen-2 fires roughly once per seven minutes in steady
state here. This is a tail-latency change; it pays at the concurrency #1980
measured at, which is not reproduced here.

* fix(engine): one name per worker process, not three

The worker side had the drift the EngineCore side just lost. Three places
produced its name and only one knew about data parallelism:

  - `set_process_title` in `AsyncIOProc.__init__`, dp-aware (`DP0TP0`)
  - the GC debug callback, `f"TP{rank}"`
  - `ModelRunner.freeze_gc_heap`, `f"TP{rank}"`

So under dp>1 every rank's worker logged as `TP0`, which is exactly the case
worth telling apart -- and `ps` disagreed with both log lines.

`engine_process_name` and `worker_process_name` now live next to
`set_process_title`, since naming a process is what that function is for, and
all five call sites take their string from them. `AsyncIOProc` names itself
once, before anything logs, which also retires a `try/except Exception` that
existed only to repeat the fallback the helper now expresses directly.

`EngineCore._process_name_for` moves out to join them rather than staying a
method, so a reader asking "what are these processes called" finds one answer.

* fix(entrypoints): the atomesh frontend was the third process to miss this

Enumerating which processes get the GC policy has now gone stale twice. #1980
reached the API server but not the disaggregated EngineCores, whose
`run_engine` does not call the base one. The first version of this branch
reached those but not atomesh, which builds its own engine in
`launch_atom_standalone` and never runs the FastAPI lifespan the API server
applies the policy from -- so that frontend had neither `tune_gc` (since
#1980) nor the freeze, while the EngineCore and workers it spawns had both.

It is the API server's counterpart, with the same profile: the tokenizer, the
per-request accumulators, ~632k tracked objects and a 265 ms gen-2 pause.

Rather than fix the third instance and wait for a fourth, the rule is now
tested: `test_every_serving_frontend_applies_the_gc_policy` walks every module
that builds an engine and serves, and fails naming the ones that do not apply
the policy. Written the obvious way it was vacuous -- an unused import
satisfies a search for the name -- so it requires the call.

The prose enumerations in `tune_gc` and the env-var docs are replaced by the
rule they kept failing to track.

* fix(engine): restore the decode freeze, dropped on a misreading

An earlier commit on this branch removed `_freeze_after_startup` from
`DecodeEngineCore` on the grounds that READY had already gone out, so freezing
could catch requests the input thread had queued and make them permanent.

That is not what the code does. `DecodeEngineCore.__init__` sets
`_ready_deferred = True` before calling `super().__init__()`, precisely so the
base's ready signal is suppressed; the real one is sent nine lines *after* the
block pool is built, once the kvcache IPC import and graph capture are done.
Nothing can have been admitted at that point, so there is no window and the
freeze is safe -- and it is what covers decode's 94,763 `Block` objects.

The comment left in its place asserted the false premise in so many words,
which would have taught the next reader the same wrong thing.

* style(entrypoints): make atomesh/server.py clean, not just no-worse

CI runs ruff through reviewdog with `-filter-mode=diff_context`, so it reports
anything landing on a changed line or in the context around it -- pre-existing
or not. Adding an import inside this file's already-unsorted block pulled a
long-standing `I001` into review and failed the check.

Comparing whole-file findings against `main` is the wrong local check for
that: identical counts still fail if one of them is near your edit. Cheaper to
leave the file with none.

Sorting the imports is the fix CI asked for. The other two go with it:
`initialize_standalone_service` declared `global engine, tokenizer` but only
reads them, which `global` is not for; and the blind except around the version
banner gets the reason it always had.

* fix(tests): the worker-RPC contract test needed no aiter, and no import

CI's non-GPU runner has no aiter, and `atom.model_engine.model_runner` imports
it. Importing `ModelRunner` inside the test to read `freeze_gc_heap` off it
therefore failed there with `ImportError: cannot import name
destroy_dist_env`.

The import-without-aiter probe missed it because that probe models collection:
it imports each test module and stops. An import inside a test body only runs
when the test does.

The contract is about what the source says -- an RPC target must return
something or `call_func(..., wait_out=True)` hangs -- so it is now read with
`ast` straight from the file. No import, no aiter, no GPU, and the whole module
drops from 5.7s to 1.6s. Re-verified by running every test file this branch
touches with `aiter` and `triton` blocked at the import hook: 142 passed.
zejunchen-zejun added a commit that referenced this pull request Aug 26, 2026
Rebasing onto main brought #1989/#1990, which turned `Sequence.token_ids`
and `Sequence.block_table` into `array("i")` for the GC pause it buys at
long context. An `array("i")` never compares equal to a list, so the two
assertions this branch added in `TestJointClaimReusesResidentBlocks` --
correct when written against a `list` block table -- started failing.

Same fix main applied to its own test in #1990
(`assert list(seq.output_tokens) == [...]`): convert the array side.

  assert list(seq.block_table[:4]) == resident

`seq.block_table = [0]` in `test_scheduler.py` is deliberately left alone:
main has two of its own at lines 469 and 489 of that file, so a lone
converted third would read as the odd one out, and it passes.

The formatting half is `black .` over the nine source files and three
tests this branch had left non-compliant; main is black-clean, so every
one of them was ours. Three of the reformats join a wrapped expression
onto one line and the rest are blank lines -- `git diff -w
--ignore-blank-lines` is empty outside those three.

Verified: `pytest tests/ --ignore=tests/plugin` -> 3780 passed, 192
skipped, 3 xfailed. The two remaining collection errors (msgpack,
aiter) are on main as well. `tests/plugin` is excluded because
`test_rtpllm_forward_context_semantics.py` and
`test_rtpllm_glm5_sparse_backend_contract.py` replace
`sys.modules["atom.utils.forward_context"]` without restoring it, which
breaks every later test that builds a Scheduler -- 66 of main's own
included. Pre-existing and not this branch's to fix.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
zejunchen-zejun added a commit that referenced this pull request Aug 27, 2026
Rebasing onto main brought #1989/#1990, which turned `Sequence.token_ids`
and `Sequence.block_table` into `array("i")` for the GC pause it buys at
long context. An `array("i")` never compares equal to a list, so the two
assertions this branch added in `TestJointClaimReusesResidentBlocks` --
correct when written against a `list` block table -- started failing.

Same fix main applied to its own test in #1990
(`assert list(seq.output_tokens) == [...]`): convert the array side.

  assert list(seq.block_table[:4]) == resident

`seq.block_table = [0]` in `test_scheduler.py` is deliberately left alone:
main has two of its own at lines 469 and 489 of that file, so a lone
converted third would read as the odd one out, and it passes.

The formatting half is `black .` over the nine source files and three
tests this branch had left non-compliant; main is black-clean, so every
one of them was ours. Three of the reformats join a wrapped expression
onto one line and the rest are blank lines -- `git diff -w
--ignore-blank-lines` is empty outside those three.

Verified: `pytest tests/ --ignore=tests/plugin` -> 3780 passed, 192
skipped, 3 xfailed. The two remaining collection errors (msgpack,
aiter) are on main as well. `tests/plugin` is excluded because
`test_rtpllm_forward_context_semantics.py` and
`test_rtpllm_glm5_sparse_backend_contract.py` replace
`sys.modules["atom.utils.forward_context"]` without restoring it, which
breaks every later test that builds a Scheduler -- 66 of main's own
included. Pre-existing and not this branch's to fix.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
PerryZhang01 pushed a commit that referenced this pull request Sep 1, 2026
`atom/models/deepseek_v4.py` calls `inverse_rope_group_quant(scale_layout=…)`
since #1989. aiter main renamed that keyword from `scale_shuffle`, so on any
image whose aiter predates the rename -- the released ATOM images freeze the
aiter of their build day -- DeepSeek-V4-Flash fails to load with

    TypeError: inverse_rope_group_quant() got an unexpected keyword argument
    'scale_layout'

`scale_shuffle=False` and `scale_layout="row"` request the same row-major
`[S, G, Ks]` scale, so a thin adapter keeps the call site working on both. It
lives in a new `atom/model_ops/aiter_compat.py` so the model file only changes
which module it imports from, leaving the traced graph untouched, and so the
next aiter rename has somewhere obvious to go. Layouts the old aiter cannot
express raise rather than silently emitting the wrong one.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
PerryZhang01 pushed a commit that referenced this pull request Sep 8, 2026
`atom/models/deepseek_v4.py` calls `inverse_rope_group_quant(scale_layout=…)`
since #1989. aiter main renamed that keyword from `scale_shuffle`, so on any
image whose aiter predates the rename -- the released ATOM images freeze the
aiter of their build day -- DeepSeek-V4-Flash fails to load with

    TypeError: inverse_rope_group_quant() got an unexpected keyword argument
    'scale_layout'

`scale_shuffle=False` and `scale_layout="row"` request the same row-major
`[S, G, Ks]` scale, so a thin adapter keeps the call site working on both. It
lives in a new `atom/model_ops/aiter_compat.py` so the model file only changes
which module it imports from, leaving the traced graph untouched, and so the
next aiter rename has somewhere obvious to go. Layouts the old aiter cannot
express raise rather than silently emitting the wrong one.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
valarLip pushed a commit that referenced this pull request Sep 9, 2026
* feat(vllm): move the OOT plugin to the vLLM 0.28.0 release

Pin `atom/plugin/vllm/vllm-version.env` to the v0.28.0 tag
(`2cf0a6915ce544dc493a0990f2ea38d81601128a`) and refresh the PyTorch 2.10
stable-ABI patch against that source.

The patch's two CMake hunks and its `csrc/libtorch_stable/cuda_view.cu` hunk
still apply verbatim; only their line offsets moved. The
`vllm/utils/torch_utils.py` hunk stopped applying because upstream grew the
XPU branch of `get_accelerator_view_from_cpu_tensor()`, which is the context
the hunk matched on. The patch's own content is unchanged.

`docker/vllm_release.dockerfile` needs no change: the two upstream
requirement edits that reach the ROCm image are already satisfied by the base
image (`huggingface_hub >= 1.27.0`, which is exactly the pinned version, and
`fastsafetensors >= 0.3.3`, installed from source at 0.3.3).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(vllm): keep the worker rendezvous on a TCPStore for aiter custom AR

vLLM 0.28 switched the multiproc and uniproc executors to a `file://`
rendezvous and keeps the previous `tcp://` one only when
`vllm.utils.network_utils.aiter_requires_tcp_store()` is true. That helper
reads vLLM's own AITER switches (`VLLM_ROCM_USE_AITER` plus
`VLLM_ROCM_USE_AITER_CUSTOM_AR`), which the ATOM plugin never sets: ATOM
initialises aiter's distributed environment itself, and aiter's
`CustomAllreduce` asserts the default store is a `TCPStore` before exchanging
IPC metadata. Every TP>1 run therefore died during model load:

    AssertionError: IPC metadata exchange requires a pure-TCP KV store
    (torch.distributed.TCPStore), got FileStore.

Pin the helper to true from `register_platform()`, which restores the 0.27.1
rendezvous. The patch is a no-op on vLLM versions without the helper, and it
can be dropped once aiter's IPC exchange accepts a `FileStore`.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(vllm): give ATOM's attention backends vLLM 0.28's customize_spec

vLLM 0.28 added `AttentionBackend.customize_spec(spec)` and calls it while
building KV cache specs, from `v1/worker/gpu/attn_utils.py`,
`v1/worker/gpu_model_runner.py` and
`model_executor/layers/attention/attention.py`. ATOM's vLLM-facing backend
surfaces are duck-typed rather than derived from vLLM's `AttentionBackend`, so
they do not inherit the default and both MiniMax-M3 and Kimi-K3 aborted with

    AttributeError: type object 'SparseMHAIndexerBackend' has no attribute
    'customize_spec'

Add the identity implementation to the four standalone surfaces, which covers
every backend derived from them. The surfaces that already derive from a vLLM
class inherit it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(vllm): build the MLA DCP manager vLLM 0.28 expects on the layer

vLLM 0.28 moved MLA's DCP collectives behind a new `MLADCPManager`
(`vllm/v1/attention/ops/dcp_utils.py`; the symbol does not exist in 0.27.1).
vLLM's own `MLAAttention.__init__` builds one when `impl.dcp_world_size > 1`,
and `MLACommonMetadataBuilder.__init__` reads it back off the registered layer
and asserts the type.

`AttentionForVllmMLA` reports `dcp_world_size = -1` on purpose, to keep vLLM's
DCP paths out of ATOM's decode kernels. vLLM therefore never built a manager,
and every DCP>1 MLA run -- Kimi-K3 DSpark DCP8 in PR CI -- aborted on that
assert while building attention metadata, before a single token was served.

Build the manager for the registered layer instead, with the arguments vLLM
passes: ATOM never feeds MLA a quantized query, so the query keeps the layer
dtype. The metadata builder is its only consumer here -- it hands the manager
the chunked-prefill workspace for the context KV all-gather.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(vllm): keep vLLM's PAD slots when localizing DSpark draft KV under DCP

vLLM 0.28 taught `_prepare_dflash_inputs_kernel` to store `PAD_SLOT_ID` for
draft rows that must not receive KV: the rejected suffix of a speculative
context span, which `num_valid_ctx` now separates from `num_ctx`, and any row
whose block-table entry is the null block after a sliding-window eviction.

The DCP patch re-derives both slot mappings after the kernel runs, to express
them in rank-local terms. Under 0.28 that unconditional rewrite resurrected
exactly those rows into real slots, so the Kimi-K3 DSpark draft would write KV
over live context -- physical block 0 included.

Whether a row is writable at all is the kernel's decision, and the DCP rewrite
has no business overturning it. Keep every `PAD_SLOT_ID` it produced and
re-localize only the rows it addressed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(v4): accept both spellings of aiter's inverse-RoPE scale layout

`atom/models/deepseek_v4.py` calls `inverse_rope_group_quant(scale_layout=…)`
since #1989. aiter main renamed that keyword from `scale_shuffle`, so on any
image whose aiter predates the rename -- the released ATOM images freeze the
aiter of their build day -- DeepSeek-V4-Flash fails to load with

    TypeError: inverse_rope_group_quant() got an unexpected keyword argument
    'scale_layout'

`scale_shuffle=False` and `scale_layout="row"` request the same row-major
`[S, G, Ks]` scale, so a thin adapter keeps the call site working on both. It
lives in a new `atom/model_ops/aiter_compat.py` so the model file only changes
which module it imports from, leaving the traced graph untouched, and so the
next aiter rename has somewhere obvious to go. Layouts the old aiter cannot
express raise rather than silently emitting the wrong one.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(vllm): harden 0.28 compatibility adapters

Co-authored-by: Cursor <cursoragent@cursor.com>

* revert(aiter): drop the inverse-RoPE and MLA head-fold compat shims

aiter main (c6ef26a5) now carries both API changes the shims bridged:
`inverse_rope_group_quant` takes `scale_layout` and no longer accepts
`scale_shuffle`, and `aiter/mla.py` defines `_fold_seqlen_indptr` and uses
it for the DCP round-robin fold. Both adapters are dead code against it.

Drop `atom/model_ops/aiter_compat.py` and call aiter directly from
`deepseek_v4.py`; drop `_aiter_supports_mla_dcp_head_fold()` and restore
`mla_dcp_kernel_num_heads` to the width it computed before, so a persistent
DCP decode takes the gathered width as-is again. Both files are byte-identical
to main after this.

`test_dense_mla_decode_pads_gathered_dcp_heads` asserted a width of 128 that
main's own code never produces for its inputs -- 12 heads over DCP8 gathers to
96, a multiple of 16, which a persistent decode folds without padding. The
assertion now states 96 and a zero pad, which is what runs.

This raises ATOM's minimum aiter to one carrying both changes; images that
froze an older aiter need a rebuild.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: perzhang <perzhang@amd.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant