Skip to content

feat(deepseek-v41): native DeepSeek-V4.1-Flash — text, vision, paged runtime and DSpark - #2314

Merged
valarLip merged 60 commits into
mainfrom
feat/deepseek-v41-flash
Sep 21, 2026
Merged

valarLip merged 60 commits into
mainfrom
feat/deepseek-v41-flash

Conversation

@valarLip

Copy link
Copy Markdown
Collaborator

What

DeepSeek-V4.1-Flash as its own model package rather than a reshape of V4.
48 commits, staged P00–P10b; recipes/DeepSeek-V4.1-Flash.md carries the stage
table and states what each stage accepted.

  • Model — atom/models/deepseek_v41/: CSA2 topology, single-pass mHC, MoE,
    Engram, vision, and the DSpark draft.
  • Ops — atom/model_ops/deepseek_v41/ and
    atom/model_ops/attentions/deepseek_v41/: compressor, paged index scorer,
    packed attention, grouped output projection, rotary. V4's BF16 attention and
    inverse RoPE are reused rather than copied.
  • Runtime — paged cache and request-state lifecycle, a whole-forward target
    graph, and a decode step that is the draft's replay plus the target's.
  • Protocol — chat, tools and numeric reasoning effort; native image requests
    with chunked prefill and request-owned embedding lifetime.
  • Quantization — group32 FP8 / W4A8 with E8M0 scales end to end
    (atom/model_ops/blockscale*), latterly on the CDNA4 microscaling matrix core.

Where it stands

P00–P10b complete. P11–P14 pending: candidate-only indexing and Engram
fusion, optional CED decoder replay, optional encoder replay with a persistent
global cache, and validated distributed/deployment combinations. Draft for that
reason.

Two accepted tradeoffs and one open regression, all already documented:

  • DSpark zero-shot raw quality is an open regression, not waived. Expanded
    fixed-length comparison over 128 questions: 90/128 without speculation versus
    80/128 with, nine gains and 19 losses, paired 95% bootstrap [−0.156, 0.0],
    exact McNemar p = 0.087. The matched five-shot comparison is identical per
    document, which does not cancel it. docs/deepseek_v41_dspark.md has the
    full account including what was ruled out.
  • Small-chunk multimodal quality (P08) and MoE precision (P09), per the recipe.

Accuracy

GSM8K, 1319 questions, fewshot 3, TP4 + whole-expert EP, DSpark-5, BF16 KV.
Six same-code runs span 0.9075–0.9287, σ ≈ 0.8pp.

A single run inside that band is the bar. Anything finer needs per-question
pairing against a same-code control: two runs of identical code disagree on
about 6.5% of questions, and a single pair's exact McNemar reaches p = 0.0026
between two runs of the same commit. The judge is whether cross-arm discordance
leaves the same-code range, not the p-value and not the headline.

Test plan

  • bash .github/scripts/run_unit_tests.sh — 8222 passed, 158 skipped,
    3 xfailed
  • tests/models/deepseek_v41/ and tests/attentions/deepseek_v41/ on GPU
  • tests.attentions.deepseek_v41.validate_runtime, eager and --graph —
    55 scheduler events each; chunked, reordered, forked, replayed from a
    missing graph, preempted, cancelled and slot-recycled routes each produce
    the tokens an uninterrupted run did
  • tests.models.deepseek_v41.validate_dspark_lifecycle --production --graph
    — reaches the verify step with accepted_drafts {'5': 10}. It still stops
    at its own restores >= 1; that assertion is pre-existing and tracked
    (the file's checkpoint interval is snapped off against the 256-column
    prefix-cache hash block, and repairing that is necessary but not
    sufficient)
  • GSM8K 1319 in the band above
  • Distributed and deployment combinations beyond TP4 + EP on MI355X (P14)

Notes for review

Arithmetic is judged against the published model by the ATOM_DSV41_REFERENCE
unit tests; the runtime validates unsupported combinations before it loads
weights. AITER is pinned in the recipe.

The two most recent commits additionally want ROCm/aiter#5703: without it the
fused dual RMSNorm keys its Triton build on the token count, which costs
113–133 ms of device idle per new shape at serving time.

Allow V4.1 to preserve the reference 64-row probability-rounding boundary while retaining the V4 default. Cover the 192/640-entry cases and single sink semantics.
Add the 40-layer offline model, owner-scoped attention caches, cross-chunk compression and host Engram preparation. Keep shared-expert TP partials in FP32 and preserve all checkpoint quantization steps.

Validation: 264 affected tests pass; controlled real-weight comparison covers five fixtures and 20 prefill/decode calls with exact residual/logit parity. Independent cross-backend quality acceptance remains open and ModelRunner registration stays gated.
Call the existing paged BF16 prefill/decode kernels and SWA ring writer.
Keep private layer windows and shared global owners in one eager row pool,
and prepare CSR inputs without concatenating the full global KV history.
Restore the generic helper API so V4 attention code matches main unchanged.

Use D512 integration fixtures and separate real backend error checks from
quantized graph composition. Validation: 136 tests, Black/Ruff, official
TileLang HIP operator comparisons, and independent 40-layer TP8 diagnostics.
Full-model numerical/quality acceptance and serving integration remain open.
Use the existing GPU kernel with FP32 frequency views, repeated batch positions, and in-place updates for strided callers. Keep the eager CPU reference path.

Validate batch/position/alias contracts and the accepted FMA rounding boundary; 23 rotary, attention, and model tests pass. The known task-score tradeoff is accepted for this substitution; this does not mark P04 complete.
Keep scaled group products and split-K partials in FP64 until output
conversion so cancellation does not discard small contributions before A8
quantization. Retain native weight storage and BF16 group dot products.
Decode E8M0 scale products exactly, including code 0 and NaN code 255.

Validation: all 64 independent cancellation/scale-boundary cases pass for
FP8 and FP4 weights, BF16/FP32 outputs, and split/unsplit reductions. The
full V4.1 suite passes 220 tests without skips. This isolated correction
does not close the remaining P04 model-quality acceptance criteria.
Evaluate intermediate normalization reductions and affine products in FP32
before one BF16 conversion at activation-quantization boundaries. Reuse the
existing V4/AITER kernel for final and index-key normalization, with a small
leaf adapter that owns CPU, empty and strided inputs without TP or collective
configuration. Keep compressor and KV normalization eager after their task
regressions; query normalization awaits its own task validation.

Validation: the index-key integration reproduces all 148 independent numerical
records from the accepted inverse baseline. ARC, HellaSwag, code, five-shot
GSM8K and all 651 Chinese items preserve every per-document metric. All 220
V4.1 tests pass without skips. The index leaf is 8.22-12.16x faster in isolated
warm operator timing; no serving speedup is claimed. P04 overall remains open.
Disable AITER custom all-reduce through its supported initialization API
before creating offline parallel groups. Mixed BF16/FP32 reductions with
changing token counts and rank skew can otherwise return incorrect values.
Keep collective policy in the offline entry point, separate from model math.

Add a model-free regression using exactly representable integer sums, mixed
dtypes and the failing tail sizes. The pinned TP4 run passes all 240 calls
per rank with zero error. The isolated custom-kernel correction is diagnostic
only; it is not enabled by this change. P04 quality acceptance remains open.
Allow full-logit callers to select the scored suffix before final normalization
and vocabulary projection. Every input token still executes through the model
and updates attention state. Validate the suffix boundary explicitly.

The regression compares cropped and full projections through actual V4
attention, then requires the next decode output and cache position to agree.
It also poisons unused full-cache storage. The full V4.1 suite passes 220 tests;
completed checkpoint/task evaluations use this API without truncating context.
No end-to-end serving performance claim is made.
Separate the pinned checkpoint oracle, harness adapter, restartable TP response
cache and paired report from runtime model code. Reject nonfinite outputs,
identity mismatches and unpaired documents, and cover long index pruning.

Document completed inverse/index-key integration, pinned dependency provenance
and the remaining P04 quality gates; P05 has not started.

Validation: 220 V4.1 tests passed, zero skips; optional dependency control
2 passed and 13 skipped; Ruff, Black and diff checks passed.
Keep query normalization eager after the complete Chinese regression loses
four raw and three normalized answers against the accepted baseline.

Record the source-audited high-margin replay and exact rational WKV calibration:
the first causal KV discrepancy is a reference FP32 rounding boundary, while
V4 attention matches at the probed token with identical inputs.

Preserve the original quality gates: P04 remains open and P05 has not started.
Keep image embeddings under request leases across chunk boundaries, prefix
forks and preemption; send descriptors after worker acknowledgement and
release the final consumer on completion or abort.

Bound small GPU wo_a/mHC projections to the validated 128-row arithmetic
inside operator helpers. Preserve native CPU, single-row and larger shapes,
and reuse the V4 attention/inverse-RoPE paths. Record the explicitly accepted
Chinese quality difference and measured padding costs.

Validation: TP4 two/three-request graph runtime, 44 affected tests, frozen
37-case numerical corpus, paired task reports, and sequential native versus
integrated runtime measurements. See docs/deepseek_v41_small_chunks.md.
Five-token DSpark for DeepSeek-V4.1-Flash: draft stages under `mtp.*` with
their own rolling-window KV, block drafting in one pass, confidence-scheduled
widths, and target verification. TP4, BF16 caches, text requests.

Bringing it up surfaced three places where V4.1 had reimplemented something V4
already owned, and each reimplementation was a source of defects rather than a
difference in the model:

Routed experts are V4's. The two models' MoE is the same layer -- sqrtsoftplus
routing with a selection-only bias, renormalized top-k, W4A8 group-32 MXFP4
experts, and the routing weight applied to the activation before quantization
-- so `models/deepseek_v41/moe.py` is now a subclass of V4's that flattens the
offline caller's batch dimension and declares `bias_vl`, the one V4.1-only
tensor. The eager expert loop and the P09 AITER adapter behind
`expert_backend` are both gone, along with the capture exclusion the eager loop
needed: `FusedMoE` is capturable at every shape, so the decode FFN and its
collectives now enter the graphs with everything else.

Weights load through `model_loader.loader.load_model` like every other model's.
The native reader kept a second set of rename rules for the same checkpoint,
and this branch spent a debugging session on the ways the two drifted: a
missing `.scale` rename dropped 46,410 tensors, a substring `.gate.bias` rule
renamed `.gate.bias_vl` to a parameter that does not exist, and shared experts
were rewritten into a fused slot the model does not allocate. `weights.py`
keeps only what reads raw tensors for the pinned-reference tests and the Engram
mmap tables; it no longer writes parameters. The offline example builds the
multimodal class unconditionally, as serving does, so the vision tensors have
an owner rather than being skipped.

Both model classes also drove every child's `process_weights_after_loading`
from a root-level hook, which `load_model`'s own parent-first traversal then
repeated -- applying each quantized Linear's post-load shuffle twice. V4 has no
such hook, and neither does this now.

mHC uses AITER's kernels, as V4 and vLLM's ROCm V4.1 both do. `mhc_post` drops
straight into `expand_residual`. The pre seam cannot use V4's single
`mhc_pre`, because the collapse applies the pre-mix carried from the previous
sublayer while the one computed here goes to the next, so it drives AITER's two
stages directly and recovers the pre gate and the delayed collapse in two
Triton kernels, following vLLM's solution to the same problem. Against the
torch bodies they replace, the worst gap is 0.18 BF16 ulp of the value it
belongs to, with one perturbation per output group proving each column is
measured.

The removed `expert_backend` option left references in the calibration
identity, in eleven scripts, and in four documents describing kernels that no
longer exist; those are swept here.
…ts that hid

DSpark's block drafter now captures through the shared `DraftGraph` instead of
declining it. The metadata builder publishes `state_slot_out` at running_bs and
binds the serving allocation at capture, as V4 does, so the captured graph bakes
the window address the drafter will replay against; `block_backbone` reads that
published mapping rather than padding a per-step tensor. Draft graphs now equal
`runner.capture_sizes`, and the `hipErrorCapturedEvent` that killed capture does
not reproduce.

Three more places where V4.1 reimplemented something V4 owns:

The LM head is V4's `ParallelHead`. The local one projected in FP32 and called
`group.all_gather` with `use_custom` defaulting to False, which is the only
reason it worked: V4.1's backend declared `use_custom_all_reduce = False`, the
runner therefore built the RCCL environment, and AITER's `all_gather` asserts on
a missing `ca_comm` rather than falling back. That declaration is gone too --
V4.1 runs the same collectives as every other model -- and the prefill last-token
slice is the head's again instead of being repeated in `compute_logits`.

The draft's Markov and confidence heads are V4's. Only the naming is V4.1's: this
checkpoint calls the low-rank tables `markov_head.{embed,head}` where V4's head
calls them `markov_w1`/`markov_w2`, so one rename rides on top of the backbone's
mapper. Sampling is V4's `forward_head`, and `draft_hidden` returns V4's seam
shape, which removes a width check that V4 documents as belonging to the shared
surface rather than to the model.

The offline entry builds the serving `Config` and holds itself to the same
admission rules. That is also what lets it set the current atom config, without
which the shared MoE cannot be constructed at all -- `load_offline_model` has
been unable to build a model since the MoE swap.

Three asserts encoded properties that no longer hold, and each was hiding the
next one behind it:

- capture must leave the cache untouched. It cannot: capture declares page 0 and
  the STATE slots as scratch and scribbles on them, which is V4's contract
  ("the data is throwaway"), safe because capture precedes admission. Now asserts
  that pages it never named are unchanged.
- paged and offline logits must be bit-identical. They cannot be: V4's fused MoE
  is not reproducible on TP4+EP, and the same forward twice already differs by as
  much as the two paths do. The floor is now measured per chunk by running the
  oracle on a second, independently advanced cache; across 17 chunks the ratio is
  0.77 to 2.20. Bisected to the commit that adopted `FusedMoE`; the eager expert
  loop it replaced was reproducible by construction.
- the dense graphs must number `3 * layers * 3`. The decode FFN entered the
  graphs when the eager loop's capture exclusion went away, so it is 12 per layer.
  Asserts the invariant -- a whole multiple, so no layer silently falls back to
  eager -- and reports the multiple.

`validate_runtime` passes for the first time since that commit, and
`validate_dspark_checkpoint` runs to completion for the first time at all.
…t a request at a time

CSA2's compressor walked one request at a time, pooled in torch, and carried the
incomplete group across forwards as a `CompressorTail` that speculation had to
stage and commit. It now runs V4's plan-driven `fused_compress_attn` over the
whole batch, and the incomplete group lives where V4 keeps it: a per-request ring
in the arena, written by `update_compressor_states`.

The tail machinery is gone -- `read_tail`, `write_tail`, `stage_tail`, the
`tail_values` / `tail_scores` fields -- because the ring is `K_pool +
speculative_tokens` wide, so a rejected round's writes land past the window the
next round reads. `TentativeState` is left staging Engram history, which is the
one thing that cannot be reconstructed from KV. `layout_id` goes to `state-v2`:
every other term was already equal, so a v1 image would restore into fields that
no longer mean what it holds.

Two things the kernel needed that V4 does not:

- `latent_out` echoes the post-norm PRE-RoPE latent. CSA2's index key projects
  from it, and V4 has no use for it because its indexer compresses its own
  projection. Without the echo the eager pool+norm would have to stay, and the
  two disagree by a BF16 ulp -- enough to change which rows the top-k selects.
- `rotated_out` echoes the rotated latent for the FP4 pool, whose interleaved
  layout the kernel's BF16 scatter cannot write. Rotating a second time in torch
  rounds differently, and the FP4 grid turns that into whole-level flips: 10.3%
  of elements, measured.

Both are constexpr'd away when unset, so V4's kernel is unchanged.

Two bugs the new ring-slack test found, both in this rewrite:

The write plan retained a request's last `K_pool` positions. That is sound only
when `K_pool` covers a whole draft span -- V4's is 8 and 128 against a span of at
most 8, so its ring stays fully populated and the cut never binds. CSA2's is the
ratio, 1 or 2, against a span of 6: four positions per round were written by
nobody, and a rejection read them back. `make_compress_plans` takes `extra_write`
and every caller states it: the speculative window on a fwd that can be rolled
back, 0 on prefill, which nothing rejects. V4's plans are byte-identical under
this, decode and prefill both, measured across every shape it runs.

`compress_batch` returned before updating the ring when a forward crossed no
compression boundary -- a single token at an even position, at ratio 2. Its
projections still owe the ring, or the round that does cross one reads a position
nobody wrote.

Reverting either fix alone turns the test red.

The metadata builder owns fixed-address plan buffers under V4's names and hands
the plans to `begin_step`; an isolated caller still gets private ones. The
paged-versus-private comparison stays bit-exact at `rtol=0, atol=0` for both
BF16 and packed, which is what says the two addressings agree.

`kv_cache_dtype=bf16` no longer round-trips the main pool and the index key
through FP4. That round-trip existed to make bf16 and fp4 numerically identical;
they are now distinct configurations, and a bf16 accuracy number does not predict
fp4.

Capture asks for a tentative step a full window in, rather than a one-token step
at position 0: `BatchStep.decode` is otherwise false for a speculative bucket, so
the capture recorded the prefill FFN while replay ran `decode_ffn` eagerly.
`dense_graphs_per_layer` goes 9.0 to 12.0 on TP4 DSpark with only that switched.

Accuracy: acceptance rate, four workloads, three repeats per arm, all overlapping
with inconsistent signs. Case 0 came back with no overlap at n=3, so it was rerun
at n=6 per arm in one session: its own run-to-run range is 0.17 to 0.35, an order
of magnitude above the difference between arms, and the sign of that difference
flips between two arms of the same code. `/app/logs_claude/p2_gate/RECORD.md`.
CSA2's index keys were BF16 rows inside the PAGE, read by a tiled torch scorer
that materialized one request's keys at a time. They are now FP8 in aiter's
MFMA preshuffle layout, and a decode step's top-k comes from
`deepgemm_fp8_paged_mqa_logits` + `top_k_per_row_decode` over the plane itself
-- no key is materialized and the per-request walk is gone from that path.

The plane had to leave the PAGE to get there. All three indexer kernels address
a block as `id * stride` with one stride, so an index plane strided by the PAGE
can only be addressed a whole page's rows at a time. Region-major -- one dense
array per owner, indexed by the same block id, which is the arrangement
DeepSeek-V4's indexer already has ("rides the same block table as its main
compressed KV") -- makes that stride the rows' own.

Half of that justification was taken away later in the same change: candidates
became dense-plus-mask rather than a paired block table, and dense scoring
makes a block id a PAGE id again. Region-major is kept because it is what a
16-row block id needs, and that is the >=128k option the measurements below say
not to foreclose; because it removes the writer's fake `cache_stride`, which
only worked because the in-block offsets never reached the end of a PAGE; and
because it was already landed and green.

A PAGE unit is therefore several byte regions rather than one run -- its main
page plus its rows in each index plane -- so a checkpoint image is cut over a
heterogeneous destination stream. `plan_segmented_copy` already takes one and
V4's MLA path already builds one.

`kv_cache_block_size` goes to 256, V4's value, now that the tile no longer ties
it to the page. The argument is not padding: a field's alignment comes out at
zero for every shape CSA2 runs and last-page fragmentation is ~0.1% of the
pool. It is the block table -- `[max_num_seqs, max_model_len / block_size]`
int32 held twice and copied to the device every forward, 8.4 MB at 256 against
134 MB at 16 for a million-token context. What it costs is prefix reuse below
256 tokens, which stops matching at all.

`MQA_LOGITS_PRESHUFFLE_ROWS` moves from `atom/config.py` to
`pool_layout/v4_pool_fields.py`: three things enforce it now -- a block size, a
pooled row count, and this plane -- and a layout declaration is what all three
can reach.

Two restrictions, both tested rather than assumed:

- `(index fp8, large_position)` is refused at startup. The paged top-k breaks
  ties by the smaller row and has no other mode, and ReLU makes exact ties
  common rather than rare, so this pair would score every decode step under a
  policy it does not implement.
- An FP8 plane needs a PAGE holding whole 16-row tiles. The ratio-2 owners are
  what make that a statement about twice the PAGE, which is why 256 and not 16.

Measured. The paged-versus-private comparison gains a third arm, `(bf16 main,
fp8 index)`, and stays bit-exact at `rtol=0, atol=0` through its decode steps.
That is stronger than equal selections: the sparse attention kernel sums a
row's prefix in the order it is handed, so an equal output means the two
scorers agreed on the rows and on their order. The indexer's query is not the
attention's, so moving the indexer to FP8 arithmetic does not weaken it. The
schedule is 69 tokens now because the old 22 fit inside one 256-token PAGE, and
a request inside a single page cannot tell a page-order mistake from correct
addressing.

Reverting either fix alone turns exactly the two FP8 arms red: writing with
`preshuffle=False` while reading with `True`, and making `restrict_to_candidates`
a no-op.

Accuracy, three arms, isolating the PAGE size from the format:

  DSpark acceptance, case 0, n=6      GSM8K 3-shot, 100 questions
  HEAD block 16    0.2348             91/100
  block 256        0.2551             89/100   McNemar p=0.754 vs HEAD
  block 256 + fp8  0.2105             90/100   McNemar p=1.000 vs HEAD

Every pair overlaps, the sign of the difference is not consistent, and each
acceptance arm's own run-to-run range is 0.19-0.48 -- four to twenty times the
difference between arms. The paired GSM8K discordance is 7-10 of 100, against
23-33 for the recorded zero-shot form of the same probe.

Neither says the precision did not drop. Acceptance rate measures how far the
draft and the target agree, and GSM8K at 100 questions carries a ~3pp binomial
error before the engine's own non-determinism; an FP8 index plane is a
reduction from BF16, and an effect below these probes' resolution on these
workloads is what they cannot see. No self-compare arm was run, so the 7-10
discordant pairs are not separated into the change and the floor.

The capacity is not a noisy number: the index plane costs 330 B/token instead
of 640, and the pool holds 243,558 PAGEs against 220,084, +10.7%.

Prefill keeps the tiled scorer over `cp_gather_indexer_k_quant_cache`. It is
not in the captured graph, and the candidate mask on a `[tokens, total_keys]`
prefill shape is the one place dense-plus-mask is not cheap.

One thing the kernel does not say about itself:
`deepgemm_fp8_paged_mqa_logits` refuses to compile below
`index_n_heads * index_head_dim = 4096` (`/app/logs_claude/probe_v41_mqa_dims.py`).
The published config is 32x128, exactly on the floor, and `wq_b` is replicated
so TP does not divide it. No startup guard: nine points is not a
characterization, and the configuration that ships is on the right side of it.
valarLip and others added 5 commits September 20, 2026 05:32
…itical path

Wires the kernels from the previous commit into the serving path, and with them
go the two blocking readbacks a decode step had, the per-request numpy and the
staging buffers that carried it all.

`EngramBatch` is this forward as the kernels see it, built from the very tensor
the model embeds so the rows Engram looks up cannot drift from the tokens it
runs. On the UVA path `prepare` stops building `EngramRequest` objects
altogether -- one `tuple(tokens)` per request, a Python int per token, then
hashed once per layer as a dict key, which a 16k prefill chunk pays in full
before a row is read -- and `_stage_uva` reads the indices where they were
produced instead of staging them through pinned memory.

The cursor moves the same way. A verify step's candidates were `T + 1` rows
staged per request on the host out of a history that had just been read back;
they are now written by `engram_cursor_rows` into the plane `commit_tentative`
already selects from, and an ordinary step's row goes straight into the cursor.
`advance_cursor`, `stage_history` and their pinned buffers are left to the host
path, which is the only one that still has a caller.

`prepare_state` gains `histories=False` for the caller that needs neither: with
a device batch nothing on the host reads this step's committed history, so it
does not need this step's verdict either. The rows ship without waiting and are
judged on the next call. That was the last blocking D2H in a decode step -- it
was 1.03 ms of host block, of which 42 us was device idle, and defending it as
"the guard has to be synchronous" was wrong twice over: the guard survives, and
a request whose state is not where the scheduler wants it is still named, one
step late, by a hard failure. Its own position check goes with it, replacing a
64-iteration Python walk over `span.position` with one pass over the batch.

Two smaller things the same path was paying: a recycled slot's state and cursor
reset cost two launches per fresh request and now cost two per step, and the
Engram staging buffers are no longer page-locked under UVA, where nothing
writes their host half once a dummy step zeroes the device one.

Measured on V4.1-Flash TP4 DSpark-5 at bs=64, against the same trace shape
throughout: the decode period falls 48.6 ms to 42.6 and GPU idle 5.5 ms to
0.4, with the busy span unmoved, which is what says only the host path
changed. `hipMemcpyWithStream` in a decode window goes 1 to 0.

Do not read finer than that from these traces. Instance to instance, the same
build moves 0.84 ms in GPU busy span alone, so the last of these changes are
below the noise and are here for what they delete, not for a number.

Gates: GSM8K 1319 questions at fewshot 3 gives flexible 0.9136, inside this
configuration's 0.9121-0.9318 band and among its other samples (0.9227 before
any of this, then 0.9181, 0.9128, 0.9189, 0.9166, 0.9181). Acceptance 0.5269
against 0.5164 on the arm before. No stale-state guard fired over 1319
questions. Unit suite 8122 passed, 158 skipped, 3 xfailed.
…, not two dozen

Before the draft's three stages run, every step assembles the same three
things: which rolling-window rows the block may attend to, the block's input
state, and each stage's view of its window. All of it was spelled out in
torch, over `[B, ring]` and `[B, width]` tensors where a launch costs more
than the arithmetic -- 26 of them per drafting step, four an `arange`
rebuilding a compile-time constant.

The attention indices are a closed form. Ring slot `j` holds the newest
position congruent to it, `anchor - ((anchor - j) mod ring)`; a slot is
readable when that position has been written, is not in the future and is
still inside the sliding window; past the ring the block's own rows index
themselves. One kernel writes a request's row of the list directly, and the
slot-to-position map it derives on the way stays a register rather than the
tensor a sub/sub/mod chain used to materialise.

Mind the remainder there: Triton's `%` keeps the sign of the DIVIDEND where
torch's follows the divisor, and `anchor - slot` is negative over most of a
partly-filled ring. Normalising it is the difference between the right rows
and silently the wrong ones, so `test_partly_filled_ring_wraps_the_way_torch_does`
drives positions below `ring` on purpose.

The block's input is an anchor followed by `width - 1` copies of one noise
token, embedded, broadcast over the mHC streams and paired with a pre-mix.
The embedding is a TP all-reduce, and it was carrying `B * width` rows of
which only `B` ever differ: the noise row is fixed once the checkpoint is
loaded. Embedding it once and keeping it takes `width` out of the collective,
and the state is written straight out instead of being expanded and copied.

The stages' windows are the same rows of the same tensor at different layers,
so they come from one broadcast gather rather than three, each of which had
been preceded by its own copy of the same `.long()` cast.

All three are index arithmetic and copies, so they are held to bit-exactness
against the chains they replace -- 324, 30 and 12 shape combinations -- not
the weaker standard a fusion that rounds would get.

Measured on MI355X (TP4, DSpark, 5 spec tokens) the prologue's torch-side
launches fall 26.0 -> 22.3 per step and 152.4us -> 138.4us, with `arange`,
`FillFunctor`, `remainder`, `compare_scalar` and `vectorized_gather` gone from
the region entirely. Scale it honestly: `propose_dspark` is ~7% of a decode
step, so this is ~0.4% end to end and below the run-to-run floor. It is here
because the work was redundant, not because the step got faster. GSM8K
3-shot holds at 0.9143-0.9249 over four runs against a 1pp noise floor.
…orch's index path

`read_windows` asks for the same rows of the same tensor at three layers, which
is one broadcast index -- and torch serves that through the generic advanced-
indexing kernel, walking the arithmetic per element as an opaque 2-byte type.
At a decode batch it spent 13.2us moving 6.3MB.

Nothing about the read needs that generality: a row is `head_dim` wide and
contiguous at both ends. One program per output row makes it a coalesced load
and store, and the read lands at 11.6us. That is not a bandwidth win and should
not be read as one -- the bytes are the same, and this access pattern tops out
near 520 GB/s whatever the block size or grid, which is where `copy_` lands on
the same scattered source. What it buys is the generic path's per-element index
arithmetic.

The `.long()` in front of it goes too. The slot table is int32, the kernel
widens each index itself, and casting the whole tensor first was a copy of it:
5.5us and a launch, for nothing.

Over the cluster this leaves, measured on MI355X (TP4, DSpark, 5 spec tokens)
against the same workload: 21.73 -> 6.67 launches per drafting step and
130.3us -> 44.9us, of which this commit is the last 2.6 launches and 7.1us.

Held to bit-exactness against `window[layers[:, None], slots[None, :]]` over 24
shape and dtype combinations -- there is nothing here to round. GSM8K 3-shot
over four runs lands 0.9174-0.9249 against a baseline's 0.9067-0.9204 on the
same build; the two overlap and four samples cannot resolve the gap between
their means, so this is "no regression", not an improvement.
DSv4.1-Flash quantizes its dense projections to E4M3 with 32x32 E8M0 blocks,
and native_quant_linear was serving them by widening both operands to BF16,
one 32-element scale group per tile, and scaling the tile sum afterwards.
CDNA4 has an instruction for exactly this shape. Handing the E8M0 codes to
tl.dot_scaled issues v_mfma_scale_f32_32x32x64_f8f6f4 and lets one tile span
several scale groups; at m=4096 the second o_proj half goes 2730us -> 245us.
BK below 64 is what to watch: Triton lowers dot_scaled to a BF16 emulation
there, which is what the old formulation effectively was.

The instruction is not free. Its block accumulator carries about 15 bits
against the block's largest term rather than the 24 an FP32 sum would, so a
group whose product is far below its neighbours rounds away before the
accumulator sees it. Against an FP32 reference the error is 1.6e-5 to 1.8e-5
of the result's peak, consistently, and it scales with the result rather than
with the element -- rows that cancel to near zero carry the same absolute
error as the rest, which no per-element rtol can express. The four tests that
compare an FP32 output against a full-precision reference now bound on the
magnitude of that output. At the BF16 the model actually asks for, the two
formulations are indistinguishable: both sit on the BF16 rounding floor.

Accumulation drops to FP32 throughout, including W4A8. The cancellation test
keeps its shape with the spread at 2**23, which is the widest an FP32
accumulator carries exactly.

Split-K comes back for FP8, gated on the launch grid against the CU count and
nothing else. A 16-workgroup kv_a at m=1 more than halves with it; a
288-workgroup shared w1 at m=128 loses 1.3x to the same split. The earlier
measurement that said split-K never pays was taken with the weight pinned in
MALL, where decode reads a third of its real time.

M becomes a runtime argument. It is the token count, so specializing on it
recompiled the kernel for every prefill chunk length: thirteen lengths
measured thirteen variants at 170-620ms each, now three.

Two kernels rather than one. FP8 spans several scale groups per tile through
the matrix core; FP4 unpacks nibbles and scales a single group by hand. Folded
together they duplicated the pointer bookkeeping and put an if in three
places. _select_fp8_tile was swept through native_quant_linear so each
candidate ran with its real split count, and with the inputs rotated.

Measured on MI355X, us, inputs rotated past MALL:

  shape                    m=1    m=32   m=128  m=1024  m=4096
  wq_a   (1280, 5120)      6.3     7.3    11.6    27.1    60.0
  wq_b   (32768, 1280)     9.6    10.3    18.4    89.5   293.9
  kv_a   (512, 5120)       4.9     5.6     7.7    17.4    29.8
  o_a    (8192, 4096)     13.0    11.6    21.5    59.7   193.5
  o_b    (5120, 8192)     11.7    18.2    30.8    93.8   244.5
  ffn.w1 (2304, 5120)      7.5     8.2    12.6    46.8   107.7
  ffn.w2 (5120, 2304)      7.7     8.1    12.6    32.3    86.5

Four things were tried and are not here, each noted where it would go: an L2
swizzle (nothing), the weight pre-transposed to (K, N) (slower -- (N, K) gives
longer contiguous runs), folding the split-K reduction into the kernel behind
an arrival counter (1.7-2.9x slower), and loading one weight-scale row and
broadcasting it (crashes TritonAMDGPUAccelerateMatmul).
Per layer the prologue ran q_norm, kv_norm, rope(query) and rope(kv) as four
kernels. The two norms read two slices of one `wqkv_a` output and neither
reads the other's result; the two rotations share a position line. Both pairs
are one launch now.

NORMS. `DualRMSNormMXFP8` wraps aiter's `fused_dual_rmsnorm_mxfp8_quant`,
which was written for this shape -- independent widths, one shared token
count -- and hands back the `(e4m3, e8m0 group-32)` pair `wq_b` reads without
quantizing again. It sits beside `DualRMSNorm` and, like it, is not an
nn.Module: it reads the weights off the two norms it is handed.

In one steady decode step the two `add_rmsnorm_quant` families lose exactly
40 launches each, 527.3 us of device time becomes 276.5, and the step falls
33.29 -> 33.03 ms. A microbenchmark predicted a third of that: back-to-back
repetition is the friendliest case those two kernels ever get.

The draft's attention takes the same pair. `context_keys` now takes the
latent already normed -- `forward` gets it beside the query, and the caller
that has only a latent norms it itself.

ROTATIONS. `RotaryEmbedding.pair` dispatches to the cached ABI's two-channel
entry, whose argument list is the one-channel one plus a tensor and whose two
head counts may differ, which is what it is for. 98 one-channel launches per
step become 18, plus 40 two-channel ones; rope device time falls 710.8 ->
443.6 us.

That 267 us does NOT show up on the clock: the step is busy 32.5 of its 33.0
ms, so the rotations were already overlapping something. The reason to keep
this is the launch count and the call site, not a speedup.

Neither fusion is bit-identical, and both differences sit on the rounding
floor: 1 FP8 code in 5,242,880 and 34 rotated lanes in 1,572,864, none of
them off by more than one ULP.

QUANTIZE, separately. `quantize_fp8_kernel` and `quantize_fp4_kernel`
declared `SIZE` a `tl.constexpr`, and `SIZE` is the token count -- so every
prefill chunk length and every batch size compiled its own kernel, 14,949
variants of the FP8 one in this box's cache. In an 8-second profiled window
the device sat idle 610 ms in front of that kernel, 65% of all idle there, in
stalls of 42-52 ms holding nothing but `hipModuleLoadData`. `SIZE` only
bounds the tiles. Output is bit-identical and the time unchanged, measured
with both spellings in ONE process -- across processes a JIT'd kernel drifts
by more than the effect being measured.

That axis was fixed once already for the GEMM ("M becomes a runtime
argument"); these two are what the sweep missed. The dual-norm kernel has it
too, on aiter's side, where the fix is not merged yet: until it is, this
commit trades one source of per-shape builds for another.

Gates. V4.1 and DSpark unit tests: 422 passed, 36 skipped, including a new
one pinning `pair`'s contract. `validate_runtime` passes eager and under CUDA
graphs, 55 events each. `validate_dspark_lifecycle --production --graph`
still reports `accepted_drafts {'5': 10}` and still stops at its own
pre-existing `restores >= 1`. Draft acceptance over 300 questions and 35,000
drafted tokens, two runs a side: 49.79 / 50.21 before, 48.91 / 50.29 after.
GSM8K 1319 reads 0.9272, 0.9189 and 0.9249 across the three steps, against a
same-code band of 0.9075-0.9287 measured the same day.
@github-actions

Copy link
Copy Markdown
Contributor

🏷️ CI Guide

Runs automatically on every eligible PR before approval:

  • ✅ Pre Checkin: Black, Ruff, catalog schema validation, non-GPU unit tests

Heavy model tests:

  • ✅ Run after the PR is approved and Pre Checkin passes
  • ✅ Run immediately when an approval review is submitted
  • ✅ Can be requested before approval with labels
Label Tests
ci:full Run all heavy PR model tests: native ATOM, vLLM, and SGLang
ci:atom Run native ATOM model accuracy tests
ci:vllm Run ATOM vLLM OOT model accuracy tests
ci:sglang Run ATOM SGLang model accuracy tests

Heavy jobs are skipped when the PR is not approved and no matching ci:* label is present.
Add labels via the sidebar or gh pr edit 2314 --add-label <label>

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remaining comments which cannot be posted as a review comment to avoid GitHub Rate Limit

ruff

⚠️ [ruff] <I001> reported by reviewdog 🐶
Import block is un-sorted or un-formatted

from atom.model_ops.attentions.deepseek_v41.backend import (
DeepseekV41MetadataBuilder,
)
from atom.model_ops.attentions.deepseek_v41.cache import PagedAttentionCache
from atom.model_ops.attentions.pool_layout.v41_pool_geometry import V41PoolGeometry
from atom.spec_decode.draft_graph import DraftGraph, StagedInput


⚠️ [ruff] <I001> reported by reviewdog 🐶
Import block is un-sorted or un-formatted

from types import SimpleNamespace
import numpy as np
import pytest
import torch
from atom.model_engine.engram_runtime import EngramInputPreparer
from atom.model_ops.attentions.deepseek_v41.metadata import RequestSpan
from tests.model_ops.test_engram import make_runtime
from .dspark_state_audit import RuntimeStateAudit


⚠️ [ruff] <I001> reported by reviewdog 🐶
Import block is un-sorted or un-formatted

import os
from contextlib import contextmanager
from copy import deepcopy
from types import SimpleNamespace
import numpy as np
import pytest
import torch
from atom.model_ops.deepseek_v41.mhc import (
SinglePassHCState,
apply_sublayer,
predict_mixes,
)
from atom.model_ops.deepseek_v41.moe import Expert, Router, weighted_swiglu
from atom.model_ops.engram import CompressedTokenizer, EngramConfig, NgramHashMapping
from atom.model_ops.engram_layer import EngramOp
from torch import nn


⚠️ [ruff] <I001> reported by reviewdog 🐶
Import block is un-sorted or un-formatted

from atom.models.deepseek_v41.weights import (
CheckpointReader,
build_weight_manifest,
checkpoint_schema,
)
from atom.config import get_hf_config


⚠️ [ruff] <I001> reported by reviewdog 🐶
Import block is un-sorted or un-formatted

from unittest.mock import Mock
import pytest
from aiter import QuantType
from atom.models.deepseek_v41 import dspark, model


⚠️ [ruff] <I001> reported by reviewdog 🐶
Import block is un-sorted or un-formatted

import pytest
import torch
from atom.model_ops.blockscale import quantize_fp8
from tests.models.deepseek_v41 import oracle_kernels
from atom.model_ops.layernorm import RMSNorm


⚠️ [ruff] <I001> reported by reviewdog 🐶
Import block is un-sorted or un-formatted

import pytest
import torch
import torch.nn.functional as F
from atom.model_ops.deepseek_v41.projections import (
grouped_output_projection,
hc_projection,
)


⚠️ [ruff] <I001> reported by reviewdog 🐶
Import block is un-sorted or un-formatted

from atom.model_ops.blockscale import native_quant_linear
from safetensors import safe_open


⚠️ [ruff] <I001> reported by reviewdog 🐶
Import block is un-sorted or un-formatted

import pytest
import torch
from atom.model_ops.deepseek_v41.rotary import RotaryEmbedding


⚠️ [ruff] <I001> reported by reviewdog 🐶
Import block is un-sorted or un-formatted

from types import SimpleNamespace
import pytest
import torch
from atom.model_ops.attentions.deepseek_v41.cache import PagedAttentionCache
from atom.model_ops.attentions.deepseek_v41.checkpoints import StateCopies
from atom.model_ops.attentions.deepseek_v41.metadata import RequestSpan
from atom.model_ops.attentions.pool_layout.v41_pool_geometry import V41PoolGeometry


⚠️ [ruff] <I001> reported by reviewdog 🐶
Import block is un-sorted or un-formatted

from atom.model_ops.attentions.deepseek_v41.packed_attention import packed_decode
from atom.model_ops.attentions.deepseek_v41.packed_rows import pack_rows
from atom.model_ops.blockscale import quantize_fp8
from atom.model_ops.v4_kernels import sparse_attn_v4_paged_decode


⚠️ [ruff] <I001> reported by reviewdog 🐶
Import block is un-sorted or un-formatted

import io
import os
import sys
from types import SimpleNamespace
import numpy as np
import pytest
import torch
from atom.model_engine.multimodal_runtime import (
embedding_indices,
multimodal_cache_seed,
)
from atom.models.deepseek_v41.image_processing import (
DeepseekV41ImageProcessor,
image_token_types,
preprocess_image,
)
from atom.models.deepseek_v41.multimodal import DeepseekV41MultimodalModel
from atom.models.deepseek_v41.vision import Aligner, ViT
from atom.models.deepseek_v41.weights import (
CheckpointReader,
build_weight_manifest,
checkpoint_schema,
)
from PIL import Image
from transformers import AutoTokenizer
from atom.config import get_hf_config
from atom.entrypoints.openai.chat_encoders import load_custom_message_encoder
from .reference import FIXTURES


⚠️ [ruff] <I001> reported by reviewdog 🐶
Import block is un-sorted or un-formatted

import itertools
import json
import os
from types import SimpleNamespace
import numpy as np
import pytest
import torch
from atom.models.deepseek_v41.weights import (
CheckpointReader,
WeightSpec,
build_weight_manifest,
checkpoint_schema,
)
from safetensors.torch import save_file
from atom.config import get_hf_config
from .reference import FIXTURES


⚠️ [ruff] <I001> reported by reviewdog 🐶
Import block is un-sorted or un-formatted

import argparse
import json
import os
from collections import Counter, deque
from contextlib import nullcontext
from pathlib import Path
from unittest.mock import patch
import torch
from tests.models.deepseek_v41.validate_dspark_runtime import diagnostic_config
from transformers import AutoTokenizer
from atom.config import (
CompilationConfig,
Config,
CUDAGraphMode,
DSparkConfig,
SpeculativeConfig,
)
from atom.model_engine.model_runner import ModelRunner
from atom.model_engine.scheduler import Scheduler
from atom.model_engine.sequence import Sequence, SequenceStatus
from atom.sampling_params import SamplingParams


⚠️ [ruff] <I001> reported by reviewdog 🐶
Import block is un-sorted or un-formatted

import argparse
import copy
import hashlib
import json
import os
from contextlib import nullcontext
from pathlib import Path
from unittest.mock import patch
import torch
from atom.models.deepseek_v41.config import validate_runtime_config
from tests.attentions.deepseek_v41.benchmark_runtime import run_case
from transformers import AutoTokenizer
from atom.config import (
CompilationConfig,
Config,
CUDAGraphMode,
DSparkConfig,
SpeculativeConfig,
)
from atom.model_engine.model_runner import ModelRunner

valarLip and others added 9 commits September 20, 2026 15:10
…cribe the code

Nine thousand of this branch's lines were scaffolding, dead state and a lab
notebook. None of it was reachable from a served request.

Dead code, introduced here and unreachable:

- `atom/distributed/rccl.py` and the switch that selected it. `96e4fb7e4`
  removed the one `use_custom_all_reduce = False` declaration -- V4.1 runs the
  same collectives as every other model -- but left behind the module, the
  ClassVar no subclass ever overrode, and the branch whose condition is
  therefore always false. Dropping the ClassVar also gives `accept_output_buffer`
  back the comment it was inserted in front of.
- `tentative_limits`. Its only reader was a commented-out `_assert_async`, while
  the buffer was still allocated per cache and written per staged request: dead
  state costing a pinned allocation and a host write every step.
- `LayerAttentionSpec.index_key_owner`, handed `kv_owner` a second time by a
  positional call and read nowhere.
- `EngramConfig.kernel_size`, parsed from a key no config declares.
- `DIAGNOSTIC175` in the model runner, whose own comment said to revert it
  before any acceptance run.
- `fixtures/text_inputs.json`, which had no consumer before this commit either.

`NativeQuantization` was a frozen dataclass whose four fields were literals in
its own constructor, whose one production caller discarded the instance, and
whose test asserted those literals back. It is `validate_native_quantization`
now -- the five refusals were always the content -- and the test exercises each
refusal instead of the constants.

Twenty-four `torchrun` scripts under `tests/` were never collected by pytest and
were reachable only from each other. They are gone, keeping the two the guides
name as manual gates -- `validate_runtime` and `validate_dspark_lifecycle`, the
second with its one borrowed helper inlined. Five test modules went with them
because their subject was a probe rather than the product. Three parameter grids
fell from 200 cases to 71 without an assertion changing: a batch axis over a
kernel that runs one program per row, and a YaRN axis that re-asserted one
shape-independent sentence at eight shapes.

The guides were written in the stage numbers of their own development. They name
capabilities now, point at neither a private log directory nor patches that were
never committed, and no longer carry superseded measurement rounds -- the open
DSpark quality regression is one table and a ruled-out list instead of eight
rounds in chronological order, and the small-chunk report, which described a
padding path that has since been removed, is folded into the vision guide as the
one tradeoff it still documents. Two referenced files that never existed are
gone, `CLAUDE.md` and the model support guide have V4.1 entries, and four
environment variables are documented, one of them newly registered rather than
read raw.

8071 unit tests pass.
…experts

The CPU pre-checks runner has neither AITER nor a GPU, and a collection error
there is not one skipped file: pytest reports `Interrupted` and the job runs
nothing at all. Under that runner's conditions this branch produced 18
collection errors against main's 1.

Two causes, both fixed here. The new test modules imported AITER-backed modules
at module scope and guarded them per function, which is too late -- the import
has already failed. And `tests/test_pd_pp.py` / `tests/test_pp.py` put a bare
`aiter` into `sys.modules` at import time so they can exercise CPU wrapper
logic, then leave it there; from that point `pytest.importorskip("aiter")`
resolves the name, the skip does not fire, and the real `from aiter import ...`
inside some later module raises ImportError during collection. Those two now
drop the top-level name once their own imports are bound. The submodules stay,
because the connectors reach for those lazily, at call time.

Modules with host-side tests worth keeping import the AITER pieces where they
are used rather than at the top, so those tests still run: `test_engram.py`
covers 34 of its 44 on a CPU runner and `test_runtime_contract.py` 29 of 31,
where both previously ran none. The rest skip at module scope, each checked for
anything a CPU runner could have executed.

Ruff is clean across every file this branch adds or touches. Beyond the
mechanical fixes -- `.exception()` rather than `.error(..., exc_info=True)`,
bare `raise`, a mutable default argument, `itertools.pairwise` -- nine
`except Exception: pass` handlers now say what they swallowed. A teardown is
allowed to fail; swallowing it without a word is what makes a hung shutdown
unexplainable.

Separately, V4.1 no longer refuses TP > 1 without expert parallelism. The
weight manifest already sharded routed experts along their intermediate
dimension when EP is off, and `FusedMoE` already took its own TP path there;
admission refused the combination on the strength of never having been run with
it. TP4 without EP scores 0.9204 on the 1,319-question GSM8K set, inside the
band six same-code EP runs span. Their throughput has not been compared.

Also removed: a `checkpoint_loader` dispatch hook with nothing behind it, which
V4.1's own runtime documents it does not use; and the router/expert and
unit-table reference implementations, which move out of `atom/model_ops/` into
the tests that are their only callers.
… pages

Whether the device lookup can be used was decided per rank in three places,
and two of them can disagree across a TP group: a rank whose head shard is
empty because the head count does not divide the world, and a rank whose
`cudaHostRegister` is refused. Either one takes the host path alone while its
peers go on to the all-gather that ends `_stage_uva`, and the job hangs there.
The local answer is now reduced with MIN over the group's gloo communicator
before anyone acts on it -- once, at load -- and a rank outvoted after
registering gives its pages back.

Registration leaked as well. A table's values and its scales are registered
separately, and the scales being refused returned before the range was
recorded, so `disable_uva` found nothing to release and the values stayed
page-locked for the life of the process: 23 GiB on DeepSeek-V4.1-Flash at TP4,
unswappable and unaddressed. `enable_uva` now unregisters what it took, and
keeps the page-aligned bases so that `disable_uva` gives back what went in
rather than deriving the same offsets a second time. `EngramHost.shutdown`
releases too, ahead of the mappings those pages belong to.

Six tests cover the two, each also run with its fix reverted to see it fail.
They need no AITER, so they run on the CPU pre-checks runner rather than
skipping there.

Also gone: a `reshape` in the staging loop to the shape its input already has,
under both a sharded and an unsharded group.
Two test files conflicted, and both conflicts belong entirely to #2315, which
reverts the Kimi-K3 P/D disaggregation of #2154. Every block in them --
`_MLABase` and the transfer-region cases in `test_gdn_state_relocation.py`,
the deferred-decode case in `test_mtp_deferred_status_queue.py` -- was added
by #2154, and this branch had only adapted them to
`KVTransferTensors.set_block_count` and to `decode_spans`. The revert is the
right side to take: the code those tests cover is gone. Both files now match
main byte for byte.

`set_block_count` itself survives the revert with its callers intact. No
construction site passes `num_blocks`, and the case pinning that a backend
cannot pass one is still there.

Gates on the merged tree. The pre-checks suite, which is what CI runs, is
6346 passed / 652 skipped / 0 errors. The full suite against a GPU is 8092
passed and 4 failed, all four in `test_qwen4_exp_mtp.py` and all four
inherited: they pass at the merge base and on this branch before the merge,
and fail on main alone. #2311 made the QSA GQA kernel's `kv_splits` adapt to
the token count, so a batched call and a per-row call now reduce in different
orders -- and that test compares them at rtol=0, with the largest difference
one bf16 ULP and the selected tokens identical. Nothing caught it upstream
because the case needs a GPU and pre-checks runs without one.
Fuse FP32 normalization, weighted dot product, gate activation, masking, and residual addition into a Triton kernel. Preserve the torch fallback and expose ATOM_ENGRAM_FUSED_GATE for comparison.

Add FP64 reference, dtype, masking, fallback, and graph replay coverage.
Snapshot token history before cursor updates and stage layer 1 and 14 hash, UVA lookup, and TP all-gather on one graph-captured side stream. Use private IPC synchronization and raw input buffers; consumers wait for complete embeddings.

Enable ATOM_ENGRAM_OVERLAP by default, with an explicit 0 opt-out and main-stream collective fallback where unsupported.

Validation: 149 regression tests; TP4 main/side collective concurrency with two graph buckets and 120 changing replays; actual BS16 trace; full GSM8K 5-shot strict 1211/1319 and flexible 1210/1319 with fusion enabled.
… after

The CPU pre-checks runner installs torch from the CPU index, and that wheel
does not bring Triton. Four modules reached Triton-backed operators at module
scope and declared their GPU requirement below that import, as a `pytestmark`
-- which runs after the import has already raised. A raise during collection
is not one skipped file: pytest reports `Interrupted` and the job runs nothing
at all.

All four are GPU-only, so moving the guard above the imports costs no
coverage. It names the GPU rather than Triton because the GPU is the
precondition; reaching Triton is a consequence of it.

`test_packed_cache.py` keeps its `import triton` for the kernel it defines,
below the guard. Its import block stays sorted: a statement between two
blocks leaves each of them to be ordered on its own.

Two more modules still fail to collect there, and both hold real host-side
tests, so the fix is to move an import boundary in the source rather than to
skip: `engram_runtime` binds its host prefetch and staging to the device
hashing kernels, and `deepseek_v41/dspark` keeps two pure-torch helpers beside
Triton kernels. Neither is in this commit.
Decorate DeepseekV41RuntimeModel with support_torch_compile and use AITER
torch_compile_guard boundaries for serving attention, Engram staging and
consumption, and shape-tuned delayed mHC. Runtime eager and compiled calls
share the same path while preserving decoder input hooks for DSpark.

Keep mHC fake shapes beside its implementation, preserve the no-post
residual alias, and admit level 3 with FULL CUDA Graphs. Cover dynamic
batches, live request state, hooks, and mHC output schemas.

Validation: 299 passed, 36 skipped in the V4.1 regression suite before the
CPU-only import guard adjustment. Final TP4 BS16 validation aborted with
GPU memory access faults; the cause remains unresolved. Earlier GSM8K
results apply to the implementation before this guard refactor.
The CPU pre-checks runner has no Triton, and `engram_runtime` bound its host
prefetch and staging to the device hashing kernels while `engram_lookup` kept
a Triton gather beside the host table. Both took the whole pytest session down
during collection rather than skipping one file, twice.

Engram now lives under `model_ops/engram/`. Its top level -- `mapping`,
`tables`, `host` -- imports numpy and torch and nothing else; `device/` holds
every module that reaches Triton. `EngramHost` takes the class that does the
device lookup rather than importing it, so a rank with no class gets the host
path and a CPU-only test says so by passing nothing.

That moves six UVA registration and group-vote tests from "CI never runs them"
to "CI runs them every time": page-locking a shard and voting on the outcome
are cudart and Gloo calls, and only the hashing and the gather ever needed a
GPU. `test_import_boundary.py` executes the rule in a child interpreter.

`deepseek_v41/dspark` gives up its three pure-torch helpers to `draft_block`
for the same reason, and `test_dspark.py` its one Triton case.

Two formatting fixes ride along in `deepseek_v41/config.py` and `model.py`.
Neither is from this change -- both reproduce on b3cb26c alone -- but one of
those files is in this diff and the other fails a repo-wide job.

No behaviour change. GSM8K 1319 5-shot 0.9318/0.9325 against 0.9181/0.9189 on
the same HEAD, inside this model's same-code spread; DSpark acceptance 49.47%
against 49.52%.
@valarLip

Copy link
Copy Markdown
Collaborator Author

@ZhangLirong-amd heads up — cf7e9eaea moves the whole Engram package, including the two files you added in 01855ed96 / 3fa486f0b. Nothing in them changed except imports and two attribute reads, but the paths did, so a local branch will conflict on rebase.

before after
model_ops/engram.py model_ops/engram/mapping.py
model_ops/engram_lookup.py model_ops/engram/tables.py (host half)
model_engine/engram_runtime.py model_ops/engram/host.py
model_ops/engram_hash.py model_ops/engram/device/hashing.py
model_engine/engram_staging.py model_ops/engram/device/staging.py
model_ops/engram_fused.py model_ops/engram/device/gate.py
model_ops/engram_layer.py model_ops/engram/device/layer.py

The rule the directory now carries: everything at the top level of model_ops/engram/ imports without Triton; everything under device/ may not. That is what the CPU pre-checks job was failing on — engram_runtime reached the hash kernels at module scope, so the runner took the whole pytest session down during collection instead of skipping one file. tests/model_ops/engram/test_import_boundary.py executes the rule now, in a child interpreter, so it fails as one test rather than as Interrupted.

Two things that touch your code directly:

  • EngramStaging.__init__ now takes the device lookup (EngramUva) instead of the host; it reads uva.hash_tables / uva.row_ids and gets the host from uva.host. The rest of the class is unchanged.
  • EngramHost no longer holds hash_tables / _row_ids / _ids_staging / overlap as its own attributes — they belong to the lookup. overlap and hash_tables remain as None-safe properties, so existing readers did not have to change.

Also worth knowing separately from this PR: b3cb26caa landed with the Ruff and Black jobs already red (models/deepseek_v41/config.py over-long line, models/deepseek_v41/model.py import block) and with test_compilation.py::test_runtime_guard_with_live_steps_and_aux_hooks[3] failing. I fixed the two formatting ones here because one of those files is in this diff and the other fails a repo-wide job. The test failure is untouched and reproduces on b3cb26caa alone — it carries skipif(not torch.cuda.is_available()), so the CPU job skips it and only a GPU box shows it.

…ks for

C408 on `test_compilation.py:174`. The file arrived in b3cb26c and is new in
this PR, so reviewdog's diff context covers all of it, not just the lines a
later commit touches -- which is why the rule lands here rather than there.

`kwargs.update(...)` below reads a dict either way.
@valarLip
valarLip marked this pull request as ready for review September 21, 2026 08:50
Skip block-table uploads when mappings and padding are unchanged. Refresh on row changes or device allocation replacement while preserving fixed buffer addresses.

Build indptrs for all configured ratios in one Triton launch and select cursor rows using int32 indices directly. Cover ratio subsets, graph replay, and block-table invalidation with regression tests.
@valarLip
valarLip merged commit b605361 into main Sep 21, 2026
45 checks passed
@valarLip
valarLip deleted the feat/deepseek-v41-flash branch September 21, 2026 11:17
valarLip added a commit that referenced this pull request Sep 22, 2026
…2356)

MiniMax-M3-MXFP8 is an fp8 (1, 32) checkpoint launched with
--online_quant_config, so every one of its attention and dense linears
matched the native group32 source shape while the model-wide online_quant
flag was set, and LinearBase raised at construction. All 14 nightly
MiniMax-M3-MXFP8 jobs have failed since #2314; run 35522504237 is the last
green one.

The flag says the run was launched with online quant, not that this layer
is affected -- an excluded layer keeps its checkpoint weight. Ask per layer
via will_online_requant(), and let a (1, 32) source that online quant will
overwrite drop the native layout for the generic per_1x32 path, whose
scale geometry it already matches. A (32, 32) source has no such twin and
keeps the error, as does every other rejection the gate made.

should_stream_online_quant was already computing this predicate inline
alongside its streaming preconditions; it now calls the named one.
ganyi1996ppo added a commit that referenced this pull request Sep 28, 2026
…strides

Two defects in the flydsl index-score integration, both found by
measurement after rebasing onto main.

1. The ubatch work-map buffer was sized with the default cp_world=1
   while `_build_ubatch_metadata` builds it with the real CP pair. The
   shard's dispatch is denser than the whole context's, so the CP map is
   LONGER, not shorter -- 128 rows against 64 at bs=8, max_block=64 --
   and the build rejects the buffer:

       ValueError: work_map: expected packed [at least 128, 2] int32

   So ubatch plus indexer CP could not start. It fails loudly rather
   than writing out of bounds, which is why it never showed as
   corruption. The non-ubatch buffer next to it already passed the pair;
   this just does the same.

2. The three score strides were `tl.constexpr`. `S_HEAD` is
   `tokens * local`, so it moves with the batch, and specializing on it
   cost a ~430ms Triton recompile for every distinct decode batch size:
   8 of 8 sizes recompiled with a cold Triton cache, 0 of 8 after. That
   is the same reason `LOCAL_BLOCKS` and `GLOBAL_BLOCKS` are already
   runtime arguments in this kernel. Triton still folds the
   divisible-by-16 and equal-to-1 cases, which is where the addressing
   win is; nothing about the layout handling changes.

tests/test_minimax_m3_index_topk.py, tests/model_ops/test_indexer_cp_parity.py
and tests/test_paged_attention_dispatch.py pass (216 passed, 21 skipped).

Not verified end to end on a server: main now imports `aiter.topk_select`
(#2244, #2314) and the aiter installed here does not have it, so the
engine cannot start from this base at all -- with or without this branch.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants