Repository navigation
feat(deepseek-v41): native DeepSeek-V4.1-Flash — text, vision, paged runtime and DSpark - #2314
Conversation
Allow V4.1 to preserve the reference 64-row probability-rounding boundary while retaining the V4 default. Cover the 192/640-entry cases and single sink semantics.
Add the 40-layer offline model, owner-scoped attention caches, cross-chunk compression and host Engram preparation. Keep shared-expert TP partials in FP32 and preserve all checkpoint quantization steps. Validation: 264 affected tests pass; controlled real-weight comparison covers five fixtures and 20 prefill/decode calls with exact residual/logit parity. Independent cross-backend quality acceptance remains open and ModelRunner registration stays gated.
Call the existing paged BF16 prefill/decode kernels and SWA ring writer. Keep private layer windows and shared global owners in one eager row pool, and prepare CSR inputs without concatenating the full global KV history. Restore the generic helper API so V4 attention code matches main unchanged. Use D512 integration fixtures and separate real backend error checks from quantized graph composition. Validation: 136 tests, Black/Ruff, official TileLang HIP operator comparisons, and independent 40-layer TP8 diagnostics. Full-model numerical/quality acceptance and serving integration remain open.
Use the existing GPU kernel with FP32 frequency views, repeated batch positions, and in-place updates for strided callers. Keep the eager CPU reference path. Validate batch/position/alias contracts and the accepted FMA rounding boundary; 23 rotary, attention, and model tests pass. The known task-score tradeoff is accepted for this substitution; this does not mark P04 complete.
Keep scaled group products and split-K partials in FP64 until output conversion so cancellation does not discard small contributions before A8 quantization. Retain native weight storage and BF16 group dot products. Decode E8M0 scale products exactly, including code 0 and NaN code 255. Validation: all 64 independent cancellation/scale-boundary cases pass for FP8 and FP4 weights, BF16/FP32 outputs, and split/unsplit reductions. The full V4.1 suite passes 220 tests without skips. This isolated correction does not close the remaining P04 model-quality acceptance criteria.
Evaluate intermediate normalization reductions and affine products in FP32 before one BF16 conversion at activation-quantization boundaries. Reuse the existing V4/AITER kernel for final and index-key normalization, with a small leaf adapter that owns CPU, empty and strided inputs without TP or collective configuration. Keep compressor and KV normalization eager after their task regressions; query normalization awaits its own task validation. Validation: the index-key integration reproduces all 148 independent numerical records from the accepted inverse baseline. ARC, HellaSwag, code, five-shot GSM8K and all 651 Chinese items preserve every per-document metric. All 220 V4.1 tests pass without skips. The index leaf is 8.22-12.16x faster in isolated warm operator timing; no serving speedup is claimed. P04 overall remains open.
Disable AITER custom all-reduce through its supported initialization API before creating offline parallel groups. Mixed BF16/FP32 reductions with changing token counts and rank skew can otherwise return incorrect values. Keep collective policy in the offline entry point, separate from model math. Add a model-free regression using exactly representable integer sums, mixed dtypes and the failing tail sizes. The pinned TP4 run passes all 240 calls per rank with zero error. The isolated custom-kernel correction is diagnostic only; it is not enabled by this change. P04 quality acceptance remains open.
Allow full-logit callers to select the scored suffix before final normalization and vocabulary projection. Every input token still executes through the model and updates attention state. Validate the suffix boundary explicitly. The regression compares cropped and full projections through actual V4 attention, then requires the next decode output and cache position to agree. It also poisons unused full-cache storage. The full V4.1 suite passes 220 tests; completed checkpoint/task evaluations use this API without truncating context. No end-to-end serving performance claim is made.
Separate the pinned checkpoint oracle, harness adapter, restartable TP response cache and paired report from runtime model code. Reject nonfinite outputs, identity mismatches and unpaired documents, and cover long index pruning. Document completed inverse/index-key integration, pinned dependency provenance and the remaining P04 quality gates; P05 has not started. Validation: 220 V4.1 tests passed, zero skips; optional dependency control 2 passed and 13 skipped; Ruff, Black and diff checks passed.
Keep query normalization eager after the complete Chinese regression loses four raw and three normalized answers against the accepted baseline. Record the source-audited high-margin replay and exact rational WKV calibration: the first causal KV discrepancy is a reference FP32 rounding boundary, while V4 attention matches at the probed token with identical inputs. Preserve the original quality gates: P04 remains open and P05 has not started.
Keep image embeddings under request leases across chunk boundaries, prefix forks and preemption; send descriptors after worker acknowledgement and release the final consumer on completion or abort. Bound small GPU wo_a/mHC projections to the validated 128-row arithmetic inside operator helpers. Preserve native CPU, single-row and larger shapes, and reuse the V4 attention/inverse-RoPE paths. Record the explicitly accepted Chinese quality difference and measured padding costs. Validation: TP4 two/three-request graph runtime, 44 affected tests, frozen 37-case numerical corpus, paired task reports, and sequential native versus integrated runtime measurements. See docs/deepseek_v41_small_chunks.md.
Five-token DSpark for DeepSeek-V4.1-Flash: draft stages under `mtp.*` with their own rolling-window KV, block drafting in one pass, confidence-scheduled widths, and target verification. TP4, BF16 caches, text requests. Bringing it up surfaced three places where V4.1 had reimplemented something V4 already owned, and each reimplementation was a source of defects rather than a difference in the model: Routed experts are V4's. The two models' MoE is the same layer -- sqrtsoftplus routing with a selection-only bias, renormalized top-k, W4A8 group-32 MXFP4 experts, and the routing weight applied to the activation before quantization -- so `models/deepseek_v41/moe.py` is now a subclass of V4's that flattens the offline caller's batch dimension and declares `bias_vl`, the one V4.1-only tensor. The eager expert loop and the P09 AITER adapter behind `expert_backend` are both gone, along with the capture exclusion the eager loop needed: `FusedMoE` is capturable at every shape, so the decode FFN and its collectives now enter the graphs with everything else. Weights load through `model_loader.loader.load_model` like every other model's. The native reader kept a second set of rename rules for the same checkpoint, and this branch spent a debugging session on the ways the two drifted: a missing `.scale` rename dropped 46,410 tensors, a substring `.gate.bias` rule renamed `.gate.bias_vl` to a parameter that does not exist, and shared experts were rewritten into a fused slot the model does not allocate. `weights.py` keeps only what reads raw tensors for the pinned-reference tests and the Engram mmap tables; it no longer writes parameters. The offline example builds the multimodal class unconditionally, as serving does, so the vision tensors have an owner rather than being skipped. Both model classes also drove every child's `process_weights_after_loading` from a root-level hook, which `load_model`'s own parent-first traversal then repeated -- applying each quantized Linear's post-load shuffle twice. V4 has no such hook, and neither does this now. mHC uses AITER's kernels, as V4 and vLLM's ROCm V4.1 both do. `mhc_post` drops straight into `expand_residual`. The pre seam cannot use V4's single `mhc_pre`, because the collapse applies the pre-mix carried from the previous sublayer while the one computed here goes to the next, so it drives AITER's two stages directly and recovers the pre gate and the delayed collapse in two Triton kernels, following vLLM's solution to the same problem. Against the torch bodies they replace, the worst gap is 0.18 BF16 ulp of the value it belongs to, with one perturbation per output group proving each column is measured. The removed `expert_backend` option left references in the calibration identity, in eleven scripts, and in four documents describing kernels that no longer exist; those are swept here.
…ts that hid
DSpark's block drafter now captures through the shared `DraftGraph` instead of
declining it. The metadata builder publishes `state_slot_out` at running_bs and
binds the serving allocation at capture, as V4 does, so the captured graph bakes
the window address the drafter will replay against; `block_backbone` reads that
published mapping rather than padding a per-step tensor. Draft graphs now equal
`runner.capture_sizes`, and the `hipErrorCapturedEvent` that killed capture does
not reproduce.
Three more places where V4.1 reimplemented something V4 owns:
The LM head is V4's `ParallelHead`. The local one projected in FP32 and called
`group.all_gather` with `use_custom` defaulting to False, which is the only
reason it worked: V4.1's backend declared `use_custom_all_reduce = False`, the
runner therefore built the RCCL environment, and AITER's `all_gather` asserts on
a missing `ca_comm` rather than falling back. That declaration is gone too --
V4.1 runs the same collectives as every other model -- and the prefill last-token
slice is the head's again instead of being repeated in `compute_logits`.
The draft's Markov and confidence heads are V4's. Only the naming is V4.1's: this
checkpoint calls the low-rank tables `markov_head.{embed,head}` where V4's head
calls them `markov_w1`/`markov_w2`, so one rename rides on top of the backbone's
mapper. Sampling is V4's `forward_head`, and `draft_hidden` returns V4's seam
shape, which removes a width check that V4 documents as belonging to the shared
surface rather than to the model.
The offline entry builds the serving `Config` and holds itself to the same
admission rules. That is also what lets it set the current atom config, without
which the shared MoE cannot be constructed at all -- `load_offline_model` has
been unable to build a model since the MoE swap.
Three asserts encoded properties that no longer hold, and each was hiding the
next one behind it:
- capture must leave the cache untouched. It cannot: capture declares page 0 and
the STATE slots as scratch and scribbles on them, which is V4's contract
("the data is throwaway"), safe because capture precedes admission. Now asserts
that pages it never named are unchanged.
- paged and offline logits must be bit-identical. They cannot be: V4's fused MoE
is not reproducible on TP4+EP, and the same forward twice already differs by as
much as the two paths do. The floor is now measured per chunk by running the
oracle on a second, independently advanced cache; across 17 chunks the ratio is
0.77 to 2.20. Bisected to the commit that adopted `FusedMoE`; the eager expert
loop it replaced was reproducible by construction.
- the dense graphs must number `3 * layers * 3`. The decode FFN entered the
graphs when the eager loop's capture exclusion went away, so it is 12 per layer.
Asserts the invariant -- a whole multiple, so no layer silently falls back to
eager -- and reports the multiple.
`validate_runtime` passes for the first time since that commit, and
`validate_dspark_checkpoint` runs to completion for the first time at all.
…t a request at a time CSA2's compressor walked one request at a time, pooled in torch, and carried the incomplete group across forwards as a `CompressorTail` that speculation had to stage and commit. It now runs V4's plan-driven `fused_compress_attn` over the whole batch, and the incomplete group lives where V4 keeps it: a per-request ring in the arena, written by `update_compressor_states`. The tail machinery is gone -- `read_tail`, `write_tail`, `stage_tail`, the `tail_values` / `tail_scores` fields -- because the ring is `K_pool + speculative_tokens` wide, so a rejected round's writes land past the window the next round reads. `TentativeState` is left staging Engram history, which is the one thing that cannot be reconstructed from KV. `layout_id` goes to `state-v2`: every other term was already equal, so a v1 image would restore into fields that no longer mean what it holds. Two things the kernel needed that V4 does not: - `latent_out` echoes the post-norm PRE-RoPE latent. CSA2's index key projects from it, and V4 has no use for it because its indexer compresses its own projection. Without the echo the eager pool+norm would have to stay, and the two disagree by a BF16 ulp -- enough to change which rows the top-k selects. - `rotated_out` echoes the rotated latent for the FP4 pool, whose interleaved layout the kernel's BF16 scatter cannot write. Rotating a second time in torch rounds differently, and the FP4 grid turns that into whole-level flips: 10.3% of elements, measured. Both are constexpr'd away when unset, so V4's kernel is unchanged. Two bugs the new ring-slack test found, both in this rewrite: The write plan retained a request's last `K_pool` positions. That is sound only when `K_pool` covers a whole draft span -- V4's is 8 and 128 against a span of at most 8, so its ring stays fully populated and the cut never binds. CSA2's is the ratio, 1 or 2, against a span of 6: four positions per round were written by nobody, and a rejection read them back. `make_compress_plans` takes `extra_write` and every caller states it: the speculative window on a fwd that can be rolled back, 0 on prefill, which nothing rejects. V4's plans are byte-identical under this, decode and prefill both, measured across every shape it runs. `compress_batch` returned before updating the ring when a forward crossed no compression boundary -- a single token at an even position, at ratio 2. Its projections still owe the ring, or the round that does cross one reads a position nobody wrote. Reverting either fix alone turns the test red. The metadata builder owns fixed-address plan buffers under V4's names and hands the plans to `begin_step`; an isolated caller still gets private ones. The paged-versus-private comparison stays bit-exact at `rtol=0, atol=0` for both BF16 and packed, which is what says the two addressings agree. `kv_cache_dtype=bf16` no longer round-trips the main pool and the index key through FP4. That round-trip existed to make bf16 and fp4 numerically identical; they are now distinct configurations, and a bf16 accuracy number does not predict fp4. Capture asks for a tentative step a full window in, rather than a one-token step at position 0: `BatchStep.decode` is otherwise false for a speculative bucket, so the capture recorded the prefill FFN while replay ran `decode_ffn` eagerly. `dense_graphs_per_layer` goes 9.0 to 12.0 on TP4 DSpark with only that switched. Accuracy: acceptance rate, four workloads, three repeats per arm, all overlapping with inconsistent signs. Case 0 came back with no overlap at n=3, so it was rerun at n=6 per arm in one session: its own run-to-run range is 0.17 to 0.35, an order of magnitude above the difference between arms, and the sign of that difference flips between two arms of the same code. `/app/logs_claude/p2_gate/RECORD.md`.
CSA2's index keys were BF16 rows inside the PAGE, read by a tiled torch scorer
that materialized one request's keys at a time. They are now FP8 in aiter's
MFMA preshuffle layout, and a decode step's top-k comes from
`deepgemm_fp8_paged_mqa_logits` + `top_k_per_row_decode` over the plane itself
-- no key is materialized and the per-request walk is gone from that path.
The plane had to leave the PAGE to get there. All three indexer kernels address
a block as `id * stride` with one stride, so an index plane strided by the PAGE
can only be addressed a whole page's rows at a time. Region-major -- one dense
array per owner, indexed by the same block id, which is the arrangement
DeepSeek-V4's indexer already has ("rides the same block table as its main
compressed KV") -- makes that stride the rows' own.
Half of that justification was taken away later in the same change: candidates
became dense-plus-mask rather than a paired block table, and dense scoring
makes a block id a PAGE id again. Region-major is kept because it is what a
16-row block id needs, and that is the >=128k option the measurements below say
not to foreclose; because it removes the writer's fake `cache_stride`, which
only worked because the in-block offsets never reached the end of a PAGE; and
because it was already landed and green.
A PAGE unit is therefore several byte regions rather than one run -- its main
page plus its rows in each index plane -- so a checkpoint image is cut over a
heterogeneous destination stream. `plan_segmented_copy` already takes one and
V4's MLA path already builds one.
`kv_cache_block_size` goes to 256, V4's value, now that the tile no longer ties
it to the page. The argument is not padding: a field's alignment comes out at
zero for every shape CSA2 runs and last-page fragmentation is ~0.1% of the
pool. It is the block table -- `[max_num_seqs, max_model_len / block_size]`
int32 held twice and copied to the device every forward, 8.4 MB at 256 against
134 MB at 16 for a million-token context. What it costs is prefix reuse below
256 tokens, which stops matching at all.
`MQA_LOGITS_PRESHUFFLE_ROWS` moves from `atom/config.py` to
`pool_layout/v4_pool_fields.py`: three things enforce it now -- a block size, a
pooled row count, and this plane -- and a layout declaration is what all three
can reach.
Two restrictions, both tested rather than assumed:
- `(index fp8, large_position)` is refused at startup. The paged top-k breaks
ties by the smaller row and has no other mode, and ReLU makes exact ties
common rather than rare, so this pair would score every decode step under a
policy it does not implement.
- An FP8 plane needs a PAGE holding whole 16-row tiles. The ratio-2 owners are
what make that a statement about twice the PAGE, which is why 256 and not 16.
Measured. The paged-versus-private comparison gains a third arm, `(bf16 main,
fp8 index)`, and stays bit-exact at `rtol=0, atol=0` through its decode steps.
That is stronger than equal selections: the sparse attention kernel sums a
row's prefix in the order it is handed, so an equal output means the two
scorers agreed on the rows and on their order. The indexer's query is not the
attention's, so moving the indexer to FP8 arithmetic does not weaken it. The
schedule is 69 tokens now because the old 22 fit inside one 256-token PAGE, and
a request inside a single page cannot tell a page-order mistake from correct
addressing.
Reverting either fix alone turns exactly the two FP8 arms red: writing with
`preshuffle=False` while reading with `True`, and making `restrict_to_candidates`
a no-op.
Accuracy, three arms, isolating the PAGE size from the format:
DSpark acceptance, case 0, n=6 GSM8K 3-shot, 100 questions
HEAD block 16 0.2348 91/100
block 256 0.2551 89/100 McNemar p=0.754 vs HEAD
block 256 + fp8 0.2105 90/100 McNemar p=1.000 vs HEAD
Every pair overlaps, the sign of the difference is not consistent, and each
acceptance arm's own run-to-run range is 0.19-0.48 -- four to twenty times the
difference between arms. The paired GSM8K discordance is 7-10 of 100, against
23-33 for the recorded zero-shot form of the same probe.
Neither says the precision did not drop. Acceptance rate measures how far the
draft and the target agree, and GSM8K at 100 questions carries a ~3pp binomial
error before the engine's own non-determinism; an FP8 index plane is a
reduction from BF16, and an effect below these probes' resolution on these
workloads is what they cannot see. No self-compare arm was run, so the 7-10
discordant pairs are not separated into the change and the floor.
The capacity is not a noisy number: the index plane costs 330 B/token instead
of 640, and the pool holds 243,558 PAGEs against 220,084, +10.7%.
Prefill keeps the tiled scorer over `cp_gather_indexer_k_quant_cache`. It is
not in the captured graph, and the candidate mask on a `[tokens, total_keys]`
prefill shape is the one place dense-plus-mask is not cheap.
One thing the kernel does not say about itself:
`deepgemm_fp8_paged_mqa_logits` refuses to compile below
`index_n_heads * index_head_dim = 4096` (`/app/logs_claude/probe_v41_mqa_dims.py`).
The published config is 32x128, exactly on the floor, and `wq_b` is replicated
so TP does not divide it. No startup guard: nine points is not a
characterization, and the configuration that ships is on the right side of it.
…itical path Wires the kernels from the previous commit into the serving path, and with them go the two blocking readbacks a decode step had, the per-request numpy and the staging buffers that carried it all. `EngramBatch` is this forward as the kernels see it, built from the very tensor the model embeds so the rows Engram looks up cannot drift from the tokens it runs. On the UVA path `prepare` stops building `EngramRequest` objects altogether -- one `tuple(tokens)` per request, a Python int per token, then hashed once per layer as a dict key, which a 16k prefill chunk pays in full before a row is read -- and `_stage_uva` reads the indices where they were produced instead of staging them through pinned memory. The cursor moves the same way. A verify step's candidates were `T + 1` rows staged per request on the host out of a history that had just been read back; they are now written by `engram_cursor_rows` into the plane `commit_tentative` already selects from, and an ordinary step's row goes straight into the cursor. `advance_cursor`, `stage_history` and their pinned buffers are left to the host path, which is the only one that still has a caller. `prepare_state` gains `histories=False` for the caller that needs neither: with a device batch nothing on the host reads this step's committed history, so it does not need this step's verdict either. The rows ship without waiting and are judged on the next call. That was the last blocking D2H in a decode step -- it was 1.03 ms of host block, of which 42 us was device idle, and defending it as "the guard has to be synchronous" was wrong twice over: the guard survives, and a request whose state is not where the scheduler wants it is still named, one step late, by a hard failure. Its own position check goes with it, replacing a 64-iteration Python walk over `span.position` with one pass over the batch. Two smaller things the same path was paying: a recycled slot's state and cursor reset cost two launches per fresh request and now cost two per step, and the Engram staging buffers are no longer page-locked under UVA, where nothing writes their host half once a dummy step zeroes the device one. Measured on V4.1-Flash TP4 DSpark-5 at bs=64, against the same trace shape throughout: the decode period falls 48.6 ms to 42.6 and GPU idle 5.5 ms to 0.4, with the busy span unmoved, which is what says only the host path changed. `hipMemcpyWithStream` in a decode window goes 1 to 0. Do not read finer than that from these traces. Instance to instance, the same build moves 0.84 ms in GPU busy span alone, so the last of these changes are below the noise and are here for what they delete, not for a number. Gates: GSM8K 1319 questions at fewshot 3 gives flexible 0.9136, inside this configuration's 0.9121-0.9318 band and among its other samples (0.9227 before any of this, then 0.9181, 0.9128, 0.9189, 0.9166, 0.9181). Acceptance 0.5269 against 0.5164 on the arm before. No stale-state guard fired over 1319 questions. Unit suite 8122 passed, 158 skipped, 3 xfailed.
…, not two dozen Before the draft's three stages run, every step assembles the same three things: which rolling-window rows the block may attend to, the block's input state, and each stage's view of its window. All of it was spelled out in torch, over `[B, ring]` and `[B, width]` tensors where a launch costs more than the arithmetic -- 26 of them per drafting step, four an `arange` rebuilding a compile-time constant. The attention indices are a closed form. Ring slot `j` holds the newest position congruent to it, `anchor - ((anchor - j) mod ring)`; a slot is readable when that position has been written, is not in the future and is still inside the sliding window; past the ring the block's own rows index themselves. One kernel writes a request's row of the list directly, and the slot-to-position map it derives on the way stays a register rather than the tensor a sub/sub/mod chain used to materialise. Mind the remainder there: Triton's `%` keeps the sign of the DIVIDEND where torch's follows the divisor, and `anchor - slot` is negative over most of a partly-filled ring. Normalising it is the difference between the right rows and silently the wrong ones, so `test_partly_filled_ring_wraps_the_way_torch_does` drives positions below `ring` on purpose. The block's input is an anchor followed by `width - 1` copies of one noise token, embedded, broadcast over the mHC streams and paired with a pre-mix. The embedding is a TP all-reduce, and it was carrying `B * width` rows of which only `B` ever differ: the noise row is fixed once the checkpoint is loaded. Embedding it once and keeping it takes `width` out of the collective, and the state is written straight out instead of being expanded and copied. The stages' windows are the same rows of the same tensor at different layers, so they come from one broadcast gather rather than three, each of which had been preceded by its own copy of the same `.long()` cast. All three are index arithmetic and copies, so they are held to bit-exactness against the chains they replace -- 324, 30 and 12 shape combinations -- not the weaker standard a fusion that rounds would get. Measured on MI355X (TP4, DSpark, 5 spec tokens) the prologue's torch-side launches fall 26.0 -> 22.3 per step and 152.4us -> 138.4us, with `arange`, `FillFunctor`, `remainder`, `compare_scalar` and `vectorized_gather` gone from the region entirely. Scale it honestly: `propose_dspark` is ~7% of a decode step, so this is ~0.4% end to end and below the run-to-run floor. It is here because the work was redundant, not because the step got faster. GSM8K 3-shot holds at 0.9143-0.9249 over four runs against a 1pp noise floor.
…orch's index path `read_windows` asks for the same rows of the same tensor at three layers, which is one broadcast index -- and torch serves that through the generic advanced- indexing kernel, walking the arithmetic per element as an opaque 2-byte type. At a decode batch it spent 13.2us moving 6.3MB. Nothing about the read needs that generality: a row is `head_dim` wide and contiguous at both ends. One program per output row makes it a coalesced load and store, and the read lands at 11.6us. That is not a bandwidth win and should not be read as one -- the bytes are the same, and this access pattern tops out near 520 GB/s whatever the block size or grid, which is where `copy_` lands on the same scattered source. What it buys is the generic path's per-element index arithmetic. The `.long()` in front of it goes too. The slot table is int32, the kernel widens each index itself, and casting the whole tensor first was a copy of it: 5.5us and a launch, for nothing. Over the cluster this leaves, measured on MI355X (TP4, DSpark, 5 spec tokens) against the same workload: 21.73 -> 6.67 launches per drafting step and 130.3us -> 44.9us, of which this commit is the last 2.6 launches and 7.1us. Held to bit-exactness against `window[layers[:, None], slots[None, :]]` over 24 shape and dtype combinations -- there is nothing here to round. GSM8K 3-shot over four runs lands 0.9174-0.9249 against a baseline's 0.9067-0.9204 on the same build; the two overlap and four samples cannot resolve the gap between their means, so this is "no regression", not an improvement.
DSv4.1-Flash quantizes its dense projections to E4M3 with 32x32 E8M0 blocks, and native_quant_linear was serving them by widening both operands to BF16, one 32-element scale group per tile, and scaling the tile sum afterwards. CDNA4 has an instruction for exactly this shape. Handing the E8M0 codes to tl.dot_scaled issues v_mfma_scale_f32_32x32x64_f8f6f4 and lets one tile span several scale groups; at m=4096 the second o_proj half goes 2730us -> 245us. BK below 64 is what to watch: Triton lowers dot_scaled to a BF16 emulation there, which is what the old formulation effectively was. The instruction is not free. Its block accumulator carries about 15 bits against the block's largest term rather than the 24 an FP32 sum would, so a group whose product is far below its neighbours rounds away before the accumulator sees it. Against an FP32 reference the error is 1.6e-5 to 1.8e-5 of the result's peak, consistently, and it scales with the result rather than with the element -- rows that cancel to near zero carry the same absolute error as the rest, which no per-element rtol can express. The four tests that compare an FP32 output against a full-precision reference now bound on the magnitude of that output. At the BF16 the model actually asks for, the two formulations are indistinguishable: both sit on the BF16 rounding floor. Accumulation drops to FP32 throughout, including W4A8. The cancellation test keeps its shape with the spread at 2**23, which is the widest an FP32 accumulator carries exactly. Split-K comes back for FP8, gated on the launch grid against the CU count and nothing else. A 16-workgroup kv_a at m=1 more than halves with it; a 288-workgroup shared w1 at m=128 loses 1.3x to the same split. The earlier measurement that said split-K never pays was taken with the weight pinned in MALL, where decode reads a third of its real time. M becomes a runtime argument. It is the token count, so specializing on it recompiled the kernel for every prefill chunk length: thirteen lengths measured thirteen variants at 170-620ms each, now three. Two kernels rather than one. FP8 spans several scale groups per tile through the matrix core; FP4 unpacks nibbles and scales a single group by hand. Folded together they duplicated the pointer bookkeeping and put an if in three places. _select_fp8_tile was swept through native_quant_linear so each candidate ran with its real split count, and with the inputs rotated. Measured on MI355X, us, inputs rotated past MALL: shape m=1 m=32 m=128 m=1024 m=4096 wq_a (1280, 5120) 6.3 7.3 11.6 27.1 60.0 wq_b (32768, 1280) 9.6 10.3 18.4 89.5 293.9 kv_a (512, 5120) 4.9 5.6 7.7 17.4 29.8 o_a (8192, 4096) 13.0 11.6 21.5 59.7 193.5 o_b (5120, 8192) 11.7 18.2 30.8 93.8 244.5 ffn.w1 (2304, 5120) 7.5 8.2 12.6 46.8 107.7 ffn.w2 (5120, 2304) 7.7 8.1 12.6 32.3 86.5 Four things were tried and are not here, each noted where it would go: an L2 swizzle (nothing), the weight pre-transposed to (K, N) (slower -- (N, K) gives longer contiguous runs), folding the split-K reduction into the kernel behind an arrival counter (1.7-2.9x slower), and loading one weight-scale row and broadcasting it (crashes TritonAMDGPUAccelerateMatmul).
Per layer the prologue ran q_norm, kv_norm, rope(query) and rope(kv) as four
kernels. The two norms read two slices of one `wqkv_a` output and neither
reads the other's result; the two rotations share a position line. Both pairs
are one launch now.
NORMS. `DualRMSNormMXFP8` wraps aiter's `fused_dual_rmsnorm_mxfp8_quant`,
which was written for this shape -- independent widths, one shared token
count -- and hands back the `(e4m3, e8m0 group-32)` pair `wq_b` reads without
quantizing again. It sits beside `DualRMSNorm` and, like it, is not an
nn.Module: it reads the weights off the two norms it is handed.
In one steady decode step the two `add_rmsnorm_quant` families lose exactly
40 launches each, 527.3 us of device time becomes 276.5, and the step falls
33.29 -> 33.03 ms. A microbenchmark predicted a third of that: back-to-back
repetition is the friendliest case those two kernels ever get.
The draft's attention takes the same pair. `context_keys` now takes the
latent already normed -- `forward` gets it beside the query, and the caller
that has only a latent norms it itself.
ROTATIONS. `RotaryEmbedding.pair` dispatches to the cached ABI's two-channel
entry, whose argument list is the one-channel one plus a tensor and whose two
head counts may differ, which is what it is for. 98 one-channel launches per
step become 18, plus 40 two-channel ones; rope device time falls 710.8 ->
443.6 us.
That 267 us does NOT show up on the clock: the step is busy 32.5 of its 33.0
ms, so the rotations were already overlapping something. The reason to keep
this is the launch count and the call site, not a speedup.
Neither fusion is bit-identical, and both differences sit on the rounding
floor: 1 FP8 code in 5,242,880 and 34 rotated lanes in 1,572,864, none of
them off by more than one ULP.
QUANTIZE, separately. `quantize_fp8_kernel` and `quantize_fp4_kernel`
declared `SIZE` a `tl.constexpr`, and `SIZE` is the token count -- so every
prefill chunk length and every batch size compiled its own kernel, 14,949
variants of the FP8 one in this box's cache. In an 8-second profiled window
the device sat idle 610 ms in front of that kernel, 65% of all idle there, in
stalls of 42-52 ms holding nothing but `hipModuleLoadData`. `SIZE` only
bounds the tiles. Output is bit-identical and the time unchanged, measured
with both spellings in ONE process -- across processes a JIT'd kernel drifts
by more than the effect being measured.
That axis was fixed once already for the GEMM ("M becomes a runtime
argument"); these two are what the sweep missed. The dual-norm kernel has it
too, on aiter's side, where the fix is not merged yet: until it is, this
commit trades one source of per-shape builds for another.
Gates. V4.1 and DSpark unit tests: 422 passed, 36 skipped, including a new
one pinning `pair`'s contract. `validate_runtime` passes eager and under CUDA
graphs, 55 events each. `validate_dspark_lifecycle --production --graph`
still reports `accepted_drafts {'5': 10}` and still stops at its own
pre-existing `restores >= 1`. Draft acceptance over 300 questions and 35,000
drafted tokens, two runs a side: 49.79 / 50.21 before, 48.91 / 50.29 after.
GSM8K 1319 reads 0.9272, 0.9189 and 0.9249 across the three steps, against a
same-code band of 0.9075-0.9287 measured the same day.
🏷️ CI GuideRuns automatically on every eligible PR before approval:
Heavy model tests:
|
There was a problem hiding this comment.
Remaining comments which cannot be posted as a review comment to avoid GitHub Rate Limit
ruff
Import block is un-sorted or un-formatted
ATOM/tests/models/deepseek_v41/test_dspark_integration.py
Lines 251 to 257 in 40b5bfa
Import block is un-sorted or un-formatted
ATOM/tests/models/deepseek_v41/test_dspark_state_audit.py
Lines 4 to 13 in 40b5bfa
Import block is un-sorted or un-formatted
ATOM/tests/models/deepseek_v41/test_math.py
Lines 4 to 20 in 40b5bfa
Import block is un-sorted or un-formatted
ATOM/tests/models/deepseek_v41/test_math.py
Lines 433 to 439 in 40b5bfa
Import block is un-sorted or un-formatted
ATOM/tests/models/deepseek_v41/test_moe_quant_config.py
Lines 4 to 8 in 40b5bfa
Import block is un-sorted or un-formatted
ATOM/tests/models/deepseek_v41/test_normalization.py
Lines 4 to 9 in 40b5bfa
Import block is un-sorted or un-formatted
ATOM/tests/models/deepseek_v41/test_projections.py
Lines 4 to 10 in 40b5bfa
Import block is un-sorted or un-formatted
ATOM/tests/models/deepseek_v41/test_quant_gpu.py
Lines 155 to 156 in 40b5bfa
Import block is un-sorted or un-formatted
ATOM/tests/models/deepseek_v41/test_rotary.py
Lines 4 to 6 in 40b5bfa
Import block is un-sorted or un-formatted
ATOM/tests/models/deepseek_v41/test_speculative_state.py
Lines 4 to 11 in 40b5bfa
Import block is un-sorted or un-formatted
ATOM/tests/models/deepseek_v41/test_speculative_state.py
Lines 299 to 303 in 40b5bfa
Import block is un-sorted or un-formatted
ATOM/tests/models/deepseek_v41/test_vision.py
Lines 4 to 34 in 40b5bfa
Import block is un-sorted or un-formatted
ATOM/tests/models/deepseek_v41/test_weights.py
Lines 2 to 20 in 40b5bfa
Import block is un-sorted or un-formatted
ATOM/tests/models/deepseek_v41/validate_dspark_lifecycle.py
Lines 4 to 26 in 40b5bfa
Import block is un-sorted or un-formatted
ATOM/tests/models/deepseek_v41/validate_dspark_runtime.py
Lines 4 to 25 in 40b5bfa
…cribe the code Nine thousand of this branch's lines were scaffolding, dead state and a lab notebook. None of it was reachable from a served request. Dead code, introduced here and unreachable: - `atom/distributed/rccl.py` and the switch that selected it. `96e4fb7e4` removed the one `use_custom_all_reduce = False` declaration -- V4.1 runs the same collectives as every other model -- but left behind the module, the ClassVar no subclass ever overrode, and the branch whose condition is therefore always false. Dropping the ClassVar also gives `accept_output_buffer` back the comment it was inserted in front of. - `tentative_limits`. Its only reader was a commented-out `_assert_async`, while the buffer was still allocated per cache and written per staged request: dead state costing a pinned allocation and a host write every step. - `LayerAttentionSpec.index_key_owner`, handed `kv_owner` a second time by a positional call and read nowhere. - `EngramConfig.kernel_size`, parsed from a key no config declares. - `DIAGNOSTIC175` in the model runner, whose own comment said to revert it before any acceptance run. - `fixtures/text_inputs.json`, which had no consumer before this commit either. `NativeQuantization` was a frozen dataclass whose four fields were literals in its own constructor, whose one production caller discarded the instance, and whose test asserted those literals back. It is `validate_native_quantization` now -- the five refusals were always the content -- and the test exercises each refusal instead of the constants. Twenty-four `torchrun` scripts under `tests/` were never collected by pytest and were reachable only from each other. They are gone, keeping the two the guides name as manual gates -- `validate_runtime` and `validate_dspark_lifecycle`, the second with its one borrowed helper inlined. Five test modules went with them because their subject was a probe rather than the product. Three parameter grids fell from 200 cases to 71 without an assertion changing: a batch axis over a kernel that runs one program per row, and a YaRN axis that re-asserted one shape-independent sentence at eight shapes. The guides were written in the stage numbers of their own development. They name capabilities now, point at neither a private log directory nor patches that were never committed, and no longer carry superseded measurement rounds -- the open DSpark quality regression is one table and a ruled-out list instead of eight rounds in chronological order, and the small-chunk report, which described a padding path that has since been removed, is folded into the vision guide as the one tradeoff it still documents. Two referenced files that never existed are gone, `CLAUDE.md` and the model support guide have V4.1 entries, and four environment variables are documented, one of them newly registered rather than read raw. 8071 unit tests pass.
…experts
The CPU pre-checks runner has neither AITER nor a GPU, and a collection error
there is not one skipped file: pytest reports `Interrupted` and the job runs
nothing at all. Under that runner's conditions this branch produced 18
collection errors against main's 1.
Two causes, both fixed here. The new test modules imported AITER-backed modules
at module scope and guarded them per function, which is too late -- the import
has already failed. And `tests/test_pd_pp.py` / `tests/test_pp.py` put a bare
`aiter` into `sys.modules` at import time so they can exercise CPU wrapper
logic, then leave it there; from that point `pytest.importorskip("aiter")`
resolves the name, the skip does not fire, and the real `from aiter import ...`
inside some later module raises ImportError during collection. Those two now
drop the top-level name once their own imports are bound. The submodules stay,
because the connectors reach for those lazily, at call time.
Modules with host-side tests worth keeping import the AITER pieces where they
are used rather than at the top, so those tests still run: `test_engram.py`
covers 34 of its 44 on a CPU runner and `test_runtime_contract.py` 29 of 31,
where both previously ran none. The rest skip at module scope, each checked for
anything a CPU runner could have executed.
Ruff is clean across every file this branch adds or touches. Beyond the
mechanical fixes -- `.exception()` rather than `.error(..., exc_info=True)`,
bare `raise`, a mutable default argument, `itertools.pairwise` -- nine
`except Exception: pass` handlers now say what they swallowed. A teardown is
allowed to fail; swallowing it without a word is what makes a hung shutdown
unexplainable.
Separately, V4.1 no longer refuses TP > 1 without expert parallelism. The
weight manifest already sharded routed experts along their intermediate
dimension when EP is off, and `FusedMoE` already took its own TP path there;
admission refused the combination on the strength of never having been run with
it. TP4 without EP scores 0.9204 on the 1,319-question GSM8K set, inside the
band six same-code EP runs span. Their throughput has not been compared.
Also removed: a `checkpoint_loader` dispatch hook with nothing behind it, which
V4.1's own runtime documents it does not use; and the router/expert and
unit-table reference implementations, which move out of `atom/model_ops/` into
the tests that are their only callers.
… pages Whether the device lookup can be used was decided per rank in three places, and two of them can disagree across a TP group: a rank whose head shard is empty because the head count does not divide the world, and a rank whose `cudaHostRegister` is refused. Either one takes the host path alone while its peers go on to the all-gather that ends `_stage_uva`, and the job hangs there. The local answer is now reduced with MIN over the group's gloo communicator before anyone acts on it -- once, at load -- and a rank outvoted after registering gives its pages back. Registration leaked as well. A table's values and its scales are registered separately, and the scales being refused returned before the range was recorded, so `disable_uva` found nothing to release and the values stayed page-locked for the life of the process: 23 GiB on DeepSeek-V4.1-Flash at TP4, unswappable and unaddressed. `enable_uva` now unregisters what it took, and keeps the page-aligned bases so that `disable_uva` gives back what went in rather than deriving the same offsets a second time. `EngramHost.shutdown` releases too, ahead of the mappings those pages belong to. Six tests cover the two, each also run with its fix reverted to see it fail. They need no AITER, so they run on the CPU pre-checks runner rather than skipping there. Also gone: a `reshape` in the staging loop to the shape its input already has, under both a sharded and an unsharded group.
Two test files conflicted, and both conflicts belong entirely to #2315, which reverts the Kimi-K3 P/D disaggregation of #2154. Every block in them -- `_MLABase` and the transfer-region cases in `test_gdn_state_relocation.py`, the deferred-decode case in `test_mtp_deferred_status_queue.py` -- was added by #2154, and this branch had only adapted them to `KVTransferTensors.set_block_count` and to `decode_spans`. The revert is the right side to take: the code those tests cover is gone. Both files now match main byte for byte. `set_block_count` itself survives the revert with its callers intact. No construction site passes `num_blocks`, and the case pinning that a backend cannot pass one is still there. Gates on the merged tree. The pre-checks suite, which is what CI runs, is 6346 passed / 652 skipped / 0 errors. The full suite against a GPU is 8092 passed and 4 failed, all four in `test_qwen4_exp_mtp.py` and all four inherited: they pass at the merge base and on this branch before the merge, and fail on main alone. #2311 made the QSA GQA kernel's `kv_splits` adapt to the token count, so a batched call and a per-row call now reduce in different orders -- and that test compares them at rtol=0, with the largest difference one bf16 ULP and the selected tokens identical. Nothing caught it upstream because the case needs a GPU and pre-checks runs without one.
Fuse FP32 normalization, weighted dot product, gate activation, masking, and residual addition into a Triton kernel. Preserve the torch fallback and expose ATOM_ENGRAM_FUSED_GATE for comparison. Add FP64 reference, dtype, masking, fallback, and graph replay coverage.
Snapshot token history before cursor updates and stage layer 1 and 14 hash, UVA lookup, and TP all-gather on one graph-captured side stream. Use private IPC synchronization and raw input buffers; consumers wait for complete embeddings. Enable ATOM_ENGRAM_OVERLAP by default, with an explicit 0 opt-out and main-stream collective fallback where unsupported. Validation: 149 regression tests; TP4 main/side collective concurrency with two graph buckets and 120 changing replays; actual BS16 trace; full GSM8K 5-shot strict 1211/1319 and flexible 1210/1319 with fusion enabled.
… after The CPU pre-checks runner installs torch from the CPU index, and that wheel does not bring Triton. Four modules reached Triton-backed operators at module scope and declared their GPU requirement below that import, as a `pytestmark` -- which runs after the import has already raised. A raise during collection is not one skipped file: pytest reports `Interrupted` and the job runs nothing at all. All four are GPU-only, so moving the guard above the imports costs no coverage. It names the GPU rather than Triton because the GPU is the precondition; reaching Triton is a consequence of it. `test_packed_cache.py` keeps its `import triton` for the kernel it defines, below the guard. Its import block stays sorted: a statement between two blocks leaves each of them to be ordered on its own. Two more modules still fail to collect there, and both hold real host-side tests, so the fix is to move an import boundary in the source rather than to skip: `engram_runtime` binds its host prefetch and staging to the device hashing kernels, and `deepseek_v41/dspark` keeps two pure-torch helpers beside Triton kernels. Neither is in this commit.
Decorate DeepseekV41RuntimeModel with support_torch_compile and use AITER torch_compile_guard boundaries for serving attention, Engram staging and consumption, and shape-tuned delayed mHC. Runtime eager and compiled calls share the same path while preserving decoder input hooks for DSpark. Keep mHC fake shapes beside its implementation, preserve the no-post residual alias, and admit level 3 with FULL CUDA Graphs. Cover dynamic batches, live request state, hooks, and mHC output schemas. Validation: 299 passed, 36 skipped in the V4.1 regression suite before the CPU-only import guard adjustment. Final TP4 BS16 validation aborted with GPU memory access faults; the cause remains unresolved. Earlier GSM8K results apply to the implementation before this guard refactor.
The CPU pre-checks runner has no Triton, and `engram_runtime` bound its host prefetch and staging to the device hashing kernels while `engram_lookup` kept a Triton gather beside the host table. Both took the whole pytest session down during collection rather than skipping one file, twice. Engram now lives under `model_ops/engram/`. Its top level -- `mapping`, `tables`, `host` -- imports numpy and torch and nothing else; `device/` holds every module that reaches Triton. `EngramHost` takes the class that does the device lookup rather than importing it, so a rank with no class gets the host path and a CPU-only test says so by passing nothing. That moves six UVA registration and group-vote tests from "CI never runs them" to "CI runs them every time": page-locking a shard and voting on the outcome are cudart and Gloo calls, and only the hashing and the gather ever needed a GPU. `test_import_boundary.py` executes the rule in a child interpreter. `deepseek_v41/dspark` gives up its three pure-torch helpers to `draft_block` for the same reason, and `test_dspark.py` its one Triton case. Two formatting fixes ride along in `deepseek_v41/config.py` and `model.py`. Neither is from this change -- both reproduce on b3cb26c alone -- but one of those files is in this diff and the other fails a repo-wide job. No behaviour change. GSM8K 1319 5-shot 0.9318/0.9325 against 0.9181/0.9189 on the same HEAD, inside this model's same-code spread; DSpark acceptance 49.47% against 49.52%.
|
@ZhangLirong-amd heads up —
The rule the directory now carries: everything at the top level of Two things that touch your code directly:
Also worth knowing separately from this PR: |
…ks for C408 on `test_compilation.py:174`. The file arrived in b3cb26c and is new in this PR, so reviewdog's diff context covers all of it, not just the lines a later commit touches -- which is why the rule lands here rather than there. `kwargs.update(...)` below reads a dict either way.
Skip block-table uploads when mappings and padding are unchanged. Refresh on row changes or device allocation replacement while preserving fixed buffer addresses. Build indptrs for all configured ratios in one Triton launch and select cursor rows using int32 indices directly. Cover ratio subsets, graph replay, and block-table invalidation with regression tests.
…2356) MiniMax-M3-MXFP8 is an fp8 (1, 32) checkpoint launched with --online_quant_config, so every one of its attention and dense linears matched the native group32 source shape while the model-wide online_quant flag was set, and LinearBase raised at construction. All 14 nightly MiniMax-M3-MXFP8 jobs have failed since #2314; run 35522504237 is the last green one. The flag says the run was launched with online quant, not that this layer is affected -- an excluded layer keeps its checkpoint weight. Ask per layer via will_online_requant(), and let a (1, 32) source that online quant will overwrite drop the native layout for the generic per_1x32 path, whose scale geometry it already matches. A (32, 32) source has no such twin and keeps the error, as does every other rejection the gate made. should_stream_online_quant was already computing this predicate inline alongside its streaming preconditions; it now calls the named one.
…strides
Two defects in the flydsl index-score integration, both found by
measurement after rebasing onto main.
1. The ubatch work-map buffer was sized with the default cp_world=1
while `_build_ubatch_metadata` builds it with the real CP pair. The
shard's dispatch is denser than the whole context's, so the CP map is
LONGER, not shorter -- 128 rows against 64 at bs=8, max_block=64 --
and the build rejects the buffer:
ValueError: work_map: expected packed [at least 128, 2] int32
So ubatch plus indexer CP could not start. It fails loudly rather
than writing out of bounds, which is why it never showed as
corruption. The non-ubatch buffer next to it already passed the pair;
this just does the same.
2. The three score strides were `tl.constexpr`. `S_HEAD` is
`tokens * local`, so it moves with the batch, and specializing on it
cost a ~430ms Triton recompile for every distinct decode batch size:
8 of 8 sizes recompiled with a cold Triton cache, 0 of 8 after. That
is the same reason `LOCAL_BLOCKS` and `GLOBAL_BLOCKS` are already
runtime arguments in this kernel. Triton still folds the
divisible-by-16 and equal-to-1 cases, which is where the addressing
win is; nothing about the layout handling changes.
tests/test_minimax_m3_index_topk.py, tests/model_ops/test_indexer_cp_parity.py
and tests/test_paged_attention_dispatch.py pass (216 passed, 21 skipped).
Not verified end to end on a server: main now imports `aiter.topk_select`
(#2244, #2314) and the aiter installed here does not have it, so the
engine cannot start from this base at all -- with or without this branch.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
What
DeepSeek-V4.1-Flash as its own model package rather than a reshape of V4.
48 commits, staged P00–P10b;
recipes/DeepSeek-V4.1-Flash.mdcarries the stagetable and states what each stage accepted.
atom/models/deepseek_v41/: CSA2 topology, single-pass mHC, MoE,Engram, vision, and the DSpark draft.
atom/model_ops/deepseek_v41/andatom/model_ops/attentions/deepseek_v41/: compressor, paged index scorer,packed attention, grouped output projection, rotary. V4's BF16 attention and
inverse RoPE are reused rather than copied.
graph, and a decode step that is the draft's replay plus the target's.
with chunked prefill and request-owned embedding lifetime.
(
atom/model_ops/blockscale*), latterly on the CDNA4 microscaling matrix core.Where it stands
P00–P10b complete. P11–P14 pending: candidate-only indexing and Engram
fusion, optional CED decoder replay, optional encoder replay with a persistent
global cache, and validated distributed/deployment combinations. Draft for that
reason.
Two accepted tradeoffs and one open regression, all already documented:
fixed-length comparison over 128 questions: 90/128 without speculation versus
80/128 with, nine gains and 19 losses, paired 95% bootstrap [−0.156, 0.0],
exact McNemar p = 0.087. The matched five-shot comparison is identical per
document, which does not cancel it.
docs/deepseek_v41_dspark.mdhas thefull account including what was ruled out.
Accuracy
GSM8K, 1319 questions, fewshot 3, TP4 + whole-expert EP, DSpark-5, BF16 KV.
Six same-code runs span 0.9075–0.9287, σ ≈ 0.8pp.
A single run inside that band is the bar. Anything finer needs per-question
pairing against a same-code control: two runs of identical code disagree on
about 6.5% of questions, and a single pair's exact McNemar reaches p = 0.0026
between two runs of the same commit. The judge is whether cross-arm discordance
leaves the same-code range, not the p-value and not the headline.
Test plan
bash .github/scripts/run_unit_tests.sh— 8222 passed, 158 skipped,3 xfailed
tests/models/deepseek_v41/andtests/attentions/deepseek_v41/on GPUtests.attentions.deepseek_v41.validate_runtime, eager and--graph—55 scheduler events each; chunked, reordered, forked, replayed from a
missing graph, preempted, cancelled and slot-recycled routes each produce
the tokens an uninterrupted run did
tests.models.deepseek_v41.validate_dspark_lifecycle --production --graph— reaches the verify step with
accepted_drafts {'5': 10}. It still stopsat its own
restores >= 1; that assertion is pre-existing and tracked(the file's checkpoint interval is snapped off against the 256-column
prefix-cache hash block, and repairing that is necessary but not
sufficient)
Notes for review
Arithmetic is judged against the published model by the
ATOM_DSV41_REFERENCEunit tests; the runtime validates unsupported combinations before it loads
weights. AITER is pinned in the recipe.
The two most recent commits additionally want ROCm/aiter#5703: without it the
fused dual RMSNorm keys its Triton build on the token count, which costs
113–133 ms of device idle per new shape at serving time.