Skip to content

[DCP][Opt] Fuse the prefill context reorg, dequant and kv_b_proj into one kernel, and take the custom AllGather - #2113

Merged
valarLip merged 1 commit into
mainfrom
hexwang/fuse_prefill_dcp_reorg
Sep 2, 2026
Merged

valarLip merged 1 commit into
mainfrom
hexwang/fuse_prefill_dcp_reorg

Conversation

@whx-sjtu

@whx-sjtu whx-sjtu commented Sep 2, 2026 •

Copy link
Copy Markdown
Contributor

Summary

The DCP cached-prefix prefill spends a lot of its budget between the AllGather and flash_attn_varlen_func: reorg_kvcache walks every (seq, rank) segment in Python on every layer, then the fp8 dequant, the kv_b_proj decompress and the k_pe concat each cost their own kernel. This collapses that whole span into a single kernel pass and moves the gather off pynccl.

Two changes:

  1. The reorg becomes an index map, and rides along with kv_b_proj. The reorg is a pure permutation, so it can be precomputed as dst_row -> src_row instead of executed as slicing and concatenation. dcp_reorg_row_indices builds it with cumsum/repeat, and the metadata builder publishes one per chunk as ag_row_indices. That map then doubles as kv_indices for the existing gather_kv_b_proj, so the row gather, KV-cache dequant, kv_b_proj, the k_nope/v split and the k_pe concat all happen in one kernel. Viewing the AllGather buffer as a block_size=1 cache is what lets the existing kernel take it unchanged — no aiter change needed.

  2. The gather goes through dcp_all_gather rather than GroupCoordinator.all_gather, which prefers aiter's custom collective. fp8 has no entry in the custom kernel's dtype enum, so the helper pairs the bytes into fp16 instead of falling back to pynccl.

Three now-dead fields (padded_local_chunk_seq_lens, local_context_lens_allranks, chunk_size) drop off MLAChunkContextMetadata. The vLLM plugin path is untouched: it has its own AiterMlaChunkedContextMetadataForVllm and keeps calling reorg_kvcache, which stays.

Results

Before:
image

After:
image

Kimi-K3, 8xMI355, tp8 + dcp8, fp8 KV cache, comm_backend=ag_rs.

before after
gsm8k, full 1319 @ 64 concurrency — 0.9621 strict / 0.9621 flexible
TTFT, 44k cached prefix, conc=1 271.9 ms 254.2 ms (-6.5%)
TTFT, 44k cached prefix, conc=4 565.1 ms 525.8 ms (-7.0%)
segment GPU compute (profiled window) ~58 ms 18.3 ms
segment host-side ops 36.8 ms 5.3 ms
kernel launches (profiled window) 165715 164286

The per-layer Python walk and its 864 cat calls in the window are gone entirely.

@github-actions

github-actions Bot commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

🏷️ CI Guide

Runs automatically on every eligible PR before approval:

  • ✅ Pre Checkin: Black, Ruff, catalog schema validation, non-GPU unit tests

Heavy model tests:

  • ✅ Run after the PR is approved and Pre Checkin passes
  • ✅ Run immediately when an approval review is submitted
  • ✅ Can be requested before approval with labels
Label Tests
ci:full Run all heavy PR model tests: native ATOM, vLLM, and SGLang
ci:atom Run native ATOM model accuracy tests
ci:vllm Run ATOM vLLM OOT model accuracy tests
ci:sglang Run ATOM SGLang model accuracy tests

Heavy jobs are skipped when the PR is not approved and no matching ci:* label is present.
Add labels via the sidebar or gh pr edit 2113 --add-label <label>

@whx-sjtu whx-sjtu changed the title [DCP][Opt] Collapse the prefill context reorg, dequant and kv_b_proj into one kernel, and take the custom AllGather [DCP][Opt] Fuse the prefill context reorg, dequant and kv_b_proj into one kernel, and take the custom AllGather Sep 2, 2026
@whx-sjtu
whx-sjtu requested a review from valarLip September 2, 2026 06:42
Signed-off-by: whx-sjtu <xiaowang990929@gmail.com>
@whx-sjtu
whx-sjtu force-pushed the hexwang/fuse_prefill_dcp_reorg branch from 3cc290a to 9624cf5 Compare September 2, 2026 07:39
@valarLip
valarLip merged commit 0093d6c into main Sep 2, 2026
28 of 41 checks passed
@valarLip
valarLip deleted the hexwang/fuse_prefill_dcp_reorg branch September 2, 2026 12:58
PerryZhang01 pushed a commit that referenced this pull request Sep 2, 2026
One conflict, in `_dcp_compute_prefill_context`. main (#1965 / #2113) replaced
the hand-written `kv_b_proj` decompress + `k_pe` concat with a fused
`_kv_b_proj_gather` writing into `k_chunk` / `v_chunk`, so this branch's `k`
and `v` are gone and the `flash_attn_varlen_func` below already consumes
main's buffers.

Taking main's side wholesale would have been wrong, though: the deleted line
also stripped the zero rope pad off `prefill_q`, which IS still used below.
`_dcp_context_kv_buffers` returns either the chunk workspace or a fresh buffer
of `self.qk_head_dim`, and for a NoPE model both are the WIDENED 320 that CK's
256 head-dim cap refuses -- so the slice is still required, now applied to
`prefill_q` and main's `k_chunk`. All five `_drop_rope_pad` call sites remain.

This path is unreachable for GLM-5.3-Flash today (it refuses DCP in
Config.__post_init__) and `_drop_rope_pad` is a no-op for every model with a
real rope, so nothing shipped changes either way -- but leaving the site
inconsistent would have set a trap for whoever enables DCP here.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
PerryZhang01 pushed a commit that referenced this pull request Sep 2, 2026
Finding 1 (raw-config chunk workspace) was already fixed; this covers 2-5.

2. Bound the decode index conversion to the request's own output region.
   `_convert_req_index_to_global_index_kernel` gated its store on
   `indice_id < kv_len`, which counts entries on the INPUT side and is not the
   width of the output region -- that is `page_kv_indptr[b+1] - [b]`, since the
   output is packed by that indptr. A pooled selection makes the two diverge
   (the row is padded to `round_up(index_topk + kpool - 1, 128)` = 2176 columns
   while the region holds at most 2051) and a long context pushes kv_len past
   both, so every column stored. The surplus columns are top-k padding, -1,
   which `tl.where(out_val >= 0, ...)` maps to cache slot 0 -- written over the
   START of request b+1. `OUT_NUMEL` only caught the final request, so it was
   silent for every other one. Now masked by `out_offset < out_kv_end`, which is
   a no-op wherever the two widths already agree.

3. Widen `_is_kda()` to both KDA models. Its two siblings in the same file
   (`gdn_attn.py:129`, `:331`) were widened for GLM-5.3-Flash and this one was
   not, so it answered "not KDA" for a KDA model and
   `_replayssm_buffer_shapes()` sized the ReplaySSM record buffers for the wrong
   head geometry -- an out-of-bounds write reachable with
   `ATOM_ENABLE_REPLAYSSM=1`. Swept the tree: no `== "kimi_linear"` test remains.

4. Let `--load_dummy` start. The KDA shard-coverage check exists to catch a
   checkpoint that supplied only part of q/k/v, but dummy loading never runs a
   weight_loader, so the set is empty by design and all 34 KDA layers refused to
   start. Skipped when `load_dummy` is set; the gate fold still runs, so the
   layer keeps the shape and dtype it would otherwise have had.

5. Guard both sparsity-off knobs the same way, at startup. `ATOM_GLM5_KPOOL=0`
   raised `NotImplementedError` from inside `forward_impl`, which takes the
   engine down mid-batch rather than refusing a request the way `envs.py` and
   `docs/environment_variables.md` both describe; `ATOM_GLM5_FORCE_DENSE_MLA`
   had no length guard at all despite being documented as short-context only, so
   the two failed in opposite directions. Both are now validated in
   `Config.__post_init__` against `max_model_len`, which already bounds
   `max_seqlen_k`: the launch fails cleanly with a message naming the fix, and a
   per-forward branch leaves a 45-layer hot path.

Unit suite unchanged at 4831 passed / 14 failed; those 14 fail identically on
plain origin/main (they arrived with #1965 / #2113, after the review's measured
4902/0). black and the diff-context ruff gate are clean.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
valarLip pushed a commit that referenced this pull request Sep 3, 2026
* [atom] GLM-5.3-Flash bring-up: k-pool DSA indexer + reference harness

First step of GLM-5.3-Flash (`glm5_next`) support. The model is not servable
under ATOM yet; this lands the one genuinely new op plus the reference oracle
the rest of the port will be built and validated against.

atom/model_ops/kpool_indexer.py
  k-pool compressed DSA indexer. GLM-5.3 scores *pools* of `index_kpool`
  consecutive keys rather than individual keys, so with index_topk=2048 and
  kpool=4 only 512 candidates are ranked to cover 2048 tokens. Closest existing
  ATOM code is DeepSeek-V4's Compressor, which pools the same way at
  compress_ratio=4 with an `ape` term but is overlapping and RoPE'd; GLM's is
  non-overlapping and NoPE (the whole text model is NoPE, qk_rope_head_dim == 0).

  Verified to select byte-identical token indices to transformers'
  Glm5NextTextIndexer on real layer-3 weights at seq 7/64/300/2048/3000 and with
  5 and 17 tokens of left padding. seq=3000 exceeds index_topk, so real sparse
  pool selection is exercised. This is the dense form; the paged/ragged variant
  the scheduler wants is the follow-up, and this is its correctness oracle.

recipes/GLM-5.3-Flash.md
  Architecture breakdown, the mapping of each component onto existing ATOM code
  (KDA -> KimiKDAAttention + aiter kimi_delta_attn; mHC -> hc_split_sinkhorn,
  which is a math-exact match down to the checkpoint tensor names; MoE, block
  FP8, MTP, vision), the checkpoint->model weight remap, and remaining work.

  Also records two upstream bugs found during bring-up:
  - transformers mis-quantizes the KDA forget gate. modules_to_not_convert names
    it model.layers.N.self_attn.f_a_proj, but the keys are under
    model.language_model.layers.N and the glm5_next conversion mapping renames
    those tensors to self_attn.forget_gate.f_a_proj before the FP8 quantizer
    runs. All 68 forget-gate linears end up wrapped in FP8Linear while holding
    BF16 weights with a freshly-initialised weight_scale_inv.
  - kernels-community/finegrained-fp8 does not compile on gfx950 (LLVM
    iota_range assert). Every block-FP8 Linear and the MoE experts route through
    it, so no FP8 glm5_next forward runs on MI355X without a substitute. ATOM
    uses its own aiter block-FP8 GEMMs and is unaffected.

recipes/glm5_3_flash/
  Reference harness: ROCm torch 2.10 + transformers 5.16.1 image, the loader
  that dumps oracle logits, a torch-only stand-in for the broken FP8 kernel, and
  the parity check. Loads in ~131s on 4x MI355X and generates coherent text.

tests/model_ops/test_kpool_indexer.py
  CPU tests for causality, padding, whole-pool expansion, tail coverage and the
  token budget. No GPU or checkpoint needed.

* [atom] GLM-5.3-Flash: run the reference on aiter block-FP8, fix multi-GPU launch bug

The reference path now uses ATOM's own block-FP8 GEMM instead of the hub Triton
kernel that does not compile on gfx950, so GLM-5.3-Flash runs end to end on
4x MI355X through aiter kernels: 4.25 tok/s vs 2.68 for the torch dequant bundle,
with the top-4 next tokens identical to the torch oracle (the small logit deltas
are FP8 activation quant, which is what the checkpoint was trained for).

fp8_aiter_backend.py routes matmul / batched_matmul / grouped_matmul to
aiter.gemm_a8w8_blockscale. aiter_fp8_check.py pins it against a torch dequant
reference on a real GLM weight (cos 0.9997 at M=1/21/256).

Third bug found while wiring this up, now in the recipe as §4c: aiter kernels
launch on the *current* CUDA device, not the tensor's device. With a multi-GPU
device_map, accelerate moves tensors to cuda:1..3 without touching the CUDA
context, so aiter reads and writes GPU 0's memory and returns NaN. Every input
was finite and well-scaled; only the output was garbage, and only past the first
device boundary -- so the model loads, runs, and silently emits nonsense.
transformers guards DeepGEMM against exactly this; aiter has no such guard.
Fixed by pinning the context with torch.cuda.device(t.device) around every call.
ATOM proper is unaffected (one device per rank) but any multi-device
single-process user of aiter is exposed.

ref_run.py gains GLM53_FP8_BACKEND / GLM53_MAX_NEW_TOKENS / GLM53_FP8_VERIFY.
The verify mode runs both backends on every call and reports the first
divergence with shapes and devices -- that is what localised the bug.

* [atom] GLM-5.3-Flash runs end to end under ATOM on MI355X

Adds atom/models/glm5_next.py and the engine wiring behind it. GLM-5.3-Flash
(320B total / 18B active) now loads and generates under ATOM on 4x MI355X:
every one of its parameters loads, and greedy decoding produces coherent text
that opens identically to the transformers oracle ("The user is asking why the
sky appears blue ..."). 21-token prompt, TP4, --enforce-eager: TTFT 3.12s,
TPOT 45ms (~22 tok/s decode) -- 5.2x the 4.25 tok/s transformers reference,
before CUDA graphs or MTP.

The model is assembly of components ATOM already had:
  KDA            -> kimi_k3.KimiKDAAttention + aiter kimi_delta_attn
  mHC            -> sparse_attn_v4.hc_split_sinkhorn + aiter mhc_pre/mhc_post
  MLA            -> attention_mla via MLAModules
  MoE            -> FusedMoE (sigmoid / noaux_tc), swiglu_limit=10
  clamped SwiGLU -> swiglu_oai_split(alpha=1, beta=0, limit=10), which is
                    exactly GLM's silu(clamp(gate)) * clamp(up)

Three decisions worth knowing about, each exact rather than approximate:

- DSA layers run DENSE. index_topk=2048 with index_kpool=4 selects 512 pools
  covering 2048 tokens, so at or below 2048 tokens the indexer selects
  everything and dense causal MLA is numerically identical. Beyond that this
  model is not yet correct. The indexer weights are loaded (Glm5NextIndexer)
  but unused; model_ops/kpool_indexer.py already implements and validates the
  selection maths the sparse path will need.
- NoPE runs on a zero-width rope slice with an identity rotary. Padding those
  lanes with zeros would also have been exact but is impossible: qk_nope_head_dim
  is already 256 and the CK prefill kernel caps head dims at 256.
- The KDA output gate is folded at load. GLM factorises it (g_b_proj @ g_a_proj)
  where Kimi-K3 has a single g_proj; both are linear with nothing between, so
  materialising the product once lets KimiKDAAttention -- and all of its state
  cache, TP and CUDA-graph integration -- be reused unchanged.

Engine wiring, all following existing precedent:
- config.py: glm5_next joins the multimodal text-config extraction, and
  glm5_next_text the set of text configs this image's transformers has no class
  for (loaded as a bare PretrainedConfig, like kimi_linear).
- model_runner.py: registers the arch, and is_kimi_linear() -- really "hybrid
  MLA + KDA", which selects KimiMLAGDNBackend -- now covers glm5_next_text.
- llm_engine.py: glm5_next_text added to per_req_cache_model_types, as its own
  startup assert demands for any model with a recurrent state pool.
- gdn_attn.py: the KDA layer lists are read 0-based for GLM (Kimi numbers them
  1-based), and the temporal state is fp32 for both, which aiter's
  chunk_kimi_delta_attn requires and reads back verbatim.

Two things that cost real debugging time and are worth recording:
- packed_modules_mapping must be a class attribute. model_runner remaps the
  quant config from the class before the model exists; doing it a second time
  from __init__ to add per-layer entries corrupts the layer pattern specs and
  silently marks every attention projection quantized. GLM needs no per-layer
  mapping anyway -- its MLA layers use q_a/q_b/kv_*, so a layer-agnostic
  .q_proj/.k_proj/.v_proj -> .in_proj mapping is unambiguous.
- Parameter names come from the module attribute path, not the `prefix` string.
  The mHC parameters are flat on the layer in the checkpoint (hc_attn_fn), so
  they are declared flat here; nesting them in a submodule renamed them to
  attn_hc.fn and left them at their init values. ATOM's loader caught this
  immediately ("1189/1190 model parameters were NOT loaded") -- that report is
  worth its weight.

Requires an aiter with chunk_kimi_delta_attn and mla_decode_fwd(causal=...);
rocm/atom-dev:nightly_202608270231 or newer.

* [atom] GLM-5.3-Flash: validate the ATOM port against the reference layer by layer

Adds the validation harness and records what it says. Headline: ATOM's forward
tracks the transformers reference to cosine >= 0.9997 at every one of the 45
layers, and does not degrade with depth -- which is the property that separates
a sound port from a mis-wired one.

  layer   0     3     7    11    19    27    35    43    44
  cosine  .99972 .99976 .99970 .99999 1.00003 .99999 1.00002 .99998 .99998

compare_layers.py drives that: it hooks the reference's decoder layers and diffs
each against ATOM's ATOM_FWD_DUMP output. Two traps it now handles, both of which
produce convincing-looking nonsense:

- ATOM_FWD_DUMP_ONE_SHOT defaults on and writes only the FIRST call, which is the
  warmup forward over 16384 dummy tokens. Comparing against that reports cosine
  0.08 at layer 0 and reads as a catastrophic bug. Dump with ONE_SHOT=0 and pick
  the call whose row count matches the prompt.
- Teacher-forced rank-1 agreement stops meaning much once two implementations
  fork. score_atom_tokens.py scores ATOM's tokens under the reference and gets
  67% rank-1 / mean p 0.58, versus a 98.4% / 0.86 baseline from rescoring the
  reference's own torch-FP8 output with the aiter-FP8 backend. That gap is mostly
  an artifact: the baseline never forks, while ATOM forks at position 4 on
  ' why' (p 0.63) vs ' about' (p 0.11) -- a genuinely split distribution -- after
  which the reference is grading a prefix it would not have written. The
  per-layer cosines are the load-bearing evidence; this number is not.

One real numerical gap found and measured, not yet fixed: GLM clamps its expert
SwiGLU at +-swiglu_limit (10.0), but only ATOM's flydsl and mori MoE paths plumb
swiglu_limit -- the default standard/CK path drops it. Instrumenting the
reference shows the clamp binds on under 0.001% of elements (max |gate| 19.6,
max |up| 14.2), so it is small but real. Dense layers 0-2 are unaffected; they go
through swiglu_oai_split.

Also: GLM53_DISABLE_FUSED_MHC=1 forces the torch mHC path for bisecting (it
produces the same output as the fused aiter kernels, at 2.8x the decode cost),
and atom_run.py gained --logprobs plus a dump of its generated ids.

recipes/GLM-5.3-Flash.md is rewritten for the new status: how to run it, the
measured numbers, and what is left (contexts past index_topk, swiglu_limit, MTP,
vision, CUDA graphs).

* [atom] GLM-5.3-Flash: vision tower

Adds atom/models/glm5_next_vl.py -- the 24-layer ViT, 2-D rope, packed
variable-length attention, 2x2 spatial merge and projection into the language
model's 4096-wide embedding space -- and wires it into
Glm5NextForConditionalGeneration behind the engine's existing multimodal
contract (get_vision_embeddings / merge_multimodal_embeddings).

Validated against transformers' Glm5NextVisionModel with the real
model.visual.* weights, on the merged [n_tokens, 4096] output that gets
scattered onto image placeholders:

  fp32 + SDPA   bit-exact on all four grids (max|d| = 0.00000)
  bf16 + aiter  cos .9941 - .9998

The fp32 row is the assertion: same kernel as the reference and no BF16
rounding, so it isolates the maths -- patch embed, the block-major 2-D rope
layout, 24 blocks, clamped SwiGLU, the merge, downsample conv and merger. The
bf16 row is the serving path, reported not asserted; it carries aiter
packed-varlen vs SDPA plus ATOM's single fused [gate|up] GEMM vs the
reference's two, compounded over 24 blocks on random patches far worse
conditioned than real image input.

Two things worth recording:

- config.py: when this image's transformers has no class for the architecture,
  get_hf_config fell through to _multimodal_config = None, which left the model
  with no vision_config and no way to build a tower. It now rebuilds the
  multimodal config from the raw dict, promoting sub-configs to
  PretrainedConfig so attribute access works either way.
- The model's packed_modules_mapping rewrites .gate_proj / .up_proj to
  .gate_up_proj by substring, so it hits the vision tower too. Leaving the
  tower's projections unfused silently dropped 98 tensors (24 blocks x 4, plus
  the merger's 2) -- the load report caught it. They are fused here, with a
  plain nn.Linear carrying a (param, tensor, shard_id) weight_loader rather
  than MergedReplicatedLinear, so the tower stays usable without an initialised
  TP group (the parity harness has none).

Load is clean: all 1614 parameters load, no checkpoint tensor dropped, and the
text path is unchanged (TTFT 3.15s, TPOT 46ms).

Still missing for actually serving an image: Glm5NextProcessor only exists in
transformers >= 5.16 while ATOM pins 5.12.1, so nothing turns an image into
pixel_values + image_grid_thw, and model_engine/multimodal.py has no glm5_next
branch. Recorded in recipes/GLM-5.3-Flash.md section 7.

* [atom] GLM-5.3-Flash: move the bring-up harness to tools/, add a raw-prompt runner

The harness is tooling, not a recipe, so it moves from recipes/glm5_3_flash/ to
tools/glm5_3_flash/ alongside the other standalone scripts. It does not belong
in tests/ either: everything there runs on CPU with aiter mocked, while these
need four GPUs and the 306 GiB checkpoint. recipes/GLM-5.3-Flash.md stays where
recipe docs live and now points at the new location.

Adds raw_prompt_offline.py: feeds an un-templated prompt through the offline
engine. atom_run.py applies the chat template, but GSM8K is scored on
/v1/completions with a plain few-shot prompt, so this isolates whether a
served-path failure is in the model or in serving -- same prompt, same weights,
one request, no scheduler.

* Fix GLM-5.3-Flash accuracy: pad the NoPE rope block to 64

GLM-5.3-Flash is NoPE (`qk_rope_head_dim == 0`), and this port expressed that
as a zero-WIDTH rope slice. Measured over all 1319 gsm8k questions that scored
flexible-extract 0.0099 / strict-match 0.0000; it now scores 0.9682 / 0.9689.

The zero-width representation fails in two independent ways:

* The paged MLA entry is sized `kv_lora_rank + qk_rope_head_dim`, so it came out
  512 while aiter's asm decode kernel is built for a 576-wide query. That kernel
  asserts the 576 only on the gfx1250 path and `cfg_mla_asm` never dispatches on
  head_size, so on gfx950 the mismatch is computed rather than rejected. 69% of
  replies came back empty and, of the rest, an extraction-free audit found the
  gold value in only 8.8% (98.0% after this change). Prefill was unaffected --
  it goes through head-dim-generic `flash_attn_varlen` -- which is why a
  prefill-only per-layer cosine check reads 0.9997 on a broken model.
* `KV_PeDim == 0` turns every `tl.arange(0, KV_PeDim)` into `arange(0, 0)`,
  which Triton rejects at compile time, and upstream aiter's three
  `gather_kv_b_proj` kernels carry no guard for it. It fires only on the
  cached-prefix prefill path, so a single-prompt run passes and any concurrent
  one dies ~15 s in with `NameError('kv_pe_data is not defined')`.

So materialize the rope block at `_ROPE_PAD = 64` lanes and hold it at zero.
The objection that ruled this out before -- `qk_nope_head_dim` is already 256
and CK caps head_dim at 256 -- conflates two constraints that apply to
different tensors: the latent/cache/decode side wants 576, the per-head
qk/prefill side wants <= 256. `MLAModules.rope_is_zero_pad` makes the MLA drop
the zero lanes at every `flash_attn_varlen_func` site, which is exact because
the lanes are identically zero, so both widths are satisfied at once. All five
sites are patched: prefill has plain, cached-single-pass and chunked
context/suffix variants, and a short-prompt smoke test only reaches one.

The padded width has to be declared to the cache allocator as
`config.mla_kv_entry_dim`. `KimiMLAGDNBackend` shadows the plain MLA allocator
and sizes the pool from the raw config, so without it the pool is built 512
wide under a 576-wide write and the server dies at startup. `aiter_mla` grows a
`mla_kv_entry_dim()` helper that prefers the declared width and falls back to
the old expression, leaving every other MLA model unchanged.

`_ZeroRopePad` appends the pad at call time and is deliberately not an
`nn.Module`: wrapping `q_b_proj` in one inserts a level into the parameter path
and the weights then silently never load.

Also widens `TritonMLAMetadataBuilder.set_mla_persistent_worker_buffers` to take
`**kwargs`. That is a pre-existing bug on main, unrelated to this model: the
aiter caller passes `is_cp_round_robin`, so the Triton MLA path dies at init for
any hybrid model and this port had no working fallback backend.

Adds the `GLM-5.3-Flash` CI accuracy entry (baseline 0.9682, threshold 0.94) so
the number is guarded, and records it in the recipe. Verified with the standard
CI config -- TP8, bf16 KV, max-num-batched-tokens 2048, 32 concurrent -- which
is the configuration that used to crash.

* Add the k-pool sparse indexer so GLM-5.3-Flash serves past 2048 tokens

Contexts at or below `index_topk` (2048) were already exact: top-k selects
every pool, so the expansion yields every token position and dense causal MLA
is numerically identical. Past that threshold the pooled selection decides what
the model attends to, and it was not implemented -- so long context was
silently wrong rather than refused.

This ports the measured implementation from the parallel GLM-5.3-Flash port:

* `model_ops/glm5_next/kpool.py` -- the pooled ops. One cached entry per
  `index_kpool` tokens: a per-dimension softmax over the pool weighted by
  `index_kpool_compress_gate` plus the per-slot `index_kpool_compress_ape`
  bias, Hadamard-rotated and FP8-quantized. `model_ops/kpool_indexer.py` stays
  as the dense `transformers`-parity oracle it was written to be, off the
  serving path.
* `Glm5NextIndexer` becomes a real `Indexer` subclass driving that path, with
  the NoPE override (no rope component to split or rotate) and a guard that
  REFUSES the regime where pooled and token-granular top-k genuinely differ
  rather than quietly returning the fallback.
* `KimiMLAGDNBackend` grows the pooled index cache sizing and the per-request
  tail slots. The tail rides the KDA state pool rather than a paged cache.
* `glm5_kpool_block_size()` sets the KV block to `index_kpool * 16 = 64`.
  `deepgemm_fp8_paged_mqa_logits` is correct only in its preshuffled layout,
  which needs the per-block index row count to be a multiple of 16; a 16-token
  block gives 4 rows and forces the index cache back to one row per token.

Three wiring points were needed beyond the ops themselves, and each failed in a
way worth recording:

* `is_deepseek_v32` -- which is what binds the index cache -- was gated on
  `use_mla`, but this model is served by the Kimi MLA+GDN hybrid backend and so
  sets `use_kimi_mla` instead. A sparse indexer is orthogonal to that choice.
  Without it `sparse_kv_indptr` stays None and decode dies in
  `mla_decode_stage1_asm_fwd`.
* MLA does not call the indexer; the model's forward has to drive it. Missing
  that call is silent: `is_sparse=True` makes MLA read a selection buffer that
  nothing ever fills, so short contexts stay perfect and only long ones break.
* The `sparse_attn_indexer_kpool` custom op and its `mutates_args` registration
  must be present, or inductor is free to hoist the MLA read above the in-place
  pooled cache write.

Measured, all 1319 gsm8k questions, chat + fewshot_as_multiturn, TP8, bf16 KV:

    3-shot  @2048  0.9682 / 0.9689   (dense, unchanged by this commit)
    16-shot @8192  0.9659 / 0.9666   (pooled)

Only the 16-shot row exercises pooling: 3-shot prompts are ~389 tokens, so they
never reach `index_topk` and the pooled selection is computed and discarded. A
short-context score shows only that the pooled writes did no harm. Long-context
retrieval additionally checked with a control at 6166 tokens -- two different
secrets in the same prompt shape, each recovered exactly, so it is retrieval
rather than a guess.

Adds the `GLM-5.3-Flash-kpool-16shot` CI entry (baseline 0.9659, threshold 0.94)
beside the 2048 one, which does not cover this path.

* Gate the pooled path in PR CI, and fix claims the last two commits falsified

`GLM-5.3-Flash-kpool-16shot` moves from `nightly` to `pr`. The pooled indexer is
the part of this port with the least static coverage -- there is no bit-exact
unit test for `model_ops/glm5_next/kpool.py` on this branch -- so it is the part
that most needs an end-to-end gate on every PR. The dense `GLM-5.3-Flash` entry
stays nightly; note that this leaves the dense-only regime (context at or below
`index_topk`) uncovered by PR CI.

Stale claims, each left false by the two commits before this one, and each of
them the kind that reads as fact to the next person:

* `atom_run.py` said "v1 of this model is only exact at or below index_topk
  (2048) tokens". The k-pool path lifted that.
* The module docstring said the vision tower was "not yet wired". It is built
  and loaded from `model.visual.*`; what is missing is upstream of it -- nothing
  turns an image into `pixel_values` -- so the tower cannot be reached at
  serving time. Naming the actual gap is the useful form.
* §2's mapping table pointed at `model_ops/kpool_indexer.py` as the k-pool
  implementation. That is the dense `transformers`-parity oracle; the serving
  path is `model_ops/glm5_next/kpool.py`.
* §7's preamble pinned `--max-model-len 2048` while the table below it lists an
  8192 row.

Also states the coverage gap plainly in §3, because it is not visible to a
reviewer: the k-pool code that serves has no bit-exact test here, and a gsm8k
score cannot pin what one would -- the pooling softmax being per-dimension
rather than per-slot, the Hadamard being orthonormal, and the pool/tail slot
arithmetic all have wrong spellings that still produce plausible keys and a
score in the right range.

Documentation and CI config only; no behaviour change.

* Silence BLE001 on the multimodal-config fallback so PR ruff passes

The ruff job runs reviewdog with `-filter-mode=diff_context -fail-on-error=true`,
so any finding on a line inside this PR's hunks fails it -- including lines the
PR only touches as context. `get_hf_config`'s broad `except Exception` is one:
unchanged itself, but sitting in the hunk that rebuilds `_multimodal_config`.

The catch is deliberate and stays. `AutoConfig.from_pretrained` raises whatever
the architecture's own code raises when this image has no class for it, so
enumerating the types would be guesswork that silently stops handling one. Marked
with the reason, matching the `# noqa: BLE001 - <why>` convention already used
across `atom/diffusion/`.

* Give a PAGE unit the index-cache regions it was priced with

PR CI's GLM-5.3-Flash-kpool-16shot job never reaches the eval: every
rank dies in `allocate_kv_cache`, before a single request.

    RuntimeError: a PAGE unit is 836352 B but this pool gives a logical
    block 11 rows x 73728 B = 811008 B; the two disagree about block
    granularity

Neither side is wrong alone. This branch put GLM-5.3's sparse indexer
key cache in the paged pool, so `sub_pool_specs` prices a block with it
in -- 11 indexer layers x 16 pooled rows x 144 B = 25,344 B on top of
the 11 MLA rows, which is the 836352/811008 gap exactly. Meanwhile main
gained `_page_unit_regions`, the destination side of K3's PAGE-backed
KDA checkpoints, and it describes a unit as the MLA rows alone -- true
for K3, which has no indexer. The two only meet in the merge, and the
equality between them is the assertion that fires.

Naming the shorter list is not a smaller, safer copy:
`units_per_checkpoint` is `ceil(image_bytes / page_unit_bytes)`, priced
against the whole block, so regions covering 811,008 of every 836,352
priced bytes leave the tail of every image with nowhere to land. So the
index cache is named here too, the way the DSV4 builder names its
indexer pools. Its block axis is the logical block a unit id already
carries -- no `block_ratio` conversion, unlike the MLA side -- so a unit
owns one contiguous region per indexer layer, the layers a fixed stride
apart: the same affine answer the MLA rows give. The region list now
sums to the priced 836,352 B exactly.

The check stays, since it is the one relation that cannot hold if the
granularity is wrong, and gains the index term in its message so the
next mismatch says which pool is short.

Tests: ten cases over a stub whose indexer shares the pool -- addresses
against what slicing gives, the region sum against the price, a unit
priced without the index cache refused, a model without an indexer left
with the MLA regions alone, both pools in the cache key, and a
store/restore round trip whose image genuinely spills into the index
regions and leaves every block past it untouched.

* Read the dense last-page lengths only where a prefill has them

`_forward_prefill_mla` derived its sequence count from
`attn_metadata.kv_last_page_lens` before splitting dense from sparse. That
array is per SEQUENCE and the prefill builder fills it only for a drafter or a
cached prefix; a sparse (DSA) prefill has neither in the common case and
carries its own per-QUERY `sparse_kv_last_page_lens` instead, which the branch
three lines below substitutes for exactly that reason. So the read landed on
`None` and took down all eight ranks:

    n_seqs = attn_metadata.kv_last_page_lens.shape[0]
    AttributeError: NoneType object has no attribute shape

GLM-5.3-Flash is the first model in CI to reach it. It is sparse, has no MTP
drafter yet, and serves with prefix caching off (its KDA state is
per-request), so every request past `index_topk` = 2048 tokens crashed the
engine -- which is every 16-shot GSM8K prompt, on the first one.

The value it computed was dead on that path anyway: the only use is cutting
`cu_seqlens_q`, and sparse replaces that with `sparse_cu_seqlens_q` regardless.
Move both into the dense branch, where the array is present because the kernel
call itself needs it.

Reproduced and verified on 8x MI355X, GLM-5.3-Flash TP8, the CI accuracy
config verbatim: gsm8k 16-shot pooled over all 1319 questions, previously dead
before the first response, now flexible-extract 0.9583 / strict-match 0.9591
against a 0.94 threshold.

* Drop the bring-up harness and the test report that rode in with it

`tools/glm5_3_flash/` was scaffolding for building a transformers oracle to
diff this port against. That job is finished and its results are in the recipe;
what would ship is 1409 lines that nothing in ATOM imports, no test exercises,
and CI never runs.

It could not be run from ATOM anyway. The harness needs `transformers>=5.16`
for `glm5_next` while ATOM pins 5.12.1, so its own Dockerfile builds a separate
image; five of the eleven files exist only to work around `finegrained-fp8`
failing to compile on gfx950 (recipe section 4b), and `vision_parity.py` covers
a tower that no input path reaches yet. `atom_run.py` and `raw_prompt_offline.py`
duplicate `atom.examples.simple_inference` and the offline engine, differing
only in using the oracle prompt. Code that cannot run in this repository's
environment and has no caller does not get maintained; it gets believed, and
then found stale.

`unit-report.xml` is a 745 KB pytest JUnit artifact. `run_unit_tests.sh` writes
it into the workspace root, and it was committed by accident in 545593a with
four failures recorded inside it. #1662 removed one of these before, so this
adds a .gitignore rule rather than relying on care.

The recipe keeps every measurement and now stands on its own: section 5 says
what rebuilding the oracle takes, including the two non-obvious parts (the FP8
kernel replacement, and that having both backends is what exposed the aiter
multi-GPU bug of section 4c); sections 3 and 6 say which results are bring-up
findings and which are the standing CI guard. The quickstart now runs
`atom.examples.simple_inference`, verified on GLM-5.3-Flash at TP4.

* Fix GLM-5.3 k-pool state consistency

Preserve partial pool state across checkpoint, relocation, and chunk boundaries so long-context requests cannot consume stale cross-request data. Align configuration and quantization edge cases with their existing contracts.

Co-authored-by: Cursor <cursoragent@cursor.com>

* Format GLM-5.3 review fixes

Apply the repository's Black formatting so the correctness fixes pass PR pre-checkin.

Co-authored-by: Cursor <cursoragent@cursor.com>

* Fix GLM-5.3 second-pass review findings

Co-authored-by: Cursor <cursoragent@cursor.com>

* Format GLM-5.3 environment settings

Co-authored-by: Cursor <cursoragent@cursor.com>

* Refactor GLM-5.3 model into ATOM style

Co-authored-by: Cursor <cursoragent@cursor.com>

* Size the chunked-prefill K workspace from mla_kv_entry_dim

The has_cached chunked-prefill workspace was still sized from the raw HF
config, `qk_nope_head_dim + qk_rope_head_dim`. That is the one place the
NoPE widening was not swept into: the pool allocator and every
flash-attention call site already go through `mla_kv_entry_dim` /
`rope_is_zero_pad`, and this buffer is fed to the gather at three call
sites, so it was the last raw-config geometry left on the MLA path.

For GLM-5.3-Flash the config deliberately holds `qk_rope_head_dim` at its
true 0 -- the indexer has to stay NoPE -- and the rope block is widened to
a 64-lane zero pad on the MLA side only. The raw sum therefore gives 256
where the kernels want 320.

It does not fail as a size mismatch. `gather_kv_b_proj` takes the rope
width from its DESTINATION buffer (`qk_nope_pe_dim = k_prefix.shape[-1]`)
and the nope width from the kv_b_proj weight, so a 256-wide workspace
against a 256-wide nope half makes `KV_PeDim = 0` and Triton rejects the
resulting `tl.arange(0, 0)` at compile time -- exactly the failure the
`_ROPE_PAD` note in glm5_next.py warns about, one kernel below where the
width is chosen. The line right after the gather already calls
`_drop_rope_pad` on this buffer, so the surrounding code was written for
the padded width; only the allocation disagreed.

`mla_qk_head_dim` derives the rope width from `mla_kv_entry_dim` instead,
which is the declared padded width. For a real-rope model that reduces to
the sum it replaces, so no shipped model changes shape.

The line dates to #911 and only became wrong when a NoPE model arrived, so
it is not reachable on main today. Note also that it needs a CACHED prefix
to fire, which `--no-enable_prefix_caching` in the CI entry suppresses --
CI would not have caught this.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Run the GLM MoE router in fp32

ATOM had no reference to `moe_router_dtype` anywhere, so the router ran at
the model dtype. That is wrong for every `glm_moe_dsa` model, and the
damage is not in the gate logits but in the `noaux_tc` correction bias.

Its 256 values sit in a narrow band riding a large offset -- measured on
GLM-5.3-MXFP4 layer 10: min 6.817, max 7.063, 238 distinct values. bf16's
ULP at ~7 is 1/64, four orders of magnitude coarser than the ~6e-6 median
spacing, so loading that tensor into a bf16 parameter collapses 238
distinct values onto 8 and discards almost all of the selection signal it
exists to carry. Every GLM-5.x checkpoint on disk ships it as fp32, and
GLM-5, 5.1, 5.2 and 5.3 all share the same [6.02, 8.11] range.

vLLM forces fp32 routing for this model_type unconditionally, including
for GLM-5/5.1/5.2, whose configs predate `moe_router_dtype` and so cannot
ask for it; keying on model_type rather than the config key matches that
and keeps those generations correct. ATOM already did this for `glm4_moe`
and `glm5_next`, both of which build the gate and the bias in fp32 --
`deepseek_v2.py` was the one GLM path that missed it.

The dtype must be set at parameter CREATION. The loader casts the
checkpoint tensor into whatever the parameter holds, so an fp32 bias
landing in a bf16 parameter is rounded once, on load, and the existing
`.to(torch.float32)` calls in the MoE backends then widen a number whose
low bits are already gone.

Gate output and bias are not independently choosable: aiter's
`biased_grouped_topk` dispatches on `gating_output.dtype()` and then
reinterpret_casts `correction_bias` to that same `scalar_t`, so an fp32
gate with a bf16 bias would read the bias buffer at double width. One
dtype governs both, and `ReplicatedLinear.forward` already accepts
`otype`. Non-GLM models resolve to None and are untouched, keeping both
`torch.empty` and the gate call byte-identical.

Measured on GSM8K, and it is NOT a scores change: paired McNemar over all
1319 questions gives b=7/c=6 (p=1.00) in chat 5-shot and b=52/c=56
(p=0.77) in 3-shot completions. Two runs of the UNFIXED code scored 0.9719
and 0.9773 in chat, a wider gap than the change produces, so aggregate
rates cannot speak to this either way. This is a faithfulness fix; GSM8K
simply does not probe routing quality. It costs +0.14-0.29 ms per decode
step across 75 MoE layers, inflated because aiter carries no tuned config
for the fp32-output gate shape and falls back to a torch path.

GLM-5/5.2 CI baselines were measured on the old routing numerics and
should be re-confirmed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Gate GLM-5.3 accuracy in CI, and drop the redundant Flash entry

GLM-5.3 (`glm_moe_dsa`) ran on ATOM but had no accuracy entry and no
recipe, and its MXFP4 checkpoint ships with no evaluation on its model
card at all. Adds GLM-5.3-FP8 (tp8, baseline 0.9719) and GLM-5.3-MXFP4
(tp4 + ptpc_fp8 online quant, baseline 0.9765), both over all 1319
questions on 8xMI355X.

Both must run the chat client, not the default raw-completions one.
GLM-5.3's template ends `<|assistant|><think>` and the server reports that
it has no switch for reasoning, so few-shot completions suppress the
reasoning the model is trained to do: the same checkpoints score 0.9136
(MXFP4) and 0.9234 (FP8) that way, which would fail this threshold on a
healthy model. Chat is also far steadier -- two runs of identical code
disagree on 13 of 1319 questions in chat versus 108 in 3-shot completions.

MXFP4 tracks its FP8 source (0.9765 vs 0.9719), so the quantization costs
nothing detectable here.

The short-context GLM-5.3-Flash entry is removed in favour of the 16-shot
kpool one. Below index_topk the indexer selects every token and
attention_mla runs dense, so that entry exercised a strict subset of the
16-shot path while costing a second 8-GPU nightly.

Thresholds are set to 0.96 as requested. This is comfortable for the two
GLM-5.3 entries (~1.6pp of headroom below the worst observed run) but
TIGHT for GLM-5.3-Flash-kpool-16shot: its baseline is 0.9659 and repeat
runs of identical code have landed at 0.9613/0.9629/0.9644, so the worst
observed run clears 0.96 by 0.0013 and that entry should be expected to
flake. It was 0.94 before.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Modernize an annotation this PR pulled into ruff's diff context

CI runs ruff through reviewdog with `-filter-mode=diff_context` and
`-fail-on-error=true`, so a finding fails the PR whenever it lands on a line
inside the diff -- including lines this PR never edited. Adding
`_moe_router_dtype` directly above `_can_fuse_indexer_wk_weights_proj` brought
that function's pre-existing `Optional[QuantizationConfig]` into the context
window, and UP045 fired on it.

`Optional` still has 71 other uses in this file, so the import stays live and
nothing else in the diff moves.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Fix GLM-5.3 third-pass review findings

Finding 1 (raw-config chunk workspace) was already fixed; this covers 2-5.

2. Bound the decode index conversion to the request's own output region.
   `_convert_req_index_to_global_index_kernel` gated its store on
   `indice_id < kv_len`, which counts entries on the INPUT side and is not the
   width of the output region -- that is `page_kv_indptr[b+1] - [b]`, since the
   output is packed by that indptr. A pooled selection makes the two diverge
   (the row is padded to `round_up(index_topk + kpool - 1, 128)` = 2176 columns
   while the region holds at most 2051) and a long context pushes kv_len past
   both, so every column stored. The surplus columns are top-k padding, -1,
   which `tl.where(out_val >= 0, ...)` maps to cache slot 0 -- written over the
   START of request b+1. `OUT_NUMEL` only caught the final request, so it was
   silent for every other one. Now masked by `out_offset < out_kv_end`, which is
   a no-op wherever the two widths already agree.

3. Widen `_is_kda()` to both KDA models. Its two siblings in the same file
   (`gdn_attn.py:129`, `:331`) were widened for GLM-5.3-Flash and this one was
   not, so it answered "not KDA" for a KDA model and
   `_replayssm_buffer_shapes()` sized the ReplaySSM record buffers for the wrong
   head geometry -- an out-of-bounds write reachable with
   `ATOM_ENABLE_REPLAYSSM=1`. Swept the tree: no `== "kimi_linear"` test remains.

4. Let `--load_dummy` start. The KDA shard-coverage check exists to catch a
   checkpoint that supplied only part of q/k/v, but dummy loading never runs a
   weight_loader, so the set is empty by design and all 34 KDA layers refused to
   start. Skipped when `load_dummy` is set; the gate fold still runs, so the
   layer keeps the shape and dtype it would otherwise have had.

5. Guard both sparsity-off knobs the same way, at startup. `ATOM_GLM5_KPOOL=0`
   raised `NotImplementedError` from inside `forward_impl`, which takes the
   engine down mid-batch rather than refusing a request the way `envs.py` and
   `docs/environment_variables.md` both describe; `ATOM_GLM5_FORCE_DENSE_MLA`
   had no length guard at all despite being documented as short-context only, so
   the two failed in opposite directions. Both are now validated in
   `Config.__post_init__` against `max_model_len`, which already bounds
   `max_seqlen_k`: the launch fails cleanly with a message naming the fix, and a
   per-forward branch leaves a 45-layer hot path.

Unit suite unchanged at 4831 passed / 14 failed; those 14 fail identically on
plain origin/main (they arrived with #1965 / #2113, after the review's measured
4902/0). black and the diff-context ruff gate are clean.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Keep the new unit tests off the AITER import path so CI can run them

The Pre Checkin unit-test job failed with pytest exit code 2 -- a COLLECTION
error, not test failures. It runs on a plain CPU runner with no AITER, and the
two tests added for the round-3 fixes imported their subjects from modules that
pull AITER at module scope (`aiter_mla.py:11`, `deepseek_v2.py:30`), so both
files errored during collection and took the whole run down.

Reproduced locally by blocking `aiter` on the meta path: "Interrupted: 2 errors
during collection", exit 2, matching CI.

`importorskip` would have turned the job green while leaving the tests unrun,
which is the complaint the last two reviews already made about this PR's test
surface. Moved the subjects instead, into modules that need nothing but the
config:

* `atom/model_ops/mla_geometry.py` -- `mla_kv_entry_dim`, `mla_qk_head_dim`,
  `aligned_index_cache_dim`. Pure integer arithmetic on config fields.
* `atom/models/moe_router.py` -- `moe_router_dtype`. Needs torch and the
  config, nothing else.

Both follow the reasoning already written down in `atom/model_ops/__init__.py`,
which resolves its attention frontend lazily for exactly this purpose.
`kimi_mla_gdn_attn` now imports the geometry helpers straight from the light
module rather than through `aiter_mla`, so no re-export is left dangling.

Verified with AITER blocked: both modules import with zero `aiter.*` in
sys.modules, and all 8 tests run and pass. Local full suite is unchanged at
4831 passed / 14 failed -- those 14 fail identically on plain origin/main.
black clean, diff-context ruff clean.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Guard the new unit tests with importorskip("aiter") instead of relocating them

Reverts the module split from 6b0e5d4 and takes the simpler route requested:
the two tests added for the round-3 fixes now guard on `pytest.importorskip
("aiter")` before importing their subjects, matching
`tests/plugin/test_vllm_mha_backend.py`.

The Pre Checkin unit-test job runs on a CPU-only runner with no AITER, and
`aiter_mla.py` and `deepseek_v2.py` both import AITER at module scope, so
collecting these files raised `ModuleNotFoundError` and pytest exited 2 --
a collection interrupt, which fails the whole job rather than one test.

`mla_geometry.py` / `moe_router.py` are gone again; `mla_kv_entry_dim`,
`mla_qk_head_dim`, `aligned_index_cache_dim` and `_moe_router_dtype` sit back
where they were, and `kimi_mla_gdn_attn` imports the geometry helpers from
`aiter_mla` as before, so the production import graph is exactly what it was
two commits ago.

Trade-off, recorded because it is the known cost rather than an oversight:
these 8 assertions now SKIP on the CPU runner instead of executing, so they
only ever run on a machine with AITER. Verified both directions -- with AITER
blocked they report as 2 skips and the run exits cleanly; with AITER present
all 8 pass.

black clean (682 files), diff-context ruff clean, no `noqa` needed: the repo's
ruff config does not enable E402, so `importorskip` ahead of the imports is
accepted as-is.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Fix GLM-5.3 fourth-pass review findings

Round 4 was taken at a883ea8, so three of its items were already fixed by
b01d423 / cf02323: `_is_kda()` widening, `_assert_kpool_regime` raising
from inside the forward, and finding 1 (the two new test files aborting the
whole CI unit run -- now guarded with `importorskip("aiter")`). The rest:

2. `_moe_router_dtype` now matches the sibling predicates in this file and
   accepts the MTP draft. `SpeculativeConfig._MTP_TYPE_MAP` rewrites
   `glm_moe_dsa` to `deepseek_mtp` while keeping the GLM-only
   `index_share_for_mtp_iteration` marker, so a model_type test alone left the
   draft router in bf16 against an fp32 target -- different experts selected on
   each side and a lower acceptance rate, which shows up as a throughput
   regression while task accuracy stays clean.

4. Fake-EPLB's synthetic router table is cast to the dtype of the logits it
   replaces. It is built once at the default dtype, and this PR's own docstring
   states the invariant it broke: `biased_grouped_topk` dispatches on
   `gating_output.dtype()` and reinterpret_casts `correction_bias` to that
   scalar_t, so a bf16 table against an fp32 bias read that buffer at half
   width. Stating an invariant and leaving a path unswept was the actual
   mistake.

5. `hybrid_mla_layers` no longer treats an empty list as the hybrid layout.
   `[] is not None`, so a config without `full_attn_layers` took the hybrid
   branch with zero MLA layers and every index-cache region was numbered on top
   of the KV regions instead of after them -- a P/D transfer would write index
   bytes into KV rows. The adjacent length check cannot catch it because both
   sides derive from the same empty list.

6. Not a defect, and now recorded so it is not re-raised. `_drop_rope_pad`
   hands non-contiguous slices to five flash-attention sites; measured on
   gfx950, the output is bit-identical to a `.contiguous()` copy (max abs diff
   0.0), so the kernel takes its row pitch from the stride, and the call is
   ~1.5% slower at 4096 tokens -- far cheaper than materializing K per chunk
   per layer. Written into the docstring with the numbers.

8. The accuracy manifest. Reverted the `0.90 -> 0.9` reformat on gpt-oss-120b
   (x2) and Llama-3.3-70B: that was a `json.load`/`json.dump` round-trip in my
   own edit script, and those three entries are now byte-identical to main
   again. Separately, `GLM-5.3-Flash-kpool-16shot` keeps threshold 0.94 rather
   than the 0.96 asked for earlier: it is a `pr`-level gate, and repeat runs of
   IDENTICAL code measured 0.9613/0.9629/0.9644/0.9659, so 0.96 clears by
   0.0013 in the worst case and would fail roughly one unrelated PR in four.
   Baseline moved to the 0.9613 measured on this tree.

Also modernized one annotation that finding 2's edit pulled into ruff's
diff_context window, the same way a883ea8 did.

Local suite unchanged at 4831 passed / 14 failed; those 14 fail identically on
plain origin/main. black clean (682 files), diff-context ruff clean, catalog
schema valid.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Modernize three more annotations ruff 0.16.5 flags in the diff context

The Ruff check failed on d77ef46 while my local replay of the same gate
reported clean. Cause: CI does `pip3 install ruff` and got 0.16.5, this
container had 0.16.3, and 0.16.5 reports UP045 on annotations the older build
left alone. Reproduced by installing 0.16.5 into a scratch venv and re-running
the gate, which surfaced the same finding CI did.

Three `Optional[...]` annotations in `DeepseekV2MoE` are now `X | None`. All
three are pre-existing lines that this PR's edits pulled into ruff's
`diff_context` window -- the same mechanism as a883ea8, which is why the
gate fails on lines the PR never meant to touch. `Optional` still has 68 uses
in the file, so the import stays live.

Checked with a +/-10 line window as well as CI's +/-3, so a slightly wider
context in reviewdog will not surface another one.

black clean, catalog valid, local suite unchanged at 4831 passed / 14 failed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: perzhang <perzhang@amd.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
zejunchen-zejun pushed a commit that referenced this pull request Sep 3, 2026
Brings in GLM-5.3-Flash (#2051), fuse prefill dcp (#2113), DSV4 PCP+MTP
fix (#2049). Two conflicts resolved: the page-unit geometry refactor
(PageUnitGeometryMixin) now carries main's sparse-indexer index-cache
region support; kept main's new kpool-tail checkpoint tests.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants