Skip to content

feat(v41): mono decode, FP4 index plane, shared mono framework - #2479

Merged
valarLip merged 3 commits into
mainfrom
feat/mono-v41-fp4-index
Oct 6, 2026
Merged

valarLip merged 3 commits into
mainfrom
feat/mono-v41-fp4-index

Conversation

@valarLip

@valarLip valarLip commented Oct 6, 2026

Copy link
Copy Markdown
Collaborator

Summary

DeepSeek-V4.1 mono decode, the V4.1 FP4 index plane, and the shared mono framework that MiniMax-M3 and V4.1 now both build on.

  • atom/mono: shared mono decode framework.
    • runtime: TP consensus, peer memory and step fence, step mailboxes, lifecycle, per-width builds, routing, debug reports, deployment refusals, check-mode compare, kernel resource readout.
    • plan: execution model, build keys, TP shard, mailbox layout, hand-off contract check.
    • device: tagged mailbox, peer all-reduce, FP8/FP4 MX, wave and math primitives, exact small-k top-k by counting (device/select), and batched strided loads (batched_rounds, rows_to_lds).
  • DeepSeek-V4.1 mono decode (atom/models/deepseek_v41/mono):
    • fused per-layer K1 / K2a / K2b kernels and the DSpark draft runner;
    • installed after load through model_runner.mono_decode_installers.
  • V4.1 FP4 index plane (--index_cache_dtype fp4), on both the original path and mono:
    • page-8 value / scale planes;
    • FP4 index writer and fused q RoPE + FP4 quant;
    • FlyDSL row-group scorer with request-level ragged rows, scoring off the request PAGE table.
    • CI configs for V4.1 move to fp4.
  • MiniMax-M3 mono:
    • on the shared framework;
    • selection rewritten as exact top-k by counting with a group-floor prefilter, with the long path from 512 blocks.
  • Default CUDAGraph capture ladder now covers every batch size of 1..8. Before, a padded batch ran the next captured size's step.

Performance

MiniMax-M3, one fused layer on 4 GPUs, µs per layer. Selections and layer outputs are byte-identical before and after.

Case Before After
S=1, 20k context 38.0 35.5
S=1, 1M context 62.3 ~58
S=16, 20k context 133.1 125.2

V4.1, server step time with every batch size of 1..8 captured (padded → own graph). bs 1/2/4/8 are unchanged.

Batch size Change
bs3 −4.3%
bs5 −14.3%
bs6 −7.5%
bs7 −3.2%

The graph pool stays at 1.70 GB, the same as with 10 graphs: the graphs share one pool sized by the largest.

V4.1 slice-stage load batching: step time −0.3 to −0.7% at conc 1/2/4/8, over two alternating server rounds.

Test plan

  • bash .github/scripts/run_unit_tests.sh: 10204 passed, 2 failed, 351 skipped. Both failures also fail on origin/main d16c457 on this machine: test_compilation.py::test_runtime_guard_with_live_steps_and_aux_hooks[3-False] and test_triton_inductor_pairing.py (torch 2.9.1 / triton 3.8.0 pairing)
  • Mono unit and contract tests, V4.1 + M3 + framework: 645 passed, 1 skipped
  • M3 whole-layer bit A/B against the pre-change tree: S = 1 / 4 / 8 / 16, 3k–1M, CP and long path, ALL EQUAL
  • V4.1 K2 output bit A/B (k2_bench.py CHECK=): S = 6 / 48, ALL EQUAL
  • V4.1 spill sweep: TP2 / TP4, every width, bound 1M, no scratch
  • V4.1 server: GSM8K and step time across conc 1/2/4/8 (FP4 index plane GSM8K 0.9242 / 0.9249, from the FP4 integration runs, not rerun for this PR)
  • CI

- atom/mono: shared mono decode framework (runtime: TP consensus, peer
  memory and step fence, step mailboxes, lifecycle, widths, routing,
  debug reports, deployment refusals, check compare; plan: execution
  model, build keys, TP shard, mailbox layout, arithmetic, hand-off
  check; device: tagged mailbox publish, peer all-reduce, FP8/FP4 MX,
  wave/math primitives)
- MiniMax-M3 mono moved onto the shared framework (peer_buffer and
  timeline removed; contract, sources, dense_post, stages added)
- DeepSeek-V4.1 mono decode (atom/models/deepseek_v41/mono): fused
  per-layer K1/K2a/K2b kernels, DSpark draft runner, installed after
  load through model_runner.mono_decode_installers
- V4.1 FP4 index plane (--index_cache_dtype fp4), original path and
  mono: page-8 values/scales planes, ATOM FP4 index writer, fused q
  RoPE + FP4 quant, FlyDSL row-group scorer; mono K1 FP4 quant and K2a
  FP4 score, whose task plan is laid out on the host by a step planner
  (DeepseekV41MetadataBuilder.add_step_planner) and published with the
  step
- FP4 index decode scoring by request (aiter row-group ragged rows via
  flydsl_pa_mqa_logits_fp4: a request's rows share each key load, lengths
  from the per-row bounds, shared out in the kernel): DeepSeek-V4 decode
  drops its per-step cta_info schedule; V4.1 FULL layers score off the
  request PAGE table (DecodeRequests), with no per-token tile table
- V4.1 MoE A8W4, mHC / Engram / DSpark / rejection sampler fixes, docs
- MiniMax-M3 mono selection: exact top-k by counting 64-bit keys with a
  group-floor prefilter (atom/mono/device/select), shared by the short
  and long paths; long path from 512 blocks; the selector's LDS in the
  activation pool. S=1 20k 38.0 -> 35.5 us a layer,
  1M 62.3 -> ~58, S=16 133 -> 125; selections and layer outputs
  byte-identical
- V4.1 lds_topk fixed-cost cuts (quad loads, pass-0 run merging, no
  passes when n <= k, keys in registers) and stage_slice loads batched
  ahead of their rounds (mono/device/ops.batched_rounds); M3's global ->
  LDS copies through ops.rows_to_lds; CM_NT on weights read once a step
- kernel resources (VGPR / SGPR / spills / scratch / LDS) from code-object
  metadata: runtime.compile.kernel_resources / launcher_resources
- default CUDAGraph capture ladder covers every batch of 1..8 (a padded
  batch ran the next size's step: V4.1 bs5 -14 %, bs6 -7.5 %; the graph
  pool is unchanged); one default for the CLI and EngineArgs
- cleanups: shared DIAG_WORDS in mono plan layout, dead seed comments
@github-actions

github-actions Bot commented Oct 6, 2026

Copy link
Copy Markdown
Contributor

🏷️ CI Guide

Runs automatically on every eligible PR before approval:

  • ✅ Pre Checkin: Black, Ruff, catalog schema validation, non-GPU unit tests

Heavy model tests:

  • ✅ Run after the PR is approved and Pre Checkin passes
  • ✅ Run immediately when an approval review is submitted
  • ✅ Can be requested before approval with labels
Label Tests
ci:full Run all heavy PR model tests: native ATOM, vLLM, and SGLang
ci:atom Run native ATOM model accuracy tests
ci:vllm Run ATOM vLLM OOT model accuracy tests
ci:sglang Run ATOM SGLang model accuracy tests

Heavy jobs are skipped when the PR is not approved and no matching ci:* label is present.
Add labels via the sidebar or gh pr edit 2479 --add-label <label>

…lect without aiter

lifecycle.py imports peer_memory, which needs aiter: test_mono_runtime.py
importing bind_agreed from it failed collection on the CPU CI runner.
bind_agreed is the TP agreement on a runner's bind, next to tp_agree;
lifecycle keeps the owned peer buffer.
The routing wrapper (MonoDecodeModel) sits outside the compiled model, so
a step's mono / original decision stays in Python at level 3 as at level 0.
@valarLip
valarLip merged commit 0873517 into main Oct 6, 2026
@valarLip
valarLip deleted the feat/mono-v41-fp4-index branch October 6, 2026 14:32
valarLip pushed a commit that referenced this pull request Oct 7, 2026
…2488)

The DeepSeek-V4 decode prep padded `cu_seqlens_q` to `running_bs` but
published `block_tables` with only `scheduled_bs` rows. Since #2479 the FP4
indexer's ragged scorer (`flydsl_pa_mqa_logits_fp4` with `query_start_loc`)
takes its batch from `cu_seqlens_q` and asserts `block_tables.shape[0] >=
batch`. So an eager DP-padded decode step, one whose rank holds fewer
requests than the padded batch, died with `AssertionError` in
`pa_mqa_logits_fp4_rowgroup`. Captured graphs replay against the full buffer
and never hit it.

Pad the table to `running_bs` (zeroed rows, as the TBO path already does)
and publish that many rows. Pad rows carry row end 0, so no page is read
through them.

Signed-off-by: yihonglie <hyi@amd.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
seungrokj added a commit to SemiAnalysisAI/InferenceX that referenced this pull request Oct 8, 2026
… every TP4 point

minimaxm3-fp4-mi355x-atom-agentic-mtp:

- Image rocm/atom-dev:nightly_202609281543 (ATOM 0.1.7.dev43+gaa0c5c3a1)
  -> nightly_202610070333 (sha256:2f605a3e8b459a0385f784edb0335c11892277fee650fe42044e4af39e36dd01;
  ATOM 0.1.7.dev60+g7fc0926c6, ROCm 7.2.4). ROCm/ATOM#2479 moves M3 mono
  decode onto the shared mono framework (mailbox flags published only after
  their stores drain; exact top-k sparse selection); ROCm/ATOM#2483 rebuilds
  the per-step mailbox clear and cross-rank fence as one FlyDSL launch,
  replacing a Triton fence that could spin forever on gfx950; ROCm/ATOM#2422
  fixes the mono long-context decision. Upstream
  recipes/MiniMax-M3-Agentic-InferenceX.md is unchanged.
- Mono decode on at every TP4 point: drop ATOM_MONO_ENABLE=0 from the
  indexer-CP points C15/C20/C24/C28. Mono serves decode steps of up to 16
  tokens (four requests with EAGLE3 K3), covering all of TP4 C1/C2/C4.
  ATOM builds mono for TP4 only, so TP2 runs the unfused path.
- Add TP2 C4 (max-num-seqs 8); GPU-resident TP2 points are C1/C2/C4.
- Record LMCache 0.5.6.dev139+gf22dec28 (the wheel in the new image) for
  both DRAM-offload arms; offload settings unchanged.
- Recipe cites NVIDIA/srt-slurm#507 (in srt-slurm since v2.36.0) for
  extra-kv-connectors instead of the dropped 507 patch.
- No data-type or precision change to the EAGLE3 draft
  (Inferact/MiniMax-M3-EAGLE3-GQA).

Risk: ROCm/ATOM#2452 (CUDA-graph pad rows in the wide MoE path, 5+ tokens)
is not in this image; every batch size 1-18 is captured, so steps of up to
four requests are never padded.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
seungrokj added a commit to SemiAnalysisAI/InferenceX that referenced this pull request Oct 8, 2026
…ghtly

dsv41flash-fp4-mi355x-atom-agentic-dspark:

- Restore DeepSeek-V4.1-Flash FP4 ATOM AgentX on MI355X (#3387) on the
  native srt-slurm recipe srt-slurm-recipes/dsv41flash/atom/mi355x-fp4-mtp/
  agentic.yaml (legacy script and master entry were removed in #3461/#3463).
  TP2 conc [1,2,4,5,6,8,16,32,64,128] (adds 4,5,6,128); TP4 conc
  [1,4,5,6,8,32] (adds 1,4,5,6; drops 2,16,64). No EP or KV offload.
  Conc 1-8 covers the batch sizes V4.1 mono decode serves (<= 8 requests).
- Image rocm/atom-dev:nightly_202609250902 -> nightly_202610070333
  (sha256:2f605a3e8b459a0385f784edb0335c11892277fee650fe42044e4af39e36dd01;
  ATOM 0.1.7.dev60+g7fc0926c6, ROCm 7.2.4):
  - ROCm/ATOM#2419: group32 FP8 weight preshuffle and BF16 RoPE tables for V4.1.
  - ROCm/ATOM#2479: V4.1 mono decode (on by default, <= 8 requests a step
    for this TP2/TP4, BF16-KV, five-token DSpark deployment) and the FP4
    index plane used by upstream recipes/DeepSeek-V4.1-Flash-Agentic.md.
  - ROCm/aiter#6174: gfx950 tuned rows for six V4.1-Flash TP4 bf16 GEMM
    shapes that fell back to torch (DSpark verify batches, TP4 lm_head).
  - ROCm/ATOM#2483: replaces the Triton mono step fence that could spin
    forever on gfx950; on the 1006 nightlies the TP2 c1 canary hung in
    warmup on the DSpark draft's first mono step (runs 37486079757,
    37493871230).
- Server settings follow the upstream recipe (BF16 KV, index cache FP8 ->
  FP4 per upstream at 7fc0926c, max-num-seqs 128, 16K batching/prefill
  chunks, prefix caching block 16, 8K state checkpoints, level 3, FULL
  graphs, dsml_v41 tool parser); every point now captures
  [1..32, 48, 64, 128]. Five-token DSpark uses golden AL 3.51 (thinking_on)
  for throughput, real acceptance for eval; 3600 s weka_062126 replay with
  five warmup requests per lane.
- DSpark draft (layers 37-39, inline): no --online_quant_config; attention
  and projection linears stay native FP8 32x32. Via the image bump, the
  draft's routed FP4 experts take MXFP8 instead of MXFP4 activations
  (ROCm/ATOM#2479) and the V4.1 RoPE tables are BF16 instead of FP32
  (ROCm/ATOM#2419). The draft has no indexer, so index-cache-dtype fp4
  does not apply to it.
- TP4 conc 32: ATOM_PREFILL_DECODE_INTERVAL=16 (local 3600 s replay:
  108.2 -> 150.9 tok/s/user, ITL p90 9.24 -> 6.63 ms, TTFT p50
  0.51 -> 1.14 s, throughput unchanged).
- TP2 conc 64: ATOM_PREFILL_DECODE_INTERVAL=4 (30.3 -> 38.8 tok/s/user,
  105.4k -> 115.2k tok/s/GPU, 6487 -> 6941 requests, TTFT p90 2.3 -> 6.4 s).
  All other points unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Jasen2201 added a commit that referenced this pull request Oct 8, 2026
Resolve the backends.py conflict: main (#2479) removed the FP4 decode
schedule, so `_publish_indexer_fp4_decode_schedule` and its EagleProposer
call sites go. Keep this branch's `_publish_dcp_token_block_tables` no-op on
CommonAttentionBuilder, with a docstring that no longer points at the
removed method.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
seungrokj added a commit to SemiAnalysisAI/InferenceX that referenced this pull request Oct 8, 2026
… every TP4 point

minimaxm3-fp4-mi355x-atom-agentic-mtp:

- Image rocm/atom-dev:nightly_202609281543 (ATOM 0.1.7.dev43+gaa0c5c3a1)
  -> nightly_202610070333 (sha256:2f605a3e8b459a0385f784edb0335c11892277fee650fe42044e4af39e36dd01;
  ATOM 0.1.7.dev60+g7fc0926c6, ROCm 7.2.4). ROCm/ATOM#2479 moves M3 mono
  decode onto the shared mono framework (mailbox flags published only after
  their stores drain; exact top-k sparse selection); ROCm/ATOM#2483 rebuilds
  the per-step mailbox clear and cross-rank fence as one FlyDSL launch,
  replacing a Triton fence that could spin forever on gfx950; ROCm/ATOM#2422
  fixes the mono long-context decision. Upstream
  recipes/MiniMax-M3-Agentic-InferenceX.md is unchanged.
- Mono decode on at every TP4 point: drop ATOM_MONO_ENABLE=0 from the
  indexer-CP points C15/C20/C24/C28. Mono serves decode steps of up to 16
  tokens (four requests with EAGLE3 K3), covering all of TP4 C1/C2/C4.
  ATOM builds mono for TP4 only, so TP2 runs the unfused path.
- Add TP2 C4 (max-num-seqs 8); GPU-resident TP2 points are C1/C2/C4.
- Record LMCache 0.5.6.dev139+gf22dec28 (the wheel in the new image) for
  both DRAM-offload arms; offload settings unchanged.
- Recipe cites NVIDIA/srt-slurm#507 (in srt-slurm since v2.36.0) for
  extra-kv-connectors instead of the dropped 507 patch.
- No data-type or precision change to the EAGLE3 draft
  (Inferact/MiniMax-M3-EAGLE3-GQA).

Risk: ROCm/ATOM#2452 (CUDA-graph pad rows in the wide MoE path, 5+ tokens)
is not in this image; every batch size 1-18 is captured, so steps of up to
four requests are never padded.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
seungrokj added a commit to SemiAnalysisAI/InferenceX that referenced this pull request Oct 8, 2026
… every TP4 point

minimaxm3-fp4-mi355x-atom-agentic-mtp:

- Image rocm/atom-dev:nightly_202609281543 (ATOM 0.1.7.dev43+gaa0c5c3a1)
  -> nightly_202610070333 (sha256:2f605a3e8b459a0385f784edb0335c11892277fee650fe42044e4af39e36dd01;
  ATOM 0.1.7.dev60+g7fc0926c6, ROCm 7.2.4). ROCm/ATOM#2479 moves M3 mono
  decode onto the shared mono framework (mailbox flags published only after
  their stores drain; exact top-k sparse selection); ROCm/ATOM#2483 rebuilds
  the per-step mailbox clear and cross-rank fence as one FlyDSL launch,
  replacing a Triton fence that could spin forever on gfx950; ROCm/ATOM#2422
  fixes the mono long-context decision. Upstream
  recipes/MiniMax-M3-Agentic-InferenceX.md is unchanged.
- Mono decode on at every TP4 point: drop ATOM_MONO_ENABLE=0 from the
  indexer-CP points C15/C20/C24/C28. Mono serves decode steps of up to 16
  tokens (four requests with EAGLE3 K3), covering all of TP4 C1/C2/C4.
  ATOM builds mono for TP4 only, so TP2 runs the unfused path.
- Add TP2 C4 (max-num-seqs 8); GPU-resident TP2 points are C1/C2/C4.
- Record LMCache 0.5.6.dev139+gf22dec28 (the wheel in the new image) for
  both DRAM-offload arms; offload settings unchanged.
- Recipe cites NVIDIA/srt-slurm#507 (in srt-slurm since v2.36.0) for
  extra-kv-connectors instead of the dropped 507 patch.
- No data-type or precision change to the EAGLE3 draft
  (Inferact/MiniMax-M3-EAGLE3-GQA).

Risk: ROCm/ATOM#2452 (CUDA-graph pad rows in the wide MoE path, 5+ tokens)
is not in this image; every batch size 1-18 is captured, so steps of up to
four requests are never padded.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
edwingao28 pushed a commit to SemiAnalysisAI/InferenceX that referenced this pull request Oct 8, 2026
… every TP4 point / MiniMax-M3 ATOM AgentX 更新至 1007 nightly 并在所有 TP4 点位启用 mono decode (#3750)

* perf(amd): MiniMax-M3 ATOM AgentX on nightly_202610061452 with mono decode at every TP4 point

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* docs: point MiniMax-M3 ATOM extra-kv-connectors at srt-slurm #507 instead of the dropped patch

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* chore: set MiniMax-M3 ATOM changelog pr-link to #3750

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* perf(amd): move MiniMax-M3 ATOM AgentX to nightly_202610070333 (ATOM #2483 mono step-fence fix)

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* perf(amd): MiniMax-M3 ATOM AgentX on 1007 nightly with mono decode at every TP4 point

minimaxm3-fp4-mi355x-atom-agentic-mtp:

- Image rocm/atom-dev:nightly_202609281543 (ATOM 0.1.7.dev43+gaa0c5c3a1)
  -> nightly_202610070333 (sha256:2f605a3e8b459a0385f784edb0335c11892277fee650fe42044e4af39e36dd01;
  ATOM 0.1.7.dev60+g7fc0926c6, ROCm 7.2.4). ROCm/ATOM#2479 moves M3 mono
  decode onto the shared mono framework (mailbox flags published only after
  their stores drain; exact top-k sparse selection); ROCm/ATOM#2483 rebuilds
  the per-step mailbox clear and cross-rank fence as one FlyDSL launch,
  replacing a Triton fence that could spin forever on gfx950; ROCm/ATOM#2422
  fixes the mono long-context decision. Upstream
  recipes/MiniMax-M3-Agentic-InferenceX.md is unchanged.
- Mono decode on at every TP4 point: drop ATOM_MONO_ENABLE=0 from the
  indexer-CP points C15/C20/C24/C28. Mono serves decode steps of up to 16
  tokens (four requests with EAGLE3 K3), covering all of TP4 C1/C2/C4.
  ATOM builds mono for TP4 only, so TP2 runs the unfused path.
- Add TP2 C4 (max-num-seqs 8); GPU-resident TP2 points are C1/C2/C4.
- Record LMCache 0.5.6.dev139+gf22dec28 (the wheel in the new image) for
  both DRAM-offload arms; offload settings unchanged.
- Recipe cites NVIDIA/srt-slurm#507 (in srt-slurm since v2.36.0) for
  extra-kv-connectors instead of the dropped 507 patch.
- No data-type or precision change to the EAGLE3 draft
  (Inferact/MiniMax-M3-EAGLE3-GQA).

Risk: ROCm/ATOM#2452 (CUDA-graph pad rows in the wide MoE path, 5+ tokens)
is not in this image; every batch size 1-18 is captured, so steps of up to
four requests are never padded.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: seungrokj <seungrok.jung@amd.com>
amd-pedghazi added a commit to amd-pedghazi/sglang that referenced this pull request Oct 10, 2026
With SGLANG_ROCM_MONO_DECODE=1, decode steps of 1-48 rows run each layer's attention all-reduce, FFN mHC seam, MoE and MoE all-reduce as one persistent FlyDSL launch. The kernels in dsv41_mono/ are copied from vllm-project/vllm#60397 (from ROCm/ATOM#2479, MIT) and keep their licence notices.
amd-pedghazi added a commit to amd-pedghazi/sglang that referenced this pull request Oct 10, 2026
With SGLANG_ROCM_MONO_DECODE=1, decode steps of 1-48 rows run each layer's attention all-reduce, FFN mHC seam, MoE and MoE all-reduce as one persistent FlyDSL launch. The kernels in dsv41_mono/ are copied from vllm-project/vllm#60397 (from ROCm/ATOM#2479, MIT) and keep their licence notices.
ZhangLirong-amd added a commit that referenced this pull request Oct 14, 2026
…-keyed piece

#2479's gfx950 row-group scorer plans its launch from the sequence count
and the query span. #2543 kept it in the num_tokens-keyed dense piece and
made the launch layout-invariant by publishing a fixed capacity: as many
sequences as the bucket has rows and max_qlen = 1 + mtp_k. Under MTP the
extra sequences are all empty yet still planned and scheduled, so the
scorer slowed 13-48% (bs 32-256, q=4, ctx 4096; same top-k). It also
applied in FULL, whose graph is keyed on (running_bs, max_q_len) and never
needed it, and raised under DP when running_bs outran the rows (#2556).

Run the scorer where its launch shape is part of the graph key instead:
- narrow PIECEWISE scores in _attn_compress, after the compressor join (it
  reads the index rows the compressor writes). That core's graph keys on
  (layer, num_tokens, bucket_bs, q_eff); plain PIECEWISE runs it eager. The
  result goes to a fixed per-microbatch buffer (fp4_decode_topk) that
  _sparse_attention reads in the dense piece.
- FULL scores inline as before, in a graph keyed on the same two values.
The scorer then takes the step's own cu_seqlens_q and max_seqlen_q, as
before #2543, and the fixed-capacity metadata is removed.

Tests: one captured scorer per (running_bs, max_q) replays rectangular,
ragged, padded and empty layouts against the oracle; a DP-padded batch
wider than its rows scores correctly.
valarLip pushed a commit that referenced this pull request Oct 14, 2026
…-keyed piece (#2572)

* perf(v4): score FP4 decode top-k in the batch-keyed core, not a token-keyed piece

#2479's gfx950 row-group scorer plans its launch from the sequence count
and the query span. #2543 kept it in the num_tokens-keyed dense piece and
made the launch layout-invariant by publishing a fixed capacity: as many
sequences as the bucket has rows and max_qlen = 1 + mtp_k. Under MTP the
extra sequences are all empty yet still planned and scheduled, so the
scorer slowed 13-48% (bs 32-256, q=4, ctx 4096; same top-k). It also
applied in FULL, whose graph is keyed on (running_bs, max_q_len) and never
needed it, and raised under DP when running_bs outran the rows (#2556).

Run the scorer where its launch shape is part of the graph key instead:
- narrow PIECEWISE scores in _attn_compress, after the compressor join (it
  reads the index rows the compressor writes). That core's graph keys on
  (layer, num_tokens, bucket_bs, q_eff); plain PIECEWISE runs it eager. The
  result goes to a fixed per-microbatch buffer (fp4_decode_topk) that
  _sparse_attention reads in the dense piece.
- FULL scores inline as before, in a graph keyed on the same two values.
The scorer then takes the step's own cu_seqlens_q and max_seqlen_q, as
before #2543, and the fixed-capacity metadata is removed.

Tests: one captured scorer per (running_bs, max_q) replays rectangular,
ragged, padded and empty layouts against the oracle; a DP-padded batch
wider than its rows scores correctly.

* perf(v4): score FP4 decode top-k straight into the fixed buffer

_attn_compress scored into a fresh tensor and copied it into
fp4_decode_topk. The core runs in a split op (eager or its own graph),
so call the scorer directly with the fixed buffer as its top-k output and
drop the copy. The graph test now captures the scorer writing into a
fixed buffer.

---------

Co-authored-by: ZhangLirong <219599949+ZhangLirong-amd@users.noreply.github.com>
zhangxinyuanliuhengyu added a commit that referenced this pull request Oct 23, 2026
…ugin metadata

SGLang calls the one-arg _config_refusal from #2479, and it builds the generic AttentionMetaData, which did not have the FP4 decode top-k buffer #2572 reads.

Co-authored-by: Cursor <cursoragent@cursor.com>
valarLip pushed a commit that referenced this pull request Oct 26, 2026
…aces (#2705)

`install_minimax_mono_plugin_refusal` rebinds
`minimax_m3.mono.dispatch._config_refusal` so SGLang refuses mono before the
TR_HYBRID cudagraph clause can read a `cudagraph_mode` it never sets.

#2479 folded the `use_index_cache` clause into `common_refusals` and dropped
`text_config` from the native signature and its one call site, but left the
stand-in taking `(atom_config, text_config)`. `MonoDecode.__init__` passes
one argument, so every SGLang MiniMax-M3 launch raised

    TypeError: install_minimax_mono_plugin_refusal.<locals>._config_refusal()
               missing 1 required positional argument: 'text_config'

at model construction, failing the `ci:sglang` accuracy job for any PR that
triggered it.

A rebind is invisible to ruff and black -- they see two unrelated functions --
and the only code path that exercises the pairing is an SGLang MiniMax-M3
launch, which runs in the heavy job rather than pre-checkin. So the pairing is
pinned by a test that reads both definitions with `ast`: no import, hence no
`importorskip`, hence it actually runs on the CPU-only runner where the drift
would otherwise go unnoticed again. A second case pins the native signature
against its call site, so a future change has to move all three together.

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
valarLip pushed a commit that referenced this pull request Oct 27, 2026
* fix(sglang): avoid HIP segfault in xgrammar json_object bitmask

Default xgrammar on ROCm applies the vocab mask through sgl_kernel's CUDA bitmask kernel, which kills the server for response_format json_object. Route that HIP binding to a Torch GPU masked_fill.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(sglang): use modern typing for xgrammar HIP bitmask patch

Satisfy Pre Checkin Ruff (UP035/UP045/UP007/UP006) on the new plugin file.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(sglang): address xgrammar HIP bitmask review feedback

Mask logits beyond bitmask coverage, rename the disable env to TORCH (keep TRITON alias), and move the unit test under tests/ so Pre Checkin runs it.

Co-authored-by: Cursor <cursoragent@cursor.com>

* test(sglang): keep xgrammar HIP bitmask tests under tests/plugin

Plugin tests stay in tests/plugin even though Pre Checkin ignores that tree.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(sglang): use the xgrammar Torch bitmask only when sgl_kernel lacks the device arch

The release image's sgl_kernel is gfx950-only, so MI308X SIGSEGVs. Keep the native kernel when common_ops.so already contains this GPU, and register the disable flag in envs.py.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(sglang): unblock MiniMax refusal and V4 fp4_decode_topk on the plugin metadata

SGLang calls the one-arg _config_refusal from #2479, and it builds the generic AttentionMetaData, which did not have the FP4 decode top-k buffer #2572 reads.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(sglang): set fp4_decode_topk on the V4 plugin metadata

Keep atom/utils/forward_context.py unchanged. The SGLang bridge publishes generic AttentionMetaData, so set the buffer there; None still uses indexer.topk.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(sglang): publish V4 decode split plans on plugin metadata

Generic AttentionMetaData has no split_plan_swa/csa/hca, so fp8 decode
raises AttributeError. Fill them in the bridge the way the native
builder does.

Co-authored-by: Cursor <cursoragent@cursor.com>

* revert(sglang): drop the CI accuracy workarounds from the bitmask fix

MiniMax refusal and DeepSeek-V4 metadata patches were only there to
turn red accuracy jobs green. Keep the xgrammar HIP bitmask stopgap.

Co-authored-by: Cursor <cursoragent@cursor.com>

---------

Co-authored-by: Cursor <cursoragent@cursor.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant