Repository navigation
feat(v41): mono decode, FP4 index plane, shared mono framework - #2479
Merged
Merged
Conversation
- atom/mono: shared mono decode framework (runtime: TP consensus, peer memory and step fence, step mailboxes, lifecycle, widths, routing, debug reports, deployment refusals, check compare; plan: execution model, build keys, TP shard, mailbox layout, arithmetic, hand-off check; device: tagged mailbox publish, peer all-reduce, FP8/FP4 MX, wave/math primitives) - MiniMax-M3 mono moved onto the shared framework (peer_buffer and timeline removed; contract, sources, dense_post, stages added) - DeepSeek-V4.1 mono decode (atom/models/deepseek_v41/mono): fused per-layer K1/K2a/K2b kernels, DSpark draft runner, installed after load through model_runner.mono_decode_installers - V4.1 FP4 index plane (--index_cache_dtype fp4), original path and mono: page-8 values/scales planes, ATOM FP4 index writer, fused q RoPE + FP4 quant, FlyDSL row-group scorer; mono K1 FP4 quant and K2a FP4 score, whose task plan is laid out on the host by a step planner (DeepseekV41MetadataBuilder.add_step_planner) and published with the step - FP4 index decode scoring by request (aiter row-group ragged rows via flydsl_pa_mqa_logits_fp4: a request's rows share each key load, lengths from the per-row bounds, shared out in the kernel): DeepSeek-V4 decode drops its per-step cta_info schedule; V4.1 FULL layers score off the request PAGE table (DecodeRequests), with no per-token tile table - V4.1 MoE A8W4, mHC / Engram / DSpark / rejection sampler fixes, docs - MiniMax-M3 mono selection: exact top-k by counting 64-bit keys with a group-floor prefilter (atom/mono/device/select), shared by the short and long paths; long path from 512 blocks; the selector's LDS in the activation pool. S=1 20k 38.0 -> 35.5 us a layer, 1M 62.3 -> ~58, S=16 133 -> 125; selections and layer outputs byte-identical - V4.1 lds_topk fixed-cost cuts (quad loads, pass-0 run merging, no passes when n <= k, keys in registers) and stage_slice loads batched ahead of their rounds (mono/device/ops.batched_rounds); M3's global -> LDS copies through ops.rows_to_lds; CM_NT on weights read once a step - kernel resources (VGPR / SGPR / spills / scratch / LDS) from code-object metadata: runtime.compile.kernel_resources / launcher_resources - default CUDAGraph capture ladder covers every batch of 1..8 (a padded batch ran the next size's step: V4.1 bs5 -14 %, bs6 -7.5 %; the graph pool is unchanged); one default for the CLI and EngineArgs - cleanups: shared DIAG_WORDS in mono plan layout, dead seed comments
Contributor
🏷️ CI GuideRuns automatically on every eligible PR before approval:
Heavy model tests:
|
…lect without aiter lifecycle.py imports peer_memory, which needs aiter: test_mono_runtime.py importing bind_agreed from it failed collection on the CPU CI runner. bind_agreed is the TP agreement on a runner's bind, next to tp_agree; lifecycle keeps the owned peer buffer.
The routing wrapper (MonoDecodeModel) sits outside the compiled model, so a step's mono / original decision stays in Python at level 3 as at level 0.
2 tasks done
valarLip
pushed a commit
that referenced
this pull request
Oct 7, 2026
…2488) The DeepSeek-V4 decode prep padded `cu_seqlens_q` to `running_bs` but published `block_tables` with only `scheduled_bs` rows. Since #2479 the FP4 indexer's ragged scorer (`flydsl_pa_mqa_logits_fp4` with `query_start_loc`) takes its batch from `cu_seqlens_q` and asserts `block_tables.shape[0] >= batch`. So an eager DP-padded decode step, one whose rank holds fewer requests than the padded batch, died with `AssertionError` in `pa_mqa_logits_fp4_rowgroup`. Captured graphs replay against the full buffer and never hit it. Pad the table to `running_bs` (zeroed rows, as the TBO path already does) and publish that many rows. Pad rows carry row end 0, so no page is read through them. Signed-off-by: yihonglie <hyi@amd.com> Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
seungrokj
added a commit
to SemiAnalysisAI/InferenceX
that referenced
this pull request
Oct 8, 2026
… every TP4 point minimaxm3-fp4-mi355x-atom-agentic-mtp: - Image rocm/atom-dev:nightly_202609281543 (ATOM 0.1.7.dev43+gaa0c5c3a1) -> nightly_202610070333 (sha256:2f605a3e8b459a0385f784edb0335c11892277fee650fe42044e4af39e36dd01; ATOM 0.1.7.dev60+g7fc0926c6, ROCm 7.2.4). ROCm/ATOM#2479 moves M3 mono decode onto the shared mono framework (mailbox flags published only after their stores drain; exact top-k sparse selection); ROCm/ATOM#2483 rebuilds the per-step mailbox clear and cross-rank fence as one FlyDSL launch, replacing a Triton fence that could spin forever on gfx950; ROCm/ATOM#2422 fixes the mono long-context decision. Upstream recipes/MiniMax-M3-Agentic-InferenceX.md is unchanged. - Mono decode on at every TP4 point: drop ATOM_MONO_ENABLE=0 from the indexer-CP points C15/C20/C24/C28. Mono serves decode steps of up to 16 tokens (four requests with EAGLE3 K3), covering all of TP4 C1/C2/C4. ATOM builds mono for TP4 only, so TP2 runs the unfused path. - Add TP2 C4 (max-num-seqs 8); GPU-resident TP2 points are C1/C2/C4. - Record LMCache 0.5.6.dev139+gf22dec28 (the wheel in the new image) for both DRAM-offload arms; offload settings unchanged. - Recipe cites NVIDIA/srt-slurm#507 (in srt-slurm since v2.36.0) for extra-kv-connectors instead of the dropped 507 patch. - No data-type or precision change to the EAGLE3 draft (Inferact/MiniMax-M3-EAGLE3-GQA). Risk: ROCm/ATOM#2452 (CUDA-graph pad rows in the wide MoE path, 5+ tokens) is not in this image; every batch size 1-18 is captured, so steps of up to four requests are never padded. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Oct 8, 2026
seungrokj
added a commit
to SemiAnalysisAI/InferenceX
that referenced
this pull request
Oct 8, 2026
…ghtly dsv41flash-fp4-mi355x-atom-agentic-dspark: - Restore DeepSeek-V4.1-Flash FP4 ATOM AgentX on MI355X (#3387) on the native srt-slurm recipe srt-slurm-recipes/dsv41flash/atom/mi355x-fp4-mtp/ agentic.yaml (legacy script and master entry were removed in #3461/#3463). TP2 conc [1,2,4,5,6,8,16,32,64,128] (adds 4,5,6,128); TP4 conc [1,4,5,6,8,32] (adds 1,4,5,6; drops 2,16,64). No EP or KV offload. Conc 1-8 covers the batch sizes V4.1 mono decode serves (<= 8 requests). - Image rocm/atom-dev:nightly_202609250902 -> nightly_202610070333 (sha256:2f605a3e8b459a0385f784edb0335c11892277fee650fe42044e4af39e36dd01; ATOM 0.1.7.dev60+g7fc0926c6, ROCm 7.2.4): - ROCm/ATOM#2419: group32 FP8 weight preshuffle and BF16 RoPE tables for V4.1. - ROCm/ATOM#2479: V4.1 mono decode (on by default, <= 8 requests a step for this TP2/TP4, BF16-KV, five-token DSpark deployment) and the FP4 index plane used by upstream recipes/DeepSeek-V4.1-Flash-Agentic.md. - ROCm/aiter#6174: gfx950 tuned rows for six V4.1-Flash TP4 bf16 GEMM shapes that fell back to torch (DSpark verify batches, TP4 lm_head). - ROCm/ATOM#2483: replaces the Triton mono step fence that could spin forever on gfx950; on the 1006 nightlies the TP2 c1 canary hung in warmup on the DSpark draft's first mono step (runs 37486079757, 37493871230). - Server settings follow the upstream recipe (BF16 KV, index cache FP8 -> FP4 per upstream at 7fc0926c, max-num-seqs 128, 16K batching/prefill chunks, prefix caching block 16, 8K state checkpoints, level 3, FULL graphs, dsml_v41 tool parser); every point now captures [1..32, 48, 64, 128]. Five-token DSpark uses golden AL 3.51 (thinking_on) for throughput, real acceptance for eval; 3600 s weka_062126 replay with five warmup requests per lane. - DSpark draft (layers 37-39, inline): no --online_quant_config; attention and projection linears stay native FP8 32x32. Via the image bump, the draft's routed FP4 experts take MXFP8 instead of MXFP4 activations (ROCm/ATOM#2479) and the V4.1 RoPE tables are BF16 instead of FP32 (ROCm/ATOM#2419). The draft has no indexer, so index-cache-dtype fp4 does not apply to it. - TP4 conc 32: ATOM_PREFILL_DECODE_INTERVAL=16 (local 3600 s replay: 108.2 -> 150.9 tok/s/user, ITL p90 9.24 -> 6.63 ms, TTFT p50 0.51 -> 1.14 s, throughput unchanged). - TP2 conc 64: ATOM_PREFILL_DECODE_INTERVAL=4 (30.3 -> 38.8 tok/s/user, 105.4k -> 115.2k tok/s/GPU, 6487 -> 6941 requests, TTFT p90 2.3 -> 6.4 s). All other points unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Jasen2201
added a commit
that referenced
this pull request
Oct 8, 2026
Resolve the backends.py conflict: main (#2479) removed the FP4 decode schedule, so `_publish_indexer_fp4_decode_schedule` and its EagleProposer call sites go. Keep this branch's `_publish_dcp_token_block_tables` no-op on CommonAttentionBuilder, with a docstring that no longer points at the removed method. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
seungrokj
added a commit
to SemiAnalysisAI/InferenceX
that referenced
this pull request
Oct 8, 2026
… every TP4 point minimaxm3-fp4-mi355x-atom-agentic-mtp: - Image rocm/atom-dev:nightly_202609281543 (ATOM 0.1.7.dev43+gaa0c5c3a1) -> nightly_202610070333 (sha256:2f605a3e8b459a0385f784edb0335c11892277fee650fe42044e4af39e36dd01; ATOM 0.1.7.dev60+g7fc0926c6, ROCm 7.2.4). ROCm/ATOM#2479 moves M3 mono decode onto the shared mono framework (mailbox flags published only after their stores drain; exact top-k sparse selection); ROCm/ATOM#2483 rebuilds the per-step mailbox clear and cross-rank fence as one FlyDSL launch, replacing a Triton fence that could spin forever on gfx950; ROCm/ATOM#2422 fixes the mono long-context decision. Upstream recipes/MiniMax-M3-Agentic-InferenceX.md is unchanged. - Mono decode on at every TP4 point: drop ATOM_MONO_ENABLE=0 from the indexer-CP points C15/C20/C24/C28. Mono serves decode steps of up to 16 tokens (four requests with EAGLE3 K3), covering all of TP4 C1/C2/C4. ATOM builds mono for TP4 only, so TP2 runs the unfused path. - Add TP2 C4 (max-num-seqs 8); GPU-resident TP2 points are C1/C2/C4. - Record LMCache 0.5.6.dev139+gf22dec28 (the wheel in the new image) for both DRAM-offload arms; offload settings unchanged. - Recipe cites NVIDIA/srt-slurm#507 (in srt-slurm since v2.36.0) for extra-kv-connectors instead of the dropped 507 patch. - No data-type or precision change to the EAGLE3 draft (Inferact/MiniMax-M3-EAGLE3-GQA). Risk: ROCm/ATOM#2452 (CUDA-graph pad rows in the wide MoE path, 5+ tokens) is not in this image; every batch size 1-18 is captured, so steps of up to four requests are never padded. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
seungrokj
added a commit
to SemiAnalysisAI/InferenceX
that referenced
this pull request
Oct 8, 2026
… every TP4 point minimaxm3-fp4-mi355x-atom-agentic-mtp: - Image rocm/atom-dev:nightly_202609281543 (ATOM 0.1.7.dev43+gaa0c5c3a1) -> nightly_202610070333 (sha256:2f605a3e8b459a0385f784edb0335c11892277fee650fe42044e4af39e36dd01; ATOM 0.1.7.dev60+g7fc0926c6, ROCm 7.2.4). ROCm/ATOM#2479 moves M3 mono decode onto the shared mono framework (mailbox flags published only after their stores drain; exact top-k sparse selection); ROCm/ATOM#2483 rebuilds the per-step mailbox clear and cross-rank fence as one FlyDSL launch, replacing a Triton fence that could spin forever on gfx950; ROCm/ATOM#2422 fixes the mono long-context decision. Upstream recipes/MiniMax-M3-Agentic-InferenceX.md is unchanged. - Mono decode on at every TP4 point: drop ATOM_MONO_ENABLE=0 from the indexer-CP points C15/C20/C24/C28. Mono serves decode steps of up to 16 tokens (four requests with EAGLE3 K3), covering all of TP4 C1/C2/C4. ATOM builds mono for TP4 only, so TP2 runs the unfused path. - Add TP2 C4 (max-num-seqs 8); GPU-resident TP2 points are C1/C2/C4. - Record LMCache 0.5.6.dev139+gf22dec28 (the wheel in the new image) for both DRAM-offload arms; offload settings unchanged. - Recipe cites NVIDIA/srt-slurm#507 (in srt-slurm since v2.36.0) for extra-kv-connectors instead of the dropped 507 patch. - No data-type or precision change to the EAGLE3 draft (Inferact/MiniMax-M3-EAGLE3-GQA). Risk: ROCm/ATOM#2452 (CUDA-graph pad rows in the wide MoE path, 5+ tokens) is not in this image; every batch size 1-18 is captured, so steps of up to four requests are never padded. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
edwingao28
pushed a commit
to SemiAnalysisAI/InferenceX
that referenced
this pull request
Oct 8, 2026
… every TP4 point / MiniMax-M3 ATOM AgentX 更新至 1007 nightly 并在所有 TP4 点位启用 mono decode (#3750) * perf(amd): MiniMax-M3 ATOM AgentX on nightly_202610061452 with mono decode at every TP4 point Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * docs: point MiniMax-M3 ATOM extra-kv-connectors at srt-slurm #507 instead of the dropped patch Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * chore: set MiniMax-M3 ATOM changelog pr-link to #3750 Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * perf(amd): move MiniMax-M3 ATOM AgentX to nightly_202610070333 (ATOM #2483 mono step-fence fix) Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * perf(amd): MiniMax-M3 ATOM AgentX on 1007 nightly with mono decode at every TP4 point minimaxm3-fp4-mi355x-atom-agentic-mtp: - Image rocm/atom-dev:nightly_202609281543 (ATOM 0.1.7.dev43+gaa0c5c3a1) -> nightly_202610070333 (sha256:2f605a3e8b459a0385f784edb0335c11892277fee650fe42044e4af39e36dd01; ATOM 0.1.7.dev60+g7fc0926c6, ROCm 7.2.4). ROCm/ATOM#2479 moves M3 mono decode onto the shared mono framework (mailbox flags published only after their stores drain; exact top-k sparse selection); ROCm/ATOM#2483 rebuilds the per-step mailbox clear and cross-rank fence as one FlyDSL launch, replacing a Triton fence that could spin forever on gfx950; ROCm/ATOM#2422 fixes the mono long-context decision. Upstream recipes/MiniMax-M3-Agentic-InferenceX.md is unchanged. - Mono decode on at every TP4 point: drop ATOM_MONO_ENABLE=0 from the indexer-CP points C15/C20/C24/C28. Mono serves decode steps of up to 16 tokens (four requests with EAGLE3 K3), covering all of TP4 C1/C2/C4. ATOM builds mono for TP4 only, so TP2 runs the unfused path. - Add TP2 C4 (max-num-seqs 8); GPU-resident TP2 points are C1/C2/C4. - Record LMCache 0.5.6.dev139+gf22dec28 (the wheel in the new image) for both DRAM-offload arms; offload settings unchanged. - Recipe cites NVIDIA/srt-slurm#507 (in srt-slurm since v2.36.0) for extra-kv-connectors instead of the dropped 507 patch. - No data-type or precision change to the EAGLE3 draft (Inferact/MiniMax-M3-EAGLE3-GQA). Risk: ROCm/ATOM#2452 (CUDA-graph pad rows in the wide MoE path, 5+ tokens) is not in this image; every batch size 1-18 is captured, so steps of up to four requests are never padded. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Co-authored-by: seungrokj <seungrok.jung@amd.com>
4 of 5 tasks
amd-pedghazi
added a commit
to amd-pedghazi/sglang
that referenced
this pull request
Oct 10, 2026
With SGLANG_ROCM_MONO_DECODE=1, decode steps of 1-48 rows run each layer's attention all-reduce, FFN mHC seam, MoE and MoE all-reduce as one persistent FlyDSL launch. The kernels in dsv41_mono/ are copied from vllm-project/vllm#60397 (from ROCm/ATOM#2479, MIT) and keep their licence notices.
amd-pedghazi
added a commit
to amd-pedghazi/sglang
that referenced
this pull request
Oct 10, 2026
With SGLANG_ROCM_MONO_DECODE=1, decode steps of 1-48 rows run each layer's attention all-reduce, FFN mHC seam, MoE and MoE all-reduce as one persistent FlyDSL launch. The kernels in dsv41_mono/ are copied from vllm-project/vllm#60397 (from ROCm/ATOM#2479, MIT) and keep their licence notices.
5 tasks
ZhangLirong-amd
added a commit
that referenced
this pull request
Oct 14, 2026
…-keyed piece #2479's gfx950 row-group scorer plans its launch from the sequence count and the query span. #2543 kept it in the num_tokens-keyed dense piece and made the launch layout-invariant by publishing a fixed capacity: as many sequences as the bucket has rows and max_qlen = 1 + mtp_k. Under MTP the extra sequences are all empty yet still planned and scheduled, so the scorer slowed 13-48% (bs 32-256, q=4, ctx 4096; same top-k). It also applied in FULL, whose graph is keyed on (running_bs, max_q_len) and never needed it, and raised under DP when running_bs outran the rows (#2556). Run the scorer where its launch shape is part of the graph key instead: - narrow PIECEWISE scores in _attn_compress, after the compressor join (it reads the index rows the compressor writes). That core's graph keys on (layer, num_tokens, bucket_bs, q_eff); plain PIECEWISE runs it eager. The result goes to a fixed per-microbatch buffer (fp4_decode_topk) that _sparse_attention reads in the dense piece. - FULL scores inline as before, in a graph keyed on the same two values. The scorer then takes the step's own cu_seqlens_q and max_seqlen_q, as before #2543, and the fixed-capacity metadata is removed. Tests: one captured scorer per (running_bs, max_q) replays rectangular, ragged, padded and empty layouts against the oracle; a DP-padded batch wider than its rows scores correctly.
valarLip
pushed a commit
that referenced
this pull request
Oct 14, 2026
…-keyed piece (#2572) * perf(v4): score FP4 decode top-k in the batch-keyed core, not a token-keyed piece #2479's gfx950 row-group scorer plans its launch from the sequence count and the query span. #2543 kept it in the num_tokens-keyed dense piece and made the launch layout-invariant by publishing a fixed capacity: as many sequences as the bucket has rows and max_qlen = 1 + mtp_k. Under MTP the extra sequences are all empty yet still planned and scheduled, so the scorer slowed 13-48% (bs 32-256, q=4, ctx 4096; same top-k). It also applied in FULL, whose graph is keyed on (running_bs, max_q_len) and never needed it, and raised under DP when running_bs outran the rows (#2556). Run the scorer where its launch shape is part of the graph key instead: - narrow PIECEWISE scores in _attn_compress, after the compressor join (it reads the index rows the compressor writes). That core's graph keys on (layer, num_tokens, bucket_bs, q_eff); plain PIECEWISE runs it eager. The result goes to a fixed per-microbatch buffer (fp4_decode_topk) that _sparse_attention reads in the dense piece. - FULL scores inline as before, in a graph keyed on the same two values. The scorer then takes the step's own cu_seqlens_q and max_seqlen_q, as before #2543, and the fixed-capacity metadata is removed. Tests: one captured scorer per (running_bs, max_q) replays rectangular, ragged, padded and empty layouts against the oracle; a DP-padded batch wider than its rows scores correctly. * perf(v4): score FP4 decode top-k straight into the fixed buffer _attn_compress scored into a fresh tensor and copied it into fp4_decode_topk. The core runs in a split op (eager or its own graph), so call the scorer directly with the fixed buffer as its top-k output and drop the copy. The graph test now captures the scorer writing into a fixed buffer. --------- Co-authored-by: ZhangLirong <219599949+ZhangLirong-amd@users.noreply.github.com>
2 of 3 tasks
1 task done
valarLip
pushed a commit
that referenced
this pull request
Oct 26, 2026
…aces (#2705) `install_minimax_mono_plugin_refusal` rebinds `minimax_m3.mono.dispatch._config_refusal` so SGLang refuses mono before the TR_HYBRID cudagraph clause can read a `cudagraph_mode` it never sets. #2479 folded the `use_index_cache` clause into `common_refusals` and dropped `text_config` from the native signature and its one call site, but left the stand-in taking `(atom_config, text_config)`. `MonoDecode.__init__` passes one argument, so every SGLang MiniMax-M3 launch raised TypeError: install_minimax_mono_plugin_refusal.<locals>._config_refusal() missing 1 required positional argument: 'text_config' at model construction, failing the `ci:sglang` accuracy job for any PR that triggered it. A rebind is invisible to ruff and black -- they see two unrelated functions -- and the only code path that exercises the pairing is an SGLang MiniMax-M3 launch, which runs in the heavy job rather than pre-checkin. So the pairing is pinned by a test that reads both definitions with `ast`: no import, hence no `importorskip`, hence it actually runs on the CPU-only runner where the drift would otherwise go unnoticed again. A second case pins the native signature against its call site, so a future change has to move all three together. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
1 task
valarLip
pushed a commit
that referenced
this pull request
Oct 27, 2026
* fix(sglang): avoid HIP segfault in xgrammar json_object bitmask Default xgrammar on ROCm applies the vocab mask through sgl_kernel's CUDA bitmask kernel, which kills the server for response_format json_object. Route that HIP binding to a Torch GPU masked_fill. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(sglang): use modern typing for xgrammar HIP bitmask patch Satisfy Pre Checkin Ruff (UP035/UP045/UP007/UP006) on the new plugin file. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(sglang): address xgrammar HIP bitmask review feedback Mask logits beyond bitmask coverage, rename the disable env to TORCH (keep TRITON alias), and move the unit test under tests/ so Pre Checkin runs it. Co-authored-by: Cursor <cursoragent@cursor.com> * test(sglang): keep xgrammar HIP bitmask tests under tests/plugin Plugin tests stay in tests/plugin even though Pre Checkin ignores that tree. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(sglang): use the xgrammar Torch bitmask only when sgl_kernel lacks the device arch The release image's sgl_kernel is gfx950-only, so MI308X SIGSEGVs. Keep the native kernel when common_ops.so already contains this GPU, and register the disable flag in envs.py. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(sglang): unblock MiniMax refusal and V4 fp4_decode_topk on the plugin metadata SGLang calls the one-arg _config_refusal from #2479, and it builds the generic AttentionMetaData, which did not have the FP4 decode top-k buffer #2572 reads. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(sglang): set fp4_decode_topk on the V4 plugin metadata Keep atom/utils/forward_context.py unchanged. The SGLang bridge publishes generic AttentionMetaData, so set the buffer there; None still uses indexer.topk. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(sglang): publish V4 decode split plans on plugin metadata Generic AttentionMetaData has no split_plan_swa/csa/hca, so fp8 decode raises AttributeError. Fill them in the bridge the way the native builder does. Co-authored-by: Cursor <cursoragent@cursor.com> * revert(sglang): drop the CI accuracy workarounds from the bitmask fix MiniMax refusal and DeepSeek-V4 metadata patches were only there to turn red accuracy jobs green. Keep the xgrammar HIP bitmask stopgap. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com>
Merged
1 task done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
DeepSeek-V4.1 mono decode, the V4.1 FP4 index plane, and the shared mono framework that MiniMax-M3 and V4.1 now both build on.
atom/mono: shared mono decode framework.device/select), and batched strided loads (batched_rounds,rows_to_lds).atom/models/deepseek_v41/mono):model_runner.mono_decode_installers.--index_cache_dtype fp4), on both the original path and mono:Performance
MiniMax-M3, one fused layer on 4 GPUs, µs per layer. Selections and layer outputs are byte-identical before and after.
V4.1, server step time with every batch size of 1..8 captured (padded → own graph). bs 1/2/4/8 are unchanged.
The graph pool stays at 1.70 GB, the same as with 10 graphs: the graphs share one pool sized by the largest.
V4.1 slice-stage load batching: step time −0.3 to −0.7% at conc 1/2/4/8, over two alternating server rounds.
Test plan
bash .github/scripts/run_unit_tests.sh: 10204 passed, 2 failed, 351 skipped. Both failures also fail onorigin/maind16c457 on this machine:test_compilation.py::test_runtime_guard_with_live_steps_and_aux_hooks[3-False]andtest_triton_inductor_pairing.py(torch 2.9.1 / triton 3.8.0 pairing)k2_bench.py CHECK=): S = 6 / 48, ALL EQUAL