Skip to content

fix(mono): zero the mailboxes and fence the ranks in one FlyDSL launch - #2483

Merged
valarLip merged 1 commit into
mainfrom
feat/mono-step-begin-flydsl
Oct 7, 2026
Merged

valarLip merged 1 commit into
mainfrom
feat/mono-step-begin-flydsl

Conversation

@valarLip

@valarLip valarLip commented Oct 7, 2026

Copy link
Copy Markdown
Collaborator

Why

CI hung on the DeepSeek-V4.1 DSpark accuracy job after --level 3 stopped turning mono off (#2479): the server never got past warmup. rocgdb on the hung workers showed two ranks inside the step fence (_step_fence_kernel, Triton) and the third already past it, waiting in the next all-reduce. Triton 3.7 on gfx950 compiled the fence's scalar poll (tl.atomic_add(own + p, 0) in an unbounded while) into two nested loops: lanes 1-63 spun in the inner one on a shared-memory copy, and lane 0, the only lane that reloads the slot, was masked off outside it (exec = 0x...fffe). A fence whose first poll found a slot short never left. Local Triton 3.8 builds a different loop, which is why it only reproduced in the CI image.

What

atom/mono/runtime/step_begin.py replaces torch._foreach_zero_ + the Triton fence with one FlyDSL launch:

  • every CTA zeroes its share of the peer data at system scope, drains its stores, and writes this step's epoch into its own flag word;
  • CTA 0's wave 0 waits for every flag (one vector load a lane), writes the epoch into its slot at every rank and polls its own slots until all ranks arrive. Both waits are wave-uniform FlyDSL loops;
  • meanwhile the other CTAs zero the scratch (this rank's alone, done before the next kernel starts on the same stream).

A flag a CTA instead of an atomic count: one address's atomics serialized and cost about 4 ns a CTA. The step's state (epoch, flags, a debug build's missing ranks) now belongs to StepMailboxes. PeerBuffer is the symmetric memory alone, and its debug argument moves to StepMailboxes. The debug build keeps its 10 s give-up and names the ranks that never arrived.

Test

4 x MI355X, TP4:

result
step begin, V4.1's buffers (5.7 MB scratch, 7.9 MB peer data), graph replay 8.50 -> 6.94 us a step
stress: one random rank delayed each step, every peer push checked after it 3000 steps, 0 lost
positive control: zeroing without the fence 435 lost pushes in 1000 steps
captured in a CUDA graph, replayed 1000 steps, 0 lost
debug build, rank 1 never begins the other ranks give up and report rank 1
DeepSeek-V4.1 GSM8K 3-shot, level 0, 8 concurrent (every decode step on mono) 0.9265 (same-code range 0.914-0.927)
MiniMax-M3 GSM8K chat 8-shot, TP4, 4 concurrent (every decode step on mono) 0.9378 (main: 0.937-0.942, #2452)
tests/test_mono_runtime.py, also without aiter / triton 60 passed

V4.1 ran at level 0 because on the development box the original path is already broken at level 3 (a torch / Triton pairing problem). Whether this clears the level-3 CI hang is for this PR's CI run to show.

A step began with torch._foreach_zero_ over the mailbox buffers, then the
peer fence, a Triton kernel. Triton 3.7 on gfx950 compiled the fence's
scalar poll into a loop whose one reloading lane was masked off: a fence
whose first poll found a slot short spun for ever. CI hung there on the
DeepSeek-V4.1 DSpark draft's first mono step.

step_begin does both in one FlyDSL launch. Every CTA zeroes its share of
the peer data at system scope and flags its epoch; CTA 0's wave 0 waits
for the flags, writes the epoch into every rank's slot and polls its own,
while the other CTAs zero the scratch. The waits are wave-uniform FlyDSL
loops. A CTA flags its own word instead of an atomic count on one
address, which serialized.

The step's state (epoch, flags, a debug build's missing ranks) is the
StepMailboxes', not the PeerBuffer's: PeerBuffer is the symmetric memory
alone, and the debug report reads mailboxes.missing_ranks().

4 x MI355X, TP4, V4.1's buffers (5.7 MB scratch, 7.9 MB peer data),
graph-replayed: 8.50 -> 6.94 us a step. A stress test that delays one
random rank a step and checks every peer push survives: 3000 steps clean,
graph replay 1000 clean; without the fence it reports wiped pushes (435
in 1000 steps). GSM8K with every decode step on mono: V4.1 0.9265 (level
0, 8 concurrent), MiniMax-M3 0.9378 (TP4, chat 8-shot, 4 concurrent).
@github-actions

github-actions Bot commented Oct 7, 2026

Copy link
Copy Markdown
Contributor

🏷️ CI Guide

Runs automatically on every eligible PR before approval:

  • ✅ Pre Checkin: Black, Ruff, catalog schema validation, non-GPU unit tests

Heavy model tests:

  • ✅ Run after the PR is approved and Pre Checkin passes
  • ✅ Run immediately when an approval review is submitted
  • ✅ Can be requested before approval with labels
Label Tests
ci:full Run all heavy PR model tests: native ATOM, vLLM, and SGLang
ci:atom Run native ATOM model accuracy tests
ci:vllm Run ATOM vLLM OOT model accuracy tests
ci:sglang Run ATOM SGLang model accuracy tests

Heavy jobs are skipped when the PR is not approved and no matching ci:* label is present.
Add labels via the sidebar or gh pr edit 2483 --add-label <label>

@valarLip
valarLip merged commit 7fc0926 into main Oct 7, 2026
@valarLip
valarLip deleted the feat/mono-step-begin-flydsl branch October 7, 2026 02:16
seungrokj added a commit to SemiAnalysisAI/InferenceX that referenced this pull request Oct 8, 2026
… every TP4 point

minimaxm3-fp4-mi355x-atom-agentic-mtp:

- Image rocm/atom-dev:nightly_202609281543 (ATOM 0.1.7.dev43+gaa0c5c3a1)
  -> nightly_202610070333 (sha256:2f605a3e8b459a0385f784edb0335c11892277fee650fe42044e4af39e36dd01;
  ATOM 0.1.7.dev60+g7fc0926c6, ROCm 7.2.4). ROCm/ATOM#2479 moves M3 mono
  decode onto the shared mono framework (mailbox flags published only after
  their stores drain; exact top-k sparse selection); ROCm/ATOM#2483 rebuilds
  the per-step mailbox clear and cross-rank fence as one FlyDSL launch,
  replacing a Triton fence that could spin forever on gfx950; ROCm/ATOM#2422
  fixes the mono long-context decision. Upstream
  recipes/MiniMax-M3-Agentic-InferenceX.md is unchanged.
- Mono decode on at every TP4 point: drop ATOM_MONO_ENABLE=0 from the
  indexer-CP points C15/C20/C24/C28. Mono serves decode steps of up to 16
  tokens (four requests with EAGLE3 K3), covering all of TP4 C1/C2/C4.
  ATOM builds mono for TP4 only, so TP2 runs the unfused path.
- Add TP2 C4 (max-num-seqs 8); GPU-resident TP2 points are C1/C2/C4.
- Record LMCache 0.5.6.dev139+gf22dec28 (the wheel in the new image) for
  both DRAM-offload arms; offload settings unchanged.
- Recipe cites NVIDIA/srt-slurm#507 (in srt-slurm since v2.36.0) for
  extra-kv-connectors instead of the dropped 507 patch.
- No data-type or precision change to the EAGLE3 draft
  (Inferact/MiniMax-M3-EAGLE3-GQA).

Risk: ROCm/ATOM#2452 (CUDA-graph pad rows in the wide MoE path, 5+ tokens)
is not in this image; every batch size 1-18 is captured, so steps of up to
four requests are never padded.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
seungrokj added a commit to SemiAnalysisAI/InferenceX that referenced this pull request Oct 8, 2026
…ghtly

dsv41flash-fp4-mi355x-atom-agentic-dspark:

- Restore DeepSeek-V4.1-Flash FP4 ATOM AgentX on MI355X (#3387) on the
  native srt-slurm recipe srt-slurm-recipes/dsv41flash/atom/mi355x-fp4-mtp/
  agentic.yaml (legacy script and master entry were removed in #3461/#3463).
  TP2 conc [1,2,4,5,6,8,16,32,64,128] (adds 4,5,6,128); TP4 conc
  [1,4,5,6,8,32] (adds 1,4,5,6; drops 2,16,64). No EP or KV offload.
  Conc 1-8 covers the batch sizes V4.1 mono decode serves (<= 8 requests).
- Image rocm/atom-dev:nightly_202609250902 -> nightly_202610070333
  (sha256:2f605a3e8b459a0385f784edb0335c11892277fee650fe42044e4af39e36dd01;
  ATOM 0.1.7.dev60+g7fc0926c6, ROCm 7.2.4):
  - ROCm/ATOM#2419: group32 FP8 weight preshuffle and BF16 RoPE tables for V4.1.
  - ROCm/ATOM#2479: V4.1 mono decode (on by default, <= 8 requests a step
    for this TP2/TP4, BF16-KV, five-token DSpark deployment) and the FP4
    index plane used by upstream recipes/DeepSeek-V4.1-Flash-Agentic.md.
  - ROCm/aiter#6174: gfx950 tuned rows for six V4.1-Flash TP4 bf16 GEMM
    shapes that fell back to torch (DSpark verify batches, TP4 lm_head).
  - ROCm/ATOM#2483: replaces the Triton mono step fence that could spin
    forever on gfx950; on the 1006 nightlies the TP2 c1 canary hung in
    warmup on the DSpark draft's first mono step (runs 37486079757,
    37493871230).
- Server settings follow the upstream recipe (BF16 KV, index cache FP8 ->
  FP4 per upstream at 7fc0926c, max-num-seqs 128, 16K batching/prefill
  chunks, prefix caching block 16, 8K state checkpoints, level 3, FULL
  graphs, dsml_v41 tool parser); every point now captures
  [1..32, 48, 64, 128]. Five-token DSpark uses golden AL 3.51 (thinking_on)
  for throughput, real acceptance for eval; 3600 s weka_062126 replay with
  five warmup requests per lane.
- DSpark draft (layers 37-39, inline): no --online_quant_config; attention
  and projection linears stay native FP8 32x32. Via the image bump, the
  draft's routed FP4 experts take MXFP8 instead of MXFP4 activations
  (ROCm/ATOM#2479) and the V4.1 RoPE tables are BF16 instead of FP32
  (ROCm/ATOM#2419). The draft has no indexer, so index-cache-dtype fp4
  does not apply to it.
- TP4 conc 32: ATOM_PREFILL_DECODE_INTERVAL=16 (local 3600 s replay:
  108.2 -> 150.9 tok/s/user, ITL p90 9.24 -> 6.63 ms, TTFT p50
  0.51 -> 1.14 s, throughput unchanged).
- TP2 conc 64: ATOM_PREFILL_DECODE_INTERVAL=4 (30.3 -> 38.8 tok/s/user,
  105.4k -> 115.2k tok/s/GPU, 6487 -> 6941 requests, TTFT p90 2.3 -> 6.4 s).
  All other points unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
seungrokj added a commit to SemiAnalysisAI/InferenceX that referenced this pull request Oct 8, 2026
… every TP4 point

minimaxm3-fp4-mi355x-atom-agentic-mtp:

- Image rocm/atom-dev:nightly_202609281543 (ATOM 0.1.7.dev43+gaa0c5c3a1)
  -> nightly_202610070333 (sha256:2f605a3e8b459a0385f784edb0335c11892277fee650fe42044e4af39e36dd01;
  ATOM 0.1.7.dev60+g7fc0926c6, ROCm 7.2.4). ROCm/ATOM#2479 moves M3 mono
  decode onto the shared mono framework (mailbox flags published only after
  their stores drain; exact top-k sparse selection); ROCm/ATOM#2483 rebuilds
  the per-step mailbox clear and cross-rank fence as one FlyDSL launch,
  replacing a Triton fence that could spin forever on gfx950; ROCm/ATOM#2422
  fixes the mono long-context decision. Upstream
  recipes/MiniMax-M3-Agentic-InferenceX.md is unchanged.
- Mono decode on at every TP4 point: drop ATOM_MONO_ENABLE=0 from the
  indexer-CP points C15/C20/C24/C28. Mono serves decode steps of up to 16
  tokens (four requests with EAGLE3 K3), covering all of TP4 C1/C2/C4.
  ATOM builds mono for TP4 only, so TP2 runs the unfused path.
- Add TP2 C4 (max-num-seqs 8); GPU-resident TP2 points are C1/C2/C4.
- Record LMCache 0.5.6.dev139+gf22dec28 (the wheel in the new image) for
  both DRAM-offload arms; offload settings unchanged.
- Recipe cites NVIDIA/srt-slurm#507 (in srt-slurm since v2.36.0) for
  extra-kv-connectors instead of the dropped 507 patch.
- No data-type or precision change to the EAGLE3 draft
  (Inferact/MiniMax-M3-EAGLE3-GQA).

Risk: ROCm/ATOM#2452 (CUDA-graph pad rows in the wide MoE path, 5+ tokens)
is not in this image; every batch size 1-18 is captured, so steps of up to
four requests are never padded.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
seungrokj added a commit to SemiAnalysisAI/InferenceX that referenced this pull request Oct 8, 2026
… every TP4 point

minimaxm3-fp4-mi355x-atom-agentic-mtp:

- Image rocm/atom-dev:nightly_202609281543 (ATOM 0.1.7.dev43+gaa0c5c3a1)
  -> nightly_202610070333 (sha256:2f605a3e8b459a0385f784edb0335c11892277fee650fe42044e4af39e36dd01;
  ATOM 0.1.7.dev60+g7fc0926c6, ROCm 7.2.4). ROCm/ATOM#2479 moves M3 mono
  decode onto the shared mono framework (mailbox flags published only after
  their stores drain; exact top-k sparse selection); ROCm/ATOM#2483 rebuilds
  the per-step mailbox clear and cross-rank fence as one FlyDSL launch,
  replacing a Triton fence that could spin forever on gfx950; ROCm/ATOM#2422
  fixes the mono long-context decision. Upstream
  recipes/MiniMax-M3-Agentic-InferenceX.md is unchanged.
- Mono decode on at every TP4 point: drop ATOM_MONO_ENABLE=0 from the
  indexer-CP points C15/C20/C24/C28. Mono serves decode steps of up to 16
  tokens (four requests with EAGLE3 K3), covering all of TP4 C1/C2/C4.
  ATOM builds mono for TP4 only, so TP2 runs the unfused path.
- Add TP2 C4 (max-num-seqs 8); GPU-resident TP2 points are C1/C2/C4.
- Record LMCache 0.5.6.dev139+gf22dec28 (the wheel in the new image) for
  both DRAM-offload arms; offload settings unchanged.
- Recipe cites NVIDIA/srt-slurm#507 (in srt-slurm since v2.36.0) for
  extra-kv-connectors instead of the dropped 507 patch.
- No data-type or precision change to the EAGLE3 draft
  (Inferact/MiniMax-M3-EAGLE3-GQA).

Risk: ROCm/ATOM#2452 (CUDA-graph pad rows in the wide MoE path, 5+ tokens)
is not in this image; every batch size 1-18 is captured, so steps of up to
four requests are never padded.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
edwingao28 pushed a commit to SemiAnalysisAI/InferenceX that referenced this pull request Oct 8, 2026
… every TP4 point / MiniMax-M3 ATOM AgentX 更新至 1007 nightly 并在所有 TP4 点位启用 mono decode (#3750)

* perf(amd): MiniMax-M3 ATOM AgentX on nightly_202610061452 with mono decode at every TP4 point

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* docs: point MiniMax-M3 ATOM extra-kv-connectors at srt-slurm #507 instead of the dropped patch

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* chore: set MiniMax-M3 ATOM changelog pr-link to #3750

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* perf(amd): move MiniMax-M3 ATOM AgentX to nightly_202610070333 (ATOM #2483 mono step-fence fix)

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* perf(amd): MiniMax-M3 ATOM AgentX on 1007 nightly with mono decode at every TP4 point

minimaxm3-fp4-mi355x-atom-agentic-mtp:

- Image rocm/atom-dev:nightly_202609281543 (ATOM 0.1.7.dev43+gaa0c5c3a1)
  -> nightly_202610070333 (sha256:2f605a3e8b459a0385f784edb0335c11892277fee650fe42044e4af39e36dd01;
  ATOM 0.1.7.dev60+g7fc0926c6, ROCm 7.2.4). ROCm/ATOM#2479 moves M3 mono
  decode onto the shared mono framework (mailbox flags published only after
  their stores drain; exact top-k sparse selection); ROCm/ATOM#2483 rebuilds
  the per-step mailbox clear and cross-rank fence as one FlyDSL launch,
  replacing a Triton fence that could spin forever on gfx950; ROCm/ATOM#2422
  fixes the mono long-context decision. Upstream
  recipes/MiniMax-M3-Agentic-InferenceX.md is unchanged.
- Mono decode on at every TP4 point: drop ATOM_MONO_ENABLE=0 from the
  indexer-CP points C15/C20/C24/C28. Mono serves decode steps of up to 16
  tokens (four requests with EAGLE3 K3), covering all of TP4 C1/C2/C4.
  ATOM builds mono for TP4 only, so TP2 runs the unfused path.
- Add TP2 C4 (max-num-seqs 8); GPU-resident TP2 points are C1/C2/C4.
- Record LMCache 0.5.6.dev139+gf22dec28 (the wheel in the new image) for
  both DRAM-offload arms; offload settings unchanged.
- Recipe cites NVIDIA/srt-slurm#507 (in srt-slurm since v2.36.0) for
  extra-kv-connectors instead of the dropped 507 patch.
- No data-type or precision change to the EAGLE3 draft
  (Inferact/MiniMax-M3-EAGLE3-GQA).

Risk: ROCm/ATOM#2452 (CUDA-graph pad rows in the wide MoE path, 5+ tokens)
is not in this image; every batch size 1-18 is captured, so steps of up to
four requests are never padded.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: seungrokj <seungrok.jung@amd.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant