Repository navigation
Mitigate Mono mailbox polling stalls during concurrent transfers - #3350
Conversation
🏷️ CI GuideRuns automatically on every eligible PR before approval:
Heavy model tests:
|
There was a problem hiding this comment.
🔵 Needs a closer look
Shared GPU mailbox synchronization affects multiple model families, but the hardware arbitration mechanism is unconfirmed and validation is limited to gfx950 reproductions, leaving full-model correctness and performance across architectures for human GPU review.
No open issues 🎉
What changed in this PR
This PR adjusts ATOM’s shared Mono mailbox polling to address hangs observed when memory-copy kernels run concurrently on another GPU stream.
Changes:
- Increases the sleep immediate between unsuccessful polls from
1to16, reducing repeated mailbox reads. - Leaves payload checks, cache policy, resource ownership, and completion handling unchanged.
The PR reports successful concurrent-copy validation on gfx950; full-model accuracy, end-to-end performance, and other GPU architectures remain unvalidated.
| File | Description |
|---|---|
atom/mono/device/sync.py |
Increases the sleep interval in _spin between unsuccessful mailbox polls. |
Review effort: Balanced
valar-cop
left a comment
There was a problem hiding this comment.
some changes needed..
decode-step latency / ttft for the 1->16 backoff on a model using the shared mailbox?
ci green.
| v = load_all() | ||
| while pending(v): | ||
| rocdl.s_sleep(1) | ||
| rocdl.s_sleep(16) |
There was a problem hiding this comment.
same loop still sleeps 1 in _spin_bounded, and the hi/qwen38/dsv4 mailbox forks spin with s_nop 0, tighter than the s_sleep(1) that hangs. apply the backoff there too or say why exempt?
| v = load_all() | ||
| while pending(v): | ||
| rocdl.s_sleep(1) | ||
| rocdl.s_sleep(16) |
There was a problem hiding this comment.
leave some comment on why 16, else someone simplifies it back to 1 and the hang returns..
| v = load_all() | ||
| while pending(v): | ||
| rocdl.s_sleep(1) | ||
| rocdl.s_sleep(16) |
There was a problem hiding this comment.
no reproducer checked in? reference the command/topology or add a minimal harness so a revert can be verified.
valarLip
left a comment
There was a problem hiding this comment.
⚠️ Action required before merge (5)
- Treat 1→16 as a mitigation, not a fix, unless you can show the mechanism. By the PR's own data, sleep(1) hung at about 1 round in 12. Zero hangs in 544 rounds at sleep(16) only bounds the rate below about 0.55% a round (95%), roughly 15x rarer. That is not "fixed" (
probe_backoff.py). Mono already treats concurrent copy kernels as a wedge risk: seeatom/mono/runtime/deployment.py:194-198, andrunner.py:531/hi_dispatch.py:105join the side stream before launching. MiniMax-M3 (minimax_m3/mono/dispatch.py:51) and DSV4.1 non-hi (deepseek_v41/mono/config.py:34-79, which already refusesATOM_DSV41_SIDE_STREAMSfor this exact CU contention) never callresident_grid_refusals, so they admit any KV connector, including Mooncake Store, whose unpack runs on a side stream (mooncake_store/worker.py:871-917). Please either order the Store copy against the Mono launch, or addresident_grid_refusals(..., connectors=...)to those models. Otherwise, retitle the PR as "reduce hang rate" and keep the root-cause issue open. - Apply the backoff at every poll site of this protocol, or justify each exemption (valar-cop).
_spin_boundedatsync.py:55still usess_sleep(1). That is the debug build you would use to chase this hang, and it now polls 16x faster than production. The same goes forstep_begin.py:105/129/135. The forks usespin_pause(s_nop 0, even tighter):hi/device/gfx950/mailbox.py:257,qwen38/.../mailbox.py:209,polls.py:52,hc.py,dual.py. Qwen3.8 is the one Mono model that explicitly admits a copy connector (qwen38/mono/config.py:36,149), and it never reachessync._spin. - Name and explain the 16 (valar-cop). Hoist it to one constant next to
POLL_MAX(sync.py:20), used by_spinand_spin_bounded. Add a comment: about 1024 clocks (64-clock units), "s_sleep(1) hung with a concurrent copy kernel on gfx950, see #3350 repro; do not lower without re-running it". - Check in or reference the reproducer (valar-cop). The gfx950 Mono plus HBM/KV-copy harness and its command or topology should be in
tools/ortests/, or at least linked, so a revert can be verified. Nothing intests/touchesatom/mono/device/sync.pytoday. - Post decode latency numbers (valar-cop). Every missed poll now waits about 1024 clocks instead of 64, which adds about 0.2 µs per critical-path hand-off. At a few hundred serial hand-offs a step that is tens to 100+ µs (
probe_backoff.py). Please give before/after TPOT/TTFT on one model using the shared mailbox (MiniMax-M3 or Kimi-K3 mono).
Reviewed at c83d06d. CI: pre-checkin green; heavy model jobs not run (no label/approval). Full finding list in result.json.
|
ci triage for 4fdec72:
|
| """Keep tagged-mailbox polling loads inside the retry loop.""" | ||
|
|
||
| llvm.InlineAsmOp(None, [], "s_nop 0", "", has_side_effects=True) | ||
| llvm.InlineAsmOp(None, [], f"s_sleep {POLL_SLEEP_UNITS}", "", has_side_effects=True) |
There was a problem hiding this comment.
Addressed in 4fdec7239 (now in the rebased history): the GLM launcher captures the shared polling source digest (_ = source_key, which FlyDSL keys as a closure scalar), so a change to POLL_SLEEP_UNITS or the shared helpers changes the JIT key; a mutation check on the real digest and key implementation is described in the PR description.
There was a problem hiding this comment.
🔵 Needs a closer look
Shared GPU polling changes span several model families, but limited gfx950 runs and substitute V4.1 timings leave unvalidated paths and residual hang and performance risks requiring maintainer judgment.
Review effort: Balanced
valarLip
left a comment
There was a problem hiding this comment.
⚠️ Action required before merge (3)
- Finish the "mitigation" path from r19 #1: open a root-cause issue and remove the safety claim your own repro disproves. The retitle and the residual-risk wording are fine, and the backoff can merge as a mitigation. Two things are still missing. First, there is no tracking issue:
atom/mono/polling.py:6-7points at this PR, which closes on merge. Open an issue (hardware root cause, plus the ordering or refusal of Mooncake Store copies next to resident-grid Mono) and cite it there. Second,resident_grid_refusals(atom/mono/runtime/deployment.py:186-188) still says allowed connectors' copy kernels "finish on their own". Your repro shows a copy kernel next to K1 wedging. Qwen3.8 admitslmcache_offloadon that premise (qwen38/mono/config.py:36,149), and MiniMax-M3 (minimax_m3/mono/dispatch.py:51) and DSV4.1 non-hi (deepseek_v41/mono/config.py:34-79) still admit any connector. Reword the docstring and list all three models in the issue. The root-cause fix (Store ordering, orresident_grid_refusals(..., connectors=...)for M3 and V4.1 non-hi) can then land as a follow-up. - Don't put the reproducer's side effects or required checks inside
assert. Intools/mono_mailbox_copy_repro.py:87,89,91,hipExtStreamCreateWithCUMask,hipExtStreamGetCUMaskandhipStreamGetFlagsare called insideassert ... == 0, and so ishipStreamDestroyat:474. Underpython -Othese calls are skipped entirely (probe_assert_O.py). The streams are never created, and the GPU-count and gfx950 guards at:60,64vanish too. This is the tool thatpolling.pysays must be re-run before anyone lowers the constant. Userc = ...; if rc: raise RuntimeError(...). - Fix the stale comments (small).
spin_pauseinmono/device/ops.py:592anddeepseek_v4/mono/kernels/primitives.py:46still describes only load-hoisting. Say that it sleepsPOLL_SLEEP_UNITS.- The repro comment at
:205-206says "Zero FP8 weights" (and has the typo "are0"), but:262,265fill them with 0x38. mono_mailbox_copy_repro.md:51-53says a run "checks bytes and Q/ring". That is only true in--mode side.- GLM: use the repo idiom
_ = source_key(as instep_begin.py:189andqwen38 emit.py:221) rather thanassert source_key. - Optional: add a CPU test that pins
POLL_SLEEP_UNITSat every gfx950 poll site andmono/polling.pyin each digest.
Re-reviewed at 4fdec72. Already fixed: #2 all gfx950 poll sites (_spin_bounded, step_begin, hi/qwen38/dsv4/GLM spin_pause, dsv4 poll) now back off; #3 a named POLL_SLEEP_UNITS with a "do not lower" comment, shared via atom/mono/polling.py (sync imports ops, so it can't live next to POLL_MAX); #4 reproducer checked in; #5 V4.1 non-hi shared-mailbox TPOT +0.54% (substitute for M3/K3, accepted); Copilot GLM cache key fixed. CI: pre-checkin green; heavy model jobs skipped (no label/approval). Full list in result.json.
|
@valarLip thanks, all three items are addressed in 1. Tracking issue and the "finish on their own" claim. Opened #3369 (hardware root cause open; Store/Mono ordering or refusal; repro commands; the mitigation and its limits). It lists the three models after I re-read the code: Qwen3.8 admits 2. 3. Stale comments. Verification: AST of the changed non-tool files equals 4fdec72 except the GLM line; Black 26.10.1 and Ruff pass on the touched files; CI-equivalent unit run on this tree vs the PR base ( |
There was a problem hiding this comment.
🔵 Needs a closer look
Maintainer judgment is needed to accept inherited V4.1 evidence for unmeasured MiniMax-M3/Kimi-K3 paths, given hardware-sensitive synchronization changes across Mono families and unresolved concurrent-copy safety.
Review effort: Balanced
valarLip
left a comment
There was a problem hiding this comment.
⚠️ Action required before merge (2)
- Rebase onto main and resolve the
deployment.pyconflict, then let Pre Checkin run on the new head. GitHub reports the PR asCONFLICTING. #2786 (Kimi-K3 mono behind a Mooncake P/D decode) rewroteresident_grid_refusals(atom/mono/runtime/deployment.py:185-207on main): it addedKVConnectorFactory.canonical_nameand changed the check toname in connectors. Your docstring edit at:186-190conflicts with it. Keep main'scanonical_namelogic and your new docstring. Because of the conflict, no Pre Checkin, ATOM, vLLM or SGLang run has started on59439b3; the green runs are all on4fdec72. Main's Kimi-K3 now passes_served_connectors(...), which admits Mooncake decode. Its receives are NIC writes, with no GPU copy kernel. Add one line to #3369 saying so, because the issue still says K3 "refuses every connector". - Tidy the reproducer (small).
- The
tools/mono_mailbox_copy_repro.py:12-13usage line omitsPYTHONPATH="$PWD". Without it, the tool's own checkout-ownership check at:76fails. Copy the command from the.md. :207inspect.signature(k1.build_attn_pre).bind(key)is redundant, because:208callsbuild_attn_pre(key)directly. Drop it, and thestr(inspect.signature(...))field at:213..md:25-26: "Install/run from a full checkout" and "Use PYTHONPATH only for this checkout" read as two different procedures. Keep the singlePYTHONPATH="$PWD"one, and document--deadline(default 15 s), which drivespending_deadline.
- The
Re-reviewed at 59439b3. Already fixed: r20 #1 tracking issue #3369 opened and cited in polling.py:7, deployment.py:190 and qwen38/mono/config.py:36; resident_grid_refusals no longer says copy kernels "finish on their own"; #3369 lists Qwen3.8, MiniMax-M3 and V4.1 non-hi. #2: no assert left in the reproducer; every HIP call and guard goes through require(). #3: spin_pause docstrings (shared, dsv4 and hi) mention POLL_SLEEP_UNITS; weights comment and .md mode wording fixed; GLM uses _ = source_key, which FlyDSL keys as a closure str scalar. Every gfx950 poll site backs off, and every digest that inlines the constant covers mono/polling.py. The optional CPU pin test was declined with a reason; that's acceptable. Full list in result.json.
59439b3 to
79f5179
Compare
The previous commit passed it unconditionally, which breaks the documented rollback checkout (an older ABI without ring_scale).
…ckoff-revert control Build the argument table from k1.ABI.names so an ABI change fails at setup, not in the watchdog loop; look buffers up by name instead of keepers[...] indices; drop the duplicate import-path check and code_identity record and the constant True fields (now in the docstring); rewrite the rollback recipe in the .md as a separate checkout with only the backoff reverted, without a literal SHA.
…e full backoff rollback
5376e56 to
c87219b
Compare
valarLip
left a comment
There was a problem hiding this comment.
⚠️ Action required before merge (2)
- Fix the two stale doc references in the rollback recipe and the helpers comment (small, non-blocking).
tools/mono_mailbox_copy_repro.md(rollback item 3) says the tworocdl.s_sleep(polling.POLL_SLEEP_UNITS)sites indeepseek_v4/mono/kernels/helpers.pyare "at lines 179 and 198". At this head they are at 180 and 199, because the comment aboveSHARED_SOURCE_KEYgrew by one line. Drop the line numbers and name the function, or fix them. The same comment (helpers.py:54-56) says "editing this file, ops.py or mono/polling.py recompiles it", butopshere is the alias forprimitives(helpers.py:28). The file it hashes isdeepseek_v4/mono/kernels/primitives.py, so writeprimitives.py. - Get the heavy accuracy jobs to finish green at this head. Pre Checkin (non-GPU unit tests) now passes, and Black, Ruff, smoke, catalog and the vLLM/SGLang/ATOMesh gates pass. The ATOM Accuracy matrix (Kimi-K3/K3.1, DeepSeek-V4/V4.1/R1, GLM-5.3) is still pending at
c87219b70. This PR changes polling in every gfx950 resident kernel, so those jobs are the only model-level check. Merge after they pass.
Re-reviewed at c87219b (full diff; this force-push is a clean rebase onto main 0819cb363, with the same per-file diff as 5376e56). Already fixed: r29 #1: atom/mono/polling.py:5-17 records the sweep (1 and 4 at 0/6; 8 at 44/44 for 128 rounds but 3/4 for 1024 rounds, hang at round 583; 16 at 44/44 and 12/12; 128 rounds cannot tell 8 from 16; +0.5% TPOT measured at 16 only). The "before lowering" rule is now --repeats 1024, timeout >= 300s, at least 12 runs alongside 16, any hang rejects, in polling.py, the .md, #3369 and the description ("Why 16"). r29 #2: the .md recipe now lists all ten changed sites (polling.py for sync.py x2 and step_begin.py x3, three spin_pause, two helpers.py sites back to rocdl.s_nop(0)), and the control built from it hung 6/6. The helpers.py comment names mono/polling.py. Every gfx950 poll site still backs off and every kernel digest still covers mono/polling.py. The gfx1250 paths and the harness sleep_ticks are untouched, as intended.
…k recipe; primitives.py in the helpers digest comment
|
@valarLip thanks. Both stale references are fixed in cf01817 (comments and docs only; an AST comparison of
|
valarLip
left a comment
There was a problem hiding this comment.
⚠️ Action required before merge (1)
- Let the ATOM Accuracy matrix finish green at this head, then merge. The code is done. Pre Checkin (non-GPU unit tests), Black, Ruff, smoke, catalog and the vLLM/SGLang/ATOMesh gates pass at
cf01817f9. The 14 heavy Accuracy jobs (Kimi-K3/K3.1, DeepSeek-V4/V4.1/R1, GLM-5.3, MiniMax-M3, Qwen3.5/Next, gpt-oss) are still pending. This PR changes polling in every gfx950 resident kernel, so those jobs are the only model-level check. If one fails, say whether it also fails on main (for example the DeepSeek-V4-Pro DSpark "VllmBackend can only be called once").
Re-reviewed at cf01817 (full diff, 16 files; this push only edits comments and docs). Already fixed: the c87219b review's #1, tools/mono_mailbox_copy_repro.md rollback item 3 now names poll() in deepseek_v4/mono/kernels/helpers.py (bounded loop under poll_timeout_ticks and the else loop, now at :180,:199) with no line numbers. The SHARED_SOURCE_KEY comment (helpers.py:55) says primitives.py. The recipe still restores exactly main's instructions at all ten changed sites. Every gfx950 poll site sleeps POLL_SLEEP_UNITS (sync.py:30,56, step_begin.py:106,130,136, three spin_pause, helpers.py:180,199). Every kernel digest covers mono/polling.py. sleep_ticks and gfx1250 are untouched, as intended. The "Why 16" evidence and the 1024-round rule are in polling.py:5-17. Copilot's GLM cache-key thread is fixed (glm52/mono/kernels/kernel.py source_digest(..., "mono/polling.py") + _ = source_key), so the thread can be resolved.
…efault 16) Unset or empty keeps 16. 1 restores main's polling instructions exactly (s_sleep 1 in the shared waits, s_nop 0 in spin_pause and the V4 helpers' poll) for a deployment without concurrent KV copies. Values outside 1..127 are an error: gfx950 sleeps on bits [6:0] of the s_sleep immediate. The value is read at import and folded into every source digest that hashes polling.py, so different values never share a JIT cache entry. Config rejects a bad value before the workers spawn and warns once below 16.
|
@valarLip one small addition on top of the version you reviewed, an operator knob asked for on our side:
Evidence (CPU compile-only on gfx950, no GPU run): the V4.1 K1 code object with the variable unset is byte-identical to This push re-queues the heavy Accuracy matrix, so I will not promise a time. Could you confirm the knob is fine? |
There was a problem hiding this comment.
🔵 Needs a closer look
Polling changes span several Mono GPU families, but the hardware cause remains unresolved and MiniMax-M3/Kimi-K3 lack model-level validation, requiring maintainer judgment on coverage gaps and residual concurrent-copy risk.
Review effort: Balanced
valarLip
left a comment
There was a problem hiding this comment.
⚠️ Action required before merge (2)
- Drop
ATOM_MONO_POLL_SLEEP_UNITSand keepPOLL_SLEEP_UNITS = 16as a plain constant (thecf01817f9code). We prefer one good default to a new env knob, and this one has no evidence behind it:- No proven need. No deployment is named, and nobody has shown that the +0.54% TPOT matters anywhere.
- It reopens the hang. The values it unlocks below 16 are the ones that hung in your own sweep: 1 and 4 in every run, 8 once. Setting
1brings back the #3369 hang with only a warning (an offload/P/D connector could be added later without anyone revisiting the variable). - It makes cache keys fragile. Today hashing
polling.pyis enough. With the knob,CACHE_TAGhas to be added by hand at three digests (thef.name == "polling.py"match inbuild_key.source_digest,helpers.SHARED_SOURCE_KEY, the hisource_digest), and any future digest that forgets it silently reuses a kernel built for another value. All of them are correct at this head, so this is not a bug yet. - It costs about 200 lines. That covers
_units/MAX, theMAIN_POLLINGsplit and_retry_pause, theConfigwarning, anenvs.pyentry that nothing reads, the docs row (which also calls this PR "before #3350"), the 89-line test file, and agetattr(sync, "POLL_SLEEP_UNITS", None)probe in the reproducer.
Rollback isgit revert. The reproducer control can go back to the separate-checkout recipe. If a real deployment needs a lower value, change the default in a PR that brings its TPOT at 16 vs the candidate and the 1024-round sweep.
- Let the full Accuracy matrix pass at the final head. At
ae000a144, 13/14 Accuracy jobs pass and Kimi-K3.1-Pro DSpark is still pending. Removing the knob changes kernel sources, so the matrix reruns. Pre Checkin, Black, Ruff and the vLLM/SGLang/ATOMesh gates pass.
Re-reviewed at ae000a1 (full diff, 21 files). Already fixed: everything from earlier rounds still holds. Every gfx950 poll site backs off by POLL_SLEEP_UNITS (sync.py:30,56, step_begin.py:106,130,136, three spin_pause, both polls in helpers.py::poll()). Every Mono kernel digest, the GLM launcher's included, covers polling.py. sleep_ticks and gfx1250 are untouched. "Why 16" and the 1024-round rule are in polling.py. The reproducer and #3369 are in place, and Copilot's GLM cache-key thread is fixed. The new tests pass on CPU.
| # the V4 helpers' poll), for a regression on a deployment with no concurrent | ||
| # KV copies (no offload or P/D connector). Below 16 beside such copies, 1 and 4 | ||
| # hung and 8 hung once (above); `Config` warns once at startup. | ||
| POLL_SLEEP_UNITS = _units(os.environ.get(ENV)) |
There was a problem hiding this comment.
let's drop this knob and keep the plain 16, no env flag if it can just be on by default
|
@valarLip thanks for the detailed review. On item 1 (dropping For the record, nothing changes for the default: unset or empty is still On item 2: agreed, the full Accuracy matrix has to pass at the final head, and I will not merge before it does. At |
valarLip
left a comment
There was a problem hiding this comment.
✅ LGTM — correction to my previous review
I withdraw item 1 of my last review ("drop ATOM_MONO_POLL_SLEEP_UNITS"). Keeping the variable configurable with a default of 16 is what we agreed with you on 11-16, and the cache keys are correct at this head (the value is in all three digests). Sorry for the noise.
Item 2 is done: at ae000a1448 all checks pass, including the full Accuracy matrix (44 pass, 0 fail).
Optional cleanups, not blocking: the envs.py entry that nothing reads; the getattr(sync, "POLL_SLEEP_UNITS", None) in tools/mono_mailbox_copy_repro.py (read the attribute directly); the docs row that calls this PR "before #3350"; and the Config warning that says every value below 16 hung (only 1, 4 and 8 were tested).
Re-reviewed at ae000a1.

Concurrent memory-copy work on another stream can leave gfx950 Mono mailbox polling waiting indefinitely. This PR inserts a named 16-unit pause between unsuccessful protocol polls, covering shared normal/bounded waits, step-start waits, and the hi, Qwen3.8 and DSV4 forks. GLM5.2 uses the shared polling helper and now includes its dependencies in its launcher cache key. Payload/tag checks, timeout diagnostics and completion/resource ownership remain intact.
Root cause and remaining exposure are tracked in #3369. This is a mitigation, not a proven hardware root-cause fix.
s_sleepdoes not release a resident CTA or CU. Hardware arbitration/cache/completion-signal details remain unresolved. This change does not order every Store copy against Mono, remove existing model admission restrictions, or establish that all concurrent-copy configurations are safe.POLL_SLEEP_UNITS = 16is shared through a dependency-free module, avoiding an ops/sync import cycle. Operators can override it withATOM_MONO_POLL_SLEEP_UNITS(default unchanged; see "Override" below). It represents about 1024 clocks in nominal 64-clock units. The code documents the gfx950 sleep-1 reproductions, the sweep below (see "Why 16"), and requires 1024-round runs, repeated, before lowering it.Override:
ATOM_MONO_POLL_SLEEP_UNITS(default unchanged)cf01817f9: the V4.1 K1 code object is byte-identical (sha25604c2367b3131525d...), and so arestep_begin(tp=8) and a probe kernel that calls each other pause site once (the threespin_pausedefinitions and the pause inpoll()ofdeepseek_v4/mono/kernels/helpers.py).1restores main's polling instructions exactly, not onlys_sleep 1:s_sleep 1insync.py/step_begin.py, ands_nop 0in the threespin_pausedefinitions and both polls ofhelpers.py::poll(). Compile-only, K1 is byte-identical to the PR base0819cb363(sha256a1397f516a5a6461...), andstep_beginand the pause probe disassemble identically to main. It is meant for a deployment that sees a regression and has no concurrent KV copies (no offload or P/D connector). With such copies, 1 is main's behaviour, which hung in the reproducer (table below).s_sleep Nat every site. 0, values above 127 and non-integers are an error at startup (Configconstruction, before the workers spawn), with a message that names the variable. The reason for the range: gfx9/CDNA (gfx950 included) sleeps for about 64 x SIMM16[6:0] clocks, sos_sleep 128assembles but sleeps like 0. ROCm 7.2.4llvm-mc -mcpu=gfx950encodess_sleep 128as0x80without a diagnostic and silently truncates65536to 0.kv_transfer_configis set. The variable is an explicit operator choice, and a refusal would stop1from being a revert for exactly those deployments. The warning anddocs/environment_variables.mdstate the risk.atom/mono/polling.pyreads the value once at import. It still imports onlyos, and spawned workers inherit the environment. The value is folded into every digest that hashespolling.py(build_key.source_digest,helpers.SHARED_SOURCE_KEY, the hisource_digest), so different values never share a JIT cache entry (checked: unset ==16, while1,8and unset give different digests at all three sites and in the GLM launcher key).polling.py's own bytes changed, so the default digests also differ fromcf01817f9's, and existing Mono JIT caches rebuild once.ATOM_MONO_POLL_SLEEP_UNITS=1on the same checkout, which replaces the manual revert recipe.code_identitylogspoll_sleep_units.Why 16 (and what it takes to change it)
Empirical, from the checked-in reproducer (
--mode side, one idle MI355X per run, fresh FlyDSL cache per run, source equal to head6a68383e2apart from the constant under test (the second sweep used a local commit on top that changes only comments and the constant), 8-GPU node, at most 2 runs in parallel). The workload is synthetic (6 token rows, 256 CTA x 512 threads, 724-block side copy), the root cause is unknown, and the sampling is finite: this is evidence about this tool only. It does not prove that the hang risk is zero, and it does not prove that 16 is safe; 16 is only the value for which no hang was observed. 5, 6, 7 and 9-15 were not tested.POLL_SLEEP_UNITSpending_deadline, GPU6)--mode side --repeats 1024(timeout>= 300s; a passing run takes about 60s) on several GPUs, at least 12 runs for the candidate and for 16 in the same batch. 128 rounds are not enough. Any hang rejects the candidate. This is in thepolling.pycomment and intools/mono_mailbox_copy_repro.md.POLL_SLEEP_UNITS = 1and the threespin_pausedefinitions, as the previous recipe said. It did not revert the tworocdl.s_sleep(polling.POLL_SLEEP_UNITS)sites indeepseek_v4/mono/kernels/helpers.py(lines 179 and 198;s_nop(0)on main), so it was not exactly the parent commit's polling. Those two sites are not on the V4.1 K1 path the tool exercises. The recipe intools/mono_mailbox_copy_repro.mdnow lists them. With the corrected recipe (all sites listed there; checked againstorigin/mainthat every changed polling instruction is back), 6 more control runs on this PR's new head source5376e56(GPU0 and GPU1, 3 each, 128 rounds, 120s timeout) hung 6/6 (rounds 0, 9, 1 and 10, 5, 7).10c8871, 128/128) and the TPOT table was measured at 16 only; the sweep is not a re-validation at other values.Validation on an owned MI355X/gfx950 with the same runtime/image, full CU masks and fresh caches:
79149c7completed three rounds, then hit the 15-second GPU completion deadline on round four. Owners remained alive until the external watchdog. Original failure evidence is preserved.10c8871completed 128/128 side-stream rounds with matching Q/ring and restored bytes, busy preconditions and positive GPU Event-window intersections, then exited normally with no remaining owned GPU process. Event-window overlap does not prove physical CU residency; finite successful runs do not establish zero hang risk.4fdec72inherits this evidence through an independently reviewed style-only delta (bb792b6) and a seven-line GLM launcher-key delta. It is not represented as another 128-round GPU run.Same-model latency comparison used the actual V4.1 non-hi shared-mailbox path (
V41MonoDecodeRunner.forward/StepMailboxes.begin_step), g23 GPU2/3, TP2/DPA, no Store, no side copy, the same checkpoint/image/settings, and identical 44 input token IDs / 64 output tokens. Each arm had two warmups followed by five formal C1 requests. Token-ID audit and the separate post-formal graph diagnostic were excluded. TTFT measures first nonempty generated SSE text; TPOT measures visible first-to-last text span divided by 63. Headers and empty role frames are excluded.79149c7bb792b6Observed TPOT difference is +0.02180 ms (+0.54%); the complete-forward diagnostic is +0.81%. With five short requests this is a limited regression probe, not a TTFT/throughput gain or precision claim. All formal requests had the expected IDs, 64 generated frames/tokens and length completion; raw SSE and both actual worker identities were independently audited. Both arms had the same four HIP registration-refusal events with successful host-copy fallback and no mlock refusal or observer error. Worker ISA extraction and isolated pure-Mono-step timing were unavailable; source/capture identities are not substituted for ISA.
Model coverage limitation: available MiniMax-M3 MXFP8 weights do not match its Mono FP4 layout, and Kimi-K3 does not fit the available supported topology. V4.1 is explicitly substitute evidence for the requested M3/K3 timing, subject to maintainer acceptance; those models have not been measured here.
CI corrections use the actual workflow versions, Black 26.10.1 and Ruff 0.17.5. On the checked merge snapshot with the final overlay, Black passed all 1939 files, and all 14 PR-touched files had zero Ruff diagnostics; the whole repository still has 399 existing diagnostics. On current head
4fdec72, remote Black, Ruff, non-GPU unit tests, offline inference smoke and catalog validation completed successfully. Copilot review completed and still lists the previously reported GLM thread as open; its requested cache-key change is in4fdec72and independently checked above. Open/outdated threads are not represented as resolved. Heavy model accuracy jobs were skipped, so these statuses do not establish full model accuracy coverage.The earlier 900-second 8P8D/C256 AgentX diagnostic used the original single-site patch plus a separately frozen #3097 runtime. It exported 6,915 completed requests without an observed GPU hang, but 43 tail-cancel credits required harness forced completion. It is not current-head full correctness, precision or fair performance validation. The physical hardware root cause (#3369) and separate cancellation-credit issue remain open.
Review follow-up (
59439b3, no kernel logic change): tracking issue #3369 is cited inpolling.py; theresident_grid_refusalsdocstring and the Qwen3.8 connector comment no longer claim allowed connectors' copy kernels cannot wedge the resident kernels (Qwen3.8lmcache_offload, MiniMax-M3 and DSV4.1 non-hi admissions are listed in #3369); the reproducer no longer runs HIP calls or guards/checks insideassert, sopython -Obehaves like a normal run; stalespin_pausedocstrings, reproducer comment and doc wording are corrected; GLM uses_ = source_key. Checked: AST of all changed non-tool files is identical to4fdec72apart from the GLM line; the reproducer ran with and without-Oon gfx950 (--setup-only,mono-alone/copy-alonex3,sidex8); CI-equivalent unit run has the same 3 failures as the PR base (environment-related). The root-cause fix (Store ordering, orconnectors=for M3/V4.1 non-hi) is left to a follow-up.Second review follow-up (rebased onto main
8220b81, head6a68383e2, no kernel logic change): the only conflict was theresident_grid_refusalsdocstring inatom/mono/runtime/deployment.py; main'sKVConnectorFactory.canonical_namelogic is kept unchanged and only the docstring from this PR is applied.git diff origin/main HEADhas the same per-file changes as the pre-rebase PR diff. Reproducer (79f5179): the usage line carriesPYTHONPATH="$PWD", the redundantinspect.signature(...)probe and its output field are removed, and the.mddocuments a singlePYTHONPATH="$PWD"procedure and--deadline(default 15 s, per round; drivespending_deadline).0ab6a3e/67cb711addring_scaleto the reproducer's K1 argument table, only when the checkout's K1 ABI has it: main's K1 ABI gained that field in #2989, and without it the tool fails with an ABI mismatch after this rebase. Checked on gfx950 (1 GPU): the--setup-only,mono-alonex3 (also on parent79149c7) andcopy-alonex3 commands from the.mdpass,sidex128 completes 128/128 with Q/ring and restored bytes matching; withoutPYTHONPATHthe tool stops at its ownership check. CI-equivalent unit run on the rebased tree has the same 3 failures as main8220b81(test_benchmark_catalogx2,test_multiprocess_metrics) and one extra passing test (the newpolling.pyimport check). #3369 now says that Kimi-K3 passes_served_connectors(...)on main.Third follow-up (
6a68383e2, tools/docs only): the reproducer now builds its K1 argument table fromk1.ABI.names(a field that is neither a buffer nor a known zero fails at setup, andring_scaleis simply absent on older ABIs), looks buffers up by name, and drops the duplicated import-path check, the duplicatedcode_identityrecord and the constantTruerecord fields (now in the docstring). The.mdrollback recipe is a separate checkout of the same revision with only the backoff reverted (POLL_SLEEP_UNITS = 1,s_nop 0in the threespin_pausedefinitions), without a literal SHA. On one gfx950 GPU at6a68383e2, all four.mdcommands exit 0 (sidex128: 128/128 rounds, 128 busy and 128 overlap samples, Q/ring and restored bytes match). In the reverted-backoff checkout the samesidex128 command stops atpending_deadline(round 17 in one run, round 0 in another) and ends with thetimeoutexit 124.Fourth follow-up (comments and docs only, no kernel logic change): records the sweep above next to
POLL_SLEEP_UNITSinpolling.pyand in "Why 16"; changes the "before lowering" check inpolling.pyand the.mdto--repeats 1024, repeated, at least 12 runs per value with 16 in the same batch; the.mdrollback recipe now also revertsdeepseek_v4/mono/kernels/helpers.py:179,198torocdl.s_nop(0)(the earlier "third follow-up" control run above used the shorter recipe);helpers.pycomment now namesmono/polling.pyas a recompile trigger. AST of changed.pyfiles is identical to6a68383e2; Black 26.10.1 and Ruff 0.17.5 pass on them. Pushing changes comments in kernel-bearing files, so GitHub CI restarts on the new head.Fifth follow-up (
306adb8a7,ae000a144; requested by the PR author as an operations switch; default behaviour unchanged): addsATOM_MONO_POLL_SLEEP_UNITS(see "Override" above). Changed:atom/mono/polling.py(env read and range check;SPIN_PAUSE_ASM,MAIN_POLLING,CACHE_TAG), the threespin_pausedefinitions andhelpers.py::poll()(_retry_pause) take the instruction from it, the three source digests foldCACHE_TAG,Config.__post_init__calls_check_mono_poll_sleep_units()(early error, one warning below 16),envs.py/docs/environment_variables.mdregister the variable, and the reproducer's rollback control andcode_identityuse it. 19 new CPU tests (tests/test_mono_poll_sleep_units.py): unset/empty -> 16; 1 ->s_nop 0/ main; invalid values (0, 128, -1, 65536,abc,1.5,16x) ->ValueErrornaming the variable, also as an import failure in a fresh process; digests of 16 / 1 / 8 differ, unset == 16; the warning appears below 16 only. Compile-only on gfx950 (CPU,dpc_sn, no GPU): default ==cf01817f9and1== main for the K1 code object,step_beginand the pause-site probe, as listed above. CI-equivalent unit run (CPU, serial): head andcf01817f9have the same 22 failures, all from this environment (21test_atomesh_spur_dispatch15 s subprocess timeouts on a loaded node andtest_multiprocess_metrics), and head adds 19 passing tests. Black 26.10.1 and Ruff 0.17.5 are clean on the changed files. No new GPU run: the behaviour of each value comes from the sweep in "Why 16". This push restarts the heavy Accuracy jobs.