Skip to content

Mitigate Mono mailbox polling stalls during concurrent transfers - #3350

Merged
valarLip merged 14 commits into
mainfrom
lirzhang/mono-mailbox-poll-backoff
Nov 17, 2026
Merged

valarLip merged 14 commits into
mainfrom
lirzhang/mono-mailbox-poll-backoff

Conversation

@ZhangLirong-amd

@ZhangLirong-amd ZhangLirong-amd commented Nov 14, 2026 •

Copy link
Copy Markdown
Collaborator

Concurrent memory-copy work on another stream can leave gfx950 Mono mailbox polling waiting indefinitely. This PR inserts a named 16-unit pause between unsuccessful protocol polls, covering shared normal/bounded waits, step-start waits, and the hi, Qwen3.8 and DSV4 forks. GLM5.2 uses the shared polling helper and now includes its dependencies in its launcher cache key. Payload/tag checks, timeout diagnostics and completion/resource ownership remain intact.

Root cause and remaining exposure are tracked in #3369. This is a mitigation, not a proven hardware root-cause fix. s_sleep does not release a resident CTA or CU. Hardware arbitration/cache/completion-signal details remain unresolved. This change does not order every Store copy against Mono, remove existing model admission restrictions, or establish that all concurrent-copy configurations are safe.

  • POLL_SLEEP_UNITS = 16 is shared through a dependency-free module, avoiding an ops/sync import cycle. Operators can override it with ATOM_MONO_POLL_SLEEP_UNITS (default unchanged; see "Override" below). It represents about 1024 clocks in nominal 64-clock units. The code documents the gfx950 sleep-1 reproductions, the sweep below (see "Why 16"), and requires 1024-round runs, repeated, before lowering it.
  • Source/cache keys include the polling constant and shared dependencies. The latest GLM correction explicitly captures its source digest in the actual launcher, preventing this scalar dependency from being omitted from the FlyDSL key. Harness-only clock delays, instruction-hazard nops and the separate gfx1250 counter wait are exempt; the gfx950 empirical value is not generalized to that architecture.
  • Checked-in reproducer and rollback commands use real K1 and public native contiguous-copy APIs. In-flight owners are retained on error until the external watchdog terminates the process.

Override: ATOM_MONO_POLL_SLEEP_UNITS (default unchanged)

  • Unset or empty: 16, the behaviour reviewed above. Compile-only on gfx950 (CPU, no GPU), a build without the variable is instruction-identical to cf01817f9: the V4.1 K1 code object is byte-identical (sha256 04c2367b3131525d...), and so are step_begin (tp=8) and a probe kernel that calls each other pause site once (the three spin_pause definitions and the pause in poll() of deepseek_v4/mono/kernels/helpers.py).
  • 1 restores main's polling instructions exactly, not only s_sleep 1: s_sleep 1 in sync.py / step_begin.py, and s_nop 0 in the three spin_pause definitions and both polls of helpers.py::poll(). Compile-only, K1 is byte-identical to the PR base 0819cb363 (sha256 a1397f516a5a6461...), and step_begin and the pause probe disassemble identically to main. It is meant for a deployment that sees a regression and has no concurrent KV copies (no offload or P/D connector). With such copies, 1 is main's behaviour, which hung in the reproducer (table below).
  • 2..127: s_sleep N at every site. 0, values above 127 and non-integers are an error at startup (Config construction, before the workers spawn), with a message that names the variable. The reason for the range: gfx9/CDNA (gfx950 included) sleeps for about 64 x SIMM16[6:0] clocks, so s_sleep 128 assembles but sleeps like 0. ROCm 7.2.4 llvm-mc -mcpu=gfx950 encodes s_sleep 128 as 0x80 without a diagnostic and silently truncates 65536 to 0.
  • Below 16, startup logs one warning that cites Mono mailbox polling can stall next to concurrent copy kernels on gfx950 (root cause open; backoff in #3350 is a mitigation) #3369 and says lower values hung beside offload / P/D copies. Mono is not refused when kv_transfer_config is set. The variable is an explicit operator choice, and a refusal would stop 1 from being a revert for exactly those deployments. The warning and docs/environment_variables.md state the risk.
  • Cache key: atom/mono/polling.py reads the value once at import. It still imports only os, and spawned workers inherit the environment. The value is folded into every digest that hashes polling.py (build_key.source_digest, helpers.SHARED_SOURCE_KEY, the hi source_digest), so different values never share a JIT cache entry (checked: unset == 16, while 1, 8 and unset give different digests at all three sites and in the GLM launcher key). polling.py's own bytes changed, so the default digests also differ from cf01817f9's, and existing Mono JIT caches rebuild once.
  • The reproducer's rollback control is now ATOM_MONO_POLL_SLEEP_UNITS=1 on the same checkout, which replaces the manual revert recipe. code_identity logs poll_sleep_units.

Why 16 (and what it takes to change it)

Empirical, from the checked-in reproducer (--mode side, one idle MI355X per run, fresh FlyDSL cache per run, source equal to head 6a68383e2 apart from the constant under test (the second sweep used a local commit on top that changes only comments and the constant), 8-GPU node, at most 2 runs in parallel). The workload is synthetic (6 token rows, 256 CTA x 512 threads, 724-block side copy), the root cause is unknown, and the sampling is finite: this is evidence about this tool only. It does not prove that the hang risk is zero, and it does not prove that 16 is safe; 16 is only the value for which no hang was observed. 5, 6, 7 and 9-15 were not tested.

POLL_SLEEP_UNITS 128-round runs (pass/total) 1024-round runs (pass/total) Hang (0-based round)
1 0/6 - all 6 hung, rounds 0-12
4 0/6 - all 6 hung, rounds 2-22
8 44/44 3/4 1 hang at round 583 (pending_deadline, GPU6)
16 44/44 12/12 none
  • 128-round runs cannot tell 8 from 16: both passed 44/44. Only the 1024-round runs separated them. Over all rounds, 8 had 1 hang in 7,495 rounds and 16 had 0 in 16,128; this difference is not statistically established, but it is the reason 8 was not adopted. Whether GPU6 is special was not tested.
  • Cost: TPOT +0.54% (complete-forward diagnostic +0.81%) is the V4.1 non-hi measurement below, taken at 16 only. TPOT at 8 was not measured.
  • Before lowering the value: run --mode side --repeats 1024 (timeout >= 300s; a passing run takes about 60s) on several GPUs, at least 12 runs for the candidate and for 16 in the same batch. 128 rounds are not enough. Any hang rejects the candidate. This is in the polling.py comment and in tools/mono_mailbox_copy_repro.md.
  • Positive control (full revert of the backoff, 128 rounds): 12/12 hung in the two sweeps, so the tool does reproduce the hang. Caveat: in those sweeps the control reverted only POLL_SLEEP_UNITS = 1 and the three spin_pause definitions, as the previous recipe said. It did not revert the two rocdl.s_sleep(polling.POLL_SLEEP_UNITS) sites in deepseek_v4/mono/kernels/helpers.py (lines 179 and 198; s_nop(0) on main), so it was not exactly the parent commit's polling. Those two sites are not on the V4.1 K1 path the tool exercises. The recipe in tools/mono_mailbox_copy_repro.md now lists them. With the corrected recipe (all sites listed there; checked against origin/main that every changed polling instruction is back), 6 more control runs on this PR's new head source 5376e56 (GPU0 and GPU1, 3 each, 128 rounds, 120s timeout) hung 6/6 (rounds 0, 9, 1 and 10, 5, 7).
  • The older validation in the next section was obtained at 16 (10c8871, 128/128) and the TPOT table was measured at 16 only; the sweep is not a re-validation at other values.

Validation on an owned MI355X/gfx950 with the same runtime/image, full CU masks and fresh caches:

  • Setup/full-byte restoration (267,822,080 bytes), Mono-only 3/3, copy-only 3/3 and side-stream 3/3 controls passed.
  • The same checked-in tool on parent 79149c7 completed three rounds, then hit the 15-second GPU completion deadline on round four. Owners remained alive until the external watchdog. Original failure evidence is preserved.
  • Candidate 10c8871 completed 128/128 side-stream rounds with matching Q/ring and restored bytes, busy preconditions and positive GPU Event-window intersections, then exited normally with no remaining owned GPU process. Event-window overlap does not prove physical CU residency; finite successful runs do not establish zero hang risk.
  • Final 4fdec72 inherits this evidence through an independently reviewed style-only delta (bb792b6) and a seven-line GLM launcher-key delta. It is not represented as another 128-round GPU run.
  • Independent mutation checks exercised real source digests and the actual FlyDSL scalar/key implementation: same-size polling 16-to-15, shared ops and GLM helper changes alter the key; an unrelated Qwen change does not; restoration returns the baseline key. This isolates the scalar contribution with the actual launcher code object and fixed geometry, without executing the full GLM factory or claiming GPU disk-cache/model validation.

Same-model latency comparison used the actual V4.1 non-hi shared-mailbox path (V41MonoDecodeRunner.forward / StepMailboxes.begin_step), g23 GPU2/3, TP2/DPA, no Store, no side copy, the same checkpoint/image/settings, and identical 44 input token IDs / 64 output tokens. Each arm had two warmups followed by five formal C1 requests. Token-ID audit and the separate post-formal graph diagnostic were excluded. TTFT measures first nonempty generated SSE text; TPOT measures visible first-to-last text span divided by 63. Headers and empty role frames are excluded.

Metric Parent 79149c7 Tested candidate bb792b6
TTFT mean 54.9607 ms 53.2052 ms
TTFT observed range 52.9591-59.1071 ms 51.8732-55.7612 ms
TPOT mean 4.02535 ms 4.04714 ms
TPOT observed range 4.01074-4.03606 ms 4.04110-4.05185 ms
Separate complete-forward-graph diagnostic mean 3.98138 ms 4.01363 ms

Observed TPOT difference is +0.02180 ms (+0.54%); the complete-forward diagnostic is +0.81%. With five short requests this is a limited regression probe, not a TTFT/throughput gain or precision claim. All formal requests had the expected IDs, 64 generated frames/tokens and length completion; raw SSE and both actual worker identities were independently audited. Both arms had the same four HIP registration-refusal events with successful host-copy fallback and no mlock refusal or observer error. Worker ISA extraction and isolated pure-Mono-step timing were unavailable; source/capture identities are not substituted for ISA.

Model coverage limitation: available MiniMax-M3 MXFP8 weights do not match its Mono FP4 layout, and Kimi-K3 does not fit the available supported topology. V4.1 is explicitly substitute evidence for the requested M3/K3 timing, subject to maintainer acceptance; those models have not been measured here.

CI corrections use the actual workflow versions, Black 26.10.1 and Ruff 0.17.5. On the checked merge snapshot with the final overlay, Black passed all 1939 files, and all 14 PR-touched files had zero Ruff diagnostics; the whole repository still has 399 existing diagnostics. On current head 4fdec72, remote Black, Ruff, non-GPU unit tests, offline inference smoke and catalog validation completed successfully. Copilot review completed and still lists the previously reported GLM thread as open; its requested cache-key change is in 4fdec72 and independently checked above. Open/outdated threads are not represented as resolved. Heavy model accuracy jobs were skipped, so these statuses do not establish full model accuracy coverage.

The earlier 900-second 8P8D/C256 AgentX diagnostic used the original single-site patch plus a separately frozen #3097 runtime. It exported 6,915 completed requests without an observed GPU hang, but 43 tail-cancel credits required harness forced completion. It is not current-head full correctness, precision or fair performance validation. The physical hardware root cause (#3369) and separate cancellation-credit issue remain open.

Review follow-up (59439b3, no kernel logic change): tracking issue #3369 is cited in polling.py; the resident_grid_refusals docstring and the Qwen3.8 connector comment no longer claim allowed connectors' copy kernels cannot wedge the resident kernels (Qwen3.8 lmcache_offload, MiniMax-M3 and DSV4.1 non-hi admissions are listed in #3369); the reproducer no longer runs HIP calls or guards/checks inside assert, so python -O behaves like a normal run; stale spin_pause docstrings, reproducer comment and doc wording are corrected; GLM uses _ = source_key. Checked: AST of all changed non-tool files is identical to 4fdec72 apart from the GLM line; the reproducer ran with and without -O on gfx950 (--setup-only, mono-alone/copy-alone x3, side x8); CI-equivalent unit run has the same 3 failures as the PR base (environment-related). The root-cause fix (Store ordering, or connectors= for M3/V4.1 non-hi) is left to a follow-up.

Second review follow-up (rebased onto main 8220b81, head 6a68383e2, no kernel logic change): the only conflict was the resident_grid_refusals docstring in atom/mono/runtime/deployment.py; main's KVConnectorFactory.canonical_name logic is kept unchanged and only the docstring from this PR is applied. git diff origin/main HEAD has the same per-file changes as the pre-rebase PR diff. Reproducer (79f5179): the usage line carries PYTHONPATH="$PWD", the redundant inspect.signature(...) probe and its output field are removed, and the .md documents a single PYTHONPATH="$PWD" procedure and --deadline (default 15 s, per round; drives pending_deadline). 0ab6a3e / 67cb711 add ring_scale to the reproducer's K1 argument table, only when the checkout's K1 ABI has it: main's K1 ABI gained that field in #2989, and without it the tool fails with an ABI mismatch after this rebase. Checked on gfx950 (1 GPU): the --setup-only, mono-alone x3 (also on parent 79149c7) and copy-alone x3 commands from the .md pass, side x128 completes 128/128 with Q/ring and restored bytes matching; without PYTHONPATH the tool stops at its ownership check. CI-equivalent unit run on the rebased tree has the same 3 failures as main 8220b81 (test_benchmark_catalog x2, test_multiprocess_metrics) and one extra passing test (the new polling.py import check). #3369 now says that Kimi-K3 passes _served_connectors(...) on main.

Third follow-up (6a68383e2, tools/docs only): the reproducer now builds its K1 argument table from k1.ABI.names (a field that is neither a buffer nor a known zero fails at setup, and ring_scale is simply absent on older ABIs), looks buffers up by name, and drops the duplicated import-path check, the duplicated code_identity record and the constant True record fields (now in the docstring). The .md rollback recipe is a separate checkout of the same revision with only the backoff reverted (POLL_SLEEP_UNITS = 1, s_nop 0 in the three spin_pause definitions), without a literal SHA. On one gfx950 GPU at 6a68383e2, all four .md commands exit 0 (side x128: 128/128 rounds, 128 busy and 128 overlap samples, Q/ring and restored bytes match). In the reverted-backoff checkout the same side x128 command stops at pending_deadline (round 17 in one run, round 0 in another) and ends with the timeout exit 124.

Fourth follow-up (comments and docs only, no kernel logic change): records the sweep above next to POLL_SLEEP_UNITS in polling.py and in "Why 16"; changes the "before lowering" check in polling.py and the .md to --repeats 1024, repeated, at least 12 runs per value with 16 in the same batch; the .md rollback recipe now also reverts deepseek_v4/mono/kernels/helpers.py:179,198 to rocdl.s_nop(0) (the earlier "third follow-up" control run above used the shorter recipe); helpers.py comment now names mono/polling.py as a recompile trigger. AST of changed .py files is identical to 6a68383e2; Black 26.10.1 and Ruff 0.17.5 pass on them. Pushing changes comments in kernel-bearing files, so GitHub CI restarts on the new head.

Fifth follow-up (306adb8a7, ae000a144; requested by the PR author as an operations switch; default behaviour unchanged): adds ATOM_MONO_POLL_SLEEP_UNITS (see "Override" above). Changed: atom/mono/polling.py (env read and range check; SPIN_PAUSE_ASM, MAIN_POLLING, CACHE_TAG), the three spin_pause definitions and helpers.py::poll() (_retry_pause) take the instruction from it, the three source digests fold CACHE_TAG, Config.__post_init__ calls _check_mono_poll_sleep_units() (early error, one warning below 16), envs.py / docs/environment_variables.md register the variable, and the reproducer's rollback control and code_identity use it. 19 new CPU tests (tests/test_mono_poll_sleep_units.py): unset/empty -> 16; 1 -> s_nop 0 / main; invalid values (0, 128, -1, 65536, abc, 1.5, 16x) -> ValueError naming the variable, also as an import failure in a fresh process; digests of 16 / 1 / 8 differ, unset == 16; the warning appears below 16 only. Compile-only on gfx950 (CPU, dpc_sn, no GPU): default == cf01817f9 and 1 == main for the K1 code object, step_begin and the pause-site probe, as listed above. CI-equivalent unit run (CPU, serial): head and cf01817f9 have the same 22 failures, all from this environment (21 test_atomesh_spur_dispatch 15 s subprocess timeouts on a loaded node and test_multiprocess_metrics), and head adds 19 passing tests. Black 26.10.1 and Ruff 0.17.5 are clean on the changed files. No new GPU run: the behaviour of each value comes from the sweep in "Why 16". This push restarts the heavy Accuracy jobs.

Copilot AI balanced review requested due to automatic review settings November 14, 2026 08:51
@github-actions

Copy link
Copy Markdown
Contributor

🏷️ CI Guide

Runs automatically on every eligible PR before approval:

  • ✅ Pre Checkin: Black, Ruff, catalog schema validation, non-GPU unit tests

Heavy model tests:

  • ✅ Run after the PR is approved and Pre Checkin passes
  • ✅ Run immediately when an approval review is submitted
  • ✅ Can be requested before approval with labels
Label Tests
ci:full Run all heavy PR model tests: native ATOM, vLLM, and SGLang
ci:atom Run native ATOM model accuracy tests
ci:vllm Run ATOM vLLM OOT model accuracy tests
ci:sglang Run ATOM SGLang model accuracy tests

Heavy jobs are skipped when the PR is not approved and no matching ci:* label is present.
Add labels via the sidebar or gh pr edit 3350 --add-label <label>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Shared GPU mailbox synchronization affects multiple model families, but the hardware arbitration mechanism is unconfirmed and validation is limited to gfx950 reproductions, leaving full-model correctness and performance across architectures for human GPU review.

No open issues 🎉

What changed in this PR

This PR adjusts ATOM’s shared Mono mailbox polling to address hangs observed when memory-copy kernels run concurrently on another GPU stream.

Changes:

  • Increases the sleep immediate between unsuccessful polls from 1 to 16, reducing repeated mailbox reads.
  • Leaves payload checks, cache policy, resource ownership, and completion handling unchanged.

The PR reports successful concurrent-copy validation on gfx950; full-model accuracy, end-to-end performance, and other GPU architectures remain unvalidated.

File Description
atom/​mono/​device/​sync.py Increases the sleep interval in _spin between unsuccessful mailbox polls.

Review effort: Balanced

@valar-cop valar-cop left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

some changes needed..

decode-step latency / ttft for the 1->16 backoff on a model using the shared mailbox?

ci green.

Comment thread atom/mono/device/sync.py Outdated
v = load_all()
while pending(v):
rocdl.s_sleep(1)
rocdl.s_sleep(16)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same loop still sleeps 1 in _spin_bounded, and the hi/qwen38/dsv4 mailbox forks spin with s_nop 0, tighter than the s_sleep(1) that hangs. apply the backoff there too or say why exempt?

Comment thread atom/mono/device/sync.py Outdated
v = load_all()
while pending(v):
rocdl.s_sleep(1)
rocdl.s_sleep(16)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

leave some comment on why 16, else someone simplifies it back to 1 and the hang returns..

Comment thread atom/mono/device/sync.py Outdated
v = load_all()
while pending(v):
rocdl.s_sleep(1)
rocdl.s_sleep(16)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no reproducer checked in? reference the command/topology or add a minimal harness so a revert can be verified.

@valarLip valarLip left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Action required before merge (5)

  1. Treat 1→16 as a mitigation, not a fix, unless you can show the mechanism. By the PR's own data, sleep(1) hung at about 1 round in 12. Zero hangs in 544 rounds at sleep(16) only bounds the rate below about 0.55% a round (95%), roughly 15x rarer. That is not "fixed" (probe_backoff.py). Mono already treats concurrent copy kernels as a wedge risk: see atom/mono/runtime/deployment.py:194-198, and runner.py:531 / hi_dispatch.py:105 join the side stream before launching. MiniMax-M3 (minimax_m3/mono/dispatch.py:51) and DSV4.1 non-hi (deepseek_v41/mono/config.py:34-79, which already refuses ATOM_DSV41_SIDE_STREAMS for this exact CU contention) never call resident_grid_refusals, so they admit any KV connector, including Mooncake Store, whose unpack runs on a side stream (mooncake_store/worker.py:871-917). Please either order the Store copy against the Mono launch, or add resident_grid_refusals(..., connectors=...) to those models. Otherwise, retitle the PR as "reduce hang rate" and keep the root-cause issue open.
  2. Apply the backoff at every poll site of this protocol, or justify each exemption (valar-cop). _spin_bounded at sync.py:55 still uses s_sleep(1). That is the debug build you would use to chase this hang, and it now polls 16x faster than production. The same goes for step_begin.py:105/129/135. The forks use spin_pause (s_nop 0, even tighter): hi/device/gfx950/mailbox.py:257, qwen38/.../mailbox.py:209, polls.py:52, hc.py, dual.py. Qwen3.8 is the one Mono model that explicitly admits a copy connector (qwen38/mono/config.py:36,149), and it never reaches sync._spin.
  3. Name and explain the 16 (valar-cop). Hoist it to one constant next to POLL_MAX (sync.py:20), used by _spin and _spin_bounded. Add a comment: about 1024 clocks (64-clock units), "s_sleep(1) hung with a concurrent copy kernel on gfx950, see #3350 repro; do not lower without re-running it".
  4. Check in or reference the reproducer (valar-cop). The gfx950 Mono plus HBM/KV-copy harness and its command or topology should be in tools/ or tests/, or at least linked, so a revert can be verified. Nothing in tests/ touches atom/mono/device/sync.py today.
  5. Post decode latency numbers (valar-cop). Every missed poll now waits about 1024 clocks instead of 64, which adds about 0.2 µs per critical-path hand-off. At a few hundred serial hand-offs a step that is tens to 100+ µs (probe_backoff.py). Please give before/after TPOT/TTFT on one model using the shared mailbox (MiniMax-M3 or Kimi-K3 mono).

Reviewed at c83d06d. CI: pre-checkin green; heavy model jobs not run (no label/approval). Full finding list in result.json.

@valar-cop

valar-cop commented Nov 14, 2026 •

Copy link
Copy Markdown

Copilot AI balanced review requested due to automatic review settings November 14, 2026 14:43

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

GLM-5.2’s launcher lacks a shared-source digest, allowing existing JIT caches to reuse the old binary and bypass the backoff change.

1 open issue

Review effort: Balanced

Comment thread atom/mono/device/ops.py Outdated
"""Keep tagged-mailbox polling loads inside the retry loop."""

llvm.InlineAsmOp(None, [], "s_nop 0", "", has_side_effects=True)
llvm.InlineAsmOp(None, [], f"s_sleep {POLL_SLEEP_UNITS}", "", has_side_effects=True)

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in 4fdec7239 (now in the rebased history): the GLM launcher captures the shared polling source digest (_ = source_key, which FlyDSL keys as a closure scalar), so a change to POLL_SLEEP_UNITS or the shared helpers changes the JIT key; a mutation check on the real digest and key implementation is described in the PR description.

@ZhangLirong-amd ZhangLirong-amd changed the title Fix Mono mailbox polling hangs during concurrent transfers Mitigate Mono mailbox polling stalls during concurrent transfers Nov 14, 2026
Copilot AI balanced review requested due to automatic review settings November 14, 2026 14:54

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

GLM-5.2’s launcher omits a shared-source digest, allowing an existing FlyDSL cache to reuse the old s_nop 0 binary and silently bypass the backoff mitigation.

1 open issue

Review effort: Balanced

Copilot AI balanced review requested due to automatic review settings November 14, 2026 15:08

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Shared GPU polling changes span several model families, but limited gfx950 runs and substitute V4.1 timings leave unvalidated paths and residual hang and performance risks requiring maintainer judgment.

1 open issue

Review effort: Balanced

@valarLip valarLip left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Action required before merge (3)

  1. Finish the "mitigation" path from r19 #1: open a root-cause issue and remove the safety claim your own repro disproves. The retitle and the residual-risk wording are fine, and the backoff can merge as a mitigation. Two things are still missing. First, there is no tracking issue: atom/mono/polling.py:6-7 points at this PR, which closes on merge. Open an issue (hardware root cause, plus the ordering or refusal of Mooncake Store copies next to resident-grid Mono) and cite it there. Second, resident_grid_refusals (atom/mono/runtime/deployment.py:186-188) still says allowed connectors' copy kernels "finish on their own". Your repro shows a copy kernel next to K1 wedging. Qwen3.8 admits lmcache_offload on that premise (qwen38/mono/config.py:36,149), and MiniMax-M3 (minimax_m3/mono/dispatch.py:51) and DSV4.1 non-hi (deepseek_v41/mono/config.py:34-79) still admit any connector. Reword the docstring and list all three models in the issue. The root-cause fix (Store ordering, or resident_grid_refusals(..., connectors=...) for M3 and V4.1 non-hi) can then land as a follow-up.
  2. Don't put the reproducer's side effects or required checks inside assert. In tools/mono_mailbox_copy_repro.py:87,89,91, hipExtStreamCreateWithCUMask, hipExtStreamGetCUMask and hipStreamGetFlags are called inside assert ... == 0, and so is hipStreamDestroy at :474. Under python -O these calls are skipped entirely (probe_assert_O.py). The streams are never created, and the GPU-count and gfx950 guards at :60,64 vanish too. This is the tool that polling.py says must be re-run before anyone lowers the constant. Use rc = ...; if rc: raise RuntimeError(...).
  3. Fix the stale comments (small).
    • spin_pause in mono/device/ops.py:592 and deepseek_v4/mono/kernels/primitives.py:46 still describes only load-hoisting. Say that it sleeps POLL_SLEEP_UNITS.
    • The repro comment at :205-206 says "Zero FP8 weights" (and has the typo "are0"), but :262,265 fill them with 0x38.
    • mono_mailbox_copy_repro.md:51-53 says a run "checks bytes and Q/ring". That is only true in --mode side.
    • GLM: use the repo idiom _ = source_key (as in step_begin.py:189 and qwen38 emit.py:221) rather than assert source_key.
    • Optional: add a CPU test that pins POLL_SLEEP_UNITS at every gfx950 poll site and mono/polling.py in each digest.

Re-reviewed at 4fdec72. Already fixed: #2 all gfx950 poll sites (_spin_bounded, step_begin, hi/qwen38/dsv4/GLM spin_pause, dsv4 poll) now back off; #3 a named POLL_SLEEP_UNITS with a "do not lower" comment, shared via atom/mono/polling.py (sync imports ops, so it can't live next to POLL_MAX); #4 reproducer checked in; #5 V4.1 non-hi shared-mailbox TPOT +0.54% (substitute for M3/K3, accepted); Copilot GLM cache key fixed. CI: pre-checkin green; heavy model jobs skipped (no label/approval). Full list in result.json.

@ZhangLirong-amd

Copy link
Copy Markdown
Collaborator Author

@valarLip thanks, all three items are addressed in 59439b3 (no kernel logic change).

1. Tracking issue and the "finish on their own" claim. Opened #3369 (hardware root cause open; Store/Mono ordering or refusal; repro commands; the mitigation and its limits). It lists the three models after I re-read the code: Qwen3.8 admits lmcache_offload via OFFLOAD_CONNECTORS; MiniMax-M3 (dispatch.py _config_refusal) and DSV4.1 non-hi (config_refusal) never call resident_grid_refusals, so any connector is admitted. polling.py now cites #3369. The resident_grid_refusals docstring now says a copy kernel beside these kernels can still wedge them and the backoff only mitigates, and I changed the same premise in the Qwen3.8 connector comment. The root-cause fix is left to a follow-up.

2. assert in the reproducer. Reproduced your -O finding before the change: on 4fdec72 with python -O ... --setup-only the first stream came back with mask [0,0,...] and the run died with TypeError: argument 5 must be int, not None; with two GPUs exposed it did not stop at the GPU-count guard either. Now all HIP calls use rc = ...; require(rc == 0, ...) and every guard/check (GPU count, gfx950, import path, byte and Q/ring checks, hipStreamDestroy) goes through require(), which raises RuntimeError instead of assert. After the change on gfx950: -O --setup-only and plain --setup-only both create both streams with full masks and exit 0; -O with two GPUs raises expose exactly one owned GPU; -O mono-alone x3, copy-alone x3 and side x8 complete and exit 0. No leftover process and VRAM 0% afterwards. I used one helper instead of repeating if rc: raise at each site; say so if you prefer the inline form.

3. Stale comments. spin_pause docstrings (mono/device/ops.py, deepseek_v4/.../primitives.py, and also the hi gfx950/ops.py copy, which had the same wording) now say they sleep POLL_SLEEP_UNITS. The weights comment says 0x38 and the typo is fixed. The .md now says Q/ring is checked when Mono runs and bytes when the copy runs, so both only in --mode side (confirmed by the mono-alone/copy-alone outputs). GLM uses _ = source_key; it is still a free variable of the launcher closure (co_freevars unchanged). Optional CPU test: not added. Pinning the constant at every poll site (hi, qwen38, dsv4 and shared forks use different emit forms) and digest membership cannot be done in about 10 lines without a fragile source-text match, and the reproducer rerun is the real check.

Verification: AST of the changed non-tool files equals 4fdec72 except the GLM line; Black 26.10.1 and Ruff pass on the touched files; CI-equivalent unit run on this tree vs the PR base (79149c7) has the same 3 failures (test_multiprocess_metrics and two test_benchmark_catalog cases, environment-related), no head-only failure. Not re-run: the 128-round side-stream stress and the V4.1 latency comparison (unchanged kernel code, so the earlier numbers still apply). GitHub CI on the new head is not yet seen.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Maintainer judgment is needed to accept inherited V4.1 evidence for unmeasured MiniMax-M3/Kimi-K3 paths, given hardware-sensitive synchronization changes across Mono families and unresolved concurrent-copy safety.

1 open issue

Review effort: Balanced

@valarLip valarLip left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Action required before merge (2)

  1. Rebase onto main and resolve the deployment.py conflict, then let Pre Checkin run on the new head. GitHub reports the PR as CONFLICTING. #2786 (Kimi-K3 mono behind a Mooncake P/D decode) rewrote resident_grid_refusals (atom/mono/runtime/deployment.py:185-207 on main): it added KVConnectorFactory.canonical_name and changed the check to name in connectors. Your docstring edit at :186-190 conflicts with it. Keep main's canonical_name logic and your new docstring. Because of the conflict, no Pre Checkin, ATOM, vLLM or SGLang run has started on 59439b3; the green runs are all on 4fdec72. Main's Kimi-K3 now passes _served_connectors(...), which admits Mooncake decode. Its receives are NIC writes, with no GPU copy kernel. Add one line to #3369 saying so, because the issue still says K3 "refuses every connector".
  2. Tidy the reproducer (small).
    • The tools/mono_mailbox_copy_repro.py:12-13 usage line omits PYTHONPATH="$PWD". Without it, the tool's own checkout-ownership check at :76 fails. Copy the command from the .md.
    • :207 inspect.signature(k1.build_attn_pre).bind(key) is redundant, because :208 calls build_attn_pre(key) directly. Drop it, and the str(inspect.signature(...)) field at :213.
    • .md:25-26: "Install/run from a full checkout" and "Use PYTHONPATH only for this checkout" read as two different procedures. Keep the single PYTHONPATH="$PWD" one, and document --deadline (default 15 s), which drives pending_deadline.

Re-reviewed at 59439b3. Already fixed: r20 #1 tracking issue #3369 opened and cited in polling.py:7, deployment.py:190 and qwen38/mono/config.py:36; resident_grid_refusals no longer says copy kernels "finish on their own"; #3369 lists Qwen3.8, MiniMax-M3 and V4.1 non-hi. #2: no assert left in the reproducer; every HIP call and guard goes through require(). #3: spin_pause docstrings (shared, dsv4 and hi) mention POLL_SLEEP_UNITS; weights comment and .md mode wording fixed; GLM uses _ = source_key, which FlyDSL keys as a closure str scalar. Every gfx950 poll site backs off, and every digest that inlines the constant covers mono/polling.py. The optional CPU pin test was declined with a reason; that's acceptable. Full list in result.json.

@ZhangLirong-amd
ZhangLirong-amd force-pushed the lirzhang/mono-mailbox-poll-backoff branch from 59439b3 to 79f5179 Compare November 16, 2026 01:21
Copilot AI balanced review requested due to automatic review settings November 16, 2026 01:21
The previous commit passed it unconditionally, which breaks the documented
rollback checkout (an older ABI without ring_scale).
…ckoff-revert control

Build the argument table from k1.ABI.names so an ABI change fails at setup, not
in the watchdog loop; look buffers up by name instead of keepers[...] indices;
drop the duplicate import-path check and code_identity record and the constant
True fields (now in the docstring); rewrite the rollback recipe in the .md as a
separate checkout with only the backoff reverted, without a literal SHA.
@ZhangLirong-amd
ZhangLirong-amd force-pushed the lirzhang/mono-mailbox-poll-backoff branch from 5376e56 to c87219b Compare November 16, 2026 08:28
Copilot AI balanced review requested due to automatic review settings November 16, 2026 08:28

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

GPU maintainer judgment is needed because the cross-model polling mitigation leaves unresolved liveness risk, with finite validation focused on V4.1 and unmeasured MiniMax-M3/Kimi-K3 impact.

1 open issue

Review effort: Balanced

@valarLip valarLip left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Action required before merge (2)

  1. Fix the two stale doc references in the rollback recipe and the helpers comment (small, non-blocking). tools/mono_mailbox_copy_repro.md (rollback item 3) says the two rocdl.s_sleep(polling.POLL_SLEEP_UNITS) sites in deepseek_v4/mono/kernels/helpers.py are "at lines 179 and 198". At this head they are at 180 and 199, because the comment above SHARED_SOURCE_KEY grew by one line. Drop the line numbers and name the function, or fix them. The same comment (helpers.py:54-56) says "editing this file, ops.py or mono/polling.py recompiles it", but ops here is the alias for primitives (helpers.py:28). The file it hashes is deepseek_v4/mono/kernels/primitives.py, so write primitives.py.
  2. Get the heavy accuracy jobs to finish green at this head. Pre Checkin (non-GPU unit tests) now passes, and Black, Ruff, smoke, catalog and the vLLM/SGLang/ATOMesh gates pass. The ATOM Accuracy matrix (Kimi-K3/K3.1, DeepSeek-V4/V4.1/R1, GLM-5.3) is still pending at c87219b70. This PR changes polling in every gfx950 resident kernel, so those jobs are the only model-level check. Merge after they pass.

Re-reviewed at c87219b (full diff; this force-push is a clean rebase onto main 0819cb363, with the same per-file diff as 5376e56). Already fixed: r29 #1: atom/mono/polling.py:5-17 records the sweep (1 and 4 at 0/6; 8 at 44/44 for 128 rounds but 3/4 for 1024 rounds, hang at round 583; 16 at 44/44 and 12/12; 128 rounds cannot tell 8 from 16; +0.5% TPOT measured at 16 only). The "before lowering" rule is now --repeats 1024, timeout >= 300s, at least 12 runs alongside 16, any hang rejects, in polling.py, the .md, #3369 and the description ("Why 16"). r29 #2: the .md recipe now lists all ten changed sites (polling.py for sync.py x2 and step_begin.py x3, three spin_pause, two helpers.py sites back to rocdl.s_nop(0)), and the control built from it hung 6/6. The helpers.py comment names mono/polling.py. Every gfx950 poll site still backs off and every kernel digest still covers mono/polling.py. The gfx1250 paths and the harness sleep_ticks are untouched, as intended.

…k recipe; primitives.py in the helpers digest comment
@ZhangLirong-amd

Copy link
Copy Markdown
Collaborator Author

@valarLip thanks. Both stale references are fixed in cf01817 (comments and docs only; an AST comparison of helpers.py against c87219b shows no change outside comments).

  1. Rollback recipe and helpers comment. I took your first option and dropped the line numbers: tools/mono_mailbox_copy_repro.md item 3 now names the function, poll() in deepseek_v4/mono/kernels/helpers.py (the bounded loop under poll_timeout_ticks and the unbounded loop in the else branch), so the next edit above it cannot make it stale again. The SHARED_SOURCE_KEY comment in helpers.py now says primitives.py, the file ops aliases there.
  2. Heavy accuracy jobs. Nothing to change on my side; this push re-queues the Accuracy matrix on the new head, so I will not promise a completion time. I will report if any job fails and whether it looks like an infrastructure problem.

Copilot AI balanced review requested due to automatic review settings November 16, 2026 12:36

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Maintainer judgment is needed on residual GPU liveness risk and cross-model coverage, since shared polling changes have V4.1-focused runtime evidence but no MiniMax-M3/Kimi-K3 measurements.

1 open issue

Review effort: Balanced

@valarLip valarLip left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Action required before merge (1)

  1. Let the ATOM Accuracy matrix finish green at this head, then merge. The code is done. Pre Checkin (non-GPU unit tests), Black, Ruff, smoke, catalog and the vLLM/SGLang/ATOMesh gates pass at cf01817f9. The 14 heavy Accuracy jobs (Kimi-K3/K3.1, DeepSeek-V4/V4.1/R1, GLM-5.3, MiniMax-M3, Qwen3.5/Next, gpt-oss) are still pending. This PR changes polling in every gfx950 resident kernel, so those jobs are the only model-level check. If one fails, say whether it also fails on main (for example the DeepSeek-V4-Pro DSpark "VllmBackend can only be called once").

Re-reviewed at cf01817 (full diff, 16 files; this push only edits comments and docs). Already fixed: the c87219b review's #1, tools/mono_mailbox_copy_repro.md rollback item 3 now names poll() in deepseek_v4/mono/kernels/helpers.py (bounded loop under poll_timeout_ticks and the else loop, now at :180,:199) with no line numbers. The SHARED_SOURCE_KEY comment (helpers.py:55) says primitives.py. The recipe still restores exactly main's instructions at all ten changed sites. Every gfx950 poll site sleeps POLL_SLEEP_UNITS (sync.py:30,56, step_begin.py:106,130,136, three spin_pause, helpers.py:180,199). Every kernel digest covers mono/polling.py. sleep_ticks and gfx1250 are untouched, as intended. The "Why 16" evidence and the 1024-round rule are in polling.py:5-17. Copilot's GLM cache-key thread is fixed (glm52/mono/kernels/kernel.py source_digest(..., "mono/polling.py") + _ = source_key), so the thread can be resolved.

…efault 16)

Unset or empty keeps 16. 1 restores main's polling instructions exactly
(s_sleep 1 in the shared waits, s_nop 0 in spin_pause and the V4 helpers'
poll) for a deployment without concurrent KV copies. Values outside 1..127
are an error: gfx950 sleeps on bits [6:0] of the s_sleep immediate. The
value is read at import and folded into every source digest that hashes
polling.py, so different values never share a JIT cache entry. Config
rejects a bad value before the workers spawn and warns once below 16.
Copilot AI balanced review requested due to automatic review settings November 16, 2026 13:42
@ZhangLirong-amd

Copy link
Copy Markdown
Collaborator Author

@valarLip one small addition on top of the version you reviewed, an operator knob asked for on our side: ATOM_MONO_POLL_SLEEP_UNITS (306adb8, docs in ae000a1). The default does not change: unset or empty means 16, the behaviour you LGTM'd.

  • Why. If a deployment without concurrent KV copies (no offload or P/D connector) sees the +0.5% TPOT cost, it can go back to main's polling without a code change.
  • What 1 means. ATOM_MONO_POLL_SLEEP_UNITS=1 restores main's instructions exactly, not just s_sleep 1: s_sleep 1 in sync.py / step_begin.py, s_nop 0 in the three spin_pause and in the V4 helpers' poll().
  • Range. Integers 1..127 only; anything else raises at startup. On gfx950 s_sleep sleeps for about 64 x SIMM16[6:0] clocks, so 128 assembles but sleeps as 0, and 0 would be no backoff.
  • Cache key. The value is read once at import in polling.py (still dependency-free); it is not in that file's bytes, so the three source digests that hash polling.py also hash the value. Builds with different values never share a JIT cache entry.
  • Guard. No refusal, only a one-time startup warning from Config when the value is below 16, pointing at Mono mailbox polling can stall next to concurrent copy kernels on gfx950 (root cause open; backoff in #3350 is a mitigation) #3369 (1 and 4 hung beside concurrent copies, 8 hung once in a 1024-round run).

Evidence (CPU compile-only on gfx950, no GPU run): the V4.1 K1 code object with the variable unset is byte-identical to cf01817f9 (sha 04c2367b), and with =1 byte-identical to main (sha a1397f51); the same holds for the spin_pause and step_begin probes. New tests (tests/test_mono_poll_sleep_units.py) cover default, 1, invalid values and distinct digests. The CI-equivalent CPU suite has the same failure list on this head and on its base (test_atomesh_spur_dispatch 15 s subprocess timeouts under node load, and test_multiprocess_metrics), with 19 more passes from the new tests. Ruff and Black are clean.

This push re-queues the heavy Accuracy matrix, so I will not promise a time. Could you confirm the knob is fine?

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Polling changes span several Mono GPU families, but the hardware cause remains unresolved and MiniMax-M3/Kimi-K3 lack model-level validation, requiring maintainer judgment on coverage gaps and residual concurrent-copy risk.

1 open issue

Review effort: Balanced

@valarLip valarLip left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Action required before merge (2)

  1. Drop ATOM_MONO_POLL_SLEEP_UNITS and keep POLL_SLEEP_UNITS = 16 as a plain constant (the cf01817f9 code). We prefer one good default to a new env knob, and this one has no evidence behind it:
    • No proven need. No deployment is named, and nobody has shown that the +0.54% TPOT matters anywhere.
    • It reopens the hang. The values it unlocks below 16 are the ones that hung in your own sweep: 1 and 4 in every run, 8 once. Setting 1 brings back the #3369 hang with only a warning (an offload/P/D connector could be added later without anyone revisiting the variable).
    • It makes cache keys fragile. Today hashing polling.py is enough. With the knob, CACHE_TAG has to be added by hand at three digests (the f.name == "polling.py" match in build_key.source_digest, helpers.SHARED_SOURCE_KEY, the hi source_digest), and any future digest that forgets it silently reuses a kernel built for another value. All of them are correct at this head, so this is not a bug yet.
    • It costs about 200 lines. That covers _units/MAX, the MAIN_POLLING split and _retry_pause, the Config warning, an envs.py entry that nothing reads, the docs row (which also calls this PR "before #3350"), the 89-line test file, and a getattr(sync, "POLL_SLEEP_UNITS", None) probe in the reproducer.
      Rollback is git revert. The reproducer control can go back to the separate-checkout recipe. If a real deployment needs a lower value, change the default in a PR that brings its TPOT at 16 vs the candidate and the 1024-round sweep.
  2. Let the full Accuracy matrix pass at the final head. At ae000a144, 13/14 Accuracy jobs pass and Kimi-K3.1-Pro DSpark is still pending. Removing the knob changes kernel sources, so the matrix reruns. Pre Checkin, Black, Ruff and the vLLM/SGLang/ATOMesh gates pass.

Re-reviewed at ae000a1 (full diff, 21 files). Already fixed: everything from earlier rounds still holds. Every gfx950 poll site backs off by POLL_SLEEP_UNITS (sync.py:30,56, step_begin.py:106,130,136, three spin_pause, both polls in helpers.py::poll()). Every Mono kernel digest, the GLM launcher's included, covers polling.py. sleep_ticks and gfx1250 are untouched. "Why 16" and the 1024-round rule are in polling.py. The reproducer and #3369 are in place, and Copilot's GLM cache-key thread is fixed. The new tests pass on CPU.

@valarLip valarLip removed the LGTM waiting pass for CI label Nov 16, 2026

@valar-cop valar-cop left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

some changes needed..

ci green.

Comment thread atom/mono/polling.py
# the V4 helpers' poll), for a regression on a deployment with no concurrent
# KV copies (no offload or P/D connector). Below 16 beside such copies, 1 and 4
# hung and 8 hung once (above); `Config` warns once at startup.
POLL_SLEEP_UNITS = _units(os.environ.get(ENV))

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

let's drop this knob and keep the plain 16, no env flag if it can just be on by default

@ZhangLirong-amd

Copy link
Copy Markdown
Collaborator Author

@valarLip thanks for the detailed review. On item 1 (dropping ATOM_MONO_POLL_SLEEP_UNITS): we discussed the environment variable offline and agreed to keep it, so I am leaving it in this PR and not reverting 306adb8a7 / ae000a144.

For the record, nothing changes for the default: unset or empty is still POLL_SLEEP_UNITS = 16, the behaviour you LGTM'd at cf01817f9, and the code object with the variable unset is byte-identical to that version.

On item 2: agreed, the full Accuracy matrix has to pass at the final head, and I will not merge before it does. At ae000a144 the last report was 13/14 with Kimi-K3.1-Pro DSpark pending; I will report if any job fails and whether it also fails on main.

@valarLip valarLip left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ LGTM — correction to my previous review

I withdraw item 1 of my last review ("drop ATOM_MONO_POLL_SLEEP_UNITS"). Keeping the variable configurable with a default of 16 is what we agreed with you on 11-16, and the cache keys are correct at this head (the value is in all three digests). Sorry for the noise.

Item 2 is done: at ae000a1448 all checks pass, including the full Accuracy matrix (44 pass, 0 fail).

Optional cleanups, not blocking: the envs.py entry that nothing reads; the getattr(sync, "POLL_SLEEP_UNITS", None) in tools/mono_mailbox_copy_repro.py (read the attribute directly); the docs row that calls this PR "before #3350"; and the Config warning that says every value below 16 hung (only 1, 4 and 8 were tested).


Re-reviewed at ae000a1.

@valarLip valarLip added the LGTM waiting pass for CI label Nov 17, 2026

@valarLip valarLip left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved at ae000a1. CI is green on this head, including the full Accuracy matrix.

@valarLip
valarLip merged commit 53402eb into main Nov 17, 2026
190 checks passed
@valarLip
valarLip deleted the lirzhang/mono-mailbox-poll-backoff branch November 17, 2026 03:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci:atom LGTM waiting pass for CI

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants