[CI Monitor] Daily Report - 2026-07-19

[amd-bot]

Daily Cross-Workflow Summary — 2026-07-19

Snapshot: 2026-07-19 22:59 UTC · Only completed runs/jobs counted in trends · Auto-updated every 30 min

TL;DR

🔴 RED · 7 active clusters (+ large known-stable set) · 1 🆕 today (R279, has in-flight fix) · 1 ✅ RECOVERED today (R274, fix merged) · 5 carrying over · in-flight fixes: R279→#31663 (open), R257→#31068 (open); R270 still has NO PR (4th day)
👉 Today's ask: (1) Open a PR for R270 — mamba transfer_kv_mamba_lf_pf NameError is 4 days old with zero in-flight fix (search returned 0), red on both nightly kernel jobs every run. (2) Chase reviewers on R279 (#31663, Qwen3-VL DP-encoder CPU all_gather crash, NEW today). (3) No action on R274 — #31675 merged 08:40 UTC and the post-merge pr-test-amd run confirms stage-c-8gpu-mi35x is green again.

Important

The Jul-19 nightly runs were still in-progress at snapshot and are excluded from trends: nightly-test-amd 29697529176 (IN-FLIGHT), nightly-test-amd-rocm720 29697452589 (IN-FLIGHT), pr-test-amd-rocm720 29697501385 (IN-FLIGHT). Their individually-concluded legs are reported with (IN-FLIGHT). Latest completed nightly runs are Jul-18. Improvements today: nightly-amd-mi355x-disagg Jul-19 29673297044 is ✅ GREEN (was ❌ yesterday → R241/R273/R276 not re-exercised), and both release-docker workflows are ✅ green. AMD AITER Scout did NOT run (last Jul-13) → R266/R267 dormant.

Workflow status

Workflow	Latest completed run	❌ jobs	Today's status	Δ vs yesterday
nightly-test-amd	Jul-18 29654601256 (failure)	6	Jul-19 29697529176 (IN-FLIGHT)	R270 + R257 persist
nightly-test-amd-rocm720	Jul-18 29654521083 (failure)	19	Jul-19 29697452589 (IN-FLIGHT)	🆕 R279; R270/R257/R262/R273 carry over
release-docker-amd-nightly	Jul-19 29686913858 ✅	0	green	✅ still green
release-docker-amd-rocm720-nightly	Jul-19 29686758040 ✅	0	green	✅ still green
nightly-amd-mi355x-disagg	Jul-19 29673297044 ✅	0	green	🟢 recovered (was ❌; R241/R273/R276 not exercised)
amd-aiter-scout	— (last Jul-13)	—	did not run	dormant
pr-test-amd	Jul-19 29686709871 (failure)	6 (post-fix)	completed ❌	✅ R274 recovered; R277 persists
pr-test-amd-rocm720	Jul-18 29654571916 (failure)	10	Jul-19 29697501385 (IN-FLIGHT)	R278 + known set persist

7d run-conclusion trend (older→newer, completed scheduled runs; ⚪=cancelled): nightly-test-amd ❌·⚪·❌·❌·❌·(IN-FLIGHT) · nightly-rocm720 ❌·⚪·❌·❌·❌·(IN-FLIGHT) · mi355x ⚪·❌·❌·❌·❌·✅ · pr-test-amd ❌·❌·❌·❌·❌·❌ · pr-test-amd-rocm720 ❌·❌·❌·❌·❌·(IN-FLIGHT) · release-docker-amd ❌·❌·✅·✅·✅·✅ · release-docker-rocm720 ❌·❌·✅·✅·✅·✅.

Failure clusters (deduplicated across all workflows)

R270 · NameError: transfer_kv_mamba_lf_pf/_pf_lf — mamba JIT transfer kernel CUDA-only import guard on AMD — nightly-amd + nightly-rocm720 kernel · ❌ none (4th day)

• Status: carry-over, 4th day (first Jul-16), never-passed on AMD. Red on every completed AMD/ROCm nightly kernel run (12 NameError subtests each). memory_pool_host.py imports the symbols under if _is_cuda: only.
• Top hypothesis: [HIGH] #30535 (b296e1a5) added a CUDA-only import guard with no HIP branch. Disconfirming: kernel may be genuinely CUDA-only, but sibling .cuh transfer kernels are HIP-enabled.
• In-flight fix: ❌ none (transfer_kv_mamba open-PR search → 0 results). Suggested triage: load_jit the mamba kernel in an MI325 container; if it builds on HIP → widen the import to _is_cuda or _is_hip; else drop register_amd_ci for this test on AMD.

Workflow	Job (shard)	Test File	Test Function	Error	Log
nightly-test-amd	nightly-test-1-gpu-kernel (IN-FLIGHT)	test/registered/jit/test_transfer_mamba.py	roundtrip (12 subtests)	NameError: transfer_kv_mamba_lf_pf → exit 255	Log
nightly-test-amd-rocm720	nightly-test-1-gpu-kernel-rocm720 (IN-FLIGHT)	test_transfer_mamba.py	same	same	Log
nightly-test-amd-rocm720	nightly-test-1-gpu-kernel-rocm720	test_transfer_mamba.py	same	same (Jul-18 completed)	Log

R257 · MI35x (gfx950) GPU hang / HW-exception family — hipErrorCapturedEvent · GPU memory access fault · watchdog SIGABRT during graph-capture / warmup / decode — nightly-amd + nightly-rocm720 MI35x · ⚠️ #31068 (open) + #30431

• Status: carry-over, 11th day (first Jul-09/10). Model- and workflow-independent; alternates pass/fail on the same physical nodes. Today re-exercised across qwen3-235b (hipErrorCapturedEvent), qwen3-30b-a3b (memory access fault), deepseek-v4-pro-mtp (HW hang), deepseek-v32-mtp (watchdog 1200s), and glm51 server-launch legs.
• Top hypothesis: [MEDIUM] MI35x node/driver GPU hang / non-capturable all-reduce on capturing stream (aiter custom all-reduce / RCCL watchdog hipEventQuery colliding with HIP graph capture) + zombie-KFD VRAM leak. Disconfirming: alternates on identical SHAs → environment over a bisectable code regression.
• In-flight fix: ⚠️ #31068 (MI35x hung-GPU pre-flight recovery, open). Suggested triage: chase #31068; A/B pre/post base-image bump at a fixed SHA; else reboot/escalate the MI35x pool.

Workflow	Job (shard)	Test File	Test Function	Error	Log
nightly-test-amd	nightly-8-gpu-mi35x-qwen3-235b-mxfp4 (IN-FLIGHT)	test_qwen3_instruct_mxfp4.py	decode graph capture	hipErrorCapturedEvent → SIGABRT	Log
nightly-test-amd-rocm720	nightly-accuracy-8-gpu-mi35x-rocm720 (IN-FLIGHT)	test_qwen3_moe_eval_mi35x.py	Qwen3-30B-A3B prefill	GPU memory access fault	Log
nightly-test-amd-rocm720	nightly-8-gpu-mi35x-deepseek-v4-pro-mtp-rocm720	test_deepseek_v4_pro_fp4_mtp.py	fp4-MoE + EAGLE MTP	GPU HW hang / memory fault	Log
nightly-test-amd-rocm720	nightly-accuracy-8-gpu-mi35x-deepseek-v32-mtp-rocm720	test_deepseek_v32_*_mtp*.py	DSA EAGLE-MTP draft-extend	watchdog 1200s hang → exit	Log
nightly-test-amd	nightly-2-gpu-mi35x-glm51-mxfp4	test_glm51_mxfp4_tp2_gsm8k_mi35x.py	setUpClass	GLM-5.1 server never healthy (GPU hang)	Log

R279 · 🆕 · Qwen3-VL DP-encoder all_gather of a CPU tensor on the RCCL device-group → RuntimeError: No backend type associated with device type cpu — nightly-rocm720 4-GPU · ✅ #31663 (open)

• Status: NEW today (first seen 2026-07-19). Signature masked by random.choice(MODELS) in test_encoder_dp.py:34 — Jul-16/17/18 "passed" only because they drew Qwen2.5-VL-72B; Qwen3-VL-32B has no green run in the Jul-3→19 window. All 4 TP ranks crash on first multimodal forward → SIGQUIT → 3600s harness timeout.
• Top hypothesis: [HIGH] on AMD hosts is_shm_available() is False (no Intel AMX / not arm64), so parallel_state.py:1118-1123 takes the input_.is_cpu branch and calls all_gather_into_tensor(group=device_group) on a CPU-resident empty/embedding tensor; the RCCL device group has no CPU backend. mm_feature_transport='cpu' on AMD keeps mm embeddings on CPU. Disconfirming: #26759 reported the identical error on NPU back on 2026-05-30 → the device/CPU-group mismatch predates any July change; the exact regressing commit is unpinned (shallow clone), so mechanism is HIGH but "regressed Jul-19" is not.
• In-flight fix: ✅ #31663 ("Place empty Qwen encoder-DP embeddings on the communication device", open, updated Jul-19) — most precise match; also #26759 / #31596. Suggested triage: cherry-pick #31663 onto d4801be4, re-run test_encoder_dp.py with model pinned to Qwen3-VL-32B; separately, pin/parametrize the model draw so failures aren't masked.

Workflow	Job (shard)	Test File	Test Function	Error	Log
nightly-test-amd-rocm720	nightly-4-gpu-rocm720 (IN-FLIGHT)	test/registered/vlm/test_encoder_dp.py	TestVLMEncoderDP.test_vlm_mmmu_benchmark (Qwen3-VL-32B)	RuntimeError: No backend type … cpu → -9 → timeout	Log

R262 · InternVL2_5-2B server crash — AttributeError: CLIPImageProcessor has no 'tokenizer' — nightly-rocm720 VLM · ❌ none

• Status: carry-over, recurring since Jul-14 (last passed Jul-05). AutoProcessor resolves to a bare CLIPImageProcessor; get_tokenizer_from_processor dereferences a missing .tokenizer, aborting the VLM MMMU suite.
• Top hypothesis: [MEDIUM] image-only-processor handling. Disconfirming: no in-window sglang commit touches the processor path → possibly upstream HF remote-code. In-flight fix: ❌ none. Suggested triage: repro InternVL2_5-2B alone; inspect the image-only-processor path in get_tokenizer_from_processor.

Workflow	Job (shard)	Test File	Test Function	Error	Log
nightly-test-amd-rocm720	nightly-accuracy-2-gpu-vlm-rocm720 (IN-FLIGHT)	test_vlms_mmmu_eval_amd.py	test_mmmu_vlm_models (InternVL2_5-2B)	AttributeError: 'CLIPImageProcessor'…'tokenizer' → exit 1	Log
nightly-test-amd-rocm720	nightly-accuracy-2-gpu-vlm-rocm720	test_vlms_mmmu_eval_amd.py	same	same (Jul-18 completed)	Log

R277 · Mooncake KV-cache transfer failure in PD-disaggregation (Failed to get kvcache from prefill instance / 30s sync timeout → KeyError: 'answer') — pr-test-amd disagg · ❌ none (intermittent)

• Status: carry-over, intermittent on stage-b-8gpu-mi35x-disaggregation. Present in both Jul-19 pr-test-amd runs (the sole mi35x failure remaining after R274's fix). Top hypothesis: [LOW] transient Mooncake transfer-engine/decode-declares-prefill-dead race. In-flight fix: ❌ none. Suggested triage: retry + monitor; if reproducible, capture prefill-instance liveness + Mooncake transfer logs at the 30s timeout boundary.

Workflow	Job (shard)	Test File	Test Function	Error	Log
pr-test-amd	stage-b-8gpu-mi35x-disagg	PD-disagg GSM8K eval	KeyError: 'answer'	Failed to get kvcache from prefill instance	Log
pr-test-amd	stage-b-8gpu-mi35x-disagg	PD-disagg GSM8K eval	same	30s sync timeout (decode declares prefill dead)	Log

R278 · PD+PP disaggregation dropped/empty generation → KeyError: 'answer' — pr-test-amd-rocm720 disagg · ❌ none

• Status: carry-over (first Jul-18). PD+PP decode output path drops a generation, GSM8K scorer keys on missing 'answer'. Top hypothesis: [LOW] empty/aborted generation in PP-decode output path. In-flight fix: ❌ none. Suggested triage: inspect the PP-decode output assembly path for dropped sequences.

Workflow	Job (shard)	Test File	Test Function	Error	Log
pr-test-amd-rocm720	stage-b-8gpu-mi35x-disagg-rocm720 (IN-FLIGHT)	PD+PP GSM8K eval	KeyError: 'answer'	dropped/aborted generation	Log

R273 · DeepSeek-V4-Flash FP4 GSM8K marginal miss at strict >0.91 gate — nightly-rocm720 · ❌ none (boundary)

• Status: carry-over, accuracy oscillates around a strict >0.91 gate; servers healthy. On mi355x today the config was not exercised (mi355x Jul-19 run green). Top hypothesis: [HIGH] boundary variance around a strict > gate. Suggested triage: re-run 2–3× at same SHA; consider a small gate tolerance.

Workflow	Job (shard)	Test File	Test Function	Error	Log
nightly-test-amd-rocm720	nightly-8-gpu-mi35x-deepseek-v4-flash-rocm720	test_deepseek_v4_flash_*	GSM8K gate	fp4 accuracy just below 0.91	Log

✅ Recovered / not-exercised today (no action) · click to expand

• R274 ✅ RECOVERED — tc_piecewise/eager MLA prefill q.view shape mismatch on MI35x 8-GPU. Fix #31675 ("[AMD] Fix DeepSeek MLA prefill shape mismatch on HIP eager fallback (missing mha_companion_layers)", root-cause #31391) merged 2026-07-19 08:40 UTC. Confirmed: post-merge pr-test-amd run 29686709871 (12:17) shows stage-c-8gpu-mi35x (0) and (1) passing. The Jul-19 00:27 pre-fix run 29666971143 failures (88138988386, 88138988403) and Jul-18 rocm720 legs are stale pre-fix.
• R241 (Kimi-K2.6 FP8 non-MTP disagg GSM8K gate) and R276 (dsv4flash-fp4 conc64 bench hang): not exercised today — nightly-amd-mi355x-disagg Jul-19 29673297044 passed green.
• R275 (chunked input-logprob assertion → test_srt_endpoint crash): did NOT recur in either completed Jul-19 pr-test-amd run (29666971143 + 29686709871); only the Jul-18 00:25 run 29623067212 showed it → treat as flaky/intermittent (still ❌ no fix; watch).

Known stable / recurring clusters (carrying over, no new action today) · click to expand

Cluster	Workflow · Rep. job	Test / fn	Status	In-flight fix
gsm8k accuracy: Mixtral-8x7B below threshold + Qwen3-4B missing-threshold config gap	nightly-amd accuracy-2-gpu + rocm720	test_gsm8k_eval_amd.py	recurring (scorer/config)	⚠️ #29933 (partial)
VLM MMMU accuracy ~chance / encoder-DP timeout on AMD (Qwen2.5/3-VL, GLM-4.1V)	nightly-amd vlm + nightly-4-gpu GLM timeout	test_vlms_mmmu_eval_amd.py / test_encoder_dp.py	never-passed ≥3wk	❌
DSA topk_v2 JIT cooperative_groups.h not found (DSV4/GLM-5.x) on gfx950	nightly-rocm720 glm51-mxfp4-rocm720	test_glm51_* / DSV4 topk_v2	recurring/widespread	✅ #31291 / #30506 (open)
DeepSeek-R1-MXFP4 TP4 MTP GSM8K accuracy below threshold	nightly-rocm720 deepseek-r1-mxfp4-tp4-rocm720	test_deepseek_r1_mxfp4_tp4_mtp_mi35x.py	never-passed	❌
DeepSeek-V3.2 basic GSM8K ~0.86–0.92 < 0.93 on MI35x (DSA/tilelang)	nightly-rocm720 deepseek-v32-rocm720	test_deepseek_v32_eval_mi35x.py	never-passed	❌
Diffusion update_weights_from_disk HTTP 500 (inference-tensor inplace)	pr-test multimodal-gen 1gpu + rocm720	test_update_weights_from_disk.py	never-passed since Jun-27	❌
ZImage disagg encoder numpy-truthiness ValueError → warmup 500	pr-test multimodal-gen 2gpu + rocm720	test_disagg_server.py	never-passed since Jul-10	⚠️ #31325 (blocked)
FLUX.2 modelopt-FP8 _scaled_mm HIPBLAS_NOT_SUPPORTED gfx942	pr-test mm-gen 2gpu + rocm720	test_server_2_gpu.py	recurring	✅ #28889 (blocked)
ModuleNotFoundError: msgpack (diffusion / pi05 vla)	pr-test-rocm720 mm-gen-unit-rocm720	test_pi05_action_api.py	never-passed since Jul-11	❌
Triton offline throughput below AMD gate (<3500)	pr-test-rocm720 stage-b-1gpu-large-rocm720	test_bench_serving_1gpu_part1.py	recurring/never-passed	❌
Diffusion 1-GPU shard exceeds 60-min timeout / runner-lost	pr-test + rocm720 ×many	test_ar_models.py / mm-gen	recurring/infra	❌ (issue #27937)
Kimi-K2.5-MXFP4 8-GPU weight-load monitored_barrier timeout / OOM	pr-test stage-c-8gpu-mi35x (0) + rocm720	Kimi-K2.5-MXFP4 TP4	recurring OOM/hang	❌
Self-hosted MI325/MI35x runner lost comm / container-start hang	nightly + pr-test ×several (N/A infra)	mixed	intermittent (heavy today)	❌ (issue #27937)
GPU out-of-resources (0 MB) during DeepSeek-V3.2 GSM8K decode 8-GPU	pr-test stage-c-8gpu (1)	DSV3.2 decode	intermittent	❌

Workflow drill-down (latest completed / concluded legs per workflow) · click to expand

nightly-test-amd · latest completed Jul-18 29654601256 (6): R270 (kernel), R257 (glm51, qwen3-235b), VLM MMMU, Mixtral+Qwen3-4B gsm8k, runner-lost (1-gpu-unit). Jul-19 29697529176 (IN-FLIGHT) concluded legs: R270 (kernel), R257 (glm51, qwen3-235b hipErrorCapturedEvent), nightly-4-gpu GLM-4.1V encoder-DP 3600s timeout, accuracy-2-gpu, deepseek-r1-mxfp4-tp4 MTP accuracy, vlm.

nightly-test-amd-rocm720 · latest completed Jul-18 29654521083 (19): R270, R257 (qwen3-235b/qwen3-30b/deepseek-v4-pro/deepseek-v32-mtp/glm51), R262 (InternVL), R273 (deepseek-v4-flash fp4), DSV4 topk_v2 JIT, + accuracy shortfalls (r1-mxfp4-mtp, v32, v32-mtp), Mixtral gsm8k. Jul-19 29697452589 (IN-FLIGHT) concluded legs: 🆕 R279 (Qwen3-VL CPU all_gather), R270, R257 (qwen3-30b memfault, glm51), R262, deepseek-v32, accuracy-2-gpu.

pr-test-amd · Jul-19 29666971143 (00:27, pre-fix) — R274 (×2, now recovered), R277 (mooncake), ZImage/diffusion/FLUX known. Jul-19 29686709871 (12:17, post-fix) — R274 gone; R277 (mooncake disagg) + diffusion/FLUX known only.

pr-test-amd-rocm720 · latest completed Jul-18 29654571916 (10) — R274 (×2, pre-fix), msgpack, triton-throughput, ZImage, diffusion timeouts. Jul-19 29697501385 (IN-FLIGHT) concluded legs: R278 (PD+PP KeyError), msgpack, triton-throughput, diffusion 60-min timeouts, container-start hang.

nightly-amd-mi355x-disagg · Jul-19 29673297044 ✅ GREEN — R241/R273/R276 not exercised.

release-docker-amd-nightly / release-docker-amd-rocm720-nightly · Jul-19 ✅ green.

How this report is generated

• Only status == "completed" runs/jobs counted in trends. Jul-19 nightly-test-amd 29697529176, nightly-test-amd-rocm720 29697452589, and pr-test-amd-rocm720 29697501385 were queued/in_progress at snapshot → labelled (IN-FLIGHT), excluded from trends; their concluded legs are reported for context.
• 🆕 today: R279 (Qwen3-VL DP-encoder CPU all_gather crash, [HIGH] mechanism, ✅ #31663). ✅ recovered today: R274 via merged #31675 (verified against post-merge run). Improved: mi355x-disagg + both release-docker green.
• Confidence: FACT/HIGH/MEDIUM/LOW/SPECULATION. Bot does NOT assign Priority — engineers decide from cluster size + persistence + fix availability.

---

Generated by amd-bot · last updated 2026-07-19 22:59 UTC

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-07-19 22:59 UTC)

---

CI Monitor — 2026-07-19

Repo: sgl-project/sglang

Monitored Workflows:

• nightly-test-amd.yml
• nightly-test-amd-rocm720.yml
• release-docker-amd-nightly.yml
• release-docker-amd-rocm720-nightly.yml
• nightly-amd-mi355x-disagg.yml
• amd-aiter-scout.yml
• pr-test-amd.yml
• pr-test-amd-rocm720.yml

Per-workflow failure reports are appended as comments below; the Daily Cross-Workflow Summary is rendered above this section.

[amd-bot]

nightly-amd-mi355x-disagg.yml — 2 failure(s)

Run started (UTC): 2026-07-18 07:28 UTC
Last scanned (UTC): 2026-07-19 00:26 UTC
Commits: sglang 44e4999

Summary

Counts: 2 failures · 2 clusters · 0 carrying over (≥3 days) · 1 new today

#	Cluster	Job	Test File	Test Function	Status	Hypothesis (confidence)
1	DSV4-Flash-FP8 dp8ep8 GSM8K accuracy below 0.91 gate	nightly-mi355x-benchmark (dsv4flash-fp8-1k1k-1p1d-dp8ep8)	benchmark harness (GSM8K gate)	dsv4flash-fp8-…-dp8ep8 GSM8K gate	Intermittent — failed 07-12 & 07-18, 2/6 runs	[MEDIUM] #29275 FP8 scale layout — tracked in #31490
2	PD-disagg KV-transfer stall at high concurrency	nightly-mi355x-benchmark (dsv4flash-fp4-1k1k-1p1d)	N/A (benchmark job)	dsv4flash-fp4-1k1k-1p1d conc=64 sweep	New today (07-18) after 3 passes	[MEDIUM] #31368 blocking wait_event.synchronize() on mori path

Both jobs ran on sglang 44e4999a, run 29630291750; aiter N/A (pre-built MI355X image, no [CI-AITER-CHECK] markers). The two failures are unrelated (distinct configs, distinct symptoms) — no shared cluster.

Cluster: DSV4-Flash-FP8 dp8ep8 GSM8K accuracy below 0.91 gate

• Affected: nightly-mi355x-benchmark (dsv4flash-fp8-1k1k-1p1d-dp8ep8) — GSM8K accuracy gate (pre-sweep).
• Shared facts: accuracy=0.906 < threshold=0.91 (0.44% miss), all 1319/1319 samples completed HTTP 200; harness printed accuracy below threshold -- failing before sweep, bench_exit=1; prefill rc=137 / decode rc=143 are post-gate teardown, not crashes. Sister non-dp8ep8 config passed today → dp8ep8/disagg-specific on gfx950.
• Top hypothesis: [MEDIUM] regression from #29275 (merged 07-08, gfx95 FP8 activation-scale layout), tracked in #31490; temporal + dp8ep8-specific fit. Disconfirming: candidate fix #31617 did not recover accuracy, so the exact as_strided mechanism at fp8_utils.py:935 isn't confirmed as the sole cause.
• In-flight fix: ⚠️ #31617 (open since 07-17) — author-marked superseded / do-not-merge (validation still 0.904). Tracked bug #31490 open, no working fix.
• Suggested triage: A/B revert #29275 on the dp8ep8 disagg nightly to reconfirm attribution + quantify recovered accuracy; if accuracy genuinely sits ~0.90–0.912, decide (human call) whether to quarantine the 0.91 gate while #31490 is worked.

Cluster: PD-disagg KV-transfer stall at high concurrency

• Affected: nightly-mi355x-benchmark (dsv4flash-fp4-1k1k-1p1d) — conc=64 concurrency sweep (benchmark, not pytest).
• Shared facts: GSM8K gate passed (0.923); conc=1/8/16/32 completed cleanly. At conc=64 (07:40:51) decode stopped emitting batches with #transfer-req: 15 pending; prefill collapsed to a single stuck request re-prefilled 137× at ~4.27 tok/s until SLURM CANCELLED … DUE TO TIME LIMIT ~2h17m later; all procs rc=143 (SIGTERM). No traceback/OOM. Config: disaggregation_transfer_backend='mori', disable_overlap_schedule=False.
• Top hypothesis: [MEDIUM] #31368 (2c64b778, merged 07-17 17:08 UTC — first scheduled run to include it) added a blocking task.wait_event.synchronize() in MoriKVSender on the exact active (mori + overlap) path; a never-completing event would hang the transfer thread and stall KV to decode. Disconfirming: no py-spy/stack confirming the park; conc≤32 exercise the same path and passed (concurrency-dependent); Jul 14 also failed this job before #31368 existed (signature unconfirmed).
• In-flight fix: ❌ none found (searched open PRs mori disaggregation, early-send KV).
• Suggested triage: revert 2c64b778 on a branch + re-run dsv4flash-fp4-1k1k-1p1d at conc64; py-spy the prefill MoriKVSender worker during a live conc64 repro to confirm it is parked in wait_event.synchronize(); if #31368 is exonerated, pull Jul 14 86998273362 mi355x-* artifacts to check for the same signature.

Suggested triage order

Rank	Triage step	Targets	Why this first
1	Revert 2c64b778 (#31368) on a branch + re-run dsv4flash-fp4-1k1k-1p1d at conc64	Cluster 2 (row 2)	Fresh regression, clear code-level hypothesis on the active mori+overlap path, no in-flight fix — cheapest A/B to confirm
2	Reopen #31490: A/B revert #29275 on the dp8ep8 disagg nightly	Cluster 1 (row 1)	Existing fix #31617 is a confirmed dead end; need to reconfirm attribution before deciding gate handling
3	py-spy prefill MoriKVSender worker during a live conc64 repro	Cluster 2 (row 2)	Direct evidence to promote/demote the wait_event.synchronize() hypothesis if the revert is inconclusive

---

Job	Failed Steps	Started
nightly-mi355x-benchmark (dsv4flash-fp8-1k1k-1p1d-dp8ep8, dsv4flash-fp8-mi355x-dp8ep8-sglang, sgl...	Launch MI355X 2N 1P1D benchmark	2026-07-18T11:12
nightly-mi355x-benchmark (dsv4flash-fp4-1k1k-1p1d, dsv4flash-fp4-mi355x-sglang, deepseek-ai/DeepS...	Launch MI355X 2N 1P1D benchmark	2026-07-18T07:28

Per-Job Analysis

nightly-mi355x-benchmark (dsv4flash-fp8-1k1k-1p1d-dp8ep8, dsv4flash-fp8-mi355x-dp8ep8-sglang, sgl... — failed step(s): Launch MI355X 2N 1P1D benchmark

Commit Info

• sglang: 44e4999a
• aiter: N/A (no [CI-AITER-CHECK] markers in this workflow's job log; the MI355X disagg benchmark runs on a pre-built image with no aiter version check)

Failed Tests

Test File	Test Function	Error	Log
test/srt/benchmark harness (GSM8K accuracy gate)	dsv4flash-fp8-1k1k-1p1d-dp8ep8 GSM8K gate	accuracy=0.906 < threshold=0.91 → "failing before sweep" → srun exit 1	Job · artifact:gsm8k.log

This is not a pytest failure — it is the benchmark harness's GSM8K accuracy gate that runs before the performance sweep. It failed the gate, so the sweep never ran (note: the artifact archive contains only _logs.tar.gz, no agg_*.json/result JSONs, unlike every passing config).

Failure Summary

The dsv4flash-fp8-1k1k-1p1d-dp8ep8 (DeepSeek-V4-Flash-FP8, dp8/ep8, 1P1D disagg) job failed its pre-sweep GSM8K accuracy gate: accuracy=0.906 against threshold=0.91 (a 0.44% miss). The harness printed accuracy below threshold -- failing before sweep and exited with code 1 (bench_exit=1). The prefill (rc=137) and decode (rc=143) server exits are the harness tearing down the servers after the gate failed — not the primary cause. Both servers stayed healthy and served all 1319 GSM8K requests with HTTP 200 through 11:21:41.

Regression Status

Known recurring / intermittent failure — accuracy straddles the 0.91 gate.

Recent history of the dsv4flash-fp8-1k1k-1p1d-dp8ep8 GSM8K gate in job nightly-mi355x-benchmark (dsv4flash-fp8-...-dp8ep8) (comparable completed scheduled runs, event=schedule, branch=main):

Date	Run	Job	Test/Gate Status	Value	Note
Jul 18	29630291750	job	❌ Failed	0.906 < 0.91	gate miss (artifact-confirmed)
Jul 17	29554632780	job	✅ Passed	—	—
Jul 16	29471180267	job	✅ Passed	—	—
Jul 15	29388634922	job	✅ Passed	—	—
Jul 14	29305625618	—	⚪ Cancelled	—	excluded
Jul 13	29223989193	job	✅ Passed	—	—
Jul 12	29179694296	job	❌ Failed	(gate miss)	artifact ~181KB, logs-only (same mode); gate value not re-downloaded

First observed failure: 2026-07-12; last known pass: 2026-07-17. Failed 2 of the 6 completed runs where this job ran (07-12, 07-18) → intermittent, threshold-straddling. The sister config dsv4flash-fp8-1k1k-1p1d (non-dp8ep8) passed today (job), so this is specific to the dp8ep8 disagg path.

Failure Cluster

DSV4-Flash-FP8 dp8ep8 GSM8K accuracy below 0.91 gate (gfx950 disagg)

Facts (verified observations)

• gsm8k.log (artifact:.../mi355x-dsv4flash-fp8-1k1k-1p1d-dp8ep8.../logs/.../gsm8k.log): Accuracy: 0.906, Invalid: 0.001, all 1319/1319 samples completed; then [gsm8k] accuracy=0.906 threshold=0.91, [gsm8k] accuracy below threshold -- failing before sweep, srun: error: mia1-p02-g09: task 0: Exited with exit code 1. Also mirrored in job-page log lines 4129-4130.
• bench_exit = 1; server_exit_prefill_mia1-p02-g09 = rc=137 (SIGKILL teardown); server_exit_decode_mia1-p02-g29 = rc=143 (SIGTERM teardown).
• Prefill/decode logs: both served requests with HTTP 200 up to 11:21:36–41; no OOM / CUDA / HIP error / traceback. Prefill's two Health check failed ... detokenizer lines (11:20:23, 11:20:48) are transient heartbeat lag — requests kept succeeding afterward. Servers healthy; exits are post-gate teardown.
• Server config (prefill log): DeepSeek-V4-Flash-FP8, kv_cache_dtype='fp8_e4m3', tp_size=8, dp_size=8, ep_size=8, attention_backend='dsv4', disaggregation_mode='prefill'.
• Reproducibility: 2/6 completed runs failed (07-12, 07-18); passed 07-13/15/16/17. Accuracy sits ~0.90–0.912, straddling the 0.91 gate.
• Regressing change is tracked upstream: issue #31490 traces the regression to merged PR #29275 (merged 2026-07-08). The exact code path it names is present: fp8_utils.py:935 x_scale = torch.as_strided(x_scale, x_scale.shape, (1, x_scale.shape[0])) inside aiter_w8a8_block_fp8_linear (line 899).

Hypothesised Causes (with confidence)

• [MEDIUM] GSM8K accuracy regression on DSV4-Flash-FP8 dp8ep8 introduced by #29275 ("Fix gfx95 bpreshuffle FP8 activation scale layout", merged 2026-07-08), tracked in issue #31490.
  • Supporting: (a) maintainer-filed bug #31490 explicitly traces the regression to #29275; (b) temporal fit — #29275 merged 07-08, this job's first failure is 07-12, accuracy now straddles the gate; (c) dp8ep8/disagg-specific (sister non-dp8ep8 config passed today), matching #29275's disagg pre-quantized-scale path; (d) observed 0.906 matches the ~0.904–0.912 range cited in #31490 / PR #31617's validation.
  • Disconfirming: the candidate one-line fix PR #31617 (replace as_strided with .contiguous() at fp8_utils.py:935) was validated on MI355X disagg and did NOT recover accuracy (fp8 0.904 / fp4 0.911) — so the precise line-level mechanism (transposed per-token scale) is not confirmed as the sole cause; the regression's exact mechanism remains open.
  • How to verify: on a branch, revert #29275 and re-run dsv4flash-fp8-1k1k-1p1d-dp8ep8 on the MI355X 2N 1P1D disagg nightly; if accuracy returns to ~0.925 the attribution holds.
• [LOW] Pure run-to-run accuracy variance at the gate boundary (flake), independent of #29275.
  • Supporting: intermittent (passed 07-13→07-17), only 0.44% below gate.
  • Disconfirming: the pass runs cluster and a tracked regression (#31490) with a plausible dp8ep8-specific code path exists; variance alone doesn't explain the dp8ep8-vs-non-dp8ep8 split.
  • How to verify: run the dp8ep8 gate N times on a fixed pre-#29275 vs post-#29275 image and compare the accuracy distribution to the 0.91 gate.

In-flight Fix Check

• In-flight fix: ⚠️ #31617 (open since 2026-07-17) — "Fix DSV4-Flash dp8ep8 GSM8K regression: row-major Triton FP8 scale on gfx95". Marked by its author as "Superseded / not working — do NOT merge" (validation still showed fp8 0.904). So there is a tracked regression (#31490, open) but no working fix yet.
• Searched open PRs with keywords dsv4flash, mi355x accuracy — #31617 is the only direct match; the rest are unrelated AMD/gfx95 work.

Suggested Triage Steps

• Confirm the gate-value regression is the sole failure by re-reading artifact:gsm8k.log for run 29179694296 (07-12) — expect the same accuracy ~0.90 < 0.91 signature (artifact size ~181KB, logs-only, already indicates a pre-sweep gate miss).
• Since fix #31617 is a confirmed dead end, reopen investigation on #31490: bisect the disagg pre-quantized FP8 path beyond the as_strided line, or A/B a full revert of #29275 on the dsv4flash-fp8-...-dp8ep8 disagg nightly to reconfirm the attribution and quantify the recovered accuracy.
• If the accuracy genuinely sits at ~0.90–0.912 with no near-term fix, decide (human call) whether to temporarily adjust/quarantine the 0.91 dp8ep8 gate to stop daily nightly noise while #31490 is worked — do NOT silently lower the gate without linking #31490.

Status

• Persistence: Intermittent — failed 2026-07-12 and 2026-07-18; passed the 4 completed runs in between (07-13, 07-15, 07-16, 07-17). Accuracy straddles the 0.91 gate (~0.90–0.912).
• Scope: dsv4flash-fp8-1k1k-1p1d-dp8ep8 job in nightly-amd-mi355x-disagg.yml only; sister non-dp8ep8 config passes. Regression is dp8ep8/disagg-specific on gfx950 (MI355X).
• Blocked work: No perf-sweep data collected for this config on failing days (gate fails before sweep); contributes to the daily run-level failure of the MI355X disagg nightly.
• In-flight fix: ⚠️ #31617 exists but is a documented non-working / do-not-merge attempt; tracked bug #31490 remains open with no working fix.

nightly-mi355x-benchmark (dsv4flash-fp4-1k1k-1p1d, dsv4flash-fp4-mi355x-sglang, deepseek-ai/DeepS... — failed step(s): Launch MI355X 2N 1P1D benchmark

Commit Info

• sglang: 44e4999a
• aiter: N/A — this SLURM-launched MI355X benchmark emits no [CI-AITER-CHECK] markers; aiter is baked into the runner image and not reported in the log.

Failed Tests

Test File	Test Function	Error	Log
N/A (benchmark job, not a pytest suite)	dsv4flash-fp4-1k1k-1p1d concurrency sweep	PD-disaggregation KV-transfer stall at concurrency=64; SLURM step CANCELLED ... DUE TO TIME LIMIT; all processes exited rc=143 (SIGTERM)	Job

Failure Summary

This is a benchmark job (DeepSeek-V4-Flash FP4, 1-prefill/1-decode PD-disaggregation on MI355X, mori transfer backend), not a unit test. The GSM8K accuracy gate passed (artifact:.../gsm8k.log → Accuracy 0.923) and the concurrency sweep completed cleanly for conc=1/8/16/32 (all four agg_*conc{1,8,16,32}*.json archived). During the concurrency=64 warmup burst at 07:40:51 the pipeline stalled: the decode server stopped emitting decode batches while KV transfers were still in flight, and the whole SLURM allocation was killed ~2h17m later at the wall-clock time limit (07:35:38 start → 09:58:50 cancel).

Regression Status

New regression (first failure today after 3 consecutive passes).

Recent history of job nightly-mi355x-benchmark (dsv4flash-fp4-1k1k-1p1d) (event=schedule, branch=main, completed runs; verdict at this matrix-job level, not run level):

Date	Run	Job Status	Note
Jul 18	29630291750	❌ Failed	conc64 KV-transfer stall → SLURM time-limit kill (this run)
Jul 17	29554632780	✅ Passed	last known pass (27ad9d11b1)
Jul 16	29471180267	✅ Passed	—
Jul 15	29388634922	✅ Passed	—
Jul 14	29305625618	❌ Failed	isolated prior failure (signature not confirmed at artifact level)
Jul 13	29223989193	✅ Passed	—
Jul 12	29179694296	✅ Passed	—
Jul 11	29139507423	✅ Passed	—

First observed (this signature): 2026-07-18. Last pass: 2026-07-17. The Jul 14 failure was preceded and followed by clean passes; I did not download its mi355x-* artifacts, so whether it shares this conc64-stall signature is unconfirmed.

Failure Cluster

PD-disaggregation KV-transfer stall / decode-loop hang at high concurrency (→ SLURM wall-clock kill, rc=143)

Facts (verified observations)

• artifact:.../logs/.../server_exit_prefill_mia1-p02-g09 → prefill@mia1-p02-g09 rc=143; server_exit_decode_mia1-p02-g29 → decode@mia1-p02-g29 rc=143; bench_exit → 143. rc=143 = 128+15 (SIGTERM), i.e. killed, not a crash.
• artifact:.../bench.log → conc=1/8/16/32 each printed a full Serving Benchmark Result (128 successful requests at conc32). === concurrency=64 === printed the aiter import + FutureWarning, then no further progress before slurmstepd: error: *** STEP 12992.2 ... CANCELLED AT 2026-07-18T09:58:50 DUE TO TIME LIMIT ***.
• artifact:.../prefill_mia1-p02-g09.log → at 07:40:50-51 the conc64 warmup burst ran (#new-seq: 8, queue draining 31→0, #inflight-req: 16). From 07:41:38 it collapsed to a single stuck request repeated 137× every 60s: Prefill batch, #new-seq: 1, #new-token: 256, #running-req: 0, #queue-req: 0, #inflight-req: 1 ... input throughput (token/s): 4.27, continuing until the kill.
• artifact:.../decode_mia1-p02-g29.log → last decode batch at 07:40:51: Decode batch, #running-req: 24 ... #transfer-req: 15. No further decode batches for ~2h18m (only GET /health 200 OK continued) until the SLURM cancel. No traceback/OOM/crash in either server log.
• artifact:.../prefill_mia1-p02-g09.log startup → disaggregation_transfer_backend='mori', disable_overlap_schedule=False (overlap scheduling enabled), disaggregation_mode='prefill', DeepSeek-V4-Flash FP4, tp_size=8.
• GSM8K gate passed: artifact:.../gsm8k.log → Accuracy: 0.923, Invalid: 0.000.
• Regression window 27ad9d11b1..44e4999ab2 = 47 commits (verified via git log).

Hypothesised Causes (with confidence + disconfirming evidence)

• [MEDIUM] PR #31368 (2c64b778, "[AMD][PD] Fix early-send cached-prefix KV racing the prefill forward on mori") introduced a blocking task.wait_event.synchronize() in MoriKVSender plus a torch.cuda.Event recorded on forward_stream in prefill.py, gated by if self.enable_overlap:.
  • Supporting: This job runs the exact modified backend (disaggregation_transfer_backend='mori') with overlap enabled (disable_overlap_schedule=False), so both edited paths are active. The commit merged Jul 17 17:08 UTC — after the last passing run (07-17 04:21 UTC) and before today's failure (07-18 04:16 UTC), i.e. today is the first scheduled run to include it. The symptom (decode stops with #transfer-req: 15 pending; prefill stuck at #inflight-req: 1) is exactly what a hung KV-transfer worker would produce: if the recorded event never completes, .synchronize() blocks the transfer thread forever and KV never reaches decode.
  • Disconfirming: No stack trace / py-spy confirming the transfer thread is actually parked in synchronize() (the hang is silent). conc=1/8/16/32 — which also exercise this path — passed, so any deadlock is concurrency-dependent (consistent with a race but not proof). The Jul 14 run also failed for this job before #31368 existed, hinting at possible prior high-concurrency instability.
  • How to verify: git revert 2c64b778 on a branch and re-run the dsv4flash-fp4-1k1k-1p1d recipe on MI355X; or attach py-spy to the prefill MoriKVSender transfer worker during a conc64 run and confirm it is blocked in wait_event.synchronize().
• [LOW] PR #31584 (2db21daf, "Fix Heartbeat Checker in KV Manager Disaggregation") removed the session_pool eviction on heartbeat failure in common/conn.py.
  • Supporting: In the disaggregation KV-manager path; also landed in the window (Jul 17 17:21 UTC).
  • Disconfirming: Only removes a session-pool del on heartbeat failure (reconnection bookkeeping); no heartbeat-failure or reconnect messages appear in the logs, and the stall begins mid-transfer while both servers are healthy — poor fit for the observed symptom.
  • How to verify: check prefill/decode logs of a repro for any heartbeat/reconnect activity; revert independently if #31368 revert does not resolve it.
• [LOW] Pure infrastructure — the SLURM allocation simply ran out of wall-clock time on a slow sweep.
  • Supporting: proximate kill reason is CANCELLED ... DUE TO TIME LIMIT.
  • Disconfirming: conc1-32 finished in seconds each; the 2h+ was spent entirely in a stalled conc64 (decode silent, prefill re-prefilling one request every 60s), so the time-limit kill is a symptom of the hang, not the root cause.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for mori disaggregation (34 results, all feature work — mooncake/NIXL legs, unified-memory KV, role-switching; none revert/fix #31368) and early-send KV (2 unrelated results). No open PR targets the mori early-send wait_event synchronize path. API calls: search/issues?q=repo:sgl-project/sglang+is:pr+is:open+mori+disaggregation and ...+early-send+KV.

Suggested Triage Steps

• Bisect / A-B the 47-commit window 27ad9d11b1..44e4999ab2, prioritizing 2c64b778 (#31368).
• Revert 2c64b778 on a branch and re-run the dsv4flash-fp4-1k1k-1p1d recipe at conc64 on MI355X to confirm/deny.
• During a live conc64 repro, py-spy the prefill MoriKVSender transfer worker; if it is parked in wait_event.synchronize(), the fix direction is to make the wait non-blocking / bounded (e.g. poll ev.query() with a timeout) rather than an unbounded .synchronize() in the transfer thread.
• If #31368 is exonerated, download the Jul 14 86998273362 mi355x-* artifacts to check whether that isolated failure shares this conc64-stall signature (would reclassify as intermittent high-concurrency hang rather than a clean regression).

Status

• Persistence: First occurrence of this conc64 KV-transfer-stall signature today (2026-07-18); this matrix job passed the previous 3 scheduled runs (Jul 15-17).
• Scope: dsv4flash-fp4-1k1k-1p1d job of nightly-amd-mi355x-disagg.yml. The suspected commit is mori-backend + overlap-specific, so other MI355X disagg matrix entries using mori (dp8ep8 / mtp / dsv4pro / fp8 variants) may share it — not verified in this per-job analysis.
• Blocked work: MI355X DeepSeek-V4-Flash FP4 1P1D disaggregation perf signal; the job also consumes the full ~2h SLURM allocation per run while hung.
• In-flight fix: ❌ none found in open PRs.

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-07-19 00:26 UTC)

[amd-bot]

nightly-test-amd.yml — 13 failure(s)

Run started (UTC): 2026-07-18 17:47 UTC
Last scanned (UTC): 2026-07-19 22:57 UTC
Commits: sglang d4801be

Summary

Counts: 13 failures · 8 clusters · 11 carrying over (≥3 days) · 2 new today

#	Cluster	Job	Test File	Test Function	Status	Hypothesis (confidence)
1	VLM MMMU accuracy pinned at random-chance	nightly-accuracy-2-gpu-vlm	test/registered/amd/accuracy/mi30x/test_vlms_mmmu_eval_amd.py	TestNightlyVLMMmmuEvalAMD.test_mmmu_vlm_models	Never-passed, ~3 wks	[MEDIUM] systemic vision/eval-harness issue — 11/11 models at ~0.28
2	VLM MMMU accuracy pinned at random-chance	nightly-accuracy-2-gpu-vlm	test/registered/amd/accuracy/mi30x/test_vlms_mmmu_eval_amd.py	TestNightlyVLMMmmuEvalAMD.test_mmmu_vlm_models	Never-passed, ~3 wks	[MEDIUM] same as row 1
3	gsm8k 2-GPU accuracy fail (Mixtral / log-truncated)	nightly-accuracy-2-gpu	test/registered/amd/accuracy/mi30x/test_gsm8k_eval_amd.py	TestNightlyGsm8KEval.test_gsm8k_all_models	Never-passed (≥Jul 4)	[MEDIUM] Mixtral 0.292 = gsm8k answer-extraction bug
4	gsm8k 2-GPU accuracy fail (Mixtral / log-truncated)	nightly-accuracy-2-gpu	test/registered/amd/accuracy/mi30x/test_gsm8k_eval_amd.py	TestNightlyGsm8KEval.test_gsm8k_all_models	Never-passed (≥Jul 4)	[no candidate ≥ MEDIUM] (log truncated)
5	GLM-5.1-MXFP4 MI35x server-launch failure	nightly-2-gpu-mi35x-glm51-mxfp4	test/registered/amd/accuracy/mi35x/test_glm51_mxfp4_tp2_gsm8k_mi35x.py	TestGLM51MXFP4TP2GSM8KMI35x.setUpClass	Persistent since Jul 7	[no candidate ≥ MEDIUM] (infra/HF-download)
6	GLM-5.1-MXFP4 MI35x server-launch failure	nightly-2-gpu-mi35x-glm51-mxfp4	test/registered/amd/accuracy/mi35x/test_glm51_mxfp4_tp2_gsm8k_mi35x.py	TestGLM51MXFP4TP2GSM8KMI35x.setUpClass	Persistent since Jul 13	[no candidate ≥ MEDIUM] (HF weight-download stall)
7	NameError: mamba JIT kernel unimported on AMD	nightly-test-1-gpu-kernel	test/registered/jit/test_transfer_mamba.py	test_mamba_kernel_* (12×)	Never-passed (Jul 16)	[FACT] import gated on if _is_cuda: (memory_pool_host.py:41)
8	NameError: mamba JIT kernel unimported on AMD	nightly-test-1-gpu-kernel	test/registered/jit/test_transfer_mamba.py	test_mamba_kernel_* (12×)	Never-passed (Jul 16)	[FACT] same as row 7
9	hipErrorCapturedEvent in aiter custom AR during graph capture	nightly-8-gpu-mi35x-qwen3-235b-mxfp4	test/registered/amd/test_qwen3_instruct_mxfp4.py	TestQwen3Instruct2507MXFP4.setUpClass	Never-passed (Jul 17)	[MEDIUM] aiter AR event-op during HIP capture
10	hipErrorCapturedEvent in aiter custom AR during graph capture	nightly-8-gpu-mi35x-qwen3-235b-mxfp4	test/registered/amd/test_qwen3_instruct_mxfp4.py	TestQwen3Instruct2507MXFP4.setUpClass	Never-passed (Jul 17)	[MEDIUM] same as row 9
11	VLM MMMU benchmark timeout on GLM-4.1V-9B-Thinking	nightly-4-gpu	test/registered/vlm/test_encoder_dp.py	TestVLMEncoderDP.test_vlm_mmmu_benchmark	Flaky (GLM draw fails)	[HIGH] random draw = GLM-4.1V too slow for 3600s cap
12	Self-hosted runner lost communication (MI325 infra)	nightly-test-1-gpu-unit	N/A (non-test)	N/A	New today (one-off)	[FACT] runner lost network comms
13	GSM8K MTP accuracy marginally below threshold	nightly-8-gpu-mi35x-deepseek-r1-mxfp4-tp4	test/registered/amd/accuracy/mi35x/test_deepseek_r1_mxfp4_tp4_mtp_mi35x.py	TestDeepSeekR1MXFP4TP4MTPMI35x.test_gsm8k	New today (0.943 vs 0.944)	[MEDIUM] sampling variance at tight threshold

Cluster: VLM MMMU accuracy pinned at random-chance (rows 1–2)

• Affected: nightly-accuracy-2-gpu-vlm & Jul 19, test_vlms_mmmu_eval_amd.py::test_mmmu_vlm_models.
• Shared facts: all 11 VLMs score 0.26–0.34 (≈4-choice random), near-deterministic run-to-run for 3+ weeks; 5/11 below threshold; no server crash.
• Top hypothesis: [MEDIUM] systemic AMD vision-path / MMMU grading issue. Disconfirming: no image-decode error in log; thresholds may simply be mis-calibrated.
• In-flight fix: ❌ none.
• Suggested triage: compare the same models on the NVIDIA VLM nightly (≥0.45 there ⇒ AMD vision path broken); dump raw MMMU responses for Qwen2.5-VL-7B on MI325.

Cluster: gsm8k 2-GPU accuracy fail (rows 3–4)

• Affected: both nightly-accuracy-2-gpu, test_gsm8k_eval_amd.py. Job red every completed run ≥Jul 4.
• Shared facts: Mixtral-8x7B scores 0.292/0.69; Qwen3-4B "FAILED" is cosmetic (no threshold entry); logs truncated by aiter untuned-GEMM flood during eager decode (bs>512).
• Top hypothesis: [MEDIUM] Mixtral miss is the gsm8k numbers[-1] answer-extraction bug (red 4+ mo cross-vendor), not an AMD regression. Disconfirming: Jul 19 tail not retrievable, so a termination/hang can't be excluded.
• In-flight fix: ✅ #29933 (Mixtral gsm8k extraction) — chase reviewers; ❌ none for Qwen3-4B threshold.
• Suggested triage: re-pull full log tail after run completes; add Qwen/Qwen3-4B threshold entry.

Cluster: GLM-5.1-MXFP4 MI35x server-launch failure (rows 5–6)

• Affected: both nightly-2-gpu-mi35x-glm51-mxfp4, test_glm51_mxfp4_tp2_gsm8k_mi35x.py::setUpClass. test_gsm8k never runs.
• Shared facts: symptom-varied server-launch failures on linux-mi35x-gpu-2 (HF-download stall/xet_get, GPU hang, OOM -9, runner comms loss) since Jul 7; last pass Jul 6 (0.94). aiter unchanged.
• Top hypothesis: [FACT] mechanism = server never ready within SERVER_LAUNCH_TIMEOUT; scheduler parked in HF weight download. Root cause (infra vs code) below MEDIUM.
• In-flight fix: ✅ #31068 (MI35x hung-GPU recovery), ✅ #30431 (VRAM-leak self-heal).
• Suggested triage: reproduce huggingface-cli download amd/GLM-5.1-MXFP4 on the runner to split infra from code; inspect runner dmesg/rocm-smi.

Cluster: NameError mamba JIT kernel unimported on AMD (rows 7–8)

• Affected: both nightly-test-1-gpu-kernel, test_transfer_mamba.py (12/12 kernel-path cases).
• Shared facts: transfer_kv_mamba_lf_pf imported only under if _is_cuda: (memory_pool_host.py:41); the io_backend="kernel" path (line 394) runs on AMD → NameError. Function exists (jit_kernel/transfer_mamba.py:64) — pure missing-import. Introduced Jul 16 (test absent Jul 14).
• Top hypothesis: [FACT] CUDA-only import guard vs unconditional call site. Disconfirming: (none) — deterministic; but whether the kernel actually runs on HIP is unverified, so broadening the guard may not be the right fix.
• In-flight fix: ❌ none.
• Suggested triage: decide AMD support — extend guard to if _is_cuda or _is_hip: (mirroring line 31) and verify HIP build, OR skip/xfail the io_backend="kernel" cases on AMD.

Cluster: hipErrorCapturedEvent in aiter custom AR during graph capture (rows 9–10)

• Affected: both nightly-8-gpu-mi35x-qwen3-235b-mxfp4, test_qwen3_instruct_mxfp4.py::setUpClass (tp4/ep2, aiter AR). Crash in custom_all_reduce.py flush_graph_buffers/_gather_ipc_meta during decode CUDA-graph capture; RCCL watchdog hipEventQuery.
• Shared facts: identical stack Jul 17/18/19 (also Jul 3); aiter unchanged (9127c94a1).
• Top hypothesis: [MEDIUM] aiter custom AR performs event/IPC op during HIP capture (ROCm hipEventQuery cross-thread capture-mode bug). Disconfirming: no green baseline for this config; #24151 fix is ROCm720-only but runner is rocm700.
• In-flight fix: ✅ #23581 (default SGLANG_USE_AITER_AR false); related #24151.
• Suggested triage: re-run with SGLANG_USE_AITER_AR=0 / --disable-custom-all-reduce; if capture passes, adopt #23581 for the mi35x/rocm700 path.

Cluster: VLM MMMU benchmark timeout on GLM-4.1V-9B-Thinking (row 11)

• Affected: nightly-4-gpu, test_encoder_dp.py::test_vlm_mmmu_benchmark.
• Shared facts: test picks 1 random model; fails on every GLM-4.1V draw (Jul 5/13/19), passes on InternVL. Server healthy, decode ~3 tok/s, killed at 3600s.
• Top hypothesis: [HIGH] GLM-4.1V thinking model too slow for the 3600s cap on MI30x 4-GPU. Disconfirming: earlier GLM runs died via exit -9 (OOM), not timeout — >1 failure path.
• In-flight fix: ❌ none.
• Suggested triage: pin the model to GLM-4.1V and reproduce; consider excluding it from the AMD 4-gpu list or raising the cap.

Cluster: GSM8K MTP accuracy marginally below threshold (row 13)

• Affected: nightly-8-gpu-mi35x-deepseek-r1-mxfp4-tp4, test_deepseek_r1_mxfp4_tp4_mtp_mi35x.py::test_gsm8k.
• Shared facts: 0.943 vs 0.944 (1/1319 miss); passed Jul 17/18 at 0.948; non-MTP sister improved to 0.959 same run; aiter unchanged; test 3 days old.
• Top hypothesis: [MEDIUM] sampling variance at a too-tight threshold. Disconfirming: only 3 runs of history; can't rule out a small MTP-only regression.
• In-flight fix: ❌ none.
• Suggested triage: re-run 3–5× on the same SHA; if it oscillates 0.943–0.948, revisit the threshold.

Cluster: Self-hosted runner lost communication (row 12)

• Affected: nightly-test-1-gpu-unit, non-test (runner died ~10 min in). Green the 6 prior runs.
• Top hypothesis: [FACT] runner ...pwzgr lost network comms; [LOW] node resource exhaustion. Head commit not implicated.
• In-flight fix: ❌ none (infra incident).
• Suggested triage: retry; if it recurs, infra checks the runner host.

Suggested triage order

Rank	Triage step	Targets	Why this first
1	Re-run with SGLANG_USE_AITER_AR=0; verify/merge #23581	Cluster A (rows 9–10)	In-flight fix already exists; 8-GPU MXFP4 signal fully blocked
2	Decide AMD support for mamba JIT kernel; fix guard or skip on AMD	Cluster B (rows 7–8)	[FACT] root cause; one-line guard vs test-skip; deterministic 12× red
3	Chase reviewers on #29933; add Qwen3-4B threshold; recover full log tail	Cluster D (rows 3–4)	In-flight fix exists; log-flood hides real signal
4	Cross-check VLM MMMU vs NVIDIA nightly; dump raw responses	Cluster E (rows 1–2)	Longest-running (3 wks); distinguishes vision-path bug vs threshold
5	Reproduce GLM-5.1-MXFP4 HF download on MI35x runner	Cluster C (rows 5–6)	Cheap infra-vs-code split; two in-flight infra PRs to validate

---

Job	Failed Steps	Started
nightly-test-1-gpu-unit	N/A	2026-07-18T19:10
nightly-test-1-gpu-kernel	Nightly Kernel Benchmark (1-GPU)	2026-07-18T19:09
nightly-accuracy-2-gpu-vlm	Nightly Accuracy Test (2-GPU VLM MMMU)	2026-07-18T19:54
nightly-accuracy-2-gpu	N/A	2026-07-18T18:07
nightly-2-gpu-mi35x-glm51-mxfp4	N/A	2026-07-18T17:47
nightly-8-gpu-mi35x-qwen3-235b-mxfp4	Accuracy Test + Performance Test MI35x (8-GPU Qwen3-235B-MXFP4)	2026-07-18T23:20
nightly-test-1-gpu-kernel	Nightly Kernel Benchmark (1-GPU)	2026-07-19T17:53
nightly-2-gpu-mi35x-glm51-mxfp4	N/A	2026-07-19T17:47
nightly-4-gpu	Nightly Test (4-GPU)	2026-07-19T18:11
nightly-accuracy-2-gpu	N/A	2026-07-19T18:27
nightly-8-gpu-mi35x-deepseek-r1-mxfp4-tp4	Accuracy Test MI35x (DeepSeek-R1-MXFP4 TP4 MTP)	2026-07-19T20:08
nightly-accuracy-2-gpu-vlm	Nightly Accuracy Test (2-GPU VLM MMMU)	2026-07-19T20:47
nightly-8-gpu-mi35x-qwen3-235b-mxfp4	Accuracy Test + Performance Test MI35x (8-GPU Qwen3-235B-MXFP4)	2026-07-19T22:34

Per-Job Analysis

nightly-test-1-gpu-unit — failed step(s): N/A

Commit Info

• sglang: b3a0185ca — "model_runner: extract post-memory-pool wiring into _init_post_memory (#31601)"
• aiter: N/A — job log blob is unavailable (BlobNotFound; parent run 29654601256 still in_progress at snapshot), so no [CI-AITER-CHECK] markers could be read.

Failed Tests

Test File	Test Function	Error	Log
N/A (non-test failure)	N/A	The self-hosted runner lost communication with the server	Job

This is not a test-level failure. The self-hosted runner terminated the job before any test results were produced; no per-test pass/fail data exists in the (unavailable) log.

Failure Summary

The nightly-test-1-gpu-unit job died ~10 min into execution (started 19:10:33Z, killed 19:20:34Z) — far short of the step's 90-min timeout. The check-run annotation reports the self-hosted MI325 runner linux-mi325-1gpu-sglang-ng8rq-runner-pwzgr lost communication with the GitHub Actions server. The test step also runs with --continue-on-error on schedule events, so an ordinary test assertion could not have failed the job — the job conclusion of failure comes from the runner dying, not from a test.

Regression Status

Infrastructure issue (runner communication loss) — not a code regression.

Recent history of nightly-test-1-gpu-unit in job nightly-test-1-gpu-unit (most recent comparable completed scheduled runs: branch=main, event=schedule, status=completed):

Date	Run	Job	Job Status	Notes
Jul 18	29654601256	nightly-test-1-gpu-unit	❌ Failed	Runner lost communication (this run)
Jul 17	29601593052	job	✅ Passed	—
Jul 16	29521595306	job	✅ Passed	—
Jul 14	29355460805	job	✅ Passed	—
Jul 13	29273241962	job	✅ Passed	—
Jul 5	28749559519	job	✅ Passed	—
Jul 4	28714602438	job	✅ Passed	—
Jul 3	28676359018	job	❌ Failed	Different signature: exit code 255 (not runner comms)

The job was green in the 6 most recent completed scheduled runs (Jul 13–17). This failure ends that streak but the cause is infrastructural, not a regression from b3a0185ca. The prior Jul 3 failure was a different signature (Process completed with exit code 255), so it is not the same recurring failure.

Failure Cluster

Self-hosted runner lost communication (MI325 1-GPU pool infra)

Facts (verified observations)

• Check-run annotation (job 88106575913) reads verbatim: "The self-hosted runner lost communication with the server. Verify the machine is running and has a healthy network connection. Anything in your workflow that terminates the runner process, starves it for CPU/Memory, or blocks its network access can cause this error."
• Job duration ≈ 10 min (started 2026-07-18T19:10:33Z, completed 2026-07-18T19:20:34Z); the test step timeout is 90 min, so no timeout was hit.
• Runner: linux-mi325-1gpu-sglang-ng8rq-runner-pwzgr (label linux-mi325-1gpu-sglang).
• The test step runs run_suite.py --suite nightly-amd-1-gpu ... --continue-on-error on schedule (workflow lines 161-168), so individual test failures cannot fail the job — corroborating a runner-level, not test-level, failure.
• Job log blob is BlobNotFound (404 from both the per-job and per-run log endpoints); parent run 29654601256 is still in_progress, so logs are not finalized.
• Sibling 1-GPU job nightly-test-1-gpu-kernel (88106575904) on a different runner (...k9wqj) failed with a different signature (exit code 255), i.e. this is not a pool-wide comms outage affecting all mi325 runners simultaneously.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [FACT] The job was terminated because the self-hosted runner ...pwzgr lost network communication with the GitHub Actions control plane (directly stated in the failure annotation).
• [LOW] Runner-host resource exhaustion (OOM/CPU starvation) or a transient network/hardware fault on that specific MI325 node killed the runner process.
  • Supporting: annotation explicitly lists "terminates the runner process, starves it for CPU/Memory, or blocks its network access" as causes; job died early (~10 min) with no timeout.
  • Disconfirming: no log is available to inspect for OOM traces; the sibling kernel job on a different runner failed with a different error, so this is likely node-local rather than a systemic pool issue.
  • How to verify: check the runner host's system logs / GPU health for ...pwzgr around 19:20:34Z; re-run the job to confirm it passes on a healthy runner.
• [SPECULATION] Code change in b3a0185ca contributed.
  • Supporting: none — it is the head commit only by coincidence.
  • Disconfirming: a Python-level regression would surface as a test failure in the log, not as a runner-comms loss; and --continue-on-error would suppress test failures anyway. No evidence links the commit to the runner dying.
  • How to verify: not actionable — no failing code path is implicated.

In-flight Fix Check

• In-flight fix: ❌ none found — this is a runner/infra incident, not a code defect, so there is no source-level fix to search for. (No open PR would address a one-off runner communication loss.)

Suggested Triage Steps

• Simply re-run the nightly-test-1-gpu-unit job; runner-communication-loss failures are typically transient and pass on retry.
• If it recurs, have infra check the health of MI325 runner linux-mi325-1gpu-sglang-ng8rq-runner-pwzgr (system/dmesg logs, GPU health, network) around 2026-07-18T19:20:34Z.
• Once the parent run 29654601256 finishes and logs finalize, re-pull the job log to confirm no test-level failure preceded the runner death (currently BlobNotFound).
• No code investigation of b3a0185ca is warranted based on current evidence.

Status

• Persistence: Single occurrence for this failure signature. Job was green in the 6 most recent completed scheduled runs (Jul 13–17); the only prior failure (Jul 3) was a different signature (exit 255).
• Scope: One job on one runner (...pwzgr). The sibling 1-GPU kernel job failed with a different, unrelated signature; other 1-GPU jobs in the run (e.g. nightly-test-1-gpu-mi35x) succeeded.
• Blocked work: Nightly 1-GPU unit-test signal missing for this run only.
• In-flight fix: ❌ none found — infra incident, resolve by retry / runner health check.

nightly-test-1-gpu-kernel — failed step(s): Nightly Kernel Benchmark (1-GPU)

Commit Info

• sglang: b3a0185c
• aiter: 9127c94a (source: dev image v0.1.17.dev110+g9127c94a1, Dev/patched version detected → skipping rebuild; [CI-AITER-CHECK] markers, job log lines 4218-4224)

Failed Tests

All 12 failures are in one file with the identical error; the other jit tests in the job passed.

Test File	Test Function	Error	Log
test/registered/jit/test_transfer_mamba.py	test_mamba_kernel_backup_load_roundtrip[page_first-dtype0]	NameError: name 'transfer_kv_mamba_lf_pf' is not defined	Log
test/registered/jit/test_transfer_mamba.py	test_mamba_kernel_backup_load_roundtrip[page_first-dtype1]	same	Log
test/registered/jit/test_transfer_mamba.py	test_mamba_kernel_backup_load_roundtrip[page_first_direct-dtype0]	same	Log
test/registered/jit/test_transfer_mamba.py	test_mamba_kernel_backup_load_roundtrip[page_first_direct-dtype1]	same	Log
test/registered/jit/test_transfer_mamba.py	test_mamba_kernel_single_item[page_first-dtype0]	same	Log
test/registered/jit/test_transfer_mamba.py	test_mamba_kernel_single_item[page_first-dtype1]	same	Log
test/registered/jit/test_transfer_mamba.py	test_mamba_kernel_single_item[page_first_direct-dtype0]	same	Log
test/registered/jit/test_transfer_mamba.py	test_mamba_kernel_single_item[page_first_direct-dtype1]	same	Log
test/registered/jit/test_transfer_mamba.py	test_mamba_kernel_full_indices[page_first-dtype0]	same	Log
test/registered/jit/test_transfer_mamba.py	test_mamba_kernel_full_indices[page_first-dtype1]	same	Log
test/registered/jit/test_transfer_mamba.py	test_mamba_kernel_full_indices[page_first_direct-dtype0]	same	Log
test/registered/jit/test_transfer_mamba.py	test_mamba_kernel_full_indices[page_first_direct-dtype1]	same	Log

Failure Summary

All 12 parametrizations of test/registered/jit/test_transfer_mamba.py fail during MambaPoolHost.backup_from_device_all_layer(..., io_backend="kernel"), which calls _copy_tensor_all_layers_lf_pf and raises NameError: name 'transfer_kv_mamba_lf_pf' is not defined at python/sglang/srt/mem_cache/memory_pool_host.py:394 (job log line 4612). The job's other jit tests (e.g. test_int8_kernel.py, kv_canary benchmarks) passed; only test_transfer_mamba.py returned exit code 1 (line 5491).

Regression Status

Never-passed (no green run found on AMD). The test file test/registered/jit/test_transfer_mamba.py AND the failing call site were both introduced by the same commit 7d0fd5101 (PR #31227, merged 2026-07-16). It has failed with the identical NameError on every comparable AMD nightly since it landed.

Recent history of test_transfer_mamba.py (test_mamba_kernel_*) in job nightly-test-1-gpu-kernel (comparable completed scheduled runs on main, event=schedule; test file did not exist before 2026-07-16):

Date	Run	Job	Test File Status	Failed Function	Error
Jul 18	29654601256	nightly-test-1-gpu-kernel	❌ Failed	all 12 test_mamba_kernel_*	NameError: transfer_kv_mamba_lf_pf (12×)
Jul 17	29601593052	nightly-test-1-gpu-kernel	❌ Failed	all 12 test_mamba_kernel_*	NameError: transfer_kv_mamba_lf_pf (12×)
Jul 16	29521595306	nightly-test-1-gpu-kernel	❌ Failed	all 12 test_mamba_kernel_*	NameError: transfer_kv_mamba_lf_pf (12×)
≤ Jul 15	—	nightly-test-1-gpu-kernel	✅ n/a	—	test file did not exist yet

First observed failure: 2026-07-16 (first AMD nightly after #31227 merged). No passing run for this test file has ever existed on AMD.

Failure Cluster

NameError: mamba JIT transfer kernel unimported on AMD (transfer_kv_mamba_lf_pf/transfer_kv_mamba_pf_lf)

Facts (verified observations, no interpretation)

• Verbatim error (job log line 4612): E NameError: name 'transfer_kv_mamba_lf_pf' is not defined, raised at ../python/sglang/srt/mem_cache/memory_pool_host.py:394: NameError. Stack: test line 218 → memory_pool_host.py:464 backup_from_device_all_layer → _copy_tensor_all_layers_lf_pf → line 394.
• In python/sglang/srt/mem_cache/memory_pool_host.py, the mamba JIT kernel names are imported under a CUDA-only guard (lines 41-45):

  if _is_cuda:
      from sglang.jit_kernel.transfer_mamba import (
          transfer_kv_mamba_lf_pf,
          transfer_kv_mamba_pf_lf,
      )
  

  _is_cuda = is_cuda() / _is_hip = is_hip() (lines 26-27). On the AMD MI325 runner _is_cuda is False, so neither name is ever bound.
• The io_backend == "kernel" branch (line 386) unconditionally calls transfer_kv_mamba_lf_pf(...) (line 394); the load path calls transfer_kv_mamba_pf_lf(...) (line 351). Neither call is gated by backend. The test forces io_backend="kernel" (test line 218-220).
• git blame confirms lines 41-45, 386-394 all originate from commit 7d0fd5101 (PR #31227, 2026-07-16). git log --diff-filter=A confirms the test file was first added by the same commit.
• python/sglang/jit_kernel/transfer_mamba.py contains no HIP/ROCm handling (grep for hip/rocm/HIP returns nothing); the JIT module is built via load_jit(..., cuda_files=["kvcacheio/transfer_mamba.cuh"], ...).
• Runner: linux-mi325-1gpu-sglang (AMD MI325, gfx942/mi30x). aiter dev image preserved at 9127c94a1 (unchanged; not overridden).
• Reproducibility: 12/12 parametrizations fail, identical error, in all 3 AMD nightly runs since the test landed (Jul 16/17/18).

Hypothesised Causes (with confidence, with disconfirming evidence)

• [FACT] The mamba JIT transfer kernel import is gated on if _is_cuda: (memory_pool_host.py:41), but the io_backend="kernel" code path (lines 351, 394) executes on AMD/HIP where _is_cuda is False, leaving transfer_kv_mamba_lf_pf/transfer_kv_mamba_pf_lf unbound → NameError. Both the guard and the call site were added by PR #31227 (7d0fd5101).
  • Supporting: the NameError names exactly the symbol that is only imported under _is_cuda; stack trace lands on line 394, the first call site; the same commit added the CUDA-only guard, the call, and the test.
  • Disconfirming: (none found) — the error is directly observable in both code and log.
  • How to verify: already conclusive from source; to confirm intent, decide whether the mamba JIT kernel is meant to support ROCm. If yes, broaden the guard to if _is_cuda or _is_hip: and confirm sglang.jit_kernel.transfer_mamba builds on gfx942. If the kernel is CUDA-only by design, the test should not force io_backend="kernel" on AMD (skip/xfail on ROCm) or the AMD kernel path needs a HIP implementation.
• [LOW] The kernel may be genuinely CUDA-only (no ROCm build exists), i.e. even fixing the import would surface a compile/launch failure on gfx942.
  • Supporting: transfer_mamba.py has no HIP handling and load_jit references only .cuh.
  • Disconfirming: not testable until the import guard is fixed; the sibling sgl_kernel.kvcacheio imports (lines 32-40) are gated on _is_cuda or _is_hip, suggesting the surrounding subsystem is intended to work on HIP.
  • How to verify: after broadening the guard, run test_transfer_mamba.py on MI325 and check whether JIT compilation of transfer_mamba.cuh succeeds under ROCm.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for transfer_kv_mamba (0 results) and transfer_mamba (0 results); a broader mamba hicache search (54 PRs) surfaced no PR addressing the AMD import guard. Needs a new PR or investigation.

Suggested Triage Steps

• Confirm whether sglang.jit_kernel.transfer_mamba is intended to run on ROCm. If yes: broaden the import guard at memory_pool_host.py:41 from if _is_cuda: to if _is_cuda or _is_hip: (mirroring the sgl_kernel.kvcacheio guard at lines 31-40), then re-run test/registered/jit/test_transfer_mamba.py on an AMD MI325 runner.
• If the mamba JIT kernel is CUDA-only by design: mark test_transfer_mamba.py's io_backend="kernel" parametrizations as skip/xfail on HIP (or gate the test registration off AMD) so the AMD nightly signal isn't permanently red, and track ROCm kernel support separately.
• Ping the author of PR #31227 (7d0fd5101) — it added the guard, the call sites, and the test in one change without an AMD path.

Status (factual, no priority assignment)

• Persistence: Failing since 2026-07-16 (first AMD nightly after PR #31227 merged); every comparable completed scheduled run since (Jul 16, 17, 18) shows the identical 12× NameError. Test file has never passed on AMD.
• Scope: nightly-test-1-gpu-kernel job of nightly-test-amd.yml; 12/12 parametrizations of one test file. Any AMD/HIP code path that reaches MambaPoolHost backup/load with io_backend="kernel" (HiCache mamba write-back) would hit the same NameError at runtime, not only the test.
• Blocked work: Mamba HiCache kernel-backend transfer signal on AMD.
• In-flight fix: ❌ none found — needs new PR or investigation.

nightly-accuracy-2-gpu-vlm — failed step(s): Nightly Accuracy Test (2-GPU VLM MMMU)

Commit Info

• sglang: b3a0185ca — "model_runner: extract post-memory-pool wiring into _init_post_memory (#31601)"
• aiter: 9127c94a1 (v0.1.17.dev110+g9127c94a1) — source: dev image preserved ([CI-AITER-CHECK] Dev/patched version detected → skipping rebuild)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi30x/test_vlms_mmmu_eval_amd.py	TestNightlyVLMMmmuEvalAMD.test_mmmu_vlm_models	5/11 VLM models below MMMU accuracy threshold → exit 1	Log

The single test method test_mmmu_vlm_models iterates 11 VLMs; 5 sub-cases scored below their configured accuracy_threshold:

Model	TP	Score	Threshold	Verdict
Qwen/Qwen2-VL-7B-Instruct	1	0.280	0.30	❌
Qwen/Qwen2.5-VL-7B-Instruct	1	0.280	0.33	❌
Qwen/Qwen3-VL-30B-A3B-Instruct	2	0.260	0.29	❌
openbmb/MiniCPM-o-2_6	1	0.280	0.32	❌
deepseek-ai/deepseek-vl2-small	1	0.280	0.31	❌

Failure Summary

The AMD nightly VLM MMMU accuracy test (test_vlms_mmmu_eval_amd.py) failed with exit code 1 because 5 of 11 models scored below threshold on the 100-sample MMMU eval (final tally: 6 passed / 5 failed). Notably, all 11 models — including the 6 "passing" ones — score in a narrow 0.26–0.34 band, which is at or near MMMU 4-choice random-chance (~0.25); the passing models pass only because their thresholds are set at 0.25–0.29.

Regression Status

Never-passed (no green run found in query window) — this is a long-standing recurring failure, NOT a regression introduced by the current commit.

Recent history of test_vlms_mmmu_eval_amd.py (test_mmmu_vlm_models) in job nightly-accuracy-2-gpu-vlm (nightly-test-amd.yml, event=schedule, branch=main, completed runs, verified at test-file level by downloading each log):

Date	Run	Test File Status	Failed Count	Notes
Jul 18	29654601256	❌ Failed	5/11	this run
Jul 17	29601593052	❌ Failed	5/11	same 5 models
Jul 16	29521595306	❌ Failed	4/11	Qwen2-VL hit exactly 0.300 → passed
Jul 14	29355460805	❌ Failed	5/11	same 5 models
Jul 14	29273241962	❌ Failed	5/11	same 5 models
Jul 05	28749559519	❌ Failed	—	job failed
Jul 04	28714602438	❌ Failed	—	job failed
Jul 03	28676359018	❌ Failed	5/11	same 5 models
Jun 28	28330859861	❌ Failed	5/11	same 5 models

Lookback window searched: 2026-06-28 → 2026-07-18 (20+ scheduled runs). No run in the window shows a green test_vlms_mmmu_eval_amd.py. The failing set is stable, and per-model scores are nearly deterministic run-to-run (e.g. Qwen3-VL-30B = 0.260 in every run; deepseek-vl2-small = 0.280 in every run).

Failure Cluster

VLM MMMU accuracy pinned at random-chance level across all models on AMD MI325 (nightly-accuracy-2-gpu-vlm)

Facts (verified observations)

• Exit path: ✗ FAILED: .../test_vlms_mmmu_eval_amd.py returned exit code 1 → First attempt failed with exit code 255 → Exit code 255 indicates test failure (not network issue), not retrying → Process completed with exit code 255 (job log tail).
• Final tally this run: Passed: 6 / Failed: 5 after 3226.7s total eval runtime.
• All 11 models score within 0.26–0.34 regardless of size/capability (Qwen3-VL-30B and DeepSeek-OCR land in the same band as 2B InternVL). A correctly functioning Qwen2.5-VL-7B typically scores ~0.45–0.55 on MMMU — these scores are at/near the ~0.25 random-chance floor for 4-choice MC.
• Scores are near-deterministic across 3+ weeks of runs (same 5 models fail with the same scores), so this is not run-to-run flakiness.
• The 6 "passing" models pass only because thresholds are 0.25–0.29 (below random chance); the 5 "failing" models have thresholds 0.30–0.33.
• Test config test_vlms_mmmu_eval_amd.py:43-123 sets per-model accuracy_threshold (comment: "AMD-verified VLM models with conservative thresholds on 100 MMMU samples").
• aiter is unchanged (dev image v0.1.17.dev110+g9127c94a1 preserved; no rebuild) — not a variable here.
• No SGLang server crash/traceback in the log; servers start, serve requests, and eval completes — the failure is purely the accuracy-vs-threshold gate.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [MEDIUM] Systemic eval-harness / vision-path correctness issue on AMD — the models are effectively answering MMMU at random (~0.28), suggesting images are not being properly consumed or the MMMU answer-extraction/grading is mis-parsing outputs, rather than 5 independent per-model accuracy regressions.
  • Supporting: 11/11 models cluster at random-chance regardless of model quality; scores are deterministic across 3+ weeks; even strong 30B/7B models don't rise above ~0.28.
  • Disconfirming: no server error or image-decode exception appears in the log (eval "succeeds" and reports scores); it is possible 100 samples is simply a very noisy/low-signal eval and the thresholds were mis-calibrated (see next hypothesis).
  • How to verify: run one model (e.g. Qwen2.5-VL-7B) locally on AMD with the same harness, dump a few raw model responses + parsed answers for MMMU items, and confirm whether the model is actually seeing the image and whether answers are being extracted correctly; compare the same run on NVIDIA.
• [LOW] Thresholds set too high relative to achievable 100-sample MMMU scores on this AMD stack (a pure test-config calibration issue).
  • Supporting: the 5 failing models miss by only ~0.02–0.05; thresholds (0.30–0.33) sit just above the observed ~0.28.
  • Disconfirming: if the harness were healthy, real VLMs should score far above 0.33 on MMMU, so lowering thresholds would mask a deeper problem rather than fix it; the near-random band across ALL models points upstream of the thresholds.
  • How to verify: check whether these same models/thresholds pass on the NVIDIA nightly (nightly-test.yml) — if NVIDIA scores ~0.45+, the thresholds are fine and the AMD vision path is the real issue.
• [SPECULATION] Current commit b3a0185ca (_init_post_memory refactor, #31601) is not implicated — the identical failure predates it by 3+ weeks (back to at least 2026-06-28), so it cannot be the cause. No bisect is warranted.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs repo:sgl-project/sglang is:pr is:open MMMU VLM and ... vlms mmmu accuracy; results were unrelated VLM feature PRs (e.g. #29237, #20548, #26606), none targeting the AMD MMMU accuracy thresholds or this test.

Suggested Triage Steps

• Compare against the NVIDIA VLM MMMU nightly: if the same models score ~0.45+ on NVIDIA, the AMD vision/eval path is broken (favor the [MEDIUM] hypothesis); if NVIDIA is also near-random, it's a shared harness issue.
• Locally reproduce one failing model (Qwen/Qwen2.5-VL-7B-Instruct) on AMD MI325 in the nightly container and dump raw model outputs + parsed MMMU answers to confirm whether the image is being consumed and answers extracted correctly.
• If the harness is confirmed healthy but scores are genuinely ~0.28, treat test_vlms_mmmu_eval_amd.py:43-123 thresholds as mis-calibrated and re-baseline them from AMD-observed scores (test-config change, human decision) — but only after ruling out the vision-path issue.
• Because this test has never been green in the queryable window, consider quarantining/marking it as a known-failing AMD accuracy gate until the root cause is resolved, so it stops masking new regressions in nightly-accuracy-2-gpu-vlm.

Status

• Persistence: Failing every completed scheduled run in the queryable window (≥20 runs, since at least 2026-06-28, ~3 weeks); never observed green.
• Scope: nightly-accuracy-2-gpu-vlm job in nightly-test-amd.yml; the sister nightly-perf-2-gpu-vlm job passes, so this is an accuracy-gate failure only, not a serving/crash failure.
• Blocked work: Blocks the AMD nightly VLM accuracy signal — the whole job is red every night, so genuine new VLM accuracy regressions on AMD would be hidden.
• In-flight fix: ❌ none found in open PRs — needs investigation (vision-path vs threshold calibration) or a new PR.

nightly-accuracy-2-gpu — failed step(s): N/A

Commit Info

• sglang: b3a0185ca
• aiter: 9127c94a1 — v0.1.17.dev110+g9127c94a1 (source: dev image preserved; matches Dockerfile default, no override — AITER_COMMIT_OVERRIDE empty)

Failed Tests

The job runs one unittest, TestNightlyGsm8KEval.test_gsm8k_all_models, in test/registered/amd/accuracy/mi30x/test_gsm8k_eval_amd.py, which loops over models as sub-tests. Per-model status from the log:

Test File	Sub-test (model)	Score / Threshold	Status	Log
test/registered/amd/accuracy/mi30x/test_gsm8k_eval_amd.py	mistralai/Mixtral-8x7B-Instruct-v0.1 (TP=2)	0.292 / 0.69	❌ real accuracy fail	Log
test/registered/amd/accuracy/mi30x/test_gsm8k_eval_amd.py	Qwen/Qwen3-4B (TP=1)	0.912 / N/A	❌ cosmetic (no threshold configured)	Log

Passing models (same run, for contrast): Llama-3.1-8B 0.819/0.80 ✅, Qwen3-8B 0.916/0.76 ✅, Llama-3.1-70B 0.932/0.89 ✅.

Failure Summary

The nightly-accuracy-2-gpu job fails the gsm8k accuracy suite. mistralai/Mixtral-8x7B-Instruct-v0.1 scores 0.292 against a 0.69 threshold — the only model with a real accuracy miss. Separately, Qwen/Qwen3-4B prints Status: ❌ FAILED but this is cosmetic: it has no entry in MODEL_SCORE_THRESHOLDS, so passed = threshold and score >= threshold evaluates to None. The returned job log is truncated by volume (massive [aiter] shape is … GEMM spam from the Mixtral MoE run) at line ~18.8k, right after Mixtral's first attempt, so the final summary table / AssertionError and the subsequent Qwen2-57B-A14B model are not visible in the fetched log.

Regression Status

Known recurring failure (job red every completed run since the test was added); Mixtral has never passed on this job.

The test file was created on 2026-07-16 by 7d0fd5101 (#31227), so only two prior completed scheduled runs are comparable at the test-file level. Underlying Mixtral gsm8k mis-scoring predates this by 4+ months on the NVIDIA h100 nightly (see In-flight Fix).

Recent history of test_gsm8k_eval_amd.py in job nightly-accuracy-2-gpu (nightly-test-amd.yml, event=schedule, branch=main):

Date	Run	Job status	Mixtral / Qwen3-4B in log
Jul 18	29654601256 [IN-FLIGHT run]	❌ failure	Mixtral 0.292; Qwen3-4B cosmetic ❌
Jul 17	29601593052	❌ failure	log truncated at Mixtral; Qwen3-4B cosmetic ❌
Jul 16	29521595306	❌ failure	log truncated at Mixtral; Qwen3-4B cosmetic ❌
Jul 15	29438351311	⏹ cancelled	not comparable
Jul 14	29355460805	❌ failure	pre-migration (different test file)

Note: in all three post-migration runs, the log cuts off at the start of the Mixtral eval and Qwen2-57B-A14B never appears — a consistent artifact of log-volume truncation (and step 6 is left in_progress in the jobs API even for jobs completed a day ago).

Failure Cluster

gsm8k accuracy failure on verbose MoE models (Mixtral) + missing-threshold config (Qwen3-4B)

Facts (verified observations)

• mistralai/Mixtral-8x7B-Instruct-v0.1 gsm8k_score = 0.2922374429223744, threshold 0.69 (current run, log L18118-18126).
• Qwen/Qwen3-4B printed Score: 0.912 (threshold: N/A) → Status: ❌ FAILED (log L12565/L12572); it ran 3 eval attempts (0.913, 0.910, 0.912) because the retry loop if threshold and score >= threshold never breaks when threshold is None.
• MODEL_SCORE_THRESHOLDS in test_gsm8k_eval_amd.py:37-71 contains Qwen/Qwen3-8B and Qwen/Qwen3-30B-A3B-Thinking-2507 but no Qwen/Qwen3-4B, while Qwen/Qwen3-4B is in the eval list DEFAULT_MODEL_NAME_FOR_NIGHTLY_EVAL_TP1 (python/sglang/test/test_utils.py:154).
• In check_model_scores (test_gsm8k_eval_amd.py:158-201), a None threshold maps to ⚠️ NO THRESHOLD and is not appended to failed_models, so Qwen3-4B alone would not raise; only models scoring below a defined threshold (i.e. Mixtral) trigger the AssertionError.
• gsm8k answer extraction get_answer_value (python/sglang/test/simple_eval_mixed_prefix_gsm8k.py:35-41) returns numbers[-1] — the last number in the response.
• Aiter is unchanged vs the Dockerfile default (no override); Llama/Qwen3 models pass on the same aiter build.

Hypothesised Causes (with confidence)

• [MEDIUM] Mixtral 0.292 is a gsm8k answer-extraction issue, not a model/AMD regression. get_answer_value takes numbers[-1]; verbose CoT models (Mixtral/Mistral) don't reliably end on the final answer, so the wrong number is extracted and the score collapses.
  • Supporting: PR #29933 documents this exact family (Mixtral-8x7B, Mistral-7B, FP8 variants) as "red for 4+ months" on the NVIDIA nightly-test-text-accuracy-2-gpu-h100 job since the mgsm→gsm8k migration (#21931); the extractor code matches the described bug.
  • Disconfirming: I could not read Mixtral's per-example transcripts (log truncated), so a genuine AMD MoE output-quality issue cannot be fully excluded; however the identical failure on NVIDIA h100 argues against an AMD-specific cause.
  • How to verify: rerun the Mixtral gsm8k eval with PR #29933 applied and compare score; or inspect /tmp/gsm8k_mistralai_Mixtral-8x7B-Instruct-v0.1.json transcripts.
• [FACT] Qwen3-4B "FAILED" is a missing-threshold config gap, not an accuracy failure. Score 0.912 is healthy; the model simply has no threshold entry.
  • Disconfirming: (none — verified directly in code and log).
  • How to verify: add "Qwen/Qwen3-4B": <thr> to MODEL_SCORE_THRESHOLDS, or remove it from the TP1 eval list.
• [LOW] The job may additionally hang/be killed after Mixtral (during Qwen2-57B-A14B). Step 6 is left in_progress and post-steps never run across all three completed runs, and Qwen2-57B never appears in any log.
  • Supporting: ~20 min of no captured output before job termination; step 6 never concludes cleanly.
  • Disconfirming: this is equally explained by GitHub truncating the high-volume aiter log; a clean AssertionError from Mixtral would also fail the job. Cannot confirm without the full log.
  • How to verify: download the full log from the Actions UI (zip) or GITHUB_STEP_SUMMARY for run 29654601256 once complete, and check whether Qwen2-57B-A14B and the summary table were reached.

In-flight Fix Check

• Mixtral gsm8k mis-scoring: ✅ #29933 "fix(eval): extract GSM8K answer from first #### marker, not last number" (open since 2026-07-02) — directly targets this cluster; chase reviewers, do NOT open a duplicate. (searched Mixtral+accuracy, gsm8k+threshold+amd.)
• Qwen3-4B missing threshold: ❌ none found in open PRs — needs a one-line config addition. (searched Qwen3-4B+threshold.)

Suggested Triage Steps

• Review/merge in-flight fix #29933, then re-run nightly-accuracy-2-gpu and check the Mixtral gsm8k score recovers toward ~0.74.
• Add a "Qwen/Qwen3-4B" entry to MODEL_SCORE_THRESHOLDS (test_gsm8k_eval_amd.py:37) — or drop it from DEFAULT_MODEL_NAME_FOR_NIGHTLY_EVAL_TP1 — to stop the cosmetic NO THRESHOLD "FAILED" and the wasteful 3× retry.
• Obtain the full job log (Actions UI zip / step summary), which the API truncates due to aiter GEMM spam, to confirm the final AssertionError list and whether Qwen2-57B-A14B ran or hung.
• Consider quieting [aiter] shape is … not found tuned config … using torch solution:0 logging for MoE models — the volume is truncating CI logs and hiding the real summary.

Status

• Persistence: nightly-accuracy-2-gpu red every completed scheduled run since the test was added on 2026-07-16 (07-16, 07-17, and the in-flight 07-18 run). Mixtral gsm8k mis-scoring itself is 4+ months old cross-vendor (per #29933).
• Scope: this job on nightly-test-amd.yml; the same Mixtral/Mistral eval issue also reds nightly-test-text-accuracy-2-gpu-h100 (NVIDIA).
• Blocked work: AMD 2-GPU nightly gsm8k accuracy signal; the real Mixtral miss is masked by a known eval-harness bug and by a Qwen3-4B config gap.
• In-flight fix: ✅ #29933 (Mixtral eval extraction); ❌ none for Qwen3-4B threshold.

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-07-19 22:57 UTC)

[amd-bot]

nightly-test-amd.yml — Per-Job Analysis (continued 2/3)

nightly-2-gpu-mi35x-glm51-mxfp4 — failed step(s): N/A

Commit Info

• sglang: b3a0185ca — model_runner: extract post-memory-pool wiring into _init_post_memory (#31601)
• aiter: v0.1.17.dev110+g9127c94a1 (source: dev image, [CI-AITER-CHECK] says "Dev/patched version detected → skipping rebuild"). ⚠️ Not confirmable from the current run's log (blob unavailable, see below); taken from the two most recent comparable runs (07-06 pass, 07-16 fail) where it is identical.

⚠️ Log availability caveat: run 29654601256 is still in_progress (one sibling job unfinished), so GitHub has not finalized this job's per-log blob — jobs/88106575932/logs returns BlobNotFound. The current-run classification below is built from the check-run annotation (GitHub-provided), the step conclusions, and timing, not the raw test log. All historical (regression) analysis IS from full logs.

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi35x/test_glm51_mxfp4_tp2_gsm8k_mi35x.py	TestGLM51MXFP4TP2GSM8KMI35x (server launch in setUpClass)	The self-hosted runner lost communication with the server … starves it for CPU/Memory …	Job

The GSM8K eval function test_gsm8k never executed — the failure is upstream of the test body, in server launch / runner infrastructure.

Failure Summary

The single test in this job (test_glm51_mxfp4_tp2_gsm8k_mi35x.py) failed because the self-hosted linux-mi35x-gpu-2 runner lost communication with GitHub during step #6 (the GLM-5.1-MXFP4 GSM8K accuracy test). Steps #6 (test) and #12 (Post Checkout) both show conclusion None (never reported completion), and the job ran only ~22 min (17:47:39→18:09:40 UTC) — consistent with the runner agent dying mid-launch rather than a test assertion. This is an infrastructure/hardware event, not an accuracy failure and not a Python-level test failure.

Regression Status

Known recurring failure — infrastructure/hardware cluster (not a code regression). This job runs exactly one test file, so job status = test-file status. The test passed 07-04/05/06 (score ~0.94) and has failed every completed run since 07-07, each via a different server-launch/hardware symptom.

Recent history of test_glm51_mxfp4_tp2_gsm8k_mi35x.py in job nightly-2-gpu-mi35x-glm51-mxfp4 (most recent completed event=schedule, branch=main runs):

Date	Run	Test File Status	Failure mode (verified from log unless noted)
07-18	29654601256	❌ Failed	Runner lost communication (annotation; log blob N/A) — current
07-17	29601593052	❌ Failed	log blob 404 — mode not verifiable
07-16	29521595306	❌ Failed	TimeoutError: Server failed to start — stuck in HF Hub download, 5405s
07-15	29438351311	⏭️ N/A	Job cancelled (excluded)
07-14	29355460805	❌ Failed	HW Exception … reason :GPU Hang, exit 134
07-13	29273241962	❌ Failed	HW Exception … reason :GPU Hang, exit 134
07-12	29202683625	❌ Failed	job-level failure (mode not individually verified)
07-11	29162254653	❌ Failed	job-level failure (mode not individually verified)
07-10	29113206892	❌ Failed	job-level failure (mode not individually verified)
07-09	29039566002	❌ Failed	TimeoutError: Server failed to start — HF download hang, 5411s
07-08	28964511688	❌ Failed	Pre-flight "VRAM usage exceeds threshold" — zombie KFD context (ROCm/aiter#2061)
07-07	28888348134	❌ Failed	Server process exited with code -9 (SIGKILL/OOM), 45s
07-06	28813271245	✅ Passed	gsm8k_score=0.9399 (≥0.92)
07-05	28749559519	✅ Passed	—
07-04	28714602438	✅ Passed	—

• First observed failure (this streak): 2026-07-07. Last known passing: 2026-07-06.
• Test function test_gsm8k has run 0 times in every failure — all failures are in setUpClass (server launch) or pre-flight, never in the eval assertion.

Failure Cluster

GLM-5.1-MXFP4 server-launch failure on linux-mi35x-gpu-2 runner (GPU hang / OOM / zombie-KFD VRAM leak / HF-download timeout / runner comms loss) — a symptom-varied infrastructure/hardware grouping, not a single root cause.

Facts (verified observations)

• Current job: check-run annotation = "The self-hosted runner lost communication with the server. … Anything in your workflow that terminates the runner process, starves it for CPU/Memory, or blocks its network access can cause this error." (FACT, GitHub-provided).
• Current job steps [CI Monitor] Daily Report - 2026-03-24 #6 (Accuracy Test MI35x (2-GPU GLM-5.1-MXFP4 GSM8K)) and [CI Monitor] Daily Report - 2026-03-31 #12 (Post Checkout code) both have conclusion None; job duration 17:47:39→18:09:40 UTC (~22 min).
• Runner: linux-mi35x-gpu-2 (linux-mi35x-gpu-2-glhqm-runner-vxfh6).
• aiter version identical across passing (07-06) and failing (07-16) runs: v0.1.17.dev110+g9127c94a1 — not a variable.
• Distinct hardware/infra symptoms across the streak: -9 OOM (07-07), zombie-KFD VRAM-not-clear (07-08), HF-download launch timeout (07-09, 07-16), GPU Hang exit 134 (07-13, 07-14).
• When the server launches, accuracy is healthy: 0.9399 on 07-06 vs 0.92 threshold.

Hypothesised Causes (with confidence)

• [HIGH] Current failure is a self-hosted runner/hardware infrastructure event, not sglang code.
  • Supporting: GitHub's own annotation states the runner lost communication (CPU/Memory starvation or machine death); steps [CI Monitor] Daily Report - 2026-03-24 #6/[CI Monitor] Daily Report - 2026-03-31 #12 never reported completion; job died at ~22 min mid-launch; part of a 10-run streak of varied hardware symptoms on the same MI35x-2GPU runner while aiter and (mostly) sglang test code are unchanged for this test.
  • Disconfirming: The current run's raw test log is unavailable, so a same-box OOM triggered by heavy model load (i.e. a workload contribution) cannot be fully excluded — the 07-07 -9 OOM shows this box can be pushed into memory starvation during GLM-5.1-MXFP4 TP2 launch.
  • How to verify: once run 29654601256 completes, re-pull jobs/88106575932/logs and inspect for -9/OOM/GPU-hang before the comms loss; check runner host dmesg/agent logs.
• [LOW] Head commit b3a0185ca (_init_post_memory model_runner refactor) contributed.
  • Supporting: it is the run's head SHA and touches memory-pool wiring.
  • Disconfirming: the failure signature is a runner-comms-loss infra event with no Python traceback attributable to it; the identical job has failed on unrelated hardware symptoms (GPU hang, HF timeout, zombie KFD) across 10 prior runs with many different head SHAs — the streak long predates this commit.
  • How to verify: not actionable as a code hypothesis until the infra symptom is ruled out.

In-flight Fix Check

Searched open PRs with mi35x+runner and GLM-5.1-MXFP4+mi35x.

• In-flight fix: ✅ [#31068](https://github.com/sgl-project/sglang/pull/31068) (open since 2026-07-13) — "[AMD] ci: detect and recover hung GPUs in MI35x pre-flight" — directly targets the GPU-hang / VRAM-not-clear class in this cluster.
• In-flight fix: ✅ [#30431](https://github.com/sgl-project/sglang/pull/30431) (open since 2026-07-07, marked Not-Merge) — "[AMD][CI] self-heal driver-level VRAM leaks + graceful GPU cleanup on shutdown" — targets the zombie-KFD VRAM leak seen on 07-08.
• No open PR found specific to GLM-5.1-MXFP4 accuracy (count 0) — expected, since this is not an accuracy failure.

Suggested Triage Steps

• Once the run completes, re-download jobs/88106575932/logs and confirm whether the comms loss was preceded by an OOM (-9), GPU Hang (exit 134), or HF-download stall — this determines which sub-symptom the current run belongs to.
• Track / chase reviewers on #31068 and #30431; they address the GPU-hang and VRAM-leak sub-symptoms driving most of this streak. Do NOT open duplicates.
• Inspect the linux-mi35x-gpu-2-glhqm-runner-vxfh6 host (dmesg, ROCm SMI, agent logs) for OOM-killer / GPU driver resets around 2026-07-18 18:00–18:10 UTC.
• If the GLM-5.1-MXFP4 TP2 launch is genuinely near the memory ceiling on this 2-GPU box, evaluate lowering --mem-fraction-static (currently 0.85) or rebalancing this job onto a healthier MI35x runner (cf. related #30639 rebalancing effort) — as a triage experiment, not a directive.

Status

• Persistence: Failing every completed scheduled run since 2026-07-07 (10 completed failures; 07-15 cancelled), after passing 07-04/05/06.
• Scope: Single job nightly-2-gpu-mi35x-glm51-mxfp4 on runner linux-mi35x-gpu-2; failures are server-launch/hardware-level, symptom-varied.
• Blocked work: No GLM-5.1-MXFP4 TP2 GSM8K accuracy signal on MI35x since 07-06 (the eval never runs).
• In-flight fix: ✅ #31068 (GPU-hang recovery) and ✅ #30431 (VRAM-leak self-heal) — both open, both target sub-symptoms of this cluster.

nightly-8-gpu-mi35x-qwen3-235b-mxfp4 — failed step(s): Accuracy Test + Performance Test MI35x (8-GPU Qwen3-235B-MXFP4)

Commit Info

• sglang: b3a0185ca
• aiter: 9127c94a1 — v0.1.17.dev110+g9127c94a1 (source: Dockerfile default; dev/patched image detected → rebuild skipped, AITER_COMMIT_OVERRIDE empty)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/test_qwen3_instruct_mxfp4.py	TestQwen3Instruct2507MXFP4.setUpClass	Exception: Server process exited with code 1 (server crashed during CUDA graph capture with HIP error: operation not permitted on an event last recorded in a capturing stream / hipErrorCapturedEvent)	Log

Failure Summary

setUpClass of TestQwen3Instruct2507MXFP4 failed because popen_launch_server never reached readiness: the SGLang server subprocess aborted during decode CUDA-graph capture. The abort is a hipErrorCapturedEvent ("operation not permitted on an event last recorded in a capturing stream") raised while the aiter custom all-reduce communicator flushed its IPC graph buffers. Ran 0 tests in 430.399s / FAILED (errors=1); the harness reported exit code 255. Server command: sglang serve --model-path amd/Qwen3-235B-A22B-Instruct-2507-mxfp4 --tp 4 --ep 2 --trust-remote-code --attention-backend aiter --device cuda.

Regression Status

Known recurring failure / Never-passed (no green run found in window).

Recent history of test/registered/amd/test_qwen3_instruct_mxfp4.py (TestQwen3Instruct2507MXFP4.setUpClass) in job nightly-8-gpu-mi35x-qwen3-235b-mxfp4 (most recent comparable completed scheduled runs: branch=main, event=schedule; runs where the job was cancelled are excluded):

Date	Run	Job	Test File Status	Failed Function	Error
Jul 18	run	job	❌ Failed	setUpClass	hipErrorCapturedEvent during graph capture (this run)
Jul 17	run	job	❌ Failed	setUpClass	Same hipErrorCapturedEvent + _gather_ipc_meta crash
Jul 16	run	job	⏭️ Cancelled	—	job cancelled (excluded from verdict)
Jul 14	run	job	❌ Failed	setUpClass	Different: GPU Hang → exit 134
Jul 13	run	job	❌ Failed	setUpClass	Different: GPU Hang → exit 134
Jul 5	run	job	❌ Failed	(setup)	Different: VRAM cleanup unsuccessful (infra)
Jul 3	run	job	❌ Failed	setUpClass	Same hipErrorCapturedEvent graph-capture crash

No passing run for this test file exists in the queryable window (Jul 3 – Jul 18) → Never-passed. The specific graph-capture-crash signature recurs on Jul 3, Jul 17, and Jul 18; other runs failed for unrelated reasons (GPU Hang, VRAM cleanup). Runs Jul 6–12 and Jul 15 were fully cancelled and are not comparable.

Failure Cluster

hipErrorCapturedEvent in aiter custom all-reduce during decode CUDA-graph capture

Facts (verified observations, no interpretation)

• Server-side crash stack (from Fatal Python error: Aborted, job log ~L5464–5488):
  • aiter/dist/device_communicators/custom_all_reduce.py:331 _gather_ipc_meta
  • aiter/dist/device_communicators/custom_all_reduce.py:303 flush_graph_buffers
  • aiter/dist/device_communicators/custom_all_reduce.py:503 capture
  • python/sglang/srt/distributed/parallel_state.py:576 graph_capture (calls ca_comm.capture() — verified at parallel_state.py:570)
  • .../parallel_state.py:1840 graph_capture
  • .../model_executor/runner/decode_cuda_graph_runner.py:852 / :407
  • .../model_executor/model_runner_components/cuda_graph_setup.py:333 / :106
  • .../model_executor/model_runner.py:838 init_cuda_graphs
• Watchdog thread error (job log L5382): [rank0] ... ProcessGroupNCCL ... HIP error: operation not permitted on an event last recorded in a capturing stream; same on rank3 (L5490). Crash occurred at Capturing batches (bs=512 avail_mem=25.14 GB): 0%.
• The crash happened during graph capture, before any test body ran (Ran 0 tests, elapsed 430s of a 3600s budget).
• aiter version verified via [CI-AITER-CHECK] markers (job log L4330–4336): v0.1.17.dev110+g9127c94a1, Dockerfile default AITER_COMMIT=9127c94a..., no override.
• Reproducibility: identical signature (_gather_ipc_meta + hipErrorCapturedEvent) in Jul 17 job and Jul 3 job.
• Config: --tp 4 --ep 2 --attention-backend aiter on 8-GPU MI35x (gfx950); model amd/Qwen3-235B-A22B-Instruct-2507-mxfp4.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [MEDIUM] The aiter custom all-reduce capture() context manager performs a disallowed event/IPC operation during HIP graph capture. On __exit__, capture → flush_graph_buffers → _gather_ipc_meta synchronizes/records an event while the stream is mid-capture, which HIP rejects with hipErrorCapturedEvent.
  • Supporting: crash is entirely inside aiter/.../custom_all_reduce.py (_gather_ipc_meta is the innermost frame); the error string is HIP's exact message for event ops during capture; sglang's role is only entering the ca_comm.capture() context at parallel_state.py:576.
  • Disconfirming: aiter version is a dev image and is not confirmed to have changed between the Jul 3 (crash) and Jul 5/13/14 (GPU Hang / VRAM) runs, so aiter alone does not explain why the signature is intermittent; the alternating GPU-Hang / VRAM-cleanup failures suggest an additional unstable-runner/OOM factor for this large model on MI35x.
  • How to verify: on a branch, launch the same server command with the aiter custom all-reduce disabled (e.g. --disable-custom-all-reduce or equivalent) and confirm graph capture completes; A/B the current dev-image aiter vs a pinned aiter build.
• [LOW] MI35x/gfx950 runner instability contributes (GPU Hang and VRAM-cleanup failures on the same test file on Jul 5/13/14).
  • Supporting: three of the sampled failing runs are hardware/infra symptoms, not the capture crash.
  • Disconfirming: the Jul 3/17/18 crashes are a deterministic HIP API error, not a hang or OOM — a genuinely flaky runner would not reproduce the same stack three times.
  • How to verify: retry the job on a different MI35x node; check dmesg/rocm-smi for the hang runs.
• This is nightly-test-amd.yml, not amd-aiter-scout.yml, so no baseline A/B (Failure Origin) classification is required.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for hipErrorCapturedEvent (0 results), flush_graph_buffers (0), custom_all_reduce graph capture (only unrelated PRs #21098, #24489), and qwen3 mxfp4 amd (quantization PRs, none addressing graph-capture crash). No matching fix in flight.

Suggested Triage Steps

• Re-run test/registered/amd/test_qwen3_instruct_mxfp4.py with the aiter custom all-reduce path disabled during graph capture; if capture succeeds, the crash is isolated to aiter custom_all_reduce.flush_graph_buffers/_gather_ipc_meta.
• File/track an aiter issue for the _gather_ipc_meta event operation running inside a HIP graph capture on gfx950 (cite aiter 9127c94a1, custom_all_reduce.py:331).
• Separately investigate the GPU-Hang (exit 134) and VRAM-cleanup failures on Jul 13/14/5 as a possible MI35x-node/OOM issue for this 235B model — likely a distinct problem from the capture crash.
• Since this test file has never passed on MI35x in the queryable window, consider quarantining it on MI35x until both the aiter graph-capture crash and the runner-stability issues are resolved.

Status (factual, no priority assignment)

• Persistence: The hipErrorCapturedEvent graph-capture signature reproduced on Jul 3, Jul 17, and Jul 18 (3 completed runs); the test file test_qwen3_instruct_mxfp4.py has never passed in the Jul 3–18 window (mixed failure causes).
• Scope: 1 job (nightly-8-gpu-mi35x-qwen3-235b-mxfp4) in nightly-test-amd.yml; MI35x/gfx950 only. Blocks the Qwen3-235B MXFP4 nightly accuracy signal on AMD MI35x.
• In-flight fix: ❌ none found in open PRs — needs new aiter-side investigation or a new PR.

nightly-test-1-gpu-kernel — failed step(s): Nightly Kernel Benchmark (1-GPU)

Commit Info

• sglang: d4801be4
• aiter: 9127c94a (v0.1.17.dev110+g9127c94a1, source: dev image preserved — [CI-AITER-CHECK] reported "Dev/patched version detected → skipping rebuild"; matches Dockerfile default)

Failed Tests

All 12 failures are in test/registered/jit/test_transfer_mamba.py, all with the identical error NameError: name 'transfer_kv_mamba_lf_pf' is not defined raised at python/sglang/srt/mem_cache/memory_pool_host.py:394.

Test File	Test Function	Error	Log
test/registered/jit/test_transfer_mamba.py	test_mamba_kernel_backup_load_roundtrip (page_first / page_first_direct × dtype0/dtype1 = 4)	NameError: name 'transfer_kv_mamba_lf_pf' is not defined	Log
test/registered/jit/test_transfer_mamba.py	test_mamba_kernel_single_item (4 params)	NameError: name 'transfer_kv_mamba_lf_pf' is not defined	Log
test/registered/jit/test_transfer_mamba.py	test_mamba_kernel_full_indices (4 params)	NameError: name 'transfer_kv_mamba_lf_pf' is not defined	Log

Suite result: 12 failed, 4 passed (the 4 passing are the test_mamba_kernel_empty_indices cases, which short-circuit before reaching the kernel call via the src_indices.numel() == 0 early return). Every other test file in the job passed. The job exit was driven solely by test_transfer_mamba.py returned exit code 1.

Failure Summary

The nightly AMD kernel suite has exactly one failing test file, test_transfer_mamba.py, contributing 12 failures. Every case that exercises the io_backend="kernel" mamba D2H/H2D transfer path raises NameError because the name transfer_kv_mamba_lf_pf is unbound on this AMD (HIP) runner. The other 13 kernel test files in the job passed.

Regression Status

Never-passed on AMD — the test file was introduced already-broken on AMD; it has failed with this identical NameError in every completed run since it first appeared.

Recent history of test/registered/jit/test_transfer_mamba.py in job nightly-test-1-gpu-kernel (branch main, event=schedule, completed runs only):

Date	Run	Job	Test File Status	Failed Function	Error
Jul 19	29697529176	nightly-test-1-gpu-kernel	❌ Failed	12× (backup_load_roundtrip/single_item/full_indices)	NameError: transfer_kv_mamba_lf_pf
Jul 18	29654601256	nightly-test-1-gpu-kernel	❌ Failed	same 12	NameError: transfer_kv_mamba_lf_pf
Jul 17	29601593052	nightly-test-1-gpu-kernel	❌ Failed	same 12	NameError: transfer_kv_mamba_lf_pf
Jul 16	29521595306	nightly-test-1-gpu-kernel	❌ Failed	same 12	NameError: transfer_kv_mamba_lf_pf
Jul 15	29438351311	nightly-test-1-gpu-kernel	⏭️ Run cancelled	—	(excluded)
Jul 14	29355460805	nightly-test-1-gpu-kernel	✅ Passed (test file absent)	—	—

First observed failure: Jul 16 2026 (the run where the test file first appears — 0 mentions on Jul 14). No completed run has ever had this test file pass on AMD. Lookback window: last 12 completed scheduled runs.

Failure Cluster

NameError: transfer_kv_mamba_lf_pf undefined on AMD (CUDA-only import guard)

Facts (verified observations)

• Verbatim error (12×): NameError: name 'transfer_kv_mamba_lf_pf' is not defined, raised at python/sglang/srt/mem_cache/memory_pool_host.py:394 inside MambaPoolHost._copy_tensor_all_layers_lf_pf, called from backup_from_device_all_layer at memory_pool_host.py:464 (job log lines 4543–4558, 5496–5508).
• The import of transfer_kv_mamba_lf_pf / transfer_kv_mamba_pf_lf is gated behind if _is_cuda: only, at python/sglang/srt/mem_cache/memory_pool_host.py:41-45 — there is no _is_hip branch. (Contrast lines 31-40, where the sgl_kernel.kvcacheio imports are guarded by if _is_cuda or _is_hip:.)
• The function is defined and callable — python/sglang/jit_kernel/transfer_mamba.py:64 — so this is purely a missing-import problem on AMD, not a missing implementation. [FACT]
• The io_backend == "kernel" branch at memory_pool_host.py:386-394 calls transfer_kv_mamba_lf_pf unconditionally once entered, so any AMD invocation of the kernel path fails. The trace shows can_use_jit = False yet the kernel path is still taken.
• The 4 passing cases (test_mamba_kernel_empty_indices) return early at memory_pool_host.py:384-385 (src_indices.numel() == 0) before reaching the unbound name.
• Reproducibility: identical 12/12 failures with identical error in 4 consecutive completed runs (Jul 16–19). NVIDIA is unaffected because _is_cuda is True there and the import succeeds.
• Environment: AMD gfx942 (mi30x), image rocm/sgl-dev:v0.5.15.post1-rocm700-mi30x-20260719; aiter dev build unchanged across passing/failing runs (irrelevant to this failure).

Hypothesised Causes (with confidence)

• [FACT] The transfer_kv_mamba_lf_pf / transfer_kv_mamba_pf_lf imports are gated by if _is_cuda: (memory_pool_host.py:41), excluding AMD/HIP, while the mamba io_backend="kernel" code path (lines 386-394, and the sibling pf_lf path at line 351) calls those names on any backend. On AMD the names are never bound → NameError.
  • Supporting: the guard is directly visible in source (line 41 if _is_cuda: vs line 31 if _is_cuda or _is_hip:); the traceback terminates exactly at the call site (line 394); the function exists (transfer_mamba.py:64), ruling out "not implemented".
  • Disconfirming: (none found) — the error is deterministic and the code guard fully explains it. Whether the JIT mamba kernel would actually run correctly on HIP if imported is a separate, unverified question; adding _is_hip to the guard is not necessarily the correct fix if the kernel is CUDA-specific.
  • How to verify: on an AMD box, python -c "from sglang.jit_kernel.transfer_mamba import transfer_kv_mamba_lf_pf" — confirm it imports/builds under HIP; then rerun test/registered/jit/test_transfer_mamba.py.
• Introducing change: the test file, the JIT kernel, and the _is_cuda-gated import were all added together in the same feature landing between the Jul 14 (✅, test absent) and Jul 16 (❌) nightly runs. The local repo is a shallow clone (100 commits) and git blame/git log -S collapse to the clone-boundary commit fc1e3797, which is not the true origin — do not cite it as the culprit. [LOW] on exact SHA; [FACT] on the Jul 14→Jul 16 window.
  • How to verify: git log --diff-filter=A -- test/registered/jit/test_transfer_mamba.py on a full (non-shallow) clone, or GitHub blame on memory_pool_host.py:41.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for transfer_kv_mamba (0 results), mamba+ROCm+hicache, and _is_hip+mamba (0 results). No open PR addresses the AMD import gating. (Open mamba/hicache PRs #31180, #31181, #29792, #30986 etc. concern unrelated radix-cache/bookkeeping work, not this guard.)

Suggested Triage Steps

• Confirm intended AMD support for the mamba JIT transfer kernel. If it should run on AMD: extend the import guard at memory_pool_host.py:41 to if _is_cuda or _is_hip: (mirroring line 31) and verify jit_kernel/transfer_mamba.py builds under HIP, then rerun test_transfer_mamba.py on gfx942.
• If the kernel is CUDA-only by design: gate test/registered/jit/test_transfer_mamba.py (or its io_backend="kernel" cases) to skip on AMD, so the AMD nightly kernel suite stops reporting a false regression.
• Either way, add an AMD-side coverage check for the io_backend="kernel" mamba path before re-enabling, since it has never been exercised successfully on AMD.

Status

• Persistence: Failing every completed scheduled run since Jul 16 2026 (4 consecutive runs); never passed on AMD.
• Scope: 1 job (nightly-test-1-gpu-kernel), 1 test file, 12 test cases. AMD-only — NVIDIA unaffected (import succeeds under _is_cuda).
• Blocked work: Blocks AMD nightly signal for the mamba HiCache kernel D2H/H2D transfer path; keeps nightly-test-1-gpu-kernel red on every nightly run.
• In-flight fix: ❌ none found — needs a new PR (guard fix or AMD test-skip) or a design decision on AMD support.

nightly-2-gpu-mi35x-glm51-mxfp4 — failed step(s): N/A

Commit Info

• sglang: d4801be4 (fix: fix vlm cuda graph shape stability (#30868))
• aiter: N/A for today — today's job log is unavailable (see below). The most recent readable run of this job (Jul 16) used the Dockerfile default 9127c94a (AITER_COMMIT_OVERRIDE empty). aiter is not implicated in this failure mode.

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi35x/test_glm51_mxfp4_tp2_gsm8k_mi35x.py	TestGLM51MXFP4TP2GSM8KMI35x.setUpClass	TimeoutError: Server failed to start within the timeout period	Log (today)

⚠️ Today's job log is unavailable (GitHub returns BlobNotFound; the overall run 29697529176 is still in-progress and this job's log blob was not finalized). The Failed Tests row above is inferred from the persistent failure pattern of this exact job. The error message and stack trace cited below are verbatim from the most recent readable run (Jul 16), not from today's run.

Failure Summary

The nightly-2-gpu-mi35x-glm51-mxfp4 job runs a single GSM8K accuracy gate that first launches an sglang server for amd/GLM-5.1-MXFP4 (TP=2) in setUpClass. Across every non-cancelled run since 2026-07-13, the server never reaches ready state within the 5400 s (SERVER_LAUNCH_TIMEOUT) budget, so setUpClass raises TimeoutError and the whole test errors out before any accuracy is measured. Today's job ran only ~24 min (17:47:47→18:11:49 UTC) and concluded failure with the accuracy step still in_progress and no uploaded log.

Regression Status

Known recurring failure — persistent since 2026-07-13 (last clean pass 2026-07-05).

Recent history of test_glm51_mxfp4_tp2_gsm8k_mi35x.py (TestGLM51MXFP4TP2GSM8KMI35x.setUpClass) in job nightly-2-gpu-mi35x-glm51-mxfp4 (most recent comparable completed scheduled runs: branch=main, event=schedule, completed only):

Date	Run	Test File Status	Failed Function	Error
Jul 19	run	❌ Failed	setUpClass	log unavailable (BlobNotFound); ~24 min then failure
Jul 18	run	❌ Failed	setUpClass	server-start stall (~22 min, step never concluded)
Jul 17	run	❌ Failed	setUpClass	server-start stall (~27 min, step never concluded)
Jul 16	run	❌ Failed	setUpClass	TimeoutError: Server failed to start (elapsed 5428 s)
Jul 15	run	⏭️ Cancelled	—	run cancelled (excluded)
Jul 14	run	❌ Failed	setUpClass	watchdog stall in HF weight download
Jul 13	run	❌ Failed	setUpClass	server-start stall
Jul 6–12	7 runs	⏭️ Cancelled	—	runs cancelled (excluded)
Jul 5	run	✅ Passed	—	—
Jul 4	run	✅ Passed	—	—

First observed failure: 2026-07-13. Last known passing: 2026-07-05. (Jul 6–12 and Jul 15 runs were cancelled and are not comparable; the Jul 3 run's accuracy step was skipped, so it is not a clean pass for this test either.)

Failure Cluster

GLM-5.1-MXFP4 MI35x server startup timeout — scheduler blocked in HuggingFace weight download

Facts (verified observations, no interpretation)

• Error (Jul 16 log, verbatim): setUpClass (__main__.TestGLM51MXFP4TP2GSM8KMI35x) → File ".../test_glm51_mxfp4_tp2_gsm8k_mi35x.py", line 52, in setUpClass / raise TimeoutError(error_msg) / TimeoutError: Server failed to start within the timeout period. Test then reports Fail. Time elapsed: 5428.10s and the step exits with code 255.
• Stall location (py-spy dumps in Jul 16 & Jul 14 logs): both TP0/TP1 schedulers are parked in the model weight download path, not in kernels/graph capture. Stack (bottom→top): scheduler.py:428 __init__ → tp_worker.py:277 → model_runner.py:349/486/844 load_model → loader.py load_model/load_weights_and_postprocess → deepseek_v2.py:2891 load_weights → deepseek_weight_loader.py:206 do_load_weights → loader.py:476 _prepare_weights → weight_utils.py:571/610 download_weights_from_hf → ci_weight_validation.py:1941/1955 ci_download_with_validation_and_retry → snapshot_download → hf_hub_download → file_download.py:563 xet_get (network I/O).
• Watchdog symptom: while the download blocks, Scheduler watchdog timeout (self.watchdog_timeout=300, self.soft=True) prints every ~5 min for both TP0 and TP1 continuously from 18:55 until the 5400 s launch timeout fires (Jul 16). --watchdog-timeout 1200 is set on the launch command, but the soft watchdog does not kill the process; setUpClass eventually times out instead.
• Launch command (Jul 16 log): sglang serve --model-path amd/GLM-5.1-MXFP4 --tp 2 --trust-remote-code --reasoning-parser glm45 --tool-call-parser glm47 --watchdog-timeout 1200 --mem-fraction-static 0.85 --kv-cache-dtype fp8_e4m3 --disable-radix-cache --model-loader-extra-config '{"enable_multithread_load": true, "num_threads": 8}' --dsa-prefill-backend tilelang --dsa-decode-backend tilelang --tokenizer-worker-num 4 ....
• Reproducibility: identical outcome in every completed run Jul 13/14/16/17/18/19 (6 runs, single job each). The Jul 13/14/16 runs reached the full ~90 min launch timeout; Jul 17/18/19 concluded failure after only ~22–27 min with the accuracy step never marked complete and no readable log — a shorter abort whose exact cause is unconfirmed.
• Test config: SERVER_LAUNCH_TIMEOUT = 5400, GSM8K_ACCURACY_THRESHOLD = 0.92, nightly=True, suite nightly-amd-2-gpu-mi35x-glm51-mxfp4 (verified in test_glm51_mxfp4_tp2_gsm8k_mi35x.py).

Hypothesised Causes (with confidence, with disconfirming evidence)

• [FACT] The mechanism of failure is that server startup does not complete within SERVER_LAUNCH_TIMEOUT, with the scheduler parked in the HuggingFace weight-download path (download_weights_from_hf → hf_hub_download → xet_get).
  • Supporting: py-spy stacks in the Jul 16 and Jul 14 logs show exactly this frame set; the timeout is raised from setUpClass.
  • Disconfirming: (none — this is directly observed).
• [LOW] The weight download stalls due to an infrastructure / network problem on the MI35x runner or the HF Hub xet backend (transient or endpoint-level), rather than an sglang code change.
  • Supporting: the hang is in external network I/O (xet_get), not in sglang kernels; no sglang code appears at the top of the stall stack.
  • Disconfirming: a pure transient network flake would not fail every completed run for 6 consecutive days — persistence points to something durable (a model-repo change, a runner-local cache/proxy/xet issue, or the download-validation path), not a one-off flake.
  • How to verify: on the MI35x runner, manually huggingface-cli download amd/GLM-5.1-MXFP4 (or run the server launch command directly) and time it; check whether the model repo grew or gained gated/xet-only files around 2026-07-06..07-13.
• [LOW] A change in the CI download/validation wrapper (ci_weight_validation.py:ci_download_with_validation_and_retry) or the model repo contents landed in the 2026-07-05..07-13 window and slowed/blocked the download.
  • Supporting: the stall passes through ci_download_with_validation_and_retry (sglang code) before entering hf_hub_download.
  • Disconfirming: the actual blocking frame is xet_get (network), below the validation wrapper — the wrapper is on the call path but not obviously the blocker. Could not enumerate the commit window: the local checkout is a shallow clone (git log 48ba79c11e..b44ac5d49a returns incomplete history), so no code-level attribution was possible.
  • How to verify: on a full clone, git log --oneline <jul05_sha>..<jul13_sha> -- python/sglang/srt/model_loader/ python/sglang/srt/model_loader/ci_weight_validation.py and inspect any download/xet/timeout changes.

No hypothesis reaches MEDIUM on root cause (only the mechanism is FACT). Needs a runner-side reproduction of the model download to separate infra from code.

In-flight Fix Check

• In-flight fix: ❌ none found targeting GLM-5.1-MXFP4 MI35x server-startup / weight-download timeout. Searched open PRs for GLM-5.1, mxfp4, xet+download, weight+download+timeout, mi35x (via search/issues?q=repo:sgl-project/sglang+is:pr+is:open+<kw>). Closest adjacent PR is #31068 ([AMD] ci: detect and recover hung GPUs in MI35x pre-flight) — MI35x infra reliability, but it targets hung GPUs, not weight-download stalls, so it likely does not address this.

Suggested Triage Steps

• Reproduce the download on the MI35x runner: run huggingface-cli download amd/GLM-5.1-MXFP4 (and then the exact server launch command from the Facts section) and measure elapsed time — this cheaply separates an infra/network stall from an sglang code issue.
• Check the model repo: inspect amd/GLM-5.1-MXFP4 on HF for recent changes (size growth, xet-only/gated files) in the 2026-07-06..07-13 window that could have made the download stall.
• Bisect the sglang window on a full clone: git log --oneline <jul05_pass_sha>..<jul13_fail_sha> -- python/sglang/srt/model_loader/ (pass_sha = 48ba79c1, fail_sha = b44ac5d4); focus on ci_weight_validation.py / weight_utils.py / xet-download changes.
• Confirm today's exact signature once the run finishes and its log finalizes — today's ~24 min abort differs from the ~90 min timeout of Jul 13/14/16 and may be a distinct earlier abort (e.g. runner kill) rather than the launch-timeout path.
• If confirmed to be a slow-but-eventually-succeeding download, consider whether pre-caching the model on the MI35x runner (not a code fix — a CI/infra decision) is warranted.

Status (factual, no priority assignment)

• Persistence: failing every completed scheduled run since 2026-07-13 (6 consecutive completed runs: Jul 13, 14, 16, 17, 18, 19); last clean pass 2026-07-05. Jul 6–12 and Jul 15 runs were cancelled (not comparable).
• Scope: 1 job in 1 workflow (nightly-test-amd.yml / nightly-2-gpu-mi35x-glm51-mxfp4). No test accuracy is ever measured because the failure is at server startup in setUpClass.
• Blocked work: the GLM-5.1-MXFP4 TP=2 GSM8K accuracy gate on MI35x/gfx950 has had no signal since 2026-07-05.
• In-flight fix: ❌ none found in open PRs — needs a runner-side download reproduction and/or new investigation.

nightly-4-gpu — failed step(s): Nightly Test (4-GPU)

Commit Info

• sglang: d4801be44 — "fix: fix vlm cuda graph shape stability (#30868)"
• aiter: 9127c94a1 (v0.1.17.dev110+g9127c94a1, source: dev/patched image preserved — [CI-AITER-CHECK] Dev/patched version detected … → skipping rebuild)

Failed Tests

Test File	Test Function	Error	Log
test/registered/vlm/test_encoder_dp.py	TestVLMEncoderDP.test_vlm_mmmu_benchmark	TIMEOUT after 3600s (harness kill, exit 255) — model under test: zai-org/GLM-4.1V-9B-Thinking	Log

Failure Summary

The nightly-4-gpu job ran a single test file, test/registered/vlm/test_encoder_dp.py (MMMU VLM benchmark, --mm-enable-dp-encoder --tp=4). The test picks one random model of four (is_in_ci() path). This run drew zai-org/GLM-4.1V-9B-Thinking. The server launched successfully (ready at 18:20:30, /health_generate → 200 at 18:21:06) and served requests, but the benchmark never finished and the harness killed it at the 3600 s wall-clock limit (✗ TIMEOUT … after 3600 seconds, Test Summary: 0/1 passed, exit 255).

Regression Status

Known recurring failure — model-selection flaky (not a new regression from today's HEAD commit).

Recent history of test/registered/vlm/test_encoder_dp.py (test_vlm_mmmu_benchmark) in job nightly-4-gpu (most recent comparable completed scheduled runs on main, event=schedule; cancelled runs where the job did not complete are excluded):

Date	Run	Model drawn	Test File Status	Elapsed	Note
Jul 19	29697529176	GLM-4.1V-9B-Thinking	❌ Failed	3600s	TIMEOUT (this run)
Jul 18	29654601256	InternVL2_5-8B	✅ Passed	398s
Jul 17	29601593052	InternVL2_5-8B	✅ Passed	453s
Jul 16	29521595306	InternVL2_5-8B	✅ Passed	454s
Jul 14	29355460805	Qwen2.5-VL-72B	✅ Passed	1483s
Jul 13	29273241962	GLM-4.1V-9B-Thinking	❌ Failed	1340s	Server exit -9
Jul 05	28749559519	GLM-4.1V-9B-Thinking	❌ Failed	1797s	Server exit -9
Jul 03	28676359018	InternVL2_5-8B	✅ Passed	397s
Jul 02	28611376093	InternVL2_5-8B	✅ Passed	412s
Jun 29	28393226564	Qwen2.5-VL-72B	❌ Failed	1579s

The verdict is at the test-file level: the test passes on every run that drew InternVL2_5-8B (fast, ~400 s) and fails on every run that drew GLM-4.1V-9B-Thinking (Jul 05, Jul 13, Jul 19). Qwen2.5-VL-72B is borderline (passed Jul 14 at 1483 s, failed Jun 29). First observed GLM failure in this window: 2026-07-05. HEAD commit d4801be44 landed after the Jul 05 / Jul 13 GLM failures, so it cannot be the origin of the GLM failure pattern.

Failure Cluster

VLM MMMU benchmark timeout on GLM-4.1V-9B-Thinking (AMD 4-GPU) — low decode throughput

Facts (verified observations)

• ✗ TIMEOUT: /sglang-checkout/test/registered/vlm/test_encoder_dp.py after 3600 seconds; Fail. Time elapsed: 3605.01s; Test Summary: 0/1 passed; job exit code 255 (Exit code 255 indicates test failure (not network issue), not retrying).
• Server started and served: The server is fired up and ready to roll! (18:20:30), /health_generate → 200 (18:21:06), multiple POST /v1/chat/completions … 200 OK (18:24, 18:41, 18:53, 18:57). The server did not crash — it was still actively decoding when the harness killed it (kill_process_tree called … pid=2411 at 19:17:31).
• Decode throughput is abnormally low: Decode batch, #running-req: 32 … gen throughput (token/s): 2.78 (19:08), 8.47 (19:10), 3.32 (19:17). At ~3 tok/s with 32 concurrent "thinking" generations, the MMMU run cannot complete in 3600 s.
• Every BF16 GEMM logs not found tuned config in /tmp/aiter_configs/bf16_tuned_gemm.csv, will use default config! using torch solution:0 — aiter is on the unoptimized torch fallback path for BF16 matmuls throughout the run.
• Model drawn this run: zai-org/GLM-4.1V-9B-Thinking (verified from Load weight begin, Found local HF snapshot for zai-org/GLM-4.1V-9B-Thinking, [CI Test Method] TestVLMEncoderDP.test_vlm_mmmu_benchmark).
• The two prior GLM runs (Jul 05, Jul 13) failed with a different symptom: Server process exited with code -9 (SIGKILL / likely OOM), not a timeout.

Hypothesised Causes (with confidence)

• [HIGH] The failure is driven by random model selection landing on GLM-4.1V-9B-Thinking, a heavy chain-of-thought VLM, which is too slow to finish the MMMU benchmark within the 3600 s cap on the AMD MI30x 4-GPU runner.
  • Supporting: 3/3 GLM draws failed; 0/N InternVL draws failed; server healthy and actively decoding at kill time; est_time=500 but GLM needs far longer. Test-file history is a clean split by model.
  • Disconfirming: prior GLM runs (Jul 05/13) failed via server exit -9 rather than timeout, so the precise failure mode differs run-to-run (crash vs. slow), i.e. GLM has more than one failure path on this runner.
  • How to verify: pin the model to zai-org/GLM-4.1V-9B-Thinking and re-run test_encoder_dp.py on AMD MI30x 4-GPU; observe whether it OOM-crashes or times out.
• [LOW] Abnormally low decode throughput (2–8 tok/s) is the proximate driver of the timeout, possibly worsened by aiter falling back to torch solution:0 for all BF16 GEMMs (no tuned config on this image).
  • Supporting: gen throughput logged at 2.78–8.47 tok/s; universal using torch solution:0 fallback.
  • Disconfirming: the same aiter fallback log appears in the passing InternVL/Qwen runs too, so the fallback alone does not explain GLM-only failure — it is more likely GLM's large per-request thinking output × 32 concurrency.
  • How to verify: compare gen throughput between a GLM run and an InternVL run for the same job.
• [LOW] HEAD commit d4801be44 (VLM cuda-graph shape stability) touches layers/attention/vision.py and cuda-graph runners exercised by this test.
  • Supporting: commit is in the VLM/cuda-graph path; decode ran with cuda graph: True.
  • Disconfirming: GLM already failed on Jul 05 and Jul 13, before this commit merged, so it cannot be the origin of the GLM failure pattern.
  • How to verify: not a priority — the pre-commit GLM failures already disconfirm this.

In-flight Fix Check

• In-flight fix: ❌ none found — no open PR targets the encoder-DP MMMU timeout / GLM-4.1V slowness on AMD. Searched open PRs for GLM-4.1V + encoder (only #17726 "fix for GLM models with special answer regex" — an accuracy fix, not a timeout/perf fix) and encoder dp vlm (14 perf/VLM PRs, none matching this timeout signature).

Suggested Triage Steps

• Reproduce deterministically: set the model to zai-org/GLM-4.1V-9B-Thinking and run python3 test/registered/vlm/test_encoder_dp.py on an AMD MI30x 4-GPU container (rocm/sgl-dev:v0.5.15.post1-rocm700-mi30x-20260719), watching gen throughput and peak VRAM to distinguish the timeout path (today) from the exit -9 path (Jul 05/13).
• If GLM is confirmed too slow/heavy for the 3600 s cap on this runner, consider either raising est_time/timeout for this suite or excluding GLM-4.1V-9B-Thinking from the AMD nightly-amd-4-gpu model list until the throughput/OOM issue is fixed — this would also remove the random-draw flakiness that makes the job pass or fail purely by which model is selected.
• Investigate the low BF16 decode throughput: check why bf16_tuned_gemm.csv has no tuned config on this image (using torch solution:0 for all GEMMs) and whether a tuned aiter config would bring GLM decode back under the time budget.

Status (factual)

• Persistence: Recurring but intermittent — fails only when the random draw selects GLM-4.1V-9B-Thinking (2026-07-05, 2026-07-13, 2026-07-19); passes on InternVL2_5-8B draws. Not continuous.
• Scope: Single job (nightly-4-gpu), single test file (test_encoder_dp.py) in nightly-test-amd.yml. Today's manifestation: 3600 s timeout; earlier GLM runs: server exit -9.
• Blocked work: Blocks a clean nightly-4-gpu signal on AMD whenever GLM is drawn; the random model selection makes the job non-deterministically red.
• In-flight fix: ❌ none found — needs a new PR or investigation.

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-07-19 22:57 UTC)

[amd-bot]

pr-test-amd-rocm720.yml — 18 failure(s)

Run started (UTC): 2026-07-18 18:11 UTC
Last scanned (UTC): 2026-07-19 19:58 UTC
Commits: sglang d4801be

Summary

Counts: 18 failures · 8 clusters · 15 carrying over (≥3 days) · 3 new/transient today (2 infra, 1 flaky)

#	Cluster	Job	Test File	Test Function	Status	Hypothesis (confidence)
1	Diffusion 1-GPU suite 60-min overrun	mm-gen-1gpu (…,3) Jul18	test_server_1_gpu.py (log lost)	N/A (hard-kill, BlobNotFound)	Never-passed 15/15 (Jul04→18)	[LOW] server proc hangs/crashes before 60-min timeout
2	Diffusion 1-GPU suite 60-min overrun	mm-gen-1gpu (…,1) Jul19	test_server_1_gpu.py	N/A (runner comms-loss)	12 consec (since Jul08)	[MEDIUM] part-1 workload doesn't fit 60-min budget / runner overload
3	Diffusion 1-GPU suite 60-min overrun	mm-gen-1gpu (…,1) Jul18	test_server_1_gpu.py	TestDiffusionServerOneGpu::test_diffusion_generation (part1)	~11 consec (since Jul08)	[HIGH] cumulative-runtime 60-min timeout
4	Diffusion 1-GPU suite 60-min overrun	mm-gen-1gpu (…,0) Jul19	test_server_1_gpu.py	test_diffusion_generation[longlive2_i2v]	≥6 days (Jul14→19)	[HIGH] shard-0 suite exceeds 60-min budget
5	Diffusion 1-GPU suite 60-min overrun	mm-gen-1gpu (…,0) Jul18	test_server_1_gpu.py	test_diffusion_generation[longlive2_i2v]	5 days (since Jul14)	[HIGH] longlive2_i2v testcase (#27639) + 60-min timeout
6	Diffusion 1-GPU suite 60-min overrun	mm-gen-1gpu (…,3) Jul19	test_server_1_gpu.py	test_diffusion_generation (part3, unconfirmed)	shard-3 red Jul16→19 [IN-FLIGHT]	[MEDIUM] longlive2 ValueError→poll-loop 60-min timeout
7	Triton throughput gate <3500	stage-b-1gpu-large (…,0) Jul18	test/registered/perf/test_bench_serving_1gpu_part1.py	test_offline_throughput_with_triton_attention_backend	Never-passed (15 days, Jul04→18)	[MEDIUM] AMD 3500 floor miscalibrated (~3100-3200 obs)
8	Triton throughput gate <3500	stage-b-1gpu-large (…,0) Jul19	test/registered/perf/test_bench_serving_1gpu_part1.py	test_offline_throughput_with_triton_attention_backend	Never-passed (since Jul16)	[MEDIUM] 3500 floor miscalibrated (fc1e3797 #31255)
9	msgpack ModuleNotFoundError	mm-gen-unit-test Jul18	test/unit/test_pi05_action_api.py	test_msgpack_roundtrip_preserves_string_keys_and_numpy_payloads	Never-passed (Jul14→18)	[HIGH] msgpack absent from ROCm720 CI image (#31227)
10	msgpack ModuleNotFoundError	mm-gen-unit-test Jul19	test/unit/test_pi05_action_api.py	test_msgpack_roundtrip_preserves_string_keys_and_numpy_payloads	Never-passed (~8 days, since Jul11)	[HIGH] msgpack in optional diffusion extra, not installed (#30633)
11	stage-b-large 45-min step timeout	stage-b-1gpu-large (…,1) Jul19	N/A (45-min step timeout)	reached file 11/12	recurring (❌Jul13, ✅14-17, ❌18-19)	[HIGH] job wall-time at 45-min boundary
12	stage-b-large 45-min step timeout	stage-b-1gpu-large (…,1) Jul18	N/A (45-min step timeout)	reached file 10/13	new-today (passed Jul14-17)	[HIGH] partition over-budget by design/variance
13	MXFP4/MLA prefill .view mismatch (#31391)	stage-c-8gpu-mi35x (…,1) Jul18	test/registered/amd/test_deepseek_r1_mxfp4_8gpu.py	TestDeepseekR1MXFP4.setUpClass	New regression (since Jul17)	[HIGH] 24a8944e (#31391) — eager path missing mha_companion_layers
14	MXFP4/MLA prefill .view mismatch (#31391)	stage-c-8gpu-mi35x (…,0) Jul18	test/registered/amd/test_kimi_k25_mxfp4_bcg_mi35x.py	TestKimiK25MXFP4BcgMI35x.test_bcg_gsm8k	Never-passed (since Jul16)	[MEDIUM] 24a8944e (#31391) head/dim metadata in BCG path
15	MI325 runner/container infra	mm-gen-2gpu (…,0) Jul18	N/A (runner comms-loss @ Set up job)	N/A	Infra, 2 days (Jul17-18)	[FACT] runner lost communication with control plane
16	MI325 runner/container infra	stage-b-1gpu-small (…,8) Jul19	N/A (Start CI container hang)	N/A	Infra, new today (passed prior 15)	[MEDIUM] transient MI325 container-start failure
17	Z-Image disagg ndarray crash	mm-gen-2gpu (…,1) Jul18	test/single_test_file/test_disagg_server.py	TestDisaggZImage1Rank::test_generates_image	~9-10 days (since Jul09)	[FACT] _is_default compares ndarray with ==
18	Z-Image disagg ndarray crash	mm-gen-2gpu (…,1) Jul18	test/single_test_file/test_disagg_server.py	TestDisaggZImage2RankDenoiser::test_generates_image_with_sp2_denoiser	~9-10 days (since Jul09)	[FACT] same encoder ndarray-bool crash
19	Z-Image disagg ndarray crash	mm-gen-2gpu (…,1) Jul18	test/single_test_file/test_disagg_server.py	TestDisaggZImageTracing::test_disagg_spans_share_trace_id	~9-10 days (since Jul09)	[FACT] same encoder ndarray-bool crash
20	PD+PP GSM8K KeyError 'answer'	stage-b-8gpu-disagg Jul19	test/registered/amd/disaggregation/test_disaggregation_pp.py	TestDisaggregationPrefillPPDynamicChunkAccuracy.test_gsm8k	Flaky (❌Jul17, ✅Jul18, ❌Jul19)	[MEDIUM] dropped/aborted generation in PD+PP path

Cluster: Diffusion 1-GPU suite 60-min overrun

• Affected: parts 0/1/3 of multimodal-gen-test-1-gpu-amd-rocm720 across both runs (rows 1-6); part 2 passes every run.
• Shared facts: the LPT-partitioned 1-GPU diffusion suite (run_suite.py --total-partitions 4) does not fit the step's timeout-minutes: 60 on the MI325 ROCm720 runner; longlive2_i2v is the last testcase and is consistently sacrificed. Two interchangeable death modes on the same step: clean 60-min timeout and runner lost communication (overload). Several shard-3 logs are BlobNotFound.
• Top hypothesis: [HIGH] structural suite-runtime overrun (last pass Jul07 finished at ~54 min, on the edge). Disconfirming: comms-loss variant (rows 2, 6-partly) implies an extra host/GPU fault beyond a clean over-budget run.
• In-flight fix: ❌ none found.
• Suggested triage: raise --total-partitions beyond 4 / raise the 60-min step timeout so logs upload; run longlive2_i2v in isolation to separate "over budget" from "hangs"; capture free -h/rocm-smi/dmesg on the comms-loss shard.

Cluster: Triton throughput gate <3500

• Affected: stage-b-test-1-gpu-large-amd-rocm720 (…,0) both runs (rows 7-8); other 13/14 files pass.
• Shared facts: test_offline_throughput_with_triton_attention_backend measures ~3100-3200 tok/s (best-ever 3212) vs the AMD floor 3500 (test_bench_serving_1gpu_part1.py:118-120, is_in_amd_ci()); never green since the floor existed; sibling non-Triton benchmark passes ~4072.
• Top hypothesis: [MEDIUM] threshold miscalibrated for MI325/ROCm720+Triton. Disconfirming: no green baseline in-window, so a pre-window regression can't be fully excluded.
• In-flight fix: ❌ none found.
• Suggested triage: check the pre-#31255 floor value; recalibrate to observed p50 with margin, or bisect Triton path using the passing default-backend number as control.

Cluster: msgpack ModuleNotFoundError

• Affected: multimodal-gen-unit-test-amd-rocm720 both runs (rows 9-10); 793/794 unit tests pass.
• Shared facts: import msgpack (lazy, protocol.py:59) raises; msgpack is declared only in the optional [diffusion] extra (pyproject.toml:108) and is not installed in the prebuilt ROCm720 image; added by #30633/#31227 after the image was provisioned.
• Top hypothesis: [HIGH] missing dep in the ROCm720 CI image. Disconfirming: (none) — 100% reproducible, never passed.
• In-flight fix: ❌ none (open msgpack PRs are feature work only).
• Suggested triage: docker … pip show msgpack; add msgpack to the ROCm720 image / amd_ci_install_dependency.sh, or pytest.importorskip; cross-check the NVIDIA unit job.

Cluster: stage-b-large 45-min step timeout

• Affected: stage-b-test-1-gpu-large-amd-rocm720 (…,1) both runs (rows 11-12); no assertion fails into a suite result.
• Shared facts: Run test step (timeout-minutes: 45) killed while running file 10-11 of 12-13; test_bench_serving_1gpu_part2.py alone eats ~24 min, inflated by test_vlm_online_latency cold-start (~275s, passes on retry). Jul17 passed at ~44 min (~1 min margin). Fail↔Pass flips day-to-day.
• Top hypothesis: [HIGH] job wall-time chronically at the 45-min boundary. Disconfirming: (none) — per-suite timings near-constant across pass/fail.
• In-flight fix: ❌ none found.
• Suggested triage: raise timeout-minutes or increase --auto-partition-size; warm the VLM benchmark so JIT builds don't count toward median latency.

Cluster: MXFP4/MLA prefill .view mismatch (#31391)

• Affected: stage-c-test-large-8-gpu-amd-mi35x-rocm720 shards 0 & 1 (rows 13-14, Jul18 run).
• Shared facts: identical crash RuntimeError: shape '[1, 16, 512]' is invalid for input of size 2048 at radix_attention.py:269 (output[:real_num_tokens].view(ret.shape)); 2048↔8192 ratio = TP=4 (TP-local vs full-head buffer). Both trace to 24a8944e (#31391) moving the MLA companion-layer swap to context.mha_companion_layers, leaving eager_runner.py (and the BCG path) without that field.
• Top hypothesis: [HIGH] for the eager/DeepSeek path (row 13); [MEDIUM] for the BCG/Kimi path (row 14, never-passed → feature-not-yet-working, not a classic regression).
• In-flight fix: ✅ #31675 (open since 2026-07-18, +1/-0 to eager_runner.py) fixes the DeepSeek eager path (row 13) exactly — chase reviewers. Row 14 (Kimi BCG) not confirmed covered.
• Suggested triage: merge #31675 and re-run row 13; verify whether it also fixes the Kimi BCG path (row 14) or git revert 24a8944e to A/B.

Cluster: MI325 runner/container infra

• Affected: multimodal-gen-test-2-gpu (…,0) comms-loss at Set up job (row 15, Jul18); stage-b-1gpu-small (…,8) container-start hang (row 16, Jul19). Both isolated (sister shards healthy).
• Shared facts: pre-test infra death — no dependency install / no test ran; logs BlobNotFound; head SHA cannot be implicated.
• Top hypothesis: [FACT]/[MEDIUM] transient self-hosted MI325 runner/container fault. Disconfirming: logs unavailable, so exact trigger (image-pull vs GPU-init vs preemption) unconfirmed.
• In-flight fix: ❌ none directly matching (infra).
• Suggested triage: re-run both jobs; if recurring, inspect MI325 pool node/kubelet logs & the container-start step log once runs finalize.

Cluster: Z-Image disagg ndarray crash

• Affected: 3 tests in test_disagg_server.py on multimodal-gen-test-2-gpu (…,1) (rows 17-19, Jul18).
• Shared facts: ENCODER role crashes — The truth value of an array with more than one element is ambiguous at scheduler_mixin.py:244 → _is_default (:155) does value == field_info.default on an ndarray → encoder emits no output → denoiser 120s timeout. Failing since Jul09; last green Jul07.
• Top hypothesis: [FACT] proximate _is_default ndarray-in-bool crash. Disconfirming: introducing commit (which Req field became an ndarray) not pinned; scheduler_mixin.py unchanged in the window (latent bug).
• In-flight fix: ✅ #31325 (open since 2026-07-15) handles numpy arrays in _is_default/extract_transfer_fields, cites this exact failure — merge, do NOT duplicate.
• Suggested triage: merge #31325, re-run, confirm 3 passed.

Cluster: PD+PP GSM8K KeyError 'answer'

• Affected: test_disaggregation_pp.py::test_gsm8k on stage-b-8gpu-disagg (row 20, Jul19).
• Shared facts: KeyError: 'answer' at few_shot_gsm8k.py:123 (a state's answer never populated); progress bar hits 200/200, so a single request dropped output. Signature hits different PP subclasses on different days (Jul17 non-chunk, Jul19 dynamic-chunk) with a clean 4/4 pass Jul18 — points to a flake, not a config-deterministic bug. Aiter/commit window (11 commits) don't touch the Qwen3-8B PP path.
• Top hypothesis: [MEDIUM] intermittent dropped generation in PD+PP-2 path. Disconfirming: offending rid not identified; rare real regression not fully excluded.
• In-flight fix: ❌ none found.
• Suggested triage: rerun 3-5× on d4801be4 to measure flake rate; log finish-reason/rid of the missing state.

Suggested triage order

Rank	Triage step	Targets	Why this first
1	Merge in-flight #31675 (eager mha_companion_layers) and re-run stage-c MI35x	rows 13 (& check 14)	Verified fix exists for a [HIGH] regression; blocks DeepSeek/Kimi MXFP4 8-GPU signal
2	Merge in-flight #31325 (ndarray-safe transfer) and re-run 2-GPU disagg	rows 17-19	Verified fix for a 3-test [FACT] cluster red ~10 days
3	Rebalance/raise timeout for the 1-GPU diffusion suite (--total-partitions, step timeout-minutes)	rows 1-6 (largest cluster)	6 jobs across both runs; also restores lost logs/observability
4	Add msgpack to ROCm720 image / install step	rows 9-10	One-line env fix; unblocks the unit-test job masking other regressions
5	Recalibrate/xfail the AMD 3500 Triton floor and the 45-min stage-b-large step	rows 7-8, 11-12	Cheap config change; both are threshold/budget miscalibrations, never a code defect

---

Job	Failed Steps	Started
stage-b-test-1-gpu-large-amd-rocm720 (linux-mi325-1gpu-sglang, 1)	Run test	2026-07-18T19:08
stage-c-test-large-8-gpu-amd-mi35x-rocm720 (linux-mi35x-gpu-8, 0)	Run test	2026-07-18T18:48
stage-c-test-large-8-gpu-amd-mi35x-rocm720 (linux-mi35x-gpu-8, 1)	Run test	2026-07-18T19:03
multimodal-gen-unit-test-amd-rocm720	Run diffusion unit tests	2026-07-18T19:26
multimodal-gen-test-2-gpu-amd-rocm720 (linux-mi325-2gpu-sglang, 0)	N/A	2026-07-18T18:14
multimodal-gen-test-2-gpu-amd-rocm720 (linux-mi325-2gpu-sglang, 1)	Run diffusion server tests (2-GPU)	2026-07-18T18:11
multimodal-gen-test-1-gpu-amd-rocm720 (linux-mi325-1gpu-sglang, 3)	N/A	2026-07-18T19:24
multimodal-gen-test-1-gpu-amd-rocm720 (linux-mi325-1gpu-sglang, 0)	Run diffusion server tests (1-GPU)	2026-07-18T19:29
stage-b-test-1-gpu-large-amd-rocm720 (linux-mi325-1gpu-sglang, 0)	Run test	2026-07-18T19:20
multimodal-gen-test-1-gpu-amd-rocm720 (linux-mi325-1gpu-sglang, 1)	N/A	2026-07-18T19:30
stage-b-test-large-8-gpu-mi35x-disaggregation-amd-rocm720 (linux-mi35x-gpu-8.fabric)	Run test_disaggregation	2026-07-19T17:47
stage-b-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang, 8)	N/A	2026-07-19T18:14
multimodal-gen-unit-test-amd-rocm720	Run diffusion unit tests	2026-07-19T18:19
stage-b-test-1-gpu-large-amd-rocm720 (linux-mi325-1gpu-sglang, 0)	Run test	2026-07-19T18:25
stage-b-test-1-gpu-large-amd-rocm720 (linux-mi325-1gpu-sglang, 1)	Run test	2026-07-19T18:26
multimodal-gen-test-1-gpu-amd-rocm720 (linux-mi325-1gpu-sglang, 3)	N/A	2026-07-19T18:29
multimodal-gen-test-1-gpu-amd-rocm720 (linux-mi325-1gpu-sglang, 0)	Run diffusion server tests (1-GPU)	2026-07-19T18:02
multimodal-gen-test-1-gpu-amd-rocm720 (linux-mi325-1gpu-sglang, 1)	N/A	2026-07-19T18:29

Per-Job Analysis

stage-b-test-1-gpu-large-amd-rocm720 (linux-mi325-1gpu-sglang, 1) — failed step(s): Run test

Commit Info

• sglang: b3a0185cab04
• aiter: 9127c94a1 v0.1.17.dev110+g9127c94a1 (source: dev/patched image detected — rebuild skipped; Dockerfile default is the same commit)

Failed Tests

Test File	Test Function	Error	Log
N/A (job-level)	N/A	##[error]The action 'Run test' has timed out after 45 minutes	Log

No individual registered test failed its assertion in this run. The job died because the Run test GitHub Actions step hit its timeout-minutes: 45 hard limit (.github/workflows/pr-test-amd-rocm720.yml:586) while partition 1/2 of the stage-b-test-1-gpu-large-amd suite was still executing. Two tests failed on their first attempt but recovered on retry (see Facts).

Failure Summary

The stage-b-test-1-gpu-large-amd-rocm720 (…, 1) partition ran for 45m13s (19:18:51 → 20:04:04 UTC) and was killed by the 45-minute step timeout. It had completed 10 of ≥13 test files and was starting the 11th (test/registered/kernels/test_dsa_metadata.py) when the action was terminated. This partition is chronically sized right at the 45-minute budget — the last passing run (Jul 17) finished the whole suite in ~44 minutes with ~1 minute of headroom.

Regression Status

New regression today (borderline capacity timeout). The failure is a job/step-level timeout, not a test-file assertion, so the history below tracks whether this partition job completed within its 45-min budget (event=schedule, branch=main, completed runs only).

Date	Run	Job	Partition Status	Note
Jul 18	29654571916	job	❌ Timed out (45m13s)	killed on file 11/13 (test_dsa_metadata.py)
Jul 17	29601526478	job	✅ Completed (~44m)	full suite finished, ~1m headroom
Jul 16	29521557976	job	✅ Completed	—
Jul 15	29438298954	job	✅ Completed	—
Jul 14	29355401408	job	✅ Completed	—

(The parent workflow pr-test-amd-rocm720.yml shows failure on all 16 recent runs, but that is driven by other partitions/jobs — this specific partition-1 job passed Jul 14–17 and first failed today.)

Failure Cluster

"stage-b-1gpu-large partition exceeds 45-min step timeout"

Facts (verified observations)

• ##[error]The action 'Run test' has timed out after 45 minutes (job log line 39515, 20:04:04 UTC). Step budget = timeout-minutes: 45 at .github/workflows/pr-test-amd-rocm720.yml:586.
• Suite start 19:18:51 → kill 20:04:04 = 45m13s. Files completed today (elapsed s): test_bench_serving_1gpu_part2.py=1471, test_lora_update.py=421, test_frozen_kv_mtp.py=174, test_bench_serving_1gpu_large.py=194, test_awq.py=215, test_eagle_constrained_decoding.py=145, plus 4 small GDN/lightning/SWA triton files (28/22/13/19). Killed while starting test_dsa_metadata.py.
• test_dsa_metadata.py, test_chunk_gated_delta_rule.py, test_resolve_swa_kv_pool.py did not run today (no signal — cut off by timeout).
• test_bench_serving_1gpu_part2.py::test_vlm_online_latency failed first attempt (AssertionError: 269735.43 not less than 16500, Qwen2.5-VL-3B, median TTFT 155,798 ms), then passed on retry (median E2E 8301.66 ms). Identical pattern in the Jul 17 passing run (first attempt 275471 ms → retry 8288 ms). So VLM perf did not regress; this file eats ~24.5 min (1.6× its 900s estimate) both days.
• test/registered/quant/test_awq.py::test_mmlu hit a transient urllib.error.URLError: <urlopen error [Errno 104] Connection reset by peer> fetching the MMLU CSV over the network, then retried (added ~30s vs Jul 17).
• Cumulative per-file overrun today vs Jul 17 (same 10 files): +~114s (part2 +29s, lora_update +28s, awq +32s incl. network retry, others +variance) — enough to push a ~44-min run past 45 min.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [HIGH] The partition is over-budget by design/variance: 10 files' cumulative runtime (~45 min) meets or exceeds the 45-min step budget, so normal run-to-run variance tips it over.
  • Supporting: step budget is timeout-minutes: 45; Jul 17 (pass) finished in ~44 min with only ~1 min headroom; today's identical file set ran ~114s longer and hit the wall. Dominated by test_bench_serving_1gpu_part2.py (24.5 min, 1.6× its estimate).
  • Disconfirming: (none found) — timing evidence is direct.
• [LOW] Transient network flake (ConnectionResetError in test_awq::test_mmlu, fetching MMLU CSV) added ~30s and contributed to the tip-over.
  • Supporting: retry cost visible in awq elapsed (215s vs 183s Jul 17).
  • Disconfirming: only ~30s of a 45-min budget; the partition would be near-timeout even without it.
• [LOW] A code change in the 41-commit window (fec6131..b3a0185) marginally slowed the suite. Candidate: #24013 (9306278f VLM ViT batching).
  • Supporting: touches the VLM encode path exercised by the heaviest file.
  • Disconfirming: VLM retry latency is essentially unchanged (8301 ms today vs 8288 ms Jul 17); per-file deltas are variance-level, not a step-change.

In-flight Fix Check

• In-flight fix: ❌ none found specifically targeting the stage-b-1gpu-large partition timeout. Keyword searches (stage-b-test-1-gpu-large, test_bench_serving_1gpu_part2) returned only broad/unrelated PRs; #29584 ("ci(amd): fix three pre-existing MI325 CI failures") mentions test_bench_serving_1gpu_part2 but is not confirmed to address the 45-min timeout. vlm_online_latency+timeout search → 0 results.

Suggested Triage Steps

• Re-run this partition once to confirm the borderline nature — if it flips to pass with no code change, this is a capacity/variance timeout, not a code regression.
• Rebalance the suite: raise the partition step timeout-minutes (currently 45, pr-test-amd-rocm720.yml:586) or increase --auto-partition-size beyond 2 so each shard has slack, since test_bench_serving_1gpu_part2.py alone consumes >half the budget.
• Investigate why test_vlm_online_latency's first attempt is always ~270s median E2E (only passes on retry) — the wasted first-attempt benchmark run (~4-5 min including retry) is the single largest budget sink; fixing/warming it would recover the most headroom.
• Harden test_awq::test_mmlu against network flakes (the MMLU CSV is fetched over the network; a ConnectionResetError retry cost ~30s) — prefer a cached/offline MMLU file on CI.

Status

• Persistence: First timeout for this partition today (2026-07-18 20:04 UTC); passed the prior 4 scheduled runs (Jul 14–17). Borderline — ~1 min headroom on the last pass.
• Scope: stage-b-test-1-gpu-large-amd-rocm720 (linux-mi325-1gpu-sglang, 1) only. Cut off 3 test files (test_dsa_metadata.py, test_chunk_gated_delta_rule.py, test_resolve_swa_kv_pool.py) which produced no signal this run.
• Blocked work: No unique test failures; the 3 un-run files lose coverage on this partition when it times out.
• In-flight fix: ❌ none found targeting the partition timeout.

stage-c-test-large-8-gpu-amd-mi35x-rocm720 (linux-mi35x-gpu-8, 0) — failed step(s): Run test

Commit Info

• sglang: b3a0185c
• aiter: 9127c94a (v0.1.17.dev110+g9127c94a1, source: dev/patched image preserved — [CI-AITER-CHECK] Dev/patched version detected → skipping rebuild)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/test_kimi_k25_mxfp4_bcg_mi35x.py	TestKimiK25MXFP4BcgMI35x.test_bcg_gsm8k (variant=bcg)	Exception: Server process exited with code -9 — underlying RuntimeError: shape '[1, 16, 512]' is invalid for input of size 2048 at radix_attention.py:269	Log

The sibling file in this partition, test/registered/cuda_graph/breakable/test_breakable_cuda_graph.py, passed (Test Summary: 1/2 passed).

Failure Summary

The GSM8K accuracy test for the Kimi-K2.5-MXFP4 breakable CUDA-graph (BCG) prefill path on MI35x (TP4, aiter prefill/decode) never reaches the accuracy assertion — the server crashes during the first prefill forward. All 4 TP ranks raise RuntimeError: shape '[1, 16, 512]' is invalid for input of size 2048 inside unified_attention_with_output (radix_attention.py:269), the scheduler dies, SIGQUIT is raised, and popen_launch_server reports exit code -9.

Regression Status

Never-passed (new test, no green run in queryable history). The test file was introduced by 7d0fd510 (#31227, Jul 16); it has failed on every scheduled run since. Note the failure mode changed: the two earlier runs never loaded the model, so the prefill shape bug was masked until Jul 18.

Recent history of test_kimi_k25_mxfp4_bcg_mi35x.py (test_bcg_gsm8k) in job stage-c-test-large-8-gpu-amd-mi35x-rocm720 (linux-mi35x-gpu-8, 0) (event=schedule, branch=main, completed runs):

Date	Run	Test File Status	Failed Function	Error
Jul 18	29654571916	❌ Failed	test_bcg_gsm8k	shape [1,16,512] invalid for size 2048 (prefill forward) → exit -9
Jul 17	29601526478	❌ Failed	test_bcg_gsm8k	ValueError: TP rank … didn't finish loading (model load OOM/slow) → exit -9
Jul 16	29521557976	❌ Failed	test_bcg_gsm8k	ValueError: TP rank … didn't finish loading (model load OOM/slow) → exit -9
≤ Jul 15	—	n/a	—	test did not exist yet

First observed failure: Jul 16 (day the test was added). No passing run exists in the lookback window.

Failure Cluster

Shape/view mismatch in BCG prefill attention (output[:real_num_tokens].view(ret.shape) — TP-local vs full head count) on Kimi-K2.5-MXFP4 MI35x.

Facts (verified observations)

• Server launch (from log): --tensor-parallel-size 4 … --prefill-attention-backend aiter --decode-attention-backend aiter --kv-cache-dtype fp8_e4m3 --enable-aiter-allreduce-fusion --cuda-graph-backend-prefill breakable --cuda-graph-backend-decode full (env SGLANG_AITER_MLA_PERSIST=1).
• Verbatim crash (all 4 ranks, job.log:17957): RuntimeError: shape '[1, 16, 512]' is invalid for input of size 2048.
• Stack trace (verified): models/kimi_k25.py:836 → mm_utils.py:1315 → deepseek_v2.py:2871/2603/2222/1872/1977 → deepseek_common/attention_forward_methods/forward_mha.py:371 (attn_mha, save_kv_cache=False) → radix_attention.py:178 → radix_attention.py:269.
• Shape arithmetic: ret.shape=[1,16,512] = 8192 elems (full 16 heads); output[:real_num_tokens] holds 2048 elems = 4 heads × 512 (TP4 local head count). Factor of 4 = TP degree — the pre-allocated output buffer is sized for TP-local heads while ret carries full heads.
• The generic banner Fail when using backend: tc_piecewise for prefill runner. is emitted from runner_backend_utils/__init__.py:24 as a wrapper suggestion regardless of the actual breakable backend selected; the authoritative signal is the traceback above.
• Line 269 sits inside the block modified by 24a8944e (#31391, Jul 17), which reworked the MHA-companion-layer selection (_pcg_mha_companion → context.mha_companion_layers[layer_id]) that supplies the head/dim metadata for this path. Confirmed ancestor of the Jul 18 run.
• NVIDIA is unaffected — this is an AMD BCG-on-ROCm path the test was explicitly written to unblock (per the test docstring, the ROCm/HIP hard block was removed and the path is default-off elsewhere).

Hypothesised Causes (with confidence)

• [MEDIUM] Head/dim metadata mismatch in the BCG MHA-companion path: the attention backend returns full-head output (ret, 16 heads) but unified_attention_with_output copies into a TP-local-head-sized buffer (output[:real_num_tokens], 4 heads), so .view(ret.shape) on 2048 elems fails.
  • Supporting: exact crash line radix_attention.py:269 is in the region rewritten by 24a8944e (#31391); the 2048↔8192 ratio equals TP=4; the failing frame is the DeepSeek MLA attn_mha (save_kv_cache=False) companion path that commit targets.
  • Disconfirming: the test has never passed, so there is no green baseline this "regressed" from — this is more accurately a feature-not-yet-working on Kimi-MXFP4-TP4-MI35x than a classic regression. The two prior runs failed earlier (at model load), so #31391 was never actually exercised at prefill before Jul 18.
  • How to verify: on an MI35x TP4 container, run the launch command above and print ret.shape vs output.shape/real_num_tokens at radix_attention.py:268; check whether context.mha_companion_layers[layer_id] yields the correct per-head-count layer for Kimi-K2.5.
• [LOW] Interaction with --enable-aiter-allreduce-fusion under PCG/BCG capture (the fusion changes tensor layout feeding attention output).
  • Supporting: open PR #27650 reports "PCG capture fail with aiter fused all-reduce+RMSNorm" on AMD.
  • Disconfirming: the error is a pure .view numel mismatch in attention output, not an all-reduce/RMSNorm fusion failure; nothing in the trace touches the fusion path.
  • How to verify: re-run the same config with --enable-aiter-allreduce-fusion removed; if the shape error persists, fusion is exonerated.

No hypothesis reaches HIGH; a code-level check of mha_companion_layers[layer_id] head counts vs ret on MI35x TP4 is needed to promote.

In-flight Fix Check

• In-flight fix: ❌ none found exactly matching this Kimi TP4 head-dim .view mismatch (searched open PRs for breakable cuda graph, kimi prefill, radix_attention companion, Kimi MXFP4 MI35x).
• Adjacent/possibly-related open PRs (not confirmed fixes): #30765 "Fix multimodal input embeddings for prefill BCG", #30872 "Enable multimodal prefill BCG for VL and audio models", #27650 "[AMD] Fix PCG capture fail with aiter fused all-reduce+RMSNorm".

Suggested Triage Steps

• Reproduce on MI35x TP4 in the ROCm 7.2 container with the exact launch command above; add a one-shot print of ret.shape, output.shape, real_num_tokens, and layer_id at radix_attention.py:268 to confirm the full-head vs TP-local-head mismatch.
• Inspect context.mha_companion_layers[layer_id] construction (model_runner_components/layer_setup.py, touched by #31391) for Kimi-K2.5 — verify the companion layer reports the same per-rank head count as the output buffer allocation.
• A/B git revert 24a8944e on a branch and re-run test_kimi_k25_mxfp4_bcg_mi35x.py to see whether the shape error changes back to the pre-#31391 behavior (helps confirm whether the companion-layer rework is where the head-count divergence enters).
• Ping the author of #31391 / #31227 (same author, Mick) — this test was added to gate the BCG feature they authored and has never gone green.
• If BCG-on-Kimi-MI35x cannot be fixed near-term, consider marking the test nightly/skip on AMD until the prefill capture path is corrected, so it stops blocking the stage-c PR-CI signal.

Status (factual)

• Persistence: Failing on every scheduled run since the test was introduced Jul 16 (3/3 runs). Never passed.
• Scope: 1 test file in 1 job (stage-c-test-large-8-gpu-amd-mi35x-rocm720, partition 0) on the pr-test-amd-rocm720.yml scheduled monitor; MI35x-only. The sibling breakable-cuda-graph unit test in the same partition passes.
• Failure evolution: model-load OOM/slow-node (Jul 16–17) → prefill shape .view mismatch (Jul 18) after the model successfully loaded.
• In-flight fix: ❌ none matching; related BCG PRs #30765 / #30872 / #27650 not confirmed to address it.

stage-c-test-large-8-gpu-amd-mi35x-rocm720 (linux-mi35x-gpu-8, 1) — failed step(s): Run test

Commit Info

• sglang: b3a0185c
• aiter: 9127c94a — v0.1.17.dev110+g9127c94a1 (source: dev image detected, rebuild skipped — [CI-AITER-CHECK] markers, AITER_COMMIT_OVERRIDE empty)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/test_deepseek_r1_mxfp4_8gpu.py	TestDeepseekR1MXFP4.setUpClass	Exception: Server process exited with code -9 → underlying RuntimeError: shape '[1, 16, 512]' is invalid for input of size 2048 (tc_piecewise prefill runner)	Log

(The other two files in this shard — test_kimi_k25_mxfp4.py, test_qwen3_coder_next_8gpu.py — passed. Test Summary: 2/3 passed; job exited 255.)

Failure Summary

The 8-GPU DeepSeek-R1-MXFP4 server failed to launch (setUpClass → popen_launch_server). Every TP rank crashed during the first prefill forward with RuntimeError: shape '[1, 16, 512]' is invalid for input of size 2048 raised at python/sglang/srt/layers/radix_attention.py:269 (output[:real_num_tokens].view(ret.shape).copy_(ret)), inside the tc_piecewise eager piecewise-CUDA-graph prefill path. The scheduler raised SIGQUIT, the server process was killed (-9), and the test errored after ~580s.

Regression Status

New regression — first failing run 2026-07-17 (run 7193); last passing run 2026-07-16 (run 7186).

Recent history of test_deepseek_r1_mxfp4_8gpu.py (TestDeepseekR1MXFP4.setUpClass) in job stage-c-test-large-8-gpu-amd-mi35x-rocm720 (linux-mi35x-gpu-8, 1) (event=schedule, branch main, completed runs only):

Date	Run	Test File Status	Failed Function	Error
Jul 18	7197	❌ Failed	setUpClass	shape '[1, 16, 512]' invalid for size 2048 (tc_piecewise)
Jul 17	7193	❌ Failed	setUpClass	same (tc_piecewise error ×24)
Jul 16	7186	✅ Passed	—	— (passed:true, elapsed 771s)
Jul 15	7175	⚪ Cancelled	—	job cancelled — excluded
Jul 14	7167	⚪ N/A	—	job aborted early (exit 134); this test not reached, no tc_piecewise error
Jul 13	7161	⚪ N/A	—	job aborted early (exit 134); this test not reached

First observed failure: 2026-07-17. Last known pass: 2026-07-16 (sha 1f34911d). Regression window: 1f34911d..fec61318.

Failure Cluster

DeepSeek MLA prefill shape mismatch under HIP eager piecewise-CUDA-graph (view [1,16,512] ← size 2048)

Facts (verified observations)

• Server command: sglang serve --model-path amd/DeepSeek-R1-MXFP4-Preview --tp 8 ... --enforce-piecewise-cuda-graph --piecewise-cuda-graph-compiler eager ... (verbatim from log). The eager compiler selects the eager piecewise path.
• Verbatim crash at radix_attention.py:269: output[:real_num_tokens].view(ret.shape).copy_(ret) → RuntimeError: shape '[1, 16, 512]' is invalid for input of size 2048. All 8 TP ranks hit the same error; SIGQUIT; Server process exited with code -9.
• Stack: eager_runner.py:322 _execute_extend → deepseek_v2.py forward → torch.ops.sglang.unified_attention_with_output → radix_attention.py:269.
• Test passed on 2026-07-16 (run 7186, passed:true) and failed on 07-17 & 07-18 with the identical error signature.
• 24a8944e (#31391) is in the window (1f34911d..fec61318), is NOT an ancestor of the last-pass sha, and is present in today's HEAD (verified via git merge-base --is-ancestor).
• 24a8944e changed the DeepSeek MLA companion-layer selection in radix_attention.py: it removed the HIP-gated attention_layer._pcg_mha_companion swap and replaced it with context.mha_companion_layers[layer_id] (guarded by context.mha_companion_layers is not None). The removed comment stated the swap is needed so "the backend sees correct head/dim metadata" for DeepSeek MLA.
• The commit added the mha_companion_layers= argument to set_tc_piecewise_forward_context and updated the prefill_cuda_graph_runner.py call sites — but the eager_runner.py:313 call site (HIP PCG eager fallback) was NOT updated and passes no mha_companion_layers (verified: 24a8944e does not touch eager_runner.py). Hence context.mha_companion_layers is None on the eager path and the MLA companion swap is silently skipped → wrong head/dim → the .view failure.

Hypothesised Causes (with confidence)

• [HIGH] 24a8944e (#31391 "enable Kimi multimodal breakable prefill cuda graph replay") broke the DeepSeek MLA attn_mha companion-layer swap on the HIP eager piecewise path by moving the swap from a per-layer attribute to context.mha_companion_layers, while leaving the eager_runner.py context setup without that field.
  • Supporting: (a) code diff modifies the exact failing function and swap mechanism; (b) the eager call site this test exercises (--piecewise-cuda-graph-compiler eager) is the one not updated → context.mha_companion_layers is None; (c) timing: authored 2026-07-17, first failure 07-17, last pass 07-16; (d) removed comment explicitly ties the swap to DeepSeek MLA head/dim metadata, and the error is a head/dim shape mismatch on a DeepSeek MLA model; (e) an author-filed fix PR #31675 with the identical diagnosis touches only eager_runner.py.
  • Disconfirming: the PR title references Kimi, not DeepSeek — but the modified code is the shared MLA/PCG path, and the DeepSeek-specific swap logic was directly removed, so this does not weaken the causal link.
  • How to verify: apply #31675 (adds mha_companion_layers= to the eager_runner.py set_tc_piecewise_forward_context call) and re-run test_deepseek_r1_mxfp4_8gpu.py on MI35x; or git revert 24a8944e on a branch and re-run.

In-flight Fix Check

• In-flight fix: ✅ #31675 "[AMD] Fix DeepSeek MLA prefill shape mismatch on HIP eager fallback (missing mha_companion_layers)" — open (non-draft), created 2026-07-18, +1/-0 to python/sglang/srt/model_executor/runner/eager_runner.py. This is the exact fix for the diagnosed cause. Chase reviewers; do NOT open a duplicate. (Searched is:pr is:open mha_companion → matched #31675.)

Suggested Triage Steps

• Verify #31675 passes stage-c-test-large-8-gpu-amd-mi35x-rocm720 and merge — it adds the missing mha_companion_layers= argument to the eager-runner context.
• If a quick confirmation is wanted before merge: git revert 24a8944e on a branch and re-run test/registered/amd/test_deepseek_r1_mxfp4_8gpu.py on MI35x (rocm720) — expect the tc_piecewise shape error to disappear.
• As a workaround per the log's own suggestion, the test could run with --cuda-graph-backend-prefill=disabled, but the correct fix is #31675 — do not mask it.

Status

• Persistence: Failing for 2 consecutive completed scheduled runs (since 2026-07-17, run 7193); passing as recently as 2026-07-16 (run 7186).
• Scope: test_deepseek_r1_mxfp4_8gpu.py in stage-c-test-large-8-gpu-amd-mi35x-rocm720; root cause is on the HIP eager piecewise-CUDA-graph path, so any DeepSeek-MLA model exercising that path on ROCm is at risk.
• Blocked work: DeepSeek-R1-MXFP4 8-GPU serving CI signal on AMD MI35x / ROCm 7.2.
• In-flight fix: ✅ #31675 (open since 2026-07-18) — awaiting merge.

multimodal-gen-unit-test-amd-rocm720 — failed step(s): Run diffusion unit tests

Commit Info

• sglang: b3a0185c
• aiter: 9127c94a (v0.1.17.dev110+g9127c94a1, source: dev/patched image preserved — [CI-AITER-CHECK] reported "Dev/patched version detected … → skipping rebuild")

Failed Tests

Test File	Test Function	Error	Log
python/sglang/multimodal_gen/test/unit/test_pi05_action_api.py	test_msgpack_roundtrip_preserves_string_keys_and_numpy_payloads	ModuleNotFoundError: No module named 'msgpack'	Log

Failure Summary

The multimodal-gen-unit-test-amd-rocm720 job ran 794 diffusion unit tests; exactly one failed: test_msgpack_roundtrip_preserves_string_keys_and_numpy_payloads. It calls pack_msgpack()/unpack_msgpack() in python/sglang/multimodal_gen/runtime/entrypoints/vla/protocol.py:59, which does a lazy import msgpack that raises ModuleNotFoundError. The msgpack package is not installed in the ROCm720 CI image. All other tests passed (793 passed, 1 skipped, 5 deselected).

Regression Status

Never-passed (on ROCm720) — the test fails in every completed scheduled run since it was introduced. The test, its import msgpack call site, and the msgpack dependency in python/pyproject.toml:108 were ALL added by #31227 (7d0fd510). No green run for this test file exists in the queryable window (checked back to the first scheduled run that contains the test, 2026-07-14).

Recent history of test_pi05_action_api.py (test_msgpack_roundtrip_preserves_string_keys_and_numpy_payloads) in job multimodal-gen-unit-test-amd-rocm720 (comparable completed scheduled runs, branch=main, event=schedule):

Date	Run	Test File Status	Failed Function	Error
Jul 18	29654571916	❌ Failed	test_msgpack_roundtrip…	ModuleNotFoundError: msgpack
Jul 17	29601526478	❌ Failed	test_msgpack_roundtrip…	ModuleNotFoundError: msgpack
Jul 16	29521557976	❌ Failed	test_msgpack_roundtrip…	ModuleNotFoundError: msgpack
Jul 15	29438298954	❌ Failed	test_msgpack_roundtrip…	ModuleNotFoundError: msgpack
Jul 14	29355401408	❌ Failed	test_msgpack_roundtrip…	ModuleNotFoundError: msgpack
≤ Jul 13	—	N/A	—	test file not present in tree

Failure Cluster

ModuleNotFoundError: msgpack — missing dependency in ROCm720 CI image

Facts (verified observations)

• Log verbatim: sglang/multimodal_gen/runtime/entrypoints/vla/protocol.py:59: in pack_msgpack / import msgpack / E ModuleNotFoundError: No module named 'msgpack' (step 7, line 5774).
• Result line: 1 failed, 793 passed, 1 skipped, 5 deselected, 7 warnings, 50 subtests passed; job exited with code 1.
• msgpack IS declared as a runtime dependency at python/pyproject.toml:108 — added by #31227 (7d0fd510), the same PR that added the test and the protocol.py import.
• CI image used: rocm/sgl-dev:v0.5.15.post1-rocm720-mi30x-20260718 (prebuilt; AITER dev build preserved, no rebuild).
• The import msgpack statements in protocol.py are lazy (inside pack_msgpack/unpack_msgpack), so only this test — which calls them directly — trips the missing module; other VLA/diffusion code paths that don't call these helpers are unaffected.
• Reproducibility: identical failure in 5/5 checked consecutive scheduled runs (Jul 14–18); single test, single job.

Hypothesised Causes (with confidence)

• [HIGH] The prebuilt ROCm720 CI docker image does not contain msgpack, because the dependency was newly added to python/pyproject.toml in #31227 and the CI runs inside a prebuilt image that was not refreshed to install the new dependency (the sglang package's deps are not re-resolved at test time on this backend).
  • Supporting: msgpack is present in pyproject.toml:108 yet import msgpack fails at runtime; the dep, the import, and the test all originate from the same PR; the image tag predates a dependency-aware rebuild for this addition; failure is 100% deterministic across all runs since introduction.
  • Disconfirming: (none found) — no run of this test file has ever passed on ROCm720; no evidence msgpack was ever installed.
  • How to verify: in container rocm/sgl-dev:v0.5.15.post1-rocm720-mi30x-20260718, run python -c "import msgpack" (expect ModuleNotFoundError) and pip show msgpack; confirm whether the ROCm720 CI install step reinstalls sglang deps.
• [LOW] Test authored CUDA/NVIDIA-first and never validated on the AMD image (an unconditional hard dependency test that assumes msgpack is available).
  • Supporting: only the AMD ROCm720 image is observed here; the codebase elsewhere prefers msgspec.msgpack rather than raw msgpack.
  • Disconfirming: root cause is environmental (missing package), not test logic — the test would pass anywhere msgpack is installed.
  • How to verify: check whether the sibling NVIDIA multimodal-gen unit job passes this same test.

In-flight Fix Check

• In-flight fix: ❌ none found — searching open PRs for msgpack (repo:sgl-project/sglang is:pr is:open msgpack, 13 results) returns only msgpack feature work (e.g. #31565 "[Diffusion] msgpack raw-bytes transport", #29656, #30005); none installs msgpack in the ROCm720 CI image or fixes this missing-dependency failure. Needs a new PR or investigation.

Suggested Triage Steps

• Confirm the gap: docker run --rm rocm/sgl-dev:v0.5.15.post1-rocm720-mi30x-20260718 python -c "import msgpack" and pip show msgpack.
• If absent, add msgpack to the ROCm720 image build / CI dependency-install step (or ensure the ROCm720 CI reinstalls sglang's pyproject.toml deps into the prebuilt image before running tests) — mirror whatever the NVIDIA multimodal-gen job does.
• Alternatively, if the intent is to avoid a hard msgpack runtime dep on AMD, consider pytest.importorskip("msgpack") in test_pi05_action_api.py::test_msgpack_roundtrip… so the test skips cleanly where the package is unavailable (maintainer's call — this masks rather than fixes the image gap).
• Verify the sibling NVIDIA multimodal-gen unit job passes this test, to confirm the failure is AMD-image-specific.

Status

• Persistence: Failing in every completed scheduled ROCm720 run since the test appeared — 5 consecutive runs (2026-07-14 → 2026-07-18); never passed.
• Scope: 1 test in the multimodal-gen-unit-test-amd-rocm720 job of pr-test-amd-rocm720.yml. Likely also affects other AMD images lacking msgpack (unverified here).
• Blocked work: Keeps the ROCm720 multimodal-gen unit job red on a single missing-dependency test, masking real regressions in that job's signal.
• In-flight fix: ❌ none found — needs a new PR or investigation.

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-07-19 19:58 UTC)

[amd-bot]

pr-test-amd-rocm720.yml — Per-Job Analysis (continued 2/4)

multimodal-gen-test-2-gpu-amd-rocm720 (linux-mi325-2gpu-sglang, 0) — failed step(s): N/A

Commit Info

• sglang: b3a0185c
• aiter: N/A — job produced no log (log blob BlobNotFound); no [CI-AITER-CHECK] markers available. The job died before any dependency install ran.

Failed Tests

Test File	Test Function	Error	Log
N/A (no test executed)	N/A	Runner lost communication with server during "Set up job" (step 1)	Job

This is not a test failure. Job 88106525157 never advanced past step 1 (Set up job, left in_progress). No checkout, no container start, no test ever ran.

Failure Summary

The self-hosted runner linux-mi325-2gpu-sglang-f4c6t-runner-mdql7 lost communication with the GitHub Actions server while the job was still in its Set up job step. The job was force-terminated exactly 10 minutes after start (18:14:01 → 18:24:01 UTC) and no log was uploaded (BlobNotFound). The two sister shards in the same run behaved independently: shard 2 88106525121 passed, and shard 1 88106525144 reached and failed at the real Run diffusion server tests (2-GPU) step.

Regression Status

Infrastructure issue — runner-communication-loss (not a code/test regression).

Recent history of job multimodal-gen-test-2-gpu-amd-rocm720 (linux-mi325-2gpu-sglang, 0) (shard 0) on pr-test-amd-rocm720.yml, most recent 8 comparable completed scheduled runs (same branch main, event=schedule):

Date	Run	Shard-0 outcome	Failing step	Classification
Jul 18	29654571916	❌ Failure	Set up job (step 1, in_progress)	Runner lost communication (INFRA) ← this job
Jul 17	29601526478	❌ Failure	Run diffusion server tests (2-GPU) (step 9, in_progress)	Runner lost communication (INFRA)
Jul 16	29521557976	❌ Failure	Run diffusion server tests (2-GPU)	Real test failure (separate issue)
Jul 15	29438298954	❌ Failure	Run diffusion server tests (2-GPU)	Real test failure
Jul 14	29355401408	❌ Failure	Run diffusion server tests (2-GPU)	Real test failure
Jul 13	29273187026	❌ Failure	Run diffusion server tests (2-GPU)	Real test failure
Jul 12	29202664092	❌ Failure	Run diffusion server tests (2-GPU)	Real test failure
Jul 11	29162225443	❌ Failure	Run diffusion server tests (2-GPU)	Real test failure

Two distinct failure modes overlap on this shard: (1) a runner-communication-loss infra failure appearing the last 2 days (Jul 17–18), and (2) a separate, persistent real test failure at the Run diffusion server tests (2-GPU) step Jul 11–16. Today's job hit mode (1) and never reached the test step, so it gives no signal on mode (2).

Failure Cluster

Self-hosted runner lost communication with server (MI325 2-GPU pool)

Facts (verified observations)

• Annotation (verbatim): "The self-hosted runner lost communication with the server. Verify the machine is running and has a healthy network connection. Anything in your workflow that terminates the runner process, starves it for CPU/Memory, or blocks its network access can cause this error." (from check-runs/88106525157/annotations).
• Job status = completed, conclusion = failure, but the only step (Set up job) is still in_progress/null — the runner died before reporting step completion.
• Duration is exactly 10m00s (18:14:01Z → 18:24:01Z), consistent with a server-side lost-communication timeout, not a workload timeout.
• Log blob returns BlobNotFound — no log was uploaded, confirming the runner never got far enough to stream logs.
• Runner: linux-mi325-2gpu-sglang-f4c6t-runner-mdql7 (ephemeral pod).
• Sister shards in the same run ran on other runner pods and progressed normally (shard 2 passed; shard 1 reached step 9), so this is isolated to this runner instance, not a run-wide outage.
• The identical annotation appeared on shard 0 the previous day (29601526478, runner died at step 9).

Hypothesised Causes (with confidence)

• [FACT] The GitHub Actions runner agent lost its connection to the control plane during job setup. This is directly stated by the failure annotation and corroborated by the missing log blob + in_progress step.
• [LOW] Underlying trigger is runner-host resource exhaustion, node preemption, or a network blip on the MI325 2-GPU pool.
  • Supporting: The annotation lists CPU/memory starvation and network loss as the typical triggers; the same signature recurred on the same shard the prior day.
  • Disconfirming: No host metrics or logs are available (log blob missing), so no evidence isolates which trigger; sister runners in the same run were healthy, arguing against a pool-wide outage.
  • How to verify: Check the MI325 runner pool's node/kubelet logs and resource metrics for pod linux-mi325-2gpu-sglang-f4c6t-runner-mdql7 around 18:14–18:24 UTC on 2026-07-18.
• [SPECULATION] Related to the sglang commit under test — rejected. The job died in Set up job before checking out or running any sglang code; the head SHA is irrelevant to a setup-phase runner loss.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for runner+communication and diffusion+amd; results were unrelated feature/model PRs, none addressing MI325 runner-communication-loss. (Infra issue; unlikely to have a code-level PR fix.)

Suggested Triage Steps

• Re-run the failed shard 0 job — a transient runner-communication-loss usually clears on retry. If it passes, close as infra flake.
• Ask AMD infra to inspect the MI325 2-GPU runner pool (node/kubelet logs, memory pressure, network) for pods ...-runner-mdql7 (Jul 18) and the Jul 17 shard-0 runner around the failure windows — the same signature 2 days running warrants a health check of the pool.
• Track separately the persistent real test failure at Run diffusion server tests (2-GPU) (Jul 11–16, and shard 1 today 88106525144) — that is a distinct code/test issue this infra failure masked and should get its own investigation.

Status

• Persistence: Runner-communication-loss on shard 0 seen 2 consecutive days (Jul 17–18); today's instance died at Set up job. Separately, the underlying diffusion test has failed every scheduled run Jul 11–16.
• Scope: Isolated to one runner pod in this run (sister shards 1 and 2 ran on healthy runners). Infra-level, pr-test-amd-rocm720.yml only in the data reviewed.
• Blocked work: This shard produced no CI signal for the 2-GPU diffusion tests today (runner died before tests ran).
• In-flight fix: ❌ none found — needs infra investigation, not a code PR.

multimodal-gen-test-2-gpu-amd-rocm720 (linux-mi325-2gpu-sglang, 1) — failed step(s): Run diffusion server tests (2-GPU)

Commit Info

• sglang: b3a0185c
• aiter: 9127c94a (v0.1.17.dev110+g9127c94a1; source: Dockerfile default, AITER_COMMIT_OVERRIDE empty)

Failed Tests

Test File	Test Function	Error	Log
test/single_test_file/test_disagg_server.py	TestDisaggZImage1Rank::test_generates_image	RuntimeError: Model generation returned no output ... DiffusionServer timeout: request not completed within 120.0s	Log (step 9)
test/single_test_file/test_disagg_server.py	TestDisaggZImage2RankDenoiser::test_generates_image_with_sp2_denoiser	same timeout	Log
test/single_test_file/test_disagg_server.py	TestDisaggZImageTracing::test_disagg_spans_share_trace_id	same timeout	Log

Failure Summary

All 3 tests in test_disagg_server.py (disaggregated Z-Image encoder/denoiser/decoder split) error out with DiffusionServer timeout: request not completed within 120.0s. The underlying cause visible in the server logs is that the ENCODER role crashes while packaging request fields for cross-role transfer: Pool ENCODER rank 0: error: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all(). Because the encoder never emits output, the denoiser waits and the request times out, surfacing as an HTTP 500 in the test client.

Regression Status

Known recurring failure (not a new regression at the current commit).

Recent history of test_disagg_server.py (TestDisaggZImage*) in job multimodal-gen-test-2-gpu-amd-rocm720 (comparable completed scheduled runs, branch=main, event=schedule; standalone partition 2/3):

Date	Run	Test File Status	Failed Function	Error
Jul 18	29654571916	❌ Failed	all 3 TestDisaggZImage*	ValueError (array) → 120s timeout
Jul 17	29601526478	❌ Failed	all 3	ValueError (array) → timeout
Jul 16	29521557976	❌ Failed	all 3	ValueError (array) → timeout
Jul 15	29438298954	❌ Failed	all 3	ValueError (array) → timeout
Jul 14	29355401408	❌ Failed	all 3	ValueError (array) → timeout
Jul 13	29273187026	❌ Failed	disagg server __enter__	server startup 500 (ValueError not seen in this partition)
Jul 10	29113205616	❌ Failed	all 3	ValueError (array) → timeout
Jul 09	29039527644	❌ Failed	all 3	ValueError (array) → timeout (first observed)
Jul 07	28888292440	✅ Passed	—	3 passed
Jul 05	28749537427	✅ Passed	—	3 passed

• Last known pass: Jul 07 (run 28888292440, head 11cea29c).
• First observed failure: Jul 09 (run 29039527644, head 2e4d6368).
• Failing on every completed scheduled run since Jul 09. This is the only lane that runs this suite (AMD 2-GPU).

Failure Cluster

Numpy-array-in-boolean-context crash in disagg encoder extract_transfer_fields → DiffusionServer 120s timeout (Z-Image disagg).

Facts (verified observations)

• Encoder-role crash, verbatim: Pool ENCODER rank 0: error (attempt 1/3): The truth value of an array with more than one element is ambiguous. Use a.any() or a.all().
• Stack trace (verbatim): scheduler_mixin.py:1049 _disagg_event_loop → :1563 _disagg_encoder_step → :244 extract_transfer_fields → if _is_default(value, f): → ValueError.
• Proximate code: scheduler_mixin.py:155 _is_default does return value == field_info.default; when value is a numpy ndarray this yields a boolean array and raises in the if at line 244. (Verified by reading source.)
• scheduler_mixin.py was not modified in the regression window 11cea29c..2e4d6368 (verified via compare API) — the comparison bug is pre-existing/latent; the trigger is a Req field newly carrying an ndarray.
• Test-file signature is identical (same ValueError + 120s timeout) in runs Jul 09→18. Test passed cleanly (3 passed) on Jul 05 and Jul 07.
• Not a scout run; AITER_COMMIT_OVERRIDE is empty, aiter is the Dockerfile default — aiter is not a variable here.

Hypothesised Causes (with confidence)

• [FACT] Proximate cause: _is_default (scheduler_mixin.py:155) compares a numpy ndarray with == and evaluates the array in a boolean context, crashing the encoder step so no output is transferred → 120s denoiser timeout → all 3 tests error.
  • Supporting: exact stack trace in the log; source read confirms the unguarded value == field_info.default.
  • Disconfirming: (none — this is directly observable).
• [LOW] Introducing commit (the change that made a Req field hold an ndarray) lies in 11cea29c..2e4d6368 (99 commits). The only commits touching the Z-Image path the test exercises are #29742 (fa185ed8, "fix z-Image accuracy", Jul 08 — also edits encoders/qwen3.py, the code that runs in the failing ENCODER role) and #27436 (33c3dfd7, BCG for DiTs, Jul 08).
  • Supporting: both merged Jul 08, squarely between the Jul 07 pass and Jul 09 fail; #29742 modifies the encoder path.
  • Disconfirming: I did not verify which specific Req field became an ndarray; neither commit touches scheduler_mixin.py, so attribution is temporal + path-proximity only. In-flight fix PR #31325's own root-cause note also does not name the introducing commit.
  • How to verify: on AMD MI325 2-GPU, git revert #29742 (or #27436) on a branch and re-run test_disagg_server.py::TestDisaggZImage1Rank; or instrument extract_transfer_fields to log which f.name holds the ndarray.

In-flight Fix Check

• In-flight fix: ✅ #31325 "[diffusion] disagg: handle numpy arrays in cross-role transfer field extraction" (open since 2026-07-15, updated 2026-07-17, by kangwangamd). Changes scheduler_mixin.py (+18/-3) to handle numpy arrays in _is_default/extract_transfer_fields. Its motivation cites this exact failure (test_disagg_server.py::TestDisaggZImage*, DiffusionServer timeout ... 120s, AMD 2-GPU lane). Chase reviewers / merge — do NOT open a duplicate. (Searched via search/issues for extract_transfer_fields, _is_default, "truth value of an array".)

Suggested Triage Steps

• Merge in-flight PR #31325 (fixes the proximate _is_default crash) and re-run multimodal-gen-test-2-gpu-amd-rocm720; confirm test_disagg_server.py returns to 3 passed.
• Optionally, to close the causal loop on which change first introduced the ndarray field, instrument extract_transfer_fields (log f.name/type(value)) or bisect 11cea29c..2e4d6368, focusing on #29742 first.
• The comparison bug is a latent robustness gap regardless of trigger — #31325's array-safe guard is the correct durable fix.

Status

• Persistence: Failing on every completed scheduled run since 2026-07-09 (~9–10 days); last green 2026-07-07.
• Scope: test_disagg_server.py (3 tests) in multimodal-gen-test-2-gpu-amd-rocm720 — the only lane running the disagg diffusion suite.
• Blocked work: Z-Image disaggregated (encoder/denoiser/decoder) CI signal on AMD.
• In-flight fix: ✅ #31325 open since 2026-07-15, awaiting merge.

multimodal-gen-test-1-gpu-amd-rocm720 (linux-mi325-1gpu-sglang, 3) — failed step(s): N/A

Commit Info

• sglang: b3a0185c
• aiter: 9127c94a — v0.1.17.dev110+g9127c94a1 (source: dev/patched image preserved; AITER_COMMIT_OVERRIDE empty, verified from [CI-AITER-CHECK] markers in the sibling shard-0 log of the same run)

Failed Tests

Test File	Test Function	Error	Log
N/A — unrecoverable	N/A	Job hard-killed ~16 min into "Run diffusion server tests"; step left in_progress, always() artifact-upload step never ran, and the job log was never finalized (GitHub returns BlobNotFound)	Job

The specific failing test file/function for shard 3 (partition 3 of 4) cannot be recovered: its GitHub log blob does not exist (BlobNotFound) and no diffusion-failures artifacts were uploaded (the upload step never executed). Partitioning is dynamic (LPT by estimated time in python/sglang/multimodal_gen/test/partitioning.py), so the model set assigned to partition 3 cannot be reconstructed statically.

Failure Summary

The multimodal-gen-test-1-gpu-amd-rocm720 shard 3 job failed after step 9 ("Run diffusion server tests (1-GPU)") ran for ~16 min (19:35:24→job end 19:51:44) and was abruptly terminated: the step never reached a conclusion, the if: always() artifact-upload step never ran, and the log was never finalized. This is a hard runner/process kill, not the 60-min step timeout and not a clean pytest assertion failure. The sibling shard 0 (same job, partition 0), whose log survived because it ran cleanly to the 60-min timeout, shows the underlying diffusion suite is broadly broken on this ROCm 7.2 config (server crashes / unsupported-op errors — see Facts).

Regression Status

Known recurring failure — Never-passed in the queryable window for shard 3.

Recent history of multimodal-gen-test-1-gpu-amd-rocm720 (…, 3) (most recent 15 comparable completed scheduled runs: branch=main, event=schedule, completed only). NOTE: comparison is at the job/shard level only — all shard-3 logs in this window return BlobNotFound, so test-file-level history cannot be reconstructed. Shard 3's step-9 hard-kill signature was confirmed identical (step9 in_progress, ~23-27 min, no end) in the 3 most recent runs.

Date	Run	Shard 3 Status	Note
Jul 18	29654571916	❌ Failed	hard-kill @ ~16 min into step 9
Jul 17	29601526478	❌ Failed	same signature (step9 in_progress, 25 min)
Jul 16	29521557976	❌ Failed	same signature (23 min)
Jul 15	29438298954	❌ Failed	same signature (23 min)
Jul 14	29355401408	❌ Failed	—
Jul 13	29273187026	❌ Failed	—
Jul 12	29202664092	❌ Failed	—
Jul 11	29162225443	❌ Failed	—
Jul 10	29113205616	❌ Failed	—
Jul 09	29039527644	❌ Failed	—
Jul 08	28964460204	❌ Failed	—
Jul 07	28888292440	❌ Failed	—
Jul 06	28813230096	❌ Failed	—
Jul 05	28749537427	❌ Failed	—
Jul 04	28714582765	❌ Failed	—

Shard 3 has failed in 15/15 comparable scheduled runs (100%). No passing run exists in the lookback range Jul 04 → Jul 18 2026; first failure predates the window. Shard 2 passes in all 15 runs; shards 0/1 are intermittent.

Failure Cluster

"Diffusion 1-GPU shard hard-killed during server test (log never finalized)"

Facts (verified observations, no interpretation)

• Step timeline (this job, from Jobs API): steps 1-8 succeeded; step 9 "Run diffusion server tests (1-GPU)" started 2026-07-18T19:35:24Z, status=in_progress, conclusion=None, no completed_at; step 10 "Upload diffusion failure artifacts" (if: always()) is pending and never started; job completed_at=2026-07-18T19:51:44Z, conclusion=failure. Elapsed in step 9 ≈ 16 min, far under the timeout-minutes: 60 on that step (.github/workflows/pr-test-amd-rocm720.yml:728).
• Log unrecoverable: GET /actions/jobs/88106525161/logs → BlobNotFound. Same for historical shard-3 jobs 87954595307 and 87699749819. Sibling shard-0 log (88106525162) IS available (it ran to the 60-min timeout).
• No artifacts: run 29654571916 has 0 artifacts; the diffusion-failure upload (retention 7 days) produced nothing because the upload step never ran.
• Reproducibility: shard 3 failed 15/15 comparable runs; the step-9 in_progress/no-end hard-kill signature is confirmed identical across the 3 most recent runs.
• Sibling shard-0 diffusion errors (same run, partition 0) — indicates the suite is broadly broken on ROCm 7.2, though these are NOT proven to be shard 3's specific failure:
  • NotImplementedError: AITer backend does not support Grouped Query Attention yet. (Cosmos3-Nano transformer load), sib log line ~4687
  • AttributeError: module diffusers has no attribute Cosmos3OmniTransformer, line ~4759
  • EOFError (server process death), line ~4782
  • Models exercised by shard 0: Z-Image-Turbo, Cosmos3-Nano, Qwen-Image-Edit, Qwen-Image-Layered (all attention_backend: aiter).

Hypothesised Causes (with confidence, with disconfirming evidence)

• [LOW] A diffusion server process assigned to partition 3 hangs or crashes hard enough to take down the runner/docker exec before the 60-min step timeout, so the log is never finalized and always() upload never runs.
  • Supporting: step 9 consistently dies at ~16-23 min with no conclusion across every recent shard-3 run; always() upload never executes (only happens on abrupt kill, not on normal step failure/timeout); shard-0 shows server-side EOFError/crash behavior on this ROCm 7.2 config.
  • Disconfirming: shard 0/1 (same job) DID reach the 60-min timeout and finalize logs, so the kill is not universal to the job — it is specific to partition 3's model set, which I could not reconstruct. No stack trace was recoverable for shard 3 itself.
  • How to verify: re-run shard 3 with live log tailing, or run python3 sglang/multimodal_gen/test/run_suite.py --suite 1-gpu --partition-id 3 --total-partitions 4 -k "not flux_2" in container ci_sglang on an MI325 1-GPU runner and watch for which model launch hangs/segfaults.
• [LOW] Underlying diffusion-suite breakage on ROCm 7.2 (AITer GQA unsupported / missing Cosmos3OmniTransformer / server EOFError seen in shard 0) also affects a model in partition 3.
  • Supporting: shard 0 of the same run shows these three concrete errors; the whole multimodal-gen-* family (1-gpu p0/p1/p3, 2-gpu p0/p1, unit-test) failed in this run.
  • Disconfirming: these errors were caught with a native fallback in shard 0 and did not by themselves stop shard 0 from continuing to the timeout, so they don't obviously explain shard 3's early hard-kill.
  • How to verify: recover a shard-3 log via rerun; confirm whether the same model/error appears in partition 3.
• No hypothesis reaches MEDIUM confidence: shard 3's own log/artifacts are unavailable, so there is no code-level or stack-trace evidence for this specific shard. This is an infrastructure/observability gap as much as a test failure.

In-flight Fix Check

• In-flight fix: ❌ none found matching the shard-3 hard-kill. Searched open PRs: multimodal-gen+amd (39 results, none address a diffusion shard hang), diffusion+partition (6, none), rocm720+diffusion (8, none). Related but not a direct fix: #29584 "ci(amd): fix three pre-existing MI325 CI failures" and #31483 "[AMD] ci: run nested multimodal_gen unit suites on AMD" — neither targets this partition-3 kill.

Suggested Triage Steps

• Fix observability first — the log/artifact loss is the primary blocker to diagnosis. On the shard-3 runner, capture step-9 output to a file and tee it, or reduce run_suite.py per-model server-launch timeout so a hang surfaces as a normal step failure (which would finalize the log and trigger the always() upload).
• Reproduce locally: run run_suite.py --suite 1-gpu --partition-id 3 --total-partitions 4 -k "not flux_2" in container ci_sglang on an MI325 1-GPU host with live stdout; identify which model launch hangs/segfaults.
• Once the offending model is known, check whether it hits the shard-0 signatures (AITer GQA NotImplementedError, missing Cosmos3OmniTransformer, server EOFError) and file/pin a targeted fix or -k skip for ROCm 7.2.
• Compare against pr-test-amd.yml (ROCm 6.x) shard 3 to see whether this is ROCm-7.2-specific.

Status (factual, no priority assignment)

• Persistence: Failing every comparable scheduled run for ≥15 days (Jul 04 → Jul 18 2026); no green shard-3 run in the queryable window.
• Scope: The multimodal-gen-* family is broadly red in this run — 1-GPU shards 0/1/3, 2-GPU shards 0/1, and the unit-test job all failed (shard 2 passed). This shard-3 job is one of 11 failed jobs in run 29654571916.
• Blocked work: Blocks diffusion/multimodal-gen 1-GPU CI signal on AMD ROCm 7.2 for partition 3; also degrades observability (no logs/artifacts recoverable).
• In-flight fix: ❌ none found — needs new investigation (start with observability fix + local repro of partition 3).

multimodal-gen-test-1-gpu-amd-rocm720 (linux-mi325-1gpu-sglang, 0) — failed step(s): Run diffusion server tests (1-GPU)

Commit Info

• sglang: b3a0185c
• aiter: 9127c94a (v0.1.17.dev110, source: Dockerfile default — AITER_COMMIT_OVERRIDE empty; not relevant to this failure)

Failed Tests

Test File	Test Function	Error	Log
python/sglang/multimodal_gen/test/server/test_server_1_gpu.py	TestDiffusionServerOneGpu::test_diffusion_generation[longlive2_i2v]	Step timed out after 60 min while the longlive2_i2v generation was still running; server warmup for this case also raised ValueError: num_frames must be divisible by num_frames_per_block for causal DMD denoising	Log

Failure Summary

All 15 earlier diffusion testcases on this shard passed. The final testcase longlive2_i2v (LongLive-2.0-5B I2V, 960x928, 61 frames) hit the workflow step's 60-minute wall clock and the job was killed (##[error]The action 'Run diffusion server tests (1-GPU)' has timed out after 60 minutes.). During this case's server warmup, a ValueError was also raised at longlive2.py:411 (non-fatal — startup continued), after which the real request was accepted and generated but never completed before the timeout.

Regression Status

New regression / Never-passed — longlive2_i2v was introduced on 2026-07-14 and has failed (identical signature) in every completed scheduled run since. It has no green run in the queryable window.

Recent history of test_server_1_gpu.py (test_diffusion_generation[longlive2_i2v]) in job multimodal-gen-test-1-gpu-amd-rocm720 (most recent comparable completed scheduled runs, branch=main, event=schedule; testcase located across shards since the job is sharded):

Date	Run	Job (shard that ran the case)	Test Status	Error
Jul 18	29654571916	shard 0	❌ Failed	warmup ValueError + 60-min timeout
Jul 17	29601526478	shard	❌ Failed	warmup ValueError + 60-min timeout
Jul 16	29521557976	shard	❌ Failed	warmup ValueError + 60-min timeout
Jul 15	29438298954	shard	❌ Failed	warmup ValueError + 60-min timeout
Jul 14	29355401408	shard	❌ Failed	warmup ValueError + 60-min timeout
Jul 13	29273187026	—	⚪ Not present	testcase did not exist yet (0 longlive2 mentions across all 4 shards)
Jul 12	29202664092	—	⚪ Not present	testcase did not exist yet
Jul 11	29162225443	—	⚪ Not present	testcase did not exist yet
Jul 10	29113205616	—	⚪ Not present	testcase did not exist yet

First observed failure: 2026-07-14. Last passing: never (the case has never produced PASSED diffusion testcase: longlive2_i2v). No 60-minute timeout occurred on this job in the Jul 10–13 runs — the timeout is new since Jul 14.

Failure Cluster

LongLive2 I2V: 60-min diffusion-suite step timeout (+ warmup num_frames_per_block divisibility ValueError)

Facts (verified observations)

• ##[error]The action 'Run diffusion server tests (1-GPU)' has timed out after 60 minutes. at 2026-07-18 20:41:04 (main log line 7652). The test step's first case started 19:41:46 (line 4381), so the whole test_server_1_gpu.py suite consumed the full 60-min budget.
• longlive2_i2v was the LAST case, starting 20:36:43 (line 7353). Its warmup raised ValueError: num_frames must be divisible by num_frames_per_block for causal DMD denoising, traceback at python/sglang/multimodal_gen/runtime/pipelines_core/stages/model_specific_stages/longlive2.py, line 411, in _forward_one_shot_common (lines 7475–7528). Server logged Synthetic server warmup failed; continuing startup then The server is fired up and ready to roll! — warmup error is non-fatal.
• After warmup, the real request (num_frames=61, 960x928) was accepted (POST /v1/videos ... 200 OK, 20:39:49); the client then polled GET /v1/videos ... 200 OK once per second from 20:39:50 to 20:41:03 (still generating) until the step was killed. No completed-video result was ever logged.
• Warmup req was 960x928x17f (17 pixel frames). num_frames_per_block=8 for LongLive2 (python/sglang/multimodal_gen/configs/models/dits/longlive2.py:79); VAE scale_factor_temporal=4 (per HF config in log). 17 pixel frames → 5 latent frames; 5 % 8 ≠ 0 → the ValueError. The 61-frame real request → 16 latent frames; 16 % 8 = 0, so the real request itself passes the divisibility check (its problem is time, not divisibility).
• Identical warmup ValueError + 60-min timeout appear verbatim in the Jul 14 earliest-failing log (longlive2.py:411, 2026-07-14 23:22:49; timeout 23:27:34).
• longlive2 does not appear at all in any of the 16 shard logs of the Jul 10–13 runs (grep count 0), and none of those shards hit a 60-min timeout.

Hypothesised Causes (with confidence)

• [HIGH] The longlive2_i2v testcase (added by #27639 "[diffusion] model: support LongLive 2.0 T2V and I2V inference", merged 2026-07-14) turned this job red.
  • Supporting: testcase + 60-min timeout are both absent Jul 10–13 and present every run Jul 14–18; PR #27639 merged exactly on the Jul 14 boundary and adds the LongLive2 model, pipeline, warmup stage, and the longlive2_i2v gpu case.
  • Disconfirming: the exact adding commit could not be byte-verified locally (shallow clone of 100 commits; git show on boundary SHAs is unreliable — the range endpoint cb47a68717 is confirmed via API to be an unrelated PD KV-cache change, not the longlive2 add). Attribution rests on log presence + PR merge date, not a diff.
  • How to verify: run test_server_1_gpu.py -k longlive2_i2v standalone on an MI325 ROCm 7.2 container with an extended timeout; confirm it is the case that exceeds budget.
• [FACT] The warmup for longlive2_i2v constructs a 17-frame request (5 latent frames) that is not divisible by num_frames_per_block=8, raising the ValueError at longlive2.py:411. This is a real warmup-request-construction defect, though non-fatal.
  • Supporting: verbatim log traceback; the divisibility math above.
  • Disconfirming: it does not by itself fail the job (startup continues; the real 61-frame request is accepted).
  • How to verify: make the warmup req use a frame count whose latent length is a multiple of 8, re-run, confirm the ValueError disappears.
• [MEDIUM] The job-failing symptom is the 60-min step timeout: longlive2_i2v is an expensive 960x928×61-frame video I2V generation placed last in an already ~60-min suite, leaving it <5 min (most of which is model load + the 90 s failed warmup); its generation is still polling when the step is killed.
  • Supporting: per-second GET /v1/videos polling for 74 s with no completion; the case never finished in 5 consecutive runs.
  • Disconfirming: we cannot tell from ≤75 s of runtime whether the generation would ever complete with more time versus being a genuine hang — no completed-video result exists in any run.
  • How to verify: run the case alone with a large timeout and observe whether it completes and how long it takes on MI325/ROCm.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for longlive2 (0), num_frames_per_block (0), and causal DMD (5 results, all unrelated: Lingbot/VAE/nsys/broadcast). No open PR addresses the warmup divisibility bug or the longlive2_i2v timeout.

Suggested Triage Steps

• Run longlive2_i2v standalone (test_server_1_gpu.py -k longlive2_i2v) on an MI325 ROCm 7.2 container with an extended timeout to determine whether the I2V generation completes at all, and how long it takes — this separates "too slow for the shared 60-min budget" from "genuine hang".
• Fix the LongLive2 I2V warmup request so its frame count yields a latent length divisible by num_frames_per_block=8 (warmup currently uses 17 frames → 5 latent frames); verify the longlive2.py:411 ValueError disappears.
• If the case is simply expensive, consider re-sharding so longlive2_i2v is not the last case in a 60-min-bounded suite (or give it its own shard / raise the step timeout for this suite).
• Ping the author of #27639 (introduced the testcase and warmup path) to own the fix, since this has blocked the shard on every run since 2026-07-14.

Status

• Persistence: Failing every completed scheduled run for 5 days (since 2026-07-14), never passed since introduction.
• Scope: multimodal-gen-test-1-gpu-amd-rocm720 on pr-test-amd-rocm720.yml (whichever shard longlive2_i2v lands on each run — shard 0 in this run). Not verified on other workflows/backends; the warmup ValueError is device-independent Python logic and would likely reproduce on NVIDIA if the same warmup path runs.
• Blocked work: makes this multimodal-gen shard red on every ROCm 7.2 scheduled run, masking the pass/fail signal of any other case sharing that shard.
• In-flight fix: ❌ none found — needs a new PR (warmup divisibility fix and/or suite time/sharding change).

stage-b-test-1-gpu-large-amd-rocm720 (linux-mi325-1gpu-sglang, 0) — failed step(s): Run test

Commit Info

• sglang: b3a0185c
• aiter: 9127c94a (source: Dockerfile default / dev image v0.1.17.dev110+g9127c94a1 preserved; AITER_COMMIT_OVERRIDE empty)

Failed Tests

Test File	Test Function	Error	Log
test/registered/perf/test_bench_serving_1gpu_part1.py	test_offline_throughput_with_triton_attention_backend	AssertionError: 3150.30 not greater than 3500 (after 1 retry: 3155.41 then 3150.30) → retry() exceed maximum number of retries	Log

Suite result: 13/14 test files passed; only test_bench_serving_1gpu_part1.py failed (exit code 1), which drove the job's exit code 255. The other 13 files (sessions, spec/eagle, cuda_graph piecewise, attention unittests, zaya) all passed.

Failure Summary

The AMD single-GPU offline-throughput perf gate test_offline_throughput_with_triton_attention_backend measured 3150–3155 token/s for Llama-3.1-8B with the triton attention backend, below its AMD assertion threshold of 3500 (test_bench_serving_1gpu_part1.py:120, guarded by is_in_amd_ci()). The test retries once and the retry also fell short, so it errored out. A transient LocalEntryNotFoundError/OSError (HF hub offline) appeared once mid-run during a server launch but the subsequent launch succeeded, and every other test file passed — so that HF blip was non-fatal and unrelated to the failure.

Regression Status

Never-passed (no green run found) — this test has failed in every comparable completed scheduled run across the entire lookback window (Jul 4 → Jul 18, 2026). It has never reached 3500 in the queryable history; the best observed value was 3212 (~8% short). This is NOT introduced by the head commit b3a0185c — the failure predates it by two weeks.

Recent history of test_bench_serving_1gpu_part1.py (test_offline_throughput_with_triton_attention_backend) in job stage-b-test-1-gpu-large-amd-rocm720 (linux-mi325-1gpu-sglang, 0) (15 most recent completed event=schedule, branch=main runs):

Date	Run	Test File Status	Failed Function	Throughput (final) vs 3500
Jul 18	7197	❌ Failed	test_offline_throughput_with_triton_attention_backend	3150.30
Jul 17	7193	❌ Failed	same	3087.65
Jul 16	7186	❌ Failed	same	3157.62
Jul 15	7175	❌ Failed	same	3162.05
Jul 14	7167	❌ Failed	same	3167.52
Jul 13	7161	❌ Failed	same	3174.25
Jul 12	7156	❌ Failed	same	3197.36
Jul 11	7152	❌ Failed	same	3150.17
Jul 10	7150	❌ Failed	same	3175.84
Jul 9	7146	❌ Failed	test_offline_throughput_default (early abort, elapsed 135s)	triton test not reached (inconclusive)
Jul 8	7145	❌ Failed	test_offline_throughput_with_triton_attention_backend	3156.68
Jul 7	7124	❌ Failed	same	2563.65
Jul 6	7117	❌ Failed	same	3131.91
Jul 5	7105	❌ Failed	same	3044.11
Jul 4	7104	❌ Failed	same	2797.34

First observed failure in window: Jul 4, 2026. No passing run for this test file in the entire 15-run window.

Failure Cluster

AMD triton-backend offline-throughput perf gate below threshold (3500)

Facts (verified observations)

• test_bench_serving_1gpu_part1.py:120 asserts output_throughput > 3500 only when is_in_amd_ci() (else > 3700); measured 3155.41 then 3150.30 (retry) in this run.
• 14 of 14 runs that reached the triton test failed it; measured throughput ranged 2563.65 – 3212.03 token/s, never ≥ 3500 (verified from each run's log).
• Run 7146 failed the same file on a different function (test_offline_throughput_default) with only 135s elapsed and 8/13 files passing overall — an early server-launch/abort, so the triton test was not cleanly measured there.
• aiter is the Dockerfile-default dev image (v0.1.17.dev110+g9127c94a1), no override; this is pr-test-amd-rocm720.yml, not a scout run — no aiter A/B applicable.
• Head commit b3a0185c merged Jul 18; the failure goes back to at least Jul 4, so the head commit is exonerated.

Hypothesised Causes (with confidence)

• [MEDIUM] The AMD threshold 3500 (test_bench_serving_1gpu_part1.py:120) is miscalibrated above the sustained throughput this MI325 / ROCm 7.2 runner delivers.
  • Supporting: 14/14 runs cluster tightly at ~3040–3210 token/s (best-ever 3212), consistently ~8–10% below 3500; no green run exists anywhere in the queryable window.
  • Disconfirming: no green baseline is visible in the 15-run window, so a genuine perf regression that predates Jul 4 cannot be excluded from the data alone; occasional dips to 2563/2797 suggest run-to-run variance/contention rather than a fixed ceiling.
  • How to verify: query scheduled runs before Jul 4 to find the last green run for this test; if none exists since the is_in_amd_ci() 3500 branch was added, the threshold was set too high for this runner.
• [LOW] Genuine throughput regression that landed before the queryable window.
  • Supporting: values sometimes drop well under 3000 (2563 on Jul 7, 2797 on Jul 4).
  • Disconfirming: recent runs are stable ~3100–3200 with no downward trend; low outliers correlate with the early-abort/contention run pattern, not a monotonic regression.
  • How to verify: bisect scheduled history backward to locate a pass_sha..fail_sha window, then git log that range for attention/moe/gemm perf changes.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for test_bench_serving+throughput (1 hit, unrelated PCG capture fix #27650) and triton+throughput+amd (feature PRs only, none targeting this perf gate). No open PR adjusts this threshold or addresses this regression.

Suggested Triage Steps

• Extend the scheduled-run history for job stage-b-test-1-gpu-large-amd-rocm720 (..., 0) back beyond Jul 4 to find the last run (if any) where test_offline_throughput_with_triton_attention_backend passed; anchor a pass_sha..fail_sha window from that.
• If a green run exists, bisect that commit range for triton-attention / GEMM / MoE perf changes reachable from the Llama-3.1-8B triton-backend path.
• If no green run exists since the AMD 3500 branch was introduced, treat the threshold as miscalibrated for MI325/ROCm 7.2: recalibrate to the observed p50 (~3150) with margin, or quarantine the assertion on AMD until a perf owner sets a realistic bound.
• Cross-check the same test on nightly-test-amd.yml (MI325) to confirm whether the 3500 gate ever passes on any AMD runner.

Status (factual)

• Persistence: Failing in every completed scheduled run for at least 15 days (since Jul 4, 2026); never green in the queryable window.
• Scope: test_offline_throughput_with_triton_attention_backend in test/registered/perf/test_bench_serving_1gpu_part1.py, partition-0 of stage-b-test-1-gpu-large-amd-rocm720 on pr-test-amd-rocm720.yml. The other 13 files in this partition pass.
• Blocked work: Keeps this AMD ROCm 7.2 stage-b perf shard permanently red, masking any real regression this gate might later catch.
• In-flight fix: ❌ none found in open PRs.

multimodal-gen-test-1-gpu-amd-rocm720 (linux-mi325-1gpu-sglang, 1) — failed step(s): N/A

Commit Info

• sglang: b3a0185c
• aiter: 9127c94a (v0.1.17.dev110+g9127c94a1; source: Dockerfile default, AITER_COMMIT_OVERRIDE empty — verified from sibling part-0 [CI-AITER-CHECK] markers)

Failed Tests

Test File	Test Function	Error	Log
python/sglang/multimodal_gen/test/server/test_server_1_gpu.py	TestDiffusionServerOneGpu::test_diffusion_generation (partition 1 of 4, -k "not flux_2")	Step wall-clock timeout after 60 minutes (not a test assertion)	Job

Note on evidence: the target job's own log blob is unavailable (GitHub returned BlobNotFound, and part 1 is absent from the run-log archive). The classification is therefore inferred from the identical-signature sibling part 0 (job 88106525162), whose log explicitly shows ##[error]The action 'Run diffusion server tests (1-GPU)' has timed out after 60 minutes. Both parts ran ~72–75 min (= 60 min step timeout + ~12–15 min setup); part 2 (lighter partition) passed in 13.7 min.

Failure Summary

The multimodal-gen-test-1-gpu-amd-rocm720 partition-1 shard did not fail on a test assertion — its Run diffusion server tests (1-GPU) step exceeded its timeout-minutes: 60 wall clock and was killed. This is a shard-level runtime-budget overrun of the LPT-partitioned 1-GPU diffusion suite (run_suite.py --suite 1-gpu --partition-id 1 --total-partitions 4), not a defect in any single model test. Partitions 0, 1, and 3 all overran; only the lighter partition 2 fit within 60 min.

Regression Status

Known recurring failure (chronic timeout, ~11 consecutive runs).

Recent history of test_server_1_gpu.py partition 1 in job multimodal-gen-test-1-gpu-amd-rocm720 (…, 1) (most recent comparable completed scheduled runs: branch=main, event=schedule):

Date	Run	Part-1 Status	Duration	Signature
Jul 18	29654571916	❌ Failed	75.0 min	60-min timeout
Jul 17	29601526478	❌ Failed	72.1 min	60-min timeout
Jul 16	29521557976	❌ Failed	71.0 min	60-min timeout
Jul 15	29438298954	❌ Failed	72.1 min	60-min timeout
Jul 14	29355401408	❌ Failed	74.0 min	60-min timeout
Jul 13	29273187026	❌ Failed	68.0 min	60-min timeout
Jul 12	29202664092	❌ Failed	64.0 min	60-min timeout
Jul 11	29162225443	❌ Failed	72.1 min	60-min timeout
Jul 10	29113205616	❌ Failed	70.0 min	60-min timeout
Jul 9	29039527644	❌ Failed	70.4 min	60-min timeout
Jul 8	28964460204	❌ Failed	71.1 min	60-min timeout
Jul 7	28888292440	✅ Passed	54.4 min	fit under 60 min (barely)
Jul 6	28813230096	❌ Failed	56.0 min	under 60 min — likely a different (real) failure

Last known passing: Jul 7 (54.4 min, just under the limit). First failure of the current streak: Jul 8. The shard has run right at the 60-min boundary for weeks, so any small runtime increase (heavier model, download variance, added case) tips it over.

Failure Cluster

60-minute step timeout of the 1-GPU diffusion suite (partition runtime overrun) — symptom-level grouping shared by partitions 0, 1, and 3 of multimodal-gen-test-1-gpu-amd-rocm720.

Facts (verified observations)

• Job conclusion failure; step 9 Run diffusion server tests (1-GPU) (timeout-minutes: 60) was the failing step; job wall-clock 75.0 min (.github/workflows/pr-test-amd-rocm720.yml:727-728).
• Target job log blob is unavailable (BlobNotFound); part 1 is missing from the run-log archive. Diagnosis uses sibling part 0.
• Part 0 (same run, same step, 71.7 min) log: ##[error]The action 'Run diffusion server tests (1-GPU)' has timed out after 60 minutes. — 11 models (zimage_image_t2i, cosmos3_nano_t2i, qwen_image_edit_ti2i, qwen_image_layered_i2i, wan2_1_t2v_1.3b, wan2_1_t2v_1.3b_upscaling_4x, wan2_1_t2v_1_3b_lora_1gpu, fast_hunyuan_video, wan2_2_ti2v_5b, fastwan2_2_ti2v_5b, sana_wm_ti2v) all PASSED, then it timed out mid-longlive2_i2v.
• Per-run part status is consistent across ≥11 runs: parts 0/1/3 ❌ (60-min timeouts), part 2 ✅ (~12–18 min). LPT partitioning is run_suite.py --total-partitions 4.
• aiter is the Dockerfile default (no override) — unchanged across passing and failing runs; this is pr-test-amd-rocm720.yml (scheduled), not an AITER Scout run.
• A non-fatal warmup error appeared in part 0's longlive2_i2v (ValueError: num_frames must be divisible by num_frames_per_block for causal DMD denoising), but the server logged Synthetic server warmup failed; continuing startup and proceeded — it did NOT cause the timeout.

Hypothesised Causes (with confidence)

• [HIGH] The failure is a cumulative-runtime timeout: partition 1's assigned 1-GPU diffusion models take longer in aggregate than the 60-min step budget on the MI325 ROCm 7.2 runner.
  • Supporting: sibling part 0 (identical 71.7-min signature) explicitly logs the 60-min timeout after running 11 models to completion; parts 0/1/3 consistently overrun while part 2 (lighter) fits; part 1 last passed at 54.4 min (right at the boundary) then flipped to timeouts once runtime crept up.
  • Disconfirming: part 1's own log is unavailable, so I cannot 100% confirm the specific in-flight model at the moment of its timeout (only that its duration matches a 60-min timeout, not a sub-60-min crash).
  • How to verify: re-run with a raised timeout-minutes (or --partition-id 1 locally) and capture per-model timing; sum estimated_full_test_time_s for the models LPT assigns to partition 1 vs. the observed MI325 ROCm720 times.
• [LOW] LPT runtime estimates in testcase_configs.py are calibrated below actual MI325 ROCm 7.2 runtimes, so partition_items_by_lpt produces shards that each exceed 60 min despite "balanced" estimates.
  • Supporting: 3 of 4 shards overrun — a balancing/estimation symptom, not a single slow test.
  • Disconfirming: no direct evidence the estimates changed; could equally be that total suite runtime grew.
  • How to verify: compare get_case_est_time() sums per partition against observed durations from the logs.
• No single-commit regression identified: last-pass→first-fail window (Jul 7→Jul 8) shows no commit to gpu_cases.py/testcase_configs.py in the (shallow) local history; this looks like gradual runtime creep, not one bad commit.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for diffusion+timeout (22 results, all feature/NPU work — none about the 1-GPU shard 60-min timeout or AMD partition rebalancing), multimodal-gen+partition (6, none relevant), and 1-gpu+diffusion+timeout in:title (0). Related but not a fix: #28584 (WIP AMD MI300 runner migration).

Suggested Triage Steps

• Confirm per-model timing on partition 1 by re-running run_suite.py --suite 1-gpu --partition-id 1 --total-partitions 4 -k "not flux_2" on an MI325 ROCm720 container with SGLANG_TEST_NUM_INFERENCE_STEPS=5 (as CI sets it) and no step timeout; sum model runtimes.
• If total > 60 min, raise timeout-minutes for the 1-GPU step and/or increase total-partitions beyond 4 so each shard fits (parts 0/1/3 all overrun today → 4-way split is under-partitioned for ROCm720).
• Recalibrate the LPT estimated_full_test_time_s for the 1-GPU cases against measured MI325 ROCm720 times so partition_items_by_lpt balances against real runtimes.
• Separately, file/track the non-fatal longlive2_i2v warmup error (num_frames must be divisible by num_frames_per_block) — it's not the timeout cause but is noise in the diffusion warmup path.

Status

• Persistence: Chronic 60-min timeout on partition 1 for ~11 consecutive scheduled runs (every run since Jul 8; last green Jul 7 at 54.4 min).
• Scope: Same timeout hits partitions 0, 1, and 3 of multimodal-gen-test-1-gpu-amd-rocm720 every run; part 2 passes. Contributes to pr-test-amd-rocm720.yml being red for 16+ consecutive scheduled runs.
• Blocked work: Blocks 1-GPU diffusion (multimodal_gen) CI signal on AMD ROCm 7.2.
• In-flight fix: ❌ none found in open PRs — needs a new PR (raise timeout / add partitions / recalibrate estimates).

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-07-19 19:58 UTC)

[amd-bot]

nightly-test-amd-rocm720.yml — 26 failure(s)

Run started (UTC): 2026-07-18 17:45 UTC
Last scanned (UTC): 2026-07-19 22:01 UTC
Commits: sglang d4801be

Summary

Counts: 26 failures · 12 clusters · 9 carrying over (≥3 days) · 3 new today

#	Cluster	Job	Test File	Test Function	Status	Hypothesis (confidence)
1	DSV3.2 basic GSM8K acc < 0.93	nightly-accuracy-8-gpu-mi35x-deepseek-v32-rocm720	test_deepseek_v32_eval_mi35x.py	test_deepseek_v32_accuracy	Never-passed (≥Jun 29), acc 0.87–0.915	[MEDIUM] 0.93 threshold miscalibrated / real DSA-tilelang acc gap
2	DSV3.2 basic GSM8K acc < 0.93	nightly-accuracy-8-gpu-mi35x-deepseek-v32-rocm720	test_deepseek_v32_eval_mi35x.py	test_deepseek_v32_accuracy	Never-passed, acc 0.855	[MEDIUM] threshold / acc gap
3	DSV3.2 basic GSM8K acc < 0.93	nightly-accuracy-8-gpu-mi35x-deepseek-v32-rocm720	test_deepseek_v32_eval_mi35x.py	test_deepseek_v32_accuracy	Never-passed, acc 0.895	[MEDIUM] threshold / acc gap
4	GPU mem fault Qwen3-30B-A3B bf16	nightly-accuracy-8-gpu-mi35x-rocm720	test_qwen3_moe_eval_mi35x.py	test_qwen3_moe_accuracy	Never-passed (≥Jul 2)	[LOW] HIP mem fault in Qwen3 bf16 MoE+aiter path (async, no kernel attribution)
5	GPU mem fault Qwen3-30B-A3B bf16	nightly-accuracy-8-gpu-mi35x-rocm720	test_qwen3_moe_eval_mi35x.py	test_qwen3_moe_accuracy	Never-passed	[LOW] same
6	GPU mem fault Qwen3-30B-A3B bf16	nightly-accuracy-8-gpu-mi35x-rocm720	test_qwen3_moe_eval_mi35x.py	test_qwen3_moe_accuracy	Never-passed	[LOW] same
7	InternVL CLIP-processor crash + VLM acc	nightly-accuracy-2-gpu-vlm-rocm720	test_vlms_mmmu_eval_amd.py	test_mmmu_vlm_models	InternVL crash since Jul 14; acc chronic	[MEDIUM] AutoProcessor→CLIPImageProcessor, common.py:521 unguarded .tokenizer
8	InternVL CLIP-processor crash + VLM acc	nightly-accuracy-2-gpu-vlm-rocm720	test_vlms_mmmu_eval_amd.py	test_mmmu_vlm_models	crash since Jul 14	[MEDIUM] same
9	InternVL CLIP-processor crash + VLM acc	nightly-accuracy-2-gpu-vlm-rocm720	test_vlms_mmmu_eval_amd.py	test_mmmu_vlm_models	crash since Jul 14	[MEDIUM] same
10	GLM-5.1-MXFP4 setUpClass crash (JIT/hang/runner)	nightly-2-gpu-mi35x-glm51-mxfp4-rocm720	test_glm51_mxfp4_tp2_gsm8k_mi35x.py	setUpClass	Never-passed (introduced Jul 16)	[HIGH] topk_impl.cuh:27 unguarded <cooperative_groups.h> on gfx950 (Jul 16 log)
11	GLM-5.1-MXFP4 setUpClass crash (JIT/hang/runner)	nightly-2-gpu-mi35x-glm51-mxfp4-rocm720	test_glm51_mxfp4_tp2_gsm8k_mi35x.py	setUpClass	this instance: runner-loss (no log)	[HIGH] runner lost mid-step; underlying JIT/hang unverified
12	GLM-5.1-MXFP4 setUpClass crash (JIT/hang/runner)	nightly-2-gpu-mi35x-glm51-mxfp4-rocm720	test_glm51_mxfp4_tp2_gsm8k_mi35x.py	setUpClass	red since Jul 13; log unavailable	[MEDIUM] GPU-Hang or cooperative_groups.h (two symptom variants)
13	2-GPU GSM8K suite (Mixtral acc / runner-loss)	nightly-accuracy-2-gpu-rocm720	test_gsm8k_eval_amd.py	test_gsm8k_all_models	recurring; Mixtral 0.307<0.69	[LOW] Mixtral MoE acc deficit on ROCm; Qwen3-4B missing-threshold cosmetic
14	2-GPU GSM8K suite (Mixtral acc / runner-loss)	nightly-accuracy-2-gpu-rocm720	test_gsm8k_eval_amd.py	test_gsm8k_all_models	runner-loss ≥3/5 runs	[FACT] MI325 runner lost communication mid-Mixtral
15	2-GPU GSM8K suite (Mixtral acc / runner-loss)	nightly-accuracy-2-gpu-rocm720	test_gsm8k_eval_amd.py	test_gsm8k_all_models	recurring; log truncated	[MEDIUM] terminating AssertionError beyond truncated log (post-Mixtral)
16	NameError transfer_kv_mamba (ROCm import)	nightly-test-1-gpu-kernel-rocm720	test_transfer_mamba.py	test_mamba_kernel_* (12)	Never-passed on ROCm (since Jul 16)	[HIGH] memory_pool_host.py:41 if _is_cuda: guard; call at :394 reachable on HIP (#30535)
17	NameError transfer_kv_mamba (ROCm import)	nightly-test-1-gpu-kernel-rocm720	test_transfer_mamba.py	test_mamba_kernel_* (12)	Never-passed	[HIGH] same
18	NameError transfer_kv_mamba (ROCm import)	nightly-test-1-gpu-kernel-rocm720	test_transfer_mamba.py	test_mamba_kernel_* (12)	Never-passed	[HIGH] same
19	HIP capture watchdog abort (Qwen3-235B)	nightly-8-gpu-mi35x-qwen3-235b-mxfp4-rocm720	test_qwen3_instruct_mxfp4.py	setUpClass	Never-passed (≥Jun 29)	[MEDIUM] aiter custom all-reduce during HIP graph capture → NCCL watchdog hipErrorCapturedEvent
20	HIP capture watchdog abort (Qwen3-235B)	nightly-8-gpu-mi35x-qwen3-235b-mxfp4-rocm720	test_qwen3_instruct_mxfp4.py	setUpClass	signature since Jul 17	[MEDIUM] same (#23581 candidate)
21	DSR1-MXFP4 MTP GSM8K acc < 0.944	nightly-8-gpu-mi35x-deepseek-r1-mxfp4-tp4-rocm720	test_deepseek_r1_mxfp4_tp4_mtp_mi35x.py	test_gsm8k	Never-passed, acc ~0.85	[MEDIUM] EAGLE/MTP verify not output-equivalent, or threshold pinned to non-MTP 0.944
22	DSR1-MXFP4 MTP GSM8K acc < 0.944	nightly-8-gpu-mi35x-deepseek-r1-mxfp4-tp4-rocm720	test_deepseek_r1_mxfp4_tp4_mtp_mi35x.py	test_gsm8k	Never-passed, acc 0.854	[MEDIUM] same
23	GPU mem fault DSV4-Pro fp4 MTP	nightly-8-gpu-mi35x-deepseek-v4-pro-mtp-rocm720	test_deepseek_v4_pro_fp4_mtp.py	test_a_gsm8k (+test_b_bs_1_speed)	recurring (symptom drifts) since Jul 13	[LOW] GPU illegal mem access fp4-MoE+EAGLE prefill; also accept-len 2.79<2.8
24	1200s watchdog hang DSA EAGLE-MTP	nightly-accuracy-8-gpu-mi35x-deepseek-v32-mtp-rocm720	test_deepseek_v32_mtp_eval_mi35x.py	test_a_gsm8k	🆕 new signature Jul 18 (passed Jul 17)	[LOW] nondeterministic GPU stall in DSA+EAGLE-MTP draft-extend
25	DSV4-Flash FP4 GSM8K acc < 0.91	nightly-8-gpu-mi35x-deepseek-v4-flash-rocm720	test_deepseek_v4_flash_fp4.py	test_a_gsm8k	🆕 new Jul 18 (passed Jul 17); 0.910 vs 0.91	[MEDIUM] threshold too tight vs variance (0.910–0.921)
26	No backend type cpu (Qwen3-VL DP vision)	nightly-4-gpu-rocm720	test_encoder_dp.py	test_vlm_mmmu_benchmark	🆕 signature Jul 19 (Qwen3-VL never green)	[HIGH] CPU tensor → RCCL device_group all_gather; is_shm_available False on AMD host

Cluster: DSV3.2 basic GSM8K acc < 0.93 (rows 1-3)

• Affected: nightly-accuracy-8-gpu-mi35x-deepseek-v32-rocm720 across 3 runs; same test/function, acc 0.855–0.895 vs 0.93.
• Shared facts: server runs clean (200/200 OK); DSA tilelang TP8; Never-passed since ≥Jun 29; MTP sister job passes.
• Top hypothesis: [MEDIUM] 0.93 miscalibrated vs achievable ~0.88. Disconfirming: no reference baseline proves 0.93 achievable → could be real acc gap.
• In-flight fix: ❌ none (WIP branch amd_fix_dpskv32_basic_0518, no PR).
• Suggested triage: run vs NVIDIA reference to decide threshold-vs-gap; A/B a non-tilelang DSA decode backend.

Cluster: GPU mem fault Qwen3-30B-A3B bf16 (rows 4-6)

• Affected: nightly-accuracy-8-gpu-mi35x-rocm720 ×3; identical Memory access fault … 0x…009000 → scheduler exit -6 at prefill result sync.
• Shared facts: only bf16 Qwen3-30B-A3B faults (gpt-oss/qwen3-coder pass same GPUs); aiter unchanged; Never-passed ≥Jul 2.
• Top hypothesis: [LOW] OOB access in bf16 MoE+aiter path. Disconfirming: async fault, no kernel frame; aiter proximity ≠ causation.
• In-flight fix: ❌ none directly (#31405 adjacent — TP2/FP8/ROCm7.0).
• Suggested triage: repro with HIP_LAUNCH_BLOCKING=1 + coredump; A/B --attention-backend triton / --moe-runner-backend triton.

Cluster: InternVL CLIP-processor crash + VLM acc (rows 7-9)

• Affected: nightly-accuracy-2-gpu-vlm-rocm720 ×3; AttributeError: 'CLIPImageProcessor' … 'tokenizer' aborts loop → 7 later models untested; Qwen2.5-VL/Qwen3-VL below threshold.
• Shared facts: crash at common.py:521; regression Jul 5→14 window; accuracy shortfalls chronic; aiter unchanged.
• Top hypothesis: [MEDIUM] AutoProcessor returns bare CLIPImageProcessor, unguarded .tokenizer. Disconfirming: same source line passed Jul 5 → likely a transformers/env bump trigger.
• In-flight fix: ❌ none for InternVL crash; ⚠️ WIP #26178 / #30008 for AMD VLM thresholds.
• Suggested triage: repro AutoProcessor.from_pretrained type on ROCm; diff transformers Jul 5 vs Jul 19 image; move popen_launch_server inside per-model try/except.

Cluster: GLM-5.1-MXFP4 setUpClass crash (rows 10-12)

• Affected: nightly-2-gpu-mi35x-glm51-mxfp4-rocm720 ×3; server exit -9 / runner-loss / GPU-Hang; Never-passed since introduced Jul 16 (last green Jul 5).
• Shared facts: Jul 16 log = topk_impl.cuh:27 'cooperative_groups.h' not found for gfx950 during decode graph capture; Jul 17/18/19 logs expired/runner-loss.
• Top hypothesis: [HIGH] (Jul 16) DSV4 topk_v2 JIT unbuildable on HIP. Disconfirming: recent instances are runner-loss/GPU-hang, not confirmed same signature.
• In-flight fix: ⚠️ #31131 (merged Jul 15, ineffective); #31260 candidate (GLM DSA on ROCm).
• Suggested triage: re-run to get fresh log; repro dpsk_v4 topk_v2 JIT build on gfx950; check HIP cooperative_groups.h shim.

Cluster: 2-GPU GSM8K suite — Mixtral acc / runner-loss (rows 13-15)

• Affected: nightly-accuracy-2-gpu-rocm720 ×3; mixed modes — Mixtral 0.307<0.69, MI325 runner-lost-communication, and log-truncation hiding final verdict.
• Shared facts: same test file; job never green in window; [aiter] torch solution:0 GEMM-log flood truncates logs; Qwen3-4B cosmetic ❌ (no threshold).
• Top hypothesis: [FACT] runner-loss (Jul 14/17/18) + [LOW] genuine Mixtral MoE acc deficit. Disconfirming: runner-loss sub-cause (mem vs net) unproven.
• In-flight fix: ❌ none (infra); ⚠️ #29933 (GSM8K answer extraction) candidate for Mixtral.
• Suggested triage: check MI325 runner dmesg/OOM; fetch GITHUB_STEP_SUMMARY table to recover truncated verdict; add Qwen3-4B threshold.

Cluster: NameError transfer_kv_mamba (rows 16-18)

• Affected: nightly-test-1-gpu-kernel-rocm720 ×3; 12/16 cases fail NameError: transfer_kv_mamba_lf_pf at memory_pool_host.py:394.
• Shared facts: if _is_cuda: import guard (line 41) excludes HIP; AOT sibling uses _is_cuda or _is_hip; introduced #30535 (Jul 16), Jul 14 passed (test absent).
• Top hypothesis: [HIGH] CUDA-gated import + ROCm-reachable call path. Disconfirming: (none found).
• In-flight fix: ❌ none found.
• Suggested triage: confirm whether JIT mamba kernel builds on HIP; if yes widen guard to _is_cuda or _is_hip, else route ROCm to direct backend / skip test.

Cluster: HIP capture watchdog abort — Qwen3-235B (rows 19-20)

• Affected: nightly-8-gpu-mi35x-qwen3-235b-mxfp4-rocm720 ×2; all 4 ranks hipErrorCapturedEvent/hipErrorStreamCaptureUnsupported from ProcessGroupNCCL watchdog at decode graph capture → exit -6.
• Shared facts: disable_custom_all_reduce=False, attention_backend=aiter; signature since Jul 17; aiter unchanged; Never-passed in window.
• Top hypothesis: [MEDIUM] aiter custom AR launches helper kernels on internal stream during HIP capture; watchdog polls captured event. Disconfirming: #23581 describes hipErrorStreamCaptureInvalidated, not this exact variant; open since Apr but signature only from Jul 17.
• In-flight fix: ⚠️ #23581 (default SGLANG_USE_AITER_AR=false).
• Suggested triage: re-run with --disable-custom-all-reduce; then --disable-cuda-graph to confirm capture-specific.

Cluster: DSR1-MXFP4 MTP GSM8K acc < 0.944 (rows 21-22)

• Affected: nightly-8-gpu-mi35x-deepseek-r1-mxfp4-tp4-rocm720 ×2; MTP step ~0.85 while non-MTP step passes 0.945 same job.
• Shared facts: accept-len ~2.8 (drafting healthy); threshold pinned to non-MTP baseline; Never-passed; aiter constant across steps.
• Top hypothesis: [MEDIUM] EAGLE verify/accept not output-equivalent on MXFP4. Disconfirming: no green MTP baseline; part could be threshold miscalibration.
• In-flight fix: ⚠️ #31214 (EAGLE verify silently greedy on HIP) candidate.
• Suggested triage: greedy-vs-EAGLE A/B at temp=0, diff GSM8K outputs; test #31214.

Cluster: GPU mem fault DSV4-Pro fp4 MTP (row 23)

• Affected: nightly-8-gpu-mi35x-deepseek-v4-pro-mtp-rocm720; Memory access fault mid-prefill → exit -6; sister _tbo_mtp.py passes.
• Top hypothesis: [LOW] illegal mem access fp4-MoE+EAGLE. Disconfirming: Reason: Unknown, symptom drifts daily (GPU-Hang at load vs fault at inference) → possible runner instability.
• In-flight fix: ⚠️ #30407 (AMD EAGLE-verify GPU fault) unconfirmed.
• Suggested triage: coredump + AMD_LOG_LEVEL=4; A/B --speculative-algorithm none.

Cluster: 1200s watchdog hang DSA EAGLE-MTP (row 24) — 🆕 new today

• Affected: nightly-accuracy-8-gpu-mi35x-deepseek-v32-mtp-rocm720; all 8 ranks hard watchdog (1200s) → SIGQUIT; py-spy parked at .item() in dsa_backend.py:743 draft-extend. Passed Jul 17.
• Top hypothesis: [LOW] nondeterministic GPU stall in DSA+EAGLE-MTP. Disconfirming: no HIP error/dump; green Jul 17 → likely flaky.
• In-flight fix: ⚠️ #29876 (EAGLE+DSA gfx950) candidate.
• Suggested triage: re-run to test determinism; if reproducible, rocgdb coredump on stuck scheduler.

Cluster: DSV4-Flash FP4 GSM8K acc < 0.91 (row 25) — 🆕 new today

• Affected: nightly-8-gpu-mi35x-deepseek-v4-flash-rocm720; fp4 0.90978 vs 0.91 (same run also produced 0.921); fp8/fp8_tbo pass. Passed Jul 17 (0.919).
• Top hypothesis: [MEDIUM] borderline threshold vs variance. Disconfirming: prior runs 0.919–0.920 slightly above line → small real regression possible.
• In-flight fix: ❌ none (#31617 adjacent fp8 dp8ep8).
• Suggested triage: re-run 3–5× to classify flaky vs regression.

Cluster: No backend type cpu — Qwen3-VL DP vision (row 26) — 🆕 new today

• Affected: nightly-4-gpu-rocm720; RuntimeError: No backend type associated with device type cpu in run_dp_sharded_mrope_vision_model → RCCL device_group all_gather; server -9 → 3600s timeout. Qwen3-VL-32B never green (random model draw masks it).
• Top hypothesis: [HIGH] CPU-resident embedding passed to device group; is_shm_available() False on AMD host (no AMX/arm64) → broken CPU branch. Disconfirming: exact commit not pinned; NPU hit same error May 30 → latent.
• In-flight fix: ✅ #31663 (primary) · #26759 · #31596 — chase reviewers.
• Suggested triage: cherry-pick #31663, re-run with model pinned to Qwen3-VL-32B; pin/parametrize model in test_encoder_dp.py.

Suggested triage order

Rank	Triage step	Targets	Why this first
1	Verify #31663 resolves crash; re-run test_encoder_dp.py pinned to Qwen3-VL-32B	row 26 (L)	HIGH-confidence root cause with an existing in-flight fix — cheapest, highest-value
2	Widen memory_pool_host.py:41 guard to _is_cuda or _is_hip (or skip ROCm), re-run kernel suite	rows 16-18 (A)	HIGH-confidence one-line class fix; clears 3 jobs / 36 cases
3	Verify #23581 (SGLANG_USE_AITER_AR=false) then --disable-custom-all-reduce repro	rows 19-20 (F)	MEDIUM code-level hypothesis + candidate PR; unblocks Qwen3-235B
4	Repro DSV4 topk_v2 JIT on gfx950; check #31131 follow-up for cooperative_groups.h	rows 10-12 (G)	HIGH-confidence JIT header issue; prior fix ineffective
5	Establish NVIDIA reference for DSV3.2/Qwen3-30B/DSR1-MXFP4-MTP to split threshold-vs-real-gap	rows 1-6, 21-22 (C,B,E)	Largest accuracy/fault clusters (8 jobs); one reference run informs all

---

Job	Failed Steps	Started
nightly-test-1-gpu-kernel-rocm720	Nightly Kernel Benchmark ROCm 7.2 (1-GPU)	2026-07-18T18:38
nightly-accuracy-2-gpu-vlm-rocm720	Nightly Accuracy Test ROCm 7.2 (2-GPU VLM MMMU)	2026-07-18T20:15
nightly-accuracy-2-gpu-rocm720	N/A	2026-07-18T17:45
nightly-2-gpu-mi35x-glm51-mxfp4-rocm720	N/A	2026-07-18T17:45
nightly-accuracy-8-gpu-mi35x-deepseek-v32-mtp-rocm720	Accuracy Test MI35x ROCm 7.2 (8-GPU DeepSeek-V3.2 TP+MTP)	2026-07-18T17:45
nightly-8-gpu-mi35x-deepseek-v4-pro-mtp-rocm720	Accuracy + MTP Accept Length Test MI35x ROCm 7.2 (8-GPU DeepSeek-V4-Pro FP8 MTP, triton)	2026-07-18T19:11
nightly-8-gpu-mi35x-deepseek-v4-flash-rocm720	Accuracy + Performance Test MI35x ROCm 7.2 (8-GPU DeepSeek-V4-Flash FP8 + FP4, triton)	2026-07-18T19:31
nightly-accuracy-8-gpu-mi35x-rocm720	Accuracy Test MI35x ROCm 7.2 (8-GPU GPT-OSS)	2026-07-18T21:44
nightly-8-gpu-mi35x-deepseek-r1-mxfp4-tp4-rocm720	Accuracy Test MI35x ROCm 7.2 (DeepSeek-R1-MXFP4 TP4 MTP)	2026-07-18T21:09
nightly-accuracy-8-gpu-mi35x-deepseek-v32-rocm720	Accuracy Test MI35x ROCm 7.2 (8-GPU DeepSeek-V3.2)	2026-07-18T21:28
nightly-8-gpu-mi35x-qwen3-235b-mxfp4-rocm720	Accuracy Test + Performance Test MI35x ROCm 7.2 (8-GPU Qwen3-235B-MXFP4)	2026-07-18T22:09
nightly-accuracy-8-gpu-mi35x-rocm720	Accuracy Test MI35x ROCm 7.2 (8-GPU GPT-OSS)	2026-07-19T08:20
nightly-test-1-gpu-kernel-rocm720	Nightly Kernel Benchmark ROCm 7.2 (1-GPU)	2026-07-19T08:30
nightly-2-gpu-mi35x-glm51-mxfp4-rocm720	N/A	2026-07-19T08:20
nightly-8-gpu-mi35x-qwen3-235b-mxfp4-rocm720	Accuracy Test + Performance Test MI35x ROCm 7.2 (8-GPU Qwen3-235B-MXFP4)	2026-07-19T08:22
nightly-8-gpu-mi35x-deepseek-r1-mxfp4-tp4-rocm720	Accuracy Test MI35x ROCm 7.2 (DeepSeek-R1-MXFP4 TP4 MTP)	2026-07-19T08:20
nightly-accuracy-8-gpu-mi35x-deepseek-v32-rocm720	Accuracy Test MI35x ROCm 7.2 (8-GPU DeepSeek-V3.2)	2026-07-19T08:36
nightly-accuracy-2-gpu-vlm-rocm720	Nightly Accuracy Test ROCm 7.2 (2-GPU VLM MMMU)	2026-07-19T11:07
nightly-accuracy-2-gpu-rocm720	N/A	2026-07-19T11:27
nightly-test-1-gpu-kernel-rocm720	Nightly Kernel Benchmark ROCm 7.2 (1-GPU)	2026-07-19T18:02
nightly-accuracy-8-gpu-mi35x-rocm720	Accuracy Test MI35x ROCm 7.2 (8-GPU GPT-OSS)	2026-07-19T17:55
nightly-2-gpu-mi35x-glm51-mxfp4-rocm720	N/A	2026-07-19T17:45
nightly-4-gpu-rocm720	Nightly Test ROCm 7.2 (4-GPU)	2026-07-19T18:06
nightly-accuracy-8-gpu-mi35x-deepseek-v32-rocm720	Accuracy Test MI35x ROCm 7.2 (8-GPU DeepSeek-V3.2)	2026-07-19T19:08
nightly-accuracy-2-gpu-vlm-rocm720	Nightly Accuracy Test ROCm 7.2 (2-GPU VLM MMMU)	2026-07-19T19:54
nightly-accuracy-2-gpu-rocm720	N/A	2026-07-19T19:58

Per-Job Analysis

nightly-test-1-gpu-kernel-rocm720 — failed step(s): Nightly Kernel Benchmark ROCm 7.2 (1-GPU)

Commit Info

• sglang: b3a0185c
• aiter: 9127c94a (v0.1.17.dev110+g9127c94a1, source: dev image preserved — AITER_COMMIT_OVERRIDE empty, Dockerfile default matched). Not relevant to this failure.

Failed Tests

All 12 failures are the same Python NameError raised in python/sglang/srt/mem_cache/memory_pool_host.py:394. test_mamba_kernel_empty_indices (4 params) passes because it returns early on empty indices before the kernel call.

Test File	Test Function	Error	Log
test/registered/jit/test_transfer_mamba.py	test_mamba_kernel_backup_load_roundtrip (page_first/page_first_direct × dtype0/dtype1)	NameError: name 'transfer_kv_mamba_lf_pf' is not defined	Log
test/registered/jit/test_transfer_mamba.py	test_mamba_kernel_single_item (4 params)	NameError: name 'transfer_kv_mamba_lf_pf' is not defined	Log
test/registered/jit/test_transfer_mamba.py	test_mamba_kernel_full_indices (4 params)	NameError: name 'transfer_kv_mamba_lf_pf' is not defined	Log

Failure Summary

The nightly-test-1-gpu-kernel-rocm720 job ran 21 test files; 20 passed and only test/registered/jit/test_transfer_mamba.py failed (exit code 1), which failed the job with exit code 255. 12 of its 16 test cases raise NameError: name 'transfer_kv_mamba_lf_pf' is not defined inside MambaPoolHost._copy_tensor_all_layers_lf_pf when exercising the io_backend="kernel" transfer path. The 4 empty_indices cases pass because they return before the kernel call.

Regression Status

Never-passed on AMD ROCm — the test file test/registered/jit/test_transfer_mamba.py was added by the same commit #31227 (7d0fd510, merged 2026-07-16) that introduced the bug, and has failed on the ROCm kernel job in every run since it was introduced. The kernel job itself went green→red on 2026-07-16.

Recent history of test/registered/jit/test_transfer_mamba.py (test_mamba_kernel_*) in job nightly-test-1-gpu-kernel-rocm720 (comparable completed scheduled runs, branch=main, event=schedule):

Date	Run	Job	Test File Status	Failed Function	Error
Jul 18	29654521083	kernel-rocm720	❌ Failed	test_mamba_kernel_* (12)	NameError: transfer_kv_mamba_lf_pf
Jul 17	29601455852	kernel-rocm720	❌ Failed	test_mamba_kernel_* (12)	NameError: transfer_kv_mamba_lf_pf
Jul 16	29521474931	kernel-rocm720	❌ Failed (first run w/ #31227)	test_mamba_kernel_* (12)	NameError: transfer_kv_mamba_lf_pf
Jul 15	29438224738	—	⏭️ Run cancelled	—	—
Jul 14	29355338821	kernel-rocm720	✅ Job passed (test did not exist yet)	—	—
Jul 13	29273096587	kernel-rocm720	✅ Job passed (test did not exist yet)	—	—

First observed failure: 2026-07-16 (run 29521474931, the first scheduled run containing #31227). No green run of this test exists on ROCm.

Failure Cluster

NameError: transfer_kv_mamba_lf_pf undefined on ROCm (CUDA-only import guard)

Facts (verified observations)

• Verbatim error: NameError: name 'transfer_kv_mamba_lf_pf' is not defined at ../python/sglang/srt/mem_cache/memory_pool_host.py:394 (log line 4780), called from backup_from_device_all_layer (memory_pool_host.py:464, log line 4767), test entry test_transfer_mamba.py:218 (log lines 4772/4852/4932/5012).
• The import is guarded by CUDA only — memory_pool_host.py:41-45:

  if _is_cuda:
      from sglang.jit_kernel.transfer_mamba import (
          transfer_kv_mamba_lf_pf,
          transfer_kv_mamba_pf_lf,
      )
  

  On this MI30x/gfx942 runner the backend is HIP: _is_hip=True, _is_cuda=False, so neither name is ever bound. (Contrast: the sgl_kernel.kvcacheio imports at line 34 use if _is_cuda or _is_hip:.)
• Both call sites that reference these names are unconditional on backend once io_backend=="kernel": transfer_kv_mamba_lf_pf( at line 394 (backup, lf→pf) and transfer_kv_mamba_pf_lf( at line 351 (load, pf→lf).
• git blame attributes lines 41-45 and 394 to 7d0fd510 (#31227); git show 7d0fd510 shows +if _is_cuda: and +transfer_kv_mamba_lf_pf( as added lines. git log shows this same commit first added test/registered/jit/test_transfer_mamba.py.
• Reproducibility: identical 12-case NameError (count verified = 12) in the kernel-rocm720 job of runs 29521474931, 29601455852, and 29654521083 — every scheduled run since #31227 landed.
• Scope: only test_transfer_mamba.py failed in this job; the other 20 test files passed (20/21 passed).

Hypothesised Causes

• [HIGH] #31227 (7d0fd510) gates the transfer_kv_mamba_lf_pf / transfer_kv_mamba_pf_lf import behind if _is_cuda: (line 41) while calling them for any io_backend=="kernel" on ROCm (lines 394, 351), so the name is unbound on HIP.
  • Supporting: three independent chains converge — (a) code-level: the import guard excludes HIP and the call site is unconditional; (b) error message names the exact missing symbol at the exact guarded import; (c) timing: the test + bug both landed in #31227 on 2026-07-16 and the job has failed every run since. The sibling sgl_kernel.kvcacheio import on the line above deliberately uses _is_cuda or _is_hip, showing the CUDA-only guard here is the anomaly.
  • Disconfirming: (none found). The only nuance is whether ROCm has a working transfer_mamba JIT kernel to import — if not, the fix is to skip/gate the kernel path on ROCm rather than widen the import.
  • How to verify: on an MI30x container, python3 -c "from sglang.srt.utils import is_hip; print(is_hip())" then check whether sglang.jit_kernel.transfer_mamba imports under HIP; run test/registered/jit/test_transfer_mamba.py after widening the guard to if _is_cuda or _is_hip: (or skipping the kernel backend on ROCm).

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for transfer_kv_mamba (0), transfer_mamba+rocm (0), mamba+hicache+hip (2, both unrelated: #16638 server_args, #24651 Qwen3.5 FP8 all-reduce). Needs a new PR or investigation.

Suggested Triage Steps

• Confirm whether sglang.jit_kernel.transfer_mamba builds/imports under ROCm/HIP. If it does, widen the guard at memory_pool_host.py:41 from if _is_cuda: to if _is_cuda or _is_hip: (matching the kvcacheio import directly above) and re-run test/registered/jit/test_transfer_mamba.py on an MI30x runner.
• If the mamba JIT kernel is CUDA-only, gate the io_backend=="kernel" mamba path (lines 351, 394) so ROCm falls back to the non-kernel copy path, and mark/skip the kernel-backend cases on ROCm in test_transfer_mamba.py.
• Ping the #31227 author (7d0fd510, Mick) since the test and bug were introduced together and the ROCm path was evidently untested pre-merge.

Status

• Persistence: Failing every completed scheduled run since 2026-07-16 (3 consecutive runs: Jul 16/17/18); test has never passed on ROCm.
• Scope: 1 job (nightly-test-1-gpu-kernel-rocm720), 1 test file, 12 test cases. NVIDIA path unaffected (import bound under _is_cuda).
• Blocked work: Blocks HiCache mamba D2H/H2D kernel-backend transfer CI signal on AMD ROCm; keeps the ROCm kernel nightly red.
• In-flight fix: ❌ none found — needs new PR or investigation.

nightly-accuracy-2-gpu-vlm-rocm720 — failed step(s): Nightly Accuracy Test ROCm 7.2 (2-GPU VLM MMMU)

Commit Info

• sglang: b3a0185c
• aiter: 9127c94a (v0.1.17.dev110+g9127c94a1, source: dev/patched image preserved — [CI-AITER-CHECK] Dev/patched version detected → skipping rebuild)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi30x/test_vlms_mmmu_eval_amd.py	test_mmmu_vlm_models (model=OpenGVLab/InternVL2_5-2B)	AttributeError: 'CLIPImageProcessor' object has no attribute 'tokenizer' → server exit code 1 → Exception: Server process exited with code 1 (unittest ERROR)	Log

Masked accuracy sub-failures (recorded as ❌ FAILED in-loop but their AssertionError was never raised — see Failure Summary):

Model (in test_mmmu_vlm_models)	Score	Threshold	Status
Qwen/Qwen2.5-VL-7B-Instruct	0.280	0.33	❌ below threshold
Qwen/Qwen3-VL-30B-A3B-Instruct	0.260	0.29	❌ below threshold
Qwen/Qwen2-VL-7B-Instruct	0.300	0.30	✅ passed (borderline)

Failure Summary

The job's exit code 255 was caused by a single unittest ERROR: launching a server for OpenGVLab/InternVL2_5-2B crashed at startup with AttributeError: 'CLIPImageProcessor' object has no attribute 'tokenizer'. In test_mmmu_vlm_models, popen_launch_server (line 216) sits outside the per-model try/except (which begins at line 225), so this exception propagated out of the entire test, aborting the model loop before the 7 models after InternVL ran and before _check_results could raise its accuracy AssertionError. Consequently the two accuracy shortfalls (Qwen2.5-VL, Qwen3-VL) are real but were masked by the earlier crash.

Regression Status

New regression (InternVL CLIPImageProcessor crash). The accuracy shortfalls are a separate, recurring issue.

Recent history of test_vlms_mmmu_eval_amd.py (test_mmmu_vlm_models) in job nightly-accuracy-2-gpu-vlm-rocm720, tracked by the InternVL CLIPImageProcessor crash signature (most recent comparable completed event=schedule runs on main):

Date	Run	Test File Status	InternVL signature	Notes
Jul 18	29654521083	❌ Failed	CLIP AttributeError (errors=1)	this run
Jul 17	29601455852	❌ Failed	CLIP AttributeError (errors=1)
Jul 16	29521474931	❌ Failed	CLIP AttributeError (errors=1)
Jul 15	29438224738	⚪ cancelled	—	run cancelled, excluded
Jul 14	29355338821	❌ Failed	CLIP AttributeError (errors=1)	first observed
Jul 13	29273096587	❌ Failed	unknown	log expired (BlobNotFound)
Jul 5	28749501468	✅ InternVL PASSED	none — FAILED (failures=1) was accuracy AssertionError on other models	last known good for InternVL
Jul 4	28714556066	✅ InternVL PASSED	none — accuracy AssertionError

• InternVL CLIPImageProcessor crash — first observed Jul 14 2026, last passing Jul 5 2026. Regression window: 48ba79c11e..cb47a68717.
• The job itself has been red every completed run in the window, but for two different reasons: Jul 4/5 = accuracy AssertionError (InternVL fine); Jul 14+ = InternVL server crash that now short-circuits the loop.

Failure Cluster

AttributeError: CLIPImageProcessor has no 'tokenizer' during InternVL server startup

Facts (verified observations)

• Verbatim crash (job log): AttributeError: 'CLIPImageProcessor' object has no attribute 'tokenizer', stack tokenizer_manager.py:3211 _get_processor_wrapper → processor.py:271 get_processor → common.py:521 get_tokenizer_from_processor → return processor.tokenizer.
• common.py:516-521 get_tokenizer_from_processor returns processor.tokenizer for any non-PreTrainedTokenizerBase; the adjacent comment (common.py:515) explicitly notes "Some models don't have an available processor, e.g.: InternVL", yet no guard exists.
• InternVL registers no customized processor (no register_customized_processor in python/sglang/srt/models/internvl.py), so processor.py:202 misses _CUSTOMIZED_MM_PROCESSOR, and processor.py:193 only special-cases "InternVL3_5" — not InternVL2_5. InternVL2_5 therefore falls to AutoProcessor.from_pretrained (processor.py:211), which returns a bare CLIPImageProcessor.
• Test structure: popen_launch_server is at test_vlms_mmmu_eval_amd.py:216, outside the try that starts at line 225; the crash escapes the loop, so models 5-11 (MiniCPM, deepseek-vl2, Janus, Kimi, MiMo, DeepSeek-OCR) were not tested this run.
• Behavioral regression confirmed from logs: InternVL2_5-2B Status: ✅ PASSED on Jul 4 & Jul 5; CLIPImageProcessor AttributeError on Jul 14, 16, 17, 18.
• Not aiter-related: this is a Python/transformers processor-loading path; aiter image was unchanged (AITER_COMMIT_OVERRIDE empty, dev image preserved).

Hypothesised Causes (with confidence)

• [FACT] The crash mechanism is get_tokenizer_from_processor calling processor.tokenizer on a CLIPImageProcessor returned by AutoProcessor.from_pretrained for InternVL2_5-2B (verified in log stack trace + source).
• [LOW] A transformers-version bump in the rebuilt ROCm nightly image (rocm/sgl-dev:v0.5.15.post1-rocm720-mi30x-20260718) changed InternVL2_5's AutoProcessor resolution to return a bare CLIPImageProcessor.
  • Supporting: the image is rebuilt daily; the regression aligns with the Jul 5→14 window; the sglang loading logic (processor.tokenizer) is unconditional and unchanged, so an env change is a plausible trigger.
  • Disconfirming: not directly verified — no transformers version diff extracted (image internals not in log).
  • How to verify: compare pip show transformers between the Jul 5 and Jul 14 images; A/B launch OpenGVLab/InternVL2_5-2B under each.
• [LOW] An sglang change to processor.py InternVL handling in 48ba79c11e..cb47a68717 narrowed InternVL support to InternVL3_5 only, dropping InternVL2_5's prior path.
  • Supporting: processor.py:193 special-cases "InternVL3_5" exclusively; InternVL2_5 has no fallback.
  • Disconfirming: local clone is shallow (window enumerates only 1 commit) so the change cannot be confirmed here; the special-case adds InternVL3_5 handling rather than removing InternVL2_5.
  • How to verify: on a full clone, git log 48ba79c11e..cb47a68717 -- python/sglang/srt/utils/hf_transformers/processor.py python/sglang/srt/models/internvl.py and inspect each diff.
• Note: PR #31227 (7d0fd510, merged Jul 16) only relocated get_tokenizer_from_processor into common.py (that's why the traceback now points there). It merged after the first failure (Jul 14), so it is not the cause — ruled out.

No hypothesis above LOW confidence for the culprit commit; needs a full-history git bisect across 48ba79c11e..cb47a68717 plus a transformers-version diff of the two images.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for InternVL processor, CLIPImageProcessor, and get_tokenizer_from_processor; no matching open PR addresses this crash. (Search: search/issues?q=repo:sgl-project/sglang+is:pr+is:open+CLIPImageProcessor returned 0 items.)

Suggested Triage Steps

• Reproduce in the current ROCm image: launch OpenGVLab/InternVL2_5-2B --trust-remote-code and confirm AutoProcessor.from_pretrained returns CLIPImageProcessor; capture pip show transformers.
• Diff transformers versions between the Jul 5 image and Jul 14+ image to test the environment hypothesis.
• On a full clone, bisect 48ba79c11e..cb47a68717 focusing on processor.py / internvl.py / customized_mm_processor_utils.py to test the sglang hypothesis.
• Independent of root cause, two robustness gaps are worth a maintainer's attention: (a) get_tokenizer_from_processor (common.py:521) does not handle the "image-processor-only" case its own comment describes; (b) test_vlms_mmmu_eval_amd.py:216 places popen_launch_server outside the try/except, letting one model's launch crash abort all remaining models and mask accuracy failures.
• Separately track the recurring accuracy shortfalls for Qwen/Qwen2.5-VL-7B-Instruct (0.280 vs 0.33) and Qwen/Qwen3-VL-30B-A3B-Instruct (0.260 vs 0.29) — these persist independent of the InternVL crash.

Status

• Persistence: InternVL CLIPImageProcessor crash failing since Jul 14 2026 (every completed run: Jul 14, 16, 17, 18); Jul 15 cancelled, Jul 13 log expired. Last known good Jul 5 2026.
• Scope: job nightly-accuracy-2-gpu-vlm-rocm720 (ROCm 7.2 nightly). The crash also blocks coverage of 7 downstream VLM models that never run once InternVL aborts the loop. Likely also affects the non-rocm720 AMD nightly VLM job if it shares this test — not verified here.
• Blocked work: full AMD VLM accuracy signal on ROCm 7.2; also hides the Qwen2.5-VL / Qwen3-VL accuracy regressions.
• In-flight fix: ❌ none found — needs new PR or investigation.

nightly-accuracy-2-gpu-rocm720 — failed step(s): N/A

Commit Info

• sglang: b3a0185c
• aiter: 9127c94a — v0.1.17.dev110+g9127c94a1 (source: dev image preserved; [CI-AITER-CHECK] reports "Dev/patched version detected → skipping rebuild", so the dev build already in the CI image ran; AITER_COMMIT_OVERRIDE empty)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi30x/test_gsm8k_eval_amd.py	TestNightlyGsm8KEval.test_gsm8k_all_models (subTest mistralai/Mixtral-8x7B-Instruct-v0.1)	AssertionError: The following models failed: — GSM8K score 0.307 < threshold 0.69	Log (score at log line ~17583)
test/registered/amd/accuracy/mi30x/test_gsm8k_eval_amd.py	TestNightlyGsm8KEval.test_gsm8k_all_models (subTest Qwen/Qwen3-4B)	Per-model print Status: ❌ FAILED — score 0.911, threshold: N/A (model missing from MODEL_SCORE_THRESHOLDS)	Log (log line ~12260)

Failure Summary

The nightly-accuracy-2-gpu-rocm720 job runs the GSM8K accuracy sweep in test_gsm8k_eval_amd.py across several models. Two models are flagged: Mixtral-8x7B-Instruct-v0.1 scored 0.307 against its 0.69 threshold (a real accuracy deficit — the assertion-triggering failure), and Qwen/Qwen3-4B prints ❌ FAILED only because it has no entry in the MODEL_SCORE_THRESHOLDS dict (its 0.911 score is healthy). Models Llama-3.1-8B (0.812), Qwen3-8B (0.915), and Llama-3.1-70B (0.926) all passed.

Note: the job-logs API caps this job's log at ~18:20 UTC (the job ran to 18:38) because the per-GEMM [aiter] ... using torch solution:0 lines flood the log. The final SUMMARY table and AssertionError are beyond the cap, but per-model result lines for all TP1 models + the first Mixtral attempt are captured, and the job conclusion is failure.

Regression Status

Known recurring failure (not a new regression at HEAD). Both signals persist across the queryable history; the job has never passed in the window (all completed scheduled runs are failure; several intervening days are cancelled).

Recent history of test_gsm8k_eval_amd.py (test_gsm8k_all_models) in job nightly-accuracy-2-gpu-rocm720 (most recent comparable completed scheduled runs: branch=main, event=schedule, completed only):

Date	Run	Test File Status	Mixtral-8x7B	Qwen3-4B (threshold N/A)
Jul 18	29654521083	❌ Failed	0.307 ❌ (<0.69)	0.911 ❌ (N/A)
Jul 17	29601455852	❌ Failed	log capped before Mixtral	0.908 ❌ (N/A)
Jul 16	29521474931	❌ Failed	log capped before Mixtral	0.909 ❌ (N/A)
Jul 15	29438224738	⏭️ cancelled	—	—
Jul 14	29355338821	❌ Failed	log capped before Mixtral	0.909 ❌ (N/A)
Jul 13	29273096587	❌ Failed	0.295 ❌ (<0.69)	0.913 ❌ (N/A)

First observed: Qwen3-4B threshold-N/A appears in every run captured (≥ Jul 13); Mixtral ≈0.30 accuracy confirmed both Jul 13 (0.295) and Jul 18 (0.307). No passing run for this test file in the window.

Failure Cluster

GSM8K accuracy sweep failure — Mixtral-8x7B below threshold + Qwen3-4B missing-threshold config gap.

Facts (verified observations)

• [CI-AITER-CHECK] markers: Dockerfile expects aiter 9127c94a…; image had v0.1.17.dev110+g9127c94a1; dev build preserved, no rebuild (job log lines 4160-4162).
• Mixtral-8x7B first attempt: Score: 0.307 / [METRIC] gsm8k_score=0.3066971080669711 model=mistralai/Mixtral-8x7B-Instruct-v0.1 (job log ~17583). Historical Jul 13: Score: 0.295 (threshold: 0.69) Status: ❌ FAILED.
• Qwen3-4B: Score: 0.911 (threshold: N/A) → Status: ❌ FAILED (job log 12260-12267).
• Code: MODEL_SCORE_THRESHOLDS in test_gsm8k_eval_amd.py has entries for Qwen3-8B (0.76) and Qwen3-30B (0.86) but no Qwen/Qwen3-4B key. Qwen/Qwen3-4B is pulled in via DEFAULT_MODEL_NAME_FOR_NIGHTLY_EVAL_TP1 at python/sglang/test/test_utils.py:154.
• Code behavior: the per-model print uses passed = threshold and score >= threshold, so a None threshold prints ❌ FAILED. However check_model_scores() classifies a missing threshold as ⚠️ NO THRESHOLD and does not add it to failed_models; therefore only Mixtral (real threshold, score below it) contributes to the final AssertionError.
• Other TP1/TP2 models in the same run/harness scored normally (0.81–0.93), so the eval harness itself is functioning for those models.

Hypothesised Causes (with confidence)

• [FACT] Qwen/Qwen3-4B ❌ FAILED is a test-config gap, not a model problem: the model is in the eval list (test_utils.py:154) but absent from MODEL_SCORE_THRESHOLDS. Score 0.911 is healthy; this row is cosmetic (NO THRESHOLD) and does not fail the assertion.
  • Supporting: threshold N/A in every captured run while score stays ~0.91; code dict lacks the key.
  • Disconfirming: (none — verified in code and 5 runs)
  • How to verify / fix direction: add "Qwen/Qwen3-4B": <thr> to MODEL_SCORE_THRESHOLDS (Qwen3-4B reports ~0.80 GSM8K → ~0.76 at −5%).
• [LOW] Mixtral-8x7B ~0.30 accuracy is a genuine correctness/accuracy problem for Mixtral MoE on ROCm 7.2, present ≥ Jul 13 (not tied to today's HEAD).
  • Supporting: 0.295 (Jul 13) and 0.307 (Jul 18), both ≈40% of the 0.69 threshold and far below the 0.744 reported score; other models unaffected → model-specific.
  • Disconfirming: cannot see the final post-retry score or per-request outputs (log capped); root component (MoE kernel vs quantization vs tokenizer add_bos_token restore vs eval parsing) not isolated.
  • How to verify: re-run only Mixtral on MI30x ROCm 7.2, dump a few /tmp/gsm8k_mistralai_Mixtral-8x7B-Instruct-v0.1.json responses, and diff answer strings vs extracted numbers.
• [LOW] The Mixtral low score is (partly) an eval answer-extraction artifact rather than model quality.
  • Supporting: open PR #29933 changes simple_eval_gsm8k.py to extract the answer from the first #### marker instead of the last number — verbose Mixtral outputs could be mis-scored by the current extractor.
  • Disconfirming: all other models score correctly with the same extractor, so a pure parsing bug would have to be specific to Mixtral's output format; the deficit is very large (0.30 vs 0.74).
  • How to verify: apply #29933 on a branch and re-run the Mixtral subTest.

In-flight Fix Check

• Mixtral accuracy: candidate — In-flight fix: ⚠️ [#29933](https://github.com/sgl-project/sglang/pull/29933) (open since 2026-07-02, edits python/sglang/test/simple_eval_gsm8k.py) — plausibly related but unconfirmed; verify against Mixtral before assuming it resolves the deficit. (searched Mixtral+accuracy, gsm8k+threshold+amd.)
• Qwen3-4B threshold: In-flight fix: ❌ none found — needs a one-line addition of Qwen/Qwen3-4B to MODEL_SCORE_THRESHOLDS. (searched Qwen3-4B+threshold.)

Suggested Triage Steps

• Add "Qwen/Qwen3-4B" to MODEL_SCORE_THRESHOLDS in test_gsm8k_eval_amd.py (or remove Qwen3-4B from the TP1 eval list if it shouldn't be gated) to clear the persistent cosmetic ❌ FAILED row.
• Isolate the Mixtral deficit: re-run only the mistralai/Mixtral-8x7B-Instruct-v0.1 subTest on MI30x ROCm 7.2, then inspect /tmp/gsm8k_mistralai_Mixtral-8x7B-Instruct-v0.1.json — compare raw responses vs extracted answers to separate a real MoE/quantization accuracy bug from an eval-parsing artifact.
• If parsing is implicated, apply #29933 on a branch and re-run Mixtral; if the score stays ~0.30, treat it as a real ROCm-720 Mixtral MoE correctness issue and bisect the MoE/attention path.
• Consider raising the job-log ceiling by quieting the per-GEMM [aiter] ... not found tuned config spam (or providing a tuned config) so the final SUMMARY/AssertionError is captured in the job-page log.

Status

• Persistence: recurring — Mixtral ≈0.30 confirmed Jul 13 & Jul 18; Qwen3-4B threshold-N/A in every captured run (≥ Jul 13). Job has no passing completed scheduled run in the window.
• Scope: single job nightly-accuracy-2-gpu-rocm720 (test_gsm8k_eval_amd.py); Qwen3-4B threshold gap would also surface in any AMD job pulling DEFAULT_MODEL_NAME_FOR_NIGHTLY_EVAL_TP1.
• Blocked work: AMD ROCm 7.2 nightly GSM8K accuracy signal is red every run, masking any new accuracy regression behind two known failures.
• In-flight fix: Mixtral → ⚠️ candidate #29933 (unconfirmed); Qwen3-4B → ❌ none.

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-07-19 22:01 UTC)

[amd-bot]

nightly-test-amd-rocm720.yml — Per-Job Analysis (continued 2/6)

nightly-2-gpu-mi35x-glm51-mxfp4-rocm720 — failed step(s): N/A

Commit Info

• sglang: b3a0185c
• aiter: unavailable for this run (job log blob expired / not in run-logs zip). For context, the first run of this job (2026-07-16) used v0.1.17.dev110+g9127c94a1 (dev image preserved, AITER_COMMIT_OVERRIDE empty, Dockerfile default 9127c94a). aiter is not implicated (see below).

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi35x/test_glm51_mxfp4_tp2_gsm8k_mi35x.py	TestGLM51MXFP4TP2GSM8KMI35x.setUpClass	Exception: Server process exited with code -9 (server crash during launch)	Job

Failure Summary

The single test in this job, test_gsm8k, never runs because setUpClass fails: popen_launch_server cannot bring up the amd/GLM-5.1-MXFP4 TP=2 server (test_glm51_mxfp4_tp2_gsm8k_mi35x.py:52). The server process exits with code -9. This job's own log blob has expired (HTTP 404 BlobNotFound) and is absent from the run-logs zip, so direct current-run evidence is unavailable; the failure mode below is taken from the first run of this identical job (run 29521474931, 2026-07-16), whose log is still retrievable and whose culprit code is unchanged at the current commit.

Regression Status

Never-passed (new job that has failed every scheduled run since introduction).

Both the test file and its workflow job were added on 2026-07-16 in #31227 (7d0fd510); only 3 comparable scheduled runs exist.

Recent history of test_glm51_mxfp4_tp2_gsm8k_mi35x.py (setUpClass/test_gsm8k) in job nightly-2-gpu-mi35x-glm51-mxfp4-rocm720 (all comparable event=schedule, branch=main, completed runs since the job was introduced):

Date	Run	Job	Test File Status	Failed Function	Error
Jul 18	run	nightly-2-gpu-mi35x-glm51-mxfp4-rocm720	❌ Failed	setUpClass	server crash (log expired; ran ~24 min)
Jul 17	run	nightly-2-gpu-mi35x-glm51-mxfp4-rocm720	❌ Failed	setUpClass	server crash (log expired; ran ~26 min)
Jul 16	run	nightly-2-gpu-mi35x-glm51-mxfp4-rocm720	❌ Failed	setUpClass	JIT compile crash → server exit -9 (log available; ran ~75 min)

No passing run exists in the queryable window (job is 3 days old). First observed failure: 2026-07-16; last known passing: never.

Failure Cluster

DSA v4 topk_v2 JIT kernel fails to compile on gfx950/ROCm (server crash during decode CUDA-graph capture)

Facts (verified observations, no interpretation)

• [FACT] In the 2026-07-16 run, the scheduler crashed during decode CUDA-graph capture with Exception: Capture cuda graph failed: ninja exited with status 1, stack: scheduler.py:428 __init__ → init_model_worker → init_all_cuda_graphs → tp_worker.py:361 → model_runner.py:748 init_cuda_graphs → cuda_graph_setup.py:332 capture_decode_graph → decode_cuda_graph_runner.py:405.
• [FACT] The underlying compiler error: python/sglang/jit_kernel/include/sgl_kernel/deepseek_v4/topk_impl.cuh:27:10: fatal error: 'cooperative_groups.h' file not found while building the JIT kernel sgl_kernel_jit_dpsk_v4_topk_v2_...__arch_9.5 (gfx950) with hipcc/ROCm 7.2. cooperative_groups.h is a CUDA-only header; HIP exposes <hip/hip_cooperative_groups.h> instead.
• [FACT] After the scheduler died, popen_launch_server reported Server process exited with code -9 and setUpClass errored at test_glm51_mxfp4_tp2_gsm8k_mi35x.py:52; Ran 0 tests.
• [FACT] The test config forces DSA backends and uses the default sgl-kernel topk backend: --dsa-prefill-backend tilelang --dsa-decode-backend tilelang, attention_backend='dsa', dsa_topk_backend='sgl-kernel' (from the server_args dump). This is the code path that triggers the dpsk_v4 topk_v2 JIT build.
• [FACT] topk_impl.cuh (+842 lines) and test_glm51_mxfp4_tp2_gsm8k_mi35x.py (+116 lines) were both added in the same commit 7d0fd510 (#31227), verified via git show --stat.
• [FACT] The unguarded #include <cooperative_groups.h> is still present at topk_impl.cuh:27 at the current commit b3a0185c (verified with git show b3a0185c:...).
• [FACT] All 3 scheduled runs of this job since introduction failed in setUpClass; the job never ran test_gsm8k.
• Reproducibility caveat: the current (Jul 18) and Jul 17 job logs are expired, so I have direct compile-error evidence for only the Jul 16 run. The Jul 18/17 runs finished in ~24-26 min vs ~75 min on Jul 16.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [HIGH] The new DSA-v4 topk_v2 JIT kernel is not ROCm-portable — topk_impl.cuh:27 unconditionally includes the CUDA-only <cooperative_groups.h>, which hipcc cannot find on gfx950. The new GLM-5.1-MXFP4 test forces the DSA sgl-kernel topk path, so the kernel is JIT-built during decode CUDA-graph capture and fails, crashing the server. Both the kernel and the test landed together in #31227, which is exactly why the job has never passed.
  • Supporting: verbatim compiler error + full crash stack in the Jul 16 log; test's forced --dsa-*/default dsa_topk_backend='sgl-kernel' config; both files added in the same commit; include still unguarded at HEAD.
  • Disconfirming: I could not confirm the identical compile error in the Jul 18 (current) or Jul 17 runs because their logs are expired; the much shorter Jul 18/17 durations (~24 min vs ~75 min) are unexplained by direct evidence. (A plausible benign explanation: the model was HF-cached on later runs, so the server reached the same CUDA-graph-capture compile failure sooner — but this is not verified.)
• [LOW] The Jul 18/17 failures are a different early failure (e.g. server OOM before capture, or an infra/VRAM issue), given the ~24-min runtime.
  • Supporting: shorter runtime than Jul 16; step 6 shows API conclusion: None on Jul 18/17 vs failure on Jul 16.
  • Disconfirming: same test, same host class (linux-mi35x-gpu-2), same forced-DSA config, same never-passed pattern; the topk_v2 compile blocker is deterministic and unchanged in code, so it must still be hit on the DSA path.
  • How to verify: re-run the job (logs currently expired) and read the fresh server log, OR reproduce locally (below).

aiter is not implicated: the failure is a compile error inside sglang's own jit_kernel tree, unrelated to aiter kernels; aiter version was unchanged (Dockerfile-default dev build) in the one run with a readable log.

In-flight Fix Check

Searched open PRs for cooperative_groups / topk_v2 / dpsk_v4 (7 results) — none addresses the ROCm/HIP missing-header compile issue. Closest are #31123 (harden top-k v1/v2 against negative seq_lens), #31115 (opt-in CuTe-DSL DSA topk backend), and #25575 (cluster contract) — all functional, not ROCm-portability fixes.

• In-flight fix: ❌ none found — needs new PR or investigation (a ROCm build guard for topk_impl.cuh, or gating this test's DSA sgl-kernel topk path off on gfx950).

Suggested Triage Steps

• Reproduce on an MI35x/gfx950 ROCm 7.2 container: launch sglang serve --model-path amd/GLM-5.1-MXFP4 --tp 2 --dsa-prefill-backend tilelang --dsa-decode-backend tilelang --kv-cache-dtype fp8_e4m3 --disable-radix-cache and confirm the dpsk_v4 topk_v2 JIT build fails with the cooperative_groups.h error during decode CUDA-graph capture.
• Inspect whether the DSA topk_v2 sgl-kernel path is meant to be enabled on ROCm at all; if the header should be HIP-portable, the fix direction is to guard the include (<hip/hip_cooperative_groups.h> under __HIP_PLATFORM_AMD__/USE_ROCM) in topk_impl.cuh:27 — note fused_add_rmsnorm.cuh:12 and qknorm_across_heads.cuh:12 have the same unguarded include and would hit the same wall if JIT-built on ROCm.
• If the DSA sgl-kernel topk kernel is not intended for gfx950, the alternative direction is to select a ROCm-supported --dsa-topk-backend in the test (or skip/quarantine this test on AMD until the kernel is ported), since the test was introduced already broken.
• Re-run the job to capture a fresh (non-expired) server log and confirm the Jul 18/17 failures share the Jul 16 root cause vs. an earlier OOM/infra failure.

Status

• Persistence: Failing every scheduled run since the job was introduced 2026-07-16 (3/3 runs; ~3 days). Never passed.
• Scope: This job nightly-2-gpu-mi35x-glm51-mxfp4-rocm720 (MI35x/gfx950, ROCm 7.2). Root-cause kernel (deepseek_v4/topk_v2) is shared, so any other AMD job forcing the DSA sgl-kernel topk path on gfx950 would be exposed to the same compile failure.
• Blocked work: No GLM-5.1-MXFP4 TP=2 GSM8K accuracy signal on MI35x/ROCm 7.2 (the accuracy gate this test was created to guard never executes).
• In-flight fix: ❌ none found in open PRs.

nightly-accuracy-8-gpu-mi35x-deepseek-v32-mtp-rocm720 — failed step(s): Accuracy Test MI35x ROCm 7.2 (8-GPU DeepSeek-V3.2 TP+MTP)

Commit Info

• sglang: b3a0185c
• aiter: 9127c94a — v0.1.17.dev110+g9127c94a1 (source: dev image preserved; Dev/patched version detected → skipping rebuild, matches gfx950 Dockerfile default; AITER_COMMIT_OVERRIDE empty)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi35x/test_deepseek_v32_mtp_eval_mi35x.py	TestDeepseekV32TPMTP.test_a_gsm8k	requests.exceptions.ConnectionError: … port=11000 … Connection refused (server dead) → FAILED (errors=1), job exit 255	Log

Failure Summary

The DeepSeek-V3.2 MTP GSM8K accuracy eval ran but the sglang server (TP8, DSA attention backend, EAGLE speculative MTP) died mid-eval. At ~52% progress (103/200) the client got Connection refused on 127.0.0.1:11000. The server exit was triggered by a hard scheduler watchdog timeout (1200 s) at 18:46 UTC across all 8 TP ranks, which raised SIGQUIT and tore down the server. The client-side KeyError: 'answer' / ConnectionError are downstream symptoms of the dead server, not the cause.

Regression Status

New occurrence of this hang signature (last green run one day earlier). The job is intermittent, but the two prior failures had a different signature.

Recent history of test_deepseek_v32_mtp_eval_mi35x.py (test_a_gsm8k) in job nightly-accuracy-8-gpu-mi35x-deepseek-v32-mtp-rocm720 (comparable completed event=schedule, branch main):

Date	Run	Test File Status	Signature
Jul 18	29654521083	❌ Failed	1200 s hard watchdog → SIGQUIT hang (this run)
Jul 17	29601455852	✅ Passed	—
Jul 16	29521474931	⚪ Cancelled	whole run cancelled (excluded)
Jul 15	29438224738	⚪ Cancelled	whole run cancelled (excluded)
Jul 14	29355338821	❌ Failed	different — no watchdog-1200/SIGQUIT/DSA in log
Jul 13	29273096587	❌ Failed	different — no watchdog-1200/SIGQUIT/DSA in log
Jul 05	28749501468	✅ Passed	—
Jul 04	28714556066	✅ Passed	—

Last known pass for this test: Jul 17 (29601455852). First observed with this watchdog-hang signature: Jul 18 (this run). Jul 15/16 whole runs cancelled → excluded per completed-run rule.

Failure Cluster

Hard watchdog timeout (1200 s) — GPU hang blocking .item() sync in DSA EAGLE-MTP draft-extend

Facts (verified observations)

• [FACT] All 8 TP schedulers hit Scheduler watchdog timeout (self.watchdog_timeout=1200.0, self.soft=False) at 18:46 UTC; SIGQUIT received … It usually means one child failed immediately after (log lines 19059–19681).
• [FACT] Py-spy dump of the stuck scheduler MainThread (TP7, TP3, etc.) shows it blocked in HIP memcpy_and_sync under THPVariable_item → init_forward_metadata (dsa_backend.py:743) → prepare_for_draft_extend (eagle_worker_common.py:172) → _draft_extend_for_decode (eagle_worker_v2.py:900) → forward_batch_generation (eagle_worker_v2.py:1205) → run_batch (scheduler.py:3332) (log ~19586–19617).
• [FACT] dsa_backend.py:743 is max_seqlen_k = int(forward_batch.seq_lens.max().item()) + draft_token_num — a device→host .item() sync. It blocks until all prior GPU work drains, so it is the sync/observation point, not the failing kernel itself.
• [FACT] Server config: attention_backend='dsa', speculative_algorithm='EAGLE', speculative_num_steps=3, speculative_num_draft_tokens=4, tp_size=8, DeepSeek-V3.2 on GPU_ARCH=mi35x (gfx950 / MI355X).
• [FACT] Eval reached 103/200 (~52%) before the server died; earlier soft watchdogs (300 s) at 17:56–18:11 correspond to model weight loading (_load_w13 (moe/fused_moe_triton/layer.py:537) in the dump), which is benign startup noise.
• [FACT] Jul 13 & Jul 14 accuracy-job logs contain zero watchdog_timeout=1200 / SIGQUIT / dsa_backend matches → those failures are a distinct mode from this hang.

Hypothesised Causes (with confidence + disconfirming evidence)

• [LOW] Nondeterministic GPU-side stall in the DSA + EAGLE-MTP decode/draft-extend path on MI355X (gfx950). The .item() is where the CPU parks waiting on a GPU op that never completes.
  • Supporting: identical stack across all 8 ranks all parked on a device sync; AMD-only combination (DSA + EAGLE spec + gfx950); test passed Jul 17 on near-identical code, consistent with an intermittent hang rather than a deterministic break.
  • Disconfirming: no HIP error/fault/assert is printed — a pure hang with no crash dump; can't localize the specific hung kernel from a py-spy sync frame alone.
  • How to verify: re-run the job; if reproducible, capture with SGLANG_... cuda-core / rocgdb coredump on the stuck scheduler PID to find the actually-hung kernel.
• [LOW] A commit in the Jul 17→Jul 18 window changed spec-v2 / draft-extend host-sync behaviour. Window fec6131844..b3a0185ca = 41 commits; spec-relevant candidates: #31468 ("DFlash: remove per-step host syncs so CPU runs a step ahead — spec-v2 overlap"), #31620, #31614, #31601.
  • Supporting: #31468 deliberately removes host syncs in the spec-v2 overlap path — the exact class of change that can turn a benign path into a CPU/GPU desync hang.
  • Disconfirming: #31468 touches dflash_worker_v2.py / eagle_info.py (7 lines) / ngram_info.py, not the eagle_worker_v2.py / eagle_worker_common.py / dsa_backend.py frames in this stack; the failing model uses speculative_algorithm='EAGLE', not DFlash. Given the green Jul 17 run and a syncless hang with no error, a flaky GPU stall is at least as likely as a code regression.
  • How to verify: git bisect / revert #31468 on a branch and re-run this job on MI355X; if it survives multiple runs, promote; otherwise it's flaky.

In-flight Fix Check

• In-flight fix (candidate): ✅ #29876 "[AMD] Fix EAGLE speculative decoding with DSA backend on gfx950 (MI355X)" (open since 2026-07-02) — targets exactly this EAGLE+DSA+gfx950 combination; worth checking whether it addresses this hang. Do NOT open a duplicate.
• Possibly related: #30217 "[fix] Populate CPU mirrors for EAGLE draft extend on gpu_only batches" (open since 2026-07-06) — touches the same seq_lens_cpu-mirror fallback region as dsa_backend.py:743; relevance to a hang is unconfirmed ([LOW]).
• No open PR was found that directly describes a "1200 s watchdog hang in DSA MTP draft-extend".

Suggested Triage Steps

• Re-run nightly-accuracy-8-gpu-mi35x-deepseek-v32-mtp-rocm720 to establish whether the hang is deterministic or flaky (green Jul 17 argues flaky).
• If reproducible, capture a rocgdb/cuda-core coredump on the stuck scheduler PID to identify the actually-hung GPU kernel (the .item() frame only marks the sync point).
• Check whether in-flight #29876 resolves it; ping its reviewers rather than filing a new PR.
• If deterministic, bisect fec6131844..b3a0185ca (41 commits), starting by reverting spec-v2/host-sync change #31468 on a branch and re-running this job.

Status

• Persistence: 1 completed run with this hang signature (Jul 18); the immediately-prior completed run (Jul 17) passed. Job is intermittent overall (also red Jul 13/14 with a different signature).
• Scope: nightly-accuracy-8-gpu-mi35x-deepseek-v32-mtp-rocm720 only; sibling nightly-8-gpu-deepseek-v32-mtp-rocm720 and nightly-perf-... were ✅ in the same Jul 18 run — so this is not a workflow-wide crash.
• Blocked work: DeepSeek-V3.2 MTP accuracy signal on MI355X (ROCm 7.2) for this run.
• In-flight fix: candidate #29876 (EAGLE+DSA gfx950); no confirmed dedicated fix for this hang.

nightly-8-gpu-mi35x-deepseek-v4-pro-mtp-rocm720 — failed step(s): Accuracy + MTP Accept Length Test MI35x ROCm 7.2 (8-GPU DeepSeek-V4-Pro FP8 MTP, triton)

Commit Info

• sglang: b3a0185c
• aiter: 9127c94a — v0.1.17.dev110+g9127c94a1 (source: Dockerfile default; dev image detected, rebuild skipped per [CI-AITER-CHECK])

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/test_deepseek_v4_pro_fp4_mtp.py	TestDeepseekV4ProFp4MTP.test_a_gsm8k	Memory access fault by GPU node-7/2/6 … Reason: Unknown → Fatal Python error: Aborted → Subprocess scheduler_0 crashed with exit code -6; retry then ConnectionRefusedError [Errno 111]	Log
test/registered/amd/test_deepseek_v4_pro_fp4_mtp.py	TestDeepseekV4ProFp4MTP.test_b_bs_1_speed	Clean attempt: AssertionError: 2.792207… not greater than 2.8 (spec accept-length). Crash attempt: ConnectionRefusedError [Errno 111] (server dead)	Log

Note: test/registered/amd/test_deepseek_v4_pro_fp4_tbo_mtp.py passed (accuracy 0.933). Final: Test Summary: 1/2 passed, job exit code 255.

Failure Summary

The job runs two test files; test_deepseek_v4_pro_fp4_mtp.py failed and ..._tbo_mtp.py passed. During test_a_gsm8k, while the server was under heavy concurrent prefill (#running-req ~250, #queue-req ~517), all GPUs hit a Memory access fault … Reason: Unknown at 21:13:54, aborting scheduler_0 (SIGABRT, exit -6) and tearing down the whole TP group. The subsequent ConnectionRefusedError on both test methods is a downstream symptom of the crashed server, not an independent failure. On a separate clean pass earlier in the same job, test_b_bs_1_speed also failed a marginal speculative accept-length assertion (2.792 vs threshold 2.8).

Regression Status

Known recurring failure — test_deepseek_v4_pro_fp4_mtp.py has failed on this MI35x/ROCm 7.2 job every completed run since at least Jul 13, with the symptom varying day to day (GPU-side crash during weight load, GPU memory fault during inference, or accept-length threshold miss). The specific "Memory access fault during prefill" symptom is new today.

Recent history of test_deepseek_v4_pro_fp4_mtp.py in job nightly-8-gpu-mi35x-deepseek-v4-pro-mtp-rocm720 (most recent comparable completed scheduled runs; event=schedule, branch=main):

Date	Run	Test File Status	Failed Function	Error
Jul 18	29654521083	❌ Failed	test_a_gsm8k (+test_b)	GPU Memory access fault during prefill → scheduler crash exit -6
Jul 17	29601455852	❌ Failed	test_b_bs_1_speed	2.51 / 2.75 not greater than 2.8 (accept-length); note first pass of the day was 2/2 green
Jul 16	29521474931	⚪ Cancelled	—	run cancelled (not counted)
Jul 15	29438224738	⚪ Cancelled	—	run cancelled (not counted)
Jul 14	29355338821	❌ Failed	weight load (_load_w13)	HW Exception by GPU node-3 … reason: GPU Hang → exit 134
Jul 13	29273096587	❌ Failed	weight load (_load_w13/default_weight_loader)	HW Exception by GPU node-3 … reason: GPU Hang → exit 134

First observed failure for this file in window: ≥ Jul 13. Last fully-green run of this file: Jul 17 first pass (a second pass the same day failed). No stable green run in the queryable window.

Failure Cluster

GPU memory access fault / HW hang on DeepSeek-V4-Pro fp4-MoE + EAGLE MTP (MI35x / gfx950, ROCm 7.2)

Facts (verified observations)

• Server [CI Monitor] Daily Report - 2026-03-21 #3 for the failing pass came up healthy: The server is fired up and ready to roll! at 21:11:40; GET /health_generate 200 OK at 21:12:01; test_a_gsm8k began immediately after.
• Last activity before the fault (21:13:53–54) was repeated Prefill batch … #running-req: ~250, #queue-req: ~517, #pending-token: ~671k, cuda graph: False; many POST /generate 200 OK were still succeeding.
• Fault at 21:13:54.74: Memory access fault by GPU node-7 … node-2 … node-6 … on address 0x…24c000. Reason: Unknown. (three GPUs report near-identical faulting addresses ending …24c000).
• Fatal Python error: Aborted (×3 ranks); Subprocess scheduler_0 (pid=17660) crashed with exit code -6 at 21:13:56; followed by TCPStore Connection reset by peer / Broken pipe across ranks 1,2,3,6,7 (secondary teardown noise).
• After the crash the class relaunched the server (server_args [CI Monitor] Daily Report - 2026-03-22 #4 at 21:15:26); the test's retry()/send_one calls in the interim got [Errno 111] Connection refused on port 11000 → FAILED (errors=2) at 21:15:07.
• test_b_bs_1_speed threshold: AVG_SPEC_ACCEPT_LENGTH_THRESHOLD=2.8; measured acc_length=2.79, speed=110.22 on the clean pass — a marginal miss.
• Weight loading itself succeeded this run (Load weight end … elapsed 237–273 s), unlike Jul 13/14 where the GPU hang occurred during MoE _load_w13.
• Config in play: DeepSeek-V4-Pro, tp_size=8, attention-backend dsv4, speculative_algorithm EAGLE (num_steps=3, topk=1, draft_tokens=4), kv_cache_dtype=fp8_e4m3, fp4 MoE, disable-radix-cache, disable-shared-experts-fusion, on gfx950 / ROCm 7.2.

Hypothesised Causes (with confidence)

• [LOW] GPU illegal memory access in the DeepSeek-V4-Pro fp4-MoE + EAGLE MTP forward path under heavy concurrent prefill on gfx950/ROCm 7.2.
  • Supporting: fault occurs mid-inference (not at load) during large batched prefill; three GPUs fault at near-identical addresses suggesting a shared indexing/offset bug in a collective or MoE/attention kernel; the platform combination (fp4 MoE + fp8 KV + EAGLE spec + dsv4 backend) is AMD-specific and this file has been red for days on this runner only.
  • Disconfirming: Reason: Unknown gives no kernel-level attribution; the fault symptom differs from Jul 13/14 (GPU Hang during weight load) — so this is not a single stable code path, which is more consistent with hardware/runtime instability than one deterministic bug. No stack trace pinpoints the faulting kernel.
  • How to verify: reproduce on MI35x with SGLANG_.../HIP AMD_LOG_LEVEL=4 + HSA_ENABLE_SDMA=0 and a coredump; bisect by disabling EAGLE (--speculative-algorithm none) and/or switching MoE runner backend to isolate whether the fault is in spec-decode, fp4 MoE, or attention.
• [LOW] test_b_bs_1_speed accept-length threshold (2.8) is too tight / spec acceptance is genuinely regressing for this model on AMD.
  • Supporting: measured 2.79/2.75/2.51 across Jul 17–18, all just under 2.8; this is an independent failure from the GPU crash.
  • Disconfirming: values vary run-to-run (2.51→2.79), consistent with sampling variance near a marginal threshold rather than a hard regression.
  • How to verify: run test_b_bs_1_speed in isolation several times to measure the accept-length distribution vs the 2.8 gate.

This is not an amd-aiter-scout.yml job, so no Failure Origin / baseline A/B section applies. aiter is the Dockerfile-default dev build (unchanged rebuild skipped), so aiter is not implicated by version change here.

In-flight Fix Check

• In-flight fix (candidate, unconfirmed): #30407 "[AMD] [Fix] Skip split-KV EAGLE-verify fast path for MLA to fix triton spec GPU …" (open since 2026-07-07) — targets AMD EAGLE-spec GPU memory faults, which overlaps this failure's symptom domain; relevance to this crash is unverified. Do not assume it fixes the prefill-time fault seen here.
• In-flight fix for the crash root cause: ❌ none clearly matching "DeepSeek-V4-Pro fp4 MTP memory access fault on MI35x" — likely needs new investigation.
• In-flight fix for the accept-length assertion: ❌ none found.

Suggested Triage Steps

• Reproduce test_a_gsm8k on an MI35x/gfx950 ROCm 7.2 container at commit b3a0185c with a GPU coredump + AMD_LOG_LEVEL=4 to capture the faulting kernel (the job-page log only shows Reason: Unknown).
• A/B the spec-decode path: rerun with --speculative-algorithm none and separately with an alternate --moe-runner-backend to localize whether the fault is in EAGLE verify, fp4 MoE, or the dsv4 attention backend.
• Check whether #30407 applies to this path; if so, cherry-pick onto a branch and re-run this job.
• Track test_b_bs_1_speed separately: measure accept-length distribution to decide whether the 2.8 gate needs adjusting for AMD or whether spec acceptance is truly regressing.
• Since the symptom changes daily (GPU Hang at load vs memory fault at inference), also rule out runner/hardware instability by checking whether other MI35x jobs on the same runner (linux-mi35x-gpu-8-crkkg-runner-nrhsk) show GPU faults in the same window.

Status

• Persistence: test_deepseek_v4_pro_fp4_mtp.py failing on this job on every completed scheduled run observed Jul 13, 14, 17, 18 (Jul 15/16 runs cancelled). No stable green run in the window.
• Scope: this job only in the queried set (nightly-8-gpu-mi35x-deepseek-v4-pro-mtp-rocm720); MI35x / gfx950 / ROCm 7.2 specific. Sister file ..._tbo_mtp.py passes, so it is isolated to the non-TBO MTP path/config.
• Symptom drift: GPU Hang during MoE weight load (Jul 13/14) → GPU Memory access fault during prefill (Jul 18) → accept-length threshold miss (Jul 17); all under DeepSeek-V4-Pro fp4-MoE + EAGLE MTP on AMD.
• In-flight fix: candidate #30407 (unconfirmed relevance); no confirmed fix for the root-cause crash.

nightly-8-gpu-mi35x-deepseek-v4-flash-rocm720 — failed step(s): Accuracy + Performance Test MI35x ROCm 7.2 (8-GPU DeepSeek-V4-Flash FP8 + FP4, triton)

Commit Info

• sglang: b3a0185c
• aiter: 9127c94a — v0.1.17.dev110+g9127c94a1 (source: dev image detected in CI container, rebuild skipped)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/test_deepseek_v4_flash_fp4.py	TestDeepseekV4Fp4.test_a_gsm8k	AssertionError: np.float64(0.9097801364670205) not greater than 0.91 → retry() exceed maximum number of retries (exit 255)	Log

test_deepseek_v4_flash_fp8.py and test_deepseek_v4_flash_fp8_tbo.py both PASSED (Test Summary: 2/3 passed).

Failure Summary

The DeepSeek-V4-Flash FP4 GSM8K accuracy test asserted accuracy > 0.91 and measured 0.90978 — a miss of ~0.00022. After exhausting retries the run raised retry() exceed maximum number of retries and the job exited 255. The sibling FP8 and FP8-TBO accuracy tests on the same model/GPU passed (0.914–0.917).

Regression Status

Flaky / borderline accuracy-threshold failure (new occurrence today for this signature).

Recent history of test_deepseek_v4_flash_fp4.py (test_a_gsm8k) in job nightly-8-gpu-mi35x-deepseek-v4-flash-rocm720 (recent comparable event=schedule, branch=main, completed runs):

Date	Run	Test File Status	fp4 GSM8K accuracy	Note
Jul 18	29654521083	❌ Failed	0.910 (fail) / 0.921 (pass) same run	accuracy-threshold miss, this report
Jul 17	29601455852	✅ Passed	0.919	3/3 passed
Jul 16	29521474931	⤫ Cancelled	—	excluded
Jul 15	29438224738	⤫ Cancelled	—	excluded
Jul 14	29355338821	❌ Failed (diff cluster)	n/a (crashed pre-eval)	GPU Hang → exit 134, NOT accuracy
Jul 13	29273096587	❌ Failed (diff cluster)	0.920 (acc passed)	GPU Hang → exit 134 later in suite

First observed for this accuracy-miss signature: Jul 18. Last fp4-accuracy pass: Jul 17 (0.919). The Jul 14/13 failures belong to a separate "GPU Hang / exit 134" cluster, so they are not passing or comparable-failing runs for this accuracy signature.

Failure Cluster

DeepSeek-V4-Flash FP4 GSM8K accuracy just below 0.91 threshold (borderline accuracy miss).

Facts (verified observations)

• assertGreater(metrics["accuracy"], 0.91) at test/registered/amd/test_deepseek_v4_flash_fp4.py:119; measured accuracy=0.9097801364670205 (verbatim from log).
• Same job run produced fp4 GSM8K Accuracy: 0.921 (passing suite pass) and Accuracy: 0.910×2 (failing suite pass) — run-to-run variance straddles the 0.91 boundary.
• Sibling tests in same job passed: fp8 Accuracy: 0.914–0.917, fp8_tbo Accuracy: 0.915–0.917.
• Final: Ran 2 tests in 536.352s / FAILED (errors=1) → ✗ FAILED: ...fp4.py returned exit code 1 → ##[error]Process completed with exit code 255.
• aiter unchanged vs Dockerfile expectation: [CI-AITER-CHECK] Dev/patched version detected: v0.1.17.dev110+g9127c94a1 → skipping rebuild.
• No GPU Hang / memory-access-fault / kernel error in this run (contrast Jul 14/13 which had HW Exception ... GPU Hang → exit 134).
• 41 commits in window fec6131844..b3a0185c.

Hypothesised Causes (with confidence)

• [MEDIUM] Flaky/borderline accuracy threshold — 0.91 is too tight vs observed GSM8K variance for this config.
  • Supporting: the same fp4 test in the same job produced 0.921 (pass) and 0.910 (fail); miss is only ~0.0002; FP8/FP8-TBO passed, so it's not a broad model breakage.
  • Disconfirming: prior scheduled runs where fp4 reached the accuracy check read 0.919–0.920, slightly above the 0.91 line — variance is small, so a small real regression could also explain a consistent ~0.910.
  • How to verify: re-run the fp4 job 3–5× on MI35x (gfx950); if accuracy oscillates across the 0.91 line without code change, it is flakiness.
• [LOW] A code change in the 41-commit window subtly shifted FP4 numerics (e.g. MoE/FP4/attention paths).
  • Supporting: window includes FP4/MoE/attention-adjacent commits, e.g. faf68940 (SM120 DSV4 flashinfer_mxfp4 moe), ec6a3163 (FP4 KV cache/SM120), 359009fa (MoE routed scaling factor).
  • Disconfirming: most listed commits are SM120/NVIDIA-scoped, not gfx950/HIP; and same-run variance (0.921 vs 0.910) argues against a deterministic code regression.
  • How to verify: bisect fec6131844..b3a0185c on MI35x, running fp4 GSM8K several times per candidate to see the accuracy mean, not a single sample.

In-flight Fix Check

• In-flight fix: ⚠️ partial — #31617 "[AMD] Fix DSV4-Flash dp8ep8 GSM8K regression: row-major Triton FP8 scale on gfx95" targets the same model + GSM8K metric + gfx95 arch, but the fp8 dp8ep8 path, not this fp4 tp8 test — relevance is [LOW], likely not a direct fix.
• In-flight fix (exact fp4 threshold): ❌ none found — no open PR adjusting the fp4 GSM8K threshold or a gfx950 fp4-accuracy fix (searched deepseek v4 fp4 accuracy, fp4 gsm8k).

Suggested Triage Steps

• Re-run job nightly-8-gpu-mi35x-deepseek-v4-flash-rocm720 3–5× to confirm whether fp4 GSM8K oscillates across 0.91 (flaky) vs sits consistently at ~0.910 (real regression).
• If flaky: consider whether the 0.91 gate has adequate margin over measured variance (0.910–0.921) for this fp4 config — a threshold review, decided by maintainers, not the bot.
• If consistently ~0.910: bisect fec6131844..b3a0185c, prioritizing FP4/MoE/attention commits, running fp4 GSM8K multiple times per step.
• Track the separate "GPU Hang → exit 134" cluster (Jul 13/14) independently — it is a different failure mode on this same job.

Status

• Persistence: New for this accuracy-miss signature on Jul 18; prior scheduled run (Jul 17) passed fp4 at 0.919.
• Scope: 1 test (test_a_gsm8k) in 1 job; fp8/fp8_tbo unaffected; NVIDIA not involved (AMD gfx950/rocm720 nightly only).
• Blocked work: DeepSeek-V4-Flash FP4 accuracy signal on MI35x ROCm 7.2.
• In-flight fix: no direct fix; #31617 is adjacent (fp8 dp8ep8) with LOW relevance.

nightly-accuracy-8-gpu-mi35x-rocm720 — failed step(s): Accuracy Test MI35x ROCm 7.2 (8-GPU GPT-OSS)

Commit Info

• sglang: b3a0185c
• aiter: 9127c94a — v0.1.17.dev110+g9127c94a1 (source: Dockerfile default; dev/patched image detected and preserved, no AITER_COMMIT_OVERRIDE)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi35x/test_qwen3_moe_eval_mi35x.py	test_qwen3_moe_accuracy (TestQwen3MoeEvalMI35x)	AssertionError: Failed models: ['Qwen/Qwen3-30B-A3B'] (server crashed: Memory access fault by GPU → Fatal Python error: Aborted)	Log

The other 3 files in the nightly-amd-8-gpu-mi35x suite passed: test_gpt_oss_eval_mi35x.py, test_gpt_oss_w4a8_mxfp4_eval_mi35x.py, test_qwen3_coder_next_eval_mi35x.py (Test Summary: 3/4 passed).

Failure Summary

The unquantized (bf16) Qwen/Qwen3-30B-A3B accuracy test (tp=8, GSM8K, threshold=0.75) is the only failure. The server process crashed during the GSM8K benchmark with a HIP GPU memory access fault (Memory access fault by GPU node-2 … on address 0x7d56a6009000. Reason: Unknown.), which aborted the scheduler process (Fatal Python error: Aborted, exit code 134/255). Because the eval never completed, the model was recorded via the exception path (❌ ERROR, accuracy=None) and pushed into failed, raising the AssertionError at test_qwen3_moe_eval_mi35x.py:226. No accuracy= line was ever printed for this model — this is a crash, not a numeric accuracy miss.

Regression Status

Known recurring failure / Never-passed in the queryable window. No green run for test_qwen3_moe_eval_mi35x.py was found in the last 15 comparable completed scheduled runs; every completed run that reached this file failed on Qwen/Qwen3-30B-A3B with the same GPU memory-access-fault family. Lookback covered runs back to 2026-07-03; runs 2026-07-06 → 2026-07-12 were all cancelled (not comparable).

Recent history of test_qwen3_moe_eval_mi35x.py (test_qwen3_moe_accuracy) in job nightly-accuracy-8-gpu-mi35x-rocm720 (completed event=schedule, branch=main):

Date	Run	Job	Test File Status	Failed Function	Error
Jul 18	29654521083	job	❌ Failed	test_qwen3_moe_accuracy	Memory access fault → Aborted; Failed models: ['Qwen/Qwen3-30B-A3B']
Jul 17	29601455852	job	❌ Failed	test_qwen3_moe_accuracy	Memory access fault → Aborted; same model
Jul 16	29521474931	job	⏭️ Cancelled	—	job cancelled (not comparable)
Jul 14	29355338821	job	❌ Failed	(suite aborted early)	HW Exception … GPU Hang → Aborted, exit 134 (same GPU-fault family)
Jul 13	29273096587	job	❌ Failed	(suite aborted early)	HW Exception … GPU Hang → Aborted, exit 134 (same family)
Jul 5	28749501468	job	❌ Failed	test_qwen3_moe_accuracy	Memory access fault → Aborted; Failed models: ['Qwen/Qwen3-30B-A3B']

First observed failure for this test file: ≥ 2026-07-05. Last known passing date: none found in window (Never-passed in queryable history). On Jul 13/14 the memory fault crashed the whole run_suite process early (exit 134) before it could emit the per-file summary, so the file's individual status can't be isolated there, but the GPU-fault symptom is identical.

Failure Cluster

GPU memory access fault → server abort during Qwen3-30B-A3B bf16 GSM8K eval (MI35x / ROCm 7.2 / tp=8)

Facts (verified observations)

• Verbatim crash (job 88106366685): Memory access fault by GPU node-2 (Agent handle: 0x57925d183cd0) on address 0x7d56a6009000. Reason: Unknown. (and node-9), followed by GPU core dump failed and Fatal Python error: Aborted.
• Crash stack (rank scheduler thread): scheduler.py:1483 run_event_loop → scheduler.py:1593 event_loop_overlap → scheduler.py:1545 pop_and_process → scheduler.py:3558 process_batch_result → batch_result_processor.py:189 process_batch_result_prefill → torch/cuda/streams.py:231 synchronize. The fault surfaces at a CUDA/HIP stream sync in prefill result processing.
• Immediately preceding log lines are aiter bf16 GEMM fallbacks: [aiter] shape is M:789, N:768/2048/128, K:… not found tuned config … using torch solution:0. (Proximity only — the sync that aborts is the next forward-pass op; not proof of aiter causation.)
• Launch config: sglang serve --model-path Qwen/Qwen3-30B-A3B --max-running-requests 128 --mem-fraction-static 0.8 --trust-remote-code --tp 8 --device cuda (server_args resolved mem_fraction_static=0.68, tp_size=8, unquantized bf16).
• The test defines a single model (test_qwen3_moe_eval_mi35x.py:56-59, threshold 0.75); "3/4 passed" is at the run_suite (4-file) level, not within this file.
• Same fault signature reproduced across Jul 5, 17, 18 (clean isolation) and Jul 13, 14 (early whole-suite abort). aiter version unchanged (Dockerfile default) across these runs.

Hypothesised Causes (with confidence)

• [LOW] Persistent HIP GPU memory access fault in the Qwen3-30B-A3B bf16 forward/decode path on MI35x (gfx950) / ROCm 7.2 at tp=8, surfacing at the prefill result CUDA-stream sync.
  • Supporting: identical fault + stack across every completed run for ~2 weeks; isolated to this one model while gpt-oss / qwen3-coder-next on the same runner pass; fault address pattern (…009000) is stable across runs/nodes.
  • Disconfirming: no code-level evidence pins a specific commit; the failure predates the queryable green history entirely (Never-passed), so it may be an environment/kernel-level issue (aiter bf16 GEMM default-config path, ROCm 7.2 driver, or model-config interaction) rather than a recent sglang regression.
  • How to verify: reproduce Qwen/Qwen3-30B-A3B bf16 tp=8 GSM8K on an MI35x/ROCm 7.2 container; capture a GPU coredump to a writable dir (/coredumps was not writable here); bisect by disabling aiter (SGLANG_USE_AITER=0 or equivalent) to confirm/exonerate the aiter bf16 GEMM fallback path.
• [LOW] aiter bf16 GEMM default-config fallback (using torch solution:0) mishandles the M:789 shapes for this model on gfx950.
  • Supporting: the fault prints immediately after these aiter fallback lines.
  • Disconfirming: log-line proximity is not causation — the aborting op is a downstream stream sync; aiter version is the Dockerfile default and unchanged across the whole failing window, so it isn't a new aiter change.
• No hypothesis reaches MEDIUM: this is a long-standing (Never-passed) AMD GPU fault, not a bisectable single-commit regression within the completed-run window.

In-flight Fix Check

• In-flight fix: ⚠️ candidate #31405 "[AMD] Gate Qwen3-30B-A3B TP2 qkv/o_proj CK block-FP8 GEMM shapes to Triton (ROCm 7.0)" — Qwen3-30B-A3B-specific GEMM gating, but targets TP2 / block-FP8 / ROCm 7.0, whereas this failure is unquantized bf16 / TP8 / ROCm 7.2 / MI35x. May be related to the same family of Qwen3-30B-A3B GEMM instability on AMD but does not clearly cover this config. (Searched is:pr is:open Qwen3-30B mi35x and qwen3_moe accuracy.)
• No open PR found that directly targets the MI35x/ROCm 7.2 bf16 tp=8 memory-access-fault. In-flight fix: ❌ none directly matching — needs investigation or a dedicated fix.

Suggested Triage Steps

• Reproduce locally: Qwen/Qwen3-30B-A3B bf16, tp=8, MI35x/ROCm 7.2 container matching image rocm/sgl-dev:v0.5.15.post1-rocm720-mi35x-20260718, run the GSM8K eval; enable a writable GPU coredump dir to capture the faulting kernel.
• A/B the aiter path: rerun with aiter GEMM disabled to confirm/exonerate the using torch solution:0 bf16 fallback for the M:789 shapes.
• Since this test has never passed in queryable history, consider whether it was ever green upstream (check the PR that added test_qwen3_moe_eval_mi35x.py); if it was introduced already-red on MI35x, treat as a model-bring-up gap and quarantine/skip on MI35x-ROCm7.2 until the GPU fault is root-caused, rather than as a fresh regression.
• Ask the AMD kernel owners whether #31405's Triton-gating approach should be extended to the bf16/TP8/ROCm7.2 shapes seen here.

Status

• Persistence: Failing on every completed scheduled run that reached this file since at least 2026-07-05 (Jul 5, 13, 14, 17, 18); no green run in the queryable window. Jul 6–12 runs were cancelled.
• Scope: 1 test file / 1 model (Qwen/Qwen3-30B-A3B) in job nightly-accuracy-8-gpu-mi35x-rocm720; other 3 accuracy files on the same runner pass. Same GPU-fault symptom also crashed the broader suite early on Jul 13/14.
• Blocked work: MI35x/ROCm 7.2 nightly accuracy signal for unquantized Qwen3-30B-A3B (bf16, tp=8).
• In-flight fix: ⚠️ #31405 is a candidate (different config); no directly-matching open PR found.

nightly-8-gpu-mi35x-deepseek-r1-mxfp4-tp4-rocm720 — failed step(s): Accuracy Test MI35x ROCm 7.2 (DeepSeek-R1-MXFP4 TP4 MTP)

Commit Info

• sglang: b3a0185c
• aiter: 9127c94a — v0.1.17.dev110+g9127c94a1 (source: dev image preserved; matches Dockerfile default per [CI-AITER-CHECK])

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi35x/test_deepseek_r1_mxfp4_tp4_mtp_mi35x.py	TestDeepSeekR1MXFP4TP4MTPMI35x.test_gsm8k	AssertionError: 0.8498862774829417 not greater than or equal to 0.944	Log (step 7)

Failure Summary

The MTP (EAGLE speculative decoding) GSM8K accuracy check for DeepSeek-R1-MXFP4 TP=4 on MI35x scored accuracy=0.850 (invalid=0.002, latency=158.6s), below the GSM8K_MTP_ACCURACY_THRESHOLD = 0.944 (test_deepseek_r1_mxfp4_tp4_mtp_mi35x.py:197). The non-MTP sister step in the same job (step 6, test_deepseek_r1_mxfp4_tp4_mi35x.py) passed at accuracy=0.945 on the same model/hardware/aiter. The server ran to completion (1319/1319 GSM8K questions); this is a pure accuracy-threshold failure on the speculative path, not a crash.

Regression Status

Never-passed (no green run found in queryable window). Every completed scheduled run of this job in which the MTP test actually executed failed at ~0.84–0.85; the remaining runs crashed before reaching the test. No passing run for this test file exists in the Jul 3 – Jul 18 window.

Recent history of test_deepseek_r1_mxfp4_tp4_mtp_mi35x.py (test_gsm8k) in job nightly-8-gpu-mi35x-deepseek-r1-mxfp4-tp4-rocm720 (most recent comparable completed scheduled runs, event=schedule, branch=main):

Date	Run	Test File Status	Accuracy	Error
Jul 18	29654521083	❌ Failed	0.850	0.850 < 0.944
Jul 17	29601455852	❌ Failed	0.854	0.854 < 0.944
Jul 16	29521474931	⏭️ Cancelled	—	run cancelled
Jul 14	29355338821	💥 Crashed (exit 134)	—	Fatal Python error: Aborted before test
Jul 13	29273096587	💥 Crashed (exit 134)	—	abort in fused_moe_triton/layer.py before test
Jul 5	28749501468	❌ Failed	0.840	0.840 < 0.944
Jul 4	28714556066	💥 Failed (exit 1, no acc recorded)	—	did not reach MTP accuracy print
Jul 3	28676295311	❌ Failed	0.849	0.849 < 0.944

First observed accuracy failure: at/before Jul 3 2026 (edge of queryable window). Last known passing date: none found. (Local repo is a shallow clone, so the commit that introduced the test/threshold cannot be enumerated locally.)

Failure Cluster

GSM8K MTP/EAGLE accuracy below threshold (DeepSeek-R1-MXFP4 TP4 MI35x)

Facts (verified observations)

• Verbatim: accuracy=0.850 invalid=0.002 latency=158.6s then AssertionError: 0.8498862774829417 not greater than or equal to 0.944 (job log, step 7).
• Same-run non-MTP baseline passed: step 6 accuracy=0.945 invalid=0.001 (job log line 13589) — same model amd/DeepSeek-R1-MXFP4-Preview, TP=4, MI35x, aiter v0.1.17.dev110+g9127c94a1.
• MTP launch config (test_deepseek_r1_mxfp4_tp4_mtp_mi35x.py:145-168): --speculative-algorithm EAGLE --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4, --attention-backend aiter, --kv-cache-dtype fp8_e4m3, GSM8K run at temperature=0.
• Speculative acceptance during the run: accept len ~2.5–2.8 / 4, accept rate ~0.52–0.60 (decode-batch log lines).
• Reproducibility: MTP accuracy landed 0.840 / 0.849 / 0.854 / 0.850 across the four runs (Jul 3, 5, 17, 18) where it executed — tightly clustered ~0.09 below threshold, never at/above 0.944.
• aiter version identical across all failing runs ([CI-AITER-CHECK] markers); the Jul 13/14 failures are a different symptom (server abort, exit 134) that never reached this test.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [MEDIUM] Threshold 0.944 is miscalibrated for the EAGLE/MTP path and/or the EAGLE verify path diverges from the target model on this MXFP4+fp8-KV+aiter config, giving a persistent ~0.09 accuracy gap.
  • Supporting: sister non-MTP step passes at 0.945 on identical model/hardware/aiter, while the MTP path is stable at ~0.85 across 4 independent runs; at temperature=0 a correct EAGLE draft/verify loop is expected to be near-lossless, so a systematic 0.09 drop points to either an unrealistic threshold (copied from the non-MTP 0.945) or a verification-path correctness issue. 0.944 ≈ 0.945 non-MTP accuracy suggests the threshold was set from the non-spec baseline.
  • Disconfirming: no green baseline was ever observed for this test, so we cannot prove the 0.85 is a regression vs the intended behavior; the test may be a never-validated port with an aspirational threshold. Cannot A/B the exact culprit without a controlled greedy-vs-EAGLE run.
  • How to verify: on MI35x, run the same launch recipe once with --speculative-algorithm EAGLE and once without (plain greedy) at temperature=0 and compare GSM8K accuracy; if greedy ≈ 0.945 and EAGLE ≈ 0.85, the verify path is diverging (bug); if both ≈ 0.85 the model/quant config caps accuracy and the threshold is wrong.
• [LOW] aiter/EAGLE kernel numerical issue specific to the speculative decode path.
  • Supporting: only the aiter-backed speculative path is affected; non-spec path is clean.
  • Disconfirming: aiter version is unchanged across passing (non-MTP) and failing (MTP) steps in the same job, so aiter alone does not explain the split.

No hypothesis reaches HIGH; a controlled greedy-vs-EAGLE A/B (above) is required to distinguish "wrong threshold" from "EAGLE verify divergence".

In-flight Fix Check

No open PR was found that clearly targets this specific test/threshold. Candidate related (EAGLE/MTP correctness) PRs, none confirmed to address MI35x DeepSeek-R1-MXFP4 GSM8K accuracy:

• Candidate: #31478 "[Bug] eagle: broadcast finalized verify decision across TP ranks" — TP-rank EAGLE verify correctness; this test is TP=4 + EAGLE, so plausibly relevant. Not verified against this test.
• Candidate: #31221 "[AMD] Derive AITER verify tokens-per-req from input shape" — AITER verify path.
• Candidate: #30407 "[AMD][Fix] Skip split-KV EAGLE-verify fast path for MLA to fix triton spec GPU fault" — MLA + EAGLE verify (targets a GPU fault, not accuracy).
• In-flight fix (exact): ❌ none found — needs investigation to confirm whether any candidate above resolves the accuracy gap.
• (Queries run: GitHub search is:pr is:open with keywords mtp accuracy mi35x, deepseek r1 mxfp4 mtp, EAGLE accuracy rocm.)

Suggested Triage Steps

• Run the greedy-vs-EAGLE A/B described above on MI35x to classify the failure as "threshold miscalibration" vs "EAGLE verify divergence" before any code change.
• If EAGLE output diverges from greedy: cherry-pick/test #31478 (TP-rank verify broadcast) on a branch and re-run this job; check whether accuracy rises toward 0.944.
• If greedy also ≈ 0.85 on this MXFP4 config: the threshold is wrong for the MTP path — recalibrate GSM8K_MTP_ACCURACY_THRESHOLD to the empirically achievable value (or mark the test xfail/quarantine on AMD MI35x) rather than treating it as a code regression.
• Separately track the Jul 13/14 exit 134 server aborts (fused_moe_triton/layer.py) — that is a distinct crash cluster, not this accuracy failure.

Status (factual)

• Persistence: MTP GSM8K accuracy below threshold on every run where the test executed within the queryable window (Jul 3 → Jul 18 2026); no green run ever observed. Accuracy stable at 0.840–0.854.
• Scope: 1 job (nightly-8-gpu-mi35x-deepseek-r1-mxfp4-tp4-rocm720), MTP/EAGLE step only; the non-MTP accuracy step in the same job passes.
• Blocked work: DeepSeek-R1-MXFP4 TP4 MTP accuracy signal on MI35x ROCm 7.2 is red/unusable; non-MTP signal is healthy.
• In-flight fix: ❌ none confirmed; candidates #31478, #31221, #30407 require verification against this test.

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-07-19 22:01 UTC)

[amd-bot]

nightly-test-amd-rocm720.yml — Per-Job Analysis (continued 3/6)

nightly-accuracy-8-gpu-mi35x-deepseek-v32-rocm720 — failed step(s): Accuracy Test MI35x ROCm 7.2 (8-GPU DeepSeek-V3.2)

Commit Info

• sglang: b3a0185c
• aiter: 9127c94a (v0.1.17.dev110+g9127c94a1; source: dev image preserved — [CI-AITER-CHECK] reports "Dev/patched version detected → skipping rebuild", matches Dockerfile default, no override)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi35x/test_deepseek_v32_eval_mi35x.py	TestDeepSeekV32EvalMI35x.test_deepseek_v32_accuracy	AssertionError: Failed models: ['deepseek-ai/DeepSeek-V3.2 (basic)'] — accuracy=0.895 threshold=0.93	Log

Failure Summary

The single GSM8K accuracy test for DeepSeek-V3.2 (basic, TP=8) on MI35x/gfx950 ROCm 7.2 measured accuracy = 0.895 against a threshold = 0.93 and raised AssertionError. The server launched cleanly, all 200/200 GSM8K completion requests returned HTTP 200, and the benchmark ran to completion — this is a numeric accuracy shortfall, not a crash or infra error.

Regression Status

Never-passed (no green run found). The test_deepseek_v32_eval_mi35x.py accuracy check has failed in every completed scheduled run in the lookback window; whenever it reached the measurement stage the accuracy landed in 0.870–0.915, always below 0.93.

Recent history of test_deepseek_v32_eval_mi35x.py (test_deepseek_v32_accuracy) in job nightly-accuracy-8-gpu-mi35x-deepseek-v32-rocm720 (most recent comparable runs: branch=main, event=schedule, completed only):

Date	Run	Test File Status	Measured accuracy	Note
Jul 18	29654521083	❌ Failed	0.895	this run
Jul 17	29601455852	❌ Failed	0.885	below threshold
Jul 16	29521474931	⏭️ Cancelled	—	not completed (excluded)
Jul 14	29355338821	❌ Failed	—	watchdog crash before measurement
Jul 13	29273096587	❌ Failed	—	watchdog crash / monitored_barrier
Jul 5	28749501468	❌ Failed	—	infra: VRAM pre-check error
Jul 4	28714556066	❌ Failed	—	infra: VRAM pre-check error
Jul 3	28676295311	❌ Failed	0.915	below threshold
Jul 2	28611268103	❌ Failed	0.875	below threshold
Jun 29	28393065818	❌ Failed	0.870	below threshold

First observed failure for this test file: at/before Jun 29 (oldest queryable completed run). Last known passing date: none found in the window → Never-passed.

Failure Cluster

DeepSeek-V3.2 GSM8K accuracy below threshold on MI35x (gfx950 ROCm 7.2)

Facts (verified observations, no interpretation)

• Verbatim log: accuracy=0.895 threshold=0.93 ❌ FAIL then AssertionError: Failed models: ['deepseek-ai/DeepSeek-V3.2 (basic)']; step exited exit code 255 → Exit code 255 indicates test failure (not network issue), not retrying.
• Threshold is a hard-coded constant in the test: accuracy_threshold=0.93 (test_deepseek_v32_eval_mi35x.py:71); benchmark is 200 GSM8K questions, 5-shot, temperature=0 (:174, :120-164).
• Server config: --tp 8 --dsa-prefill-backend tilelang --dsa-decode-backend tilelang --mem-fraction-static 0.85, kv_cache_dtype='bfloat16', fp8 per_1x128 MoE (from [aiter] [fused_moe] logs). No server crash in this run; 200/200 requests completed.
• aiter version identical across the window (dev image v0.1.17.dev110+g9127c94a1, no override, no rebuild) per [CI-AITER-CHECK] markers — aiter is not a changing variable here.
• Completed-run accuracy measurements: 0.870, 0.875, 0.915, 0.885, 0.895 — 5 data points, all below 0.93, spread ≈ 0.045.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [MEDIUM] The 0.93 threshold is set higher than the accuracy this configuration actually achieves on MI35x gfx950 ROCm 7.2 (test config issue).
  • Supporting: every completed measurement (5 runs, Jun 29–Jul 18) falls in 0.870–0.915; even the best run (0.915) failed; the test has literally never passed since it entered this queryable window.
  • Disconfirming: "threshold too high" and "a real, persistent backend/kernel accuracy gap" produce the identical symptom — a consistently low mean does not by itself prove the threshold is wrong rather than the numerics being degraded. No known-good baseline was compared.
  • How to verify: compare against the DeepSeek-V3.2 basic GSM8K accuracy on a reference backend (NVIDIA nightly, or the DSA reference). If reference ≈0.93+ while MI35x sits ≈0.89, it is a real accuracy gap; if reference is also ≈0.89, the threshold is simply miscalibrated.
• [LOW] Numerical accuracy gap in the DSA tilelang prefill/decode path or aiter fp8 per_1x128 MoE kernels on gfx950.
  • Supporting: AMD-specific config (--dsa-*-backend tilelang, fp8 MoE via aiter); GSM8K exact-match is sensitive to small logit drift.
  • Disconfirming: outputs are valid (no NaN/INVALID surge, no crash), and the 0.870–0.915 spread looks like sampling noise around a low mean rather than a specific broken kernel; aiter is unchanged across the whole window so no new kernel regression is implicated.
  • How to verify: dump per-question predictions and diff against a reference backend; A/B a non-tilelang DSA backend if available.
• No hypothesis reaches HIGH; the failure is stable and long-standing, so this is a standing accuracy gap / calibration issue rather than a fresh regression.

In-flight Fix Check

• In-flight fix: ❌ none found in open PRs — searched is:pr is:open with DeepSeek-V3.2 mi35x accuracy, deepseek v32 threshold, mi35x deepseek eval (via /search/issues); no open PR targets this accuracy check.
• Note: a remote branch amd_fix_dpskv32_basic_0518 appears in the job's git fetch output (suggesting an in-progress AMD effort on DeepSeek-V3.2 basic), but it has no associated open PR — worth chasing the author rather than opening a duplicate.

Suggested Triage Steps

• Establish a reference: pull the DeepSeek-V3.2 basic GSM8K accuracy from the NVIDIA/reference nightly to decide whether 0.93 is achievable on this config at all.
• If reference confirms the model tops out ~0.89 on MI35x, treat accuracy_threshold=0.93 (test_deepseek_v32_eval_mi35x.py:71) as miscalibrated and recalibrate to the observed distribution (with margin) — a config change, decided by the maintainer.
• If reference is materially higher, bisect the DSA tilelang / aiter fp8 MoE path on gfx950 for a numerical gap; dump per-question predictions vs reference on AMD MI35x.
• Chase the owner of branch amd_fix_dpskv32_basic_0518 to see whether a fix is already in progress before starting new work.

Status (factual, no priority assignment)

• Persistence: Failing in every completed scheduled run in the lookback window (since ≥ Jun 29 2026); never observed passing. Measured accuracy 0.870–0.915 vs 0.93 threshold.
• Scope: 1 job / 1 test file — nightly-accuracy-8-gpu-mi35x-deepseek-v32-rocm720 in nightly-test-amd-rocm720.yml. Sibling MTP accuracy job and perf jobs are separate and not affected by this specific assertion.
• Blocked work: DeepSeek-V3.2 basic accuracy signal on MI35x ROCm 7.2 is permanently red, so genuine future regressions in this path are masked.
• In-flight fix: ❌ none found in open PRs (possible WIP branch amd_fix_dpskv32_basic_0518, no PR).

nightly-8-gpu-mi35x-qwen3-235b-mxfp4-rocm720 — failed step(s): Accuracy Test + Performance Test MI35x ROCm 7.2 (8-GPU Qwen3-235B-MXFP4)

Commit Info

• sglang: b3a0185c
• aiter: 9127c94a — v0.1.17.dev110+g9127c94a1 (source: dev/patched CI image; matches Dockerfile default, no rebuild, [CI-AITER-CHECK] markers)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/test_qwen3_instruct_mxfp4.py	TestQwen3Instruct2507MXFP4.setUpClass	RuntimeError: Rank 0 scheduler died during initialization (exit code: -6) → server crash during decode CUDA-graph capture	Log

Failure Summary

The single test test_qwen3_instruct_mxfp4.py (model amd/Qwen3-235B-A22B-Instruct-2507-mxfp4, tp=4 ep=2, attention-backend=aiter, full decode CUDA graph) never started serving: all 4 ranks aborted (SIGABRT, exit -6) during server initialization while capturing the decode CUDA graph. setUpClass then raised Exception: Server process exited with code 1, giving FAILED (errors=1) and job exit 255. This is a server-launch crash, not an accuracy assertion.

Regression Status

Never-passed (no green run found in queryable window). Lookback: Jun 29 – Jul 18 2026 (event=schedule, branch=main, completed runs only; cancelled runs excluded). The test file has failed in every completed run where the job actually ran; the dominant symptom is the HIP stream-capture watchdog abort, with GPU-Hang and infra (VRAM-cleanup) variants interspersed.

Recent history of test_qwen3_instruct_mxfp4.py (setUpClass) in job nightly-8-gpu-mi35x-qwen3-235b-mxfp4-rocm720:

Date	Run	Test File Status	Symptom
Jul 18	29654521083	❌ Failed	HIP capture watchdog abort, exit -6 (this run)
Jul 17	29601455852	❌ Failed	HIP capture watchdog abort
Jul 16	29521474931	⏭️ Cancelled	(excluded — job did not run to completion)
Jul 14	29355338821	❌ Failed	GPU Hang, exit 134
Jul 13	29273096587	❌ Failed	GPU Hang, exit 134
Jul 5	28749501468	❌ Failed	infra: VRAM cleanup unsuccessful (test not reached)
Jul 4	28714556066	❌ Failed	exit 1
Jul 3	28676295311	❌ Failed	exit 1
Jul 2	28611268103	❌ Failed	infra: VRAM cleanup unsuccessful (test not reached)
Jun 29	28393065818	❌ Failed	HIP capture watchdog abort

Failure Cluster

RCCL/NCCL watchdog HIP-capture abort during decode CUDA-graph capture (operation not permitted on an event last recorded in a capturing stream).

Facts (verified observations)

• All 4 ranks print [E718 ...] ProcessGroupNCCL.cpp:2057 ... Process group watchdog thread terminated with exception: HIP error: operation not permitted on an event last recorded in a capturing stream (hipErrorCapturedEvent) and/or operation not permitted when stream is capturing (hipErrorStreamCaptureUnsupported), followed by terminate called ... c10::DistBackendError and Fatal Python error: Aborted (log).
• Crash stack (main thread) is inside decode CUDA-graph capture (verified from log, lines ~5057–5096):
  scheduler.py:__init__ → init_model_worker → init_all_cuda_graphs → tp_worker.init_cuda_graphs → model_runner.py:838 init_cuda_graphs → cuda_graph_setup.py:106 capture_cuda_graphs → cuda_graph_setup.py:333 capture_decode_graph → decode_cuda_graph_runner.py __init__/capture/run_once → qwen3_moe.py:994 forward → qwen2_moe.py:891 forward → vocab_parallel_embedding.py:574 forward → communication_op.py:20 tensor_model_parallel_all_reduce → parallel_state.py all_reduce/outplace_all_reduce → aiter custom_all_reduce.py:608/589 all_reduce.
• The fatal exception is raised by the NCCL watchdog thread (ProcessGroupNCCL::WorkNCCL::finishedGPUExecutionInternal → isCompleted → Watchdog::runLoop), i.e. a background thread polling HIP event completion concurrently with graph capture on the main thread — not by the aiter all_reduce call itself.
• Config: cuda_graph_config decode backend='full', attention_backend='aiter', disable_custom_all_reduce=False, tp=4 ep=2, mem_fraction_static=0.7735 (server_args).
• aiter version identical (v0.1.17.dev110+g9127c94a1) across the failing runs Jun 29 / Jul 13 / Jul 14 / Jul 17 / Jul 18 (verified via [CI-AITER-CHECK] markers in each log) — the failure is not correlated with an aiter version change.
• No green run exists in the entire queryable window (Jun 29 – Jul 18), so no pass_sha..fail_sha regression window can be computed.

Hypothesised Causes (with confidence)

• [MEDIUM] ROCm limitation: the NCCL/RCCL watchdog thread queries HIP events (finishedGPUExecutionInternal) while the main thread has a stream in HIP-graph capture; on ROCm this raises hipErrorCapturedEvent/hipErrorStreamCaptureUnsupported, terminating the watchdog and aborting the process. Triggered by full decode CUDA-graph capture that contains a collective all-reduce (custom aiter all_reduce) at the vocab-embedding layer.
  • Supporting: fatal frame is ProcessGroupNCCL::Watchdog::runLoop on all ranks; error text is specifically about events "recorded in a capturing stream"; main-thread stack is squarely in capture_decode_graph; reproduces identically on Jun 29 and Jul 17/18 with unchanged aiter.
  • Disconfirming: the intervening Jul 13/14 completed runs aborted with a different symptom (GPU Hang, exit 134) rather than the capture watchdog error, so a single deterministic cause does not explain every failing run — the runner/hardware state may also contribute.
  • How to verify: relaunch the same server command with --disable-cuda-graph (or decode backend='disabled') on an MI35x ROCm 7.2 runner; if it initializes, the capture path is implicated. Then A/B with --disable-custom-all-reduce to isolate the collective-in-capture interaction.
• [LOW] Runner/hardware instability on the mi35x ... rocm720 8-GPU runner (GPU Hang, VRAM-cleanup failures).
  • Supporting: Jul 13/14 = GPU Hang exit 134; Jul 2/5 = VRAM cleanup unsuccessful after 3 attempts before the test even reached serving.
  • Disconfirming: the capture watchdog abort is reproducible across different runs/dates, which points to a software path rather than pure hardware flakiness.
  • How to verify: check runner GPU health / dmesg and whether other jobs on the same runner class are also unstable in the same window.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for capturing+stream+watchdog, hipErrorCapturedEvent (0 results), qwen3+235b+mxfp4, and custom_all_reduce+graph+capture; no open PR targets this HIP-capture watchdog abort or this MXFP4 test. Needs new investigation.

Suggested Triage Steps

• Reproduce with --disable-cuda-graph on an MI35x ROCm 7.2 runner using the logged command (sglang serve --model-path amd/Qwen3-235B-A22B-Instruct-2507-mxfp4 --tp 4 --ep 2 --attention-backend aiter) to confirm the decode-graph capture path is the trigger.
• If capture is confirmed, A/B --disable-custom-all-reduce to determine whether the collective-inside-capture is the specific conflict, then investigate suppressing/pausing the NCCL watchdog event polling during HIP graph capture on ROCm.
• Separately investigate runner health for the GPU Hang (exit 134) and VRAM cleanup unsuccessful variants — these may be an independent infra issue on the MI35x ROCm 7.2 8-GPU runner rather than the same code path.
• Since this test has never passed on this runner in the queryable window, consider quarantining test_qwen3_instruct_mxfp4.py on the MI35x ROCm 7.2 nightly until the capture/runner issues are resolved, so it stops masking other regressions.

Status

• Persistence: Never-passed across the full queryable window (Jun 29 – Jul 18 2026); every completed run where the job ran has failed.
• Scope: nightly-8-gpu-mi35x-qwen3-235b-mxfp4-rocm720 job in nightly-test-amd-rocm720.yml; blocks the Qwen3-235B-MXFP4 8-GPU accuracy/perf signal on MI35x ROCm 7.2.
• In-flight fix: ❌ none found in open PRs.

nightly-accuracy-8-gpu-mi35x-rocm720 — failed step(s): Accuracy Test MI35x ROCm 7.2 (8-GPU GPT-OSS)

Commit Info

• sglang: b3a0185c (model_runner: extract post-memory-pool wiring into _init_post_memory (#31601))
• aiter: 9127c94a — v0.1.17.dev110+g9127c94a1 (source: dev image preserved; AITER_COMMIT_OVERRIDE empty, [CI-AITER-CHECK] reports "Dev/patched version detected → skipping rebuild")

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi35x/test_qwen3_moe_eval_mi35x.py	test_qwen3_moe_accuracy (model Qwen/Qwen3-30B-A3B)	Memory access fault by GPU node-4/5/7/8 … Reason: Unknown → Fatal Python error: Aborted → AssertionError: Failed models: ['Qwen/Qwen3-30B-A3B']	Log

The other 3 files in this job passed: test_gpt_oss_eval_mi35x.py (acc 0.510/0.865), test_gpt_oss_w4a8_mxfp4_eval_mi35x.py (acc 0.830), test_qwen3_coder_next_eval_mi35x.py (acc 0.965/0.970). Job Test Summary: 3/4 passed, process exit code 255.

Failure Summary

During the unquantized bf16 Qwen/Qwen3-30B-A3B MoE GSM8K eval (TP8, attention_backend=aiter, SGLANG_USE_AITER=1), the scheduler process died with a GPU memory access fault on multiple GPU nodes (4/5/7/8), producing Fatal Python error: Aborted. The Python fault surfaces at torch/cuda/streams.py:231 synchronize called from batch_result_processor.py:189 (result.copy_done.synchronize() — the async-error surface point), inside process_batch_result_prefill. The test then reports the model as failed. This is a crash, not an accuracy-below-threshold miss.

Regression Status

Known recurring failure / Never-passed in queryable window. The same test file + function has failed with the identical Memory access fault symptom in every comparable completed scheduled run where the test was actually reached. No passing run for this test exists in the last 15-run window (07-02 → 07-18).

Recent history of test_qwen3_moe_eval_mi35x.py (test_qwen3_moe_accuracy, Qwen/Qwen3-30B-A3B) in job nightly-accuracy-8-gpu-mi35x-rocm720 (comparable completed event=schedule, branch=main runs):

Date	Run	Test File Status	Failed Function	Error
Jul 19	29654521083	❌ Failed	test_qwen3_moe_accuracy	Memory access fault (GPU node-4/5/7/8) → Aborted
Jul 18	29601455852	❌ Failed	test_qwen3_moe_accuracy	Memory access fault (GPU node-3/6/7/8) → Aborted
Jul 16	29521474931	⏭ Job cancelled	—	test not reached (excluded)
Jul 15	29438224738	⏭ Run cancelled	—	excluded
Jul 14	29355338821	⚠ Not reached	—	HW Exception … GPU Hang → exit 134 during gpt-oss (earlier test)
Jul 13	29273096587	⚠ Not reached	—	HW Exception … GPU Hang → exit 134 during gpt-oss (earlier test)
Jul 5	28749501468	❌ Failed	test_qwen3_moe_accuracy	Memory access fault (GPU node-5) → Aborted
Jul 4	28714556066	❌ Failed	test_qwen3_moe_accuracy	Memory access fault (GPU node-3/4/5/6/7/8/9) → Aborted
Jul 3	28676295311	⚠ Not reached	—	job died at setup (562-line log)
Jul 2	28611268103	❌ Failed	test_qwen3_moe_accuracy	Memory access fault (GPU node-2/3/4/9) → Aborted

First observed failure: at least 2026-07-02 (earliest queryable non-cancelled run). Last known passing run for this test: none found in the window (multiple runs also crashed on GPU Hang/faults before reaching it).

Failure Cluster

GPU memory access fault during Qwen3-30B-A3B (bf16 MoE, aiter) GSM8K prefill on MI35x ROCm 7.2

Facts (verified observations)

• Memory access fault by GPU node-8 … on address 0x7b2222009000. Reason: Unknown. (+ nodes 7/5/4), immediately followed by GPU core dump failed and Fatal Python error: Aborted (job log ~08:42:23).
• Faulting thread stack: torch/cuda/streams.py:231 synchronize ← batch_result_processor.py:189 process_batch_result_prefill ← scheduler.py:3558 process_batch_result ← event_loop_overlap. Line 189 in current source is result.copy_done.synchronize() — the async-error surface, not the faulting kernel itself.
• Only the bf16 unquantized Qwen/Qwen3-30B-A3B run (TP8, attention_backend=aiter, moe_runner_backend=auto) crashed; the two GPT-OSS runs and Qwen3-Coder-Next in the same job on the same GPUs passed in this run.
• Reproducible: same test + same fault symptom in ≥5 runs (07-19, 07-18, 07-05, 07-04, 07-02); 07-13/07-14 crashed even earlier with HW Exception … GPU Hang (exit 134).
• aiter version unchanged (dev image v0.1.17.dev110+g9127c94a1) across these runs; sglang HEAD varied.

Hypothesised Causes (with confidence)

• [LOW] Faulty/unstable kernel in the Qwen3 MoE bf16 + aiter attention forward path on gfx950 (MI35x) triggering an out-of-bounds GPU access during prefill.
  • Supporting: fault is isolated to this one model/config; surfaces at the prefill result sync; identical address-pattern faults (0x…009000) recur across weeks; the model runs with aiter attention + bf16 tuned-GEMM fallback (not found tuned config … using torch solution:0 logged repeatedly right before the fault).
  • Disconfirming: the fault is asynchronous — copy_done.synchronize() only reports it, so the log-line proximity to the aiter GEMM messages is not evidence of aiter causation; no stack frame identifies the actual faulting kernel; failure predates the HEAD commit by weeks so no single sglang commit explains it.
  • How to verify: rerun with CUDA_LAUNCH_BLOCKING=1 / HIP_LAUNCH_BLOCKING=1 to localize the faulting kernel; A/B toggle --attention-backend triton vs aiter and --moe-runner-backend triton on Qwen/Qwen3-30B-A3B TP8 on an MI35x ROCm 7.2 box.
• [LOW] MI35x runner hardware/driver instability (GPU Hang / memory faults) on this workflow generally.
  • Supporting: 07-13 and 07-14 runs died with HW Exception … GPU Hang (exit 134) during the earlier gpt-oss test, showing the runner class is prone to GPU-level faults independent of this test.
  • Disconfirming: on 07-19/07-18/07-05/07-04/07-02 the other 3 tests passed on the same GPUs in the same job, so a pure hardware explanation doesn't cover why only this model faults consistently.
• No hypothesis reaches MEDIUM. This is an asynchronous GPU fault with no code-level attribution; needs a launch-blocking repro to localize the kernel.

In-flight Fix Check

• In-flight fix: ❌ none found targeting this specific bf16 Qwen3-30B-A3B / aiter memory-access-fault on MI35x. Nearest open PRs are unrelated paths: #31405 (Qwen3-30B-A3B TP2 CK block-FP8 GEMM, ROCm 7.0 — different quant/GEMM/ROCm), #30407 (EAGLE-verify MLA triton fault), #18263 (AITER KV-cache OOM). Searched search/issues with keywords Qwen3-30B-A3B+amd, mi35x+memory+access+fault, Qwen3+MoE+accuracy+AMD.

Suggested Triage Steps

• Reproduce on an MI35x ROCm 7.2 box with HIP_LAUNCH_BLOCKING=1 running only test_qwen3_moe_eval_mi35x.py to pin the exact faulting kernel (attention vs MoE GEMM).
• A/B the backends: rerun Qwen/Qwen3-30B-A3B TP8 with --attention-backend triton and separately --moe-runner-backend triton; if triton passes, the fault is in the aiter path.
• If aiter-path is implicated, bisect the aiter dev build (9127c94a) against the prior passing aiter for this model; note no passing run exists in the window, so also consider whether this test ever passed on MI35x (possible never-passed AMD port).
• Given no green run in the queryable window, consider quarantining Qwen/Qwen3-30B-A3B in this suite until the GPU fault is root-caused, to restore signal for the other 3 (passing) accuracy tests.

Status

• Persistence: Failing on every run that reached the test since at least 2026-07-02 (07-19, 07-18, 07-05, 07-04, 07-02); no passing run found in the 15-run window. Intervening 07-13/07-14 runs crashed earlier with GPU Hang.
• Scope: 1 test file (test_qwen3_moe_eval_mi35x.py, model Qwen/Qwen3-30B-A3B) in job nightly-accuracy-8-gpu-mi35x-rocm720; the job's other 3 accuracy tests pass. This single failure fails the whole job (exit 255).
• Blocked work: Blocks green signal for the MI35x ROCm 7.2 8-GPU nightly accuracy job.
• In-flight fix: ❌ none found — needs new investigation (launch-blocking repro / backend A/B).

nightly-test-1-gpu-kernel-rocm720 — failed step(s): Nightly Kernel Benchmark ROCm 7.2 (1-GPU)

Commit Info

• sglang: b3a0185cab0411bdfe26a2ae9f3485499e03d6cd
• aiter: 9127c94a18e4398e1eba91f6639e910f0994ad02 (v0.1.17.dev110+g9127c94a1; source: dev image preserved — [CI-AITER-CHECK] Dev/patched version detected → skipping rebuild). Not relevant to this failure.

Failed Tests

All in test/registered/jit/test_transfer_mamba.py; all fail with the identical NameError at python/sglang/srt/mem_cache/memory_pool_host.py:394. 12 failing parametrizations across 3 functions (the 4 test_mamba_kernel_empty_indices cases PASS — they hit the numel()==0 early-return before the undefined call).

Test File	Test Function	Error	Log
test/registered/jit/test_transfer_mamba.py	test_mamba_kernel_backup_load_roundtrip (page_first/page_first_direct × dtype0/dtype1)	NameError: name 'transfer_kv_mamba_lf_pf' is not defined	Log
test/registered/jit/test_transfer_mamba.py	test_mamba_kernel_single_item (× 4 params)	NameError: name 'transfer_kv_mamba_lf_pf' is not defined	Log
test/registered/jit/test_transfer_mamba.py	test_mamba_kernel_full_indices (× 4 params)	NameError: name 'transfer_kv_mamba_lf_pf' is not defined	Log

Job overall: Test Summary: 20/21 passed; test_transfer_mamba.py (exit code 1) → step Process completed with exit code 255.

Failure Summary

The registered JIT kernel suite ran on AMD ROCm 7.2 (mi30x/gfx942). The single test file test/registered/jit/test_transfer_mamba.py failed: every case that exercises io_backend="kernel" raised NameError: name 'transfer_kv_mamba_lf_pf' is not defined inside MambaPoolHost._copy_tensor_all_layers_lf_pf (memory_pool_host.py:394). All 20 other kernel test files passed.

Regression Status

Never-passed (no green run found) on AMD ROCm — the test file did not exist on this backend before 2026-07-16 and has failed on every completed scheduled run since it was added.

Recent history of test/registered/jit/test_transfer_mamba.py in job nightly-test-1-gpu-kernel-rocm720 (comparable completed event=schedule, branch=main runs):

Date	Run	Job	Test File Status	Failed Function	Error
Jul 19	run	kernel-rocm720	❌ Failed	test_mamba_kernel_* (12)	NameError: transfer_kv_mamba_lf_pf
Jul 17	run	kernel-rocm720	❌ Failed	test_mamba_kernel_*	NameError: transfer_kv_mamba_lf_pf
Jul 16	run	kernel-rocm720	❌ Failed	test_mamba_kernel_*	NameError: transfer_kv_mamba_lf_pf
Jul 15	run	—	⏭️ Cancelled	—	(excluded)
Jul 14	run	kernel-rocm720	✅ Passed	—	test absent (20/20 passed)
Jul 13	run	kernel-rocm720	✅ Passed	—	test absent

First observed failure: 2026-07-16. Last run without this failure: 2026-07-14 (the test file did not yet exist in the suite; 20/20 passed).

Failure Cluster

NameError: transfer_kv_mamba_lf_pf undefined on ROCm (CUDA-only import guard)

Facts (verified observations)

• Verbatim error (all 12 cases): E NameError: name 'transfer_kv_mamba_lf_pf' is not defined, raised at ../python/sglang/srt/mem_cache/memory_pool_host.py:394 (job log ~line 4689).
• Call site memory_pool_host.py:394 invokes transfer_kv_mamba_lf_pf(...) inside the if io_backend == "kernel": branch of MambaPoolHost._copy_tensor_all_layers_lf_pf.
• Import guard (memory_pool_host.py:41-45):

  if _is_cuda:
      from sglang.jit_kernel.transfer_mamba import (
          transfer_kv_mamba_lf_pf,
          transfer_kv_mamba_pf_lf,
      )

  Both symbols are imported only under if _is_cuda. On AMD ROCm _is_hip=True and _is_cuda=False (memory_pool_host.py:26-27), so neither name is ever bound → NameError when the kernel path executes. The sibling load path transfer_kv_mamba_pf_lf (line 351) is undefined on ROCm for the same reason.
• The 4 test_mamba_kernel_empty_indices cases pass because _copy_tensor_all_layers_lf_pf returns early at if src_indices.numel() == 0 (line 384-385) before reaching the undefined call.
• The JIT kernel source is python/sglang/jit_kernel/transfer_mamba.py (defines both functions) backed by CUDA header .../jit_kernel/csrc/kvcacheio/transfer_mamba.cuh.
• Both the test file and the kernel path were introduced together in PR #31227 (7d0fd510, 2026-07-16 12:31 UTC), which added transfer_mamba.cuh, transfer_mamba.py, test_transfer_mamba.py, and the if _is_cuda: guarded import + the io_backend=="kernel" call site.
• Timeline: #31227 merged 07-16 12:31 UTC; first failing scheduled run started 07-16 17:51 UTC. The prior run (07-14) had no such test (20/20 passed).
• Pure-Python failure; aiter version is unchanged/irrelevant (dev image preserved).

Hypothesised Causes (with confidence)

• [HIGH] The if _is_cuda: import guard at memory_pool_host.py:41 excludes AMD/HIP, leaving transfer_kv_mamba_lf_pf / transfer_kv_mamba_pf_lf undefined, while the io_backend="kernel" code path (line 394) is reachable on ROCm — introduced by PR #31227 (7d0fd510).
  • Supporting: (a) code-level evidence — the guard and the call are in the same commit and directly produce the exact NameError; (b) temporal — first failure is the first scheduled run after the merge; (c) the test never passed on ROCm (existed only after the merge). All three evidence chains converge.
  • Disconfirming: (none found). NVIDIA is unaffected because _is_cuda is True there, consistent with the AMD-only symptom.
  • How to verify: on an AMD ROCm box, python3 test/registered/jit/test_transfer_mamba.py reproduces the NameError; import-level repro: with _is_hip true, from sglang.srt.mem_cache.memory_pool_host import transfer_kv_mamba_lf_pf fails. Open question for the maintainer: is the transfer_mamba JIT kernel intended to compile on HIP (then the guard should be if _is_cuda or _is_hip:), or is it CUDA-only (then test_transfer_mamba.py should be skipped / not registered on ROCm)?

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for transfer_kv_mamba, transfer_mamba+rocm, transfer_mamba+hip, test_transfer_mamba, MambaPoolHost, memory_pool_host+mamba (via search/issues?q=repo:sgl-project/sglang+is:pr+is:open+<kw>); none address this AMD import-guard NameError. (PR #31180 is an unrelated MambaPoolHost relocation refactor.)

Suggested Triage Steps

• Confirm intent of the transfer_mamba JIT kernel on ROCm. If it is meant to run on AMD, widen the guard at memory_pool_host.py:41 to if _is_cuda or _is_hip: (matching the sgl_kernel.kvcacheio guard on line 31) and re-run the kernel suite on AMD MI30x.
• If the kernel is CUDA-only, gate the io_backend="kernel" mamba path and/or skip test/registered/jit/test_transfer_mamba.py on ROCm so it does not register a hard NameError on AMD.
• Ping the author of PR #31227 (7d0fd510); no duplicate fix PR currently exists.

Status

• Persistence: Failing every completed scheduled run since 2026-07-16 (first failing run 29521474931; also 07-17 and today 07-19). Never passed on AMD ROCm.
• Scope: nightly-test-1-gpu-kernel-rocm720 job of nightly-test-amd-rocm720.yml; 12 parametrizations in one test file. AMD/HIP-specific — NVIDIA CUDA unaffected (symbols imported there).
• Blocked work: Mamba HiCache host-pool kernel transfer (io_backend="kernel") unusable/untested on ROCm; blocks kernel CI signal on AMD ROCm 7.2.
• In-flight fix: ❌ none found — needs new PR or investigation.

nightly-2-gpu-mi35x-glm51-mxfp4-rocm720 — failed step(s): N/A

Commit Info

• sglang: b3a0185c (model_runner: extract post-memory-pool wiring into _init_post_memory (#31601))
• aiter: N/A — the failing job uploaded no log ([CI-AITER-CHECK] markers unavailable; see Failure Summary).

Failed Tests

Test File	Test Function	Error	Log
N/A (no log for this job instance)	N/A	Runner lost mid-step — accuracy step never reached a conclusion	Job

The underlying test this job runs (observed failing in the most recent completed run, Jul 16) is:

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi35x/test_glm51_mxfp4_tp2_gsm8k_mi35x.py	TestGLM51MXFP4TP2GSM8KMI35x.setUpClass	Exception: Server process exited with code -9 (CUDA graph capture failed → JIT ninja build error)	Log

Failure Summary

This specific job instance (88173371049) is an infrastructure/runner-loss failure, not a captured test failure. The job started at 08:20:30Z and was marked failure at 08:43:31Z, but step 6 (Accuracy Test MI35x ROCm 7.2 (2-GPU GLM-5.1-MXFP4 GSM8K)) is frozen at in_progress/None, step 12 (Post Checkout code) is pending, and the job log blob returns BlobNotFound. No per-step log was uploaded for this job even in the run-level log archive. Separately, the underlying test has been genuinely red since ~Jul 8/9 (last green Jul 5) with a JIT kernel compile error on gfx950.

Regression Status

This instance: Infrastructure issue (runner loss). Underlying test: Known recurring failure (red for ~10 days; last green Jul 5, sha 48ba79c1).

Recent history of test_glm51_mxfp4_tp2_gsm8k_mi35x.py (TestGLM51MXFP4TP2GSM8KMI35x.setUpClass) in job nightly-2-gpu-mi35x-glm51-mxfp4-rocm720 (most recent comparable event=schedule, branch main runs):

Date	Run	Test File Status	Signature
Jul 18–19	29654521083	❌ This run — infra	Runner-loss: acc step in_progress, no log (BlobNotFound), attempt 2
Jul 17	29601455852	❌ Failed — infra	Runner-loss: acc step in_progress/None (same signature as this run)
Jul 16	29521474931	❌ Failed — test	JIT compile cooperative_groups.h not found → server exit -9
Jul 15	29438224738	⏹ Cancelled	Job cancelled (excluded)
Jul 14	29355338821	❌ Failed — test	acc step completed/failure (signature not individually re-verified)
Jul 13	29273096587	❌ Failed — test	acc step completed/failure
Jul 13	29202600186	❌ Failed — test	acc step completed/failure
Jul 11	29162179641	❌ Failed — test	acc step completed/failure
Jul 10	29113137016	❌ Failed — test	acc step completed/failure
Jul 09	29039471605	❌ Failed — test	acc step completed/failure
Jul 08	28964405353	❌ Failed	acc step skipped (upstream step failed)
Jul 05	28749501468	✅ Passed	—
Jul 04	28714556066	✅ Passed	—
Jul 03	28676295311	✅ Passed	—

First observed failure: Jul 8 2026. Last known pass: Jul 5 2026 (48ba79c1). The two most recent instances (Jul 17 and this run) are runner-losses, so the JIT test error could not be re-observed in them.

Failure Cluster

"Runner lost during 2-GPU MXFP4 accuracy test (no log uploaded)" — for this instance. Underlying test cluster: "DSV4 topk JIT build fails on gfx950 (cooperative_groups.h not found) → CUDA graph capture → server exit -9".

Facts (verified observations)

• Job 88173371049: conclusion=failure; step 6 Accuracy Test … GLM-5.1-MXFP4 GSM8K = in_progress, conclusion=None, started_at=08:25:46Z, no completed_at; step 12 Post Checkout code = pending; job completed_at=08:43:31Z. Runner linux-mi35x-gpu-2-glhqm-runner-w6tfd; run_attempt=2.
• Job log endpoint returns <Error><Code>BlobNotFound</Code>. The run-level log archive contains logs for 3 other jobs (nightly-8-gpu-mi35x-qwen3-235b-mxfp4-rocm720, nightly-accuracy-8-gpu-mi35x-rocm720, nightly-test-1-gpu-kernel-rocm720) but none for nightly-2-gpu-mi35x-glm51-mxfp4-rocm720 — confirming no log was ever uploaded.
• The parent run 29654521083 is still status=queued (other jobs pending); this job’s failure is final regardless.
• Jul 16 completed-failure log (verbatim): topk_impl.cuh:27:10: fatal error: 'cooperative_groups.h' file not found when compiling for gfx950; → Exception: Capture cuda graph failed: ninja exited with status 1; → Server process exited with code -9; → FAILED (errors=1) in setUpClass.
• In the current tree, python/sglang/jit_kernel/include/sgl_kernel/deepseek_v4/topk_impl.cuh:27 is #include <cooperative_groups.h> and line 33 namespace cg = cooperative_groups; — matching the compile error location exactly.

Hypothesised Causes (with confidence)

• [HIGH] This job instance failed due to a lost/terminated runner, not a test outcome.
  • Supporting: accuracy step never received a conclusion (in_progress/None), downstream step left pending, log blob BlobNotFound, job still marked failure, run_attempt=2. This is the canonical "runner disconnected mid-step" signature. The identical signature also occurred Jul 17.
  • Disconfirming: (none found) — no partial log survives that would indicate an in-test crash instead of infra loss.
  • How to verify: re-run the job; check ROCm/runner host health for linux-mi35x-gpu-2-glhqm-runner-w6tfd around 08:25–08:43Z Jul 19.
• [MEDIUM] Underlying test is broken by a DSV4 topk JIT build failure on gfx950/ROCm 7.2 (cooperative_groups.h unavailable to the HIP/clang JIT include path).
  • Supporting: Jul 16 completed run shows the exact compile error at the topk_impl.cuh:27 include, reproducible in current source; test red since Jul 8 after a Jul 5 green.
  • Disconfirming: #31131 "[AMD] Fix DSV4 JIT build on rocm" merged Jul 15 (a8b60433), yet the Jul 16 run still failed with the same error — so either that fix was insufficient for the cooperative_groups.h/gfx950 case, or a different regression is in play. The Jul 9–14 failures were not individually log-verified to share this signature.
  • How to verify: bisect 48ba79c1.. focusing on python/sglang/jit_kernel/**/deepseek_v4/**; attempt a local JIT build of sgl_kernel_jit_dpsk_v4_topk_v2 in the ROCm 7.2 gfx950 container.

In-flight Fix Check

• Runner-loss (this instance): In-flight fix: ❌ none found — infrastructure/runner issue, not code; re-run the job.
• Underlying JIT build error: In-flight fix: ⚠️ [#31131](https://github.com/sgl-project/sglang/pull/31131) merged Jul 15 but the Jul 16 run still failed — the merged AMD DSV4-JIT fix did not resolve the cooperative_groups.h failure. Open PR #25575 (dsv4 topk_v2 cluster contract) is unrelated (predates regression, different symptom). No open PR targeting cooperative_groups.h/gfx950 include resolution found.
• Searches run: search/issues?q=repo:sgl-project/sglang+is:pr+is:open+cooperative_groups, +glm51+mxfp4, +deepseek_v4+topk.

Suggested Triage Steps

• Re-run this job first — its failure is a runner-loss (no log); a green re-run or a captured test failure will disambiguate infra vs. the underlying test bug.
• If the re-run reproduces the JIT error, bisect 48ba79c1..b3a0185c over python/sglang/jit_kernel/**deepseek_v4**, and check whether #31131 needs a follow-up to make cooperative_groups.h resolvable under the ROCm 7.2 gfx950 JIT include path.
• Investigate runner-host stability for the linux-mi35x-gpu-2 pool — two consecutive nights (Jul 17, Jul 19) of mid-accuracy-test runner loss on this 2-GPU MXFP4 job suggests a possible OOM/GPU-hang/host-eviction pattern during the ~18-min warmup window.

Status

• Persistence: Underlying test red since Jul 8 2026 (last green Jul 5); the two most recent instances (Jul 17, Jul 19) are runner-losses with no uploaded log.
• Scope: nightly-2-gpu-mi35x-glm51-mxfp4-rocm720 in nightly-test-amd-rocm720.yml. The sibling GLM5.1 jobs nightly-8-gpu-glm51-rocm720 and nightly-8-gpu-mi35x-glm51-rocm720 passed on Jul 17, so this is specific to the 2-GPU MXFP4 (TP2) variant.
• Blocked work: GLM-5.1-MXFP4 TP2 GSM8K accuracy signal on MI35x / ROCm 7.2.
• In-flight fix: runner-loss → none (re-run); JIT error → #31131 merged but ineffective, needs follow-up.

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-07-19 22:01 UTC)

[amd-bot]

pr-test-amd.yml — 34 failure(s)

Run started (UTC): 2026-07-18 00:25 UTC
Last scanned (UTC): 2026-07-19 17:01 UTC
Commits: sglang a03ca46

Summary

Counts: 34 failures · 12 clusters · 21 carrying over (≥3 days) · 13 recent regressions/flakes (<3 days, clusters E/F/G/H/K/L)

#	Cluster	Job	Test File	Test Function	Status	Hypothesis (confidence)
1	1-GPU diffusion shard hang (no log)	mm-gen-1gpu (…,0)	test_server_1_gpu.py	N/A (hang, no log)	~8d persistent	[LOW] heavy diffusion case hangs MI325; log lost
2	1-GPU diffusion shard hang (no log)	mm-gen-1gpu (…,1)	test_server_1_gpu.py	N/A (hang, no log)	since 07-09 (~10d)	[LOW] breakable-CUDA-graph reloc 0562ccb1a
3	1-GPU diffusion shard hang (no log)	mm-gen-1gpu (…,1)	test_server_1_gpu.py	test_diffusion_generation (server-ready timeout)	since 07-11+	[MEDIUM] sana/ltx/lingbot server not ready <1200s
4	1-GPU diffusion shard hang (no log)	mm-gen-1gpu (…,0)	N/A (hang, no log)	N/A	since 07-11	[LOW] no log; needs py-spy/artifact capture
5	1-GPU diffusion shard hang (no log)	mm-gen-1gpu (…,1)	N/A (runner lost comm.)	N/A	since 07-10 (~9d)	[MEDIUM] host OOM on MI325 during model load
6	1-GPU diffusion shard hang (no log)	mm-gen-1gpu (…,0)	N/A (frozen step, no log)	N/A	14/14 runs	[MEDIUM] abnormal runner termination
7	1-GPU diffusion shard hang (no log)	mm-gen-1gpu (…,0)	N/A (hang→timeout)	N/A	≥7 runs	[MEDIUM] server hang → 90-min kill
8	1-GPU diffusion shard hang (no log)	mm-gen-1gpu (…,1)	test_server_1_gpu.py	N/A (hang→timeout)	14/14 runs	[MEDIUM] DiffusionServer 120s timeout accumulation
9	MLA prefill head-dim mismatch (HIP eager)	stage-c-mi35x (…,1)	test_deepseek_r1_mxfp4_8gpu.py	setUpClass	since 07-17 (2d)	[HIGH] #31391 24a8944e1 missing mha_companion_layers
10	MLA prefill head-dim mismatch (HIP eager)	stage-c-mi35x (…,1)	test_deepseek_r1_mxfp4_8gpu.py	test_a_gsm8k	since 07-17	[HIGH] #31391 eager-fallback swap skipped
11	MLA prefill head-dim mismatch (HIP eager)	stage-c-mi35x (…,1)	test_deepseek_r1_mxfp4_8gpu.py	setUpClass	since 07-17	[HIGH] #31391
12	MLA prefill head-dim mismatch (HIP eager)	stage-c-mi35x (…,0)	test_kimi_k25_mxfp4_bcg_mi35x.py	test_bcg_gsm8k	since 07-17	[HIGH] #31391
13	MLA prefill head-dim mismatch (HIP eager)	stage-c-mi35x (…,0)	test_kimi_k25_mxfp4_bcg_mi35x.py	test_bcg_gsm8k	since 07-17 (4 runs)	[HIGH] #31391
14	disagg encoder numpy truthiness	mm-gen-2gpu (…,1)	test_disagg_server.py	TestDisaggZImage* (3)	never-passed ~8d	[HIGH] _is_default value==default on ndarray
15	disagg encoder numpy truthiness	mm-gen-2gpu (…,1)	test_disagg_server.py	TestDisaggZImage* (3)	never-passed	[MEDIUM] cross-role transfer stall / _is_default
16	disagg encoder numpy truthiness	mm-gen-2gpu (…,1)	test_disagg_server.py	TestDisaggZImage* (3)	never-passed	[FACT] scheduler_mixin.py:155 ndarray truthiness
17	disagg encoder numpy truthiness	mm-gen-2gpu (…,1)	test_disagg_server.py	TestDisaggZImage* (3)	never-passed 9d	[HIGH] _is_default ndarray
18	update_weights 500 (InferenceMode)	mm-gen-1gpu (…,3)	test_update_weights_from_disk.py	TestUpdateWeightsFromDisk* (5F/4E)	never-passed since 07-16	[HIGH] inplace copy_ on inference tensor
19	update_weights 500 (InferenceMode)	mm-gen-1gpu (…,3)	test_update_weights_from_disk.py	TestUpdateWeightsFromDisk* (5F/4E)	never-passed	[HIGH] weights_updater.py inplace copy
20	update_weights 500 (InferenceMode)	mm-gen-1gpu (…,3)	test_update_weights_from_disk.py	TestUpdateWeightsFromDisk* (5F/4E)	8/8 runs	[HIGH] inplace copy outside InferenceMode
21	update_weights 500 (InferenceMode)	mm-gen-1gpu (…,3)	test_update_weights_from_disk.py	TestUpdateWeightsFromDisk* (5F/4E)	recurring	[HIGH] inplace copy outside InferenceMode
22	FLUX.2 FP8 hipBLAS unsupported	mm-gen-2gpu (…,0)	test_server_2_gpu.py	test_diffusion_generation[flux2_modelopt_fp8_tp2_t2i] + [joy_echo_t2v_2gpu]	never-passed ≥07-16	[HIGH] gfx942 FP8 _scaled_mm NOT_SUPPORTED
23	FLUX.2 FP8 hipBLAS unsupported	mm-gen-2gpu (…,0)	test_server_2_gpu.py	[flux2_modelopt_fp8_tp2_t2i] + [joy_echo_t2v_2gpu]	15/15 runs	[MEDIUM] FLUX.2 FP8 GEMM + JoyAI HF download
24	FLUX.2 FP8 hipBLAS unsupported	mm-gen-2gpu (…,0)	test_server_2_gpu.py	[flux2_modelopt_fp8_tp2_t2i] (inferred, no log)	≥9d never-passed	[MEDIUM] FLUX.2 FP8 + teardown cascade
25	PP disagg gsm8k KeyError(answer)	stage-b-mi35x-disagg	test_disaggregation_pp.py	TestDisaggregationPrefillPPAccuracy.test_gsm8k	flaky ~3/5	[MEDIUM] mooncake KV transfer 30s timeout
26	PP disagg gsm8k KeyError(answer)	stage-b-mi35x-disagg	test_disaggregation_pp.py	…PrefillPPDynamicChunkAccuracy.test_gsm8k	flaky, trending	[MEDIUM] mooncake KVTransferError / faulty NIC
27	PP disagg gsm8k KeyError(answer)	stage-b-mi35x-disagg	test_disaggregation_pp.py	TestDisaggregationDecodePPAccuracy.test_gsm8k	2/4 (recovered next run)	[LOW] dropped request in mooncake PP path
28	Start CI container hang (infra)	mm-gen-2gpu (…,0)	N/A (container start)	N/A	one-off (underlying: disagg #31325)	[LOW] runner provisioning stall
29	Start CI container hang (infra)	stage-c-8gpu-amd (…,0)	N/A (container start)	N/A	1/9 isolated	[MEDIUM] transient container/registry flake
30	remote TransferEngine zeroed weights	stage-b-2gpu-large (…,1)	test_load_weights_from_remote_instance.py	test_load_weights_from_remote_instance	flaky 3/10 since 07-13	[MEDIUM] AMD Mooncake reports success, dst zeroed
31	DSV3.2 GSM8K GPU OOM	stage-c-8gpu-amd (…,1)	test_deepseek_v32_basic.py	TestDeepseekV32TP.test_a_gsm8k	≥7 runs, recovered next	[MEDIUM] HBM 0 MB at parallel=1400
32	Kimi MXFP4 barrier timeout	stage-c-mi35x (…,0)	test_kimi_k25_mxfp4_bcg_mi35x.py	test_bcg_gsm8k	3 runs since 07-17	[HIGH] TP3 load >480s monitored_barrier margin
33	logprob-length assert crash	stage-b-1gpu-small (…,4)	test_srt_endpoint.py	test_logprob_mixed	1 occurrence (flaky)	[MEDIUM] input-logprob race under chunked prefill
34	kv_canary verify_position violation	extra-a-2gpu-large	test_e2e_pd.py	test_pd_transfer_canary_clean	new 07-19	[MEDIUM] #31662 10908a679 ChunkCache release

Cluster: 1-GPU diffusion shard hang (no log) — rows 1-8

• Affected: multimodal-gen-test-1-gpu-amd parts 0 & 1 across all 4 runs (part 2 passes, part 3 fails separately in cluster "update_weights 500").
• Shared facts: step 9 Run diffusion server tests (1-GPU) stuck in_progress, if: always() upload never runs, job-log blob BlobNotFound; 60–91 min then runner killed below the 90-min timeout. One row (row 5) carries GitHub's "self-hosted runner lost communication" annotation.
• Top hypothesis: [MEDIUM] heavy T2V/T2I case hangs/OOMs the MI325 host so the runner dies before flushing logs; disconfirming: zero shard log → hang site unverified, cannot exclude infra.
• In-flight fix: ❌ none. Suggested triage: add per-test timeout + persisted stdout/py-spy/rocm-smi so a log survives the kill; then bisect the diffusion window / pin the wedging case.

Cluster: MLA prefill head-dim mismatch (HIP eager) — rows 9-13

• Affected: MI35x 8-GPU stage-c shards 0/1, DeepSeek-R1-MXFP4 & Kimi-K2.5-MXFP4; all 8 (or 4) TP ranks crash at radix_attention.py:269 .view() (512 vs 128 head-dim) on the tc_piecewise/eager-fallback prefill path.
• Shared facts: clean pass→fail at dc0b3eb6→85ac56c8; 24a8944e1 (#31391) is the sole window commit touching the path; HIP-only, aiter unchanged.
• Top hypothesis: [HIGH] #31391 dropped the attn_mqa→attn_mha companion swap on the eager_runner.py HIP fallback (context.mha_companion_layers is None); disconfirming: no revert A/B run (relies on code read + fix PR).
• In-flight fix: ✅ #31675 (open 07-18, one-line eager_runner.py fix, cites these jobs). Note row 13's per-job analysis reported "none found" for the Kimi variant, but #31675 is the same root cause. Suggested triage: verify/merge #31675; if incomplete, git revert 24a8944e1 on a branch.

Cluster: disagg encoder numpy truthiness — rows 14-17

• Affected: multimodal-gen-test-2-gpu-amd shard 1, all 3 TestDisaggZImage* tests, every run.
• Shared facts: encoder crashes ValueError: truth value of an array … ambiguous at scheduler_mixin.py:244→_is_default (value == field_info.default on ndarray) → 120s DiffusionServer timeout → 500; predates 7d0fd5101 (present 07-11).
• Top hypothesis: [FACT/HIGH] ndarray-unsafe _is_default; disconfirming: none. In-flight fix: ✅ #31325 (open 07-15, mergeable_state: unstable). Suggested triage: stabilize & merge #31325.

Cluster: update_weights 500 (InferenceMode) — rows 18-21

• Affected: multimodal-gen-test-1-gpu-amd shard 3, test_update_weights_from_disk.py, identical 5F/1P/4E every run since 07-16 (new file, never-passed).
• Shared facts: server RuntimeError: Inplace update to inference tensor outside InferenceMode via weights_updater.py→linear.py inplace copy_; plus 4 FLUX No weights dir for transformer fixture-setup errors (separate sub-bug).
• Top hypothesis: [HIGH] weight update runs outside torch.inference_mode(). In-flight fix: ✅ #31263 (open 07-15) — covers the 500s only; FLUX fixture ERRORs need a separate fix. Suggested triage: merge #31263; file follow-up for the FLUX perturbed-dir fixture.

Cluster: FLUX.2 FP8 hipBLAS unsupported — rows 22-24

• Affected: multimodal-gen-test-2-gpu-amd shard 0; [flux2_modelopt_fp8_tp2_t2i] → HIPBLAS_STATUS_NOT_SUPPORTED on gfx942, plus [joy_echo_t2v_2gpu] HF/Xet download error and a port-5555/OOM startup cascade; log often not uploaded on hard crash.
• Top hypothesis: [HIGH] FP8 rowwise _scaled_mm unsupported on MI325. In-flight fix: ✅ #28889 (flux2, open since 06-22, blocked); ❌ none for JoyAI download / teardown cascade. Suggested triage: unblock/merge #28889; pre-cache or xfail JoyAI-Echo; fix inter-case server teardown.

Cluster: PP disagg gsm8k KeyError(answer) — rows 25-27

• Affected: stage-b-...-mi35x-disaggregation-amd, test_disaggregation_pp.py::test_gsm8k (mooncake, pp=2/tp=2). Surface KeyError: 'answer'; rows 25/26 show KVTransferError (30s mooncake timeout), row 27 a dropped/empty request with no KVTransferError.
• Shared facts: intermittent — passes and fails on adjacent/same SHAs, no clean regression window; both servers stay healthy.
• Top hypothesis: [MEDIUM] intermittent mooncake RDMA transfer stall (possible faulty NIC). In-flight fix: ✅ #29127 (isolate faulty rdma5 NIC); candidates #29984, #31144, #30551. Suggested triage: land #29127, correlate failures with NIC.

Cluster: Start CI container hang (infra) — rows 28-29

• Isolated Start CI container stalls (no log), one-off per job; next runs pass. In-flight fix: ❌ none (infra). Triage: re-run; inspect runner Docker/registry health.

Singleton clusters

• row 30 — remote TransferEngine zeroed weights: flaky 3/10; ✅ #30035 (draft) pins AMD test to NCCL — un-draft/merge, then fix core Mooncake write-completion.
• row 31 — DSV3.2 GSM8K GPU OOM: 0 MB HBM at parallel=1400, recovers next run; ❌ none — retest with lower mem-fraction-static/parallel.
• row 32 — Kimi MXFP4 barrier timeout: TP3 load exceeds the 480s post-load barrier (17s margin in last green); ❌ none — raise UNBALANCED_MODEL_LOADING_TIMEOUT_S or fix TP0/TP3 NUMA load skew. Distinct from cluster "MLA prefill mismatch" despite the same test file.
• row 33 — logprob-length assert crash: single flaky input_top_logprobs_val length mismatch under chunked prefill; ❌ none — loop-repro on MI325.
• row 34 — kv_canary verify_position violation: new regression in the 7-commit window d7b942552927..99f5a6f46b14; [MEDIUM] #31662 (10908a679) skips freeing protected-prefix slots in ChunkCache decode; ❌ none — git revert 10908a679 + re-run.

Suggested triage order

Rank	Triage step	Targets	Why this first
1	Verify/merge #31675 (one-line eager-fallback fix)	rows 9-13 (cluster E, 5 jobs)	Fresh MI35x regression from #31391, HIGH-confidence, fix ready
2	Stabilize & merge #31325	rows 14-17 (cluster B, 4 jobs)	Never-green 9d, deterministic, fix exists (unblock its CI)
3	Merge #31263 + file FLUX-fixture follow-up	rows 18-21 (cluster A, 4 jobs)	Never-green since 07-16, fix ready
4	Add per-test timeout + persisted log/py-spy to 1-GPU diffusion step	rows 1-8 (cluster C, 8 jobs)	Largest cluster but undiagnosable — evidence capture unblocks all root-causing
5	git revert 10908a679 A/B for kv_canary; land #29127 for PP disagg	rows 34, 25-27	Row 34 is a fresh, well-localized regression; PP flake has a known runner-NIC suspect

---

Job	Failed Steps	Started
multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 3)	Run diffusion server tests (1-GPU)	2026-07-18T13:51
multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 0)	N/A	2026-07-18T13:54
multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 1)	N/A	2026-07-18T13:46
multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 1)	Run diffusion server tests (2-GPU)	2026-07-18T13:50
multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 0)	N/A	2026-07-18T12:34
stage-b-test-2-gpu-large-amd (linux-mi325-2gpu-sglang, 1)	Run test	2026-07-18T12:51
stage-c-test-large-8-gpu-amd-mi35x (linux-mi35x-gpu-8, 0)	Run test	2026-07-18T12:24
multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 1)	Run diffusion server tests (2-GPU)	2026-07-18T00:25
stage-c-test-large-8-gpu-amd-mi35x (linux-mi35x-gpu-8, 1)	Run test	2026-07-18T12:24
multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 0)	Run diffusion server tests (2-GPU)	2026-07-18T00:52
multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 3)	Run diffusion server tests (1-GPU)	2026-07-18T01:28
multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 1)	N/A	2026-07-18T01:25
multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 0)	N/A	2026-07-18T01:24
stage-b-test-large-8-gpu-mi35x-disaggregation-amd (linux-mi35x-gpu-8.fabric)	Run test_disaggregation	2026-07-18T06:38
stage-b-test-1-gpu-small-amd (linux-mi325-1gpu-sglang, 4)	Run test	2026-07-18T01:30
stage-c-test-large-8-gpu-amd (linux-mi325-8gpu-sglang, 1)	Run test	2026-07-18T00:25
stage-c-test-large-8-gpu-amd (linux-mi325-8gpu-sglang, 0)	N/A	2026-07-18T00:32
stage-c-test-large-8-gpu-amd-mi35x (linux-mi35x-gpu-8, 0)	Run test	2026-07-18T10:23
stage-c-test-large-8-gpu-amd-mi35x (linux-mi35x-gpu-8, 1)	Run test	2026-07-18T09:53
stage-b-test-large-8-gpu-mi35x-disaggregation-amd (linux-mi35x-gpu-8.fabric)	Run test_disaggregation	2026-07-19T00:28
multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 3)	Run diffusion server tests (1-GPU)	2026-07-19T00:49
stage-c-test-large-8-gpu-amd-mi35x (linux-mi35x-gpu-8, 1)	Run test	2026-07-19T00:56
multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 1)	Run diffusion server tests (2-GPU)	2026-07-19T00:47
call-pr-test-amd-extra / extra-a-test-2-gpu-large-amd (linux-mi325-2gpu-sglang)	Run test	2026-07-19T00:59
stage-c-test-large-8-gpu-amd-mi35x (linux-mi35x-gpu-8, 0)	Run test	2026-07-19T00:50
multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 1)	N/A	2026-07-19T00:48
multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 0)	N/A	2026-07-19T00:33
multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 0)	N/A	2026-07-19T00:43
stage-b-test-large-8-gpu-mi35x-disaggregation-amd (linux-mi35x-gpu-8.fabric)	Run test_disaggregation	2026-07-19T12:27
multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 3)	Run diffusion server tests (1-GPU)	2026-07-19T13:27
multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 1)	Run diffusion server tests (2-GPU)	2026-07-19T13:51
multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 0)	N/A	2026-07-19T13:16
multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 1)	N/A	2026-07-19T13:18
multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 0)	N/A	2026-07-19T15:10

Per-Job Analysis

multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 3) — failed step(s): Run diffusion server tests (1-GPU)

Commit Info

• sglang: d7b942552
• aiter: 9127c94a1 — v0.1.17.dev110+g9127c94a1 (source: dev/patched image, rebuild skipped; AITER_COMMIT_OVERRIDE empty). Matches Dockerfile default.

Failed Tests

Job: multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 3) — 5 failed, 1 passed, 4 errors in the single test file test/single_test_file/test_update_weights_from_disk.py.

Test File	Test Function	Error	Log
python/sglang/multimodal_gen/test/single_test_file/test_update_weights_from_disk.py	TestUpdateWeightsFromDisk::test_update_weights_from_disk_default[Qwen-Image]	assert 500 == 200 (server returned HTTP 500)	Log
same	TestUpdateWeightsFromDisk::test_update_weights_specific_modules[Qwen-Image]	AssertionError: Update failed: {'success': False, 'message': 'Inplace update to inference tensor outside InferenceMode is not allowed...'} (500)	Log
same	TestUpdateWeightsFromDisk::test_corrupted_weights_rollback[Qwen-Image]	assert (500 == 200)	Log
same	TestUpdateWeightsFromDiskWithOffload::test_update_weights_with_offload_enabled[FLUX.2-klein-base-4B]	AssertionError: Expected 200, got 500	Log
same	TestUpdateWeightsFromDiskWithOffload::test_update_weights_with_offload_enabled[Qwen-Image]	AssertionError: Expected 200, got 500	Log
same	TestUpdateWeightsFromDisk::test_update_weights_from_disk_default[FLUX.2-klein-base-4B] (ERROR at setup)	AssertionError: No weights dir for transformer in /tmp/sglang_perturbed_vae_* (fixture diffusion_server_no_offload line 410 → _compute_checksum_from_disk line 220)	Log
same	TestUpdateWeightsFromDisk::test_update_weights_specific_modules[FLUX.2-klein-base-4B] (ERROR at setup)	same fixture error	Log
same	TestUpdateWeightsFromDisk::test_update_weights_rejects_invalid_requests[FLUX.2-klein-base-4B] (ERROR at setup)	same fixture error	Log
same	TestUpdateWeightsFromDisk::test_corrupted_weights_rollback[FLUX.2-klein-base-4B] (ERROR at setup)	same fixture error	Log

Failure Summary

The standalone test file test_update_weights_from_disk.py produced 5 failures + 4 setup errors. Two distinct symptoms: (1) the update_weights_from_disk HTTP endpoint returns 500 for every Qwen-Image case and both offload cases; (2) all four no-offload FLUX.2-klein-base-4B cases error during fixture setup because the perturbed-clone temp dir has no transformer/ subdirectory. The single pass was test_update_weights_rejects_invalid_requests[Qwen-Image].

Regression Status

Never-passed (no green run found). The test file was added on 2026-07-16 in 7d0fd5101 (#31227, "misc optimizations") and has failed in every scheduled run since. No passing run exists in the queryable window.

Recent history of test_update_weights_from_disk.py in job multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 3) (scheduled, branch=main, completed runs; the file only exists in the 5 most recent):

Date	Run	Test File Status	Failed Functions	Error
Jul 18 12:17	29644059344	❌ Failed	5 failed + 4 setup errors	500 / "No weights dir for transformer"
Jul 18 00:25	29623067212	❌ Failed	same set	same
Jul 17 12:25	29580198489	❌ Failed	same set	same
Jul 17 00:27	29544840981	❌ Failed	same set	same
Jul 16 12:31	29498496614	❌ Failed (first run with file)	same set	same ("Inplace update to inference tensor" confirmed in log)
Jul 11–16 (10 earlier runs)	—	⚪ N/A	test file did not exist yet	—

First observed failure: 2026-07-16 12:31 UTC (run 29498496614). No last-passing date exists.

Failure Cluster

Two symptom-level clusters:

• A: update_weights_from_disk HTTP 500 — "Inplace update to inference tensor outside InferenceMode" (5 tests: all Qwen + both offload)
• B: fixture setup error — perturbed clone missing transformer/ dir for FLUX.2 (4 FLUX no-offload ERRORs)

Facts (verified observations)

• Summary line: 5 failed, 1 passed, 11 warnings, 4 errors in 1182.40s; job exit code 1.
• Cluster A verbatim server error (from earliest run 29498496614 at 2026-07-16 16:24:39 and again this run): RuntimeError: Inplace update to inference tensor outside InferenceMode is not allowed. You can make a clone to get a normal tensor before doing inplace update. — raised in "scheduler event loop", surfaced to the test as HTTP 500.
• Cluster B verbatim: AssertionError: No weights dir for transformer in /tmp/sglang_perturbed_vae_n3lklmuj at test_update_weights_from_disk.py:220, invoked from the diffusion_server_no_offload fixture sanity loop at :410. Qwen does NOT hit this (Qwen fixture setup succeeds and reaches the endpoint) → cluster B is FLUX-specific.
• Code path for A: python/sglang/multimodal_gen/runtime/post_training/weights_updater.py:294 / :302 perform param.data.copy_(...) (inplace) inside load_weights_into_model, called from _apply_weights → update_weights_from_disk.
• AITER unchanged vs prior runs (dev image v0.1.17.dev110+g9127c94a1, no override); NVIDIA-vs-AMD not comparable (diffusion multimodal_gen suite).
• Reproducibility: identical failure set in all 5 runs since the file was introduced (5/5).

Hypothesised Causes (with confidence)

• [HIGH] Cluster A is a pre-existing bug in the diffusion weight-update path: weights_updater.py calls param.data.copy_() on inference-mode tensors (created during server warmup) outside an inference_mode/no_grad context, which PyTorch forbids.
  • Supporting: exact error string Inplace update to inference tensor outside InferenceMode; the in-flight fix #31263 modifies exactly weights_updater.py to "run weight update under torch.inference_mode()"; the inplace copy_ sites are visible at weights_updater.py:294,302.
  • Disconfirming: (none found) — error is deterministic and present in the very first run of the test.
  • How to verify: apply #31263 on a branch, re-run test_update_weights_from_disk.py on AMD MI325.
• [MEDIUM] Cluster B is a test-fixture/model-layout bug: _clone_model_with_modified_module(local_source, perturbed_vae_model_dir, _VAE_MODULE, ...) does not reproduce a transformer/ subdirectory for the FLUX.2 source-model layout, so the _DIFFERING_MODULES sanity loop (transformer, vae) fails on transformer. Qwen's layout clones correctly.
  • Supporting: assertion is on the temp clone dir /tmp/sglang_perturbed_vae_*, not the HF cache; only FLUX errors, Qwen's same fixture succeeds; FLUX transformer does exist in the HF snapshot (.../transformer/diffusion_pytorch_model.safetensors loaded at server start) — so the gap is in the clone, not the source.
  • Disconfirming: full clone internals of _clone_model_with_modified_module not traced line-by-line; could instead be a race (clone runs in a background thread joined before the sanity loop, so a race is unlikely but not excluded).
  • How to verify: run only the FLUX no-offload fixture locally and inspect the contents of perturbed_vae_model_dir.
• Note: cluster B is not covered by #31263 (that PR only touches the server weight-update path, not the test fixture).

In-flight Fix Check

• Cluster A: ✅ #31263 "[diffusion] post_training: run weight update under torch.inference_mode()" — open (non-draft) since 2026-07-15, mergeable_state: unstable, changes only weights_updater.py (+31/-16). Chase reviewers / green the checks; do NOT open a duplicate. (Search: is:open InferenceMode multimodal.)
• Cluster B: ❌ none found — the FLUX clone/fixture issue needs a new PR or investigation. (Search: is:open update weights disk diffusion.)

Suggested Triage Steps

• Merge-blocker verification for A: apply #31263 on a branch, re-run test_update_weights_from_disk.py on AMD MI325; expect the 3 Qwen + 2 offload 500s to clear.
• For B (independent of #31263): inspect _clone_model_with_modified_module behavior for the FLUX.2-klein-base-4B source layout; confirm whether it copies non-target module subdirs (transformer/) for FLUX.
• Consider whether this newly-added test should have been gated/skipped on AMD until #31263 lands, since it was introduced red into the scheduled AMD lane (broken since 2026-07-16).

Status

• Persistence: Never-passed; failing every scheduled run since 2026-07-16 12:31 UTC (5/5 runs since the file was introduced in 7d0fd5101).
• Scope: 1 job (multimodal-gen-test-1-gpu-amd shard 3), 1 test file, 9 test cases (5 failed / 4 errored). Two independent causes (A: server weight-update path; B: FLUX test fixture).
• Blocked work: diffusion update_weights_from_disk CI signal on AMD is red.
• In-flight fix: A → ✅ #31263 (awaiting merge); B → ❌ none.

multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 0) — failed step(s): N/A

Commit Info

• sglang: d7b942552 (#31636, "[NPU] fix: skip Triton embedding kernel on NPU")
• aiter: N/A — shard 0's log was never uploaded (see below), so no [CI-AITER-CHECK] markers are readable for this job.

Failed Tests

Test File	Test Function	Error	Log
N/A (log not captured)	N/A	Job hung and was terminated mid-test; step "Run diffusion server tests (1-GPU)" left in_progress, no output uploaded	Job

This is not a capturable test failure — it is a runtime hang/termination with no log. The specific failing test/function on shard 0 cannot be pinned from logs because none exist for it.

Failure Summary

Job multimodal-gen-test-1-gpu-amd (…, 0) failed after 74 min (88079191158, 13:54:06→15:08:09 UTC). Step 9 "Run diffusion server tests (1-GPU)" (90-min step timeout) was still in_progress and steps 10/20 pending at termination, i.e. the job died while a diffusion test was running. The job-log blob returns BlobNotFound, and shard 0's file is absent from the run-log archive (only shards 2 and 3 uploaded logs) — the hang prevented log upload. Sibling shard 1 (71 min) failed identically with no log; shard 2 passed (13 min); shard 3 failed (30 min) with a captured real error.

Regression Status

Known recurring failure / effectively Never-passed in the queryable window. Shard 0 of this job has failed in every completed scheduled run for the last ~6 days, always with a long duration (66–91 min) and (for the long runs) no uploaded log — the signature of a persistent hang. No green shard-0 run exists in the last 15 scheduled runs.

Recent history of multimodal-gen-test-1-gpu-amd (…, 0) (most recent comparable completed scheduled runs: branch=main, event=schedule, completed only). Test-file status is unknown where the log was never captured:

Date (UTC)	Run	Job	Test File Status	Duration	Note
07-18 12:17	run	shard 0	❌ Failed (no log)	74 min	this job
07-18 00:25	run	shard 0	❌ Failed (no log; hit 90-min step timeout)	91 min
07-17 12:25	run	shard 0	❌ Failed (no log)	85 min
07-17 00:27	run	shard 0	❌ Failed (no log)	73 min
07-16 12:31	run	shard 0	❌ Failed (no log)	75 min
07-15 12:28	run	shard 0	❌ Failed	10 min	fast fail (different signature)
07-15 00:22	run	shard 0	❌ Failed (no log)	79 min
07-14 12:26	run	shard 0	❌ Failed (no log)	76 min
07-14 00:24	run	shard 0	❌ Failed (no log)	68 min
07-13 12:48	run	shard 0	❌ Failed (no log)	66 min
07-13 00:28	run	shard 0	❌ Failed (no log)	67 min

First observed in this query window: 07-13 00:28 (still failing then; likely older). No passing shard-0 run found in the 15-run lookback ⇒ marked Never-passed (in window).

Failure Cluster

"multimodal-gen 1-GPU diffusion shard hangs to termination (no log captured) on AMD MI325"

Facts (verified observations)

• Job conclusion failure, duration 74 min; step "Run diffusion server tests (1-GPU)" (timeout-minutes: 90) left in_progress, steps "Upload diffusion failure artifacts" / "Post Checkout code" pending (verified via jobs API steps[]).
• Shard 0 job-log endpoint returns BlobNotFound; shard 0 (and shard 1) files are absent from the run-log archive runs/29644059344/logs — only 1-gpu shards 2 and 3 uploaded logs.
• Sibling shards in the same run: shard 1 ❌ 71 min (no log), shard 2 ✅ 13 min, shard 3 ❌ 30 min (log present).
• No mass-cancellation: failed jobs completed at spread-out times (shard 1 at 14:57:06, shard 0 at 15:08:09) — each hung independently, not a single workflow-level cancel.
• The 1-gpu suite (run_suite.py --suite 1-gpu … --total-partitions 4) partitions cases by LPT estimated runtime (gpu_cases.py:990-1016), so shard 0 receives the heaviest cases (large video/image gen: Wan2.1-T2V / HunyuanVideo / Qwen-Image / FLUX.1 from ONE_GPU_CASES in test_server_1_gpu.py). This assignment is data-driven, not static — the exact shard-0 case set for this run is not recoverable without the log.
• Sibling shard 3 (different shard, captured log) failed with a real AssertionError at python/sglang/multimodal_gen/runtime/layers/linear.py:410 (assert param_data.shape == loaded_weight.shape) during test_update_weights_from_disk.py weight update — this is a separate failure, not shard 0's.

Hypothesised Causes (with confidence)

• [LOW] Shard 0 hangs inside a long-running diffusion generation case (video/image T2V/I2V) on ROCm, never completing, so the runner is reclaimed/terminated between ~66 and ~91 min with no log upload.
  • Supporting: 74-min runtime with step still in_progress; identical multi-day pattern (66–91 min, no logs) on shards 0 and 1 while shorter shards (2/3) finish and upload; one run hit exactly the 90-min step timeout (91 min total).
  • Disconfirming: with zero shard-0 log, there is no stack trace, no test name, no error — the hang location is entirely inferred. Cannot exclude a runner/container infrastructure fault (GPU wedged, HF download stall) independent of the test code.
  • How to verify: reproduce run_suite.py --suite 1-gpu --partition-id 0 --total-partitions 4 -k "not flux_2" on an MI325 in the CI container with a per-case timeout + py-spy dump on hang; inspect which case wedges.
• [LOW] Shared root cause with the shard-3 weight-loader AssertionError (linear.py:410) if shard 0 also runs a weight-update or model-load path.
  • Supporting: same suite, same commit; shard 3 shows a concrete model-loading shape mismatch.
  • Disconfirming: shard 3 failed fast with a captured error (30 min), whereas shard 0 hangs (74 min, no output) — different symptom class; LPT keeps the 1200s-estimated test_update_weights_from_disk.py on one shard (shard 3 here), so it is unlikely on shard 0.
  • How to verify: recover a shard-0 log via a manual rerun; compare failing case set.

No hypothesis reaches MEDIUM — the missing shard-0 log blocks code-level attribution. Root cause needs a manual rerun that captures the log (or a per-case timeout so the harness emits partial output before the hang).

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for multimodal diffusion timeout (18 hits) and multimodal-gen hang (9 hits); none target the 1-GPU shard hang / missing-log problem (results are disagg-diffusion, NPU, and FLUX.2-FP8 features). API: search/issues?q=repo:sgl-project/sglang+is:pr+is:open+multimodal+diffusion+timeout.

Suggested Triage Steps

• Manually rerun multimodal-gen-test-1-gpu-amd shard 0 with a per-case timeout and py-spy/SIGTERM-triggered stack dump so the hang produces output — the current failure is undiagnosable because no log is ever captured.
• Add a step to upload partial pytest output/--timeout (pytest-timeout) so a hanging case fails the individual test instead of wedging the whole 90-min step; this converts the recurring silent hang into a test-level signal.
• Once a log exists, pin the wedging case and bisect the 1-gpu case history; check whether it is a ROCm-specific hang in a video-generation pipeline (Wan T2V / HunyuanVideo) under the CI env (SGLANG_TEST_NUM_INFERENCE_STEPS=5, aiter attention backend).
• Separately, triage the sibling shard-3 weight_loader shape AssertionError at linear.py:410 (distinct failure, but same job family).

Status

• Persistence: shard 0 has failed every completed scheduled run for ≥6 days (07-13 → 07-18), always 66–91 min, no green run in the 15-run window.
• Scope: shards 0 and 1 of multimodal-gen-test-1-gpu-amd both hang with no logs; shard 3 fails separately with a captured error; shard 2 passes. Confined to the 1-GPU diffusion suite on pr-test-amd.yml.
• Blocked work: no reliable PR-CI signal for the heaviest 1-GPU diffusion cases on AMD MI325 (they never complete and never log).
• In-flight fix: ❌ none found — needs new investigation/PR.

multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 1) — failed step(s): N/A

Commit Info

• sglang: d7b942552
• aiter: N/A — the job log blob is missing (see below), so [CI-AITER-CHECK] markers are unrecoverable for this run.

Failed Tests

Test File	Test Function	Error	Log
N/A (log unrecoverable)	N/A	Abnormal job termination during Run diffusion server tests (1-GPU) — step ran ~71 min then died without a step conclusion; per-job log never uploaded (BlobNotFound)	Job

The exact failing test cannot be recovered for this shard: GitHub returns BlobNotFound for the shard-1 job log, and the run-level log archive contains only the shards that terminated cleanly (shards 2 and 3), not shards 0 and 1. Nearest observable evidence is sibling shard 3 (see Facts).

Failure Summary

Job multimodal-gen-test-1-gpu-amd (…, 1) failed during the Run diffusion server tests (1-GPU) step (run_suite.py --suite 1-gpu --partition-id 1 --total-partitions 4). The step ran ~71 min (under its 90-min timeout), step 9 recorded no conclusion (None), there is no "Complete job" step, and the log blob is missing — the signature of a hung/force-terminated job rather than a clean pytest failure. Sibling shard 0 behaved identically; shards 2 (pass) and 3 (clean failure, 30 min) completed normally.

Regression Status

Known recurring / persistent regression — began 2026-07-09, failing every completed scheduled run since.

Recent history of job multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 1) (most recent comparable completed scheduled runs: branch=main, event=schedule, completed only). Because the log blobs are missing for the hung runs, status is reported at job-conclusion + hang-signature level (step 9 conclusion, runtime); test-file granularity is unrecoverable:

Date (UTC)	Run	Job concl.	Step 9	Runtime	Signature
07-18 12:17	29644059344	❌ failure	None	71m	hang/abnormal (this run)
07-18 00:25	29623067212	❌ failure	None	72m	hang/abnormal
07-17 12:25	29580198489	❌ failure	None	75m	hang/abnormal
07-17 00:27	29544840981	❌ failure	n/a	71m	hang/abnormal
07-16 12:31	29498496614	❌ failure	None	71m	hang/abnormal
07-15 12:28	29415344552	❌ failure	None	69m	hang/abnormal
07-15 00:22	29378873430	❌ failure	None	68m	hang/abnormal
07-14 12:26	29332511138	❌ failure	None	71m	hang/abnormal
07-14 00:24	29295847781	❌ failure	None	86m	hang/abnormal
07-13 12:48	29251251853	❌ failure	None	63m	hang/abnormal
07-13 00:28	29215394044	❌ failure	None	66m	hang/abnormal
07-12 12:18	29192376210	❌ failure	None	60m	hang/abnormal
07-10 00:32	29060215979	❌ failure	n/a	74m	hang/abnormal
07-09 12:49	29019359571	❌ failure	n/a	81m	first failure
07-09 00:31	28985427127	✅ success	success	55m	last pass

• First observed failure: run 29019359571 — 2026-07-09 12:49 UTC (head 6ab7a65d9).
• Last known pass: run 28985427127 — 2026-07-09 00:31 UTC (head 074bb928f).
• Regression window: 074bb928f...6ab7a65d9 — 36 commits, ~17 touching python/sglang/multimodal_gen/.

Failure Cluster

Diffusion 1-GPU test shard hang / abnormal job termination (no step conclusion, missing logs)

Facts (verified observations, no interpretation)

• [FACT] Shard-1 per-job log returns BlobNotFound (HTTP 404); run-level archive contains 1-GPU shards 2 & 3 only, not shards 0 & 1.
• [FACT] Shard 1 (this job): step 9 Run diffusion server tests (1-GPU) conclusion = None; no Complete job step; started 13:46:04Z, completed 14:57:06Z (~71 min).
• [FACT] Sibling shard 0 88079191158 has the identical signature (step 9 None, no Complete-job, ~74 min, log BlobNotFound).
• [FACT] Sibling shard 3 88079191154 terminated cleanly (step 9 = failure, Complete-job = success, ~30 min, log present); shard 2 88079191166 = success.
• [FACT] Shard 3's clean failures are RuntimeError: Inplace update to inference tensor outside InferenceMode is not allowed in test/single_test_file/test_update_weights_from_disk.py (many TestUpdateWeightsFromDisk params → HTTP 500). This is a distinct symptom from shard 1's hang.
• [FACT] Shard 1 has failed with the hang signature in every completed scheduled run from 2026-07-09 12:49 through 2026-07-18; last green was 2026-07-09 00:31.
• [FACT] Regression window 074bb928f...6ab7a65d9 modifies diffusion runtime code including breakable_cuda_graph/runner.py, breakable_cuda_graph/replay_token.py, layers/attention/layer.py, layers/attention/backends/amx_attn.py.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [LOW] Commit 0562ccb1a "Move breakable CUDA graph back into model_executor/runner_backend_utils" (#30586) — lands in the regression window and modifies multimodal_gen/runtime/breakable_cuda_graph/runner.py + adds replay_token.py + touches attention/layer.py; broken CUDA-graph replay is a classic hang source on the diffusion inference path.
  • Supporting: reachable from the diffusion server forward pass; symptom is a hang, consistent with a stuck graph replay/capture.
  • Disconfirming: no shard-1 stack trace to confirm the hang is in graph replay; shards 2/3 running the same code did not hang, so any hang must be test/model-specific to shard 1's LPT-assigned cases.
  • How to verify: re-run run_suite.py --suite 1-gpu --partition-id 1 --total-partitions 4 -k "not flux_2" in the AMD MI325 container with py-spy dump on the diffusion server after ~60 min to capture the hung frame; A/B revert 0562ccb1a on a branch and re-run.
• [LOW] Commit 177c048c6 "[Diffusion][CPU] Adding AMX optimizations for CPU platform" (#28527) — in-window, touches diffusion attention/layer.py, linear.py, VAE/text-encoder loaders and platforms/interface.py.
  • Supporting: broad diffusion loader/attention surface changed in the same window.
  • Disconfirming: change is CPU/AMX-targeted; unlikely to alter the AMD GPU (ROCm) path that this shard exercises.
  • How to verify: same repro; low priority relative to the CUDA-graph candidate.
• [LOW] Whole 36-commit window is unbisected; no candidate reaches MEDIUM without the shard-1 log. Needs py-spy/coredump capture or git bisect across 074bb928f...6ab7a65d9.

Note: the InferenceMode error (#31263) is not advanced as the shard-1 cause — it produces clean HTTP-500 test failures (shard 3 completed cleanly in 30 min), not a 71-min hang.

In-flight Fix Check

• Shard-1 hang: In-flight fix: ❌ none clearly matching. Closest is #31361 "[diffusion] fix: don't self-kill diffusion worker when PID 1 is the real parent" (open since 2026-07-15) — a diffusion worker SIGKILL -9 bug that leaves "the server never becomes ready", which would cause a client-side hang. Disconfirming: it triggers only when the diffusion server is PID 1; sibling shards 2/3 in the same run got servers ready and returned responses, so this is unlikely to be the shard-1 root cause. (searched is:pr is:open multimodal_gen diffusion.)
• Sibling InferenceMode cluster (shard 3, context only): In-flight fix: ✅ #31263 "[diffusion] post_training: run weight update under torch.inference_mode()" (open since 2026-07-15) — its body explicitly names multimodal-gen-test-1-gpu running test_update_weights_from_disk.py with this exact error. Chase reviewers; do not duplicate. (searched is:pr is:open InferenceMode.)

Suggested Triage Steps

• Re-run the shard-1 command in the AMD MI325 container and attach py-spy dump (or rocgdb) to the diffusion server process after ~60 min to capture the hung frame — this is the only way to identify which LPT-assigned test in partition 1 hangs, since the CI log is unrecoverable.
• Once the hung test is known, bisect the diffusion-runtime commits in 074bb928f...6ab7a65d9, starting with the breakable-CUDA-graph relocation 0562ccb1a (#30586); A/B revert on a branch and re-run partition 1.
• File/track a CI-infra follow-up: shards 0 & 1 lose their logs entirely on this hang path (BlobNotFound, no artifact upload). Consider a hard per-test timeout inside run_suite.py so a hung case fails fast with an uploaded log instead of consuming ~71 min and dropping evidence.
• Track #31263 separately for the InferenceMode failures affecting the other 1-GPU shards.

Status

• Persistence: shard-1 hang failing every completed scheduled run since 2026-07-09 12:49 UTC (~9 days); last green 2026-07-09 00:31 UTC.
• Scope: 1-GPU diffusion shards 0 and 1 (both hang/abnormal) in this run; shard 3 fails cleanly with a separate InferenceMode symptom; 2-GPU diffusion shards 0 and 1 also failed this run (not analysed here).
• Evidence gap: shard-1 and shard-0 logs are unrecoverable (BlobNotFound); root cause requires a live repro with process-dump instrumentation.
• In-flight fix: hang — ❌ none matching (#31361 is a weak, likely-non-applicable candidate); InferenceMode sibling cluster — ✅ #31263.

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-07-19 17:01 UTC)

[amd-bot]

pr-test-amd.yml — Per-Job Analysis (continued 2/7)

multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 1) — failed step(s): Run diffusion server tests (2-GPU)

Commit Info

• sglang: d7b942552
• aiter: 9127c94a1 (v0.1.17.dev110+g9127c94a1, source: dev/patched image preserved — [CI-AITER-CHECK] reported "Dev/patched version detected → skipping rebuild"; AITER_COMMIT_OVERRIDE empty)

Failed Tests

Test File	Test Function	Error	Log
python/sglang/multimodal_gen/test/single_test_file/test_disagg_server.py	TestDisaggZImage1Rank::test_generates_image	ERROR at setup → RuntimeError: Warmup request failed for zimage_1rank: 500 Server Error	Log
python/sglang/multimodal_gen/test/single_test_file/test_disagg_server.py	TestDisaggZImage2RankDenoiser::test_generates_image_with_sp2_denoiser	ERROR at setup → RuntimeError: Warmup request failed for zimage_sp2: 500 Server Error	Log
python/sglang/multimodal_gen/test/single_test_file/test_disagg_server.py	TestDisaggZImageTracing::test_disagg_spans_share_trace_id	ERROR at setup → RuntimeError: Warmup request failed for zimage_trace: 500 Server Error	Log

Failure Summary

All 3 disaggregated-diffusion (ZImage) tests in test_disagg_server.py error during setUpClass, not in the test bodies. The warmup image-generation request to the DiffusionServer returns HTTP 500 because the request is never completed and the orchestrator raises DiffusionServer timeout: request <id> not completed within 120.0s (orchestrator.py:567). Each of the 3 clusters (zimage_1rank, zimage_sp2, zimage_trace) is retried 3× and times out every attempt, so the whole shard errors and the job exits 1.

Regression Status

Never-passed (no green run found in the queryable window). This is shard 1 of multimodal-gen-test-2-gpu-amd, which runs test_disagg_server.py (3 items) in every run. It has failed in every completed scheduled run back to at least 2026-07-12 with the identical DiffusionServer timeout symptom. (Shard 2 — a different test file — passed on Jul 16–18 but is unrelated to this test file.)

Recent history of test_disagg_server.py (all 3 ZImage functions) in job multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 1) (most recent comparable completed scheduled runs on main, event=schedule):

Date (UTC)	Run	Test File Status	Failed Functions	Error
Jul 18 12:17	29644059344	❌ Failed	all 3 ZImage tests	DiffusionServer timeout 120s (verified)
Jul 18 00:25	29623067212	❌ Failed	all 3 ZImage tests	same (verified)
Jul 17 12:25	29580198489	❌ Failed	all 3 ZImage tests	same (verified)
Jul 17 00:27	29544840981	❌ Failed	(shard failure)	same cluster
Jul 16 12:31	29498496614	❌ Failed	(shard failure)	same cluster
Jul 15 12:28	29415344552	❌ Failed	(shard failure)	same cluster
Jul 15 00:22	29378873430	❌ Failed	(shard failure)	same cluster
Jul 14 12:26	29332511138	❌ Failed	all 3 ZImage tests	same (verified)
Jul 14 00:24	29295847781	❌ Failed	all 3 ZImage tests	DiffusionServer timeout 120s (verified)
Jul 13 12:48	29251251853	❌ Failed	(shard failure)	same cluster
Jul 13 00:28	29215394044	❌ Failed	(shard failure)	same cluster
Jul 12 12:18	29192376210	❌ Failed	(shard failure)	same cluster

Lookback window: last 15 completed scheduled runs (Jul 12–18); no passing run found for this test file. First observed failure ≥ Jul 12; the shallow CI clone prevents dating the true origin.

Failure Cluster

Disagg ZImage diffusion request never completes → DiffusionServer timeout (120s) → 500 on warmup.

Facts (verified observations)

• Error verbatim: RuntimeError: Model generation returned no output. Error from scheduler: DiffusionServer timeout: request <id> not completed within 120.0s (job log lines 5930/6001/6069/6141).
• The 500 propagates from runtime/entrypoints/openai/utils.py:355 → image_api.py:305; the timeout is raised in runtime/disaggregation/orchestrator.py:567 (_handle_timeouts).
• All three transfer roles log topology.cpp:205 No RDMA devices found and transfer_metadata.cpp:912 Local segment descriptor not found; the Mooncake transfer engine falls back to TCP (tcp_transport.cpp: listen on port …).
• Models load successfully (Loaded transformer: ZImageTransformer2DModel … 11.46 GB) and all roles report "ready"/"event loop started" — the stall is at request completion, not startup.
• Each cluster (zimage_1rank, zimage_sp2, zimage_trace) failed all 3 warmup retries; 0 passed, 3 errored.
• aiter unchanged (dev image preserved); not an AITER Scout run, so no aiter override is in play.

Hypothesised Causes (with confidence)

• [MEDIUM] The disaggregated encoder→denoiser→decoder pipeline stalls in cross-role tensor transfer, so the orchestrator's request tracker never marks the request complete and hits the 120s timeout.
  • Supporting: timeout raised in orchestrator.py:567 _handle_timeouts; transfer_metadata.cpp:912 Local segment descriptor not found appears for every role; servers are healthy (GET /v1/models → 200) yet the generation request never returns; symptom is identical across 1-rank, sp2-denoiser, and tracing configs, pointing at the shared transfer path rather than any single model config.
  • Disconfirming: "No RDMA devices found" is expected on this runner and the engine does fall back to TCP, so RDMA absence alone is not proof of the transfer being the fault; the exact hang site inside the transfer/denoise loop is not shown in the log tail.
  • How to verify: run test_disagg_server.py on an AMD MI325 2-GPU box in image rocm/sgl-dev:v0.5.15.post1-rocm700-mi30x-20260717, enable per-role debug logging in runtime/disaggregation/orchestrator.py / transfer manager to see which role stops receiving; check whether a single-process (non-disagg) ZImage generation succeeds on the same runner to isolate transfer vs model.
• [LOW] Environment/runner issue specific to AMD (transfer engine + ROCm) rather than an sglang code regression.
  • Supporting: never-passed on AMD in the whole queryable window; failure is confined to the disagg transfer path.
  • Disconfirming: no NVIDIA baseline gathered here to confirm it passes elsewhere; cannot rule out a genuine sglang bug in the disagg orchestrator.
  • How to verify: compare against the NVIDIA multimodal-gen disagg job (if one exists) on the same commit.

In-flight Fix Check

• In-flight fix: 🟡 candidate #31325 "[diffusion] disagg: handle numpy arrays in cross-role transfer field extraction" (open, created 2026-07-15, not draft) — touches exactly the disagg cross-role transfer path that is stalling. Cannot confirm it resolves this timeout without a targeted rerun; do not assume, but chase/verify before opening a duplicate. (Searched sglang open PRs for disagg diffusion and DiffusionServer timeout.)

Suggested Triage Steps

• Verify whether #31325 fixes the warmup timeout: check out that branch and run test_disagg_server.py on an AMD MI325 2-GPU runner in container rocm/sgl-dev:v0.5.15.post1-rocm700-mi30x-20260717.
• Add per-role debug logging around runtime/disaggregation/orchestrator.py:555-567 and the transfer manager to pin the exact role/step where the request stalls (encoder handoff vs denoiser vs decoder).
• Run a single-process (non-disagg) ZImage generation on the same runner to isolate whether the ZImage model itself works, narrowing the fault to the disagg transfer layer.
• If confirmed to be a persistent AMD-transfer limitation with no near-term fix, consider quarantining test_disagg_server.py on AMD until the disagg transfer path is fixed, so the shard stops masking other regressions.

Status

• Persistence: Failing every completed scheduled run since at least 2026-07-12 (≥7 days); no green run for this test file in the 15-run window.
• Scope: multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 1) in pr-test-amd.yml; all 3 ZImage disagg tests in test_disagg_server.py.
• Blocked work: Disaggregated-diffusion (ZImage) CI signal on AMD is fully red; this shard cannot validate any disagg changes.
• In-flight fix: 🟡 candidate #31325 (open since 2026-07-15) — verify before duplicating.

multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 0) — failed step(s): N/A

Commit Info

• sglang: d7b942552
• aiter: 9127c94a1 (v0.1.17.dev110+g9127c94a1, source: dev image preserved — matches Dockerfile default; from sibling shard-1 [CI-AITER-CHECK] markers, since this job's own log is unavailable)

Failed Tests

Test File	Test Function	Error	Log
N/A (job never reached tests)	N/A	Job hung at Start CI container step (step 5, in_progress, never completed); job terminated after ~10 min. Job log artifact is unavailable (BlobNotFound).	Job

The underlying persistent failure for this suite (observed in the sibling shard, multimodal-gen-test-2-gpu-amd (…, 1), whose log is available) is test-level:

Test File	Test Function	Error	Log
sglang/multimodal_gen/test/single_test_file/test_disagg_server.py	TestDisaggZImage1Rank.test_generates_image	RuntimeError: Warmup request failed … 500 … DiffusionServer timeout: request not completed within 120.0s	Log
test_disagg_server.py	TestDisaggZImage2RankDenoiser.test_generates_image_with_sp2_denoiser	same (500 / 120s timeout)	Log
test_disagg_server.py	TestDisaggZImageTracing.test_disagg_spans_share_trace_id	same (500 / 120s timeout)	Log

Failure Summary

The target job multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 0) concluded failure but never reached its tests: step 5 Start CI container stayed in_progress and never completed, and the job's log blob is unavailable (BlobNotFound), so no test-level output exists for this run. Its sibling shard-1 (same image, same runner lane) did reach the diffusion suite and failed with DiffusionServer timeout HTTP 500 on all three TestDisaggZImage* warmup requests — the persistent problem this lane has shown for the entire query window.

Regression Status

Known recurring failure (persistent / never-passed in window). The multimodal-gen-test-2-gpu-amd (…, 0) job has failed in every completed scheduled pr-test-amd.yml run in the 15-run window; the 2-GPU disagg diffusion suite is the only lane that runs it and has been red continuously. This particular run is the only one where the job died at container-start rather than in the diffusion-test step — i.e. an additional infra hiccup layered on top of the standing failure.

Recent history of multimodal-gen-test-2-gpu-amd (…, 0) (same branch main, event=schedule, completed runs only):

Date (UTC)	Run	Test Step Status	Failing Step	Note
Jul 18 12:17	29644059344	❌ not reached	Start CI container (hang)	this run — infra, log unavailable
Jul 18 00:25	29623067212	❌ Failed	Run diffusion server tests (2-GPU)	diffusion suite
Jul 17 12:25	29580198489	❌ Failed (hang)	Run diffusion server tests (2-GPU)	diffusion suite
Jul 17 00:27	29544840981	❌ Failed	Run diffusion server tests (2-GPU)	diffusion suite
Jul 16 12:31	29498496614	❌ Failed	Run diffusion server tests (2-GPU)	diffusion suite
Jul 16 00:25	29461421122	— cancelled	—	excluded
Jul 15 12:28	29415344552	❌ Failed (hang)	Run diffusion server tests (2-GPU)	diffusion suite
Jul 15 00:22	29378873430	❌ Failed (hang)	Run diffusion server tests (2-GPU)	diffusion suite
Jul 14 12:26	29332511138	❌ Failed	Run diffusion server tests (2-GPU)	diffusion suite
Jul 14 00:24	29295847781	❌ Failed (hang)	Run diffusion server tests (2-GPU)	diffusion suite
Jul 13 12:48	29251251853	❌ Failed (hang)	Run diffusion server tests (2-GPU)	diffusion suite

No passing run found in the query window (back to Jul 13; older Jul 5–7 scheduled runs were also failure at the workflow level). PR #31325's description independently confirms this suite "has been red every nightly on the AMD 2-GPU lane."

Failure Cluster

Two distinct signatures in this job's run:

1. Start CI container hang (infra) — this shard, this run only.
2. Disagg diffusion warmup HTTP 500 / DiffusionServer timeout (Z-Image encoder→denoiser transfer) — the persistent lane failure, seen in the sibling shard and all prior runs.

Facts (verified observations, no interpretation)

• This job (88079191167): step 5 Start CI container is status=in_progress, conclusion=None; steps 6–20 pending; job ran 12:34:29→12:44:30 UTC (~10 min) then concluded failure. Its log fetch returns BlobNotFound (verified: /actions/jobs/88079191167/logs → XML <Code>BlobNotFound</Code>).
• Sibling shard-1 (88079191163) failing step is Run diffusion server tests (2-GPU); log shows RuntimeError: … DiffusionServer timeout: request … not completed within 120.0s and three ERROR at setup of TestDisaggZImage* → 500 Server Error … /v1/images/generations, exit code 1.
• Root error in the sibling denoiser/encoder log (verbatim): Pool ENCODER rank 0: error (attempt 1/3): The truth value of an array with more than one element is ambiguous. Use a.any() or a.all(), raised at python/sglang/multimodal_gen/runtime/disaggregation/scheduler_mixin.py line 244 (_is_default) via extract_transfer_fields (line 1563) in _disagg_encoder_step / _disagg_event_loop (line 1049).
• Model under test: Tongyi-MAI/Z-Image-Turbo, attention_backend=aiter, disagg encoder/denoiser/decoder split.
• 9/9 checked prior completed scheduled runs failed this exact job; 1 cancelled excluded.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [HIGH] Persistent lane failure: numpy-array values in Req cross-role transfer fields make _is_default() in scheduler_mixin.py:244 evaluate an array in boolean context → ValueError, so the encoder step never completes → denoiser/decoder never receive input → warmup request times out at 120s → HTTP 500 → all TestDisaggZImage* error at setup.
  • Supporting: sibling stack trace points exactly at scheduler_mixin.py:244 (_is_default inside extract_transfer_fields); open PR #31325 targets that exact file with title "handle numpy arrays in cross-role transfer field extraction" and its body cites the identical symptom (DiffusionServer timeout … 120s, red every nightly on AMD 2-GPU lane).
  • Disconfirming: the target shard-0 in this run died at container-start and never produced this trace, so for this specific run the ValueError→timeout chain is inferred from the sibling, not observed in shard-0's own (missing) log.
  • How to verify: apply PR #31325 on a branch, re-run test_disagg_server.py::TestDisaggZImage* on AMD MI325 2-GPU.
• [LOW] Start CI container infra hang (shard-0, this run): the container-start step never completed (~10 min) and the log blob was never uploaded, both classic signs of a runner/container provisioning stall rather than a code defect.
  • Supporting: steps 6+ never started; log = BlobNotFound; no other run in the window shows shard-0 failing at container-start (all others reach the test step), so this is a one-off.
  • Disconfirming: no log is available to confirm the cause; cannot rule out an early crash mis-reported as a hang.
  • How to verify: re-run the single job; if it reaches the test step it confirms a transient infra stall (and the job then fails on the standing diffusion bug above).

In-flight Fix Check

• In-flight fix: ✅ #31325 "[diffusion] disagg: handle numpy arrays in cross-role transfer field extraction" — open (non-draft) since 2026-07-15, modifies exactly python/sglang/multimodal_gen/runtime/disaggregation/scheduler_mixin.py, body describes this exact DiffusionServer timeout on the AMD 2-GPU lane. Chase reviewers; do not open a duplicate. (API call: search/issues?q=repo:sgl-project/sglang+is:pr+is:open+diffusion+disagg+encoder.)
• For the shard-0 container-start hang: ❌ none found — infra/transient, not a code fix target.

Suggested Triage Steps

• Verify #31325 merges cleanly and re-run test_disagg_server.py::TestDisaggZImage* on AMD MI325 2-GPU — this should clear the standing lane failure.
• Re-run the single job 88079191167 to confirm the Start CI container hang was a one-off runner stall (its log being BlobNotFound prevents post-mortem of this run).
• If container-start hangs recur on linux-mi325-2gpu-sglang, escalate to CI/infra (runner provisioning / log-upload), separate from the diffusion bug.

Status (factual, no priority assignment)

• Persistence: multimodal-gen-test-2-gpu-amd (…, 0) has failed every completed scheduled run in the 15-run window (since at least Jul 13, no green found; older Jul 5–7 runs also red). The Start CI container hang is unique to this run (Jul 18 12:17 UTC).
• Scope: 2-GPU AMD disagg diffusion lane — 5 multimodal-gen jobs failed in this run (2-GPU shards 0,1; 1-GPU shards 0,1,3); it is the only lane running the Z-Image disagg suite.
• Blocked work: blocks the entire multimodal-gen disagg-diffusion CI signal on AMD.
• In-flight fix: ✅ #31325 (open since 2026-07-15) addresses the persistent diffusion failure; the container-start hang has no fix (transient infra).

stage-b-test-2-gpu-large-amd (linux-mi325-2gpu-sglang, 1) — failed step(s): Run test

Commit Info

• sglang: d7b942552 ([NPU] fix: skip Triton embedding kernel on NPU #31636)
• aiter: 9127c94a1 — v0.1.17.dev110+g9127c94a1 (source: dev image preserved, [CI-AITER-CHECK] "Dev/patched version detected → skipping rebuild")

Failed Tests

Test File	Test Function	Error	Log
test/registered/model_loading/test_load_weights_from_remote_instance.py	TestLoadWeightsFromRemoteInstance.test_load_weights_from_remote_instance	AssertionError: sgl_dp_1_dst_params rank 1 → Exception: retry() exceed maximum number of retries.	Log

Other 6 files in this partition passed (Test Summary: 6/7 passed).

Failure Summary

The 2-GPU remote-instance weight-loading test failed: after loading weights from the seed instance to the destination instance over the transfer_engine backend, the destination parameters on rank 1 came back all-zero (Params2: [9.18e-41, 0, 0, …]) while the seed parameters (Params1) held real values, so np.allclose failed. The single test method retried once and then errored (Ran 1 test in 187.741s … FAILED (errors=1)), exiting the suite with code 1/255.

Regression Status

Flaky / intermittent — same test file + function + signature has failed in some but not all recent completed scheduled runs. It PASSED in the immediately preceding run and in the 4 most recent runs before this one, then failed here with the exact signature already seen on Jul 13 and Jul 15.

Recent history of test_load_weights_from_remote_instance.py (test_load_weights_from_remote_instance) in job stage-b-test-2-gpu-large-amd (linux-mi325-2gpu-sglang, 1) (comparable completed scheduled runs on main, event=schedule):

Date	Run	Test File Status	Failed Function	Error
Jul 18 12:17	29644059344	❌ Failed	test_load_weights_from_remote_instance	sgl_dp_1_dst_params rank 1 (dst all-zero)
Jul 18 00:25	29623067212	✅ Passed	—	—
Jul 17 12:25	29580198489	✅ Passed	—	—
Jul 17 00:27	29544840981	✅ Passed	—	—
Jul 16 12:31	29498496614	✅ Passed	—	—
Jul 16 00:25	29461421122	⏭️ Cancelled (excluded)	—	—
Jul 15 12:28	29415344552	❌ Failed	test_load_weights_from_remote_instance	same signature (dst all-zero)
Jul 15 00:22	29378873430	✅ Passed	—	—
Jul 14 12:26	29332511138	✅ Passed	—	—
Jul 14 00:24	29295847781	✅ Passed	—	—
Jul 13 12:48	29251251853	❌ Failed	test_load_weights_from_remote_instance	same signature (dst all-zero)

Failure ratio: 3 of 10 completed runs (~30%). First observed in this window: Jul 13 12:48. Not tied to the current HEAD commit (an NPU-only fix, unrelated to remote weight loading).

Failure Cluster

AMD remote-instance TransferEngine reports success but leaves destination weights zeroed

Facts (verified observations)

• Verbatim: AssertionError: sgl_dp_1_dst_params rank 1; dst Params2: [9.18354962e-41 0. 0. …] (uninitialized/zeroed), seed Params1 holds real values (log lines 17600, 17472).
• Stack: test_load_weights_from_remote_instance.py:331 → verify_params_close → :48 assert np.allclose(...); wrapped by retry() in python/sglang/srt/utils/common.py:3165/3180 → retry() exceed maximum number of retries (log 17585-17619).
• Server launched with --remote-instance-weight-loader-start-seed-via-transfer-engine (log 17583); the transfer_engine backend path is exercised, not NCCL.
• Byte-identical failure signature (9.18354962e-41 leading value) in prior failures 87352016133 (Jul 15) and 86820131635 (Jul 13).
• Last passing run 88021853232 ran this exact file and passed ("passed": true, "elapsed": 134), confirming intermittency.
• aiter unchanged across passing/failing runs (dev image v0.1.17.dev110+g9127c94a1 preserved every run).

Hypothesised Causes (with confidence)

• [MEDIUM] The AMD TransferEngine (Mooncake) remote-instance weight-loader path can return success while never writing the weights into the rank-1 destination buffer, leaving it zeroed. The test randomly selects the transfer_engine backend, so the failure is non-deterministic.
  • Supporting: dst buffer is exactly zero (not garbage) → transfer never landed; identical signature recurs only on some runs; the server explicitly uses --remote-instance-weight-loader-start-seed-via-transfer-engine; open PR #30035 describes this exact behavior ("AMD Mooncake path can report success while leaving destination weights zeroed").
  • Disconfirming: no code-level proof of which commit introduced the race; the same test passed on the immediately adjacent commit, so it is timing/environment-sensitive rather than a clean code regression.
  • How to verify: run test_load_weights_from_remote_instance.py in a loop (~10×) on an MI325 2-GPU runner forcing remote_instance_loader_backend=transfer_engine; expect intermittent all-zero rank-1 params.
• [LOW] Fresh regression from HEAD commit d7b942552.
  • Supporting: it is this run's HEAD.
  • Disconfirming: HEAD is an NPU-only Triton-embedding fix that does not touch weight loading; the same failure predates it (Jul 13, Jul 15) — so it cannot be the cause.
  • How to verify: n/a — already disconfirmed by history.

In-flight Fix Check

• In-flight fix: ✅ #30035 ("test: avoid AMD remote-instance TransferEngine flake", open draft since 2026-07-03) — pins the AMD CI-reduced test to the NCCL backend until the AMD TransferEngine path has a deterministic regression test + core fix. Directly targets this flake; chase the author to un-draft/merge, do NOT open a duplicate. (Searched search/issues?q=…is:pr+is:open+remote+instance+weight+loader.)

Suggested Triage Steps

• Un-draft and land #30035 to stop the CI flake, then separately track the core AMD Mooncake TransferEngine zeroed-weights bug (do not let the test workaround close the underlying issue).
• To confirm the core bug: loop the test on MI325 forcing the transfer_engine backend and assert rank-1 dst params are non-zero; capture whether the transfer API returns success while the buffer stays zero.
• If confirmed, investigate the AMD Mooncake write/completion path in the remote-instance weight loader (why the transfer completes "successfully" without landing bytes on rank 1).

Status

• Persistence: Intermittent — 3 of 10 completed scheduled runs failed in the Jul 13–18 window (Jul 13, Jul 15, Jul 18 12:17); passed in the 4 runs immediately prior.
• Scope: This single job/test file (stage-b-test-2-gpu-large-amd partition 1); only the AMD transfer_engine remote-loader path.
• Blocked work: Intermittently red-flags pr-test-amd scheduled runs; masks any genuine remote-instance-loader regression on AMD.
• In-flight fix: ✅ #30035 (open draft since 2026-07-03) — awaiting un-draft/merge.

stage-c-test-large-8-gpu-amd-mi35x (linux-mi35x-gpu-8, 0) — failed step(s): Run test

Commit Info

• sglang: d7b942552
• aiter: 9127c94a1 (v0.1.17.dev110+g9127c94a1, source: dev image preserved — [CI-AITER-CHECK] Dev/patched version detected → skipping rebuild)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/test_kimi_k25_mxfp4_bcg_mi35x.py	test_bcg_gsm8k (variant='bcg')	Exception: Server process exited with code -9 (underlying ValueError from dist_barrier_after_load — post-load monitored_barrier timed out; TP3 never finished loading)	Log

(The other file in this shard, test/registered/cuda_graph/breakable/test_breakable_cuda_graph.py, passed. Shard summary: 1/2 passed.)

Failure Summary

The amd/Kimi-K2.5-MXFP4 TP4 server never became healthy. Kimi MXFP4 (quark) MoE weights load at wildly uneven per-rank speeds via HIP memcpy_and_sync: TP2 finished in 380s, TP1 in 396s, TP0 in 1348s, and TP3 never finished within the window. Rank 0 (the barrier monitor) entered dist_barrier_after_load's monitored_barrier at 12:53:49 after its own slow load, waited the full UNBALANCED_MODEL_LOADING_TIMEOUT_S=480s, TP3 still hadn't arrived, so the barrier raised at 13:01:49 → SIGQUIT → kill_process_tree → server exit -9.

Regression Status

Known recurring failure (marginal / flaky timeout margin) — first observed 2026-07-17 12:25 UTC, last passing 2026-07-17 00:27 UTC.

Recent history of test_kimi_k25_mxfp4_bcg_mi35x.py (test_bcg_gsm8k) in job stage-c-test-large-8-gpu-amd-mi35x (linux-mi35x-gpu-8, 0) (comparable completed scheduled runs, event=schedule, branch=main):

Date	Run	Test File Status	Failed Function	Detail
Jul 18 12:17	29644059344	❌ Failed	test_bcg_gsm8k	exit -9; TP3 never finished load
Jul 18 00:25	29623067212	❌ Failed	test_bcg_gsm8k	exit code 1 (elapsed 1841s)
Jul 17 12:25	29580198489	❌ Failed	test_bcg_gsm8k	exit code 1 (elapsed 1792s) — first fail
Jul 17 00:27	29544840981	✅ Passed	—	elapsed 2022s; TP3 finished 463s after TP0 (17s under barrier)
Jul 14–15 (×3)	87352016564 etc.	⏭️ N/A	—	job aborted early (exit 134 Fatal Python error: Aborted in a different test); Kimi test not reached

Failure Cluster

Kimi-K2.5-MXFP4 TP4 post-load monitored_barrier timeout from a slow straggler rank (uneven HIP host→device MoE weight copy on MI35x).

Facts (verified observations)

• UNBALANCED_MODEL_LOADING_TIMEOUT_S = 480 at python/sglang/srt/model_executor/model_runner_components/load_model_utils.py:40; barrier raise site at load_model_utils.py:270. [FACT]
• Failing run per-rank Load weight end: TP2 elapsed=380s (12:37:41), TP1 396s (12:37:57), TP0 1348.75s (12:53:49). No TP3 Load weight end — final py-spy at 13:01:17 shows TP3 (PID 3183) still in do_load_weights → _load_w13 (moe/fused_moe_triton/layer.py:537) → c10::hip::memcpy_and_sync (libamdhip64.so). [FACT]
• Timeline math: rank 0 entered monitored_barrier at 12:53:49; error printed 13:01:49 = exactly +480s. Gloo error: Rank 2 successfully reached monitoredBarrier, but received errors while waiting for send/recv from rank 0. [FACT]
• Passing run (Jul 17 00:27) had the identical skew: TP2 357s, TP1 379s, TP0 1325.76s, TP3 1788.65s — TP3 arrived 463s after TP0, i.e. 17s inside the 480s window. Both pass and fail runs logged 30 soft-watchdog timeouts during load. [FACT]
• Same NUMA pinning both runs: TP0=CPU0-31, TP1=32-63, TP2=64-95, TP3=96-127; the two slow ranks are consistently TP0 and TP3. [FACT]
• mem_fraction_static resolved to 0.65025; weights fit fine (mem usage=133.35 GB, avail mem=145.88 GB after load) — not an OOM. [FACT]

Hypothesised Causes (with confidence)

• [HIGH] Marginal/infra timeout: TP3's MoE weight load on the MI35x (gfx950) runner is pathologically slow (~1788s+ vs ~380s for TP1/TP2), and the 480s post-load barrier window is razor-thin relative to the last-rank spread.
  • Supporting: pass and fail runs have near-identical load timings; the only difference is TP3 finishing 463s (pass) vs >480s (fail) after rank 0 entered the barrier. Load itself did not regress. Slowness localizes to HIP memcpy_and_sync on NUMA-pinned ranks TP0/TP3.
  • Disconfirming: none — the 17s margin in the last green run directly demonstrates the failure is a timing-margin crossing, not a functional break.
  • How to verify: on the MI35x runner, launch the same command and record per-rank Load weight end elapsed; check whether TP3 consistently exceeds TP0_finish + 480s.
• [LOW] A commit in the regression window dc0b3eb68..85ac56c82 (34 commits) slowed weight loading.
  • Supporting: window brackets the pass→fail transition; contains Kimi-touching commits e.g. 24a8944e1 (#31391), 7d0fd5101 (#31227).
  • Disconfirming: per-rank load elapsed is essentially unchanged between the passing and failing runs (TP0 1325s→1348s, TP3 1788s→hung), so load speed did not regress; #31391 touches cuda-graph replay (post-load), not the load path.
  • How to verify: not the leading lever; only pursue if the timing margin proves stable and a load-path slowdown is later observed.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for UNBALANCED_MODEL_LOADING_TIMEOUT (0 results) and kimi slow load (8 unrelated results, none address the barrier timeout or MI35x load skew).

Suggested Triage Steps

• Confirm the margin: re-run this job 2–3× and log per-rank Load weight end elapsed; if TP3 routinely lands within ~5% of the 480s barrier after rank 0, this is a flaky timeout, not a functional bug.
• Consider whether UNBALANCED_MODEL_LOADING_TIMEOUT_S (currently 480s, load_model_utils.py:40) is adequate for MI35x Kimi-K2.5-MXFP4 loads where the last-rank spread already reached 463s in the green run — raise on a branch and re-run test_kimi_k25_mxfp4_bcg_mi35x.py.
• Investigate why TP0/TP3 HIP host→device MoE copies are ~4–5× slower than TP1/TP2 (NUMA/PCIe/host-mem bandwidth asymmetry); if reproducible, the fix is on the runner/load-path side, not the test.
• Cross-check whether enable_multithread_load: true helps or hurts here by comparing per-rank elapsed with it disabled.

Status

• Persistence: Failing 3 consecutive completed scheduled runs since 2026-07-17 12:25 UTC (last green 2026-07-17 00:27 UTC, which passed with only a 17s barrier margin).
• Scope: 1 test file in job stage-c-test-large-8-gpu-amd-mi35x (linux-mi35x-gpu-8, 0) on pr-test-amd.yml; the sibling test_breakable_cuda_graph.py in the same shard passes.
• Blocked work: Kimi-K2.5-MXFP4 accuracy signal on MI35x (gfx950).
• In-flight fix: ❌ none found — needs new PR or investigation.

multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 1) — failed step(s): Run diffusion server tests (2-GPU)

Commit Info

• sglang: 238b2b2c9
• aiter: 9127c94a1 (v0.1.17.dev110+g9127c94a1, source: Dockerfile default — AITER_COMMIT_OVERRIDE empty)

Failed Tests

Test File	Test Function	Error	Log
test/single_test_file/test_disagg_server.py	TestDisaggZImage1Rank::test_generates_image	RuntimeError: Warmup request failed for zimage_1rank: 500 Server Error (ERROR at setup)	Log
test/single_test_file/test_disagg_server.py	TestDisaggZImage2RankDenoiser::test_generates_image_with_sp2_denoiser	RuntimeError: Warmup request failed for zimage_sp2: 500 Server Error (ERROR at setup)	Log
test/single_test_file/test_disagg_server.py	TestDisaggZImageTracing::test_disagg_spans_share_trace_id	RuntimeError: Warmup request failed for zimage_trace: 500 Server Error (ERROR at setup)	Log

Full path: python/sglang/multimodal_gen/test/single_test_file/test_disagg_server.py. Result: 7 warnings, 3 errors in 1995.80s; job exited with code 1.

Failure Summary

All 3 disaggregated Z-Image server tests in this shard error at setUpClass because the warmup request returns HTTP 500. The 500 originates in the disagg ENCODER pool, which raises ValueError: The truth value of an array with more than one element is ambiguous while extracting cross-role transfer fields; the encoder retries 3× and never produces output, so each request hits DiffusionServer timeout ... not completed within 120.0s and warmup fails.

Regression Status

Never-passed (no green run found in the queryable window) — the identical failure is present in every one of the last 15 comparable completed scheduled runs (branch main, event=schedule), from 2026-07-11 through 2026-07-18. No passing run for test_disagg_server.py exists in this window.

Recent history of test_disagg_server.py (TestDisaggZImage*) in job multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 1) (most recent comparable completed scheduled runs):

Date	Run	Test File Status	Failed Function	Error
Jul 18	29623067212	❌ Failed	all 3 TestDisaggZImage*	ValueError in encoder → 500 warmup
Jul 17	29580198489	❌ Failed	all 3	same
Jul 17	29544840981	❌ Failed	all 3	same
Jul 16	29498496614	❌ Failed	all 3	same
Jul 15	29415344552	❌ Failed	all 3	same
Jul 15	29378873430	❌ Failed	all 3	same
Jul 14	29332511138	❌ Failed	all 3	same
Jul 14	29295847781	❌ Failed	all 3	same
Jul 13	29251251853	❌ Failed	all 3	same
Jul 13	29215394044	❌ Failed	all 3	same
Jul 12	29173750641	❌ Failed	all 3	same
Jul 11	29132618067	❌ Failed	all 3	same

Lookback range: 2026-07-11 → 2026-07-18 (12 completed runs where the job ran; one intervening 2026-07-16 run was cancelled and excluded). No passing run observed for this test file.

Failure Cluster

ValueError (numpy array truth-value ambiguous) in disagg encoder transfer-field extraction → warmup 500 → DiffusionServer timeout

Facts (verified observations, no interpretation)

• Encoder-side traceback (verbatim), repeated for every request:

  File ".../disaggregation/scheduler_mixin.py", line 1563, in _disagg_encoder_step
    tensor_fields, scalar_fields = extract_transfer_fields(req_result)
  File ".../disaggregation/scheduler_mixin.py", line 244, in extract_transfer_fields
    if _is_default(value, f):
  ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
  

• Downstream symptom in the router/harness: RuntimeError: Model generation returned no output. Error from scheduler: DiffusionServer timeout: request <id> not completed within 120.0s, then 500 Server Error ... /v1/images/generations, 3 warmup attempts per cluster.
• Offending code at python/sglang/multimodal_gen/runtime/disaggregation/scheduler_mixin.py:153-159:

  def _is_default(value, field_info) -> bool:
      if field_info.default is not dataclasses.MISSING:
          return value == field_info.default   # line 155
      ...

  When value is a numpy array and field_info.default is a scalar/None, value == default yields an element-wise boolean array; returning it as a bool raises the ambiguous-truth-value error.
• Reproducibility: all 3 tests in the shard fail; the same signature appears in all 12 comparable completed runs 07-11 → 07-18.
• aiter unchanged across the window (Dockerfile default 9127c94a1, no override).

Hypothesised Causes (with confidence, with disconfirming evidence)

• [HIGH] The bug is in _is_default (scheduler_mixin.py:155): value == field_info.default produces a numpy array (not a scalar bool) when a Req field holds a numpy array, and coercing it to bool raises ValueError. The failing stack trace passes directly through this line, and the in-flight fix PR #31325 targets exactly this file with the description "handle numpy arrays in cross-role transfer field extraction".
  • Supporting: exact file+line+function match between traceback and source; open PR titled to fix this precise case; deterministic across every run.
  • Disconfirming: (none found) — the traceback, source, and fix PR all converge.
  • How to verify: apply PR #31325 on a branch and re-run test_disagg_server.py on AMD MI325 2-GPU.
• [LOW] Suspicion of the local file-creating commit 7d0fd5101 (#31227) — in this checkout it is the only commit touching scheduler_mixin.py.
  • Supporting: git log --follow attributes the whole file to it.
  • Disconfirming: the identical buggy _is_default code and the identical CI failure are present in runs whose head SHAs predate it (495ae9aaa, 50d1edaa7 on 07-15, and back to 07-11); git cat-file confirms the file already existed at those SHAs. So #31227's attribution here is a rebase/squash topology artifact, not the origin — do not blame it.

In-flight Fix Check

• In-flight fix: ✅ #31325 "[diffusion] disagg: handle numpy arrays in cross-role transfer field extraction" (open since 2026-07-15, base main, not draft, mergeable_state=unstable) — modifies exactly scheduler_mixin.py. Chase reviewers / green up CI; do NOT open a duplicate. (Found via search/issues?q=repo:sgl-project/sglang+is:pr+is:open+extract_transfer_fields.)

Suggested Triage Steps

• Verify #31325 resolves the encoder ValueError: check out its head, re-run the 3 test_disagg_server.py::TestDisaggZImage* tests on AMD MI325 2-GPU, confirm warmup returns 200.
• Investigate why #31325 is unstable (its own CI red) before merge — the fix touches the same file, so its checks must pass.
• Confirm which Req field carries the numpy array reaching _is_default (image latents/embeddings from the encoder) so the guard in _is_default covers ndarray/tensor values generally, not just this one field.

Status

• Persistence: Failing every comparable completed scheduled run for ~7 days (since at least 2026-07-11, oldest run in the 15-run window; never observed passing).
• Scope: 1 job/shard multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 1) in pr-test-amd.yml; all 3 disagg Z-Image tests blocked at setup.
• Blocked work: multimodal-gen disaggregation (Z-Image 1-rank / SP2-denoiser / tracing) CI signal on AMD.
• In-flight fix: ✅ #31325 (open since 2026-07-15, mergeable_state=unstable) — needs reviewers + its own CI green.

stage-c-test-large-8-gpu-amd-mi35x (linux-mi35x-gpu-8, 1) — failed step(s): Run test

Commit Info

• sglang: d7b942552
• aiter: 9127c94a1 (v0.1.17.dev110+g9127c94a1, source: dev image preserved — AITER_COMMIT_OVERRIDE empty, [CI-AITER-CHECK] reports "Dev/patched version detected → skipping rebuild"; image rocm/sgl-dev:v0.5.15.post1-rocm700-mi35x-20260717)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/test_deepseek_r1_mxfp4_8gpu.py	TestDeepseekR1MXFP4.setUpClass	Exception: Server process exited with code -9 (root: RuntimeError: shape '[1, 16, 512]' is invalid for input of size 2048 in unified_attention_with_output, all 8 TP ranks)	Log

The other two files in this shard passed: test_kimi_k25_mxfp4.py ✅, test_qwen3_coder_next_8gpu.py ✅ (Test Summary: 2/3 passed).

Failure Summary

The 8-GPU MI35x server for amd/DeepSeek-R1-MXFP4-Preview crashed during warmup/startup: every TP rank's scheduler hit RuntimeError: shape '[1, 16, 512]' is invalid for input of size 2048 at radix_attention.py:269 (output[:real_num_tokens].view(ret.shape).copy_(ret)) while using the tc_piecewise prefill CUDA-graph backend, reached via the eager fallback (eager_runner.py:203 → _execute_extend:322). The server therefore exited with code -9 and setUpClass raised, so 0 tests ran.

Regression Status

New regression (test is new — added 2026-07-16 in 7d0fd5101 / #31227; had exactly one green run before regressing).

Recent history of test/registered/amd/test_deepseek_r1_mxfp4_8gpu.py in job stage-c-test-large-8-gpu-amd-mi35x (linux-mi35x-gpu-8, 1) (comparable completed scheduled runs, branch=main, event=schedule; test-file-level status):

Date (UTC)	Run	head_sha	Test File Status	Failed Function	Error
Jul 18 12:17	29644059344	d7b942552	❌ Failed	setUpClass	shape '[1, 16, 512]' invalid (warmup)
Jul 18 00:25	29623067212	238b2b2c9c17	❌ Failed	test_a_gsm8k	shape '[-1, 16, 576]' invalid
Jul 17 12:25	29580198489	85ac56c82308	❌ Failed	test_a_gsm8k	shape '[-1, 16, 576]' invalid ← first fail
Jul 17 00:27	29544840981	dc0b3eb68fa9	✅ Passed (3/3)	—	— ← last pass
Jul 16 12:31	29498496614	7d0fd5101	⏭️ Cancelled (job cancelled, not comparable)	—	test first added in this commit

First observed failure: 2026-07-17 12:25 UTC. Last known passing: 2026-07-17 00:27 UTC. Prior to Jul 16 the test did not exist. Deterministic (fails on all 8 ranks every run since first fail), not flaky. Note the symptom shifted from a test_a_gsm8k-time crash ([-1,16,576]) to a setUpClass/warmup crash ([1,16,512]) in the latest run — same code path, see hypotheses.

Failure Cluster

DeepSeek MLA prefill view/shape mismatch in unified_attention_with_output (tc_piecewise PCG, missing MHA-companion swap)

Facts (verified observations)

• Root error verbatim (all 8 TP ranks): RuntimeError: shape '[1, 16, 512]' is invalid for input of size 2048, at radix_attention.py:269, function unified_attention_with_output; followed by Fail when using backend: tc_piecewise for prefill runner. and SIGQUIT received ... one child failed.
• Crash stack passes through model_runner.py:1457 _forward_raw → eager_runner.py:203 execute → eager_runner.py:322 _execute_extend → deepseek_v2.py forward.
• Test-level failure: ERROR: setUpClass (TestDeepseekR1MXFP4) → Exception: Server process exited with code -9; Ran 0 tests in 559.695s.
• Server launch: --tp 8 --chunked-prefill-size 131072 --enforce-piecewise-cuda-graph --piecewise-cuda-graph-compiler eager --piecewise-cuda-graph-max-tokens 8192 (i.e. tc_piecewise PCG is on).
• Regression window dc0b3eb68fa9..85ac56c82308 = 34 commits. Only two touch the PCG/attention code: 24a8944e1 (#31391, modifies the exact crash function) and 681c22357 (#31439, attention_backend_setup only).
• 24a8944e1 replaced the HIP-gated _pcg_mha_companion attribute swap with a context-threaded context.mha_companion_layers list in unified_attention_with_output (verified via git show).
• aiter unchanged (dev image preserved, no override) — CUDA/NVIDIA lanes unaffected; this is AMD MI35x-specific.

Hypothesised Causes (with confidence)

• [HIGH] 24a8944e1 (#31391, "enable Kimi multimodal breakable prefill cuda graph replay") replaced the per-layer _pcg_mha_companion attribute (swapping attn_mqa→attn_mha so the backend sees correct MHA head/dim on the save_kv_cache=False path) with a context-threaded mha_companion_layers list, but missed the HIP PCG eager-fallback call site in eager_runner.py. On that path context.mha_companion_layers is None, so no swap occurs; the MHA q is reshaped with attn_mqa's dims → the .view() fails.
  • Supporting: (a) the only window commit modifying the crash function; (b) crash stack goes through eager_runner.py (the un-fixed call site); (c) error is a head/dim mismatch (512=kv_lora_rank, 576=512+64 rope, 192=128+64 MHA q); (d) AMD/HIP-only, aiter unchanged; (e) in-flight fix #31675 cites this exact job and confirms this root cause.
  • Disconfirming: current head populates mha_companion_layers unconditionally for DeepSeek in layer_setup.py and passes it at the three prefill_cuda_graph_runner.py call sites — so the bug survives only on the eager-fallback path; not reproduced locally (no MI35x).
  • How to verify: apply #31675 (one-line mha_companion_layers=... in eager_runner.py) on a branch and re-run test_deepseek_r1_mxfp4_8gpu.py on MI35x TP8.
• [LOW] 6c6175fab (#31487, "avoid excessive prefill CUDA graph padding") — landed after the first fail; likely explains only the symptom shift (crash moved from gsm8k-time [-1,16,576] to warmup-time [1,16,512] via changed padding/token counts), not the regression itself.
  • Supporting: touches prefill PCG padding; timing matches the shape/shift change.
  • Disconfirming: test already failed before this commit landed, so it cannot be the root regressor.
  • How to verify: subsumed by #31675 verification — if that fixes it, this is irrelevant to root cause.

In-flight Fix Check

• In-flight fix: ✅ #31675 "[AMD] Fix DeepSeek MLA prefill shape mismatch on HIP eager fallback (missing mha_companion_layers)" (open since 2026-07-18, by yctseng0211) — one-line change to eager_runner.py; its description links this exact job and root-causes it to #31391. Chase reviewers; do NOT open a duplicate. (Search: search/issues?q=...is:pr+is:open+piecewise+mla+shape and ...+mha_companion.)

Suggested Triage Steps

• Verify/merge in-flight fix #31675: confirm it adds mha_companion_layers=model_runner.mha_companion_layers to the set_tc_piecewise_forward_context(...) call in the HIP eager-fallback path of eager_runner.py, then re-run test_deepseek_r1_mxfp4_8gpu.py on MI35x TP8.
• If #31675 does not fully clear it, bisect dc0b3eb68fa9..85ac56c82308 focusing on 24a8944e1; optionally revert it on a branch and re-run.
• If the warmup vs gsm8k symptom persists to differ, check 6c6175fab (#31487) padding interaction as a secondary factor.

Status

• Persistence: Failing since 2026-07-17 12:25 UTC — 3 consecutive completed scheduled runs; deterministic on all 8 TP ranks. Test is new (added 2026-07-16), had one green run (Jul 17 00:27).
• Scope: stage-c-test-large-8-gpu-amd-mi35x (linux-mi35x-gpu-8, 1) in pr-test-amd.yml; AMD MI35x TP8 only (aiter unchanged, NVIDIA unaffected). Blocks DeepSeek-R1-MXFP4 8-GPU piecewise-CUDA-graph CI signal on MI35x.
• In-flight fix: ✅ #31675 (open since 2026-07-18) — awaiting merge.

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-07-19 17:01 UTC)

[amd-bot]

pr-test-amd.yml — Per-Job Analysis (continued 3/7)

multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 0) — failed step(s): Run diffusion server tests (2-GPU)

Commit Info

• sglang: 238b2b2c9c17647cad98ff2203d0fdbaf2e0dce2
• aiter: 9127c94a18e4398e1eba91f6639e910f0994ad02 (v0.1.17.dev110+g9127c94a1, source: dev image preserved, no override — AITER_COMMIT_OVERRIDE empty)

Failed Tests

Test File	Test Function	Error	Log
python/sglang/multimodal_gen/test/server/test_server_2_gpu.py	TestDiffusionServerTwoGpu::test_diffusion_generation[flux2_modelopt_fp8_tp2_t2i]	RuntimeError: CUDA error: HIPBLAS_STATUS_NOT_SUPPORTED when calling HIPBLAS_STATUS_NOT_SUPPORTED (at torch._scaled_mm, fp8_utils.py:1878)	Log
python/sglang/multimodal_gen/test/server/test_server_2_gpu.py	TestDiffusionServerTwoGpu::test_diffusion_generation[joy_echo_t2v_2gpu]	ValueError: Could not find model at jdopensource/JoyAI-Echo and failed to download from HF Hub: … CAS Client Error: Format error: I/O error: error decoding response body (server-startup ERROR)	Log

Failure Summary

The 2-GPU diffusion server suite (shard 0) failed with 1 failed + 1 error out of 24 executed cases (22 passed, 1 skipped). flux2_modelopt_fp8_tp2_t2i failed at inference time when the FLUX.2 modelopt-FP8 to_q linear hit torch._scaled_mm, which hipBLAS rejected with HIPBLAS_STATUS_NOT_SUPPORTED on the MI325 (gfx942). joy_echo_t2v_2gpu errored during server startup because the model jdopensource/JoyAI-Echo could not be downloaded from HF Hub (Xet/CAS reconstruction I/O error after 5 retries).

Regression Status

Known recurring failure for both tests — no green run for either in the queryable window (both consistently land in 2-GPU shard 0, partition 1/3).

Recent history of test_server_2_gpu.py cases flux2_modelopt_fp8_tp2_t2i (FAILED) / joy_echo_t2v_2gpu (ERROR) in job multimodal-gen-test-2-gpu-amd (…, 0) (most recent comparable completed event=schedule, branch=main runs):

Date (UTC)	Run	Job (shard 0)	flux2 Status	joy_echo Status	Note
Jul 18 00:25	29623067212	shard 0	❌ FAILED	❌ ERROR	current run (verified)
Jul 18 12:17	29644059344	shard 0	❌ (inferred)	❌ (inferred)	shard-0 job failed; log expired (BlobNotFound); tests absent from shards 1/2
Jul 17 12:25	29580198489	shard 0	❌ (inferred)	❌ (inferred)	shard-0 job failed; log expired
Jul 17 00:27	29544840981	shard 0	❌ FAILED	❌ ERROR	verified (same signatures)
Jul 16 12:31	29498496614	shard 0	❌ FAILED	❌ ERROR	verified (same signatures)
Jul 16 00:25	29461421122	—	⏭️ cancelled	⏭️ cancelled	run cancelled — excluded
Jul 15 12:28	29415344552	shard 0	❌ (inferred)	❌ (inferred)	shard-0 job failed; log expired

• flux2: HIPBLAS_STATUS_NOT_SUPPORTED at torch._scaled_mm verified byte-identical across runs 29623067212, 29544840981, 29498496614. First observed failure ≥ Jul 16; no passing run in window → treat as Never-passed on AMD gfx942 (corroborated by in-flight fix #28889 open since Jun 22).
• joy_echo: HF Hub download failure verified in the same three runs (5 download attempts exhausted each time). Persistent, but the underlying error is a transient network/CAS I/O failure, so this behaves as a flaky-but-recurring infra failure.
• The head commit 238b2b2c ("Remove QServe and FBGEMM FP8 quantization") is not the cause — both failures predate it (present in the Jul 16 run at head 7d0fd510).

Failure Cluster

• flux2: "hipBLAS FP8 rowwise _scaled_mm NOT_SUPPORTED on gfx942 (FLUX.2 modelopt-FP8)"
• joy_echo: "HF Hub / Xet model-download failure (JoyAI-Echo)"

Facts (verified observations)

• flux2 stack trace (verbatim tail): linear.py:426 → modelopt_quant.py:432 apply_fp8_linear → fp8_utils.py:1878 output = torch._scaled_mm(...) → RuntimeError: CUDA error: HIPBLAS_STATUS_NOT_SUPPORTED when calling HIPBLAS_STATUS_NOT_SUPPORTED.
• fp8_utils.py:1864-1871 is the ROCm rowwise-scaling torch._scaled_mm branch (hipBLASLt path, comment: "For now validated on ROCm platform … ROCm 6.3 / torch 2.7+").
• joy_echo stack trace (verbatim tail): hf_diffusers_utils.py:896 maybe_download_model → snapshot_download → huggingface_hub file_download.py:563 xet_get → RuntimeError: … CAS Client Error: Format error: I/O error: error decoding response body, re-raised as ValueError: Could not find model at jdopensource/JoyAI-Echo …. Scheduler process then died (Rank 0 scheduler is dead, Exit code: 1, parent got EOFError).
• Both failures reproduce in 2-GPU shard 0 only; shards 1 and 2 passed (1 passed each) and do not contain these two cases.
• aiter is a preserved dev build (v0.1.17.dev110+g9127c94a1); no AITER_COMMIT_OVERRIDE (this is pr-test-amd.yml, not a scout run).

Hypothesised Causes (with confidence, with disconfirming evidence)

• [HIGH] flux2: hipBLAS(Lt) does not support the FP8 rowwise-scaled torch._scaled_mm shape/dtype emitted by FLUX.2 modelopt-FP8 on gfx942 (MI325).
  • Supporting: the failing frame is exactly torch._scaled_mm in the ROCm rowwise branch (fp8_utils.py:1878); error is a hipBLAS not-supported status; open PR #28889 "[AMD] Use native e4m3fnuz fp8 GEMM for FLUX.2 modelopt-FP8 on gfx942" targets this exact path; failure is identical across all verified runs.
  • Disconfirming: (none found) — no green run exists in the window to contradict it.
  • How to verify: check out PR #28889 branch and re-run test_diffusion_generation[flux2_modelopt_fp8_tp2_t2i] on MI325.
• [MEDIUM] joy_echo: infrastructure — HF Hub / Xet CAS backend returned a corrupt/undecodable response while fetching jdopensource/JoyAI-Echo, exhausting all 5 retries.
  • Supporting: error originates entirely inside huggingface_hub xet_get (CAS Client Error … error decoding response body); no sglang code frame in the download failure; recurs but with retry exhaustion consistent with network/CDN flakiness.
  • Disconfirming: it has failed in every verified run (Jul 16–18), which is more persistent than typical transient flakes — could instead be a gated/renamed/removed repo or a runner-side Xet config issue rather than pure network noise.
  • How to verify: from the MI325 runner, run huggingface-cli download jdopensource/JoyAI-Echo (and retry with HF_HUB_DISABLE_XET=1) to distinguish Xet-CDN flakiness from a repo-access problem.

In-flight Fix Check

• flux2: In-flight fix: ✅ #28889 "[AMD] Use native e4m3fnuz fp8 GEMM for FLUX.2 modelopt-FP8 on gfx942" (open since 2026-06-22, mergeable_state: blocked) — chase reviewers / unblock; do NOT open a duplicate. (search: is:pr is:open HIPBLAS_STATUS_NOT_SUPPORTED and flux2 modelopt fp8)
• joy_echo: In-flight fix: ❌ none found (search: is:pr is:open JoyAI-Echo → 0 results) — needs investigation of the JoyAI-Echo download path / runner Xet config.

Suggested Triage Steps

• flux2: verify PR #28889 resolves the failure — rebase it onto current main, unblock its CI, and re-run test_diffusion_generation[flux2_modelopt_fp8_tp2_t2i] on MI325 (gfx942). If the fix is not close to merge, consider temporarily skipping this case on AMD gfx942 until #28889 lands.
• joy_echo: reproduce huggingface-cli download jdopensource/JoyAI-Echo on the runner; if it's Xet-CDN flakiness, retry with HF_HUB_DISABLE_XET=1 or pre-cache the model on the runner; if the repo is gated/moved, update the model reference in the test config. Since it's a setup-time download error, it should not block flux2's classification.

Status

• Persistence: both failing in 2-GPU shard 0 of pr-test-amd.yml scheduled runs continuously since at least Jul 16 (every completed run in the window; Jul 16 00:25 run was cancelled and excluded).
• Scope: 1 job (multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 0)) in pr-test-amd.yml; shards 1 and 2 pass. flux2 is a real AMD FP8 kernel-support gap; joy_echo is an HF-download/infra failure.
• Blocked work: FLUX.2 modelopt-FP8 and JoyAI-Echo (JoyAI-Echo T2V) diffusion coverage on AMD MI325.
• In-flight fix: flux2 → ✅ #28889 (blocked, open since 2026-06-22); joy_echo → ❌ none.

multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 3) — failed step(s): Run diffusion server tests (1-GPU)

Commit Info

• sglang: 238b2b2c9
• aiter: v0.1.17.dev110+g9127c94a1 (source: dev/patched CI image preserved; Dockerfile default 9127c94a1, AITER_COMMIT_OVERRIDE empty — verified from [CI-AITER-CHECK] markers, log lines 4258-4260)

Failed Tests

Job: multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 3) — pytest result 5 failed, 1 passed, 4 errors (standalone file test_update_weights_from_disk.py).

Test File	Test Function	Error	Log
python/sglang/multimodal_gen/test/single_test_file/test_update_weights_from_disk.py	TestUpdateWeightsFromDisk::test_update_weights_from_disk_default[Qwen-Image]	assert 500 == 200 (server RuntimeError: Inplace update to inference tensor outside InferenceMode is not allowed)	Log
same	TestUpdateWeightsFromDisk::test_update_weights_specific_modules[Qwen-Image]	AssertionError: Update failed: {...'Inplace update to inference tensor...'} → assert 500 == 200	Log
same	TestUpdateWeightsFromDisk::test_corrupted_weights_rollback[Qwen-Image]	assert (500 == 200)	Log
same	TestUpdateWeightsFromDiskWithOffload::test_update_weights_with_offload_enabled[FLUX.2-klein-base-4B]	AssertionError: Expected 200, got 500	Log
same	TestUpdateWeightsFromDiskWithOffload::test_update_weights_with_offload_enabled[Qwen-Image]	AssertionError: Expected 200, got 500	Log
same	TestUpdateWeightsFromDisk::test_update_weights_from_disk_default[FLUX.2-klein-base-4B] (ERROR at setup)	AssertionError: No weights dir for transformer in /tmp/sglang_perturbed_vae_* (fixture diffusion_server_no_offload, line 410 → _compute_checksum_from_disk line 220)	Log
same	TestUpdateWeightsFromDisk::test_update_weights_specific_modules[FLUX.2-klein-base-4B] (ERROR at setup)	same No weights dir for transformer	Log
same	TestUpdateWeightsFromDisk::test_update_weights_rejects_invalid_requests[FLUX.2-klein-base-4B] (ERROR at setup)	same No weights dir for transformer	Log
same	TestUpdateWeightsFromDisk::test_corrupted_weights_rollback[FLUX.2-klein-base-4B] (ERROR at setup)	same No weights dir for transformer	Log

Failure Summary

The standalone diffusion test file test_update_weights_from_disk.py reported 5 failed, 1 passed, 4 errors. Two distinct symptoms: (1) all Qwen-Image update tests plus both offload-enabled tests get HTTP 500 from the server because the weight update raises RuntimeError: Inplace update to inference tensor outside InferenceMode is not allowed; (2) the four FLUX.2-klein-base-4B non-offload tests ERROR during fixture setup with AssertionError: No weights dir for transformer when the fixture builds a perturbed-VAE model dir.

Regression Status

Never-passed on AMD — the test file was added by #31227 (7d0fd5101) and has failed with the identical 5 failed / 1 passed / 4 errors signature in every scheduled run since it first appeared in CI. No green run exists in the queryable window.

Recent history of test_update_weights_from_disk.py in job multimodal-gen-test-1-gpu-amd (…, 3) (all completed, event=schedule, branch=main; test file did not exist before 2026-07-16 12:31):

Date	Run	Job	Test File Status	Failed Functions	Error
Jul 18 12:17	29644059344	shard 3	❌ Failed	5F/4E (same set)	inference-tensor 500 + No weights dir
Jul 18 00:25	29623067212	shard 3	❌ Failed	5F/4E (this run)	inference-tensor 500 + No weights dir
Jul 17 12:25	29580198489	shard 3	❌ Failed	5F/4E (same set)	same
Jul 17 00:27	29544840981	shard 3	❌ Failed	5F/4E (same set)	same
Jul 16 12:31	29498496614	shard 3	❌ Failed (first appearance, head=7d0fd5101)	5F/4E (same set)	same
≤ Jul 16 00:25	—	—	N/A	—	test file not yet in repo

Failure Cluster

"Diffusion update_weights_from_disk: HTTP 500 from in-place inference-tensor copy (+ FLUX perturbed-weights fixture setup error)"

Facts (verified observations)

• Pytest tail: ======== 5 failed, 1 passed, 11 warnings, 4 errors in 869.54s ======== (log line 12953).
• Server-side dominant error (Qwen-Image + offload): RuntimeError: Inplace update to inference tensor outside InferenceMode is not allowed. You can make a clone to get a normal tensor before doing inplace update. (log lines 5947, 7139, etc.), surfaced to the client as assert 500 == 200 (lines 12814-12815, 12836).
• Stack trace of the 500 path: weights_updater.py:419 _apply_weights → :188 _load_weights_into_module → :280 load_weights_into_model → linear.py:410 weight_loader, where the loader executes param_data.copy_(loaded_weight) — an in-place write on param.data (verified in source python/sglang/multimodal_gen/runtime/layers/linear.py:409-410).
• Second, distinct symptom (FLUX.2-klein-base-4B non-offload, 4 ERRORS): fixture diffusion_server_no_offload (test_update_weights_from_disk.py:410) → _compute_checksum_from_disk (:220) → AssertionError: No weights dir for transformer in /tmp/sglang_perturbed_vae_* (log lines 12759-12835). These tests never reach the weight-update call.
• Reproducibility: identical 5F/1P/4E across all 5 scheduled runs since the test file was introduced (2026-07-16 → 2026-07-18); only shard 3 of this job carries it.
• aiter unchanged across all these runs (dev image v0.1.17.dev110+g9127c94a1, no override) — rules out an aiter delta.
• Non-fatal noise seen alongside (not the reported failures): AssertionError: assert param_data.shape == loaded_weight.shape (linear.py:410, lines 5503/6695) and one HF HFValidationError/ValueError: Could not find model at /sgl-data/hf-cache/... snapshot-path error (lines 12705-12753).

Hypothesised Causes (with confidence, with disconfirming evidence)

• [HIGH] The dominant 500 failures are caused by the weight-update path performing an in-place param_data.copy_() on tensors that were allocated as inference tensors (created under torch.inference_mode() at model load), which PyTorch forbids to mutate outside inference mode.
  • Supporting: the error string names InferenceMode directly; linear.py:410 is an unconditional in-place copy_; an open PR (#31263) authored 2026-07-15 does exactly run weight update under torch.inference_mode() and touches only weights_updater.py — three converging chains.
  • Disconfirming: (none found) for the Qwen-Image/offload rows. Note this hypothesis does NOT explain the 4 FLUX non-offload setup ERRORS, which fail earlier in the fixture.
  • How to verify: apply #31263 on a branch and re-run test_update_weights_from_disk.py on AMD MI325; confirm the Qwen-Image + offload tests flip to pass.
• [LOW] The FLUX.2-klein-base-4B setup ERRORS (No weights dir for transformer) are a separate test-fixture/environment problem where the perturbed-VAE model dir is built without a transformer weights subdir on AMD.
  • Supporting: distinct stack (fixture setup, not the update RPC); only affects FLUX non-offload variant.
  • Disconfirming: no code evidence yet that this is AMD-specific vs. a general fixture bug; #31263 does not touch the fixture, so it likely will NOT fix these four.
  • How to verify: inspect test_update_weights_from_disk.py fixture _perturb/checksum helpers; check whether the FLUX transformer weights are written under the expected /tmp/sglang_perturbed_vae_*/transformer path on AMD.

In-flight Fix Check

• In-flight fix: ✅ #31263 [diffusion] post_training: run weight update under torch.inference_mode() (open, non-draft, created 2026-07-15, last updated 2026-07-16; +31/-16 in weights_updater.py) — directly targets the Inplace update to inference tensor error. Chase reviewers; do NOT open a duplicate. (Search: search/issues?q=repo:sgl-project/sglang+is:pr+is:open+inference+tensor+InferenceMode.)
• No open PR found matching the FLUX No weights dir for transformer fixture setup error.

Suggested Triage Steps

• Verify #31263 resolves the 5 HTTP-500 rows: check it out on a branch, run test_update_weights_from_disk.py on AMD MI325 (container rocm/sgl-dev:v0.5.15.post1-rocm700-mi30x-20260717), confirm Qwen-Image + offload tests pass.
• Separately investigate the 4 FLUX.2-klein-base-4B setup ERRORS — inspect the fixture that writes the perturbed VAE/transformer weights; #31263 will likely NOT clear these, so they need their own fix.
• Since the test entered CI already-red (introduced by #31227), consider whether it should be marked nightly/skipped on AMD until #31263 and the fixture fix land, to restore signal on this shard.

Status

• Persistence: Never-passed on AMD — failing since first CI appearance 2026-07-16 12:31 UTC (run 29498496614); identical signature every completed scheduled run since (5 runs).
• Scope: 1 job / 1 shard (multimodal-gen-test-1-gpu-amd shard 3) in pr-test-amd.yml; 9 of 10 test items in test_update_weights_from_disk.py.
• Blocked work: diffusion update_weights_from_disk CI signal on AMD.
• In-flight fix: ✅ #31263 (open) covers the 5 inference-tensor 500 rows; ❌ none for the 4 FLUX fixture-setup ERRORS.

multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 1) — failed step(s): N/A

Commit Info

• sglang: 238b2b2c9
• aiter: 9127c94a1 (v0.1.17.dev110+g9127c94a1, source: dev image, matches Dockerfile default — AITER_COMMIT_OVERRIDE empty)

Failed Tests

This job hung: step 9 Run diffusion server tests (1-GPU) never concluded (status: in_progress, conclusion: None) and the job was killed at ~72 min. The current-run log returns BlobNotFound (never finalized because the step never completed), so the exact failing case cannot be read from this run's log. The failure signature was recovered from the most recent shard-1 run whose step did finalize (job 85320514218, 2026-07-06), which had the identical hang-to-timeout pattern:

Test File	Test Function	Error	Log
test/server/test_server_1_gpu.py	TestDiffusionServerOneGpu::test_diffusion_generation[sana_wm_ti2v]	TimeoutError: Server not ready within 1200.0s (server startup)	Log
test/server/test_server_1_gpu.py	TestDiffusionServerOneGpu::test_diffusion_generation[ltx_2_3_hq_pipeline]	TimeoutError: Server not ready within 1200.0s (server startup)	Log
test/server/test_server_1_gpu.py	TestDiffusionServerOneGpu::test_diffusion_generation[lingbot_world_realtime_plastic_beach]	TimeoutError: Server not ready within 1200.0s (server startup)	Log

The affected case ids (sana_wm_ti2v, ltx_2_3_hq_pipeline, lingbot_world_realtime_plastic_beach) still exist in python/sglang/multimodal_gen/test/server/gpu_cases.py:384 and are auto-partitioned by deterministic LPT, so they land in partition 1 across runs. [MEDIUM] confidence that these are the current-run culprits (log-unrecoverable for this run; inferred from the identical duration + step-9-never-concludes signature).

Failure Summary

Shard 1 (partition 1) of the 1-gpu diffusion server suite hangs: multiple diffusion models fail to reach "server ready" within the 1200 s per-case deadline, each burning ~20 min, which accumulates past the step's 90-min limit. In this run the runner was lost / killed at ~72 min before the 90-min step timeout could even fire, so no log was uploaded. Sibling shard 0 hung identically (91 min → 90-min step timeout), shard 2 passed, and shard 3 (the standalone test_update_weights_from_disk.py) had an unrelated real failure. This is a standing failure of the parametrized 1-GPU diffusion cases, unrelated to HEAD commit 238b2b2c9 (QServe/FBGEMM FP8 removal — no diffusion code touched).

Regression Status

Never-passed (no green run found) in the queryable window — and specifically a hang in every run since ~2026-07-11.

Recent history of test/server/test_server_1_gpu.py (TestDiffusionServerOneGpu::test_diffusion_generation) in job multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 1) (most recent completed event=schedule, branch=main runs):

Date (UTC)	Run	Test File Status	Signature	Dur
Jul 18 00:25	29623067212	❌ Failed (hang)	step 9 never concluded; log BlobNotFound	72 min
Jul 17 12:25	29580198489	❌ Failed (hang)	step 9 in_progress/None	75 min
Jul 17 00:27	29544840981	❌ Failed	71 min	71 min
Jul 16 12:31	29498496614	❌ Failed (hang)	step 9 in_progress/None	71 min
Jul 15 12:28	29415344552	❌ Failed (hang)	step 9 in_progress/None	69 min
Jul 15 00:22	29378873430	❌ Failed (hang)	step 9 in_progress/None	68 min
Jul 14 12:26	29332511138	❌ Failed (hang)	step 9 in_progress/None	71 min
Jul 14 00:24	29295847781	❌ Failed (hang)	step 9 in_progress/None	86 min
Jul 13 12:48	29251251853	❌ Failed (hang)	step 9 in_progress/None	63 min
Jul 13 00:28	29215394044	❌ Failed (hang)	step 9 in_progress/None	66 min
Jul 12 12:18	29192376210	❌ Failed (hang)	step 9 in_progress/None	60 min
Jul 12 00:27	29173750641	❌ Failed (hang)	step 9 in_progress/None	64 min
Jul 11 12:17	29152390308	❌ Failed	64 min	64 min
Jul 11 00:26	29132618067	❌ Failed	68 min	68 min
Jul 06 01:45	28760739527	❌ Failed (completed)	Server not ready within 1200.0s → step timed out after 90 min	98 min

No green run for this shard exists back to at least 2026-07-01 (lookback: pages 1–2 of completed scheduled runs, ~2026-07-01 → 2026-07-18). Earliest recoverable server-timeout evidence is 2026-07-06; from ~2026-07-11 the step stops finalizing at all (pure hang).

Failure Cluster

Diffusion server startup timeout in 1-GPU suite (Server not ready within 1200.0s) → 90-min step hang / runner loss.

Facts (verified observations, no interpretation)

• Step 9 Run diffusion server tests (1-GPU) = status: in_progress, conclusion: None; started 2026-07-18T01:33:40Z, job marked failure at 02:37:15Z (~63.5 min into the step, before the 90-min step timeout). Steps 10 (if: always()) and 20 (Post Checkout, always runs) are pending — the job terminated abnormally (runner lost / killed), not a clean test exit.
• Current-run log for job 88021852955 returns BlobNotFound (never finalized). 0 artifacts on the run.
• Same run, sibling shards (verified via jobs API): shard 0 88021852991 failed, ran 91 min, step 9 in_progress/None (hit 90-min step timeout); shard 2 88021852976 passed (14 min); shard 3 88021852974 failed with a different, finalized failure in the standalone test_update_weights_from_disk.py (5 failed, 4 errors in 869 s).
• Partition layout (verified from run_suite.py + step invocation --total-partitions 4, 1 standalone file): partitions 0/1/2 run parametrized test_server_1_gpu.py cases via LPT; partition 3 runs the standalone weights test. So shards 0 and 1 both hang on parametrized 1-GPU server cases.
• Recovered signature (finalized log, job 85320514218, 2026-07-06): E TimeoutError: Server not ready within 1200.0s. at setup of test_diffusion_generation[sana_wm_ti2v], [ltx_2_3_hq_pipeline], [lingbot_world_realtime_plastic_beach]; final line ##[error]The action 'Run diffusion server tests (1-GPU)' has timed out after 90 minutes.
• aiter for this run: v0.1.17.dev110+g9127c94a1 (dev image preserved), from [CI-AITER-CHECK] markers in sibling shard-3 log; AITER_COMMIT_OVERRIDE empty — not a scout run.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [MEDIUM] Specific diffusion models (sana_wm_ti2v, ltx_2_3_hq_pipeline, lingbot_world_realtime_plastic_beach) fail to start their server on MI325 within the 1200 s deadline; ≥3 such 20-min blocks push partition 1 past the 90-min step limit and the shard hangs.
  • Supporting: recovered 2026-07-06 shard-1 log shows exactly these three cases erroring with Server not ready within 1200.0s and the step timing out at 90 min; identical duration/step-never-concludes signature in every run since; shard 2 (different case subset) passes, so the hang is case-specific, not whole-runner infra.
  • Disconfirming: the current run's own log is unrecoverable, so the exact current-run case cannot be confirmed; the 2026-07-06 case set could have shifted if case est-times/config changed since. The current run was killed at 63 min (before 90-min timeout), which is more consistent with runner-loss than a clean per-case timeout accumulation — so an intermittent MI325 runner/GPU health component cannot be excluded.
  • How to verify: re-run partition 1 alone on an MI325 container with per-case server logs captured (SGLANG_DIFFUSION_ARTIFACT_DIR), and launch each of the three servers manually (sglang serve --model-path <sana/ltx/lingbot> on 1×MI325) to see whether startup completes < 1200 s.
• [LOW] HEAD commit 238b2b2c9 (Remove QServe/FBGEMM FP8) caused it.
  • Supporting: it is this run's HEAD.
  • Disconfirming: the shard has never passed back to at least 2026-07-01 across dozens of unrelated commits; the commit touches quantization, not diffusion server startup. Effectively ruled out as the cause.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for diffusion+server+not+ready+timeout, multimodal_gen+server+startup, sana+ltx+diffusion+amd, 1-gpu+diffusion+timeout; all hits are unrelated features/quant PRs, none addressing 1-GPU diffusion server startup timeouts on AMD.

Suggested Triage Steps

• Reproduce partition 1 in isolation on 1×MI325 (run_suite.py --suite 1-gpu --partition-id 1 --total-partitions 4 -k "not flux_2") with server logs persisted, to confirm which case(s) hang in the current code (the current-run log is gone).
• Manually launch each candidate server (sana_wm_ti2v, ltx_2_3_hq_pipeline, lingbot_world_realtime_plastic_beach) on MI325 and inspect startup — determine whether it's a model-load/compile hang (MIOpen/aiter kernel build) vs a real deadlock.
• If a specific case is confirmed broken on AMD, consider quarantining/skipping it on the AMD 1-GPU suite (e.g. via -k exclusion like the existing not flux_2) until separately fixed, so partition 1 stops eating a 90-min timeout every run.
• Independently, investigate the MI325 runner-loss angle: this run was killed at ~63 min (before the 90-min timeout) with no log upload — check runner/GPU health (rocm-smi, dmesg for HIP faults) on linux-mi325-1gpu-sglang for the interval.

Status

• Persistence: Failing every completed scheduled run for ≥17 days (no green run since at least 2026-07-01); a pure step-9 hang since ~2026-07-11.
• Scope: Partitions 0 and 1 of multimodal-gen-test-1-gpu-amd in pr-test-amd.yml (parametrized 1-GPU diffusion server cases); partition 2 passes, partition 3 fails on a separate cluster. Blocks 1-GPU diffusion server CI signal on AMD.
• Blocked work: Reliable AMD 1-GPU diffusion server-generation coverage; the recurring 60–90 min hang also consumes MI325 runner time twice per scheduled run.
• In-flight fix: ❌ none found in open PRs.

multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 0) — failed step(s): N/A

Commit Info

• sglang: 238b2b2c9
• aiter: 9127c94a1 (v0.1.17.dev110+g9127c94a1, source: dev image preserved — AITER_COMMIT_OVERRIDE empty, per sibling part-3 [CI-AITER-CHECK] markers; part-0's own log is unavailable)

Failed Tests

Test File	Test Function	Error	Log
N/A (partition 0 hang)	N/A	Step 9 Run diffusion server tests (1-GPU) ran ~79 min then the runner terminated; no test result was recorded	Job

The specific failing test for part 0 cannot be identified: the job log blob returns BlobNotFound, step 9 is frozen at in_progress even though the job concluded failure, and step 10 (Upload diffusion failure artifacts) never ran. The step log was never flushed because the runner died mid-step.

Cross-partition context for the same job in this run (matrix part):

Part	Result	Duration	Signature
0 (this job)	❌ failure	~91m (step 9 ~79m)	hang, no log
1	❌ failure	~72m	hang, no log (BlobNotFound)
2	✅ success	~14m	—
3	❌ failure	~24m	real test failure: test_update_weights_from_disk.py — FLUX.2-klein-base-4B ERRORs (transformer loaded 0.0 GB), Qwen-Image FAILED

Failure Summary

The multimodal-gen-test-1-gpu-amd (part 0) job failed because step 9 (Run diffusion server tests (1-GPU), --partition-id 0 --total-partitions 4) hung for ~79 minutes and the runner was terminated before the step could finish or flush its log (below the 90-min step timeout and the 360-min job default, so this is a runner/GPU-level termination, not a clean timeout). No log is recoverable for this partition, so the exact hanging diffusion test cannot be pinned. Sibling partitions in the same run show the 1-GPU diffusion suite is broadly unhealthy: part 1 hung identically, part 3 hit real model-loading failures, and only part 2 passed.

Regression Status

Known recurring failure (not introduced by 238b2b2c9). Part 0 has failed in every completed event=schedule run on main in the queryable window; the failure mode became a hang (step 9 stuck in_progress, 65-85 min, no log) starting 2026-07-11.

Recent history of multimodal-gen-test-1-gpu-amd (part 0) (most recent comparable completed scheduled runs: branch=main, event=schedule). Test-file status cannot be resolved for the hang rows (no logs); status is job-level with the step-9 signature noted:

Date (UTC)	Run	Result	Dur	Step 9 signature
Jul 18 12:17	29644059344	❌	74m	hang (in_progress, no log)
Jul 18 00:25	29623067212	❌	91m	hang (this job)
Jul 17 12:25	29580198489	❌	85m	hang
Jul 17 00:27	29544840981	❌	73m	hang
Jul 16 12:31	29498496614	❌	75m	hang
Jul 16 00:25	29461421122	❌	8m	step 9 skipped (different/upstream failure)
Jul 15 12:28	29415344552	❌	10m	step 9 not reached (different/upstream failure)
Jul 15 00:22	29378873430	❌	79m	hang
Jul 14 12:26	29332511138	❌	76m	hang
Jul 14 00:24	29295847781	❌	68m	hang
Jul 13 12:48	29251251853	❌	66m	hang
Jul 13 00:28	29215394044	❌	67m	hang
Jul 12 12:18	29192376210	❌	70m	hang
Jul 12 00:27	29173750641	❌	67m	hang
Jul 11 00:26	29132618067	❌	65m	hang (first hang)
Jul 10 12:44	29093581297	✅	51m	last pass (step 9 completed/success), sha 94de28764c88

First hang observed: Jul 11 00:26 UTC. Last known passing run for part 0: Jul 10 12:44 UTC (94de28764c88). (Note: earlier Jul 9-10 runs had completed step-9 failures with logs; the pure-hang mode is the Jul 11+ signature.)

Failure Cluster

Diffusion 1-GPU server test hang → runner termination (no log flushed)

Facts (verified observations)

• curl .../jobs/88021852991/logs → BlobNotFound; step 9 status is in_progress/completed_at=None while the job is completed/failure (verified via job API).
• Step 9 started 2026-07-18T01:36:25Z; job completed 2026-07-18T02:55:33Z → ~79 min in step 9. Step 9 timeout-minutes: 90; job has no job-level timeout (GitHub default 360m). So neither timeout was hit — the runner was terminated.
• Step 10 Upload diffusion failure artifacts is pending (never ran).
• Sibling part 1 (88021852955) shows the identical signature: ~72m, log BlobNotFound.
• Sibling part 3 (88021852974) log (recovered from the run archive) shows real failures in sglang/multimodal_gen/test/single_test_file/test_update_weights_from_disk.py: FLUX.2-klein-base-4B variants ERROR with memory_usages_gb={..., 'transformer': 0.0, ...} (transformer weights not loaded), Qwen-Image variants FAILED. Note the step's -k "not flux_2" filter does not exclude the FLUX.2-klein param IDs.
• aiter in the CI image: v0.1.17.dev110+g9127c94a1 (dev build preserved, no override) — unchanged across passing (Jul 10) and failing runs, so not a regression variable here.
• Part 0 has failed in 15/15 completed scheduled runs since Jul 11; last green Jul 10 12:44.

Hypothesised Causes (with confidence)

• [LOW] A diffusion model in partition 0's case set hangs during server startup / warmup / inference on MI325, making the runner unresponsive so GitHub terminates the job before the 90-min step timeout.
  • Supporting: consistent 65-85 min run-then-die with no log across all recent runs; part 1 hangs identically; warmup for one model alone took 150s in part 3; part 3 shows real diffusion server/model-load failures in the same suite.
  • Disconfirming: no part-0 log exists to confirm which model/stage hangs; part 2 passes, so it is not a whole-suite/image break; the specific failing test is unverified.
  • How to verify: re-run part 0 with a shorter per-test/server watchdog and artifact upload on hang; or run run_suite.py --suite 1-gpu --partition-id 0 --total-partitions 4 -k "not flux_2" locally on an MI325 container and watch which case stalls.
• [LOW] A commit in 94de28764c88..238b2b2c9c17 (73 commits) regressed the diffusion path; only a few touch multimodal_gen — e.g. d38903933 (#31233, Qwen/Wan multi-output conditioning) and 7d0fd5101 (#31227).
  • Supporting: the hang mode begins Jul 11, shortly after the Jul 10 last-pass; these commits touch diffusion runtime.
  • Disconfirming: no part-0 stack/log ties the hang to any of these; the current head 238b2b2c9 itself only removes quantization code unrelated to diffusion hangs; the regression may equally be a runner/environment change, not a commit.
  • How to verify: bisect the window on an MI325 runner against partition 0's case set.

No hypothesis above LOW confidence — root-causing is blocked by the missing part-0 log; a hang-time artifact/py-spy dump is needed before a commit can be implicated.

In-flight Fix Check

• Searched open PRs: multimodal_gen+hang, diffusion+amd+hang, multimodal-gen+timeout, diffusion+timeout. Closest is #29584 (open, ci(amd): fix three pre-existing MI325 CI failures) but it targets stage-b AMD jobs, not multimodal-gen-test-1-gpu.
• In-flight fix: ❌ none found targeting the multimodal-gen 1-GPU part-0 hang — needs new PR or investigation.

Suggested Triage Steps

• Add a per-server / per-test watchdog and force artifact + py-spy dump on hang in step 9 so partition 0 produces a diagnosable log (the single biggest blocker here is zero log).
• Once a hang log exists, identify the stalling model/case in partition 0's LPT-assigned set and check it against 94de28764c88..238b2b2c9c17; bisect if a specific case regressed on Jul 11.
• Separately track the part-3 real failure: test_update_weights_from_disk.py FLUX.2-klein-base-4B transformer loading 0.0 GB + the -k "not flux_2" filter not excluding FLUX.2-klein IDs.
• Consider adding a job-level timeout-minutes so hangs fail fast with an uploaded log instead of running to runner termination.

Status

• Persistence: Failing every completed scheduled run since 2026-07-11 00:26 UTC (last green 2026-07-10 12:44 UTC); pure-hang mode for ~8 days.
• Scope: multimodal-gen-test-1-gpu-amd parts 0 and 1 hang; part 3 has real test failures; part 2 passes. Confined to the 1-GPU diffusion suite in pr-test-amd.yml.
• Blocked work: No reliable 1-GPU diffusion CI signal on AMD MI325 for partitions 0/1.
• In-flight fix: ❌ none found targeting this hang.

stage-b-test-large-8-gpu-mi35x-disaggregation-amd (linux-mi35x-gpu-8.fabric) — failed step(s): Run test_disaggregation

Commit Info

• sglang: 238b2b2c9c17647cad98ff2203d0fdbaf2e0dce2
• aiter: 9127c94a18e4398e1eba91f6639e910f0994ad02 (source: dev/patched CI image — v0.1.17.dev110+g9127c94a1 detected, rebuild skipped; Dockerfile default matched this commit)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/disaggregation/test_disaggregation_pp.py	TestDisaggregationDecodePPAccuracy.test_gsm8k	KeyError: 'answer' (via run_eval → few_shot_gsm8k.py:123), all retries exhausted → exit code 1	Log

The other 3 files in the suite passed (test_disaggregation_basic.py, test_mori_transfer_engine_e2e.py, test_nixl_transfer_engine_e2e.py) — suite result 3/4 passed.

Failure Summary

In the PD-disaggregation pipeline-parallel accuracy test (Qwen3-8B, --tp-size 2 --pp-size 2 --disaggregation-transfer-backend mooncake --attention-backend aiter), test_gsm8k runs a 200-question GSM8K eval. The eval progress bar reached 100% (200/200) but few_shot_gsm8k.run_eval raised KeyError: 'answer' at few_shot_gsm8k.py:123 — at least one of the 200 request states never had its gen("answer") variable populated. The test's built-in retry (max 1) reran the full eval and hit the same KeyError again, so it exited with exit code 1.

Regression Status

Flaky / intermittent correctness failure (recently surfaced). The test_disaggregation_pp.py file only began producing clean pass/fail verdicts on 2026-07-15; earlier runs aborted with exit code 134 before it completed.

Recent history of test_disaggregation_pp.py (test_gsm8k) in job stage-b-test-large-8-gpu-mi35x-disaggregation-amd (most recent comparable completed scheduled runs: main, event=schedule):

Date (UTC)	Run	Test File Status	Failed Function	Error
07-18 12:17	29644059344	✅ Passed (4/4)	—	—
07-18 00:25	29623067212 (this)	❌ Failed	test_gsm8k	KeyError: 'answer'
07-17 12:25	29580198489	❌ Failed	test_gsm8k	KeyError: 'answer'
07-17 00:27	29544840981	⚠️ job cancelled — no pp verdict	—	—
07-16 12:31	29498496614	⚠️ job cancelled — no pp verdict	—	—
07-15 12:28	29415344552	⚠️ job aborted (exit 134) before pp completed	—	—
07-15 00:22	29378873430	✅ Passed (4/4)	—	—
07-14 12:26	29332511138	⚠️ job aborted (exit 134)	—	—
07-14 00:24	29295847781	⚠️ job aborted (exit 134)	—	—
07-13 → 07-12 (4 runs)	29251251853 … 29173750641	⚠️ all job aborted (exit 134) before pp completed	—	—

• Clean pp verdicts in window: 2 pass, 2 fail. First observed KeyError failure: 2026-07-17 12:25. Last known pp pass: 2026-07-18 12:17 (the run after this failure) and 2026-07-15 00:22.
• The separate exit 134 (Fatal Python error: Aborted) job crash is a distinct, longer-standing failure cluster that masked the pp signal on 07-11…07-15; it is not the same failure as this KeyError.

Failure Cluster

Dropped/empty GSM8K request in PD-disaggregation PP eval (KeyError: 'answer')

Facts (verified observations)

• Verbatim error (occurs 3× — initial + retry attempts): KeyError: 'answer' at few_shot_gsm8k.py:123 → interpreter.py:357 return self.variables[name]. Final wrapper: Exception: retry() exceed maximum number of retries. → FAILED (errors=1) → suite exit code 1.
• Config under test (from log line 15941–15942): prefill+decode Qwen3-8B, --tp-size 2 --pp-size 2 --disable-overlap-schedule --attention-backend aiter --disaggregation-transfer-backend mooncake --disaggregation-ib-device rdma0.
• The eval completed all 200 iterations (100%|██████████| 200/200) before the KeyError, i.e. the batch finished but ≥1 state lacked the answer variable — a request that returned without producing generation output, not a hang/timeout.
• The identical KeyError: 'answer' in TestDisaggregationDecodePPAccuracy.test_gsm8k also occurred in the prior scheduled run 29580198489 (verified in that log at lines 19936/20821/20843).
• The next scheduled run 29644059344 (sha d7b942552927, which includes every commit in 238b2b2) passed pp 4/4.
• Commit ordering (verified via git log): 50d1eda (07-14, pp pass) → 85ac56c (07-17, pp fail) → 238b2b2 (this, fail) → d7b94255 (07-18, pp pass). 73 commits in 50d1eda..238b2b2.

Hypothesised Causes (with confidence)

• [LOW] Intermittent dropped request in the mooncake PD-disaggregation PP transfer path — under tp=2/pp=2, one request occasionally returns with no generated text, so gen("answer") is never set and states[i]["answer"] is missing.
  • Supporting: eval reached 200/200 (no hang) yet a single state lacked answer; failure is transient (2 consecutive fails bracketed by passes); this is the only backend/config combo (mooncake + pp=2 + tp=2) in the suite that failed while nixl/mori/basic passed.
  • Disconfirming: the run immediately after (same code, sha d7b94255) passed pp 4/4, so this is not deterministic; no server-side error/stack was emitted for the dropped request in the log.
  • How to verify: rerun test_disaggregation_pp.py on linux-mi35x-gpu-8.fabric 5–10× at sha 238b2b2; measure KeyError frequency. Add defensive logging in the test to dump the offending states[i] (meta_info/finish_reason) when answer is absent.
• [LOW] A disagg-touching commit in 50d1eda..238b2b2 perturbed PD reliability — candidates: #31584 (2db21daf8 "Fix Heartbeat Checker in KV Manager Disaggregation") and #31495 (d28e35b1a "Fix num_running_reqs gauge on disagg prefill servers").
  • Supporting: both land in the regression window and modify disaggregation code paths.
  • Disconfirming (strong): sha d7b94255 contains both commits and passed pp — a hard regression from either would still fail there. This makes a deterministic commit-cause unlikely; at most it raised flake probability.
  • How to verify: only pursue if the repeat-run frequency test above shows a clear failure-rate step change; otherwise treat as flake.
• No hypothesis above LOW confidence. The pass-after-fail at d7b94255 is decisive disconfirming evidence against any hard regression.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for disaggregation+pp+test, KeyError+answer+gsm8k, disaggregation+flaky. Nearest-but-unrelated: #31543 "[PD] pool decode bootstrap HTTP sessions" and #25921 "fix(disagg): disable cuda graph correctly when running prefill" — neither targets this KeyError. Needs new investigation.

Suggested Triage Steps

• Rerun test_disaggregation_pp.py 5–10× at sha 238b2b2 on linux-mi35x-gpu-8.fabric to quantify the flake rate before assigning any commit cause.
• Harden the test/eval: in few_shot_gsm8k.run_eval (or the test) log states[i].get("meta_info") / finish_reason when answer is missing, so the next occurrence reveals whether the request errored, was aborted, or returned empty.
• If the flake rate is non-trivial, bisect disagg-touching commits in 50d1eda..238b2b2 — start with #31584 and #31495 — but note the pass at d7b94255 argues against a deterministic culprit.
• Separately track the recurring exit code 134 (Fatal Python error: Aborted) job crashes (07-11…07-15) as a distinct issue — they block clean pp signal on this job.

Status

• Persistence: KeyError: 'answer' in pp test_gsm8k observed in 2 consecutive completed scheduled runs (2026-07-17 12:25 and this 2026-07-18 00:25), then recovered in the next run (2026-07-18 12:17). Clean pp verdicts in window: 2 fail / 2 pass → intermittent.
• Scope: 1 test file (test_disaggregation_pp.py), 1 function (test_gsm8k), 1 job (stage-b-test-large-8-gpu-mi35x-disaggregation-amd) in pr-test-amd.yml. Only the mooncake + pp=2/tp=2 config; nixl/mori/basic disagg tests pass.
• Blocked work: PD-disaggregation pipeline-parallel accuracy signal on MI35x is unreliable (intermittently red).
• In-flight fix: ❌ none found in open PRs.

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-07-19 17:01 UTC)

[amd-bot]

pr-test-amd.yml — Per-Job Analysis (continued 4/7)

stage-b-test-1-gpu-small-amd (linux-mi325-1gpu-sglang, 4) — failed step(s): Run test

Commit Info

• sglang: 238b2b2c9
• aiter: 9127c94a1 — v0.1.17.dev110+g9127c94a1 (source: dev/patched image detected, rebuild skipped)

Failed Tests

Test File	Test Function	Error	Log
test/registered/core/test_srt_endpoint.py	test_logprob_mixed (TestSRTEndpoint)	AssertionError in scheduler → server crash → ConnectionError: retry() exceed maximum number of retries in test client	Log

The other 9 test files in this shard passed (Test Summary: 9/10 passed).

Failure Summary

test_logprob_mixed sends 8 concurrent generate requests with mixed input lengths (up to 50000 tokens) and top_logprobs_num values. During heavy concurrent chunked prefill, the scheduler process hit an AssertionError at python/sglang/srt/managers/scheduler_components/logprob_result_processor.py:252 (len(req.logprob.input_top_logprobs_val) == relevant_tokens_len), received SIGQUIT, and exited. Subsequent client requests then failed with Connection refused / RemoteDisconnected, surfacing as a test ERROR. The connection errors are downstream symptoms; the primary fault is the scheduler-side assertion.

Regression Status

Flaky / intermittent. The same test file passed in the immediately preceding and following comparable scheduled runs.

Recent history of test/registered/core/test_srt_endpoint.py (test_logprob_mixed) in job stage-b-test-1-gpu-small-amd small-shard family (branch main, event=schedule, completed runs; shard index varies run-to-run due to dynamic test partitioning):

Date	Run	Head SHA	Test File Status	Failed Function	Error
Jul 18 12:17	29644059344	d7b942552	✅ Passed (shard 4, 10/10)	—	—
Jul 18 00:25	29623067212	238b2b2c9	❌ Failed (shard 4)	test_logprob_mixed	AssertionError (scheduler)
Jul 17 12:25	29580198489	85ac56c82	✅ Passed (shard 13)	—	—

Notes: The two prior runs' shard-7 failures were a different, unrelated failure (AssertionError: <throughput> not greater than 2000), not this logprob assertion — verified at file level. First (and so far only) observed failure of this test file/function: 2026-07-18 00:25 UTC. Last known passing: 2026-07-18 12:17 UTC (after) and 2026-07-17 12:25 UTC (before).

Failure Cluster

Logprob-length assertion crash in scheduler during concurrent chunked prefill (input_top_logprobs_val length mismatch)

Facts (verified observations)

• Scheduler traceback (verbatim, log line 6274-6299): crash in logprob_result_processor.py:252, add_input_logprob_return_values → assertion len(req.logprob.input_top_logprobs_val) == relevant_tokens_len fails; bare AssertionError.
• Call chain: event_loop_overlap → pop_and_process → process_batch_result → batch_result_processor.process_batch_result_prefill (batch_result_processor.py:251) → _apply_prefill_logprobs (:418) → add_logprob_return_values → add_input_logprob_return_values.
• Crash occurs mid-run under concurrent prefill (log shows many Prefill batch lines with #new-token: 16384, #queue-req: 4, cuda graph: False immediately before the exception at 01:48:58).
• After the scheduler SIGQUIT, client sees RemoteDisconnected then ConnectionRefused on port 11000 (log 6319, 6455); test client reports retry() exceed maximum number of retries (log 6543) and FAILED (errors=1) (log 6548). 12 tests ran, 1 errored.
• Same test file passed at file level in the adjacent scheduled runs before (29580198489, shard 13) and after (29644059344, shard 4 10/10).
• The test itself (test/registered/core/test_srt_endpoint.py:245-273) shuffles args and drives 8 threads with input_len up to 50000 and top_logprobs_num ∈ {0,5}, heavily exercising input-logprob accounting across prefill chunks.
• aiter unchanged/default dev build ([CI-AITER-CHECK] markers, log 4229-4235); not an aiter override run.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [MEDIUM] Pre-existing race/edge-case in input-logprob accounting under the overlap scheduler + chunked prefill: when top_logprobs_num > 0 and prefill is split across chunks at specific concurrent-batch boundaries, input_top_logprobs_val length diverges from relevant_tokens_len, tripping the assert at logprob_result_processor.py:252.
  • Supporting: exact assert + stack trace verified in log; test maximizes chunk-boundary and concurrency variety; failure appears only intermittently (timing/scheduling dependent).
  • Disconfirming: passed at the same/newer SHAs immediately before and after with no code change to the logprob path between them (logprob_result_processor.py last touched by #31227 7d0fd5101, which is older than the last passing run) — so this is not a fresh regression at 238b2b2c9.
  • How to verify: re-run test/registered/core/test_srt_endpoint.py::TestSRTEndpoint::test_logprob_mixed in a loop (10-20×) on AMD MI325 to reproduce the intermittent assert; if reproducible, add logging of len(input_top_logprobs_val) vs relevant_tokens_len and the chunk boundaries at logprob_result_processor.py:247-253.
• [LOW] Commit 238b2b2c9 (Remove QServe/FBGEMM FP8 quant, #31109) introduced the failure.
  • Supporting: it is this run's head SHA.
  • Disconfirming: the commit only removes quantization backends and does not touch the logprob/scheduler path; the very next run at a newer SHA passed 10/10, so the HEAD SHA does not deterministically cause the failure. Effectively rules this out.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for logprob assertion (17 results, none match this assert; nearest #31567 bounds top_logprobs_num vs vocab size, a different DoS issue), input_top_logprobs (0 results), and test_srt_endpoint (1 result, unrelated CPU-skip PR #20894). Note: a remote branch cursor/logprobs-assertion-failure-37d6 appears in the checkout (log 703) but has no associated open PR found.

Suggested Triage Steps

• Reproduce by running test_logprob_mixed in a loop on AMD MI325 (container rocm/sgl-dev:v0.5.15.post1-rocm700-mi30x-20260717) to confirm intermittency; if it cannot be reproduced quickly, treat as low-frequency flake and monitor.
• If reproduced, instrument logprob_result_processor._process_input_top_logprobs / _calculate_relevant_tokens_len (:57, :122) and the assert site (:247-253) to capture the divergent lengths and the prefill-chunk state; determine whether the top-logprob list is short/long by exactly the chunk overlap.
• Check whether disabling the overlap scheduler or chunked prefill in this test's server config makes the failure disappear (isolates the concurrency/chunking dimension).
• Investigate the existing cursor/logprobs-assertion-failure-37d6 branch — it may already contain a repro/fix for this exact assertion.

Status

• Persistence: Single occurrence — failed once in run 29623067212 (2026-07-18 00:25 UTC); passed in the comparable runs immediately before and after.
• Scope: 1 test function (test_logprob_mixed) in 1 shard of pr-test-amd.yml small-test stage; not observed in other workflows in this window.
• Blocked work: Intermittently red-flags the AMD 1-GPU small-test signal; low blast radius but can cause spurious PR-CI failures since a scheduler crash fails the whole test_srt_endpoint.py file.
• In-flight fix: ❌ none found — needs a repro-driven investigation (or check the cursor/logprobs-assertion-failure-37d6 branch).

stage-c-test-large-8-gpu-amd (linux-mi325-8gpu-sglang, 1) — failed step(s): Run test

Commit Info

• sglang: 238b2b2c9 — "Remove QServe and FBGEMM FP8 quantization (#31109)"
• aiter: 9127c94a (v0.1.17.dev110+g9127c94a1) — source: dev/patched image detected, rebuild skipped ([CI-AITER-CHECK])

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/test_deepseek_v32_basic.py	TestDeepseekV32TP.test_a_gsm8k	Scheduler subprocess crashed (exit -6 / SIGABRT) with HSA_STATUS_ERROR_OUT_OF_RESOURCES: Available Free mem : 0 MB at 99% (1311/1319) of GSM8K; test then errored KeyError: 'answer' → ConnectionRefused → retry() exceed maximum number of retries	Log

Final result: FAILED (errors=1, skipped=2) · Test Summary: 0/1 passed · job exit code 255. (The DP variant TestDeepseekV32DP is skipped on AMD; test_bs_1_speed never ran because the server died.)

Failure Summary

On DeepSeek-V3.2 TP=8 (MI325, 8-GPU), the GSM8K accuracy eval (num_questions=1400, parallel=1400, max_new_tokens=512) ran to 99% (1311/1319) and then a scheduler subprocess (scheduler_3, pid 3360) aborted with an HSA out-of-resources / 0 MB free GPU memory error during a decode CUDA-graph forward. The server crashed, so the remaining requests got connection-refused and the harness raised KeyError: 'answer' on the missing generation output. This is a GPU memory-exhaustion crash, not an accuracy assertion failure.

Regression Status

Known recurring failure (test-file level) — chronically unstable on this shard, with two distinct signatures.

Recent history of test/registered/amd/test_deepseek_v32_basic.py (TestDeepseekV32TP.test_a_gsm8k) in job stage-c-test-large-8-gpu-amd (linux-mi325-8gpu-sglang, 1) (most recent comparable completed scheduled runs: same branch main, event=schedule):

Date (UTC)	Run	Test File Status	Signature
Jul 18 12:17	29644059344	✅ Passed	Accuracy 0.956
Jul 18 00:25	29623067212	❌ Failed (THIS RUN)	OOM crash @99% GSM8K
Jul 17 12:25	29580198489	❌ Failed	watchdog-timeout hang (300s), no accuracy
Jul 17 00:27	29544840981	❌ Failed	watchdog-timeout hang
Jul 16 12:31	29498496614	❌ Failed	watchdog-timeout hang
Jul 15 12:28	29415344552	❌ Failed	watchdog-timeout hang
Jul 15 00:22	29378873430	❌ Failed	watchdog-timeout hang
Jul 14 12:26	29332511138	❌ Failed	watchdog-timeout hang

• First observed failure (this window): ≥ Jul 14 12:26 UTC (shard-1 was also failure on Jul 13–14 runs, so likely earlier). No passing run found before this run in the window.
• First known pass: the next run, Jul 18 12:49 (Accuracy 0.956) — i.e. the test recovered immediately after this run.
• Key nuance: this specific run's OOM crash is a different symptom from the Jul 14–17 watchdog hangs (verified: zero OUT_OF_RESOURCES and no Accuracy printed in those logs — the server never completed the eval). The test is resource-marginal / flaky on this large DSV3.2 8-GPU shard rather than deterministically broken.

Failure Cluster

GPU out-of-resources (0 MB free) crash during DeepSeek-V3.2 GSM8K decode. (Distinct from the sibling "DSV3.2 8-GPU server watchdog-timeout hang" cluster seen on Jul 14–17.)

Facts (verified observations)

• Verbatim: :0:rocdevice.cpp :3671: … Aborting with error : HSA_STATUS_ERROR_OUT_OF_RESOURCES: … Code: 0x1008 Available Free mem : 0 MB at progress 1311/1319.
• [2026-07-18 01:08:42] Subprocess scheduler_3 (pid=3360) crashed with exit code -6. Triggering SIGQUIT for cleanup...
• Crash stack (scheduler forward at time of abort): compute_dsa_seqlens (python/sglang/srt/layers/attention/dsa/utils.py:64) ← _apply_cuda_graph_metadata (dsa_backend.py:1644) ← init_forward_metadata_out_graph (dsa_backend.py:712) ← decode_cuda_graph_runner.py load_batch:1199/execute:1255 ← model_runner.py _forward_raw:1403/forward:1271 ← tp_worker.py:533 ← scheduler.py run_batch:3332.
• Downstream harness error: few_shot_gsm8k.py:123 → interpreter.py get_var → KeyError: 'answer', then URLError: [Errno 111] Connection refused (server dead), then Exception: retry() exceed maximum number of retries.
• Test config (source): TestDeepseekV32TP launches deepseek-ai/DeepSeek-V3.2 with --tp 8, --dsa-prefill-backend tilelang --dsa-decode-backend tilelang; eval sends all 1400 questions concurrently (parallel=1400).
• aiter version unchanged (dev image) across passing and failing runs; NVIDIA path unaffected — AMD 8-GPU only.
• Reproducibility: same test file failed in ≥7 consecutive completed scheduled runs (Jul 14–18 00:25); recovered on the next run.

Hypothesised Causes (with confidence)

• [MEDIUM] GPU memory exhaustion from the eval's extreme concurrency on a very large model — DeepSeek-V3.2 TP=8 with parallel=1400 (all GSM8K questions in flight at once) drives HBM to 0 MB free on the final decode batches.
  • Supporting: crash lands at 99% (last batch) with literal Available Free mem : 0 MB; the fault is in the decode CUDA-graph metadata path (compute_dsa_seqlens), i.e. an allocation during active decode; test config sends 1400 requests concurrently.
  • Disconfirming: the immediately-following run (Jul 18 12:49) with the identical config passed at 0.956 → not a deterministic config bug; points to nondeterministic memory pressure / fragmentation, and the sibling failures are a hang, not an OOM.
  • How to verify: rerun the shard with reduced --mem-fraction-static (or lower parallel) and capture peak HBM; check whether OOM disappears.
• [LOW] DSA tilelang decode/prefill backend under-reserves / mis-accounts KV-indexer memory, leaving no headroom for the last batches.
  • Supporting: crash frame is inside the tilelang DSA path; open DSA memory-accounting PRs exist (#31218, #28752).
  • Disconfirming: no code-level evidence that either PR touches this exact OOM; test passes intermittently with the same code.
  • How to verify: A/B the run with a non-tilelang DSA backend, or apply #31218/#28752 on a branch and rerun.
• Not a head-commit regression: 238b2b2c9 only removes QServe/FBGEMM FP8 quant (unrelated to the AMD DSA/DeepSeek path), and the failure spans dozens of commits — so no single suspicious commit in this window.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for DeepSeek-V3.2+OOM, deepseek_v32_basic, dsa+memory, deepseek+watchdog. DSA memory PRs #31218 and #28752 exist but do not target this test's OOM/hang. No PR addresses the DSV3.2 8-GPU AMD instability.

Suggested Triage Steps

• Rerun test_deepseek_v32_basic.py on MI325 8-GPU with lower --mem-fraction-static (or reduce eval parallel below 1400) and monitor peak HBM to confirm the memory-headroom hypothesis.
• If OOM persists, A/B the --dsa-decode-backend/--dsa-prefill-backend tilelang setting against an alternative backend to isolate DSA-indexer memory accounting.
• Treat this test as flaky on AMD (2 failure modes: OOM crash + watchdog hang, ~1/8 runs green) — consider quarantining/relaxing on AMD until memory headroom is fixed, rather than chasing a single-commit bisect (failure is not tied to the head commit).
• If DSA memory accounting is confirmed as the culprit, evaluate whether #31218/#28752 help on this shard.

Status

• Persistence: test-file-level failing since ≥ Jul 14 12:26 UTC (≥7 consecutive completed scheduled runs through this Jul 18 00:25 run); recovered on the next run (Jul 18 12:49, 0.956).
• Scope: stage-c-test-large-8-gpu-amd shard 1 only (DeepSeek-V3.2 TP=8, MI325); AMD-only, NVIDIA unaffected. Sibling shards 0/2/3 of the same job are unrelated.
• Signatures: 2 distinct — OOM crash (this run) vs watchdog-timeout hang (Jul 14–17).
• In-flight fix: ❌ none found.

stage-c-test-large-8-gpu-amd (linux-mi325-8gpu-sglang, 0) — failed step(s): N/A

Commit Info

• sglang: 238b2b2c9 ("Remove QServe and FBGEMM FP8 quantization" #31109)
• aiter: N/A — the job never reached the Install dependencies step, so aiter was never installed and no [CI-AITER-CHECK] markers exist. The job log blob was never uploaded (BlobNotFound from the logs API).

Failed Tests

Test File	Test Function	Error	Log
N/A (non-test failure)	N/A	Job hung at Start CI container step (stuck in_progress, never completed) and was terminated ~12.5 min later	Job

No test ran. This is an infrastructure/runner-level failure at container startup, not a test failure.

Failure Summary

The job stage-c-test-large-8-gpu-amd (linux-mi325-8gpu-sglang, 0) failed before executing any tests. Step-level status shows Set up job, Checkout code, and Ensure VRAM is clear succeeded, but Start CI container entered in_progress at 00:32:29Z and never completed; the job was terminated at 00:45:13Z (~12m 44s total). The downstream steps Install dependencies, Test RCCL multi-GPU communication, and Run test all remained pending. The job log blob does not exist (BlobNotFound), consistent with an abrupt kill at the container-start stage before any log could be uploaded.

Regression Status

Infrastructure issue (isolated / transient flake) — not a code regression.

Recent history of this job (stage-c-test-large-8-gpu-amd (linux-mi325-8gpu-sglang, 0)) in pr-test-amd.yml (comparable completed scheduled runs, same branch main, event=schedule). Since no test ran, tracking is at the Start CI container / Run test step level:

Date	Run	Container step	Run test step	Job result
Jul 18 12:17	29644059344	✅ success	✅ success	✅ Passed
Jul 18 00:25	29623067212	❌ hung (in_progress)	⏸ pending	❌ Failed
Jul 17 12:25	29580198489	✅ success	✅ success	✅ Passed
Jul 17 00:27	29544840981	✅ success	✅ success	✅ Passed
Jul 16 12:31	29498496614	✅ success	✅ success	✅ Passed
Jul 15 12:28	29415344552	✅ success	✅ success	✅ Passed
Jul 15 00:22	29378873430	✅ success	✅ success	✅ Passed
Jul 14 12:26	29332511138	✅ success	✅ success	✅ Passed
Jul 14 00:24	29295847781	✅ success	✅ success	✅ Passed

This job passed in all 8 comparable scheduled runs checked (both before and after the failing run). The failing run is a single, isolated occurrence; the very next scheduled run 12h later passed with the container step succeeding.

Failure Cluster

Container startup hang on 8-GPU MI325 runner (Start CI container never completes; no log uploaded).

Facts (verified observations, no interpretation)

• Job stage-c-test-large-8-gpu-amd (linux-mi325-8gpu-sglang, 0) (id 88021853407) conclusion = failure (verified via jobs API).
• Step status (verified via jobs API): Set up job ✅, Checkout code ✅, Ensure VRAM is clear ✅, Start CI container = in_progress / conclusion None (started 2026-07-18T00:32:29Z, no completed_at), Install dependencies / Test RCCL multi-GPU communication / Run test all pending.
• Job wall-clock: started 2026-07-18T00:32:22Z, completed 2026-07-18T00:45:13Z (~12m 51s); the container-start step alone hung ~12.5 min before the job was terminated.
• Job log blob returns BlobNotFound from GET /actions/jobs/88021853407/logs — no log was uploaded, consistent with an abrupt termination at container start.
• Reproducibility: 1/9 comparable scheduled runs failed for this job; the other 8 (before and after) passed with Start CI container = success.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [MEDIUM] Transient runner/infrastructure failure while pulling or starting the CI Docker container on the MI325 8-GPU runner (image pull stall, runner disk/network hiccup, or runner killed by the platform).
  • Supporting: failure is at the Start CI container step, before any sglang/aiter code runs; the step hung ~12.5 min then the job was terminated; no log blob was uploaded (abrupt kill); no test ever executed.
  • Disconfirming: exact failure mode (pull vs. daemon vs. timeout) cannot be confirmed because the log blob is gone — (none contradicting the infra hypothesis found).
  • How to verify: re-run the job on the same runner pool; if it passes, confirms a transient flake. Check runner host logs / disk & registry availability for the 00:32–00:45Z window on 2026-07-18.
• [SPECULATION] Commit 238b2b2c9 is implicated.
  • Supporting: it was the head SHA of the failing run.
  • Disconfirming: the failure occurred at container startup before any repository code is imported or executed, so the commit content is not on the failing code path; adjacent scheduled runs with different SHAs passed on this job. This hypothesis is effectively ruled out.

In-flight Fix Check

• In-flight fix: ❌ none applicable — this is a transient container-startup infra flake, not a code defect; no source fix is warranted. (No PR search performed since there is no code target; a maintainer may re-run the job to confirm the flake.)

Suggested Triage Steps

• Re-run the failed job 88021853407; a green re-run confirms a one-off container-start flake and closes this out.
• If container-start hangs recur on linux-mi325-8gpu-sglang, inspect the runner host: Docker daemon health, image registry reachability, and disk space during 2026-07-18 00:32–00:45Z.
• No sglang/aiter code investigation needed — the failure precedes dependency install and test execution.

Status (factual, no priority assignment)

• Persistence: single isolated occurrence on 2026-07-18 00:25Z run; not seen in the 8 other comparable scheduled runs checked (Jul 14–18), including the immediately following run which passed.
• Scope: this one job/shard only (stage-c-test-large-8-gpu-amd (linux-mi325-8gpu-sglang, 0)); container never started, so no tests were affected at the code level.
• Blocked work: none ongoing — subsequent scheduled run passed.
• In-flight fix: ❌ none applicable (transient infra flake).

stage-c-test-large-8-gpu-amd-mi35x (linux-mi35x-gpu-8, 0) — failed step(s): Run test

Commit Info

• sglang: 238b2b2c9
• aiter: 9127c94a1 (v0.1.17.dev110+g9127c94a1; source: Dockerfile default, dev image preserved — [CI-AITER-CHECK] reported "Dev/patched version detected → skipping rebuild")

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/test_kimi_k25_mxfp4_bcg_mi35x.py	TestKimiK25MXFP4BcgMI35x.test_bcg_gsm8k (variant='bcg')	KeyError: 'answer' (surface) — caused by scheduler crash RuntimeError: shape '[8192, 16, 512]' is invalid for input of size 16777216 at radix_attention.py:269	Log (step 6)

The companion test test/registered/cuda_graph/breakable/test_breakable_cuda_graph.py passed (1/2 files passed overall).

Failure Summary

The Kimi-K2.5-MXFP4 GSM8K accuracy test crashed all TP schedulers during prefill. The eval surfaced KeyError: 'answer' because the server never returned generations — the schedulers had already died with RuntimeError: shape '[8192, 16, 512]' is invalid for input of size 16777216 in unified_attention_with_output (radix_attention.py:269, output[:real_num_tokens].view(ret.shape).copy_(ret)). 16777216 = 8192 × 16 × 128 (v_head_dim=128), while the backend returned ret with head-dim 512 (the MLA latent / attn_mqa dim).

Regression Status

New regression. The test file was added Jul 16 in #31227 (7d0fd5101); it passed exactly once, then regressed.

Recent history of test_kimi_k25_mxfp4_bcg_mi35x.py (test_bcg_gsm8k) in job stage-c-test-large-8-gpu-amd-mi35x (linux-mi35x-gpu-8, 0) (all completed event=schedule, branch=main runs since the test was added — verified at test-file level):

Date (run created)	Run	head_sha	Test File Status	Failure detail
Jul 18 12:17	29644059344	d7b9425529	❌ Failed (different cluster)	ValueError: TP rank ... didn't finish loading (likely OOM/slow node) at load_model_utils.py:270 — NOT the shape mismatch
Jul 18 00:25	29623067212	238b2b2c9	❌ Failed	shape mismatch [8192, 16, 512] (this run)
Jul 17 12:25	29580198489	85ac56c823	❌ Failed	shape mismatch [6461, 16, 512] (same cluster)
Jul 17 00:27	29544840981	dc0b3eb68f	✅ Passed	2/2 passed
Jul 16 12:31	29498496614	7d0fd5101	⚠️ Cancelled	inconclusive (test newly added; run cancelled)

• First observed (shape-mismatch cluster): Jul 17 12:25 (85ac56c823). Last passing: Jul 17 00:27 (dc0b3eb68f).
• Regression window: dc0b3eb68f..85ac56c823 (~33 commits).
• Note: the latest run (Jul 18 12:17) fails the same test file with a different failure (a distributed model-load ValueError, likely OOM/slow-node), which masks the shape mismatch there.

Failure Cluster

DeepSeek/Kimi MLA prefill head-dim mismatch on HIP eager fallback (missing MHA companion swap)

Facts (verified observations)

• Verbatim crash (radix_attention.py:269): RuntimeError: shape '[8192, 16, 512]' is invalid for input of size 16777216. 8192×16×128 = 16777216; backend ret head-dim = 512 (≠ v_head_dim 128).
• Full stack: kimi_k25.py:826 forward → mm_utils.py:1177 general_mm_embed_routine → deepseek_v2.py:2598/2221 forward → forward_mha.py:371 forward_normal_core (attn_mha(...), save_kv_cache=False) → radix_attention.py:178 unified_attention_with_output → line 269.
• Crash occurred in the eager extend path (eager_runner.py:203/336 _execute_extend), not inside a captured graph.
• Server config (from log): --tensor-parallel-size 4 --prefill-attention-backend aiter --decode-attention-backend aiter --cuda-graph-backend-prefill breakable --kv-cache-dtype fp8_e4m3 on MI35x (gfx950).
• All 4 TP ranks (TP0–TP3) hit the identical error, then SIGQUIT → schedulers died → GSM8K eval raised KeyError: 'answer'.
• Reproduced in 2 consecutive runs at the test-file level (85ac56c823, 238b2b2c9).
• #31391 (24a8944e1) is confirmed in first-failing 85ac56c823 and absent from last-passing dc0b3eb68f (verified via git merge-base --is-ancestor).

Hypothesised Causes (with confidence)

• [HIGH] #31391 (24a8944e1) "enable Kimi multimodal breakable prefill cuda graph replay" removed the MHA companion swap on the HIP eager fallback path.
  • Supporting: The PR replaced the per-layer attn_layer._pcg_mha_companion attribute (set once at model setup, guarded by _is_hip, available on every forward path) with a per-forward context list context.mha_companion_layers[layer_id] (radix_attention.py:226-229). It updated the three set_tc_piecewise_forward_context(...) callers in prefill_cuda_graph_runner.py to pass mha_companion_layers=, but the eager HIP-PCG fallback caller at eager_runner.py:327-334 still omits it, so it defaults to None. With mha_companion_layers is None, the attn_mqa → attn_mha swap is skipped, the backend uses attn_mqa metadata (MLA latent head-dim 512) while the outer RadixAttention.forward sized the output buffer for attn_mha v_head_dim=128 (radix_attention.py:170) → the .view() mismatch. Symptom (512 vs 128), timing (pass at dc0b3eb68f, fail at 85ac56c823), and code path all converge. Independently confirmed by open fix PR #31675, whose root-cause description matches exactly.
  • Disconfirming: No revert-and-rerun A/B was performed in this analysis (relies on code reading + the fix PR). The latest run (Jul 18 12:17) does not reproduce this cluster because an earlier model-load ValueError masks it — but the shape mismatch is reproduced in the two prior runs.
  • How to verify: apply #31675 (adds mha_companion_layers=model_runner.mha_companion_layers to the eager_runner.py:327 call) on a branch and re-run test_kimi_k25_mxfp4_bcg_mi35x.py on MI35x.

In-flight Fix Check

• In-flight fix: ✅ #31675 "[AMD] Fix DeepSeek MLA prefill shape mismatch on HIP eager fallback (missing mha_companion_layers)" — open since 2026-07-18, 1-line change (+1/-0), mergeable_state: blocked (awaiting CI/review). Chase reviewers; do NOT open a duplicate. (Search: is:pr is:open 31391 and is:pr is:open attn_mha+companion.)

Suggested Triage Steps

• Verify #31675 passes the AMD MI35x lane and merge it — it directly threads mha_companion_layers into the eager fallback context.
• After merge, re-run stage-c-test-large-8-gpu-amd-mi35x and confirm test_kimi_k25_mxfp4_bcg_mi35x.py returns to 2/2.
• Separately triage the latest-run (Jul 18 12:17) dist_barrier_after_load ValueError (load_model_utils.py:270, "rank didn't finish loading — likely OOM/slow node") on Kimi-K2.5-MXFP4 TP=4 — a distinct cluster (possible OOM/mem-fraction or flaky node) not addressed by #31675.

Status

• Persistence: shape-mismatch cluster failing since Jul 17 12:25 UTC (first fail 85ac56c823), reproduced in the 2 completed runs that reached this test with that code; last green Jul 17 00:27 (dc0b3eb68f).
• Scope: test_bcg_gsm8k on AMD MI35x (gfx950), TP=4, breakable-prefill + aiter backend; all 4 TP ranks affected. HIP eager-fallback path only.
• Blocked work: Kimi-K2.5-MXFP4 breakable-prefill CI signal on MI35x.
• In-flight fix: ✅ #31675 (open since 2026-07-18, blocked on CI/review).

stage-c-test-large-8-gpu-amd-mi35x (linux-mi35x-gpu-8, 1) — failed step(s): Run test

Commit Info

• sglang: 238b2b2c9c17647cad98ff2203d0fdbaf2e0dce2
• aiter: 9127c94a18e4398e1eba91f6639e910f0994ad02 (v0.1.17.dev110+g9127c94a1, source: dev image preserved; AITER_COMMIT_OVERRIDE empty)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/test_deepseek_r1_mxfp4_8gpu.py	TestDeepseekR1MXFP4.test_a_gsm8k	RuntimeError: shape '[-1, 16, 576]' is invalid for input of size 35782656 (server crash on all 8 TP ranks) → then KeyError: 'answer' → ConnectionRefused on retry	Log

The other two files in this shard passed: test_kimi_k25_mxfp4.py (✅) and test_qwen3_coder_next_8gpu.py (✅, accuracy 0.965). Job summary: 2/3 passed.

Failure Summary

The DeepSeek-R1-MXFP4 8-GPU server crashed on the very first GSM8K prefill. All 8 TP scheduler ranks raised RuntimeError: shape '[-1, 16, 576]' is invalid for input of size 35782656 inside the tc_piecewise prefill runner, at aiter_backend.py:2108 (q.view(-1, layer.tp_q_head_num, layer.qk_head_dim)), reached via radix_attention.py:257 unified_attention_with_output. With the schedulers dead, the eval requests produced no answer variable (KeyError: 'answer'), and the retry's /flush_cache call got Connection refused because the server had already exited.

Regression Status

New regression. Test file test_deepseek_r1_mxfp4_8gpu.py passed 3/3 at sglang dc0b3eb6 and first failed with this exact reshape signature at 85ac56c8.

Recent history of test_deepseek_r1_mxfp4_8gpu.py (TestDeepseekR1MXFP4.test_a_gsm8k) in job stage-c-test-large-8-gpu-amd-mi35x (linux-mi35x-gpu-8, 1) (comparable completed event=schedule runs on main, ordered by commit time):

Commit date	Run	Job	Test File Status	Failed Function	Error
Jul 18 00:25	29623067212	job	❌ Failed (THIS RUN, 238b2b2c)	test_a_gsm8k	shape '[-1, 16, 576]'
Jul 17 12:25	29580198489	job	❌ Failed (85ac56c8)	test_a_gsm8k	shape '[-1, 16, 576]' (size 36003840)
Jul 17 00:27	29544840981	job	✅ Passed 3/3 (dc0b3eb6)	—	—
Jul 15 12:28	29415344552	job	⚠️ Job aborted (exit 134) before deepseek result — inconclusive	—	—
Jul 15 00:22	29378873430	job	⚠️ Job aborted (exit 134) — inconclusive	—	—
Jul 14 12:26	29332511138	job	⚠️ Job aborted (exit 134) — inconclusive	—	—
Jul 14 00:24	29295847781	job	✅ Passed 3/3	—	—

First observed reshape failure: 85ac56c8 (Jul 17). Last known passing: dc0b3eb6 (Jul 17). Note: the newer run 29644059344 (d7b94255, Jul 18 12:30) also failed this file but at setUpClass (a different, later-stage symptom), not the reshape.

Failure Cluster

DeepSeek MLA prefill q.view shape mismatch on HIP tc_piecewise eager fallback

Facts (verified observations)

• Verbatim error (all 8 TP ranks): RuntimeError: shape '[-1, 16, 576]' is invalid for input of size 35782656, prefixed by Fail when using backend: tc_piecewise for prefill runner. (log).
• Stack trace (verified): scheduler.py → tp_worker.py → model_runner.py:1457 _forward_raw → eager_runner.py:203 execute → eager_runner.py:336 _execute_extend → deepseek_v2.py:2869 forward → compile.py:197 trampoline (tc_piecewise) → radix_attention.py:257 unified_attention_with_output → aiter_backend.py:2108 forward_extend → q.view(-1, layer.tp_q_head_num, layer.qk_head_dim).
• tp_q_head_num=16 (128 heads / TP8), qk_head_dim=576 (DeepSeek MLA: 512 kv_lora + 64 rope). q element count is not divisible by 16×576=9216, i.e. q carries metadata for a different attention layer than layer.
• Only 1 of 34 window commits touches any file on the failing path: 24a8944e1 modifies radix_attention.py + eager_runner.py-adjacent context wiring; no other window commit touches radix_attention.py, eager_runner.py, aiter_backend.py, or the tc_piecewise context_manager.py.
• HIP/AMD-specific: NVIDIA lanes are unaffected; the crash path is the # HIP PCG eager fallback block at eager_runner.py:327.
• Reproducibility: crash occurred identically on all 8 TP ranks, in 2 consecutive scheduled runs (85ac56c8, 238b2b2c).

Hypothesised Causes (with confidence)

• [HIGH] Commit 24a8944e1 (PR #31391 "fix: enable Kimi multimodal breakable prefill cuda graph replay") broke the DeepSeek MLA MHA-companion layer swap on the HIP eager path.
  • Supporting: The commit replaced the device-agnostic swap if _is_hip and not save_kv_cache and hasattr(attention_layer, "_pcg_mha_companion") with if not save_kv_cache and context.mha_companion_layers is not None in radix_attention.py. It threaded mha_companion_layers into PrefillCudaGraphRunner's context, but the # HIP PCG eager fallback call to set_tc_piecewise_forward_context in eager_runner.py:327 does not pass mha_companion_layers, so it defaults to None. With the companion never swapped, the aiter backend receives attn_mqa's head/dim metadata instead of attn_mha's, so q.view(-1, 16, 576) fails. The commit's own removed comment states this swap exists precisely so "the backend sees correct head/dim metadata." Temporal correlation: passes at dc0b3eb6 (excludes the commit), fails at 85ac56c8 (includes it, merged Jul 17 19:13).
  • Disconfirming: (none found) — the fix PR below confirms the same mechanism; the later run d7b94255 shows a different setUpClass symptom, but that is a downstream/secondary failure, not counter-evidence for the reshape.
  • How to verify: on AMD MI35x, revert 24a8944e1 (or apply #31675) and re-run test/registered/amd/test_deepseek_r1_mxfp4_8gpu.py.

In-flight Fix Check

In-flight fix: ✅ [#31675](https://github.com/sgl-project/sglang/pull/31675) "[AMD] Fix DeepSeek MLA prefill shape mismatch on HIP eager fallback (missing mha_companion_layers)" (open, non-draft, created 2026-07-18) — chase reviewers, do NOT open a duplicate. It is a one-line change to python/sglang/srt/model_executor/runner/eager_runner.py (+1/-0) adding the missing mha_companion_layers argument, exactly matching the HIGH-confidence hypothesis above. (Searched search/issues for mha_companion, companion+layer, tc_piecewise+prefill.)

Suggested Triage Steps

• Verify #31675 rebases cleanly on 238b2b2c and re-run test_deepseek_r1_mxfp4_8gpu.py on AMD MI35x to confirm it resolves the reshape crash.
• Confirm whether the setUpClass failure in the later run 29644059344 (d7b94255) is a separate issue or a downstream effect that #31675 also clears; if it persists after the fix, open a separate investigation (likely server startup / GPU-memory state).
• Bisect dc0b3eb6..85ac56c8 only if #31675 does not fully resolve it (all path-relevant changes isolate to 24a8944e1).

Status

• Persistence: Reshape failure present since 85ac56c8 (Jul 17 12:25 scheduled run) through this run (238b2b2c, Jul 18 00:25) — 2 consecutive completed scheduled runs.
• Scope: test_deepseek_r1_mxfp4_8gpu.py::test_a_gsm8k on the MI35x 8-GPU large-test shard; HIP/AMD-only (tc_piecewise prefill eager fallback). Kimi and Qwen3-Coder tests in the same shard pass.
• Blocked work: DeepSeek-R1-MXFP4 8-GPU accuracy signal on AMD MI35x.
• In-flight fix: ✅ #31675 (open since 2026-07-18) — awaiting merge.

stage-b-test-large-8-gpu-mi35x-disaggregation-amd (linux-mi35x-gpu-8.fabric) — failed step(s): Run test_disaggregation

Commit Info

• sglang: 99f5a6f46b14e244d90fcbd751a5e238c1ba1290
• aiter: 9127c94a18e4398e1eba91f6639e910f0994ad02 (v0.1.17.dev110+g9127c94a1; source: Dockerfile default, AITER_COMMIT_OVERRIDE empty, version matches)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/disaggregation/test_disaggregation_pp.py	TestDisaggregationPrefillPPAccuracy.test_gsm8k	KVTransferError: Failed to get kvcache from prefill instance, it might be dead → KeyError: 'answer' → Exception: retry() exceed maximum number of retries	Log

The other 3 files in this job passed: test_disaggregation_basic.py, test_mori_transfer_engine_e2e.py, test_nixl_transfer_engine_e2e.py (job Test Summary: 3/4 passed).

Failure Summary

The 8-GPU MI35x disaggregation job failed only in test_disaggregation_pp.py, the pipeline-parallel PD test (Qwen3-8B, prefill tp=2 pp=2, decode tp=2, mooncake transfer backend, aiter attention). During the gsm8k eval, the mooncake KV-cache transfer between prefill and decode timed out (transfer_engine_py.cpp:614] Sync batch data transfer timeout after 30000001692ns ≈ 30 s). The decode server then declared the prefill instance dead, returned HTTP 500s, and the gsm8k harness could not populate the answer variable (KeyError: 'answer'), failing after retries. The pp=1 sibling tests (nixl/mori/basic) in the same job passed.

Regression Status

Flaky / intermittent — mooncake KV-transfer timeout in the PP disaggregation path. The test passes when only a few requests time out (the gsm8k accuracy > 0.70 threshold tolerates some failures) and fails when a burst of transfers stall.

Recent history of test_disaggregation_pp.py (TestDisaggregationPrefillPPAccuracy.test_gsm8k) in job stage-b-test-large-8-gpu-mi35x-disaggregation-amd (most recent comparable completed event=schedule, branch=main runs; verdict is at the test-file level, not the job's overall status):

Date (UTC)	Run	Test File Status	Detail
Jul 19 00:27	29666971143	❌ Failed	transfer timeout → KeyError 'answer'
Jul 18 12:17	29644059344	✅ Passed	4/4 passed
Jul 18 00:25	29623067212	❌ Failed	same signature
Jul 17 12:25	29580198489	❌ Failed	same signature
Jul 17 00:27	29544840981	⚪ Not comparable	job cancelled
Jul 16 12:31	29498496614	⚪ Not comparable	job cancelled
Jul 15 12:28	29415344552	⚠️ Not reached	job failed before pp ran
Jul 15 00:22	29378873430	✅ Passed	passed with 5 transfer timeouts tolerated

Among runs where the pp test actually ran: 3 failed / 5 passed-or-failed (fail: Jul 17 12:25, Jul 18 00:25, Jul 19 today; pass: Jul 15 00:22, Jul 18 12:17). Note the test passed Jul 18 12:17 and failed both immediately before (Jul 18 00:25) and after (today) — pass/fail alternates across runs, so there is no clean regression window. First transfer-timeout occurrence observed Jul 15 00:22 (tolerated); first test-level failure Jul 17 12:25.

Failure Cluster

Mooncake KV-cache transfer 30s sync timeout in PP disaggregation (decode declares prefill dead)

Facts (verified observations)

• I0719 00:41:58.349304 26609 transfer_engine_py.cpp:614] Sync batch data transfer timeout after 30000001692ns (mooncake transfer engine, ~30 s) — verbatim from log.
• Decode transfer failed ... KVTransferError(...): Failed to get kvcache from prefill instance, it might be dead → POST /generate HTTP/1.1 500 Internal Server Error.
• gsm8k harness stack: few_shot_gsm8k.py:123 → interpreter.py get_var → KeyError: 'answer'; then common.py:3188 raise Exception("retry() exceed maximum number of retries"); unittest result Ran 2 tests ... FAILED (errors=1).
• pp test config (from source test_disaggregation_pp.py): Qwen3-8B, prefill --tp-size 2 --pp-size 2 --disable-overlap-schedule --attention-backend aiter, decode --tp-size 2 --base-gpu-id 4; transfer backend defaults to mooncake (disaggregation_fixture.py:63).
• Prefill was serving normally at gsm8k start (PP0/PP1 prefill batches at 00:41:27), then transfers began stalling at ~00:41:58 (127/200 progress).
• Identical transfer_engine_py.cpp:614 timeout + KVTransferError + KeyError: 'answer' signature confirmed in the Jul 18 00:25 and Jul 17 12:25 failing runs.
• aiter version is the Dockerfile default and unchanged across passing and failing runs.
• Same-job sibling disaggregation tests (nixl, mori, basic — all pp=1) passed in this run.

Hypothesised Causes (with confidence)

• [MEDIUM] Intermittent mooncake KV-cache transfer stall on the MI35x disaggregation runner (RDMA/NIC or mooncake transfer reliability), aggravated by the PP (pp=2+tp=2) topology.
  • Supporting: the crash is a 30 s mooncake sync-transfer timeout, not a model/kernel error; failure recurs intermittently on the same head SHA (passes and fails across adjacent runs); only the pp path fails while pp=1 nixl/mori/basic pass; open debug PR #29127 ("Isolate faulty RDMA NIC (rdma5) on MI35x disaggregation runner") indicates a known NIC-reliability suspicion on this exact runner.
  • Disconfirming: the specific NIC cannot be confirmed from this log (only generic UCX ERROR failed to get interface index for tw-ethN messages, which also appear in the passing sibling tests, so they are not by themselves the cause); the passing runs use the same runner/config.
  • How to verify: re-run test_disaggregation_pp.py several times on the MI35x disagg runner in isolation; correlate timeouts with a specific RDMA NIC (per #29127); check whether raising the mooncake transfer timeout removes the failures.
• [LOW] A recent PD/mooncake code change regressed transfer reliability under PP.
  • Supporting: first test-level failure appeared Jul 17.
  • Disconfirming: no clean pass→fail SHA boundary (same-SHA runs both pass and fail); transfer timeouts were already present (tolerated) in the Jul 15 00:22 passing run, predating a single obvious culprit.
  • How to verify: only worth pursuing if the runner/NIC hypothesis is ruled out; then bisect PD/mooncake commits over Jul 14–17.

In-flight Fix Check

No PR is a confirmed fix for this exact flake, but several open PRs target the same area (searched mooncake+timeout, KVTransferError, disaggregation+pp, transfer+timeout+disaggregation):

• Infra/runner: #29127 — [DEBUG] Isolate faulty RDMA NIC (rdma5) on MI35x disaggregation runner (directly relevant to the runner hypothesis).
• Mooncake transfer reliability: #31357 Drain Mooncake transfers before releasing KV pages; #31144 Prevent decode scheduler from blocking on ZMQ sends to a stalled prefill peer; #30769 Add decode-side bootstrap timeout cleanup for Mooncake; #29984 Fix race on shared ZMQ sockets in mooncake KV manager senders.
• In-flight fix: ⚠️ candidates only — none verified to resolve this pp-disaggregation transfer timeout; do NOT assume any closes it without a repro.

Suggested Triage Steps

• Re-run test_disaggregation_pp.py in isolation on the MI35x disagg runner N× to quantify the flake rate and confirm intermittency (expected ~3/5 fail rate based on history).
• Correlate mooncake transfer timeouts with a specific RDMA NIC on this runner (align with #29127); if a faulty NIC is confirmed, treat as an infra fix rather than a code regression.
• Capture the prefill-side scheduler/mooncake logs at the moment of the 30 s stall (this job log only shows the decode-side "prefill dead" symptom) to determine whether the prefill actually hung or the transfer channel stalled.
• Evaluate whether #31144 / #31357 reduce the timeout rate under the pp=2 config.
• If the runner/NIC is exonerated and the flake persists, bisect PD/mooncake commits Jul 14–17; otherwise consider quarantining the pp gsm8k assertion until transfer reliability is addressed.

Status

• Persistence: Intermittent since ~Jul 17 12:25 UTC (transfer timeouts first seen, tolerated, Jul 15 00:22). Fails ~3 of the last 5 runs where the pp test actually ran; passed as recently as Jul 18 12:17.
• Scope: One test file (test_disaggregation_pp.py, test_gsm8k) in one job (stage-b-test-large-8-gpu-mi35x-disaggregation-amd) on pr-test-amd.yml; pp=1 disaggregation variants unaffected.
• Blocked work: Blocks reliable green signal for PP prefill/decode disaggregation on MI35x; other disaggregation coverage remains green.
• In-flight fix: ⚠️ candidate PRs only (#29127, #31357, #31144) — none confirmed to fix this flake.

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-07-19 17:01 UTC)

[amd-bot]

pr-test-amd.yml — Cross-Run Pattern Summary (4 runs in lookback window)

Last scanned (UTC): 2026-07-19 17:04 UTC

Agent assessment

Persistent clusters (top by job count)

1. Diffusion 1-GPU shard hang / abnormal runner termination (no logs) — the largest cluster.

• Jobs (every run): multimodal-gen-test-1-gpu (…,0) + (…,1). Shard 1 also surfaced as self-hosted runner lost communication / Server not ready within 1200s — same hang family.
• 4/4 runs; shard-1 failing since 2026-07-09 (last green 28985427127). Step 9 left in_progress, log blob BlobNotFound → no test-level evidence.
• Top hypothesis: [LOW] a heavy LPT-assigned diffusion case (video/image gen) wedges on ROCm; no stack trace, cannot exclude runner/container infra fault. In-flight fix: ❌ none.

2. DeepSeek/Kimi MLA prefill view/shape mismatch (tc_piecewise, missing MHA-companion swap) — now resolved.

• Jobs: stage-c-8gpu-mi35x (…,0) + (…,1), failing runs 29623067212, 29644059344, 29666971143 (3/4).
• RuntimeError: shape '[…,16,512]' is invalid … at radix_attention.py:269.
• [MEDIUM] resolved by #31675 (c68392c5): it is an ancestor of the latest run's SHA a03ca46a, and both mi35x stage-c shards passed in that latest run while failing in all 3 prior. Disconfirming: correlation only — no A/B revert. Verify by watching the next mi35x run stays green.

3. AMD 2-GPU diffusion suite (multiple concurrent modes) — 4/4 runs, split across two shards:

• (…,1): disagg ZImage ValueError: truth value of an array … ambiguous in extract_transfer_fields → warmup 500 / DiffusionServer timeout 120s. In-flight fix: ✅ #31325.
• (…,0): FLUX.2 modelopt-FP8 HIPBLAS_STATUS_NOT_SUPPORTED on gfx942 (+ JoyAI-Echo HF download). In-flight fix: ✅ #28889.
• Sibling 1-GPU (…,3): update_weights_from_disk HTTP 500, Inplace update to inference tensor outside InferenceMode. In-flight fix: ✅ #31263.

Regression candidates (latest run only, completed)

(none) — the latest run 29686709871 introduced no failure not already seen earlier in the window.

Flakes / intermittent

• PD/PP disaggregation transfer timeout — stage-b-8gpu-mi35x-disaggregation, 3/4 runs; symptom drifts (Mooncake 30s sync timeout / KeyError:'answer' GSM8K / Failed to get kvcache from prefill). Likely a NIC/transfer infra flake (#29127 isolates a faulty rdma5 NIC). Investigate.
• kv_canary verify_position ChunkCache crash — extra-a-test-2-gpu-large, 1/4 (run 29666971143); attributed to decode-ChunkCache regression #31662, no in-flight fix. Investigate — did not recur in latest run.
• AMD remote-instance TransferEngine leaves dst weights zeroed — stage-b-2gpu-large (…,1), 1/4; in-flight fix (draft) #30035. Retry/quarantine.
• DeepSeek-V3.2 GPU OOM (0 MB free) in decode — stage-c-8gpu (…,1), 1/4; no fix. Investigate.
• Logprob length assertion (concurrent chunked prefill) — stage-b-1gpu-small (…,4), 1/4. Retry, quarantine if recurs.
• 8-GPU MI325 container startup hang — stage-c-8gpu (…,0), 1/4; pure infra flake, no code target. Rerun to confirm.

Suggested next actions

1. Confirm MLA fix stuck — verify mi35x stage-c (…,0)/(…,1) stay green over the next 2 runs now that #31675 is merged; if so, close the cluster.
2. Chase the 3 open diffusion fixes — #31325, #31263, #28889 — merging all three clears most of the 2-GPU/1-GPU-shard-3 diffusion red.
3. Instrument the 1-GPU diffusion hang (cluster 1, largest, no fix) — add pytest per-case --timeout + py-spy dump so the wedging case produces an uploaded log; currently undiagnosable (BlobNotFound).
4. Triage stage-b disaggregation transfer flake — correlate the 3/4-run timeouts with the rdma5 NIC (#29127); decide isolate-NIC vs code.
5. Watch kv_canary #31662 — no in-flight fix and it crashes the disagg-decode scheduler; needs an owner if it recurs.

---

Runs analysed

Run	Started (UTC)	Failed jobs
29623067212	2026-07-18 00:25	11
29644059344	2026-07-18 12:24	8
29666971143	2026-07-19 00:28	9
29686709871	2026-07-19 12:27	6

Persistent failures (in every run)

• multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 0) — in 4/4 runs (3067212, 4059344, 6971143, 6709871)
• multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 1) — in 4/4 runs (3067212, 4059344, 6971143, 6709871)
• multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 3) — in 4/4 runs (3067212, 4059344, 6971143, 6709871)
• multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 0) — in 4/4 runs (3067212, 4059344, 6971143, 6709871)
• multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 1) — in 4/4 runs (3067212, 4059344, 6971143, 6709871)

Latest-only failures (potential regressions)

(none)

Flaky / intermittent failures

• call-pr-test-amd-extra / extra-a-test-2-gpu-large-amd (linux-mi325-2gpu-sglang) — in 1/4 runs (6971143)
• stage-b-test-1-gpu-small-amd (linux-mi325-1gpu-sglang, 4) — in 1/4 runs (3067212)
• stage-b-test-2-gpu-large-amd (linux-mi325-2gpu-sglang, 1) — in 1/4 runs (4059344)
• stage-b-test-large-8-gpu-mi35x-disaggregation-amd (linux-mi35x-gpu-8.fabric) — in 3/4 runs (3067212, 6971143, 6709871)
• stage-c-test-large-8-gpu-amd (linux-mi325-8gpu-sglang, 0) — in 1/4 runs (3067212)
• stage-c-test-large-8-gpu-amd (linux-mi325-8gpu-sglang, 1) — in 1/4 runs (3067212)
• stage-c-test-large-8-gpu-amd-mi35x (linux-mi35x-gpu-8, 0) — in 3/4 runs (3067212, 4059344, 6971143)
• stage-c-test-large-8-gpu-amd-mi35x (linux-mi35x-gpu-8, 1) — in 3/4 runs (3067212, 4059344, 6971143)

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-07-19 17:04 UTC)

[amd-bot]

pr-test-amd.yml — Per-Job Analysis (continued 5/7)

multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 3) — failed step(s): Run diffusion server tests (1-GPU)

Commit Info

• sglang: 99f5a6f46
• aiter: 9127c94a1 — v0.1.17.dev110+g9127c94a1 (source: dev image detected, rebuild skipped; matches Dockerfile default per [CI-AITER-CHECK] markers)

Failed Tests

Job multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 3) · step 9 · pytest summary 5 failed, 1 passed, 4 errors in 970.65s. All in python/sglang/multimodal_gen/test/single_test_file/test_update_weights_from_disk.py.

Test File	Test Function	Error	Log
test_update_weights_from_disk.py	TestUpdateWeightsFromDisk::test_update_weights_specific_modules[Qwen-Image]	AssertionError: Update failed: {'success': False, 'message': 'Inplace update to inference tensor outside InferenceMode is not allowed...'} (assert 500==200), line 454	Log
test_update_weights_from_disk.py	TestUpdateWeightsFromDisk::test_update_weights_from_disk_default[Qwen-Image]	assert 500 == 200, line 432	Log
test_update_weights_from_disk.py	TestUpdateWeightsFromDisk::test_corrupted_weights_rollback[Qwen-Image]	assert (500 == 200), line 535	Log
test_update_weights_from_disk.py	TestUpdateWeightsFromDiskWithOffload::test_update_weights_with_offload_enabled[FLUX.2-klein-base-4B]	AssertionError: Expected 200, got 500, line 638	Log
test_update_weights_from_disk.py	TestUpdateWeightsFromDiskWithOffload::test_update_weights_with_offload_enabled[Qwen-Image]	AssertionError: Expected 200, got 500, line 638	Log
test_update_weights_from_disk.py	TestUpdateWeightsFromDisk::test_update_weights_from_disk_default[FLUX.2-klein-base-4B]	ERROR at setup — AssertionError: No weights dir for transformer in /tmp/sglang_perturbed_vae_..., line 220 (fixture diffusion_server_no_offload:410)	Log
test_update_weights_from_disk.py	TestUpdateWeightsFromDisk::test_update_weights_specific_modules[FLUX.2-klein-base-4B]	ERROR at setup — same No weights dir for transformer	Log
test_update_weights_from_disk.py	TestUpdateWeightsFromDisk::test_update_weights_rejects_invalid_requests[FLUX.2-klein-base-4B]	ERROR at setup — same No weights dir for transformer	Log
test_update_weights_from_disk.py	TestUpdateWeightsFromDisk::test_corrupted_weights_rollback[FLUX.2-klein-base-4B]	ERROR at setup — same No weights dir for transformer	Log

Failure Summary

The update_weights_from_disk server endpoint returns HTTP 500 during the diffusion post-training weight-update tests, so every test asserting a 200 fails, and the offload variant fails identically. The server-side stack trace shows RuntimeError: Inplace update to inference tensor outside InferenceMode is not allowed raised in the weight loader, both on the primary update and again inside _rollback (weights_updater.py:461). The four FLUX setup errors are a separate fixture-level failure (No weights dir for transformer in the perturbed-model temp dir).

Regression Status

Known recurring failure — the same 5 failed, 1 passed, 4 errors signature with the same Inplace update to inference tensor RuntimeError appears in every completed scheduled run of this shard in the queryable window. No passing run found (Never-passed within window; lookback = 8 completed scheduled runs, 2026-07-14 → 2026-07-19).

Recent history of test_update_weights_from_disk.py in job multimodal-gen-test-1-gpu-amd (…, 3) (most recent comparable completed scheduled runs: event=schedule, branch=main, completed only):

Date	Run	Job	Test File Status	Failed Function	Error
Jul 19	29666971143	job	❌ Failed (5f/1p/4e)	test_update_weights_specific_modules +8	Inplace update to inference tensor → 500
Jul 18	29644059344	job	❌ Failed (5f/1p/4e)	same	same
Jul 18	29623067212	job	❌ Failed (5f/1p/4e)	same	same
Jul 17	29580198489	job	❌ Failed (5f/1p/4e)	same	same
Jul 16	29498496614	job	❌ Failed (5f/1p/4e)	same	same
Jul 15	29415344552	job	❌ Failed (5f/1p/4e)	same	same
Jul 15	29378873430	job	❌ Failed (5f/1p/4e)	same	same
Jul 14	29295847781	job	❌ Failed (5f/1p/4e)	same	same

Failure Cluster

update_weights_from_disk 500 — Inplace update to inference tensor outside InferenceMode (diffusion post-training)

Facts (verified observations, no interpretation)

• Server-side stack trace (log lines ~12724–12744): _apply_weights → _rollback (weights_updater.py:455) → maybe_download_model (hf_diffusers_utils.py:962) raises ValueError/HFValidationError on FLUX; and separately _rollback (weights_updater.py:461) → _load_weights_into_module raises RuntimeError: Inplace update to inference tensor outside InferenceMode is not allowed (18 occurrences in the log, lines 5901, 7094, 8288, 8350, 9557, …).
• The endpoint returns 500 Internal Server Error for POST /update_weights_from_disk (log line 12744), so assert status_code == 200 fails for the Qwen-Image cases and both offload cases.
• Source: weights_updater.py:461 calls _load_weights_into_module(module, weights_iter), whose weight_loader does an in-place param copy (linear.py:410) — the inference-tensor path.
• The four FLUX setup ERRORs (No weights dir for transformer in /tmp/sglang_perturbed_vae_...) come from fixture diffusion_server_no_offload (line 410) via _compute_checksum_from_disk (line 220) — the perturbed-model temp dir is missing the transformer subdir. This is a distinct symptom from the 500s.
• Reproducibility: identical 5 failed / 1 passed / 4 errors + identical inference-tensor error count in all 8 completed scheduled runs Jul 14–19.
• aiter version is unchanged across all runs (dev image v0.1.17.dev110+g9127c94a1); NVIDIA-vs-AMD comparison not evaluated.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [HIGH] The diffusion post-training weight update copies weights in-place into model parameters that are inference-mode tensors (allocated under torch.inference_mode()), which PyTorch forbids outside an inference-mode context → RuntimeError → HTTP 500. Fix is to run the update under torch.inference_mode().
  • Supporting: RuntimeError: Inplace update to inference tensor outside InferenceMode printed verbatim (FACT); it originates at weights_updater.py:461 _load_weights_into_module; open PR #31263 modifies exactly weights_updater.py and is titled "run weight update under torch.inference_mode()", directly targeting this error.
  • Disconfirming: the four FLUX No weights dir for transformer setup ERRORs are a fixture-data problem, not the inference-tensor error, so #31263 alone may not turn this job fully green; the FLUX rollback also hits a separate HFValidationError (local snapshot path passed to snapshot_download).
  • How to verify: apply #31263 on a branch and re-run test_update_weights_from_disk.py on AMD MI325; confirm the Qwen-Image 500s clear.
• [MEDIUM] The FLUX setup ERRORs are an independent fixture bug: the perturbed-model dir (/tmp/sglang_perturbed_vae_*) contains only a perturbed vae and no transformer, so _compute_checksum_from_disk(..., "transformer") asserts.
  • Supporting: setup assertion at test_update_weights_from_disk.py:220 via fixture line 410; only FLUX parametrizations hit it, Qwen-Image gets past setup to the 500.
  • Disconfirming: could be downstream of the same broken update/rollback flow if the fixture reuses the update path to build the perturbed dir — not confirmed from log alone.
  • How to verify: read the diffusion_server_no_offload fixture body (lines ~380–410) and how perturbed_vae_model_dir is populated.

In-flight Fix Check

• In-flight fix: ✅ #31263 "[diffusion] post_training: run weight update under torch.inference_mode()" (open since 2026-07-15, updated 2026-07-16, non-draft, modifies weights_updater.py) — chase reviewers; matches the primary Inplace update to inference tensor error. Do NOT open a duplicate. (Search: search/issues?q=...+inference+tensor+InferenceMode.)
• Note: #31263 may not cover the FLUX setup ERRORs or the FLUX rollback HFValidationError — a follow-up may be needed for those.

Suggested Triage Steps

• Verify #31263 applies cleanly on 99f5a6f46 and re-run test_update_weights_from_disk.py on AMD MI325 (linux-mi325-1gpu-sglang shard 3) — expect the three Qwen-Image failures + two offload failures to clear.
• Independently investigate the FLUX No weights dir for transformer setup ERRORs by reading the diffusion_server_no_offload fixture (lines ~380–410) — determine why the perturbed-model dir lacks a transformer subdir; likely a separate fix from #31263.
• Inspect the FLUX rollback path weights_updater.py:455 → maybe_download_model(self.pipeline.model_path) where model_path is a local HF-cache snapshot dir being passed to snapshot_download (raises HFValidationError); may need a local-path guard in maybe_download_model (hf_diffusers_utils.py:962).

Status (factual, no priority assignment)

• Persistence: Failing in every completed scheduled run in the queryable window — at least since 2026-07-14 (8/8 runs), no green run for this test file found.
• Scope: multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 3) in pr-test-amd.yml; blocks the diffusion post-training update_weights_from_disk CI signal on AMD.
• In-flight fix: ✅ #31263 (open since 2026-07-15) addresses the primary inference-tensor error; FLUX setup/rollback errors may need a separate follow-up.

stage-c-test-large-8-gpu-amd-mi35x (linux-mi35x-gpu-8, 1) — failed step(s): Run test

Commit Info

• sglang: 99f5a6f46
• aiter: 9127c94a1 (v0.1.17.dev110+g9127c94a1, source: Dockerfile default / dev image preserved — AITER_COMMIT_OVERRIDE empty)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/test_deepseek_r1_mxfp4_8gpu.py	TestDeepseekR1MXFP4.setUpClass	Exception: Server process exited with code -9 (server crashed during prefill: RuntimeError: shape '[1, 16, 512]' is invalid for input of size 2048 at radix_attention.py:269)	Log

The other two files in this job passed: test_kimi_k25_mxfp4.py ✅ and test_qwen3_coder_next_8gpu.py ✅ (Test Summary: 2/3 passed).

Failure Summary

The DeepSeek-R1-MXFP4 8-GPU server fails to launch: during prefill warmup every TP rank (TP0–TP7) crashes with RuntimeError: shape '[1, 16, 512]' is invalid for input of size 2048 at python/sglang/srt/layers/radix_attention.py:269 (output[:real_num_tokens].view(ret.shape).copy_(ret)) on the tc_piecewise prefill runner path (the test launches with --enforce-piecewise-cuda-graph --piecewise-cuda-graph-compiler eager). The scheduler exits, popen_launch_server sees exit code -9, and setUpClass errors before any test runs (Ran 0 tests).

Regression Status

New regression — first failed 2026-07-17.

Recent history of test_deepseek_r1_mxfp4_8gpu.py in job stage-c-test-large-8-gpu-amd-mi35x (linux-mi35x-gpu-8, 1) (most recent comparable completed scheduled runs: branch=main, event=schedule):

Date (UTC)	Run	Test File Status	Failed Function	Error
07-19	29666971143	❌ Failed	setUpClass	shape '[1, 16, 512]' invalid for size 2048
07-18 12:17	29644059344	❌ Failed	setUpClass	shape '[1, 16, 512]' invalid for size 2048
07-18 00:25	29623067212	❌ Failed	prefill crash	shape '[-1, 16, 576]' invalid for size 35782656
07-17 12:25	29580198489	❌ Failed (first fail)	prefill crash	shape '[-1, 16, 576]' invalid for size 36003840
07-17 00:27	29544840981	✅ Passed (3/3)	—	—
07-16 12:31	29498496614	⚪ Cancelled	—	job cancelled

Last known pass: run 29544840981, head dc0b3eb68fa9. First fail: run 29580198489, head 85ac56c82308. The error size shifted from [-1,16,576] to [1,16,512] on 07-18 12:17 but is the same code path and same root cause under different token/batch shapes.

Failure Cluster

DeepSeek MLA prefill view/reshape shape mismatch on AMD tc_piecewise piecewise-CUDA-graph path

Facts (verified observations)

• Crash is identical across all 8 TP ranks at radix_attention.py:269, output[:real_num_tokens].view(ret.shape).copy_(ret) — verified in job log (lines ~6624–7304).
• Runner emits: Fail when using backend: tc_piecewise for prefill runner. — the failing path is the piecewise prefill CUDA graph, enabled by the test's --enforce-piecewise-cuda-graph flag.
• Test-file-level history: the test passed 3/3 at head dc0b3eb68fa9 (run 29544840981) and first failed at head 85ac56c82308 (run 29580198489) — a clean pass→fail transition.
• Regression window dc0b3eb68fa9..85ac56c82308 contains 24a8944e1 (#31391), which modified both radix_attention.py and runner/prefill_cuda_graph_runner.py.
• Current tree: radix_attention.py:226 gates the attn_mqa→attn_mha swap on context.mha_companion_layers is not None; #31391 added mha_companion_layers=self.mha_companion_layers only to the prefill_cuda_graph_runner.py call site (verified via git show 24a8944e1).
• aiter version is unchanged (dev image v0.1.17.dev110+g9127c94a1) across passing and failing runs — not an aiter-side change.

Hypothesised Causes (with confidence)

• [HIGH] PR #31391 (24a8944e1, "enable Kimi multimodal breakable prefill cuda graph replay") replaced the per-layer _pcg_mha_companion attribute with a context-threaded mha_companion_layers list, but did not update the HIP PCG eager-fallback call site in eager_runner.py. On AMD's eager-fallback prefill path context.mha_companion_layers is None, so radix_attention.py:226-227 never swaps attn_mqa→attn_mha; the backend then reshapes the MHA q using attn_mqa's head/dim metadata and the .view() fails.
  • Supporting: commit lands exactly in the pass→fail window; modifies the two files in the failing path; radix_attention.py:226 depends on the new field; the shape numbers match MLA MHA (head_dim 192) vs MQA (qk 576 / kv_lora 512) metadata; the PR author of the fix independently diagnosed the identical cause and links failing run 29644059344.
  • Disconfirming: none found — the last green run predates #31391 and the fix PR reproduces the same root cause.
  • How to verify: apply #31675 (adds mha_companion_layers to the eager_runner.py HIP fallback call site) on a branch and re-run test_deepseek_r1_mxfp4_8gpu.py on MI35x 8-GPU.
• [LOW] #31487 (6c6175fab, "avoid excessive prefill CUDA graph padding") lands in the 238b2b2c9c17..d7b942552927 window where the error size changed [-1,16,576]→[1,16,512].
  • Supporting: changes prefill CUDA graph token padding, which alters real_num_tokens/buffer sizes seen at the failing .view().
  • Disconfirming: does not introduce the failure (test already failing before it); only changes the reported shape numbers, not the root cause.
  • How to verify: not needed for root cause — subsumed by #31675.

In-flight Fix Check

• In-flight fix: ✅ #31675 "[AMD] Fix DeepSeek MLA prefill shape mismatch on HIP eager fallback (missing mha_companion_layers)" (open, non-draft, created 2026-07-18, base main). Directly targets this exact failure and cites failing run 29644059344. Chase reviewers — do NOT open a duplicate.
• Search: curl ".../search/issues?q=repo:sgl-project/sglang+is:pr+is:open+piecewise+prefill".

Suggested Triage Steps

• Merge/verify in-flight fix #31675: confirm it adds mha_companion_layers to the HIP eager-fallback set_tc_piecewise_forward_context(...) call site in eager_runner.py, then re-run test_deepseek_r1_mxfp4_8gpu.py on linux-mi35x-gpu-8.
• If #31675 stalls, bisect dc0b3eb68fa9..85ac56c82308 around 24a8944e1 (#31391) to A/B-confirm; a git revert 24a8944e1 on a branch + re-run should restore green.

Status

• Persistence: Failing since 2026-07-17 (first fail run 29580198489), every completed scheduled run since (~2 days, 4 consecutive runs).
• Scope: test_deepseek_r1_mxfp4_8gpu.py on the AMD MI35x 8-GPU tc_piecewise prefill path; other files in the same job pass. Likely affects any DeepSeek-MLA-on-AMD test using --enforce-piecewise-cuda-graph.
• Blocked work: DeepSeek-R1-MXFP4 8-GPU CI signal on AMD MI35x.
• In-flight fix: ✅ #31675 (open since 2026-07-18) — awaiting merge.

multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 1) — failed step(s): Run diffusion server tests (2-GPU)

Commit Info

• sglang: 99f5a6f46
• aiter: 9127c94a1 v0.1.17.dev110+g9127c94a1 (source: dev image — [CI-AITER-CHECK] Dev/patched version detected → skipping rebuild)

Failed Tests

Test File	Test Function	Error	Log
test/single_test_file/test_disagg_server.py	TestDisaggZImage1Rank::test_generates_image	RuntimeError: Warmup request failed for zimage_1rank: 500 (server-side DiffusionServer timeout … not completed within 120.0s)	Log
test/single_test_file/test_disagg_server.py	TestDisaggZImage2RankDenoiser::test_generates_image_with_sp2_denoiser	RuntimeError: Warmup request failed for zimage_sp2: 500 (same timeout)	Log
test/single_test_file/test_disagg_server.py	TestDisaggZImageTracing::test_disagg_spans_share_trace_id	RuntimeError: Warmup request failed for zimage_trace: 500 (same timeout)	Log

Pytest result: 3 errors in 1998.91s. All 3 error at setUpClass/warmup — the server never becomes healthy.

Failure Summary

The multimodal-gen-test-2-gpu-amd shard-1 job runs test_disagg_server.py (3 Z-Image disaggregated-serving tests). All 3 error during warmup: the client gets a 500 Internal Server Error on /v1/images/generations, because the disagg ENCODER role crashes every request. The underlying encoder error is ValueError: The truth value of an array with more than one element is ambiguous raised inside extract_transfer_fields → _is_default in scheduler_mixin.py, which prevents the encoder from producing/transferring output to the denoiser, so the request times out after 120s and the server returns 500 for all 3 warmup attempts.

Regression Status

Known recurring failure / Never-passed in window — NOT a regression from the current commit or from the recent 7d0fd5101.

Recent history of test/single_test_file/test_disagg_server.py (TestDisaggZImage*) in job multimodal-gen-test-2-gpu-amd (…, 1) (comparable completed scheduled runs, branch=main, event=schedule; shard-1 ran this file in every checked run):

Date	Run	Job	Test File Status	Failed Function	Error
Jul 19	29666971143	shard 1	❌ Failed	all 3 TestDisaggZImage*	truth value of array ambiguous @ _is_default (line 244)
Jul 18	29644059344	shard 1	❌ Failed	all 3	same signature
Jul 16	29498496614	shard 1	❌ Failed	all 3	same (culprit 7d0fd5101 present)
Jul 15	29415344552	shard 1	❌ Failed	all 3	same, _is_default @ line 226 (pre-7d0fd5101)
Jul 11	29132618067	shard 1	❌ Failed	all 3	same, extract_transfer_fields @ line 226

No green run found for this test file anywhere in the 15-run lookback window (2026-07-11 → 2026-07-19). First observed failure ≥ 2026-07-11 (window start); likely older. The 7d0fd5101 commit (2026-07-16) only shifted the crashing lines (226→244, 1545→1563) — it did not introduce the bug, since the identical error appears in the 2026-07-11 and 2026-07-15 runs that predate it.

Failure Cluster

Disagg ENCODER crash — numpy-array truthiness in extract_transfer_fields → warmup 500 / DiffusionServer timeout.

Facts (verified observations)

• Encoder error (verbatim, current run): Pool ENCODER rank 0: error (attempt 1/3): The truth value of an array with more than one element is ambiguous. Use a.any() or a.all().
• Stack trace (current run): scheduler_mixin.py:1049 _disagg_event_loop → :1563 _disagg_encoder_step → tensor_fields, scalar_fields = extract_transfer_fields(req_result) → :244 extract_transfer_fields → if _is_default(value, f):.
• Source: python/sglang/multimodal_gen/runtime/disaggregation/scheduler_mixin.py:155 — _is_default does return value == field_info.default. For a numpy ndarray value this returns an elementwise boolean array; using it in the caller's if _is_default(...) (line 244) raises the ambiguous-truth ValueError.
• Downstream symptom: encoder retries 3× and fails → denoiser never receives fields → DiffusionServer timeout: request … not completed within 120.0s → HTTP 500 on /v1/images/generations → warmup fails → all 3 tests error at setup.
• Reproducibility: identical test file + same 3 functions + same extract_transfer_fields/_is_default stack trace in 5/5 spot-checked scheduled runs spanning 2026-07-11 → 2026-07-19 (15 occurrences of the error string per run log).
• aiter is unchanged (dev image v0.1.17.dev110+g9127c94a1 in this run) and is not in the failing stack trace.

Hypothesised Causes (with confidence)

• [FACT] The failure is a Python-level bug in _is_default (scheduler_mixin.py:155): value == field_info.default on a numpy-array-valued Req field yields an array, and evaluating its truthiness at scheduler_mixin.py:244 raises ValueError: truth value of an array … is ambiguous.
  • Supporting: stack trace points exactly at _is_default/extract_transfer_fields; the source line matches; an explicit maintainer fix PR (#31325, "handle numpy arrays in cross-role transfer field extraction") targets this exact path.
  • Disconfirming: (none found) — code, stack trace, and the fix PR all agree.
  • How to verify: apply #31325 on a branch, re-run test_disagg_server.py on AMD MI325 2-GPU.
• [FACT] NOT caused by the current commit 99f5a6f46 nor by 7d0fd5101 — the identical error predates both (present 2026-07-11 and 2026-07-15).

In-flight Fix Check

• In-flight fix: ✅ #31325 "[diffusion] disagg: handle numpy arrays in cross-role transfer field extraction" — open (created 2026-07-15, not draft, mergeable_state: unstable). Directly fixes the numpy-array truthiness in transfer-field extraction. Chase reviewers / stabilize its own CI; do NOT open a duplicate. (Search: is:pr is:open extract_transfer_fields → 1 match.)

Suggested Triage Steps

• Verify #31325 resolves this cluster: rebase it on 99f5a6f46, re-run multimodal-gen-test-2-gpu-amd (…, 1) / test_disagg_server.py; its unstable mergeable state should be investigated so it can land.
• If #31325 stalls, a minimal local guard in _is_default (scheduler_mixin.py:155) — treat numpy/tensor values as non-default without == truthiness (e.g. np.array_equal or isinstance guard) — is the fix direction to confirm.
• Because this test has no green run in the 15-run window, consider tracking it as a persistent AMD disagg-serving gate rather than a fresh regression.

Status

• Persistence: Failing in every checked scheduled run 2026-07-11 → 2026-07-19 (≥ 8 days); no green run in the 15-run window.
• Scope: test_disagg_server.py (all 3 TestDisaggZImage* tests) in multimodal-gen-test-2-gpu-amd shard 1 on pr-test-amd.yml; blocks AMD Z-Image disaggregated-serving CI signal.
• In-flight fix: ✅ #31325 open since 2026-07-15 — awaiting merge/stabilization.

call-pr-test-amd-extra / extra-a-test-2-gpu-large-amd (linux-mi325-2gpu-sglang) — failed step(s): Run test

Commit Info

• sglang: 99f5a6f46b14e244d90fcbd751a5e238c1ba1290
• aiter: 9127c94a18e4398e1eba91f6639e910f0994ad02 (v0.1.17.dev110+g9127c94a1, source: dev image preserved — no override, Dockerfile default). aiter is unchanged vs the last passing run, so it is not a suspect.

Failed Tests

Test File	Test Function	Error	Log
test/registered/mock_model/test_e2e_pd.py	test_pd_transfer_canary_clean	AssertionError: 500 != 200 (decode server crashed → file exit code 137)	Log

Failure Summary

The PD (prefill/decode disaggregation) end-to-end test test_pd_transfer_canary_clean failed on the extra-a-test-2-gpu-large-amd job. The decode server's scheduler crashed with a RuntimeError: kv_canary violation during event_loop_overlap_disagg_decode; the fixture fail-fast then killed the decode process (rc=-9), so /generate requests returned 500 and the assertion result.status_code == 200 failed. The test file returned exit code 137. The other 5 files in the job (kv_canary perturb/baseline + mock_model tp/pp) all passed.

Regression Status

New regression. The extra-a-test-2-gpu-large-amd job (and test_e2e_pd.py specifically) passed in the previous 5 comparable completed scheduled runs and failed for the first time in this run.

Recent history of test/registered/mock_model/test_e2e_pd.py (test_pd_transfer_canary_clean) in job call-pr-test-amd-extra / extra-a-test-2-gpu-large-amd (most recent comparable completed scheduled runs: branch=main, event=schedule, completed only):

Date	Run	Job	Test File Status	Failed Function	Error
Jul 19 00:27	29666971143	extra-a-test-2-gpu-large-amd	❌ Failed	test_pd_transfer_canary_clean	kv_canary violation → decode rc=-9 → 500
Jul 18 12:17	29644059344	extra-a-test-2-gpu-large-amd	✅ Passed	—	—
Jul 18 00:25	29623067212	extra-a-test-2-gpu-large-amd	✅ Passed	—	—
Jul 17 12:25	29580198489	extra-a-test-2-gpu-large-amd	✅ Passed	—	—
Jul 17 00:27	29544840981	extra-a-test-2-gpu-large-amd	✅ Passed	—	—
Jul 16 12:31	29498496614	extra-a-test-2-gpu-large-amd	✅ Passed	—	—

First observed failure: 2026-07-19 00:27 UTC. Last known passing: 2026-07-18 12:17 UTC (d7b942552927). Regression window: d7b942552927..99f5a6f46b14 — only 7 commits.

Failure Cluster

kv_canary verify_position violation crashing disagg-decode scheduler (ChunkCache release)

Facts (verified observations, no interpretation)

• Decode scheduler traceback (verbatim, log ~01:29:03): RuntimeError: kv_canary violation: launch_tag=TAIL_K_FULL fail_reason=verify_position slot_idx=129928 position=0 stored_token=0 expected_token=-1 stored_chain_hash=0x0…0 expected_aux=0x0…0; total_violations=8194 ring_overflow=True step_when_pumped=35. Stack: scheduler.py:4469 dispatch_event_loop → disaggregation/decode.py:2059 event_loop_overlap_disagg_decode → run_batch → model_runner.py:1268 forward → kv_canary/runner/canary_manager.py:270 _post_ops_outside_graph → violation_reporter.py:67 log_or_raise_violation.
• Test config (log ~01:28:36): decode server --disaggregation-mode decode … --kv-canary raise, model Qwen/Qwen3-0.6B, --load-format dummy, mooncake transfer; [2026-07-19 01:28:32] KV cache is forced as chunk cache for decode server — decode server uses ChunkCache.
• Consequence: [FIXTURE FAIL-FAST] decode (pid=20376) exited rc=-9; aborting. → many POST /generate → 500 Internal Server Error (mini_lb ServerDisconnectedError) → AssertionError: 500 != 200 at test_e2e_pd.py:125 → ✗ FAILED … returned exit code 137.
• In the last passing run 29644059344, test_e2e_pd.py passed ({"passed": true, "elapsed": 62}); its 2648 kv_canary violation log lines all belong to the perturb tests (which expect violations), not the clean PD test.
• expected_token=-1 at position=0 indicates the canary read a slot its tracking table considers unassigned/stale — a slot-lifecycle mismatch, not a numeric mismatch.
• Reproducibility: single run so far, but violation is massive/systematic (total_violations=8194, ring_overflow=True), not a one-off flake.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [MEDIUM] Commit 10908a679 "[Fix] Respect cache_protected_len in ChunkCache and disabled-radix release paths" (#31662) changed ChunkCache.cache_finished_req to free req_to_token[idx, cache_protected_len : kv_len_to_handle] instead of [idx, :kv_len_to_handle] (python/sglang/srt/mem_cache/chunk_cache.py:85).
  • Supporting:
    • It is the ONLY commit in the 7-commit window that touches a FULL-KV/ChunkCache free path used by the decode server; the other 6 are flashinfer-SWA (NVIDIA; clean test is FULL, swa_window_size=0), a mechanical model_runner init refactor, metrics labels, two mamba-radix resets (Qwen3-0.6B is not mamba; decode uses ChunkCache with no radix), and an NPU-only fix.
    • The change is provably not a no-op here: the disagg decode path sets decode_req.req.cache_protected_len = total_prefix_len (nonzero) at python/sglang/srt/disaggregation/decode.py:1068, so the protected-prefix slots are now skipped on free.
    • The pre-change comment stated the decode-server intent explicitly ("For decode server: … we want to free all req.origin_input_ids"); ChunkCache retains no prefixes (disable=True, protected_size()==0), so for the decode server there is no "owned elsewhere" path to free that prefix → prefix slots are now leaked/never released, and their canary metadata is never reset, matching verify_position expected_token=-1 on slot recycle.
    • Temporal: test passed on d7b942552927 (which does NOT contain this commit, verified via git merge-base --is-ancestor) and failed on the very next scheduled run that includes it.
  • Disconfirming: the error string does not literally name cache_protected_len; the naive read of "skip freeing" implies a slot leak rather than a stale read, so the exact leak→recycle→canary mechanism is inferred, not directly traced. The PR added a passing unit test, but it covers PureSWAChunkCache, not plain ChunkCache decode with nonzero cache_protected_len. No revert A/B was run, so this stays MEDIUM (not HIGH).
  • How to verify: git revert 10908a679 on a branch and re-run test/registered/mock_model/test_e2e_pd.py (test_pd_transfer_canary_clean) on AMD MI325 2-GPU; expect the canary verify_position violations to disappear.
• [LOW] Commit b3a0185ca model_runner: extract post-memory-pool wiring into _init_post_memory (#31601) — touches the frozen model_runner.py through which the canary is installed.
  • Supporting: canary install/forward runs through model_runner.
  • Disconfirming: it is a mechanical init extraction with no change to KV free/slot lifecycle; the violation is in the per-forward slot-verify path, not init.
  • How to verify: only if the revert of 10908a679 does not clear the failure.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for cache_protected_len (6 results, all predate the regression or unrelated), ChunkCache+decode, kv_canary+decode, test_e2e_pd (0). None targets the #31662 decode-ChunkCache regression.

Suggested Triage Steps

• Revert 10908a679 (#31662) on a branch and re-run test/registered/mock_model/test_e2e_pd.py on AMD MI325 2-GPU (container rocm/sgl-dev:v0.5.15.post1-rocm700-mi30x-20260718) to confirm/deny the hypothesis.
• If confirmed, the fix likely needs ChunkCache's decode path to still free the protected prefix (since ChunkCache has no radix "owner" for it), i.e. special-case disable/ChunkCache so cache_protected_len is only skipped when a radix cache actually owns and later frees it.
• If disproven, bisect the remaining window d7b942552927..99f5a6f46b14, starting with b3a0185ca.

Status

• Persistence: New — first failure 2026-07-19 00:27 UTC; passed in the prior 5 consecutive scheduled runs (since at least 2026-07-16 12:31 UTC).
• Scope: 1 test file (test_e2e_pd.py::test_pd_transfer_canary_clean) in 1 job (extra-a-test-2-gpu-large-amd) on pr-test-amd.yml. Blocks the PD-disaggregation KV-transfer integrity (kv_canary "clean") signal on AMD.
• In-flight fix: ❌ none found — needs a new PR or investigation.

stage-c-test-large-8-gpu-amd-mi35x (linux-mi35x-gpu-8, 0) — failed step(s): Run test

Commit Info

• sglang: 99f5a6f46b14e244d90fcbd751a5e238c1ba1290
• aiter: 9127c94a18e4398e1eba91f6639e910f0994ad02 (v0.1.17.dev110+g9127c94a1, source: dev image — Dockerfile default preserved, dev/patched version detected, no rebuild)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/test_kimi_k25_mxfp4_bcg_mi35x.py	test_bcg_gsm8k (variant=bcg)	Exception: Server process exited with code -9 — server crashed during warmup on RuntimeError: shape '[1, 16, 512]' is invalid for input of size 2048	Log

Failure Summary

The 8-GPU MI35x large-test job failed because test_bcg_gsm8k could not launch the amd/Kimi-K2.5-MXFP4 server (--tensor-parallel-size 4, --prefill-attention-backend aiter, --cuda-graph-backend-prefill breakable). Every TP scheduler process hit a shape/view error in the DeepSeek MLA attention output path and died, so the launcher reported the server exited with code -9. The remaining test in the shard (test_breakable_cuda_graph.py) passed.

Regression Status

New regression — first failing at sglang 85ac56c8; last green at dc0b3eb6. The test file was added 2026-07-16 in #31227 and passed once before regressing.

Recent history of test_kimi_k25_mxfp4_bcg_mi35x.py (test_bcg_gsm8k) in job stage-c-test-large-8-gpu-amd-mi35x (linux-mi35x-gpu-8, 0) (ordered by head-commit date; event=schedule, branch=main, completed runs; job-execution time differs from run-created time due to 8-GPU queueing):

Commit date (UTC)	Run	head_sha	Test File Status	Failed Function	Error
2026-07-16 22:57	29544840981	dc0b3eb6	✅ Passed	—	—
2026-07-17 11:39	29580198489	85ac56c8	❌ Failed	test_bcg_gsm8k	shape '[6461, 16, 512]' is invalid for input of size 13232128
2026-07-18 00:10	29623067212	238b2b2c	❌ Failed	test_bcg_gsm8k	shape '[8192, 16, 512]' is invalid for input of size 16777216
2026-07-18 11:32	29644059344	d7b94255	❌ Failed	test_bcg_gsm8k	server exited -9 (same warmup crash)
2026-07-18 20:33	29666971143	99f5a6f4	❌ Failed (current)	test_bcg_gsm8k	shape '[1, 16, 512]' is invalid for input of size 2048

First observed failure: 2026-07-17 (head 85ac56c8). Last known passing: 2026-07-16 (head dc0b3eb6). Regression window: dc0b3eb6..85ac56c8 (34 commits).

Failure Cluster

DeepSeek/Kimi MLA attention output view shape mismatch (v_head_dim vs absorbed qk_head_dim) → server warmup crash

Facts (verified observations, no interpretation)

• Verbatim crash (all TP ranks): RuntimeError: shape '[<N>, 16, 512]' is invalid for input of size <N*2048>, raised at python/sglang/srt/layers/radix_attention.py:269: output[:real_num_tokens].view(ret.shape).copy_(ret). 512 is the MLA absorbed head dim (kv_lora_rank); 2048 = 16 heads × 128 v_head_dim. The failing element count is always real_num_tokens × 2048 while ret wants real_num_tokens × 16 × 512.
• Stack trace (verified from log): eager_runner._execute_extend → kimi_k25.py:836 → deepseek_v2.py:2603/2222/1872/1977 → forward_mha.py:371 forward_normal_core (attn_mha, save_kv_cache=False) → radix_attention.py:178 (non-breakable unified_attention_with_output) → :269. Crash is on the eager extend path, not graph replay.
• Server command: --model-path amd/Kimi-K2.5-MXFP4 --tensor-parallel-size 4 --prefill-attention-backend aiter --decode-attention-backend aiter --kv-cache-dtype fp8_e4m3 --cuda-graph-backend-prefill breakable --cuda-graph-backend-decode full.
• The only commit in the regression window touching radix_attention.py (or forward_mha.py) is 24a8944e1 (#31391, "fix: enable Kimi multimodal breakable prefill cuda graph replay", merged 2026-07-17 11:13 UTC).
• #31391 rewrote the MHA-companion selection inside unified_attention_with_output and removed the comment "use the companion attn_mha so the backend sees correct head/dim metadata".
• Reproducibility: identical symptom in 4 consecutive completed runs on 3 distinct head SHAs; all 4 TP ranks crash simultaneously. aiter version identical (v0.1.17.dev110+g9127c94a1) across the failing runs (from [CI-AITER-CHECK] markers).

Hypothesised Causes (with confidence, with disconfirming evidence)

• [HIGH] #31391 (24a8944e1) broke the MLA attn_mha companion-layer swap, so the aiter backend computes with the wrong (absorbed, dim 512) head metadata while the output buffer is sized for v_head_dim (128) → view fails.
  • Supporting:
    • Before: unified_attention_with_output swapped to the companion whenever _is_hip and not save_kv_cache and hasattr(attention_layer, "_pcg_mha_companion") — an attribute reachable on every call (incl. eager extend). After: it swaps only when context.mha_companion_layers is not None, where context = get_tc_piecewise_forward_context(). The eager extend path (the actual crash site, line 178) does not go through the TC-piecewise context that #31391 populates via set_tc_piecewise_forward_context(mha_companion_layers=...), so the companion swap is now skipped and the backend uses the attn_mqa head/dim (512).
    • The removed comment explicitly tied this logic to "correct head/dim metadata"; the failure is exactly a head/dim mismatch (512 vs 128).
    • It is the sole window commit touching the file; last-green dc0b3eb6 lacks it, first-red 85ac56c8 contains it (verified via git merge-base --is-ancestor).
  • Disconfirming: #31391 is titled as a breakable-cuda-graph fix, yet the crash is on the eager (non-graph) extend path — so the regression appears to be an unintended side-effect on the shared eager path rather than the graph path the PR targeted. No revert-and-rerun A/B has been performed, and I did not byte-verify aiter was identical in the last-green run (it is a dev image build). The other 33 window commits do not touch this file, weakening any alternative.
  • How to verify: git revert 24a8944e1 on a branch and re-run python3 test/registered/amd/test_kimi_k25_mxfp4_bcg_mi35x.py on an MI35x (gfx950) 4-GPU runner with the same server flags; confirm the warmup view error disappears.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for kimi+breakable (4 results, none address this shape error), mha_companion (0 results), and Kimi+MXFP4 (no matching fix). Needs a new PR or investigation.

Suggested Triage Steps

• Revert 24a8944e1 (#31391) on a branch and re-run test_kimi_k25_mxfp4_bcg_mi35x.py on MI35x to confirm it is the culprit.
• If confirmed, the likely fix is to ensure the attn_mha companion is selected on the eager extend path too — either restore the HIP companion swap in radix_attention.py:226-229 for the non-piecewise path, or populate context.mha_companion_layers for eager extend (the context currently only carries it under the TC-piecewise/breakable path per the context_manager.py / layer_setup.py changes in #31391).
• Ping the #31391 author (Mick) — the removed "correct head/dim metadata" comment describes exactly this failure mode.

Status

• Persistence: Failing since 2026-07-17 (head 85ac56c8); 4 consecutive completed runs red on this test.
• Scope: stage-c-test-large-8-gpu-amd-mi35x (linux-mi35x-gpu-8, 0) in pr-test-amd.yml; blocks the Kimi-K2.5-MXFP4 breakable-prefill GSM8K signal on MI35x. Symptom is in shared DeepSeek/Kimi MLA code, so other MLA-model tests using the same eager MHA path on ROCm may be at risk.
• In-flight fix: ❌ none found — needs new PR or investigation.

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-07-19 17:01 UTC)