[CI Monitor] Daily Report - 2026-07-25

[amd-bot]

Daily Cross-Workflow Summary — 2026-07-25

Snapshot: 2026-07-25 23:45 UTC · Only completed runs/jobs counted in trends · Auto-updated every 30 min

TL;DR

🟡 YELLOW · 2 🆕 today (R290 mi355x accuracy gate, R291 GHA platform error) · carry-over nightly families (R257 17d, R270 10d, R273, R262, R289) · R288 resolved (DCP hang did not recur) · in-flight fixes: R270→#31741 (blocked), R257→#31068/#32003, R290→#31727 (candidate)
👉 Today's ask: rerun the mi355x dsv4flash-fp8 leg 2–3× to separate R290 gate-flake from a real 3-day accuracy slide (0.917→0.914→0.905); rerun pr-test-amd to clear the transient R291 GHA platform error (it skipped all 22 AMD test jobs → no test signal on the latest run). Keep chasing #31741 (R270, 10th day) and #31068/#32003 (R257, 17th day). No fresh code regression.

Caution

pr-test-amd latest completed run 30157771919 (Jul-25 12:19) has NO test signal — a GitHub Actions internal platform error hit check-changes+gate jobs (R291), so all 22 downstream AMD test jobs were skipped, not tested. The real test-family failures come from the prior run 30136388246 (Jul-25 00:29). Rerun/update the branch before trusting a "green".

Caution

Jul-25 nightly-test-amd 30168349328 and nightly-test-amd-rocm720 30168262867 are IN-FLIGHT (queued, head 69a3c54c) → excluded from trends. Latest completed nightlies are the Jul-24 runs (30115195247 / 30115085856). pr-test-amd-rocm720 30168318210 also IN-FLIGHT; latest completed = Jul-24 30115153295.

Note

release-docker-amd-rocm720-nightly 30157811305 = startup_failure today (infra — workflow never provisioned, 0 jobs; was green Jul-18→24). release-docker-amd-nightly 30157965906 ✅ green. amd-aiter-scout: no run (cron Mon/Thu; today Sat).

Workflow status

Workflow	Latest completed run	❌ jobs	7d trend (completed only, old→new)	Δ vs yesterday
nightly-test-amd	Jul-24 30115195247 (failure)	8	❌·❌·⊘·❌·❌·❌·❌	~same (R289 confirmed intermittent)
nightly-test-amd-rocm720	Jul-24 30115085856 (failure)	10	❌·❌·⊘·❌·❌·❌·❌	R288 gone (DCP hang not reproduced)
nightly-amd-mi355x-disagg	Jul-25 30143908399 (failure)	3	✅·✅·✅·✅·✅·✅·❌	🔴 went RED (R290 🆕)
release-docker-amd-nightly	Jul-25 30157965906 ✅	0	✅·✅·✅·✅·⚠️·✅·✅	✅ still green
release-docker-amd-rocm720-nightly	Jul-25 30157811305 ⚠️ startup_failure	0 (infra)	✅·✅·✅·✅·✅·✅·⚠️	⚠️ startup_failure (infra)
nightly-amd-mi355x-disagg	(see above)
amd-aiter-scout	(no run today)	—	n/a (cron Mon/Thu)	—
pr-test-amd	Jul-25 30157771919 (failure — R291 infra)	4 (gate/infra)	❌·❌·❌·❌·❌·❌·❌	+R291 🆕 (GHA platform error)
pr-test-amd-rocm720	Jul-24 30115153295 (failure)	8	❌·❌·❌·❌·❌·❌·❌	all known families

Failure clusters (deduplicated across all workflows)

R290 · 🆕 · GSM8K accuracy gate miss — DeepSeek-V4-Flash-FP8 1P1D (MI355X disagg) — mi355x-disagg · ⚠️ candidate #31727

• Status: first RED for this workflow since Jul-18; config passed the prior 6 runs (Jul-19→24). This exact gate signature is intermittent: 3/13 completed runs (Jul-12, Jul-18, Jul-25). Marginal — 0.905 vs 0.91 gate.
• Top hypothesis: [HIGH] the 0.91 GSM8K gate is too tight for this DSV4-Flash-FP8 DP8/EP8 config (real accuracy 0.905–0.914, ±0.005 sampling variance); runs straddle the bar. Disconfirming: the 3-day trend is monotonic (0.917→0.914→0.905), weakly more consistent with a small real drift than pure noise — a rerun is needed to decide.
• In-flight fix: ❌ no threshold-retune PR; ⚠️ candidate #31727 (DSV4-Flash FP8 fused-RMS scale metadata on gfx950) could nudge numerics. Suggested triage: rerun the leg 2–3× at HEAD ebcb74ab; if it oscillates ≥0.91 it's gate flake, else bisect 99b29bf1..ebcb74ab (48 commits, FP8/MoE/DSV4 only).

Workflow	Job (shard)	Test File	Test Function	Error	Log
nightly-amd-mi355x-disagg	dsv4flash-fp8-1k1k-1p1d-dp8ep8	GSM8K gate (few_shot_gsm8k.py, 1319q)	N/A (accuracy gate)	accuracy=0.905 threshold=0.91 → failing before sweep	Log
nightly-amd-mi355x-disagg	collect-results	rollup	N/A	fails because the leg above failed	Log
nightly-amd-mi355x-disagg	dsv4pro-fp4-1k1k-1p1d	N/A (leg cancelled during setup)	N/A	cancelled ~7s (runner mia1-p01-g20 infra) — separate, benchmark never ran	Log

R291 · 🆕 · GitHub Actions internal platform error (gate jobs, 0 steps) — pr-test-amd · ❌ transient infra

• Status: isolated to the single run 30157771919 (Jul-25 12:22 UTC, 44 s window); prior run 12h earlier passed these jobs normally. Not a code regression.
• Top hypothesis: [FACT] GitHub Actions platform internal error failed check-changes+gate/finish jobs before any step ran (0 steps, no runner, identical annotation on all 4 jobs). Because check-changes never emitted its change-filter outputs, all 22 downstream AMD test jobs were skipped → this run has zero test signal. Disconfirming: (none — directly observable).
• In-flight fix: ❌ none (nothing to fix; platform-side). Suggested triage: simply rerun the run / update the branch; if it recurs, escalate to GitHub Actions infra.

Workflow	Job (shard)	Test File	Test Function	Error	Log
pr-test-amd	call-gate / pr-gate	N/A (non-test)	N/A	GitHub Actions has encountered an internal error…	Log
pr-test-amd	check-changes	N/A (non-test)	N/A	same internal error → 22 test jobs skipped	Log

R257 · MI35x (gfx950) GPU-hang / HW-exception family — hipErrorCapturedEvent · GPU mem-fault · runner-loss — nightly-amd + nightly-rocm720 · ⚠️ #31068/#32003

• Status: carry-over, 17th day (first Jul-09/10), never-passed on MI35x. Largest cluster: ~9 job-legs across both nightlies on the Jul-24 completed runs. Model-/workflow-independent; alternates on identical SHAs → environment/HW over a bisectable code regression.
• Top hypothesis: [MEDIUM] a TP/MoE collective (aiter custom_all_reduce / IPC-meta gather) is captured into the decode HIP graph; the RCCL ProcessGroupNCCL watchdog's isCompleted()→hipEventQuery on that event is illegal mid-capture → hipErrorCapturedEvent → uncaught → SIGABRT (exit -6); compounded by MI35x node/driver hangs & runner-loss. Disconfirming: alternates pass/fail on identical SHAs; Jul-13 showed a distinct GPU Hang exit 134 → not a clean code regression.
• In-flight fix: ⚠️ #31068 (hung-GPU pre-flight recovery), #32003 (route scheduled jobs to MI300). Suggested triage: make MoE/TP all-reduce capture-safe on ROCm; chase #31068/#32003; validate MI35x pool health.

Workflow	Job (shard)	Test File	Test Function	Error	Log
nightly-test-amd	nightly-8-gpu-mi35x-qwen3-235b-mxfp4	test_qwen3_instruct_mxfp4.py	TestQwen3Instruct2507MXFP4.setUpClass	hipErrorCapturedEvent during decode graph capture → -6	Log
nightly-test-amd	nightly-2-gpu-mi35x-glm51-mxfp4	test_qwen3_instruct_mxfp4.py	setUpClass	hipErrorCapturedEvent → -6	Log
nightly-test-amd-rocm720	nightly-8-gpu-mi35x-qwen3-235b-mxfp4-rocm720	test_qwen3_instruct_mxfp4.py	setUpClass	hipErrorCapturedEvent (aiter custom_all_reduce) → -6	Log
nightly-test-amd-rocm720	nightly-accuracy-8-gpu-mi35x-rocm720	test_qwen3_moe_eval_mi35x.py	test_qwen3_moe_accuracy	GPU memory access fault on Qwen3-30B-A3B prefill → abort	Log
nightly-test-amd-rocm720	nightly-2-gpu-mi35x-glm51-mxfp4-rocm720	test_glm51_mxfp4_tp2_gsm8k_mi35x.py	setUpClass	runner lost communication (log blob missing)	Log

R270 · Mamba JIT transfer kernel NameError on ROCm (CUDA-only import gate) — nightly-amd + nightly-rocm720 kernel · ✅ #31741 (blocked)

• Status: carry-over, 10th day (first Jul-16), never-passed on AMD. 12 NameError subtests per kernel leg; identical on both Jul-24 completed runs.
• Top hypothesis: [FACT] transfer_kv_mamba_* imports gated behind if _is_cuda: while the call site is reachable on HIP → NameError at runtime. Disconfirming: (none — directly observable).
• In-flight fix: ✅ #31741 (open since Jul-20, mergeable_state: blocked). Suggested triage: rebase/merge #31741, then verify the JIT kernel actually compiles under ROCm, not just that the import resolves.

Workflow	Job (shard)	Test File	Test Function	Error	Log
nightly-test-amd	nightly-test-1-gpu-kernel	test/registered/kernels/ops/mamba/test_transfer_mamba.py	12 subtests (roundtrip/single_item/full_indices)	NameError: transfer_kv_mamba_lf_pf → exit 255	Log
nightly-test-amd-rocm720	nightly-test-1-gpu-kernel-rocm720	test_transfer_mamba.py	same 12 subtests	same	Log

R289 · DeepSeek-V3.2 MTP GPU hang → scheduler watchdog timeout (MI35x, DSA backend) — nightly-amd · ⚠️ intermittent, no fix

• Status: carry-over from Jul-24 (now 2nd data point), confirmed intermittent — passes ~4 of 6 recent nights; the identical signature already occurred Jul-22 on an earlier SHA, so no Jul23→24 commit introduced it. Not a fresh regression.
• Top hypothesis: [LOW] intermittent MI35x (gfx950) GPU hang in the MTP draft-extend decode path — CPU parked on a HIP D2H sync (dsa_backend.py:793 .item()); classic GPU-side stall. Disconfirming: no window commit touches dsa_backend.py/eagle_worker; the .item() is where CPU waits on the GPU, not the root cause.
• In-flight fix: ❌ none (#32209 is a different model/config). Suggested triage: rerun 2–3× same SHA to confirm flakiness before any bisect; sync with the MI35x debug-branch owners.

Workflow	Job (shard)	Test File	Test Function	Error	Log
nightly-test-amd	nightly-accuracy-8-gpu-mi35x-deepseek-v32-mtp	test/registered/amd/accuracy/mi35x/test_deepseek_v32_mtp_eval_mi35x.py	TestDeepseekV32TPMTP.test_a_gsm8k	GSM8K stalls at 128/200 → watchdog → KeyError:'answer' → exit 255	Log

Known stable / recurring clusters (carrying over, no new action today) · click to expand

Cluster	Workflow · Rep. job	Test / fn	Status	In-flight fix
R273 accuracy near-misses: Qwen3-4B missing-threshold config gap, Mixtral/Mistral gsm8k answer-extraction, VLM MMMU shortfall, MiniMax-M2.5 (~0.929), DeepSeek-R1-MXFP4 MTP (~0.85), DeepSeek-V3.2 tilelang (~0.86–0.90)	nightly-amd acc-2gpu · nightly-rocm720 dsv32	test_gsm8k_eval_amd.py / test_*mxfp4*.py / test_vlms_mmmu_eval_amd.py	marginal/boundary gates, recurring	⚠️ recalib precedent #31702; Qwen3-4B gap #26730
R262 InternVL2_5-2B CLIPImageProcessor has no .tokenizer — server crash	nightly-rocm720 vlm	test_vlms_mmmu_eval_amd.py (InternVL2_5-2B)	recurring since Jul-14	❌
DeepSeek-V4-Pro NextN/MTP server-launch timeout (slow CPU FP8 wo_a dequant)	nightly-rocm720 dsv4-pro-mtp	test_deepseek_v4_pro_*_mtp.py	recurring (triton-only)	❌
Diffusion (multimodal-gen) family: 1-/2-GPU runner-loss, 60/90-min step timeout, FLUX.2 FP8 HIPBLAS_STATUS_NOT_SUPPORTED, update_weights_from_disk 400 shape-mismatch	pr-test-amd mm-gen 1gpu (0) · pr-rocm720 mm-gen 1gpu (0) ×many	test_server_1_gpu.py / test_server_2_gpu.py	never-passed/recurring	⚠️ skip #31925
Disaggregation family: Mooncake KV-transfer timeout, PD+PP event-loop hang (KeyError:'answer'), Kimi-K2.5-MXFP4 load monitored_barrier SIGKILL	pr-test-amd disagg mi35x · stage-c kimi	test_disaggregation_pp.py / test_kimi_k25_mxfp4_bcg_mi35x.py	recurring/flaky	⚠️ #31800
ModuleNotFoundError: msgpack on ROCm 7.2 image (diffusion extra not installed)	pr-rocm720 mm-gen-unit	test_msgpack_roundtrip_*	never-passed since Jul-11	✅ #31899
MXFP4 GSM8K just below strict gate (Qwen3-30B-A3B online FP8→MXFP4 requant)	pr-rocm720 stage-b-small-mi35x	test_*_mxfp4_*.py	borderline eval flake	❌
Triton offline throughput below AMD 3500 tok/s gate	pr-rocm720 stage-b-1gpu-large	test_bench_serving_*	never-passed	❌
GPU-Hang HW-exception / HSA-OOM (infra): sgl-kernel build exit-134, unified-radix 2-GPU watchdog, DeepSeek-V3.2 TP8 OUT_OF_RESOURCES	pr-test-amd stage-b-1gpu-small (8) · pr-rocm720 stage-c-8gpu	infra/timeout	recurring	⚠️ #31068, #32003

Workflow drill-down · click to expand

nightly-test-amd · latest completed 30115195247 (Jul-24) · 8 failures: R257 (qwen3-235b-mxfp4 ×2 legs, hipErrorCapturedEvent) · R270 (kernel) · R289 (dsv32-mtp watchdog hang) · R273 (acc-2gpu Mixtral+Qwen3-4B config, acc-2gpu-vlm MMMU, minimax-m25, nightly-4-gpu GLM-4.1V MMMU timeout). Jul-25 30168349328 IN-FLIGHT.

nightly-test-amd-rocm720 · latest completed 30115085856 (Jul-24) · 10 failures: R270 (kernel) · R257 (qwen3-235b-mxfp4, acc-8gpu mem-fault, glm51 runner-loss) · R262 (InternVL2_5) · R273 (dsv32, dsr1-mxfp4 MTP, minimax-m25, Mixtral/Qwen3-4B) · dsv4-pro-mtp launch timeout · dsv4-flash. R288 (DCP hang) did NOT recur — qwen35-triton-dcp absent from today's failures. Jul-25 30168262867 IN-FLIGHT.

nightly-amd-mi355x-disagg · 30143908399 (Jul-25) · 3 failures = R290 🆕 (dsv4flash-fp8 gate 0.905<0.91) + collect-results rollup + 1 unrelated leg cancellation (runner mia1-p01-g20 infra). Was green Jul-19→24.

pr-test-amd · latest completed 30157771919 (Jul-25 12:19) · R291 🆕 GHA platform error → 4 gate/finish jobs failed, 22 test jobs skipped (no signal). Prior run 30136388246 (Jul-25 00:29): diffusion + disagg families only.

pr-test-amd-rocm720 · latest completed 30115153295 (Jul-24) · 8 failures: diffusion family (FLUX.2 FP8/timeout/runner-loss) · msgpack ModuleNotFound · triton throughput gate · MXFP4 borderline · stage-c-8gpu HSA-OOM (DeepSeek-V3.2). Jul-25 30168318210 IN-FLIGHT.

How this report is generated

• Only status == "completed" runs/jobs counted in trends. Jul-25 nightly-amd/rocm720 and pr-rocm720 runs are queued (IN-FLIGHT) → excluded; latest completed nightlies are the Jul-24 runs. mi355x-disagg Jul-25 and pr-test-amd Jul-25 12:19 completed and are used directly.
• NEW today: R290 mi355x GSM8K gate miss (workflow flipped green→red, marginal/flaky 3/13, candidate #31727); R291 GitHub Actions platform error (pr-test-amd, isolated single run, 22 test jobs skipped).
• Resolved/not-reproduced: R288 DCP all-reduce hang — qwen35-triton-dcp-rocm720 passed in the Jul-24 completed run.
• Carry-over: R257 MI35x hang family (17th day), R270 mamba NameError (10th day, #31741 blocked), R273 accuracy near-misses, R262 InternVL, R289 DSV3.2-MTP (confirmed intermittent).
• Infra: release-docker-amd-rocm720-nightly startup_failure (Jul-25) — platform-side, 0 jobs, no cluster.
• Confidence: FACT/HIGH/MEDIUM/LOW/SPECULATION. Bot does NOT assign Priority — engineers decide from cluster size + persistence + fix availability.

---

Generated by amd-bot · last updated 2026-07-25 23:45 UTC

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-07-25 23:45 UTC)

---

CI Monitor — 2026-07-25

Repo: sgl-project/sglang

Monitored Workflows:

• nightly-test-amd.yml
• nightly-test-amd-rocm720.yml
• release-docker-amd-nightly.yml
• release-docker-amd-rocm720-nightly.yml
• nightly-amd-mi355x-disagg.yml
• amd-aiter-scout.yml
• pr-test-amd.yml
• pr-test-amd-rocm720.yml

Per-workflow failure reports are appended as comments below; the Daily Cross-Workflow Summary is rendered above this section.

[amd-bot]

nightly-test-amd.yml — 13 failure(s)

Run started (UTC): 2026-07-24 18:00 UTC
Last scanned (UTC): 2026-07-25 23:43 UTC
Commits: sglang 69a3c54

Summary

Counts: 13 failures · 9 clusters · 7 carrying over (≥3 days) · 1 newly-surfaced (Qwen3-4B threshold gap) · spans 2 runs (30115195247 Jul-24, 30168349328 Jul-25)

#	Cluster	Job	Test File	Test Function	Status	Hypothesis (confidence)
1	HIP capturing-stream crash	nightly-8-gpu-mi35x-qwen3-235b-mxfp4	test/registered/amd/test_qwen3_instruct_mxfp4.py	TestQwen3Instruct2507MXFP4.setUpClass	Persistent since Jul-09	[MEDIUM] TP collective captured into decode HIP graph; RCCL watchdog hipEventQuery illegal in capture → SIGABRT
2	HIP capturing-stream crash	nightly-2-gpu-mi35x-glm51-mxfp4	test/registered/amd/test_qwen3_instruct_mxfp4.py	TestQwen3Instruct2507MXFP4.setUpClass	Persistent since Jul-09	[MEDIUM] same (job mislabeled; actually launches Qwen3-235B-mxfp4)
3	HIP capturing-stream crash	nightly-8-gpu-mi35x-qwen3-235b-mxfp4	test/registered/amd/test_qwen3_instruct_mxfp4.py	TestQwen3Instruct2507MXFP4.setUpClass	Persistent since Jul-09	[MEDIUM] aiter custom all-reduce ⨯ RCCL watchdog in capture
4	VLM MMMU below threshold (MI30x)	nightly-accuracy-2-gpu-vlm	test/registered/amd/accuracy/mi30x/test_vlms_mmmu_eval_amd.py	test_mmmu_vlm_models	5/11 models, since ≥Jul-16	[MEDIUM] per-model thresholds > reproducible AMD score (calibration)
5	VLM MMMU below threshold (MI30x)	nightly-accuracy-2-gpu-vlm	test/registered/amd/accuracy/mi30x/test_vlms_mmmu_eval_amd.py	test_mmmu_vlm_models	same 5 models	[MEDIUM] same calibration gap
6	VLM encoder-DP 3600s timeout	nightly-4-gpu	test/registered/vlm/test_encoder_dp.py	whole-file run	Intermittent (model-dependent)	[MEDIUM] workload legitimately exceeds 3600s/file cap
7	VLM encoder-DP 3600s timeout	nightly-4-gpu	test/registered/vlm/test_encoder_dp.py	TestVLMEncoderDP.test_vlm_mmmu_benchmark	fails when GLM-4.1V drawn	[HIGH] GLM-4.1V-9B-Thinking low decode tput (1–6 tok/s) > 3600s
8	gsm8k 2-gpu step hard-killed	nightly-accuracy-2-gpu	test/registered/amd/accuracy/mi30x/test_gsm8k_eval_amd.py	TestNightlyGsm8KEval.test_gsm8k_all_models	7/7 recent runs killed	[MEDIUM] full sweep exceeds runner wall-clock; step reclaimed (~55–71 min)
9	Mamba JIT transfer NameError (ROCm)	nightly-test-1-gpu-kernel	test/registered/kernels/ops/mamba/test_transfer_mamba.py	test_mamba_kernel_* (12/16)	Persistent since ≥Jul-21	[FACT] CUDA-only import guard; skipif uses torch.cuda.is_available() (True on ROCm)
10	EAGLE draft-extend gsm8k hang	nightly-accuracy-8-gpu-mi35x-deepseek-v32-mtp	mtp gsm8k accuracy	test_a_gsm8k	Intermittent (~4/6 pass)	[LOW] HIP dev→host sync stall in dsa_backend.py:793 (symptom, not cause)
11	MiniMax-M2.5 gsm8k marginal	nightly-4-gpu-mi35x-minimax-m25	test/registered/amd/accuracy/mi35x/test_minimax_m25_tp4_eval_mi35x.py	test_minimax_m25_tp4_accuracy	Flaky (0.928–0.936 vs 0.93)	[MEDIUM] threshold inside ±0.007 noise band
12	Qwen3-4B missing threshold (config gap)	nightly-accuracy-2-gpu	test/registered/amd/accuracy/mi30x/test_gsm8k_eval_amd.py	test_gsm8k_all_models (Qwen3-4B subTest)	Newly surfaced (Jul-24/25)	[FACT] no Qwen3-4B key in MODEL_SCORE_THRESHOLDS → threshold: N/A = auto-FAIL
13	MI35x runner lost communication	nightly-2-gpu-mi35x-glm51-mxfp4	N/A (infra)	N/A	This run (recurs on runner)	[FACT] runner lost comms; step frozen in_progress, log blob BlobNotFound

Cluster: HIP capturing-stream crash (hipErrorCapturedEvent)

• Affected: rows 1–3 — 8gpu-qwen3-235b, 2gpu-glm51 (Jul-24), 8gpu-qwen3-235b (Jul-25). All run test_qwen3_instruct_mxfp4.py::setUpClass.
• Shared facts: identical RCCL stack — ProcessGroupNCCL::Watchdog::runLoop → WorkNCCL::isCompleted → hipErrorCapturedEvent → SIGABRT (exit -6), fired during decode CUDA-graph capture on MI35x mxfp4 (tp=4, ep=2, attention-backend=aiter). Biggest blast radius; persistent since 2026-07-09 (last pass Jul-05).
• Top hypothesis: [MEDIUM] a TP collective (aiter custom all-reduce IPC-meta gather) is enqueued during capture; watchdog hipEventQuery on the capturing-stream event is illegal. Disconfirming: the soft-watchdog py-spy dump during weight load is a red herring, not the crash point.
• In-flight fix: ❌ none (#29946 targets the NV symm-mem path, not this AMD aiter path).
• Suggested triage: bisect 28749559519(pass, Jul-05)..first-fail Jul-09 window; on a branch try disabling aiter custom all-reduce during graph capture and re-run on MI35x.

Cluster: VLM MMMU below threshold on AMD MI30x

• Affected: rows 4–5 — same test_vlms_mmmu_eval_amd.py::test_mmmu_vlm_models, same 5 VLMs (Qwen2-VL, Qwen2.5-VL, Qwen3-VL-30B, MiniCPM-o, deepseek-vl2) under threshold in both runs.
• Top hypothesis: [MEDIUM] per-model thresholds in AMD_VLM_MODELS (test_vlms_mmmu_eval_amd.py:44-116) exceed reproducible AMD scores. Disconfirming: not yet A/B'd against a bisect confirming no real accuracy regression.
• In-flight fix: ❌ none. Triage: recalibrate thresholds vs measured AMD scores, OR bisect to rule out an inference regression before adjusting.

Cluster: VLM encoder-DP 3600s per-file timeout

• Affected: rows 6–7 — same test_encoder_dp.py. Passes when Qwen VLMs are drawn (Jul-21/23), times out on GLM-4.1V-9B-Thinking (Jul-22/24/25).
• Top hypothesis: [HIGH] GLM-4.1V-9B-Thinking long-reasoning generation + low decode throughput blows the 3600s cap. In-flight fix: ❌ none (#31735 only edits est_time, not the hard cap). Triage: raise --timeout-per-file for this file or split the GLM-4.1V case out.

Single-job clusters

• gsm8k 2-gpu hard-kill (row 8) — [MEDIUM] full sweep exceeds runner budget, step reclaimed before finalize; 7/7 recent runs. In-flight ❌. Triage: pull the full step-6 log tail (/actions/runs/30115195247/logs) for the kill signal (timeout vs OOM vs reclaim).
• Mamba NameError (row 9) — [FACT] CUDA-only import reached on ROCm via wrong skipif. In-flight fix ✅ #31741 (open Jul-20, blocked) — chase reviewers, do NOT duplicate.
• EAGLE gsm8k hang (row 10) — [LOW] HIP .item() sync stall in dsa_backend.py:793; intermittent (~4/6 pass), not a fresh regression (same signature Jul-22 on earlier SHA). In-flight ❌.
• MiniMax-M2.5 marginal (row 11) — [MEDIUM] 0.93 gate inside ±0.007 noise band. In-flight ❌. Triage: widen threshold or average N runs.
• Qwen3-4B missing threshold (row 12) — [FACT] no Qwen3-4B key → auto-FAIL despite score ~0.909. In-flight ⚠️ #26730 touches this test — verify it adds the key. Triage: add Qwen3-4B to MODEL_SCORE_THRESHOLDS.
• Runner lost comms (row 13) — [FACT] infra; sister deepseek-r1-mxfp4-tp2 passed on same runner label this run. In-flight ❌. Triage: compare launch args/memory footprint vs the passing sister suite; check runner health.

Suggested triage order

Rank	Triage step	Targets	Why this first
1	Unblock & merge #31741	row 9 (Mamba)	In-flight fix already exists; cheapest green
2	Bisect 28749559519..Jul-09; test aiter custom all-reduce vs capture	rows 1–3 (cluster A)	Largest cluster, persistent 16 days, blocks all MXFP4 nightlies
3	Add Qwen3-4B threshold key (verify #26730)	row 12	One-line config fix; [FACT] root cause
4	Recalibrate VLM MMMU thresholds after bisect check	rows 4–5	2-job cluster; confirm no real accuracy regression
5	Raise/split --timeout-per-file for GLM-4.1V encoder-DP	rows 6–7	[HIGH] cause identified; deflakes nightly-4-gpu

---

Job	Failed Steps	Started
nightly-8-gpu-mi35x-qwen3-235b-mxfp4	Accuracy Test + Performance Test MI35x (8-GPU Qwen3-235B-MXFP4)	2026-07-24T18:56
nightly-accuracy-2-gpu	N/A	2026-07-24T20:25
nightly-2-gpu-mi35x-glm51-mxfp4	N/A	2026-07-24T18:00
nightly-4-gpu	Nightly Test (4-GPU)	2026-07-24T18:00
nightly-4-gpu-mi35x-minimax-m25	Accuracy Test MI35x (4-GPU MiniMax-M2.5)	2026-07-24T20:53
nightly-accuracy-2-gpu-vlm	Nightly Accuracy Test (2-GPU VLM MMMU)	2026-07-24T19:06
nightly-test-1-gpu-kernel	Nightly Kernel Benchmark (1-GPU)	2026-07-24T19:46
nightly-accuracy-8-gpu-mi35x-deepseek-v32-mtp	Accuracy Test MI35x (8-GPU DeepSeek-V3.2 TP+MTP)	2026-07-24T21:34
nightly-2-gpu-mi35x-glm51-mxfp4	N/A	2026-07-25T17:47
nightly-4-gpu	Nightly Test (4-GPU)	2026-07-25T18:02
nightly-accuracy-2-gpu	N/A	2026-07-25T20:50
nightly-accuracy-2-gpu-vlm	Nightly Accuracy Test (2-GPU VLM MMMU)	2026-07-25T21:09
nightly-8-gpu-mi35x-qwen3-235b-mxfp4	Accuracy Test + Performance Test MI35x (8-GPU Qwen3-235B-MXFP4)	2026-07-25T23:16

Per-Job Analysis

nightly-8-gpu-mi35x-qwen3-235b-mxfp4 — failed step(s): Accuracy Test + Performance Test MI35x (8-GPU Qwen3-235B-MXFP4)

Commit Info

• sglang: 2428f561
• aiter: 9127c94a (v0.1.17.dev110+g9127c94a1, source: dev image preserved — [CI-AITER-CHECK] Dev/patched version detected → skipping rebuild; Dockerfile default was the same SHA)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/test_qwen3_instruct_mxfp4.py	TestQwen3Instruct2507MXFP4.setUpClass	Exception: Server process exited with code 1 → RuntimeError: Rank 0 scheduler died during initialization (exit code: -6)	Log

Ran 0 tests in 480.461s / FAILED (errors=1) — the server never came up, so no accuracy/perf test executed.

Failure Summary

The sglang serve launch for amd/Qwen3-235B-A22B-Instruct-2507-mxfp4 (TP=4, EP=2, --attention-backend aiter, gfx950/MI35x) crashed during startup. All four TP scheduler ranks reached decode CUDA-graph capture, then the RCCL ProcessGroupNCCL watchdog thread threw HIP error: operation not permitted on an event last recorded in a capturing stream (hipErrorCapturedEvent), which was uncaught and called terminate() → SIGABRT (exit -6). setUpClass therefore failed and the job exited 255.

Regression Status

Known recurring failure (never-passed in the queryable window).

Recent history of test_qwen3_instruct_mxfp4.py (TestQwen3Instruct2507MXFP4.setUpClass) in job nightly-8-gpu-mi35x-qwen3-235b-mxfp4 (most recent comparable completed event=schedule, branch=main runs):

Date	Run	Job	Test File Status	Failed Function	Error
Jul 24	30115195247	…mi35x-qwen3-235b-mxfp4	❌ Failed	setUpClass	hipErrorCapturedEvent → scheduler exit -6
Jul 23	30031602981	same	❌ Failed	setUpClass	hipErrorCapturedEvent → exit -6 (8×)
Jul 22	29944275726	same	❌ Failed	setUpClass	(job failure)
Jul 21	29855050299	same	❌ Failed	setUpClass	hipErrorCapturedEvent → exit -6 (8×)
Jul 19	29697529176	same	❌ Failed	setUpClass	(job failure)
Jul 18	29654601256	same	❌ Failed	setUpClass	(job failure)
Jul 17	29601593052	same	❌ Failed	setUpClass	hipErrorCapturedEvent → exit -6 (8×)
Jul 16	29521595306	same	⏭️ Cancelled	—	—
Jul 14	29355460805	same	❌ Failed	setUpClass	(job failure)
Jul 13	29273241962	same	❌ Failed	setUpClass	different: HW Exception … GPU Hang, exit 134

No passing run found in the window. The hipErrorCapturedEvent-during-capture signature is confirmed identical on Jul 17 / 21 / 23 / 24 (8 occurrences each). The Jul 13 run failed with a different symptom (GPU Hang), so the runner/gfx950 environment has shown more than one failure mode.

Failure Cluster

RCCL ProcessGroup watchdog HIP error during CUDA-graph capture (hipErrorCapturedEvent)

Facts (verified observations)

• server_args: tp_size=4, ep, attention_backend=aiter, mem_fraction_static=0.7735, Qwen3MoeForCausalLM, quant=quark (mxfp4), device gfx950/MI35x. Soft watchdog enabled (Set soft_watchdog_timeout since in CI).
• Weight load was slow (Load weight end. elapsed=366.76 s … 426.43 s). The sglang soft watchdog dump at 19:08:42 (watchdog_timeout=300, self.soft=True) fired during weight loading and dumped py-spy (all ranks in _load_w13 → memcpy_and_sync), but did not kill — weight loading continued to completion. This dump is a red herring, not the crash.
• KV cache allocated 19:10:57; decode CUDA-graph capture began 19:10:58 (Capture target decode CUDA graph begin. backend=full).
• At 19:11:14, all 4 ranks logged multimem all-gather disabled (module 'torch.distributed._symmetric_memory' has no attribute 'set_signal_pad_size'), immediately followed by:
  [E … ProcessGroupNCCL.cpp:2055] … Process group watchdog thread terminated with exception: HIP error: operation not permitted on an event last recorded in a capturing stream.
• Stack is entirely torch/RCCL: c10_hip_check_implementation → WorkNCCL::finishedGPUExecutionInternal → WorkNCCL::isCompleted → ProcessGroupNCCL::Watchdog::runLoop → terminate() → c10::DistBackendError → SIGABRT (exit code: -6).
• Reproducibility: identical on 4 verified runs (Jul 17/21/23/24); 8 hipErrorCapturedEvent lines per run.

Hypothesised Causes (with confidence)

• [MEDIUM] A TP collective (all-gather / all-reduce) is being captured into the decode HIP graph, and the RCCL ProcessGroupNCCL watchdog thread's periodic isCompleted() → hipEventQuery on that work's event is illegal while the stream is capturing → hipErrorCapturedEvent, uncaught → SIGABRT.
  • Supporting: stack trace is 100% in ProcessGroupNCCL::Watchdog::runLoop → finishedGPUExecutionInternal; error text literally says "event last recorded in a capturing stream"; timing lands exactly at capture start (begin 19:10:58, crash 19:11:14) right after the multimem all-gather disabled fallback line on all ranks.
  • Disconfirming: from the log alone the exact captured collective cannot be pinned; this is a torch/RCCL-vs-HIP-graph interaction not clearly attributable to a specific recent sglang commit; the same job also failed Jul 13 with an unrelated GPU Hang, indicating the gfx950/MI35x runner or RCCL/torch build itself may be unstable.
• [LOW] gfx950/MI35x runner or RCCL/torch driver instability (the Jul 13 GPU Hang, exit 134 on the identical job/config points to environment flakiness independent of this code path).

No hypothesis above MEDIUM; a definitive root cause needs a controlled repro with AMD_SERIALIZE_KERNEL=3 and identification of which collective is captured.

In-flight Fix Check

• In-flight fix: ❌ none found — search for open PRs matching hipErrorCapturedEvent returned 0. Tangentially-related graph-capture-vs-collective PRs exist but none target this signature/job: #29946 (fall back to NCCL all-reduce for symm-mem during CUDA-graph capture), #32106 (skip EPLB pass during CUDA-graph capture). API calls: search/issues q=…+hipErrorCapturedEvent (0) and …+capturing+stream (26, none matching this job).

Suggested Triage Steps

• Reproduce on an MI35x/gfx950 container at SHA 2428f561 with AMD_SERIALIZE_KERNEL=3 and TORCH_USE_HIP_DSA to get an accurate synchronous stack and identify the captured collective.
• Test whether excluding the TP collective from decode graph capture (or the RCCL watchdog querying it) avoids the fault — e.g. verify whether disabling the PG watchdog event-query during capture, or forcing the collective outside the captured region, clears it; confirm on test_qwen3_instruct_mxfp4.py.
• Since this job has never passed in the queryable window (no pass_sha), a main-branch bisect has no green anchor — instead determine whether this MI35x mxfp4 config ever passed; if not, treat as a new-runner bring-up issue rather than a regression, and disable/quarantine the job until the graph-capture + RCCL-watchdog interaction is fixed.
• Cross-check the Jul 13 GPU Hang separately — if the runner intermittently hangs, hardware/driver health on linux-mi35x-gpu-8-* should be validated before attributing all failures to the capture path.

Status

• Persistence: Failing every completed run in the window; hipErrorCapturedEvent signature confirmed on Jul 17, 21, 23, 24. No passing run found; Jul 13 failed with a different GPU Hang symptom.
• Scope: Single job nightly-8-gpu-mi35x-qwen3-235b-mxfp4 (gfx950/MI35x, Qwen3-235B mxfp4, TP=4/EP=2, aiter backend). Blocks the MI35x Qwen3-235B-MXFP4 accuracy+perf signal entirely (0 tests run).
• In-flight fix: ❌ none found — needs new investigation/PR.

nightly-accuracy-2-gpu — failed step(s): N/A

Commit Info

• sglang: 2428f561
• aiter: 9127c94a (v0.1.17.dev110+g9127c94a1; source: Dev/patched CI image, preserved — AITER_COMMIT_OVERRIDE empty)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi30x/test_gsm8k_eval_amd.py	TestNightlyGsm8KEval.test_gsm8k_all_models	Test step killed mid-run (never finalized) during mistralai/Mixtral-8x7B eval; job → failure	Log

Non-test observation (NOT the job-failing cause): Qwen/Qwen3-4B reported Status: ❌ FAILED with threshold: N/A despite Score: 0.913 — see Facts. Log

Failure Summary

The single suite test_gsm8k_all_models iterates ~14+ instruct models sequentially. The captured log shows Llama-3.1-8B (0.811 ✅), Qwen3-8B (0.911 ✅), Qwen3-4B (0.913, threshold N/A ❌-print), Llama-3.1-70B (0.932 ✅) all completing, then the log cuts off during mistralai/Mixtral-8x7B-Instruct-v0.1 evaluation (~59 min into the test step). The GitHub API reports step 6 "Nightly Test (2-GPU)" as still in_progress (never got a conclusion) and the post-steps as pending, while the job overall concluded failure — the signature of the runner/process being hard-killed, not a clean test exit.

Regression Status

Known recurring failure / Never-passed (no green run found in query window).

Recent history of test_gsm8k_eval_amd.py (test_gsm8k_all_models) in job nightly-accuracy-2-gpu (most recent comparable completed scheduled runs, same branch main, event=schedule):

Date	Run	Job	Test File Status	Test-step state	Job dur
Jul 24	30115195247	nightly-accuracy-2-gpu	❌ Failed	in_progress (never finalized)	~71 min
Jul 23	30031602981	nightly-accuracy-2-gpu	❌ Failed	in_progress (never finalized)	~63 min
Jul 22	29944275726	nightly-accuracy-2-gpu	❌ Failed	in_progress (never finalized)	~55 min
Jul 21	29855050299	nightly-accuracy-2-gpu	❌ Failed	in_progress (never finalized)	~59 min
Jul 19	29697529176	nightly-accuracy-2-gpu	❌ Failed	in_progress (never finalized)	~68 min
Jul 18	29654601256	nightly-accuracy-2-gpu	❌ Failed	in_progress (never finalized)	~70 min
Jul 17	29601593052	nightly-accuracy-2-gpu	❌ Failed	(step absent — killed)	~64 min

No passing run for this job found in the query window (last ~8 comparable scheduled runs). First-observed failure predates the window. The failure signature — test step never finalizing, job dying at a variable 55-71 min mark — is identical across all runs.

Failure Cluster

nightly-accuracy-2-gpu test step hard-killed before completion (gsm8k full-model sweep never finishes)

Facts (verified observations, no interpretation)

• Job 89554026354 conclusion = failure; step 6 "Nightly Test (2-GPU)" status = in_progress, conclusion = None; step "Post Checkout code" = pending (verified via /actions/jobs/89554026354 API). [FACT]
• The captured log (4.2 MB, 18647 lines) ends abruptly at 21:17:03 inside the Mixtral-8x7B decode loop (aiter GEMM spam), with no SUMMARY table, no AssertionError, and no github_summary.md echo. The test step began 20:37:27 and the job ended 21:36:48. [FACT]
• The workflow invokes the suite with --continue-on-error (schedule event) and --timeout-per-file 7200; the nightly-accuracy-2-gpu job has no timeout-minutes at step or job level (.github/workflows/nightly-test-amd.yml:261-268). Because --continue-on-error is set, per-model threshold assertions do NOT fail the step — so the job failure is not from an accuracy assertion. [FACT]
• Qwen/Qwen3-4B is present in DEFAULT_MODEL_NAME_FOR_NIGHTLY_EVAL_TP1 (python/sglang/test/test_utils.py:153-155) but is absent from MODEL_SCORE_THRESHOLDS (test_gsm8k_eval_amd.py:37-67). [FACT]
• Consequences of that gap, per source (test_gsm8k_eval_amd.py:273-301): threshold = MODEL_SCORE_THRESHOLDS.get(model) → None; the retry loop for attempt in range(3) never early-breaks (line 279 if threshold and ... is falsy), so gsm8k ran 3 times (log scores 0.910, 0.910, 0.913 at lines 11147/12053/13602 — ~4 min wasted); passed = threshold and score >= threshold → None → prints Status: ❌ FAILED (line 13621) despite the high score. [FACT]
• In check_model_scores (test_gsm8k_eval_amd.py:165-177) a missing threshold maps to ⚠️ NO THRESHOLD and is not appended to failed_models, so Qwen3-4B does not contribute to the AssertionError at line 199-201. [FACT]
• Models observed completing before the kill: Llama-3.1-8B 0.811 (thr 0.80 ✅), Qwen3-8B 0.911 (thr 0.76 ✅), Qwen3-4B 0.913 (thr N/A), Llama-3.1-70B 0.932 (thr 0.89 ✅). Server startups were expensive: Llama-70B 621.8 s, Mixtral 421.6 s. Many models (Qwen2-57B, all FP8 TP1/TP2, AMD TP1/TP2 incl. Qwen3-30B) had not yet run when the log ends. [FACT]

Hypothesised Causes (with confidence, with disconfirming evidence)

• [MEDIUM] The full gsm8k sweep cannot finish within the runner's effective wall-clock budget, and the job is reclaimed/killed (~55-71 min) before the test step finalizes → failure.
  • Supporting: identical signature across ≥7 consecutive scheduled runs (step never finalizes, post-steps pending); the model list is large with several 400-620 s startups; the log always dies mid-evaluation with no SUMMARY. The Qwen3-4B 3×-retry bug adds ~4 min of pure waste to every run.
  • Disconfirming: the death time is variable (55-71 min), which is inconsistent with a fixed timeout-minutes (and none is configured). A variable kill could equally be a runner reclaim, container/GPU crash, or OOM rather than pure wall-clock. The truncated log contains no explicit timeout/OOM/kill marker to confirm.
  • How to verify: fetch the complete step-6 log via the run-logs zip (/actions/runs/30115195247/logs) and inspect the final 200 lines for a timeout/Killed/exit -9/OOM signal; check the self-hosted linux-mi325-2gpu-sglang runner's job-duration/idle limits.
• [LOW] A hang or GPU crash on a specific model (e.g. Mixtral-8x7B TP2, where the log consistently dies) stalls the suite until the runner kills it.
  • Supporting: today's log truncates precisely during Mixtral eval.
  • Disconfirming: other runs die at different models/times, so it is not Mixtral-specific; no crash stack trace captured.
  • How to verify: run test_gsm8k_eval_amd.py locally on MI325 2-GPU limited to mistralai/Mixtral-8x7B-Instruct-v0.1 and watch for hang/crash.
• [FACT, separate issue] Qwen/Qwen3-4B missing from MODEL_SCORE_THRESHOLDS — real harness defect (misleading ❌ print + 3× wasted eval time), but not the job-failing cause (see Facts).

In-flight Fix Check

• Job-kill / suite-duration: In-flight fix: ❌ none found matching "nightly accuracy 2-gpu timeout" in open PRs.
• Qwen3-4B threshold gap: In-flight fix: ⚠️ possibly related — #26730 "test(amd): run full GSM8K for accuracy checks" is an open PR touching this exact AMD gsm8k accuracy test; verify whether it adds a Qwen3-4B threshold or otherwise reshapes the sweep. No open PR explicitly adds the missing Qwen/Qwen3-4B threshold key.

Suggested Triage Steps

• Download the complete step-6 log (curl -L .../actions/runs/30115195247/logs, unzip, read the nightly-accuracy-2-gpu file tail) to capture the definitive kill signal (timeout vs. OOM vs. runner reclaim) — the /jobs/{id}/logs text endpoint truncated at ~18.6k lines.
• Confirm the linux-mi325-2gpu-sglang runner's max job duration; if a runner-level limit is reclaiming the job, either add an explicit timeout-minutes for observability or shard the model sweep across jobs so each fits the budget.
• Add the missing "Qwen/Qwen3-4B" threshold to MODEL_SCORE_THRESHOLDS (or drop it from DEFAULT_MODEL_NAME_FOR_NIGHTLY_EVAL_TP1) to stop the 3× retry waste and the misleading ❌ FAILED print; coordinate with #26730.
• If a specific large model (Mixtral-8x7B / Qwen2-57B / 70B-class) is confirmed to hang, quarantine it on AMD 2-GPU until fixed.

Status

• Persistence: nightly-accuracy-2-gpu has failed on every completed scheduled run sampled Jul 17 → Jul 24 (≥7 runs), each with the test step never finalizing and the job killed 55-71 min in. No green run in the query window.
• Scope: single job (nightly-accuracy-2-gpu) in nightly-test-amd.yml; blocks the 2-GPU AMD gsm8k accuracy signal. The sibling nightly-accuracy-8-gpu passed in this run, so the issue is specific to this job/runner/sweep.
• Blocked work: no accuracy sign-off for the TP1/TP2 instruct-model gsm8k sweep on AMD MI325 2-GPU.
• In-flight fix: none for the job kill; #26730 possibly relevant to the Qwen3-4B threshold gap.

nightly-2-gpu-mi35x-glm51-mxfp4 — failed step(s): N/A

Commit Info

• sglang: 2428f561
• aiter: 9127c94a — v0.1.17.dev110+g9127c94a1 (source: dev build preserved in CI image; Dockerfile default, AITER_COMMIT_OVERRIDE empty)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/test_qwen3_instruct_mxfp4.py	TestQwen3Instruct2507MXFP4.setUpClass	Exception: Server process exited with code 1 → RuntimeError: Rank 0 scheduler died during initialization (exit code: -6) (SIGABRT)	Log

The server never came up (Ran 0 tests in 480s); the failure is a scheduler crash during CUDA-graph capture, not an accuracy/assert failure. Note the job is named …glm51-mxfp4 but actually launches amd/Qwen3-235B-A22B-Instruct-2507-mxfp4 (--tp 4 --ep 2 --attention-backend aiter).

Failure Summary

During server init on MI35x, rank 0's scheduler aborted (exit -6) while capturing decode CUDA graphs. The fatal thread was inside aiter's custom all-reduce IPC-meta gather, and the RCCL watchdog thread simultaneously threw HIP error: operation not permitted on an event last recorded in a capturing stream (hipErrorCapturedEvent). All 4 ranks aborted, so popen_launch_server timed out and setUpClass errored.

Regression Status

Known recurring failure — job-level persistent since 2026-07-09. Last job-level pass 2026-07-05 (run 28749559519). The test did pass regularly in June, so this is a regression, not never-passed. ⚠️ Caveat: GitHub log blobs for runs older than ~2 days have expired (BlobNotFound), so historical rows below are job-level status only — I could not re-verify that each historical failure carries the same HIP-capture symptom. This job runs exactly one test file, so job status maps 1:1 to test_qwen3_instruct_mxfp4.py.

Recent comparable completed scheduled runs (nightly-test-amd.yml, event=schedule, branch=main; cancelled runs excluded):

Date	Run	Test File Status	Note
Jul 24	job	❌ Failed	HIP capture crash (verified this run)
Jul 23	job	❌ Failed	job-level (log expired)
Jul 22	job	❌ Failed	job-level
Jul 21	job	❌ Failed	job-level
Jul 19	job	❌ Failed	job-level
Jul 18	job	❌ Failed	job-level
Jul 17	job	❌ Failed	job-level
Jul 16	job	❌ Failed	job-level
Jul 14	job	❌ Failed	job-level
Jul 13	job	❌ Failed	job-level
Jul 12	run	❌ Failed	job-level
Jul 11	run	❌ Failed	job-level
Jul 10	run	❌ Failed	job-level
Jul 09	run	❌ Failed	first of persistent streak
Jul 05	run	✅ Passed	last known pass

(Earlier July was flaky — pass on Jul 04–05, fail on Jul 02–03, pass Jun 28–29 — so the test was intermittently unstable on this config before going persistently red on Jul 09.)

Failure Cluster

hipErrorCapturedEvent: RCCL collective on an event recorded in a capturing stream during CUDA-graph capture

Facts (verified observations)

• Crash step stack (verbatim from log): cuda_graph_setup.py:106 capture_cuda_graphs → model_runner.py:904 init_cuda_graphs → tp_worker.py:396 init_cuda_graphs → scheduler.py:846 init_all_cuda_graphs → scheduler.py:859 init_model_worker → scheduler.py:431 __init__.
• Fatal (main) thread was in /sgl-workspace/aiter/aiter/dist/device_communicators/custom_all_reduce.py:336 _gather_ipc_meta ← :303 flush_graph_buffers — i.e. aiter's own custom all-reduce, not sglang's custom_all_reduce.py.
• Watchdog thread that raised: c10d::ProcessGroupNCCL::WorkNCCL::isCompleted() → finishedGPUExecutionInternal() (ProcessGroupNCCL.cpp:2061), error text operation not permitted on an event last recorded in a capturing stream, hipErrorCapturedEvent.
• All 4 ranks (PG ID 5) aborted with Fatal Python error: Aborted; parent got EOFError on the scheduler pipe → RuntimeError: Rank 0 scheduler died (exit code: -6).
• Immediately preceding line on every rank: multimem all-gather disabled (module 'torch.distributed._symmetric_memory' has no attribute 'set_signal_pad_size') — a benign symm-mem capability disable, not the crash itself.
• aiter version unchanged relative to Dockerfile expectation ([CI-AITER-CHECK] Dev/patched version detected: v0.1.17.dev110+g9127c94a1 → skipping rebuild); no override in play.

Hypothesised Causes (with confidence)

• [MEDIUM] An RCCL collective (all_gather inside aiter _gather_ipc_meta/flush_graph_buffers) is enqueued during decode-graph capture; the resulting WorkNCCL event is recorded on the capturing stream, and the ProcessGroupNCCL watchdog polling isCompleted() hits hipErrorCapturedEvent.
  • Supporting: main-thread fatal stack is precisely aiter _gather_ipc_meta under capture_cuda_graphs; watchdog stack is WorkNCCL::isCompleted → finishedGPUExecutionInternal; error string explicitly names "event recorded in a capturing stream".
  • Disconfirming: the crashing code lives in aiter (dev build baked into the CI image), which is not bisectable from the sglang tree; sglang's own custom_all_reduce.register_graph_buffers already defers the IPC gather to after capture (custom_all_reduce.py:185,234), so sglang's path is not the one crashing; cannot confirm the Jul-09 onset shares this exact symptom (logs expired).
  • How to verify: rerun the server with --disable-custom-all-reduce (flag exists, server_args.py:1884) or force an NCCL/RCCL all-reduce fallback; if capture completes, aiter's custom all-reduce IPC path during capture is confirmed as the trigger.
• [LOW] A sglang commit in the 48ba79c1..2e4d6368 window (Jul 05 last pass → Jul 09 first persistent fail, ~ tens of commits) altered graph-capture ordering and began exercising the aiter collective during capture.
  • Supporting: clean job-level pass→fail flip at that boundary.
  • Disconfirming: symptom is inside aiter, not sglang; test was already flaky earlier in July; historical symptom unverifiable (logs expired).
  • How to verify: bisect 48ba79c1..2e4d6368 running this test on MI35x.
• [LOW] A change in the aiter dev build's custom_all_reduce.flush_graph_buffers behavior.
  • Supporting: crash frame is aiter code.
  • Disconfirming: the same image aiter version is reported across the failing window; not bisectable from this repo.
  • How to verify: A/B two aiter builds via AITER_COMMIT_OVERRIDE around the Jul-09 boundary.

No hypothesis exceeds MEDIUM; a definitive answer needs the --disable-custom-all-reduce rerun above.

In-flight Fix Check

• In-flight fix: ❌ none found directly. The closest open PR, #29946 "fall back to NCCL all-reduce for symm-mem during CUDA graph capture" (open since 2026-07-02), targets the NV --enable-symm-mem path, not this AMD aiter custom-all-reduce path, so it is unlikely to fix this. (Searched is:open PRs for capturing+stream, custom_all_reduce+capture, qwen3+instruct+mxfp4, hipErrorCapturedEvent.)

Suggested Triage Steps

• Reproduce with custom all-reduce disabled: launch amd/Qwen3-235B-A22B-Instruct-2507-mxfp4 --tp 4 --ep 2 --attention-backend aiter --disable-custom-all-reduce on MI35x; if capture succeeds, confirms the aiter IPC-gather-during-capture trigger.
• If confirmed, investigate why aiter flush_graph_buffers/_gather_ipc_meta runs a collective inside the capture region on this MoE-MXFP4 config, and whether sglang should guard the aiter custom all-reduce out of the capture stream (mirror sglang's own deferred register_graph_buffers).
• If the disable rerun still crashes, bisect sglang 48ba79c1..2e4d6368 on MI35x to locate the graph-capture ordering change.
• Separately confirm whether sibling job nightly-8-gpu-mi35x-qwen3-235b-mxfp4 (also persistently red) shares this cluster — if so, treat as one aiter-capture issue spanning both Qwen3-MXFP4 jobs.

Status (factual)

• Persistence: Job-level red every completed scheduled run since 2026-07-09 (last pass 2026-07-05); today's run confirmed as HIP-capture crash.
• Scope: nightly-2-gpu-mi35x-glm51-mxfp4 in nightly-test-amd.yml; a likely-related persistent failure exists in nightly-8-gpu-mi35x-qwen3-235b-mxfp4 (same model family/quant, unverified same symptom).
• Blocked work: Qwen3-235B-Instruct-2507-MXFP4 accuracy signal on MI35x.
• In-flight fix: ❌ none found directly (#29946 is a different, NV symm-mem path).

nightly-4-gpu — failed step(s): Nightly Test (4-GPU)

Commit Info

• sglang: 2428f561
• aiter: 9127c94a — v0.1.17.dev110+g9127c94a1 (source: dev image; [CI-AITER-CHECK] reports Dev/patched version detected → skipping rebuild, Dockerfile default was the same SHA)

Failed Tests

Test File	Test Function	Error	Log
test/registered/vlm/test_encoder_dp.py	whole-file run (python3 test_encoder_dp.py, multi-model VLM encoder-DP suite)	✗ TIMEOUT after 3600 seconds (est_time=500s), Test Summary: 0/1 passed, job exit code 255	Log

Failure Summary

The single enabled test in the nightly-4-gpu suite, test/registered/vlm/test_encoder_dp.py, was killed by the 3600s per-file timeout (elapsed=3600, estimated_time=500.0). The process was not hung — the server started successfully (The server is fired up and ready to roll! at 18:12:43) and was actively emitting aiter GEMM work right up to the kill at 19:09:36, so the file overran its wall-clock budget rather than deadlocking. The job then exited 255 (Exit code 255 indicates test failure (not network issue), not retrying).

Regression Status

Flaky / borderline timeout (test is newly introduced; 2 of its 3 nightly runs have timed out).

The test file was added 2026-07-22 by commit 0a6d1930 ([Attention Backend] Add HPC-Ops attention backend (#30540)), so runs before Jul 22 did not contain it.

Recent history of test/registered/vlm/test_encoder_dp.py in job nightly-4-gpu (all comparable completed event=schedule, branch=main runs since the test was introduced):

Date	Run	Job	Test File Status	Failed Function	Error
Jul 24	30115195247	nightly-4-gpu	❌ Failed	whole-file	TIMEOUT after 3600s (elapsed=3600)
Jul 23	30031602981	nightly-4-gpu	✅ Passed	—	1/1 passed, elapsed=2919s
Jul 22	29944275726	nightly-4-gpu	❌ Failed	whole-file	TIMEOUT after 3600s (elapsed=3600)
≤ Jul 21	—	—	⛔ N/A	—	test did not exist (added Jul 22)

First observed failure for this test file: 2026-07-22 (its very first nightly run). Only known pass: 2026-07-23, and that pass took 2919s — within ~680s of the 3600s cap. Lookback range: Jul 22–24 (the full life of the test); no stable green baseline exists.

Failure Cluster

VLM encoder-DP nightly test exceeds 3600s per-file timeout (borderline-slow, not a crash)

Facts (verified observations, no interpretation)

• ✗ TIMEOUT: /sglang-checkout/test/registered/vlm/test_encoder_dp.py after 3600 seconds; Test Summary: 0/1 passed; ##[error]Process completed with exit code 255 (current job log).
• Server reached readiness: [2026-07-24 18:12:43] The server is fired up and ready to roll!; a /health_generate returned 200 OK at 18:13:23. The first model launched was zai-org/GLM-4.1V-9B-Thinking (Glm4vForConditionalGeneration), tp_size=4, attention_backend='aiter', mm_enable_dp_encoder=True.
• Work continued until the kill: aiter GEMM log lines are present up to 19:09:35, immediately before kill_process_tree called ... 19:09:36 — the file was progressing, not stalled.
• Throughout the run the log repeats, for essentially every bf16 GEMM shape: not found tuned config in /tmp/aiter_configs/bf16_tuned_gemm.csv, will use default config! using torch solution:0 — the untuned Torch fallback GEMM path is in use.
• One transient Health check failed. Server couldn't get a response from detokenizer for last 20 seconds at 18:13:09 coincided with a 33.4s aiter JIT kernel build (finish build [mha_batch_prefill_...], cost 33.4s); the server recovered and served requests afterward (200 OK at 18:13:23), so this is not the cause of the timeout.
• Historical elapsed times for the same file: 3600s (Jul 22, timeout), 2919s (Jul 23, pass), 3600s (Jul 24, timeout). est_time registered for the file is 500s.
• The Jul 23 (pass) run's first model was Qwen2_5_VLForConditionalGeneration (weight load 225–227s); the Jul 22 and Jul 24 (fail) runs' first model was Glm4vForConditionalGeneration (weight load ~54s). Model set/ordering appears to differ across runs.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [MEDIUM] The test's total workload (multiple VLM models launched sequentially, each with tp=4 weight load + CUDA-graph capture + benchmark) legitimately exceeds the 3600s per-file cap on this MI325 4-GPU runner; the file is fundamentally borderline, not broken.
  • Supporting: Jul 23 passed at 2919s — only ~19% under the 3600s cap; Jul 22 and Jul 24 crossed it. est_time=500s is a gross underestimate (real time is 6–7×). Server never crashed; work progressed to the moment of the kill.
  • Disconfirming: run-to-run variance in which model runs first (Qwen2.5-VL vs GLM-4V) means the two timeouts may not be identical workloads; a genuine per-model hang (rather than aggregate slowness) cannot be fully excluded from the job log alone.
  • How to verify: re-run test/registered/vlm/test_encoder_dp.py alone on an MI325 4-GPU container with per-model timing/logging; confirm the sum of per-model durations approaches/exceeds 3600s and identify the slowest model.
• [LOW] The aiter bf16 GEMM untuned fallback (using torch solution:0, no entry in bf16_tuned_gemm.csv) inflates per-token latency enough to push the file over the cap.
  • Supporting: the fallback message fires for nearly every GEMM shape across the whole run; the Torch default path is slower than a tuned aiter kernel.
  • Disconfirming: the same untuned fallback was almost certainly present in the Jul 23 passing run (same aiter dev image lineage), so it cannot by itself flip pass→fail; it is at most a constant slow-down, not the trigger.
  • How to verify: A/B the file with a populated /tmp/aiter_configs/bf16_tuned_gemm.csv (tuned configs) vs empty, comparing total elapsed.
• [LOW] A specific model in the suite regressed in speed between Jul 23 and Jul 24 (sglang delta d0b96898..2428f561).
  • Supporting: Jul 24 is one nightly newer than the Jul 23 pass.
  • Disconfirming: Jul 22 (older than the pass) also timed out, so the failure predates any Jul 23→24 delta — a plain time-ordered regression does not fit. The alternating fail/pass/fail pattern points to variance/borderline budget, not a monotonic regression.
  • How to verify: git log --oneline d0b9689805..2428f561 -- python/sglang/srt/models/glm4v* python/sglang/srt/managers/ and check for VLM/encoder-DP perf changes; compare per-model elapsed across the three runs.

In-flight Fix Check

• In-flight fix: ❌ none found directly — search repo:sgl-project/sglang is:pr is:open test_encoder_dp and encoder_dp+timeout both returned 0 results.
• Possibly related (not a direct fix): #30008 [AMD] WIP - Set REQUEST_TIMEOUT=30 for AMD to deflake multimodal tests — targets per-request timeout deflaking for AMD multimodal tests, but does not raise the 3600s per-file cap that this failure hits. Treat as adjacent, not a resolution.

Suggested Triage Steps

• Re-run test/registered/vlm/test_encoder_dp.py in isolation on an MI325 4-GPU container at commit 2428f561 with per-model wall-clock logging to confirm it is aggregate slowness vs a single-model hang, and to see how far over 3600s it lands.
• If aggregate-slow is confirmed, decide between (a) raising --timeout-per-file / the file's registered timeout for this VLM suite on AMD, or (b) splitting test_encoder_dp.py into smaller per-model registered tests so each fits the budget — discuss with the test owner (added in #30540).
• Check whether populating /tmp/aiter_configs/bf16_tuned_gemm.csv with tuned configs for these VLM GEMM shapes meaningfully cuts runtime (measures the [LOW] aiter-fallback hypothesis).
• Track #30008 but do not rely on it to fix this file-level timeout.

Status (factual, no priority assignment)

• Persistence: Timed out on 2 of the 3 nightly nightly-4-gpu runs since the test was introduced 2026-07-22 (❌ Jul 22, ✅ Jul 23 at 2919s, ❌ Jul 24). No stable green baseline; the one pass was within ~680s of the cap.
• Scope: nightly-4-gpu job of nightly-test-amd.yml; this is the only enabled test in that suite (the other registered file, test_engine_dumper_comparator_e2e.py, is skipped), so the whole nightly-4-gpu job is red whenever this file times out.
• Blocked work: Blocks the AMD 4-GPU nightly VLM encoder-DP signal.
• In-flight fix: ❌ none found directly; #30008 is adjacent (per-request timeout deflake) but does not address the 3600s per-file cap.

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-07-25 23:43 UTC)

[amd-bot]

nightly-test-amd.yml — Per-Job Analysis (continued 2/3)

nightly-4-gpu-mi35x-minimax-m25 — failed step(s): Accuracy Test MI35x (4-GPU MiniMax-M2.5)

Commit Info

• sglang: 2428f561
• aiter: 9127c94a (source: Dockerfile default, AITER_COMMIT_OVERRIDE empty)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi35x/test_minimax_m25_tp4_eval_mi35x.py	test_minimax_m25_tp4_accuracy	AssertionError: Failed models: ['MiniMaxAI/MiniMax-M2.5 (TP4+UnifiedAttn+FP8KV)'] — GSM8K accuracy=0.929 threshold=0.93	Log

Failure Summary

The single test file in job nightly-4-gpu-mi35x-minimax-m25 ran MiniMax-M2.5 (TP=4, unified attention, FP8 KV cache) against the full 1319-question GSM8K completion benchmark and scored 0.929, missing the 0.93 threshold by 0.001. The benchmark itself completed cleanly (1319/1319 questions, invalid=0.000); the job failed purely on the accuracy gate, not a crash or infra error.

Regression Status

Known recurring failure (flaky / marginal accuracy — score oscillates around the 0.93 threshold).

Recent history of test_minimax_m25_tp4_eval_mi35x.py (test_minimax_m25_tp4_accuracy) in job nightly-4-gpu-mi35x-minimax-m25 (comparable completed scheduled runs, event=schedule, branch=main):

Date	Run	Test File Status	Accuracy	Error
Jul 24	30115195247	❌ Failed	0.929	below 0.93
Jul 23	30031602981	❌ Failed	0.929	below 0.93
Jul 22	29944275726	✅ Passed	0.933	—
Jul 21	29855050299	❌ Failed	0.928	below 0.93
Jul 20	29766761021	⏭️ Cancelled	—	run cancelled (excluded)
Jul 19	29697529176	✅ Passed	0.936	—
Jul 18	29654601256	✅ Passed	0.936	—
Jul 17	29601593052	✅ Passed	(pass)	—
Jul 14	29355460805	❌ Failed	N/A	different cluster — Fatal Python error: Aborted before accuracy check

First observed accuracy-miss for this test file in the window: Jul 21 (0.928). Last passing: Jul 22 (0.933). The score sequence 0.936 · 0.936 · 0.928 · 0.933 · 0.929 · 0.929 sits directly on top of the 0.93 gate. (Jul 14 was a distinct crash-signature failure, not the accuracy cluster.)

Failure Cluster

GSM8K accuracy marginally below threshold (MiniMax-M2.5 TP4 UnifiedAttn+FP8KV, ~0.001 miss).

Facts (verified observations)

• Log: accuracy=0.929 threshold=0.93 FAIL, then AssertionError: Failed models: ['MiniMaxAI/MiniMax-M2.5 (TP4+UnifiedAttn+FP8KV)']; job exit code 255, "not retrying" (test failure, not network).
• Benchmark fully completed: 100%|██████████| 1319/1319, invalid=0.000, latency=104.9s. No crash, no OOM.
• Threshold is hard-coded accuracy_threshold: float = 0.93 at test_minimax_m25_tp4_eval_mi35x.py:44,66; benchmark uses num_questions=1319 (GSM8K_NUM_QUESTIONS default, line 179).
• Score straddles the gate across runs: 0.928 (F), 0.929 (F), 0.929 (F), 0.933 (P), 0.936 (P), 0.936 (P). Binomial std error on 1319 questions ≈ √(0.93·0.07/1319) ≈ 0.007, so ±1σ noise spans ~0.923–0.937 — i.e. the threshold sits inside the noise band.
• aiter version unchanged across passing and failing runs (Dockerfile default 9127c94a, no override) — not a variable here.
• A transient Health check failed. Server couldn't get a response from detokenizer for last 20 seconds appeared at 21:21:18 during heavy load, but the server recovered and finished all 1319 requests; same warning also appears in runs that passed → benign, not the cause.

Hypothesised Causes (with confidence)

• [MEDIUM] Threshold set too tight for the model's true accuracy — marginal/flaky, not a code regression. The 0.93 gate lies within the ±0.007 sampling-noise band of the measured accuracy (0.928–0.936), so individual runs cross it randomly.
  • Supporting: score oscillates run-to-run with pass/fail interleaved (P,P,F,P,F,F); every failure misses by ≤0.002; benchmark completes with invalid=0.000.
  • Disconfirming: there is a mild downward drift (0.936→0.933→0.929) that could hint at a small real regression rather than pure noise — but the drift magnitude is within one std error and Jul 22 (0.933) still passed after any recent code landed.
  • How to verify: re-run the job 3–5× at the current head SHA; if accuracy clusters at ~0.929±0.005 with mixed pass/fail, it is threshold-marginal. Consider lowering threshold or raising num_questions to shrink variance.
• [LOW] Small real accuracy drift from a commit in d4801be447..2428f561 (85 commits). Candidate area: attention/FP8-KV path exercised by this config.
  • Supporting: monotone-ish drift 0.936→0.929 over the window; 0a6d1930 ("[Attention Backend] Add HPC-Ops attention backend", #30540) touches both this test and minimax model code.
  • Disconfirming: the test already failed at 0.928 on Jul 21, before 0a6d1930 landed (Jul 22 14:06 UTC), and the Jul 22 run passed (0.933) after it landed — so that commit does not correlate with the failures. No single commit tracks the pass/fail pattern.
  • How to verify: bisect d4801be447..2428f561 running this test on MI35x TP4, but only after confirming the failure is reproducible (not noise) via repeated runs.

No hypothesis above MEDIUM points to a specific commit; evidence favors a threshold-marginal flake over a code regression.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for minimax+m25+mi35x (3 hits: #30984, #27209, #20894) and minimax+accuracy+threshold (7 hits incl. #30613, #31820); none targets the MiniMax-M2.5 TP4 accuracy gate on MI35x.

Suggested Triage Steps

• Re-run job nightly-4-gpu-mi35x-minimax-m25 3–5× at head 2428f561 to measure the accuracy distribution; if it clusters at ~0.929 with mixed pass/fail, treat as threshold-marginal.
• If confirmed marginal: consider lowering accuracy_threshold (e.g. 0.92) or increasing GSM8K_NUM_QUESTIONS to shrink sampling variance, in test_minimax_m25_tp4_eval_mi35x.py:44/66/179.
• If the repeated runs show a stable ~0.929 (below the Jul 18–19 level of 0.936): bisect d4801be447..2428f561 focusing on attention / FP8-KV / MiniMax model commits, running this test per candidate.
• Ignore the transient detokenizer health-check warning — it is benign (also present in passing runs).

Status

• Persistence: Accuracy-miss cluster intermittent since Jul 21 (0.928); failed Jul 21, 23, 24; passed Jul 22 — 3 of last 4 completed runs failed, all by ≤0.002.
• Scope: 1 job / 1 test file on nightly-test-amd.yml (MI35x 4-GPU). No other jobs affected. NVIDIA/other backends not involved.
• Blocked work: MiniMax-M2.5 TP4 accuracy signal on MI35x nightly is red; no correctness impact demonstrated (miss is within measurement noise).
• In-flight fix: ❌ none found in open PRs.

nightly-accuracy-2-gpu-vlm — failed step(s): Nightly Accuracy Test (2-GPU VLM MMMU)

Commit Info

• sglang: 2428f56145
• aiter: 9127c94a18 (v0.1.17.dev110+g9127c94a1, source: dev image preserved — [CI-AITER-CHECK] Dev/patched version detected → skipping rebuild)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi30x/test_vlms_mmmu_eval_amd.py	test_mmmu_vlm_models	AssertionError: The following models failed (5/11 VLMs below MMMU threshold)	Log

Per-model breakdown (this run — 6 passed, 5 failed):

Model	Score	Threshold	Status
Qwen/Qwen2-VL-7B-Instruct	0.2901	0.30	❌ FAIL
Qwen/Qwen2.5-VL-7B-Instruct	0.2802	0.33	❌ FAIL
Qwen/Qwen3-VL-30B-A3B-Instruct	0.2601	0.29	❌ FAIL
openbmb/MiniCPM-o-2_6	0.2802	0.32	❌ FAIL
deepseek-ai/deepseek-vl2-small	0.2802	0.31	❌ FAIL
InternVL2_5-2B / MiniCPM-v-2_6 / Janus-Pro-7B / Kimi-VL-A3B / MiMo-VL-7B / DeepSeek-OCR-2	≥ threshold	—	✅ PASS

Failure Summary

The single test test_mmmu_vlm_models in test_vlms_mmmu_eval_amd.py iterates over 11 VLMs and raises AssertionError if any score below its per-model MMMU threshold. Five models (Qwen2-VL-7B, Qwen2.5-VL-7B, Qwen3-VL-30B-A3B, MiniCPM-o-2_6, deepseek-vl2-small) scored below threshold, so the test failed at _check_results (test_vlms_mmmu_eval_amd.py:380). The measured scores are stable and reproducible across runs, not noisy.

Regression Status

Known recurring failure / Never-passed — no green run for this test file in the queryable window.

Recent history of test_vlms_mmmu_eval_amd.py (test_mmmu_vlm_models) in job nightly-accuracy-2-gpu-vlm (most recent comparable completed scheduled runs: branch=main, event=schedule; Jul-20 and Jul-15 runs were cancelled and excluded):

Date	Run	Test File Status	Failed Models (count)	Note
Jul 24	run	❌ Failed	5 (Qwen2-VL, Qwen2.5-VL, Qwen3-VL-30B, MiniCPM-o, deepseek-vl2)	this run
Jul 23	run	❌ Failed	5 (same set)	—
Jul 22	run	❌ Failed	5 (same set)	—
Jul 21	run	❌ Failed	5 (same set)	—
Jul 19	run	❌ Failed	5 (same set)	—
Jul 18	run	❌ Failed	5 (same set)	—
Jul 17	run	❌ Failed	5 (same set)	—
Jul 16	run	❌ Failed	4 (Qwen2-VL passed at exactly 0.30)	—

First observed failure: at/before the earliest queryable run (Jul 16 2026). No passing run for this test file found in the 8-run window (Jul 16 – Jul 24). Four of the five models — Qwen2.5-VL-7B, Qwen3-VL-30B-A3B, MiniCPM-o-2_6, deepseek-vl2-small — fail in every run; Qwen2-VL-7B is borderline (0.2799–0.30, passed once on Jul 16).

Failure Cluster

VLM MMMU accuracy below threshold (test-config calibration) on AMD MI30x

Facts (verified observations)

• Verbatim assertion (this run, line 15599): AssertionError: The following models failed: followed by the 5 model/score/threshold lines above. Summary line: Passed: 6 / Failed: 5; step exit code 1 → job exit 255.
• Scores are deterministic across the week: Qwen2.5-VL ≈ 0.2802/0.2901, Qwen3-VL-30B ≈ 0.2601/0.2700, MiniCPM-o ≈ 0.2700–0.2802, deepseek-vl2 ≈ 0.2802 in every run examined (Jul 16–24). Variance is ≤ ±0.01, well inside 100-sample MMMU sampling noise — this is a stable gap, not a flake.
• The gap to threshold is small and fixed: Qwen2.5-VL is ~0.05 short (0.28 vs 0.33), Qwen3-VL ~0.03 short (0.26 vs 0.29), MiniCPM-o ~0.04 short (0.28 vs 0.32), deepseek-vl2 ~0.03 short (0.28 vs 0.31).
• Same test file failed with the same model set in all 8 completed comparable runs; no green run exists in the window.
• AITER version is unchanged (dev image v0.1.17.dev110+g9127c94a1 preserved) — not an override, so this is not a scout run; no A/B origin analysis applies.
• The 6 passing models (InternVL2_5-2B, MiniCPM-v-2_6, Janus-Pro-7B, Kimi-VL-A3B, MiMo-VL-7B-RL, DeepSeek-OCR-2) clear their thresholds every run, so server launch / eval harness / GPU are healthy — the failure is model-specific accuracy, not an infra crash.

Hypothesised Causes (with confidence)

• [MEDIUM] Per-model thresholds in AMD_VLM_MODELS (test_vlms_mmmu_eval_amd.py:44-116) are set higher than the model's reproducible AMD/ROCm score for these 4-5 VLMs (threshold-calibration mismatch).
  • Supporting: scores are stable to ±0.01 across 8 runs and sit just below thresholds; the passing models clear their (lower) thresholds cleanly; comment in source calls them "conservative thresholds on 100 MMMU samples," implying they were hand-set.
  • Disconfirming: this does not distinguish "threshold too high" from "genuine accuracy regression that landed before Jul 16." No green baseline exists in the window to anchor the correct value, so the intended score is unknown.
  • How to verify: check the commit that introduced each model's threshold (e.g. Qwen3-VL-30B, MiniCPM-o entries) and whether a green run ever existed on AMD at that value; compare against the same models' NVIDIA MMMU nightly scores.
• [LOW] Persistent VLM quality/accuracy gap on ROCm for these specific models (multimodal preprocessing, image-token handling, or a kernel producing slightly-off logits on gfx942).
  • Supporting: only a subset of VLMs is affected while text and other VLMs pass; the deficit is consistent.
  • Disconfirming: no crash or error, scores are only marginally low and internally consistent; cannot be separated from a pure threshold miscalibration without a known-good reference score.
  • How to verify: run the same 100-sample MMMU eval for Qwen2.5-VL-7B on an NVIDIA nightly runner and compare; if NVIDIA also ≈ 0.28, the threshold is simply too high.

No hypothesis reaches HIGH: the true regression point (if any) predates the 8-run queryable window, so a code culprit cannot be bisected from available history without extending the lookback further back than Jul 16.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for VLM MMMU threshold, test_vlms_mmmu_eval_amd, and VLM accuracy AMD (via search/issues); 0 relevant open PRs. The 4 hits for the first query (#29202 MTP, #13166 NVIDIA Eagle, #27667 CI sandbox, #18186 generic AMD CI fix) do not touch these thresholds.

Suggested Triage Steps

• Extend the history lookback beyond Jul 16 for nightly-accuracy-2-gpu-vlm to find the last run (if any) where these 4 models passed; if none exists, treat as a never-passed calibration issue and recalibrate thresholds down to just below observed stable scores (e.g. Qwen2.5-VL ~0.27, Qwen3-VL ~0.25, MiniCPM-o ~0.26, deepseek-vl2 ~0.27) on a branch, then re-run.
• If a prior green run is found, bisect the sglang commits between that run's head SHA and the first failing SHA, focusing on multimodal / VL preprocessing and attention paths, to determine whether a real accuracy regression landed.
• Cross-check the same models' MMMU scores on an NVIDIA nightly run: if NVIDIA scores match the AMD ~0.28, the thresholds are miscalibrated (config fix); if NVIDIA is materially higher, investigate a ROCm-specific VLM accuracy gap.

Status

• Persistence: Failing continuously across the full queryable window — all 8 completed comparable scheduled runs (Jul 16 → Jul 24 2026); no green run found. Four models fail every run; Qwen2-VL-7B is borderline.
• Scope: 1 test file / 1 test function in job nightly-accuracy-2-gpu-vlm (nightly-test-amd.yml). Blocks the 2-GPU VLM MMMU accuracy signal on AMD MI30x. Other accuracy jobs in the same run (nightly-accuracy-2-gpu, nightly-4-gpu, etc.) fail independently and are separate clusters.
• In-flight fix: ❌ none found — needs a new PR (threshold recalibration) or a bisect to confirm/deny an accuracy regression.

nightly-test-1-gpu-kernel — failed step(s): Nightly Kernel Benchmark (1-GPU)

Commit Info

• sglang: 2428f561
• aiter: 9127c94a (v0.1.17.dev110+g9127c94a1, source: dev image preserved — [CI-AITER-CHECK] reports "Dev/patched version detected → skipping rebuild")

Failed Tests

Test File	Test Function	Error	Log
test/registered/kernels/ops/mamba/test_transfer_mamba.py	test_mamba_kernel_backup_load_roundtrip[page_first/page_first_direct × dtype0/1] (4)	NameError: name 'transfer_kv_mamba_lf_pf' is not defined	Log
test/registered/kernels/ops/mamba/test_transfer_mamba.py	test_mamba_kernel_single_item[…] (4)	same NameError	Log
test/registered/kernels/ops/mamba/test_transfer_mamba.py	test_mamba_kernel_full_indices[…] (4)	same NameError	Log

12 failed, 4 passed (the 4 test_mamba_kernel_empty_indices cases PASS because they return early on src_indices.numel()==0 before reaching the kernel call).

Failure Summary

The AMD kernel nightly job fails in test/registered/kernels/ops/mamba/test_transfer_mamba.py: 12 of 16 cases raise NameError: name 'transfer_kv_mamba_lf_pf' is not defined at python/sglang/srt/mem_cache/memory_pool_host.py:407. The same undefined-name error hits the load path (transfer_kv_mamba_pf_lf, line 364) whenever a case exercises the io_backend='kernel' code path. All other test files in the job pass.

Regression Status

Known recurring failure — Never-passed on AMD (the test exercises a CUDA-only kernel that is never imported on ROCm).

Recent history of test_transfer_mamba.py (test_mamba_kernel_*) in job nightly-test-1-gpu-kernel (comparable completed scheduled runs, branch=main, event=schedule):

Date	Run	Job	Test File Status	Failed Function	Error
Jul 24	30115195247	nightly-test-1-gpu-kernel	❌ Failed (12/16)	test_mamba_kernel_backup_load_roundtrip +2	NameError: transfer_kv_mamba_lf_pf
Jul 23	30031602981	nightly-test-1-gpu-kernel	❌ Failed (12/16)	same	same NameError
Jul 22	29944275726	nightly-test-1-gpu-kernel	❌ Failed (12/16)	same (path was registered/jit/)	same NameError
Jul 21	29855050299	nightly-test-1-gpu-kernel	❌ Failed (12/16)	same (path was registered/jit/)	same NameError

No passing run for this test file exists in the queryable window (Jul 21–24); marked Never-passed on AMD. The test file was moved from test/registered/jit/test_transfer_mamba.py to test/registered/kernels/ops/mamba/test_transfer_mamba.py by #32128 on Jul 23 — the failure is identical before and after the move.

Failure Cluster

NameError: transfer_kv_mamba_lf_pf undefined — Mamba JIT transfer kernel not imported on ROCm

Facts (verified observations, no interpretation)

• Error verbatim: NameError: name 'transfer_kv_mamba_lf_pf' is not defined at ../python/sglang/srt/mem_cache/memory_pool_host.py:407 (job log line 5422/6270).
• memory_pool_host.py:41 guards the mamba kernel imports behind if _is_cuda: — transfer_kv_mamba_lf_pf / transfer_kv_mamba_pf_lf are imported only on true CUDA, not HIP (verified in source, lines 41–45).
• The call sites at memory_pool_host.py:364 (transfer_kv_mamba_pf_lf) and :407 (transfer_kv_mamba_lf_pf) are inside the io_backend == "kernel" branch and are not guarded by any _is_cuda/_is_hip check (verified in source).
• The test's skip guard is pytest.mark.skipif(not torch.cuda.is_available(), reason="…require CUDA.") (test_transfer_mamba.py:21-23). On ROCm, torch.cuda.is_available() returns True, so the test is not skipped, while the production is_cuda() used for the import returns False.
• The test explicitly opts into AMD nightly: register_amd_ci(est_time=10, suite="nightly-amd-kernel-1-gpu", nightly=True) (test_transfer_mamba.py:19).
• 4 test_mamba_kernel_empty_indices cases pass (early-return on empty indices); the 12 that reach the kernel call fail. Identical 12 failed, 4 passed in all 4 runs Jul 21–24.
• The mamba concurrency io transfer kernel was introduced by #30535 (b296e1a5, Jul 16). #32015 (977ea336, Jul 22) only changed the import path; the if _is_cuda: guard already existed (verified from the commit patch).

Hypothesised Causes (with confidence, with disconfirming evidence)

• [FACT] The Mamba JIT transfer kernel is CUDA-only by import (if _is_cuda: at memory_pool_host.py:41), but the ROCm/AMD nightly runs the test because its skipif uses torch.cuda.is_available() (True on ROCm) instead of is_cuda(). On AMD the names transfer_kv_mamba_lf_pf/pf_lf are never bound, and the io_backend='kernel' call sites (lines 364/407) reference them → NameError.
  • Supporting: the guard, the call sites, and the skip condition are all directly readable in source; error message and line number match memory_pool_host.py:407; failure is 100% reproducible across 4 consecutive nightly runs with identical 12 failed, 4 passed.
  • Disconfirming: (none found) — the mismatch between is_cuda() (import) and torch.cuda.is_available() (skip) fully explains the symptom.
  • How to verify: on an AMD MI30x container, python -c "from sglang.srt.utils import is_cuda; print(is_cuda(), __import__('torch').cuda.is_available())" → expect False True; then run the test and observe the NameError.

In-flight Fix Check

In-flight fix: ✅ #31741 "[AMD] Enable mamba JIT transfer kernel on ROCm (fix transfer_kv_mamba NameError)" — open since 2026-07-20, mergeable_state: blocked (last updated 2026-07-23). This directly targets the failure; chase reviewers / unblock rather than opening a duplicate.

• Related: #31780 "Fix mamba mem_cache tests on non-CUDA (XPU) devices" — addresses the same skip-guard class of issue on XPU; may inform the ROCm fix.

Search: curl ".../search/issues?q=repo:sgl-project/sglang+is:pr+is:open+transfer_kv_mamba" → 1 open PR (#31741).

Suggested Triage Steps

• Verify #31741 resolves this: check out the branch, rebuild the AMD image, and re-run test/registered/kernels/ops/mamba/test_transfer_mamba.py on an MI30x runner. Determine why it is blocked (required CI / review) and unblock.
• If a full ROCm kernel port is not yet ready, an interim option to consider is tightening the test skip guard from torch.cuda.is_available() to an is_cuda()-equivalent (true-CUDA-only) check so AMD nightly stops reporting a Never-passed failure — the maintainer decides whether to fix-forward (enable on ROCm) or gate the test.
• Confirm scope: grep memory_pool_host.py call sites (lines 364, 407) — both transfer_kv_mamba_pf_lf and transfer_kv_mamba_lf_pf are affected; ensure any fix covers both the load (pf_lf) and backup (lf_pf) paths.

Status

• Persistence: Failing every completed scheduled run in the queryable window (Jul 21 → Jul 24, 4/4 runs), same NameError, same 12 failed, 4 passed. Never observed passing on AMD.
• Scope: 1 test file (test_transfer_mamba.py), 12 cases, in job nightly-test-1-gpu-kernel on nightly-test-amd.yml. NVIDIA is unaffected (kernel is imported on true CUDA).
• Blocked work: Blocks clean AMD kernel nightly signal for the HiCache Mamba host-transfer kernel; the ROCm code path for io_backend='kernel' mamba transfers is unusable until the kernel is enabled on HIP.
• In-flight fix: ✅ #31741 (open since 2026-07-20, currently blocked).

nightly-accuracy-8-gpu-mi35x-deepseek-v32-mtp — failed step(s): Accuracy Test MI35x (8-GPU DeepSeek-V3.2 TP+MTP)

The analysis is complete. The report above is my final output — a background log-download task just finished confirming its completion (exit 0), which doesn't change any findings.

Summary of the investigation: The nightly-accuracy-8-gpu-mi35x-deepseek-v32-mtp job failed because test_a_gsm8k hung mid-eval. The GSM8K run progressed cleanly to ~64% (128/200), then the scheduler froze with a 47-request batch and made zero progress for ~30 minutes until the server was torn down, ending in KeyError: 'answer' and exit 255.

py-spy dumps pin the stall unambiguously to a HIP device→host sync (.item()) at dsa_backend.py:793 inside the EAGLE draft-extend path (prepare_for_draft_extend → _draft_extend_for_decode). That sync is where the CPU waits on the GPU, not the root cause.

Key judgment calls:

• Intermittent, not a fresh regression — the identical signature already occurred Jul 22 with an earlier SHA, and the job passes ~4 of 6 recent nights, so no Jul23→Jul24 commit could have introduced it.
• The 21:46–22:01 watchdog lines are just model-loading barriers, not the failure.
• No on-point in-flight fix exists (#32209 is a different model/config).

nightly-2-gpu-mi35x-glm51-mxfp4 — failed step(s): N/A

Commit Info

• sglang: 69a3c54c
• aiter: N/A — the job log blob does not exist (BlobNotFound), so no [CI-AITER-CHECK] markers are recoverable. The runner died before uploading its log.

Failed Tests

Test File	Test Function	Error	Log
N/A (non-test infra failure)	N/A	The self-hosted runner lost communication with the server during step 6 "Accuracy Test MI35x (2-GPU GLM-5.1-MXFP4 GSM8K)"	Job

This is not a test-level failure. The runner (linux-mi35x-gpu-2-glhqm-runner-969ww) lost communication with GitHub ~22 min into the accuracy step, so the job was force-failed and no test log was uploaded. The intended workload was run_suite.py --hw amd --suite nightly-amd-2-gpu-mi35x-glm51-mxfp4 --nightly (.github/workflows/nightly-test-amd.yml:294-302).

Failure Summary

Job nightly-2-gpu-mi35x-glm51-mxfp4 was force-failed by GitHub with the check-run annotation "The self-hosted runner lost communication with the server." The step Accuracy Test MI35x (2-GPU GLM-5.1-MXFP4 GSM8K) is frozen in in_progress and Post Checkout code never ran. The job log blob returns BlobNotFound, confirming the runner died before log upload. Started 17:47:59Z, killed 18:10:00Z (~22 min).

Regression Status

Known recurring failure (infrastructure) — never-passed in the queryable window. This job has conclusion failure in every completed scheduled run queried. Two distinct failure modes are present: a runner-lost mode (step frozen in_progress, this failure's signature) that has recurred every run since 2026-07-17, and an earlier genuine test failure mode (step actually completed=failure) on 2026-07-16 and before.

Recent history of nightly-2-gpu-mi35x-glm51-mxfp4 (accuracy step) — most recent comparable completed scheduled runs (branch=main, event=schedule, completed only):

Date	Run	Job Concl	Acc-step state	Failure mode
Jul 25	30168349328	❌ failure	frozen in_progress	runner lost communication (this run)
Jul 24	30115195247	❌ failure	frozen in_progress	runner lost communication
Jul 23	30031602981	❌ failure	frozen in_progress	runner lost communication
Jul 22	29944275726	❌ failure	frozen in_progress	runner lost communication
Jul 21	29855050299	❌ failure	frozen in_progress	runner lost communication
Jul 19	29697529176	❌ failure	frozen in_progress	runner lost communication
Jul 18	29654601256	❌ failure	—	runner lost communication
Jul 17	29601593052	❌ failure	—	runner lost communication
Jul 16	29521595306	❌ failure	completed=failure	genuine test failure (step ran ~1h37m)
Jul 14	29355460805	❌ failure	completed=failure	genuine test failure
Jul 13	29273241962	❌ failure	completed=failure	genuine test failure

No passing run found in the query window → Never-passed for this job. The runner-lost signature has persisted for 8+ consecutive scheduled runs since 2026-07-17.

Failure Cluster

Self-hosted MI35x runner lost communication during GLM-5.1-MXFP4 2-GPU GSM8K accuracy test

Facts (verified observations, no interpretation)

• Check-run annotation (FACT): failure | The self-hosted runner lost communication with the server. Verify the machine is running and has a healthy network connection. Anything in your workflow that terminates the runner process, starves it for CPU/Memory, or blocks its network access can cause this error.
• Job log endpoint returns BlobNotFound (215-byte XML error) on repeated fetches — no log was uploaded.
• Job steps: step 6 Accuracy Test MI35x (2-GPU GLM-5.1-MXFP4 GSM8K) = in_progress/null; step 12 Post Checkout code = pending/null. Earlier steps (Set up, Checkout, VRAM clear, Setup docker, Install dependencies) all = success.
• Timing: started 2026-07-25T17:47:59Z, force-completed 2026-07-25T18:10:00Z (~22 min); runner linux-mi35x-gpu-2-glhqm-runner-969ww, label linux-mi35x-gpu-2.
• Sister job nightly-2-gpu-mi35x-deepseek-r1-mxfp4-tp2 on the same runner label linux-mi35x-gpu-2 (different physical runner -d84jd) succeeded in this same run.
• The identical runner-lost annotation appears in 8+ consecutive scheduled runs of this specific job (Jul 17–25); on Jul 16 and earlier the same job instead failed with the accuracy step actually completing (completed=failure), a different mode.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [LOW] The GLM-5.1-MXFP4 2-GPU GSM8K workload itself is crashing or starving its runner host (host/GPU OOM, GPU fault, or driver hang taking the machine offline), rather than a generic network flake on the runner pool.
  • Supporting: failure is highly reproducible (8+ consecutive runs) and always at the same accuracy step ~20–30 min in; the sister 2-GPU MI35x job on the same runner label succeeded, so the label pool is not globally broken; GitHub's own annotation text lists "terminates the runner process / starves it for CPU/Memory" as causes.
  • Disconfirming: no log was uploaded, so there is zero direct evidence of an OOM/GPU fault — the runner-lost signature is equally consistent with a host/network hardware fault on that specific physical runner. We cannot see the workload's last output.
  • How to verify: on the runner host, watch dmesg/rocm-smi/host memory during a manual re-run of run_suite.py --hw amd --suite nightly-amd-2-gpu-mi35x-glm51-mxfp4 --nightly; check whether the GitHub Actions runner agent (or the machine) dies. Correlate with GPU/host OOM-killer logs.
• [LOW] Physical-runner or network hardware fault specific to linux-mi35x-gpu-2-glhqm-runner-969ww.
  • Supporting: annotation explicitly cites network/machine health; log-upload failure is consistent with an abrupt host death.
  • Disconfirming: the failure recurs across many days and (per historical annotations) different job IDs/runners for the same GLM workload, so it is unlikely to be one bad physical host; it tracks the workload, not the host.
  • How to verify: cross-check which physical runner names hit this annotation across the Jul 17–25 runs; if they differ, host-specific fault is unlikely.

No hypothesis reaches MEDIUM — root cause cannot be pinned without runner-side host/GPU logs, which are not in the GitHub job artifacts.

In-flight Fix Check

• In-flight fix: ❌ none found for the runner-lost failure. Searches run:
  • is:pr is:open GLM-5.1-MXFP4 in:title → 1 result, #29873 "docs(GLM-5.1): add --enable-aiter-allreduce-fusion to MXFP4 MI355X command" — docs only, not a runner-stability fix (though the allreduce-fusion flag it documents may be worth trying in the suite command — see triage).
  • is:pr is:open mi35x runner in:title → 1 result, #29127 "[DEBUG] Isolate faulty RDMA NIC (rdma5) on MI35x disaggregation runner" — different (disaggregation) runner context, not this 2-GPU accuracy job.

Suggested Triage Steps

• Reproduce manually on an MI35x 2-GPU host: run run_suite.py --hw amd --suite nightly-amd-2-gpu-mi35x-glm51-mxfp4 --nightly --timeout-per-file 7200 inside the CI container while monitoring dmesg -w, rocm-smi, and host free -g; determine whether the runner agent, the container, or the whole host dies (~20–30 min mark).
• Confirm the runner-lost annotation spans multiple distinct physical runner names across the Jul 17–25 runs (I confirmed -969ww for today); if it does, rule out a single bad host and focus on the workload.
• Compare against the sister nightly-2-gpu-mi35x-deepseek-r1-mxfp4-tp2 which passed on the same label — diff the two suites' launch args/memory footprint to see what the GLM-5.1-MXFP4 path does differently (2-GPU MXFP4 GLM-5.1 vs DeepSeek-R1 MXFP4).
• Separately, investigate the pre-2026-07-17 genuine accuracy failure (step completed=failure, e.g. Jul 16 run) — that log likely still exists and may reveal whether GLM-5.1-MXFP4 was ever passing on this suite.

Status (factual, no priority assignment)

• Persistence: runner-lost signature failing every completed scheduled run since 2026-07-17 (8+ consecutive; job never green in the queryable window).
• Scope: 1 job (nightly-2-gpu-mi35x-glm51-mxfp4) on runner label linux-mi35x-gpu-2; a sibling job on the same label passed this run, so scope is workload-specific, not label-wide.
• Blocked work: GLM-5.1-MXFP4 2-GPU GSM8K accuracy signal on MI35x is unavailable (no test result produced).
• In-flight fix: ❌ none found for the runner-lost failure (#29873 is docs-only; #29127 is a different runner).

nightly-4-gpu — failed step(s): Nightly Test (4-GPU)

Commit Info

• sglang: 69a3c54c
• aiter: 9127c94a (v0.1.17.dev110+g9127c94a1; source: dev image preserved — [CI-AITER-CHECK] reports "Dev/patched version detected → skipping rebuild", matches Dockerfile default)

Failed Tests

Test File	Test Function	Error	Log
test/registered/vlm/test_encoder_dp.py	TestVLMEncoderDP.test_vlm_mmmu_benchmark	TIMEOUT after 3600s (est. 500s) → exit 255	Log

Failure Summary

The single test file in the nightly-4-gpu suite, test_encoder_dp.py, ran the MMMU VLM benchmark and was killed by the per-file harness timeout after 3600s (estimated 500s). The server started cleanly (TP=4, all 4 ranks loaded, CUDA graphs captured) and processed requests for the full hour without crashing — but decode throughput stayed abnormally low (1–6 tok/s), so the benchmark never completed. In CI the test randomly selects one of 4 VLMs; this run selected zai-org/GLM-4.1V-9B-Thinking.

Regression Status

Known recurring failure — flaky by random model selection. The outcome tracks 100% with which model the random.choice(MODELS) picks (test_encoder_dp.py:34). GLM-4.1V-9B-Thinking has never passed in the queryable window (all 4 picks timed out); the two Qwen picks passed (one only barely, at 2919s).

Recent history of test/registered/vlm/test_encoder_dp.py (test_vlm_mmmu_benchmark) in job nightly-4-gpu (most recent comparable completed scheduled runs, same branch, event=schedule):

Date	Run	Job	Test File Status	Model Picked	Elapsed
Jul 25	run	nightly-4-gpu	❌ TIMEOUT	GLM-4.1V-9B-Thinking	3600s
Jul 24	run	nightly-4-gpu	❌ TIMEOUT	GLM-4.1V-9B-Thinking	3600s
Jul 23	run	nightly-4-gpu	✅ Passed	Qwen2.5-VL-72B	2919s
Jul 22	run	nightly-4-gpu	❌ TIMEOUT	GLM-4.1V-9B-Thinking	3600s
Jul 21	run	nightly-4-gpu	✅ Passed	Qwen3-VL-32B	632s
Jul 19	run	nightly-4-gpu	❌ TIMEOUT	GLM-4.1V-9B-Thinking	3600s

First observed GLM timeout in this window: Jul 19. GLM-4.1V-9B-Thinking = Never-passed across the queryable window (lookback Jul 19–25).

Failure Cluster

VLM MMMU benchmark timeout on GLM-4.1V-9B-Thinking (low decode throughput)

Facts (verified observations, no interpretation)

• Harness output: ✗ TIMEOUT: .../test_encoder_dp.py after 3600 seconds; {"file": ".../test_encoder_dp.py", "passed": false, "elapsed": 3600}; First attempt failed with exit code 255 (job log ~L8483–8500).
• Server never crashed: continuous Prefill batch / Decode batch log lines from 18:26 through 19:11 UTC (final batch at 19:11:39, timeout fired 19:12:20).
• Decode throughput logged as gen throughput (token/s): 1.29 … 1.51 … 2.28 … 5.90 … 5.45 throughout the run — abnormally low for a 9B model on 4×MI30x.
• Config: zai-org/GLM-4.1V-9B-Thinking, --mm-enable-dp-encoder --tp=4 --cuda-graph-max-bs-decode 32, attention_backend='aiter' (verified in ServerArgs dump).
• Model selection is random.choice(MODELS) under is_in_ci() (test_encoder_dp.py:33-34); MODELS = {Qwen2.5-VL-72B, Qwen3-VL-32B, InternVL2_5-8B, GLM-4.1V-9B-Thinking}.
• Outcome correlates 100% with model picked across 6 runs (table above): all 4 GLM picks timed out; both Qwen picks passed.
• aiter dev image v0.1.17.dev110+g9127c94a1 is identical in passing (Jul 21/23) and failing (Jul 19/22/24/25) runs ([CI-AITER-CHECK] markers), so aiter is not implicated.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [HIGH] GLM-4.1V-9B-Thinking cannot finish the MMMU benchmark within the 3600s per-file timeout because it is a long-reasoning "Thinking" model (large generation length per MMMU question) combined with very low observed decode throughput (1–6 tok/s).
  • Supporting: every run that picked GLM timed out at exactly 3600s; the two runs that picked Qwen models passed; server was healthy and generating the whole time (no crash), just slowly.
  • Disconfirming: no direct baseline showing GLM ever completed faster in the queryable window, so it is unproven whether throughput regressed vs GLM being intrinsically too slow for this timeout — the near-miss Qwen2.5-VL-72B run (2919s) suggests the whole suite is close to the timeout ceiling regardless of model.
  • How to verify: run test_encoder_dp.py pinned to GLM-4.1V-9B-Thinking on 4×MI30x in container rocm/sgl-dev:v0.5.16-rocm700-mi30x and measure wall-clock + steady-state decode tok/s; compare against a known-good sglang tag to check for a throughput regression.
• [LOW] A decode-throughput regression (attention backend / cuda-graph / MoE path) slowed GLM specifically in a recent sglang commit.
  • Supporting: 1–6 tok/s is low for a 9B model on MI30x.
  • Disconfirming: no green GLM run exists to bound a pass_sha..fail_sha window; the failure spans ≥6 days with many differing head SHAs, and the same low-throughput symptom would be expected to hit Qwen too (it did not clearly).
  • How to verify: bisect GLM-pinned throughput across the Jul 19→25 sglang history; if flat and always >3600s, this is a fixed-timeout/model-fit issue, not a regression.

In-flight Fix Check

• Searched open PRs with keywords encoder dp timeout, GLM-4.1V Thinking, mmmu timeout amd, vlm nightly timeout.
• In-flight fix: ❌ none found — no open PR targets this timeout. (PR #31735 "update CI test est_time values" only adjusts the estimate, not the 3600s hard --timeout-per-file, so it would not resolve this.)

Suggested Triage Steps

• Reproduce with the model pinned to GLM-4.1V-9B-Thinking (bypass random.choice) on 4×MI30x and capture steady-state decode tok/s — confirm whether the model is intrinsically over-budget or throughput regressed.
• If throughput is the issue: bisect GLM-pinned decode tok/s across sglang commits Jul 19→25 to check for a VLM/decode regression; inspect the aiter attention backend path for GLM (the run logs Using aiter_attn as multimodal attention backend and low tok/s).
• If GLM is simply too slow for MMMU at this timeout: consider (maintainer decision) either raising --timeout-per-file for the nightly-amd-4-gpu suite, capping GLM's MMMU sample/generation length, or excluding GLM-4.1V-9B-Thinking from the random AMD pool until throughput is addressed — the current random.choice makes the job non-deterministically red ~½ the time.

Status (factual, no priority assignment)

• Persistence: GLM-4.1V-9B-Thinking picks have timed out on every occurrence since at least Jul 19 (4/4 GLM runs in the Jul 19–25 window); job flips red/green purely on random model selection.
• Scope: 1 job (nightly-4-gpu) / 1 test file (test_encoder_dp.py) in nightly-test-amd.yml. The CUDA sister registration (nightly-4-gpu, register_cuda_ci) may share the same risk but was not queried here.
• Blocked work: intermittently fails the AMD 4-GPU nightly VLM signal (~50% of nights).
• In-flight fix: ❌ none found in open PRs.

nightly-accuracy-2-gpu — failed step(s): N/A

Commit Info

• sglang: 69a3c54c ("Fix SWA admission livelock on cached-prefix resumes (#32379)")
• aiter: 9127c94a1 — v0.1.17.dev110+g9127c94a1 (source: dev image detected, rebuild skipped; matches Dockerfile default AITER_COMMIT=9127c94a1)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi30x/test_gsm8k_eval_amd.py	TestNightlyGsm8KEval.test_gsm8k_all_models (subTest model=Qwen/Qwen3-4B)	AssertionError: The following models failed: Qwen/Qwen3-4B — Score 0.909, threshold: N/A → ❌ FAILED	Log

Failure Summary

The nightly-accuracy-2-gpu job runs a 5-model GSM8K accuracy suite. Four models passed (Llama-3.1-8B 0.814, Qwen3-8B 0.916, Llama-3.1-70B 0.931; Mixtral eval was still running where the captured log ends). Qwen/Qwen3-4B was marked ❌ FAILED with "threshold: N/A" despite scoring 0.909 — a strong score. test_gsm8k_all_models then raises AssertionError, failing the job (job conclusion = failure, completed 21:55:31 UTC).

Regression Status

Known recurring failure (config-level, not a code regression in the tested SHA).

Recent history of test_gsm8k_eval_amd.py (subTest Qwen/Qwen3-4B) in job nightly-accuracy-2-gpu (branch main, event=schedule, completed runs):

Date	Run	Job	Test File Status	Failed Function	Error
Jul 25	30168349328	nightly-accuracy-2-gpu	❌ Failed	Qwen/Qwen3-4B subTest	Score 0.909, threshold N/A
Jul 24	30115195247	nightly-accuracy-2-gpu	❌ Failed	Qwen/Qwen3-4B subTest	Score 0.913, threshold N/A

Both verified at test-file level (identical Qwen/Qwen3-4B → "threshold: N/A" → FAILED pattern). Earlier nightly-test-amd runs (Jul 13–23) show workflow-level failure but I did not test-verify the Qwen3-4B subtest in each. This is a static config defect, so it fails on every run that evaluates Qwen3-4B; no green run exists for this subtest since Qwen3-4B was added to the default TP1 list.

Failure Cluster

Qwen3-4B GSM8K marked FAILED due to missing score threshold (config gap, not accuracy loss)

Facts (verified observations)

• Log line 13814–13822: 📈 Results for Qwen/Qwen3-4B: Score: 0.909 (threshold: N/A) … Status: ❌ FAILED. Two earlier Qwen3-4B eval passes in the same run scored 0.909 and 0.911 (lines 11298, 12270) — the model's accuracy is healthy.
• MODEL_SCORE_THRESHOLDS in test/registered/amd/accuracy/mi30x/test_gsm8k_eval_amd.py:37-67 contains Qwen/Qwen3-8B: 0.76 but no Qwen/Qwen3-4B key.
• Pass/fail logic test_gsm8k_eval_amd.py:288-289: threshold = MODEL_SCORE_THRESHOLDS.get(model) → None; passed = threshold and score >= threshold → None (falsy) → prints Status: ❌ FAILED and appends to failed_models.
• Qwen/Qwen3-4B is present in the evaluated list via python/sglang/test/test_utils.py:153-154: DEFAULT_MODEL_NAME_FOR_NIGHTLY_EVAL_TP1 = "meta-llama/Llama-3.1-8B-Instruct,Qwen/Qwen3-8B,Qwen/Qwen3-4B".
• The same model does have a threshold (0.77) in the generic eval file test/registered/eval/test_text_models_gsm8k_eval.py:33 (82.41% measured − 5%), confirming 0.909 would comfortably pass a correct threshold.
• Secondary (non-fatal): a soft scheduler watchdog fired during Llama-3.1-70B load — [TP0] Scheduler watchdog timeout (self.watchdog_timeout=300, self.soft=True) (line 13141). Stack (lines 13124-13185) shows MainThread blocked in snapshot_download→xet_get (HuggingFace Hub weight download); the CI download lock for Llama-3.1-70B was held from 21:19:47 to >21:24 (lines 13017, 13127). Llama-3.1-70B nonetheless loaded and PASSED 0.931 (line 17061), so this slow-download watchdog did not fail the job.
• AITER dev image preserved; [CI-AITER-CHECK] Dev/patched version detected: v0.1.17.dev110+g9127c94a1 → skipping rebuild (lines 4302-4303).

Hypothesised Causes

• [FACT] Qwen3-4B is evaluated but has no entry in MODEL_SCORE_THRESHOLDS, so the harness marks any score as FAILED (threshold: N/A).
  • Supporting: threshold dict (test_gsm8k_eval_amd.py:37-67) lacks Qwen/Qwen3-4B; pass logic passed = threshold and score >= threshold (line 289) evaluates None → falsy; log shows threshold: N/A → ❌ FAILED with a passing score of 0.909; identical on Jul 24 (0.913).
  • Disconfirming: none. The accuracy itself is fine (0.909/0.911/0.913 across runs) — this is purely a missing-config defect, not a model/kernel/aiter regression.
  • How to verify: add "Qwen/Qwen3-4B": 0.77 to MODEL_SCORE_THRESHOLDS (mirroring test_text_models_gsm8k_eval.py:33) and re-run nightly-accuracy-2-gpu; the subtest should pass.

In-flight Fix Check

• In-flight fix: ❌ none found — search repo:sgl-project/sglang is:pr is:open Qwen3-4B threshold returned 4 PRs (#16638, #28045, #30960, #31735), all unrelated (server_args refactor, spec decoding, CI est_time updates). No open PR adds the missing Qwen3-4B threshold.

Suggested Triage Steps

• Add a Qwen/Qwen3-4B entry to MODEL_SCORE_THRESHOLDS in test/registered/amd/accuracy/mi30x/test_gsm8k_eval_amd.py (a value near 0.77 is consistent with test/registered/eval/test_text_models_gsm8k_eval.py:33), then re-run nightly-accuracy-2-gpu on MI325 to confirm the subtest passes.
• Alternatively, if Qwen3-4B is not intended on AMD nightly, remove it from DEFAULT_MODEL_NAME_FOR_NIGHTLY_EVAL_TP1 (python/sglang/test/test_utils.py:154) or filter it in this AMD file — but adding the threshold is the lower-risk fix since accuracy is healthy.
• Consider hardening the harness so a missing threshold is a loud config error / skip rather than a silent FAILED (line 289), to avoid future model-list/threshold drift.
• Separately (does not fail the job): monitor HF Hub download latency for Llama-3.1-70B on linux-mi325-2gpu runners — the soft watchdog at 300s during xet_get/snapshot_download indicates >5 min weight pulls; if it worsens it could start tripping hard timeouts.

Status

• Persistence: Qwen/Qwen3-4B subtest fails on every completed nightly-accuracy-2-gpu run that evaluates it — verified Jul 24 and Jul 25; it is a static config gap with no passing run.
• Scope: nightly-accuracy-2-gpu job of nightly-test-amd.yml. The other 4 models in the suite pass; the single Qwen3-4B config failure is what turns the job red.
• Blocked work: masks the real GSM8K accuracy signal for the AMD 2-GPU nightly — a genuine accuracy regression in any of the 5 models would be hard to distinguish from this persistent red.
• In-flight fix: ❌ none found — needs a new PR adding the Qwen3-4B threshold.

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-07-25 23:43 UTC)

[amd-bot]

pr-test-amd-rocm720.yml — 10 failure(s)

Run started (UTC): 2026-07-24 17:59 UTC
Last scanned (UTC): 2026-07-25 23:13 UTC
Commits: sglang 69a3c54

Summary

Counts: 10 failures · 7 clusters · 9 carrying over (≥3 days) · 1 new today

#	Cluster	Job	Test File	Test Function	Status	Hypothesis (confidence)
1	2-GPU diffusion part-0 hang / lost runner	multimodal-gen-test-2-gpu (…,0)	python/sglang/multimodal_gen/test/server/test_server_2_gpu.py	test_diffusion_generation[flux2_modelopt_fp8_tp2_t2i] (+joy_echo cascade)	Never-passed, ≥15 runs (since ~Jul 10)	[FACT] FLUX.2 FP8 HIPBLAS_STATUS_NOT_SUPPORTED + joy_echo port-5555 cascade
2	2-GPU diffusion part-0 hang / lost runner	multimodal-gen-test-2-gpu (…,0)	N/A (runner hang, no log)	N/A	Never-passed, ≥9 runs (Jul 25)	[MEDIUM] runner lost mid-step, no log flushed
3	Diffusion 1-GPU suite 60-min step timeout	multimodal-gen-test-1-gpu (…,0)	sglang/multimodal_gen/test/server/test_server_1_gpu.py	test_diffusion_generation (partition 0)	Never-passed, 16-run window (since ~Jul 9)	[HIGH] suite exceeds 60-min budget
4	Diffusion 1-GPU suite 60-min step timeout	multimodal-gen-test-1-gpu (…,1)	N/A (whole partition 1)	N/A	Never-passed, 8-run window (since Jul 17)	[HIGH] suite exceeds 60-min budget
5	GSM8K MXFP4 MoE accuracy gate flake (0.92)	stage-b-1gpu-small-mi35x	test/registered/quant/test_quark_mxfp4.py	TestFP8ToMXFP4MOETP1.test_gsm8k	Flaky 2/4 (since Jul 21)	[MEDIUM] 0.92 gate too tight vs sampling variance
6	GSM8K MXFP4 MoE accuracy gate flake (0.92)	stage-b-1gpu-small-mi35x	test/registered/quant/test_quark_mxfp4.py	TestFP8ToMXFP4MOETP1.test_gsm8k	Flaky ~3/6 (since Jul 20)	[HIGH] 0.92 gate too tight vs sampling variance
7	msgpack ModuleNotFoundError (ROCm image dep gap)	multimodal-gen-unit-test	multimodal_gen msgpack roundtrip test	test_msgpack_roundtrip_preserves_string_keys_and_numpy_payloads	Never-passed (since Jul 11)	[FACT] msgpack (diffusion extra) not installed in ROCm 7.2 image
8	1-GPU diffusion part-3 runner lost (infra)	multimodal-gen-test-1-gpu (…,3)	N/A (runner lost communication)	N/A	Persistent, ≥12 runs (since Jul 13)	[MEDIUM] self-hosted runner lost mid-test
9	AMD Triton offline throughput < 3500 gate	stage-b-1gpu-large (…,0)	test/registered/perf/test_bench_serving_1gpu_part1.py	test_offline_throughput_with_triton_attention_backend	Never-passed, 9/9 (since ≥Jul 16)	[MEDIUM] 3500 gate mis-calibrated / long-standing perf gap
10	DeepSeek-V3.2 TP8 GPU OOM mid-GSM8K	stage-c-large-8gpu (…,1)	test/registered/amd/test_deepseek_v32_basic.py	TestDeepseekV32TP.test_a_gsm8k	🆕 New today, 1/8 (green Jul 17–23)	[no candidate ≥ MEDIUM]

Cluster: 2-GPU diffusion part-0 hang / lost runner

• Affected: rows 1–2 — same job multimodal-gen-test-2-gpu-amd-rocm720 (…,0) on Jul 24 (89553949923) and Jul 25 (89705019759).
• Shared facts: part-0 red every run ≥Jul 10 while parts 1/2 pass; recent runs (Jul 19–25) share step-9-stuck-in_progress + BlobNotFound (no log uploaded), ~99–110 min, under the 150-min step timeout. Retrievable Jul 13/15/16 logs show the true trigger: FLUX.2 modelopt-FP8 HIPBLAS_STATUS_NOT_SUPPORTED + joy_echo_t2v_2gpu 20-min startup hang leaking scheduler port 5555 → 68× --strict-ports cascade.
• Top hypothesis: [FACT] FLUX.2 FP8 GEMM hipBLAS unsupported on gfx942 + joy_echo port-5555 leak; [MEDIUM] recent runs now hang/lose the runner before logs flush. Disconfirming: current logs missing, so an independent infra fault can't be excluded.
• In-flight fix: ✅ #28889 (FLUX.2 e4m3fnuz FP8 GEMM on gfx942); joy_echo/port cascade → ❌ none.
• Suggested triage: land/verify #28889; add per-case watchdog in run_suite.py so a failed launch reaps port 5555 and flushes logs; pull runner host dmesg/rocm-smi.

Cluster: Diffusion 1-GPU suite 60-min step timeout

• Affected: rows 3–4 — partitions 0 (89553950036) & 1 (89553950015), same run.
• Shared facts: step Run diffusion server tests (1-GPU) (timeout-minutes: 60, .github/workflows/pr-test-amd-rocm720.yml:728) killed at 60 min; jobs run 70–80 min; every one of the last 8 runs failed. HF cache shows ✗ NOT CACHED for all models → multi-GB re-download inside the timed step.
• Top hypothesis: [HIGH] partition workload doesn't fit the 60-min budget; [MEDIUM] empty HF cache is a major contributor. Disconfirming: partition 2 (same empty cache) passes in budget.
• In-flight fix: ❌ none found.
• Suggested triage: confirm wall-clock by summing per-case durations; pre-warm/verify /sgl-data/hf-cache mount; raise --total-partitions or timeout-minutes.

Cluster: GSM8K MXFP4 MoE accuracy gate flake (0.92)

• Affected: rows 5–6 — stage-b-test-1-gpu-small-amd-mi35x-rocm720, test_quark_mxfp4.py::TestFP8ToMXFP4MOETP1.test_gsm8k, on Jul 24 (89553949886) & Jul 25 (89705019718).
• Shared facts: accuracy over Jul 20–25 = {0.900…0.936}, straddling the hard-coded 0.92 gate (test_quark_mxfp4.py:251); passes when a run samples ≥0.92, fails otherwise; aiter (v0.1.17.dev110+g9127c94a1) and threshold constant across pass/fail.
• Top hypothesis: [HIGH] 0.92 gate inside the run-to-run GSM8K variance band → test-calibration flake, not a code regression. Disconfirming: never reaches the 0.948 reference even when green (possible real MXFP4 degradation, but present in passing runs too).
• In-flight fix: ❌ none found.
• Suggested triage: re-run shard 5–10× at the head SHA to confirm ~50% flake; quantify true MXFP4-requant vs FP8 delta before loosening the gate.

Cluster: msgpack ModuleNotFoundError (ROCm image dep gap)

• Affected: row 7 — multimodal-gen-unit-test-amd-rocm720.
• Shared facts: ModuleNotFoundError: No module named 'msgpack'; msgpack declared in the diffusion extra (pyproject.toml:110) but absent from the ROCm 7.2 CI image; failing every run since the test was added 2026-07-11.
• Top hypothesis: [FACT] dependency not installed in image — environment gap, not a code fault.
• In-flight fix: ✅ #31899 — chase reviewers, do not duplicate.
• Suggested triage: land #31899 (install msgpack in ROCm image or move to base deps).

Cluster: 1-GPU diffusion part-3 runner lost (infra)

• Affected: row 8 — multimodal-gen-test-1-gpu-amd-rocm720 (…,3).
• Shared facts: job annotation confirms the self-hosted MI325 runner lost communication mid-test; log blob 404 BlobNotFound; ≥12 consecutive scheduled runs since 2026-07-13; same run also had part-1 lost / part-0 timeout.
• Top hypothesis: [MEDIUM] runner/host-side termination (GPU hang / OOM / infra), not a code regression. Disconfirming: no log/diagnostics to pin the cause.
• In-flight fix: ❌ none found.
• Suggested triage: pull runner _diag/host dmesg/rocm-smi for the part-3 runner; re-run via workflow_dispatch and watch live.

Cluster: AMD Triton offline throughput < 3500 gate

• Affected: row 9 — stage-b-test-1-gpu-large-amd-rocm720 (…,0).
• Shared facts: assertGreater(output_throughput, 3500) (test_bench_serving_1gpu_part1.py:120, is_in_amd_ci()); measured 3087–3244 tok/s in 9/9 runs (benchmark completes, 500/500 requests); no green in window.
• Top hypothesis: [MEDIUM] 3500 AMD gate above achievable throughput / long-standing perf gap. Disconfirming: no green baseline in window, so "over-tight gate" vs "old regression" indistinguishable from history alone.
• In-flight fix: ❌ none found.
• Suggested triage: query pr-test-amd-rocm720 history before Jul 16 for any green run — decides gate-recalibration vs bisect.

Cluster: DeepSeek-V3.2 TP8 GPU OOM mid-GSM8K

• Affected: row 10 — stage-c-test-large-8-gpu-amd-rocm720 (…,1).
• Shared facts: HSA_STATUS_ERROR_OUT_OF_RESOURCES / Available Free mem : 0 MB → Fatal Python error: Aborted at 810/1319 questions; config tp=8, attention_backend=dsa, kv_cache_dtype=bfloat16, mem_fraction_static=0.896; green the prior 7 runs (0.957 on Jul 23) — genuinely new signature (distinct from the older Jul 14–16 setUpClass startup cluster).
• Top hypothesis: no candidate ≥ MEDIUM. [LOW] borderline mem config tipping into fragmentation OOM on long decode; [LOW] mem-footprint regression in d0b96898..2428f561. Disconfirming: passed 7 straight days at identical config.
• In-flight fix: ❌ none found.
• Suggested triage: re-run 3–5× to classify flaky vs deterministic, then bisect the window if deterministic; probe lower mem_fraction_static (0.85).

Suggested triage order

Rank	Triage step	Targets	Why this first
1	Land / chase in-flight #31899 (install msgpack)	Cluster F (row 7)	Fix already exists; one-line dependency gap, cheapest green recovery
2	Land / verify #28889 + add port-5555 reap in run_suite.py; re-run part-0	Cluster C (rows 1–2)	Largest + most persistent cluster (≥15 runs); half has a ready fix
3	Re-partition / raise timeout-minutes; pre-warm HF cache	Cluster B (rows 3–4)	Structural 60-min timeout blocking 1-GPU signal, no code fix needed
4	Re-run 3–5× to classify flaky vs deterministic, then bisect	Cluster G (row 10)	Only NEW-today failure; cheap flakiness check before a bisect
5	Re-run MXFP4 shard 5–10× + quantify MXFP4-vs-FP8 delta before gate change	Cluster A (rows 5–6)	Confirm calibration flake vs real degradation before touching threshold

---

Job	Failed Steps	Started
stage-b-test-1-gpu-small-amd-mi35x-rocm720 (linux-mi35x-gpu-1)	Run test	2026-07-24T17:59
multimodal-gen-unit-test-amd-rocm720	Run diffusion unit tests	2026-07-24T19:41
stage-b-test-1-gpu-large-amd-rocm720 (linux-mi325-1gpu-sglang, 0)	Run test	2026-07-24T19:53
multimodal-gen-test-2-gpu-amd-rocm720 (linux-mi325-2gpu-sglang, 0)	N/A	2026-07-24T18:46
multimodal-gen-test-1-gpu-amd-rocm720 (linux-mi325-1gpu-sglang, 0)	Run diffusion server tests (1-GPU)	2026-07-24T19:57
stage-c-test-large-8-gpu-amd-rocm720 (linux-mi325-8gpu-sglang, 1)	Run test	2026-07-24T18:30
multimodal-gen-test-1-gpu-amd-rocm720 (linux-mi325-1gpu-sglang, 1)	N/A	2026-07-24T20:04
multimodal-gen-test-1-gpu-amd-rocm720 (linux-mi325-1gpu-sglang, 3)	N/A	2026-07-24T20:16
stage-b-test-1-gpu-small-amd-mi35x-rocm720 (linux-mi35x-gpu-1)	Run test	2026-07-25T17:47
multimodal-gen-test-2-gpu-amd-rocm720 (linux-mi325-2gpu-sglang, 0)	N/A	2026-07-25T21:09

Per-Job Analysis

stage-b-test-1-gpu-small-amd-mi35x-rocm720 (linux-mi35x-gpu-1) — failed step(s): Run test

Commit Info

• sglang: 2428f5614
• aiter: 9127c94a1 v0.1.17.dev110+g9127c94a1 (source: dev image preserved — [CI-AITER-CHECK] Dev/patched version detected → skipping rebuild, matches Dockerfile default)

Failed Tests

Test File	Test Function	Error	Log
test/registered/quant/test_quark_mxfp4.py	TestFP8ToMXFP4MOETP1.test_gsm8k	AssertionError: 0.914 not greater than 0.92 (both attempts) → retry() exceed maximum number of retries	Log

Other 4 files in this shard passed (test_dsa_skip_logits_clean.py, test_verify_splitkv.py, test_mamba_state_scatter_triton.py, test_deepseek_mla_dispatch.py). Job Test Summary: 4/5 files passed → exit 255.

Failure Summary

A single GSM8K accuracy assertion failed: the online FP8→MXFP4-requantized Qwen/Qwen3-30B-A3B-Instruct-2507-FP8 model scored 0.914 on both the initial attempt and the retry, just below the hard-coded accuracy_threshold=0.92 (test_quark_mxfp4.py:251). The 3 earlier tests in the file (TestOnlineQuantizationMemoryLoad* test_gsm8k) were skipped (FAILED (errors=1, skipped=3)). No crash, no kernel error — purely a threshold miss of 0.006 (≈4 of 500 questions).

Regression Status

Flaky / borderline threshold — the test oscillates above and below 0.92 across consecutive daily runs on differing SHAs. The test itself was only introduced 2026-07-21 by #28291 (dcd9014f15, reland of "Online MXFP4 quantization 2/N"), so only 4 comparable scheduled runs exist.

Recent history of test/registered/quant/test_quark_mxfp4.py (TestFP8ToMXFP4MOETP1.test_gsm8k) in job stage-b-test-1-gpu-small-amd-mi35x-rocm720 (linux-mi35x-gpu-1) (all comparable event=schedule, branch=main, completed runs since the test was added):

Date	Run	Job	Test File Status	MOETP1 accuracy (attempts)	Error
Jul 24	run	job	❌ Failed	0.914, 0.914	0.914 not greater than 0.92
Jul 23	run	job	✅ Passed	0.914 → retry passed	—
Jul 22	run	job	❌ Failed	0.900, 0.918	0.918 not greater than 0.92
Jul 21	run	job	✅ Passed	hit 0.928	—

First observed failure of this test file: 2026-07-22. Last known pass: 2026-07-23. Failing 2 of 4 runs since introduction. (Job's overall conclusion is failure in all 4 runs due to unrelated shards; verdict above is at the test-file level.)

Failure Cluster

GSM8K accuracy just below threshold on online FP8→MXFP4 MoE quantization (Qwen3-30B-A3B).

Facts (verified observations)

• Log verbatim: AssertionError: 0.914 not greater than 0.92; metrics={'accuracy': 0.914, 'invalid': 0.0, ...}; retry also 0.914 → Exception: retry() exceed maximum number of retries. (test_quark_mxfp4.py:251 → :171).
• Threshold is a hard-coded constant: self._test_gsm8k(accuracy_threshold=0.92) for model Qwen/Qwen3-30B-A3B-Instruct-2507-FP8; source comment states reference accuracy ~0.948.
• Eval config (_test_gsm8k): num_questions=500, num_shots=8, max_new_tokens=512, parallel=128 — batched/parallel decode, so run-to-run non-determinism is expected.
• Observed MOETP1 accuracy across the 4 daily runs: 0.928 (Jul21 pass), 0.900/0.918 (Jul22 fail), 0.914→retry-pass (Jul23), 0.914/0.914 (Jul24 fail) — steady-state clusters ~0.90–0.93, straddling the 0.92 line; never reached the 0.948 reference even on passing runs.
• aiter version is identical (v0.1.17.dev110+g9127c94a1) in passing and failing runs ([CI-AITER-CHECK] markers).
• The earlier [Health check failed ... detokenizer for last 20 seconds] (18:08:30) is benign warmup — caused by a 25s aiter JIT kernel build (mha_batch_prefill...); server reached ready immediately after and served all 500 GSM8K requests with 200 OK.

Hypothesised Causes (with confidence)

• [MEDIUM] Over-tight accuracy threshold (0.92) relative to the model's true CI steady-state (~0.91) → a test-calibration/flakiness issue, not a code regression.
  • Supporting: same test file passes and fails on alternating days with different HEAD SHAs; observed accuracies (0.900–0.928) are tightly clustered around 0.92; the retry sometimes flips the verdict (Jul 23 passed on retry). GSM8K over 500 questions with parallel=128 has inherent sampling variance of several questions (each = 0.002).
  • Disconfirming: could mask a genuine accuracy regression in the MXFP4 requant path — but no single sglang commit correlates (distinct passing and failing SHAs), and the threshold+model were introduced together on Jul 21.
  • How to verify: re-run this shard 5× on the current SHA and record the accuracy distribution; if it centers ~0.91 with spread across 0.92, the threshold is miscalibrated.
• [LOW] Real numerical accuracy loss in the FP8→MXFP4 online requant or aiter MoE kernel on gfx950, keeping accuracy ~0.03 below the 0.948 reference.
  • Supporting: observed accuracy never reaches the documented reference of 0.948 even when passing (max 0.928).
  • Disconfirming: aiter version is unchanged across all pass/fail runs; the ~0.91 level is the steady-state from the very first run (Jul 21), so there is no fresh drop to attribute to a commit.
  • How to verify: compare GSM8K accuracy of the BF16/FP8 baseline vs the MXFP4-requantized model locally on MI35x to quantify the quantization accuracy delta.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for quark mxfp4 (37 open, none touch this GSM8K threshold/flake) and mxfp4 accuracy (124 open, none address test_quark_mxfp4 calibration). API: search/issues?q=repo:sgl-project/sglang+is:pr+is:open+quark+mxfp4 and ...+mxfp4+accuracy.

Suggested Triage Steps

• Re-run the shard 5–10× on SHA 2428f5614 to measure the accuracy distribution for TestFP8ToMXFP4MOETP1.test_gsm8k; confirm it straddles 0.92.
• If confirmed flaky, lower accuracy_threshold in test_quark_mxfp4.py:251 toward the observed floor (~0.88, matching the sibling TestOnlineQuantizationMemoryLoadMOE 0.89 and Dense 0.868 thresholds) and/or raise num_questions to tighten the estimate — decision for the PR author of #28291.
• Separately, if the ~0.03 gap vs the 0.948 reference is deemed a real quality concern, benchmark the MXFP4 requant path against the FP8 baseline on MI35x. (Bisecting is not useful — no clean pass→fail SHA window exists.)

Status

• Persistence: Flaky since test introduction 2026-07-21; failed 2 of 4 comparable scheduled runs (Jul 22, Jul 24), passed Jul 21 & Jul 23.
• Scope: One test file / one test function in the stage-b-test-1-gpu-small-amd-mi35x-rocm720 (linux-mi35x-gpu-1) shard of pr-test-amd-rocm720.yml. No crash; other 4 files in the shard pass.
• Blocked work: Intermittently reds the ROCm 7.2 MI35x quantization shard; no product functionality blocked.
• In-flight fix: ❌ none found — needs threshold recalibration PR or accuracy investigation.

multimodal-gen-unit-test-amd-rocm720 — failed step(s): Run diffusion unit tests

The background task already completed and I incorporated its results earlier. My analysis is complete — the report above is final.

Summary: the multimodal-gen-unit-test-amd-rocm720 job failed on a single test, test_msgpack_roundtrip_preserves_string_keys_and_numpy_payloads, with ModuleNotFoundError: No module named 'msgpack'. This is a never-passed environment/dependency gap (not a code regression): msgpack is declared in the diffusion extra (pyproject.toml:110) but is not installed in the ROCm 7.2 CI image. It has failed every scheduled run since the test was introduced on 2026-07-11. An in-flight fix already exists — #31899.

stage-b-test-1-gpu-large-amd-rocm720 (linux-mi325-1gpu-sglang, 0) — failed step(s): Run test

Commit Info

• sglang: 2428f5614
• aiter: 9127c94a1 (source: dev image — v0.1.17.dev110+g9127c94a1 detected and preserved, rebuild skipped)

Failed Tests

Test File	Test Function	Error	Log
test/registered/perf/test_bench_serving_1gpu_part1.py	test_offline_throughput_with_triton_attention_backend	AssertionError: 3239.82 not greater than 3500 (retry: 3223.43 not greater than 3500) → retry() exceed maximum number of retries	Log

The other 2 tests in this file passed (Ran 3 tests … FAILED (errors=1)); the suite step reported 9/10 passed, with only this file at exit code 1.

Failure Summary

The AMD-MI325 offline-throughput benchmark with the Triton attention backend measured output_throughput ≈ 3239.8 token/s (and 3223.4 on the single retry), both below the hard-coded AMD CI gate of 3500 at test/registered/perf/test_bench_serving_1gpu_part1.py:120. The benchmark itself ran to completion (500/500 successful requests); this is a throughput-threshold assertion, not a crash or accuracy failure.

Regression Status

Known recurring failure / Never-passed in queryable window. This exact test has failed with the same < 3500 throughput symptom in every comparable completed scheduled run examined (9 consecutive runs, 2026-07-16 → 2026-07-24). No green run for this test exists in the lookback window. Measured throughput is stable at ~3087–3244 token/s — consistently ~7–12% under the 3500 gate — not a fresh drop.

Recent history of test_bench_serving_1gpu_part1.py (test_offline_throughput_with_triton_attention_backend) in job stage-b-test-1-gpu-large-amd-rocm720 (linux-mi325-1gpu-sglang, 0) (most recent comparable completed scheduled runs: branch=main, event=schedule, completed only):

Date	Run	Test File Status	Failed Function	Measured throughput
Jul 24	30115153295	❌ Failed	test_offline_throughput_with_triton_attention_backend	3239.82 / 3223.43
Jul 23	30031591211	❌ Failed	same	3244.00 / 3226.39
Jul 22	29944235234	❌ Failed	same	3168.52 / 3177.18
Jul 21	29855003895	❌ Failed	same	3177.95 / 3196.09
Jul 20	29766743590	❌ Failed	same	3159.69 / 3156.87
Jul 19	29697501385	❌ Failed	same	3174.22 / 3167.25
Jul 18	29654571916	❌ Failed	same	3155.41 / 3150.30
Jul 17	29601526478	❌ Failed	same	3104.10 / 3087.65
Jul 16	29521557976	❌ Failed	same	3157.62 / 3160.33

(Regression verdict is at the test-file level: same test, same symptom, every run. First observed failure ≥ 2026-07-16; no passing run found in the queried window — lookback covered the 9 most recent scheduled runs.)

Failure Cluster

AMD Triton-backend offline throughput below 3500 gate (perf-threshold failure)

Facts (verified observations, no interpretation)

• Verbatim: AssertionError: 3239.8221247809747 not greater than 3500 (log line 8415), retry AssertionError: 3223.429489288858 not greater than 3500 (line 9799), then Exception: retry() exceed maximum number of retries. (line 9818) → FAILED (errors=1).
• Failing assertion is self.assertGreater(res["output_throughput"], 3500) under if is_in_amd_ci(): at test/registered/perf/test_bench_serving_1gpu_part1.py:120.
• Server config: --attention-backend triton --context-length 8192, num_prompts=500, request_rate=inf, model meta-llama/Llama-3.1-8B-Instruct (from ServerArgs dump, line 8421). Benchmark completed: Successful requests: 500.
• Reproducibility: same test + same symptom in 9/9 examined scheduled runs; throughput band 3087–3244 token/s, always below the 3500 gate; low run-to-run variance.
• aiter unchanged as a cause: dev image v0.1.17.dev110+g9127c94a1 preserved across runs ([CI-AITER-CHECK] markers, lines in step 6).
• The current file path was created 2026-07-22 by #30540 (0a6d1930c), but the same test/symptom already failed at its prior location before that date, so the file move is not the cause.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [MEDIUM] The 3500 AMD gate is above what this MI325 runner + Triton backend actually delivers for Llama-3.1-8B offline throughput; the test is effectively mis-calibrated (or a slow, pre-window perf regression left it permanently red).
  • Supporting: 9 consecutive runs cluster tightly at 3087–3244 token/s with low variance — characteristic of a stable achievable-throughput ceiling below the gate, not intermittent noise. The passing sibling tests in the same file use lower gates (e.g. 2600) and pass.
  • Disconfirming: no green baseline exists in the queryable window, so it cannot be proven the AMD gate was ever met — a genuine perf regression predating 2026-07-16 would look identical. Cannot distinguish "threshold too high" from "old regression" from history alone.
  • How to verify: extend the scheduled-run history search before 2026-07-16 to find the last run (if any) where this test passed on AMD; if a green run exists, bisect pass_sha..fail_sha for a Triton-decode/attention perf regression; if none exists in a long window, treat as an un-calibrated AMD gate.
• [LOW] A Triton attention-backend decode-path regression specific to ROCm 7.2 / gfx942.
  • Supporting: failure is isolated to the --attention-backend triton variant; the non-Triton throughput tests in the file pass.
  • Disconfirming: throughput is flat across 9 runs spanning many sglang commits — no step-change correlating with any single commit, which a code-level regression would usually show.
  • How to verify: run bench_serving locally on MI325 with --attention-backend triton vs the default backend on the same SHA and compare token/s.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for test_bench_serving + throughput (1 hit, #27650, unrelated PCG/all-reduce fix) and offline + throughput + triton (20 hits, none targeting this threshold/test). No PR adjusts the 3500 AMD gate or the Triton offline-throughput path for this test.

Suggested Triage Steps

• Query scheduled pr-test-amd-rocm720.yml history before 2026-07-16 to determine whether this test ever passed on AMD; the answer decides between "over-tight gate" and "long-standing perf regression".
• If a green baseline is found: bisect the pass_sha..fail_sha window for Triton attention / decode-throughput changes and re-run test_offline_throughput_with_triton_attention_backend on MI325 container.
• If no green baseline is found in a wide window: treat the 3500 AMD gate at test_bench_serving_1gpu_part1.py:120 as mis-calibrated for this runner and have a maintainer decide whether to re-baseline it to the observed ~3100–3250 band (with headroom) or quarantine the AMD branch until the perf gap is understood.
• Confirm the runner is not thermally/contention throttled by comparing the MI325 GPU clocks/rocm-smi across a passing sibling test in the same job.

Status (factual, no priority assignment)

• Persistence: Failing every completed scheduled run for ≥ 9 days (since at least 2026-07-16), no green run in the queried window.
• Scope: This one test in job stage-b-test-1-gpu-large-amd-rocm720 (linux-mi325-1gpu-sglang, 0) on pr-test-amd-rocm720.yml; benchmark completes successfully, only the AMD throughput gate trips.
• Blocked work: Keeps the stage-b-test-1-gpu-large-amd-rocm720 job (and thus the ROCm 7.2 scheduled run) red; masks any new regression that would surface in the same job.
• In-flight fix: ❌ none found — needs threshold re-calibration decision or a perf investigation.

multimodal-gen-test-2-gpu-amd-rocm720 (linux-mi325-2gpu-sglang, 0) — failed step(s): N/A

Commit Info

• sglang: 2428f5614 — [Bugfix] Fix Kimi-Linear state transfer across heterogeneous TP (#32262)
• aiter: N/A for this run — the job log blob for 89553949923 is expired (BlobNotFound) and part 0 was not uploaded to the run-level log archive, so [CI-AITER-CHECK] markers cannot be read for this run. (For reference, the same job on Jul 16 ran aiter dev v0.1.17.dev110+g9127c94a1, Dockerfile default 9127c94a, dev image preserved.)

Failed Tests

Direct per-test evidence is from the last retrievable run of this exact job/part (Jul 16, run 29521557976 / job 87699749934); the current run's own log is unavailable.

Test File	Test Function	Error	Log
python/sglang/multimodal_gen/test/server/test_server_2_gpu.py	TestDiffusionServerTwoGpu::test_diffusion_generation[flux2_modelopt_fp8_tp2_t2i]	RuntimeError: CUDA error: HIPBLAS_STATUS_NOT_SUPPORTED during denoise warmup	Log
python/sglang/multimodal_gen/test/server/test_server_2_gpu.py	TestDiffusionServerTwoGpu::test_diffusion_generation[joy_echo_t2v_2gpu]	FAILED during server startup after ~20 min hang; leaves scheduler port 5555 occupied	Log
python/sglang/multimodal_gen/test/server/test_server_2_gpu.py	16 remaining test_diffusion_generation[...] cases (wan2_1_, mova_, ltx_, zimage_, flux_*, fsdp-inference)	RuntimeError: Scheduler port 5555 is unavailable and --strict-ports is enabled (68× cascade)	Log

Failure Summary

The multimodal-gen-test-2-gpu-amd-rocm720 part 0 shard fails on the test_server_2_gpu.py diffusion server suite. In the last runs whose logs are retrievable, two genuine failures occur: flux2_modelopt_fp8_tp2_t2i errors with HIPBLAS_STATUS_NOT_SUPPORTED during FP8 denoise, and joy_echo_t2v_2gpu hangs ~20 min at server startup and fails, leaving a scheduler bound to port 5555 that is never released — so every subsequent test case in the partition errors out with --strict-ports port-5555 conflicts. Parts 1 and 2 of the same job passed. For the current run (Jul 24) the job log was never uploaded and the step hung, so the exact per-test failure cannot be read directly.

Regression Status

Known recurring / Never-passed — part 0 of this job failed in all 15 of the most recent comparable completed scheduled runs (branch=main, event=schedule). No green run exists in the queryable window (lookback: 2026-07-10 → 2026-07-24; job-metadata shows the same failing job back to 2026-07-05). This is not a new regression.

Recent history of test_server_2_gpu.py (test_diffusion_generation[...]) in job multimodal-gen-test-2-gpu-amd-rocm720 (…, 0) (most recent 15 comparable completed scheduled runs; same branch, event=schedule):

Date	Run	Test File Status	Failed Function(s)	Error / Note
Jul 24	run	❌ Failed (hang)	not readable (log not uploaded)	step 9 stuck in_progress, ~105 min, runner-lost signature
Jul 23	run	❌ Failed (hang)	not readable	step 9 in_progress, ~110 min
Jul 22	run	❌ Failed (hang)	not readable	step 9 in_progress, ~107 min
Jul 21	run	❌ Failed (hang)	not readable	step 9 in_progress, ~109 min
Jul 20	run	❌ Failed (hang)	not readable	step 9 in_progress, ~102 min
Jul 19	run	❌ Failed (hang)	not readable	step 9 in_progress, ~101 min
Jul 18	run	⚠️ Not reached	—	job failed at ~10 min before test step (step 9 never started)
Jul 17	run	⚠️ Not reached	—	job failed at ~86 min before test step (step 9 never started)
Jul 16	run	❌ Failed	flux2_modelopt_fp8_tp2_t2i + joy_echo_t2v_2gpu + cascade	HIPBLAS + port-5555 (verified in log)
Jul 15	run	❌ Failed	same	HIPBLAS + port-5555 (verified)
Jul 14	run	❌ Failed	same (completed/failure)	~90 min
Jul 13	run	❌ Failed	same	HIPBLAS + port-5555 (verified)
Jul 12	run	❌ Failed	same (completed/failure)	~129 min
Jul 11	run	❌ Failed	same (completed/failure)	~120 min
Jul 10	run	❌ Failed	same (completed/failure)	~118 min

First observed failure for this test file: on/before 2026-07-10 (earliest in window; job-level failures extend to 2026-07-05). Last known passing run for this test file: none found in the 15-run window.

Failure Cluster

"2-GPU diffusion server suite (part 0) fails — FLUX.2 FP8 GEMM HIPBLAS_STATUS_NOT_SUPPORTED + joy_echo startup hang leaking scheduler port 5555 → strict-ports cascade (recently manifesting as a mid-run hang / lost runner)."

Facts (verified observations)

• Current run log is unavailable: GET /actions/jobs/89553949923/logs → HTTP 404 BlobNotFound; the run-level log zip for 30115153295 contains 2-gpu parts 1 and 2 only — part 0 was never uploaded. Job step 9 "Run diffusion server tests (2-GPU)" is stuck in_progress, step 10 "Upload diffusion failure artifacts" (if: always()) never ran, job concluded failure after ~105 min (18:46:30→20:31:35 UTC). Step timeout is 150 min, so this is not a step timeout.
• Sibling shards passed: parts 1 and 2 of the same job completed success in ~20 min each (part1, part2). Only part 0 fails.
• Direct per-test evidence (Jul 13/15/16, same job/part, identical config):
  • test_server_2_gpu.py::…[flux2_modelopt_fp8_tp2_t2i]: [denoising_step_0] Error … CUDA error: HIPBLAS_STATUS_NOT_SUPPORTED when calling HIPBLAS_STATUS_NOT_SUPPORTED → RuntimeError (FLUX.2 modelopt-FP8, tp2). Model = black-forest-labs/FLUX.2-dev.
  • …[joy_echo_t2v_2gpu]: [server-test] FAILED during server startup after a fixed ~20 min (1200s) startup wait (22:03:50→22:23:51 on Jul 16; identical 20-min interval on Jul 13 & 15).
  • After joy_echo, all remaining cases print RuntimeError: Scheduler port 5555 is unavailable and --strict-ports is enabled — 68 occurrences per run on Jul 15 & 13; ##[error]Process completed with exit code 1.
  • Retry harness ran 2 attempts (of up to 7); attempt 2 re-collected 25 items and still ERRORed on the same 18 items.
• Consistency: the flux2 HIPBLAS failure + joy_echo 20-min hang + port-5555 cascade signature is identical across Jul 13, Jul 15, Jul 16 (three independent retrievable logs).
• Recent runs (Jul 19–24) all share the "step 9 in_progress, ~101–110 min, log not uploaded" signature — distinct from the earlier "completed/failure ~116–129 min" signature.

Hypothesised Causes (with confidence + disconfirming evidence)

• [FACT] flux2_modelopt_fp8_tp2_t2i fails with HIPBLAS_STATUS_NOT_SUPPORTED during FP8 denoise on MI325 (gfx942). Verified verbatim in Jul 13/15/16 logs.
  • Supporting: MI325 = gfx942; error is raised from the hipBLAS FP8 GEMM path during the first denoising step of FLUX.2 modelopt-FP8.
  • Disconfirming: (none — directly observed).
  • How to verify: run only -k flux2_modelopt_fp8_tp2_t2i on an MI325/gfx942 container; confirm the hipBLAS FP8 GEMM call is the throwing frame.
• [FACT] joy_echo_t2v_2gpu server startup hangs the full ~20 min startup budget, fails, and its scheduler on port 5555 is not torn down → all later cases error under --strict-ports.
  • Supporting: fixed 20-min interval before "FAILED during server startup"; the very next case is the first port 5555 is unavailable and it cascades to every subsequent case (68 hits).
  • Disconfirming: (none — directly observed for the completed runs). The cascade is a test-harness cleanup gap (leaked port), not necessarily a product bug in each downstream model.
  • How to verify: reproduce joy_echo alone; check whether the launcher/child scheduler process holding port 5555 is reaped on startup-failure, or add a port-free/--strict-ports-off teardown between cases.
• [MEDIUM] The current run (Jul 24) failed for the same underlying part-0 sequence, but this time the shard hung and the runner was lost before the log/artifacts could upload.
  • Supporting: same job/part/matrix and identical env; step 9 stuck in_progress at ~105 min with if:always() upload never executing is a classic lost-runner/hang signature; 6 consecutive runs (Jul 19–24) show this identical metadata pattern, temporally following the joy_echo-hang era.
  • Disconfirming: the current log is unreadable, so an unrelated infra fault (GPU RAS event, host OOM, network drop) independent of the diffusion tests cannot be excluded; the symptom (hang → lost runner) differs from the earlier clean exit 1.
  • How to verify: re-run part 0 with log/heartbeat streaming (or shorter per-case startup timeout) to capture where it hangs; check runner host/dmesg for GPU/OOM events around 20:31 UTC.

In-flight Fix Check

• FLUX.2 FP8 HIPBLAS failure → In-flight fix: ✅ #28889 "[AMD] Use native e4m3fnuz fp8 GEMM for FLUX.2 modelopt-FP8 on gfx942" (open since 2026-06-22, updated 2026-07-21, not draft) — directly targets this hipBLAS FP8 path on gfx942. Chase reviewers; do not duplicate. (Search: is:pr is:open HIPBLAS_STATUS_NOT_SUPPORTED and flux2 fp8 rocm.)
• joy_echo startup hang / port-5555 strict-ports cascade → In-flight fix: ❌ none found (searched is:pr is:open joy_echo → 0; strict-ports diffusion / diffusion port 5555 → only unrelated #24227). Needs a new PR (harness port-cleanup fix and/or joy_echo startup investigation).

Suggested Triage Steps

• Land / prioritize #28889 and re-run part 0 -k flux2_modelopt_fp8_tp2_t2i on MI325 to confirm the hipBLAS FP8 GEMM error clears.
• Investigate joy_echo_t2v_2gpu 2-GPU startup separately — why it consumes the full ~20 min startup timeout and fails; this is the trigger that leaks port 5555.
• Fix the harness so a failed/timed-out server launch reaps its scheduler child and frees port 5555 between cases (or don't run remaining cases under --strict-ports after a startup failure) to stop the 68-case cascade masking real per-model signal.
• For the current lost-runner symptom: enable per-case log streaming / lower the startup timeout so a future part-0 run uploads a diagnosable log; inspect the MI325 runner host logs around 2026-07-24 20:31 UTC for GPU/OOM/RAS events.
• Because this shard has never passed in the queryable window, consider quarantining/splitting the known-red cases (flux2 FP8, joy_echo) off the blocking path until fixes land, so part 0 can yield signal on the other models.

Status

• Persistence: part-0 shard red in all 15 most-recent comparable scheduled runs (≥ since 2026-07-10; job-level failures back to 2026-07-05). No green in window.
• Scope: multimodal-gen-test-2-gpu-amd-rocm720 part 0 only (parts 1 & 2 green) on pr-test-amd-rocm720.yml. Same flux2_modelopt_fp8 FP8/gfx942 pattern likely also affects other AMD diffusion shards that exercise FLUX.2 modelopt-FP8.
• Behavior shift: Jul 10–16 failed cleanly (exit 1, ~116–129 min, logs uploaded); Jul 19–24 hang mid-step (~101–110 min, logs/artifacts not uploaded) — reduced diagnosability for recent runs, current run included.
• In-flight fix: flux2 FP8 → ✅ #28889; joy_echo/port-cascade → ❌ none.

multimodal-gen-test-1-gpu-amd-rocm720 (linux-mi325-1gpu-sglang, 0) — failed step(s): Run diffusion server tests (1-GPU)

Commit Info

• sglang: 2428f5614
• aiter: 9127c94a1 — v0.1.17.dev110+g9127c94a1 (source: dev/patched image detected, rebuild skipped; no override)

Failed Tests

Test File	Test Function	Error	Log
sglang/multimodal_gen/test/server/test_server_1_gpu.py	TestDiffusionServerOneGpu::test_diffusion_generation (partition 0/4, whole step)	Step timeout: The action 'Run diffusion server tests (1-GPU)' has timed out after 60 minutes (executing longlive2_i2v at cutoff)	Log

The primary failure is a 60-minute job-step timeout, not a test assertion. Two secondary in-suite signals (neither directly marks the job failed):

• cosmos3_nano_t2i → SKIPPED: AttributeError: module diffusers has no attribute Cosmos3OmniTransformer (log) — installed diffusers predates the Cosmos3OmniTransformer class.
• longlive2_i2v → warmup ValueError: num_frames must be divisible by num_frames_per_block for causal DMD denoising (warmup uses 17f) (log) — non-fatal ("continuing startup").

Failure Summary

Partition 0 of the multimodal-gen-test-1-gpu-amd-rocm720 job runs ~12 diffusion server test cases sequentially in test_server_1_gpu.py. The step began 20:07:42 UTC and was killed by the 60-minute step timeout (timeout-minutes: 60, .github/workflows/pr-test-amd-rocm720.yml:728) at 21:07:54 UTC, while the last case longlive2_i2v was still generating. Eleven cases passed and one (cosmos3_nano_t2i) was skipped before the wall-clock ran out; the suite simply does not fit in 60 minutes.

Regression Status

Known recurring failure (structural timeout, not tied to this commit). Partition 0 of this job has failed in every completed scheduled run inspected; the exact test case executing at the 60-minute cutoff drifts by a few minutes run-to-run. Never-passed in the queryable 16-run window (2026-07-09 → 2026-07-24, all event=schedule, branch=main, completed).

Recent history of test_server_1_gpu.py partition 0 (test_diffusion_generation suite) in job multimodal-gen-test-1-gpu-amd-rocm720 (…, 0) (most recent comparable completed scheduled runs):

Date	Run	Job	Test File Status	Cutoff case at timeout	Error
Jul 24	run	job	❌ Failed	longlive2_i2v	60-min step timeout
Jul 23	run	job	❌ Failed	sana_wm_ti2v	60-min step timeout
Jul 22	run	job	❌ Failed	longlive2_i2v	60-min step timeout
Jul 21	run	job	❌ Failed	longlive2_i2v	60-min step timeout
Jul 20	run	job	❌ Failed	(conclusion=failure; not log-verified)	—
Jul 19	run	job	❌ Failed	(conclusion=failure; not log-verified)	—

(Partition 2 passes in most runs; partitions 1 and 3 also fail — a different case distribution. The Job column is for reference; the verdict is based on partition-0 Test File Status.)

Failure Cluster

60-minute step timeout in multimodal-gen diffusion server suite (partition 0)

Facts (verified observations)

• Step Run diffusion server tests (1-GPU) (step 9) started 2026-07-24T20:07:42Z, killed at 2026-07-24T21:07:54Z with ##[error]The action 'Run diffusion server tests (1-GPU)' has timed out after 60 minutes (log).
• Step timeout is configured timeout-minutes: 60 (.github/workflows/pr-test-amd-rocm720.yml:728); invocation is --partition-id 0 --total-partitions 4 -k "not flux_2" (:753).
• 11 cases PASSED before cutoff (zimage_image_t2i, qwen_image_edit_ti2i, qwen_image_layered_i2i, wan2_1_t2v_1.3b, wan2_1_t2v_1.3b_upscaling_4x, wan2_1_t2v_1_3b_lora_1gpu, fast_hunyuan_video, wan2_2_ti2v_5b, fastwan2_2_ti2v_5b, sana_wm_ti2v); cosmos3_nano_t2i SKIPPED. Slowest cases: fast_hunyuan_video ~12.5 min, wan2_2_ti2v_5b ~8.5 min.
• cosmos3_nano_t2i skip is caused by env: native load falls back after NotImplementedError: AITer backend does not support Grouped Query Attention yet then hits AttributeError: module diffusers has no attribute Cosmos3OmniTransformer — identical in Jul 22/23 logs (verified diffusers too old for that class).
• longlive2_i2v warmup ValueError (num_frames 61 not divisible by num_frames_per_block) reproduces on Jul 22 and Jul 24 logs; it is caught and startup continues, so it does not itself fail the job.
• AITER unchanged: dev image v0.1.17.dev110+g9127c94a1, no AITER_COMMIT_OVERRIDE.

Hypothesised Causes (with confidence)

• [HIGH] The partition-0 case set exceeds the 60-minute step budget on MI325; the job is killed mid-run regardless of individual case correctness.
  • Supporting: 60-min timeout is the terminal error in every inspected run; 11 cases + skips consume the full hour; the case executing at cutoff drifts (sana_wm_ti2v on Jul 23, longlive2_i2v on Jul 22/24) — i.e. it's wall-clock, not a specific test hang.
  • Disconfirming: none found — no single case shows an infinite hang; each completes and advances.
  • How to verify: sum per-case durations; re-run partition 0 with more partitions (e.g. --total-partitions 6) or raise timeout-minutes and confirm the suite completes.
• [LOW] longlive2_i2v adds wasted time via a failing warmup request before the real generation, pushing the last case past the deadline.
  • Supporting: warmup ValueError precedes the real request; longlive2 is frequently the cutoff case.
  • Disconfirming: Jul 23 timed out at sana_wm_ti2v before longlive2_i2v even started, so longlive2 is not necessary for the timeout.
  • How to verify: fix warmup num_frames/config alignment and re-time the suite.
• [LOW] cosmos3_nano_t2i skip (diffusers/GQA) is a separate pre-existing env gap, not a timeout contributor (it skips in ~90s).
  • Disconfirming: it is SKIPPED, not counted as a failure, and consumes little time.

This is not an amd-aiter-scout.yml run, so no Failure Origin / Baseline A/B section applies. AITER is unchanged across passing and failing history and is not implicated in the timeout.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for multimodal gen timeout, longlive2, Cosmos3OmniTransformer diffusers. PR #29584 ("ci(amd): fix three pre-existing MI325 CI failures") matched the keyword but targets stage-b FP8 tests, not this diffusion job — not applicable.

Suggested Triage Steps

• Confirm the wall-clock hypothesis: sum per-case durations in this log (step 9); if total ≈ 60 min, the fix is capacity, not correctness.
• Increase parallelism (--total-partitions > 4) or raise the step timeout-minutes for multimodal-gen-test-1-gpu-amd-rocm720 in .github/workflows/pr-test-amd-rocm720.yml:728, then re-run and verify partition 0 completes.
• Separately address the longlive2_i2v warmup num_frames config (align warmup frame count with num_frames_per_block) to remove wasted warmup time.
• Separately track the cosmos3_nano_t2i diffusers-version gap (Cosmos3OmniTransformer missing) — a container/dependency bump, independent of the timeout.

Status

• Persistence: Failing every completed scheduled run in the 16-run window (since at least 2026-07-19, verified via logs Jul 21–24); never passed in-window.
• Scope: Partition 0 of multimodal-gen-test-1-gpu-amd-rocm720 on pr-test-amd-rocm720.yml; partitions 1 and 3 also fail (partition 2 passes). Structural timeout, unrelated to commit 2428f5614.
• Blocked work: Blocks the AMD ROCm 7.2 multimodal-gen 1-GPU CI signal.
• In-flight fix: ❌ none found — needs a new PR (raise timeout or add partitions).

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-07-25 23:13 UTC)

[amd-bot]

pr-test-amd-rocm720.yml — Per-Job Analysis (continued 2/2)

stage-c-test-large-8-gpu-amd-rocm720 (linux-mi325-8gpu-sglang, 1) — failed step(s): Run test

Commit Info

• sglang: 2428f5614
• aiter: v0.1.17.dev110+g9127c94a1 (source: dev/patched image — [CI-AITER-CHECK] reported "Dev/patched version detected → skipping rebuild"; Dockerfile default was 9127c94a)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/test_deepseek_v32_basic.py	TestDeepseekV32TP.test_a_gsm8k	HSA_STATUS_ERROR_OUT_OF_RESOURCES: Available Free mem : 0 MB → Fatal Python error: Aborted → server crash → client ConnectionRefusedError: [Errno 111] → retry() exceed maximum number of retries	Log

Ran 3 tests in 2405.955s → FAILED (errors=1, skipped=2). The other 2 tests (test_bs_1_speed, and one more) were skipped because the server died during test_a_gsm8k.

Failure Summary

On DeepSeek-V3.2 (TP=8, MI325, ROCm 7.2, attention_backend=dsa, kv_cache_dtype=bfloat16, mem_fraction_static=0.896), the GSM8K accuracy eval crashed ~61% of the way through (810/1319 questions). A GPU ROCm queue aborted with HSA_STATUS_ERROR_OUT_OF_RESOURCES / Available Free mem : 0 MB, killing a scheduler rank; the resulting gloo Connection reset by peer (TP3) tore down the server, and all subsequent client requests got connection-refused, erroring the test.

Regression Status

New regression (runtime OOM signature) — first seen today after a 7-day green streak for this test file.

Recent history of test_deepseek_v32_basic.py (TestDeepseekV32TP.test_a_gsm8k) in job stage-c-test-large-8-gpu-amd-rocm720 (linux-mi325-8gpu-sglang, 1) (comparable completed scheduled runs, same branch, event=schedule):

Date	Run	Job	Test File Status	Failed Function	Note
Jul 24	30115153295	job	❌ Failed	test_a_gsm8k	OOM mid-eval (this run)
Jul 23	30031591211	job	✅ Passed	—	Accuracy 0.957, 1/1 passed
Jul 22	29944235234	job	✅ Passed	—	—
Jul 21	29855003895	job	✅ Passed	—	—
Jul 20	29766743590	job	✅ Passed	—	—
Jul 19	29697501385	job	✅ Passed	—	—
Jul 18	29654571916	job	✅ Passed	—	—
Jul 17	29601526478	job	✅ Passed	—	—
Jul 16	29521557976	job	❌ Failed	setUpClass	Different signature — server-startup failure (line 115), not an OOM
Jul 15	29438298954	job	❌ Failed	(startup-era)	Older failure cluster, distinct from today
Jul 14	29355401408	job	❌ Failed	(startup-era)	Older failure cluster, distinct from today

• First observed (this OOM signature): 2026-07-24. Last known passing: 2026-07-23 (Accuracy: 0.957).
• The Jul 14–16 failures are a separate setUpClass startup failure, not this runtime OOM — so today is genuinely a new signature, not a resumption of the old one.
• Note (context, not a data point): the overall workflow has a red X every day, but from other jobs/test files — this specific job+test file was green Jul 17–23.

Failure Cluster

GPU out-of-resources / HSA_STATUS_ERROR_OUT_OF_RESOURCES (0 MB free) mid-GSM8K-eval on DeepSeek-V3.2 TP8.

Facts (verified observations)

• Crash marker (verbatim): :0:rocdevice.cpp :3582: ... Aborting with error : HSA_STATUS_ERROR_OUT_OF_RESOURCES: The runtime failed to allocate the necessary resources ... Code: 0x1008 Available Free mem : 0 MB, immediately followed by Fatal Python error: Aborted, at progress 61%| 810/1319.
• Server config (from server_args): tp_size=8, attention_backend='dsa', dsa_prefill_backend='tilelang', dsa_decode_backend='tilelang', kv_cache_dtype='bfloat16' (bf16 KV cache — 2× the memory of fp8), mem_fraction_static=0.896 (aggressive), page_size=64, chunked_prefill_size=16384.
• Crash cascade: [TP3] Scheduler hit an exception ... broadcast_pyobj ... dist.broadcast ... RuntimeError: gloo ... Read error [172.17.0.2]:12716: Connection reset by peer; then SIGQUIT received ... It usually means one child failed. The faulting-rank py-spy stack was inside process_batch_result_decode (batch_result_processor.py:730) → synchronize (torch/cuda/streams.py:231) — i.e. the abort surfaced on a decode-time CUDA stream sync.
• Reproducibility: 1 of the last 8 comparable scheduled runs (Jul 17–24) for this test file. Passed the previous 7 consecutive days with valid accuracy (0.957 on Jul 23).
• Earlier soft-watchdog noise (18:46–19:11, TP4 Scheduler watchdog timeout ... self.soft=True, repeated py-spy dump failures on TP5 PID 4639) occurred during/after weight loading (do_load_weights → load_weights (deepseek_v2.py:2924)) and is not the crash — the server subsequently served 200 OK for ~35 min before the OOM.

Hypothesised Causes (with confidence + disconfirming evidence)

• [LOW] Borderline/flaky memory config tipping into hard OOM. The mid-run OOM (after 810 successful requests, 0 MB free) at mem_fraction_static=0.896 with a bf16 KV cache is the classic signature of memory-fragmentation/high-water-mark exhaustion during a long decode workload rather than a static allocation regression.
  • Supporting: crash at 61% (not at startup); server ran ~35 min and served hundreds of requests first; only 1/8 runs affected; identical config passed the prior 7 days.
  • Disconfirming: it did pass 7 straight days at this exact config, so if purely borderline it should have flaked before; a small memory-usage bump from a recent commit could equally explain "just now started tipping over."
  • How to verify: re-run this job 3–5× on MI325 ROCm 7.2 at the same SHA; if it passes ≥80%, treat as flaky and consider lowering mem_fraction_static for this test. If it fails deterministically, it's a regression → bisect the window below.
• [LOW] A memory-footprint regression among the 44 commits in d0b96898..2428f561. Candidate touch-points (none code-path-confirmed for the DSA decode path this test uses):
  • 5da0b6ec3 "Write-back policy fix for unified tree (#31845)" — edits mem_cache/unified_radix_cache.py (+92). Disconfirming: this run uses radix_cache_backend=None (default radix cache, not the unified backend), so this code is likely not exercised.
  • eac7c7d7c "fix(attention): read per-runner kv cache dtype off model_runner (#32251)" — touches KV-dtype plumbing across attention backends. Disconfirming: it does not modify the DSA/tilelang decode path used here, and is a dtype-source refactor, not an allocation change.
  • How to verify: git bisect the window running test_deepseek_v32_basic.py on MI325 ROCm 7.2; focus first on any commit that increases per-request or KV/graph memory.
• No hypothesis reaches MEDIUM confidence; needs a repeat-run flakiness check first, then a bisect of d0b96898..2428f561 if deterministic.

In-flight Fix Check

• In-flight fix: ❌ none found. Searched open PRs for DeepSeek-V3.2 + memory (32 hits, none addressing this AMD gsm8k OOM) and deepseek_v32 (4 hits: docs/CI/DSA-feature PRs, none a fix for this crash). No existing PR targets a DeepSeek-V3.2 TP8 runtime-OOM on AMD.

Suggested Triage Steps

• Re-run the job at SHA 2428f5614 3–5× on MI325 ROCm 7.2 to classify flaky vs deterministic before any bisect.
• If deterministic: git bisect d0b96898..2428f561 running python3 test/registered/amd/test_deepseek_v32_basic.py, prioritizing memory/KV/attention commits.
• As a mitigation probe (not a fix): retry with a lower mem_fraction_static (e.g. 0.85) to confirm the failure is headroom-sensitive; if it clears the OOM, the test config is running too close to the memory ceiling for bf16 KV on this GPU.
• Keep the Jul 14–16 setUpClass startup failures tracked separately — they are a different cluster and should not be conflated with this OOM.

Status

• Persistence: 1 occurrence (2026-07-24); this test file was green the prior 7 consecutive scheduled runs (Jul 17–23).
• Scope: 1 test (test_a_gsm8k) in 1 job (stage-c-test-large-8-gpu-amd-rocm720, matrix index 1) on pr-test-amd-rocm720.yml. Not yet observed on other workflows/backends.
• Blocked work: DeepSeek-V3.2 8-GPU accuracy signal on AMD MI325 ROCm 7.2; server crash also skips test_bs_1_speed in the same file.
• In-flight fix: ❌ none found — needs a flakiness check then, if deterministic, a new investigation/bisect.

multimodal-gen-test-1-gpu-amd-rocm720 (linux-mi325-1gpu-sglang, 1) — failed step(s): N/A

Commit Info

• sglang: 2428f5614561c9e3990782e99a76bfaaff56324c
• aiter: 9127c94a18e4398e1eba91f6639e910f0994ad02 (v0.1.17.dev110+g9127c94a1, source: Dockerfile default — AITER_COMMIT_OVERRIDE empty, no dev/patched build)

Failed Tests

Not a unit-test assertion — this is a step-level timeout of a diffusion server benchmark suite, so there is no single failing test_* function.

Test File	Test Function	Error	Log
N/A — sglang/multimodal_gen/test/run_suite.py --suite 1-gpu --partition-id 1 --total-partitions 4 -k "not flux_2"	N/A (whole partition)	The action 'Run diffusion server tests (1-GPU)' has timed out after 60 minutes	Job

⚠️ Log availability: the failing job's own log is unavailable — the per-job blob has expired (BlobNotFound) and the job is absent from the run-level archive because it was cancelled (steps 9/10/20 all have a null conclusion). The timeout classification below is therefore reconstructed from job metadata + the sibling shard-0 log in the same run, not from shard-1's own log.

Failure Summary

The step "Run diffusion server tests (1-GPU)" (partition 1 of 4) did not finish within its 60-minute step timeout and was killed. The job ran 80 minutes total (2026-07-24T20:04:18Z → 21:24:22Z): ~20 min container start / dependency install + the 60-min diffusion step that timed out. Step 9 was the in-flight step when the job was cancelled. This is a non-test, whole-suite timeout, not a correctness failure of a specific test.

Regression Status

Known recurring failure (infrastructure / capacity timeout — not a fresh code regression).

Recent history of the diffusion 1-GPU suite, partition 1 step in job multimodal-gen-test-1-gpu-amd-rocm720 (linux-mi325-1gpu-sglang, 1) (most recent 8 comparable completed scheduled runs: same branch main, event=schedule, completed only):

Date	Run	Job	Step 9 Status	Job Duration	Signature
Jul 24	30115153295	job	❌ timeout (null / cancelled)	80 min	60-min step timeout
Jul 23	30031591211	job	❌ failure	77 min	60-min step timeout
Jul 22	29944235234	job	❌ failure	72 min	60-min step timeout
Jul 22	29855003895	job	❌ failure	77 min	60-min step timeout
Jul 21	29766743590	job	❌ failure	71 min	60-min step timeout
Jul 19	29697501385	job	❌ failure	75 min	60-min step timeout
Jul 18	29654571916	job	❌ failure	75 min	60-min step timeout
Jul 17	29601526478	job	❌ failure	72 min	60-min step timeout

First observed failure in the query window: 2026-07-17 (earliest checked); no passing run of this partition exists in the last 8 scheduled runs — failed every time, always with a ~71-80 min duration consistent with the 60-min step timeout. This is a persistent timeout, not a regression introduced by commit 2428f56145.

Failure Cluster

Diffusion 1-GPU suite step timeout (60 min) on MI325 ROCm 7.2

Facts (verified observations)

• Sibling shard-0 in the SAME run timed out identically: job 89553950036 logged ##[error]The action 'Run diffusion server tests (1-GPU)' has timed out after 60 minutes. (shard-0 archive line 7749; step 9 = failure, duration 70 min). This is the direct signature for the shared step that shard-1 also runs.
• Per-run job outcomes for this run (multimodal-gen-test-1-gpu, 4 partitions): shard 0 ❌ 60-min timeout, shard 1 ❌ (this job, 80 min, timeout), shard 2 ✅ pass (job 89553950021), shard 3 ❌ fast-fail in 25 min (job 89553950119) — shard 3 is a different failure (not a timeout).
• All diffusion models were NOT cached: shard-2 "Diagnose HF cache" step printed ✗ NOT CACHED for all five (Wan2.1-T2V-1.3B, HunyuanVideo, Qwen-Image, FLUX.1-dev, FLUX.2-dev); shard-0/2 logs also show Model found in cache but incomplete, will download from HF Hub. Each partition therefore re-downloads multi-GB weights during the timed step.
• Model runtimes are large: shard-2 log shows single generations taking 124.47s (Z-Image), plus 30s+ text-encoding and 20s+ decode stages; several server (re)starts each set distributed timeout to 3600 seconds. Multiple such models per partition easily exceed 60 min.
• Step config: timeout-minutes: 60 on the "Run diffusion server tests (1-GPU)" step (.github/workflows/pr-test-amd-rocm720.yml:728).
• aiter is the Dockerfile default (no override) — unchanged and irrelevant to a wall-clock timeout.

Hypothesised Causes (with confidence)

• [HIGH] The partition-1 workload of the 1-GPU diffusion suite does not fit in the 60-minute step budget on the MI325 ROCm 7.2 runner.
  • Supporting: job ran the full 80 min; the identical step in sibling shard-0 of the same run printed the explicit "timed out after 60 minutes" error; every one of the last 8 scheduled runs of this partition failed with a ~71-80 min duration.
  • Disconfirming: shard-1's own log is unavailable, so the exact model that was mid-run at the 60-min mark cannot be pinned; shard-3 failed fast (25 min), showing not all partition failures in this job are timeouts.
  • How to verify: re-run the job and watch the step, or read shard-1's diffusion-failures artifact (diffusion-failures-amd-rocm720-1gpu-1-<attempt>) which uploads on always().
• [MEDIUM] Empty HF cache is a major contributor — every model shows ✗ NOT CACHED, so the timed step pays multi-GB download cost before/while running inference.
  • Supporting: shard-2 cache-diagnostic printed NOT CACHED for all 5 models; /sgl-data persistent-cache marker logic exists specifically to avoid this, implying the cache mount was empty/not populated on this runner.
  • Disconfirming: shard-2 (same empty cache) still passed within budget, so cache-miss alone isn't sufficient — the partition's model set/size matters too.
  • How to verify: confirm whether /sgl-data/hf-cache is a live mount on the shard-1 runner; pre-warm the cache and re-time the partition.
• [LOW] Uneven partitioning: partition 1 (and 0, 3) draw the heavier/slower models while partition 2 gets light ones.
  • Supporting: shard-2 finished with Current partition used fallback estimate: 300.0s and passed quickly; shards 0/1/3 all failed.
  • Disconfirming: no shard-1 log to enumerate its assigned models; partition assignment is dynamic.
  • How to verify: inspect run_suite.py --suite 1-gpu partition assignment for --partition-id 1 --total-partitions 4.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for multimodal-gen+timeout, diffusion+timeout+amd, diffusion+cache (via search/issues?q=repo:sgl-project/sglang+is:pr+is:open+…); no open PR targets the ROCm720 1-GPU diffusion suite timeout / HF-cache warming. (#29584 "fix three pre-existing MI325 CI failures" is unrelated to this step.)

Suggested Triage Steps

• Read the uploaded artifact diffusion-failures-amd-rocm720-1gpu-1-<run_attempt> from run 30115153295 to recover shard-1's model-by-model timing and see which model was in flight at the 60-min mark (compensates for the missing job log).
• Verify whether /sgl-data/hf-cache is a live persistent mount on the MI325 ROCm720 runners; if models re-download every run, pre-warming the cache may bring the partition under 60 min.
• If the workload genuinely exceeds 60 min even with a warm cache, consider re-balancing partitions (increase --total-partitions, or move a heavy model off partition 1) or raising the step timeout-minutes — a maintainer decision.
• Separately triage shard-3 (job 89553950119): it fast-failed in 25 min, a different signature from this timeout.

Status

• Persistence: Failing every scheduled run for ≥8 consecutive runs (since at least 2026-07-17), always ~71-80 min → 60-min step timeout.
• Scope: 3 of 4 partitions of multimodal-gen-test-1-gpu-amd-rocm720 failed in this run (shards 0, 1, 3); shard 2 passed. Shards 0 & 1 are the timeout cluster; shard 3 is a distinct fast-fail.
• Blocked work: Blocks 1-GPU diffusion (multimodal_gen) CI signal on AMD MI325 ROCm 7.2.
• In-flight fix: ❌ none found in open PRs.

multimodal-gen-test-1-gpu-amd-rocm720 (linux-mi325-1gpu-sglang, 3) — failed step(s): N/A

The report is complete. The background download that this notification refers to already succeeded and was used in the analysis.

To summarize the finding: this is not a test failure — the multimodal-gen-test-1-gpu-amd-rocm720 part-3 job failed because the self-hosted MI325 runner lost communication with the GitHub server mid-test (job annotation confirms it), which is why the job log blob is 404 BlobNotFound. It's a persistent infrastructure/runner-health issue affecting the diffusion 1-GPU shards (≥12 consecutive scheduled runs since 2026-07-13; part 1 lost the runner and part 0 timed out in the same run), unrelated to the head commit. No in-flight fix exists — it needs runner-side host/GPU log investigation, not a code PR.

stage-b-test-1-gpu-small-amd-mi35x-rocm720 (linux-mi35x-gpu-1) — failed step(s): Run test

Commit Info

• sglang: 69a3c54c
• aiter: 9127c94a (v0.1.17.dev110+g9127c94a1, source: dev image preserved — matches Dockerfile default)

Failed Tests

Test File	Test Function	Error	Log
test/registered/quant/test_quark_mxfp4.py	TestFP8ToMXFP4MOETP1.test_gsm8k	AssertionError: 0.912 not greater than 0.92 → retry() exceed maximum number of retries	Log

Failure Summary

The single failing test file in this job is test_quark_mxfp4.py. Its TestFP8ToMXFP4MOETP1.test_gsm8k case (model Qwen/Qwen3-30B-A3B-Instruct-2507-FP8, online quark_mxfp4 requant, TP1) runs a GSM8K eval and asserts accuracy > 0.92. Both the initial attempt (0.900) and the single retry (0.912) landed below the gate, so the retry wrapper raised and the test errored. The other 4 tests in the file passed or were skipped (Ran 5 tests … FAILED (errors=1, skipped=3)).

Regression Status

Known recurring / flaky (marginal accuracy gate) — not a new regression and not a code fault; the GSM8K accuracy for this MOE case oscillates on both sides of the fixed 0.92 threshold across runs.

Recent history of test_quark_mxfp4.py (TestFP8ToMXFP4MOETP1.test_gsm8k) in job stage-b-test-1-gpu-small-amd-mi35x-rocm720 (linux-mi35x-gpu-1) (most recent comparable completed scheduled runs: main, event=schedule):

Date	Run	Test File Status	Failed Function	Observed accuracy (attempt/retry)
Jul 25	30168318210	❌ Failed	TestFP8ToMXFP4MOETP1.test_gsm8k	0.900 / 0.912 (both < 0.92)
Jul 24	30115153295	❌ Failed	TestFP8ToMXFP4MOETP1.test_gsm8k	0.914 / 0.914
Jul 23	30031591211	✅ Passed	—	0.914 → 0.936 (passed on retry)
Jul 22	29944235234	❌ Failed	TestFP8ToMXFP4MOETP1.test_gsm8k	0.918 / 0.910
Jul 21	29855003895	✅ Passed	—	0.928 (first try)
Jul 20	29766743590	✅ Passed	—	0.924 (first try)
Jul 19	29697501385	⚪ N/A	—	test file not in this shard
Jul 18	29654571916	⚪ N/A	—	test file not in this shard

First observed in this shard: Jul 20. Failing in ~3 of the last 6 runs where it ran; no green streak — it flips pass/fail depending on GSM8K sampling. The Job column's overall pass/fail is not used; verdict is at the TestFP8ToMXFP4MOETP1.test_gsm8k level.

Failure Cluster

GSM8K accuracy gate flake on online-MXFP4 MOE (Qwen3-30B-A3B-FP8, threshold 0.92)

Facts (verified observations)

• Exact assertion (verbatim): AssertionError: 0.912 not greater than 0.92, followed by Exception: retry() exceed maximum number of retries. (job log lines 9216–9243).
• Failing test: TestFP8ToMXFP4MOETP1.test_gsm8k; [CI Test Method] TestFP8ToMXFP4MOETP1.test_gsm8k at log line 6232; model launched with --quantization quark_mxfp4, model_path='Qwen/Qwen3-8B-FP8' for the Dense case and Qwen/Qwen3-30B-A3B-Instruct-2507-FP8 for the failing MOE case.
• Threshold is a hard-coded constant in the test at 69a3c54c: test/registered/quant/test_quark_mxfp4.py:251 → self._test_gsm8k(accuracy_threshold=0.92) (class TestFP8ToMXFP4MOETP1, test/registered/quant/test_quark_mxfp4.py:239-251). The in-source comment states reference accuracy ~0.948.
• Observed GSM8K accuracy for this exact test across Jul 20–25: {0.900, 0.910, 0.912, 0.914, 0.918, 0.924, 0.928, 0.936} — spanning both sides of 0.92, i.e. the gate sits inside the run-to-run variance band.
• The sibling TestFP8ToMXFP4DenseTP1.test_gsm8k (threshold 0.868) passed this run at 0.902; only the MOE case (0.92 gate) fails.
• aiter is unchanged (Dockerfile default dev build preserved, no override): AITER_COMMIT_OVERRIDE: empty; [CI-AITER-CHECK] Dev/patched version detected … → skipping rebuild.

Hypothesised Causes (with confidence)

• [HIGH] The 0.92 accuracy gate for the online-MXFP4 MOE case is too tight relative to inherent GSM8K sampling variance, making the test intermittently fail.
  • Supporting: measured accuracy over 6 consecutive days brackets 0.92 (0.900–0.936); the test passes when a run samples ≥0.92 and fails otherwise, with no code change correlating to pass↔fail flips; aiter and the test threshold are constant across passing and failing runs.
  • Disconfirming: (none found) — no green baseline where accuracy is comfortably above the gate; values cluster just below the reference ~0.948, consistent with expected MXFP4 online-requant degradation rather than a regression.
  • How to verify: rerun this shard 5–10× at the same SHA; expect ~50% failure with accuracy in [0.90, 0.936].
• [LOW] Genuine accuracy degradation in the online MXFP4 requant path for Qwen3-30B-A3B MOE.
  • Supporting: absolute accuracy (~0.90–0.93) is ~2–5 pts under the ~0.948 reference.
  • Disconfirming: this gap is present in passing runs too (0.924, 0.928, 0.936 all below reference), so it does not explain the intermittent failures; the loss is expected and documented (Requantizing … may incur a loss in accuracy).
  • How to verify: compare accuracy against the BF16/FP8 baseline for the same model to quantify the true MXFP4 delta before adjusting the gate.

In-flight Fix Check

• Searched open PRs: is:pr is:open quark_mxfp4 (2 results) and is:pr is:open mxfp4 accuracy (125 results). None targets the TestFP8ToMXFP4MOETP1 GSM8K threshold. PR #29328 is the online-MXFP4-requant feature that introduced this test, not a threshold fix.
• In-flight fix: ❌ none found — needs new PR or investigation

Suggested Triage Steps

• Rerun test/registered/quant/test_quark_mxfp4.py on the linux-mi35x runner 5–10× at 69a3c54c to confirm the ~50% flake rate and capture the accuracy distribution.
• If confirmed flaky, consider loosening the TestFP8ToMXFP4MOETP1 gate (test/registered/quant/test_quark_mxfp4.py:251) to a value below the observed floor (e.g. ≤0.89, matching the sibling non-FP8 MOE class at 0.89), and/or increase the GSM8K sample count to reduce variance — maintainer's call.
• Before relaxing the gate, quantify the true MXFP4 online-requant accuracy delta vs the FP8 baseline for Qwen/Qwen3-30B-A3B-Instruct-2507-FP8 to ensure the lower floor still catches real regressions.

Status

• Persistence: Intermittent since at least Jul 20; failed Jul 22, 24, 25 (passed Jul 20, 21, 23) — ~3/6 recent scheduled runs, no stable green.
• Scope: Single test file / single function (test_quark_mxfp4.py :: TestFP8ToMXFP4MOETP1.test_gsm8k) in job stage-b-test-1-gpu-small-amd-mi35x-rocm720 (linux-mi35x-gpu-1); MI35x-only (gated behind mxfp_supported()).
• Blocked work: Flakes the pr-test-amd-rocm720 stage-b MI35x shard roughly half the time; no correctness signal lost on the passing sibling MXFP4 tests.
• In-flight fix: ❌ none found in open PRs.

multimodal-gen-test-2-gpu-amd-rocm720 (linux-mi325-2gpu-sglang, 0) — failed step(s): N/A

Commit Info

• sglang: 69a3c54c
• aiter: N/A — the job log blob does not exist (GitHub returns BlobNotFound/404), so no [CI-AITER-CHECK] markers are readable.

Failed Tests

Test File	Test Function	Error	Log
N/A — non-test / runner-level failure	N/A	Runner terminated mid-step; no test log uploaded	Job (log blob missing)

This is not a diagnosable pytest failure. The job runs python3 sglang/multimodal_gen/test/run_suite.py --suite 2-gpu --partition-id 0 --total-partitions 3 (step "Run diffusion server tests (2-GPU)", 150-min step timeout). The 2-gpu suite = parametrized test_server_2_gpu.py cases + standalone test/.../single_test_file/test_disagg_server.py and test_ar_models.py, runtime-partitioned into 3 shards. Which specific test in part 0 was running cannot be determined because no log was uploaded.

Failure Summary

Job shard (…, 0) of the 2-GPU diffusion suite is marked conclusion=failure, but its step 9 "Run diffusion server tests (2-GPU)" is still in_progress and steps 10/20 are pending in the GitHub API — i.e. the job never finished the test step cleanly. The log blob is absent (BlobNotFound). The job ran ~99 min (21:09:55→22:49:00 UTC), well under the 150-min step timeout, so this is not a step timeout. Sibling shards part 1 and part 2 of the same 2-GPU test passed in this run.

Regression Status

Known recurring failure (part-0 job level) — logs are never uploaded, so test-file granularity is unobtainable for any run in the window.

Recent history of job multimodal-gen-test-2-gpu-amd-rocm720 (linux-mi325-2gpu-sglang, 0) (most recent comparable completed scheduled runs: branch=main, event=schedule):

Date	Run	Job conclusion	Step-9 signature	Log
Jul 25	30168318210	❌ failure	step 9 stuck in_progress, 10/20 pending	missing (BlobNotFound)
Jul 24	30115153295	❌ failure	step 9 stuck in_progress, 10/20 pending	missing
Jul 23	30031591211	❌ failure	step 9 stuck in_progress, 10/20 pending	missing
Jul 22	29944235234	❌ failure	(job-level)	missing
Jul 21	29855003895	❌ failure	(job-level)	missing
Jul 20	29766743590	❌ failure	(job-level)	missing
Jul 19	29697501385	❌ failure	(job-level)	missing
Jul 18	29654571916	❌ failure	(job-level)	missing
Jul 17	29601526478	❌ failure	(job-level)	missing

Part 0 failed in every one of the last ~9 completed scheduled runs. No passing run found in the query window → treat as Never-passed within the queryable window at the job level. For the 3 most recent runs I verified the identical step-9-stuck / no-log signature (07-25, 07-24, 07-23), confirming a stable failure mode rather than varied per-run test failures. Durations were ~99–110 min each, all under the 150-min step timeout. (Note: workflow-level conclusion is failure for all runs, but that reflects the whole workflow; the row above is the specific part-0 job.)

Failure Cluster

2-GPU diffusion (part 0) runner termination — job fails with test step stuck in_progress and no log uploaded.

Facts (verified observations)

• Job 89705019759: status=completed, conclusion=failure; step 9 Run diffusion server tests (2-GPU) = in_progress, steps 10 & 20 = pending (GitHub jobs API, verified).
• Job log endpoint returns <Code>BlobNotFound</Code> (215-byte error body, HTTP 404) — no log was uploaded.
• Duration 21:09:55→22:49:00 UTC = ~99 min; step timeout is 150 min (.github/workflows/pr-test-amd-rocm720.yml:867). So not a step timeout.
• Sibling shards in the same run passed: multimodal-gen-test-2-gpu-amd-rocm720 (…, 1) ✅ and (…, 2) ✅.
• Same step-9-stuck + missing-log signature reproduced in jobs 89553949923 (Jul 24, ~105 min) and 89289299688 (Jul 23, ~110 min).
• Overall run 30168318210 is still in progress (33/50 jobs queued) at snapshot time; this specific job is completed.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [MEDIUM] The self-hosted MI325 2-GPU runner assigned to part 0 is lost/killed mid-test (runner-agent disconnect, host/GPU hang, or OOM reaping the runner), so GitHub marks the job failed while the step is still in_progress and no log is ever flushed.
  • Supporting: identical signature across ≥3 consecutive runs (step 9 stuck, steps 10/20 never start, log blob absent); runs ~100 min then dies under the 150-min timeout; only part 0 affected while parts 1/2 finish and upload logs normally.
  • Disconfirming: without a log or runner diagnostics I cannot confirm why the runner dies (GPU hang vs OOM vs infra); "same signature" is symptom-level, not proven single root cause.
  • How to verify: pull the runner-side diagnostic/_diag logs or host dmesg/rocm-smi for the runner that served part 0; re-run just part 0 via workflow_dispatch (target_stage=multimodal-gen-test-2-gpu-amd-rocm720) and watch the runner live.
• [LOW] A specific heavy case in part 0 (e.g. test_disagg_server.py or test_ar_models.py, each weighted 600 s) hangs and wedges the GPU, causing the runner to be reaped before the step completes.
  • Supporting: part 0 is the heavy-standalone shard per the workflow comment; ~100-min runtime is consistent with a long hang.
  • Disconfirming: no log to confirm which case was running; partition contents are runtime-computed and not pinned to part 0.
  • How to verify: reproduce part 0 locally/on a runner with SGLANG_DIFFUSION_ARTIFACT_DIR set and step timeout raised, capturing stdout so the hanging case is identifiable.

Because logs are missing for the current AND all historical runs, no hypothesis exceeds MEDIUM; a live re-run with runner-side capture is required to get a real stack.

In-flight Fix Check

• In-flight fix: ❌ none found targeting the part-0 runner termination / missing-log hang. Searched open PRs for diffusion 2-gpu amd, multimodal_gen rocm, diffusion runner hang. Related but non-matching AMD diffusion PRs exist (e.g. #31483, #31899, #27587); none address this failure mode.

Suggested Triage Steps

• Retrieve the runner-side diagnostic logs (Actions runner _diag/worker logs) and host dmesg/rocm-smi for the runner that ran part 0 in 30168318210 — the GitHub job log is empty, so root cause lives on the runner host.
• Manually re-run only this stage via workflow_dispatch with target_stage_select=multimodal-gen-test-2-gpu-amd-rocm720 and observe the runner live to catch whether/where it hangs.
• Add a shorter internal timeout / per-case watchdog inside run_suite.py for the 2-gpu part-0 cases so the harness self-terminates and flushes logs before the runner is reaped (would restore log visibility for future triage).
• If part 0 consistently corresponds to test_disagg_server.py/test_ar_models.py, run those two files in isolation on an MI325 2-GPU box to determine whether one wedges the GPU.

Status (factual)

• Persistence: Part-0 job has failed in every completed scheduled run in the queried window (Jul 17–25, ≥9 runs); identical step-9-stuck + missing-log signature verified for Jul 23/24/25.
• Scope: Isolated to shard (…, 0) of multimodal-gen-test-2-gpu-amd-rocm720; parts 1 & 2 pass. AMD ROCm 7.2 pipeline only.
• Blocked work: Removes CI signal for whatever tests land in 2-GPU diffusion partition 0 (disagg/AR/standalone 2-GPU cases).
• In-flight fix: ❌ none found — needs runner-side investigation or a harness-level watchdog to restore diagnosability.

---

⏳ 33 job(s) still running (run) — will update when complete

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-07-25 23:13 UTC)

[amd-bot]

nightly-test-amd-rocm720.yml — 18 failure(s)

Run started (UTC): 2026-07-24 17:58 UTC
Last scanned (UTC): 2026-07-25 22:47 UTC
Commits: sglang 69a3c54

Summary

Counts: 18 failures · 12 clusters · 9 clusters carrying over (≥3 days) · 1 new today (MiniMax-M2.5)

#	Cluster	Job	Test File	Test Function	Status	Hypothesis (confidence)
1	GLM-5.1-MXFP4 TP2 MI35x bring-up wedges runner (no logs)	nightly-2-gpu-mi35x-glm51-mxfp4 (Jul24)	N/A (infra)	N/A	Never-green ≥15 runs (Jul10–24); BlobNotFound	[LOW] runner starved/GPU-hang; no log to bisect
2	GLM-5.1-MXFP4 TP2 MI35x bring-up wedges runner (no logs)	nightly-2-gpu-mi35x-glm51-mxfp4 (Jul25)	test_glm51_mxfp4_tp2_gsm8k_mi35x.py	setUpClass	Never-green; log blob missing	[MEDIUM] DSA dpsk_v4_topk JIT fails for gfx950 / GPU hang
3	DeepSeek-V3.2 GSM8K < 0.93 (dsa/tilelang MI35x)	nightly-accuracy-8-gpu-mi35x-deepseek-v32 (Jul24)	test_deepseek_v32_eval_mi35x.py	test_deepseek_v32_accuracy	Never-passed since ~Jul13; 0.860	[MEDIUM] 0.93 threshold miscalibrated for this path
4	DeepSeek-V3.2 GSM8K < 0.93 (dsa/tilelang MI35x)	nightly-accuracy-8-gpu-mi35x-deepseek-v32 (Jul25)	test_deepseek_v32_eval_mi35x.py	test_deepseek_v32_accuracy	Never-passed; 0.875	[MEDIUM] threshold miscalibrated
5	InternVL CLIPImageProcessor no .tokenizer + Qwen VLM below threshold	nightly-accuracy-2-gpu-vlm (Jul24)	test_vlms_mmmu_eval_amd.py	test_mmmu_vlm_models (InternVL2_5-2B + 3 Qwen)	Never-passed since ≥Jul17	[MEDIUM] common.py:529 returns processor.tokenizer unguarded
6	InternVL CLIPImageProcessor no .tokenizer + Qwen VLM below threshold	nightly-accuracy-2-gpu-vlm (Jul25)	test_vlms_mmmu_eval_amd.py	test_mmmu_vlm_models (InternVL2_5-2B + 3 Qwen)	Never-passed	[HIGH] common.py:527-529 unguarded image-only processor
7	GPU memory access fault — bf16 Qwen3-30B-A3B MoE (MI355X)	nightly-accuracy-8-gpu-mi35x (Jul24)	test_qwen3_moe_eval_mi35x.py	test_qwen3_moe_accuracy	Never-passed since Jul17 (7/7)	[LOW] OOB in bf16 MoE/attn kernel gfx950; async fault, no stack
8	GPU memory access fault — bf16 Qwen3-30B-A3B MoE (MI355X)	nightly-accuracy-8-gpu-mi35x (Jul25)	test_qwen3_moe_eval_mi35x.py	test_qwen3_moe_accuracy	Never-passed (8/8)	[LOW] same; no captured kernel frame
9	hipErrorCapturedEvent in decode CUDA-graph capture (Qwen3-235B MXFP4)	nightly-8-gpu-mi35x-qwen3-235b-mxfp4 (Jul24)	test_qwen3_instruct_mxfp4.py	setUpClass	Never-passed since Jul14	[MEDIUM] embedding all-reduce (RCCL) polled mid-capture
10	hipErrorCapturedEvent in decode CUDA-graph capture (Qwen3-235B MXFP4)	nightly-8-gpu-mi35x-qwen3-235b-mxfp4 (Jul25)	test_qwen3_instruct_mxfp4.py	server launch (0 tests)	Never-passed	[MEDIUM] EP all-reduce falls back to torch.distributed.all_reduce, not capture-safe
11	EAGLE/MTP GSM8K accuracy below gate (DeepSeek-R1-MXFP4)	nightly-8-gpu-mi35x-deepseek-r1-mxfp4-tp4 (Jul24)	test_deepseek_r1_mxfp4_tp4_mtp_mi35x.py	test_gsm8k	Never-passed since Jul18; 0.848	[MEDIUM] spec path lossy / 0.944 threshold too high (sibling passes 0.95)
12	EAGLE/MTP GSM8K accuracy below gate (DeepSeek-R1-MXFP4)	nightly-8-gpu-mi35x-deepseek-r1-mxfp4-tp4 (Jul25)	test_deepseek_r1_mxfp4_tp4_mtp_mi35x.py	TestDeepSeekR1MXFP4TP4MTPMI35x.test_gsm8k	Never-passed (8/8); 0.838	[MEDIUM] spec-decode quality gap vs 0.951 sibling
13	GSM8K answer-extraction bug — run-on models (Mixtral)	nightly-accuracy-2-gpu (Jul24)	test_gsm8k_eval_amd.py	test_gsm8k_all_models (Mixtral-8x7B)	Persistent since Apr gsm8k migration; ≈0.31	[HIGH] get_answer_value() takes last number
14	NameError transfer_kv_mamba_lf_pf undefined on ROCm	nightly-test-1-gpu-kernel (Jul24)	test_transfer_mamba.py	test_mamba_kernel_* (12)	Never-passed since Jul17	[HIGH] import gated if _is_cuda: (missing _is_hip) memory_pool_host.py:41
15	GSM8K model force-FAILED — missing threshold (Qwen3-4B config gap)	nightly-accuracy-2-gpu (Jul25)	test_gsm8k_eval_amd.py	test_gsm8k_all_models (Qwen3-4B)	Since ~Jul22; score healthy 0.914	[HIGH] no MODEL_SCORE_THRESHOLDS entry → passed=None
16	DeepSeek-V4-Pro MTP server launch timeout (slow CPU FP8 wo_a dequant)	nightly-8-gpu-mi35x-deepseek-v4-pro-mtp (Jul24)	test_deepseek_v4_pro_fp4_mtp.py	test_a_gsm8k, test_b_bs_1_speed	Regressed Jul23 (passed Jul21–22); marginal	[MEDIUM] CPU _dequant_fp8 pushes launch past timeout
17	DeepSeek-V4-Flash GSM8K gate too-tight (0.91)	nightly-8-gpu-mi35x-deepseek-v4-flash (Jul24)	test_deepseek_v4_flash_fp8.py / _tbo.py	test_a_gsm8k (0.9098), test_gsm8k_tbo (0.9030)	2-day-old test; band straddles gate	[MEDIUM] 0.91 gate inside variance band
18	MiniMax-M2.5 GSM8K marginally below threshold	nightly-4-gpu-mi35x-minimax-m25 (Jul25)	test_minimax_m25_tp4_eval_mi35x.py	test_minimax_m25_tp4_accuracy	New today (passed Jul17–24); 0.929 vs 0.93	[MEDIUM] marginal-threshold flake (−0.001)

Cluster: GLM-5.1-MXFP4 TP2 MI35x bring-up wedges runner (no logs)

• Affected: rows 1–2, job nightly-2-gpu-mi35x-glm51-mxfp4-rocm720 both runs. Same suite (test_glm51_mxfp4_tp2_gsm8k_mi35x.py), same runner class linux-mi35x-gpu-2. Both today's logs are BlobNotFound; historical logs show dpsk_v4_topk JIT-compile fail (exit -9) and GPU Hang (exit 134). DeepSeek-R1-MXFP4 sibling on same runner passes → workload-specific, not host-wide.
• In-flight fix: ✅ runner-health #31068; GLM-5.x/DSA enablement #32175, #30808, #32098, #31727. No dedicated gate fix.
• Triage: repro exact launch cmd on gfx950; capture dmesg/rocm-smi; fix log-upload so failures are diagnosable.

Cluster: DeepSeek-V3.2 GSM8K < 0.93 (dsa/tilelang MI35x)

• Affected: rows 3–4, test_deepseek_v32_eval_mi35x.py::test_deepseek_v32_accuracy. 7/7 eval-reaching runs land 0.855–0.905 (mean ≈0.88), never 0.93; server healthy (200/200). In-flight: ❌ none.
• Triage: re-run at N=1000 to cut variance; A/B vs CUDA reference to decide threshold-too-tight vs real DSA quality gap.

Cluster: InternVL CLIPImageProcessor no .tokenizer + Qwen VLM below threshold

• Affected: rows 5–6, test_vlms_mmmu_eval_amd.py::test_mmmu_vlm_models. Hard error is InternVL2_5-2B launch crash at common.py:529 return processor.tokenizer (in-code comment already names InternVL); crash aborts suite before 7 of 11 models run. Co-located Qwen2-VL/2.5-VL/3-VL all below MMMU thresholds. In-flight: ❌ none.
• Triage: guard get_tokenizer_from_processor for image-only processors, or add InternVL to AMD_FAILING_VLM_MODELS; move popen_launch_server inside per-model try/except so one crash stops masking the rest.

Cluster: GPU memory access fault — bf16 Qwen3-30B-A3B MoE (MI355X)

• Affected: rows 7–8, test_qwen3_moe_eval_mi35x.py::test_qwen3_moe_accuracy. Fault (…009000, all 8 ranks) on 2nd prefill batch after graph capture; only bf16 MoE faults (quantized MoE siblings pass). Async fault, no kernel stack. In-flight: ❌ none.
• Triage: repro with HIP_LAUNCH_BLOCKING=1 + --disable-cuda-graph + GPU coredump to pin the faulting kernel; image/date-based bisection (no green run to bisect).

Cluster: hipErrorCapturedEvent in decode CUDA-graph capture (Qwen3-235B MXFP4)

• Affected: rows 9–10, test_qwen3_instruct_mxfp4.py. RCCL ProcessGroupNCCL watchdog polls an event recorded inside the capturing stream during a MoE/embedding all-reduce; torch symmetric-memory disabled logged → EP all-reduce falls through to raw torch.distributed.all_reduce (parallel_state.py:960). In-flight: ❌ (adjacent #23581, #27650 target the inverse aiter-AR path).
• Triage: repro with AMD_SERIALIZE_KERNEL=3; force a graph-safe EP all-reduce or exclude it from capture.

Cluster: EAGLE/MTP GSM8K accuracy below gate (DeepSeek-R1-MXFP4)

• Affected: rows 11–12, test_deepseek_r1_mxfp4_tp4_mtp_mi35x.py::test_gsm8k. MTP ~0.84–0.86 vs non-MTP sibling ~0.95 same run/model/GPU; accept rate ~0.6 (not crashing). Threshold 0.944 > sibling's own 0.93. In-flight: ❌ (#31214 is temperature>0, adjacent not fix).
• Triage: diff greedy tokens spec-on vs spec-off at temp=0; decide spec-path fix vs threshold re-baseline with owner.

Cluster: GSM8K answer-extraction bug — run-on models (Mixtral)

• Affected: row 13, test_gsm8k_eval_amd.py::test_gsm8k_all_models (Mixtral ≈0.31/0.69). get_answer_value() returns the last number; run-on models emit extra #### N pairs. Same signature on NVIDIA nightly-test-text-accuracy-2-gpu-h100. In-flight: ✅ #29933 (open draft since Jul02) — un-draft + merge.

Cluster: NameError transfer_kv_mamba_lf_pf undefined on ROCm

• Affected: row 14, test_transfer_mamba.py::test_mamba_kernel_* (12). Import gated if _is_cuda: at memory_pool_host.py:41 never binds symbol on HIP. In-flight: ✅ #31741 (open since Jul20, blocked) — unblock; confirm a ROCm kernel exists before widening the gate.

Cluster: GSM8K model force-FAILED — missing threshold (Qwen3-4B config gap)

• Affected: row 15, test_gsm8k_eval_amd.py::test_gsm8k_all_models (Qwen3-4B). Score healthy 0.914 but absent from MODEL_SCORE_THRESHOLDS → passed=None → unconditional FAIL. In-flight: ❌.
• Triage: add "Qwen/Qwen3-4B" threshold entry, or make missing-threshold a skip not a FAIL. (Distinct from the Mixtral extraction cluster despite same file.)

Cluster: DeepSeek-V4-Pro MTP server launch timeout (slow CPU FP8 wo_a dequant)

• Affected: row 16, test_deepseek_v4_pro_fp4_mtp.py (test_a_gsm8k, test_b_bs_1_speed). py-spy shows CPU _dequant_fp8 (deepseek_v4.py:3143) during NextN load; 224 soft-watchdog timeouts; pass/fail is a timing boundary (Jul21 passed with same signature). In-flight: ✅ #28729 (offloads wo_a dequant to GPU).

Cluster: DeepSeek-V4-Flash GSM8K gate too-tight (0.91)

• Affected: row 17, test_deepseek_v4_flash_fp8.py::test_a_gsm8k (0.9098) + _tbo.py::test_gsm8k_tbo (0.9030). 2-day-old test; band 0.903–0.926 straddles 0.91 gate. In-flight: ❌. Triage: same-SHA reruns to separate variance from regression before touching threshold.

Cluster: MiniMax-M2.5 GSM8K marginally below threshold

• Affected: row 18, test_minimax_m25_tp4_eval_mi35x.py::test_minimax_m25_tp4_accuracy. New today: 0.929 vs 0.93 (−0.001); last 4 = 0.929/0.945/0.942/0.931; invalid=0; aiter unchanged; 37-commit window has no AMD-path change. In-flight: ❌. Triage: rerun 2–3× at HEAD; expect recovery to ~0.94.

Suggested triage order

Rank	Triage step	Targets	Why this first
1	Un-draft + merge #29933; A/B Mixtral extractor	row 13 (Mixtral)	Verified [HIGH] cause + ready fix, also fixes NVIDIA nightly
2	Unblock #31741 (confirm ROCm mamba kernel exists)	row 14 (mamba NameError)	[HIGH] structural, in-flight fix already open
3	Add Qwen/Qwen3-4B threshold (or skip-on-missing)	row 15 (config gap)	Trivial [HIGH] fix; a healthy model is force-failing the whole job
4	Verify #28729 rebases; re-run V4-Pro MTP 3–5×	row 16 (dequant timeout)	In-flight fix + quantify the timing-boundary flake
5	Repro capture-crash + GPU-fault with AMD_SERIALIZE_KERNEL=3/HIP_LAUNCH_BLOCKING=1	rows 7–10 (2 largest kernel clusters, 4 jobs)	Biggest blast radius; needs synchronous stack to localize
6	Calibrate accuracy gates via reference/high-N reruns	rows 3–4, 11–12, 17–18	Distinguish threshold-too-tight vs real quality/spec-decode gap before editing gates

---

Job	Failed Steps	Started
nightly-test-1-gpu-kernel-rocm720	Nightly Kernel Benchmark ROCm 7.2 (1-GPU)	2026-07-24T19:35
nightly-accuracy-2-gpu-vlm-rocm720	Nightly Accuracy Test ROCm 7.2 (2-GPU VLM MMMU)	2026-07-24T19:37
nightly-accuracy-2-gpu-rocm720	N/A	2026-07-24T19:37
nightly-2-gpu-mi35x-glm51-mxfp4-rocm720	N/A	2026-07-24T17:58
nightly-8-gpu-mi35x-deepseek-v4-pro-mtp-rocm720	Accuracy + MTP Accept Length Test MI35x ROCm 7.2 (8-GPU DeepSeek-V4-Pro FP8 MTP, triton)	2026-07-24T17:58
nightly-8-gpu-mi35x-deepseek-v4-flash-rocm720	Accuracy + Performance Test MI35x ROCm 7.2 (8-GPU DeepSeek-V4-Flash FP8 + FP4, triton)	2026-07-24T18:23
nightly-accuracy-8-gpu-mi35x-rocm720	Accuracy Test MI35x ROCm 7.2 (8-GPU GPT-OSS)	2026-07-24T17:58
nightly-8-gpu-mi35x-deepseek-r1-mxfp4-tp4-rocm720	Accuracy Test MI35x ROCm 7.2 (DeepSeek-R1-MXFP4 TP4 MTP)	2026-07-24T21:59
nightly-8-gpu-mi35x-qwen3-235b-mxfp4-rocm720	Accuracy Test + Performance Test MI35x ROCm 7.2 (8-GPU Qwen3-235B-MXFP4)	2026-07-24T20:38
nightly-accuracy-8-gpu-mi35x-deepseek-v32-rocm720	Accuracy Test MI35x ROCm 7.2 (8-GPU DeepSeek-V3.2)	2026-07-24T19:28
nightly-2-gpu-mi35x-glm51-mxfp4-rocm720	N/A	2026-07-25T17:45
nightly-accuracy-8-gpu-mi35x-rocm720	Accuracy Test MI35x ROCm 7.2 (8-GPU GPT-OSS)	2026-07-25T18:21
nightly-accuracy-8-gpu-mi35x-deepseek-v32-rocm720	Accuracy Test MI35x ROCm 7.2 (8-GPU DeepSeek-V3.2)	2026-07-25T18:22
nightly-accuracy-2-gpu-rocm720	N/A	2026-07-25T19:26
nightly-4-gpu-mi35x-minimax-m25-rocm720	Accuracy Test MI35x ROCm 7.2 (4-GPU MiniMax-M2.5)	2026-07-25T20:51
nightly-accuracy-2-gpu-vlm-rocm720	Nightly Accuracy Test ROCm 7.2 (2-GPU VLM MMMU)	2026-07-25T20:34
nightly-8-gpu-mi35x-qwen3-235b-mxfp4-rocm720	Accuracy Test + Performance Test MI35x ROCm 7.2 (8-GPU Qwen3-235B-MXFP4)	2026-07-25T21:40
nightly-8-gpu-mi35x-deepseek-r1-mxfp4-tp4-rocm720	Accuracy Test MI35x ROCm 7.2 (DeepSeek-R1-MXFP4 TP4 MTP)	2026-07-25T21:22

Per-Job Analysis

nightly-test-1-gpu-kernel-rocm720 — failed step(s): Nightly Kernel Benchmark ROCm 7.2 (1-GPU)

Commit Info

• sglang: 2428f561
• aiter: N/A (kernel-benchmark job log emits no [CI-AITER-CHECK] markers)

Failed Tests

All 12 failures share the identical NameError at memory_pool_host.py:407. Test file: test/registered/kernels/ops/mamba/test_transfer_mamba.py.

Test File	Test Function	Error	Log
test/registered/kernels/ops/mamba/test_transfer_mamba.py	test_mamba_kernel_backup_load_roundtrip (page_first/page_first_direct × float16/bfloat16)	NameError: name 'transfer_kv_mamba_lf_pf' is not defined	Log
test/registered/kernels/ops/mamba/test_transfer_mamba.py	test_mamba_kernel_single_item (4 params)	NameError: name 'transfer_kv_mamba_lf_pf' is not defined	Log
test/registered/kernels/ops/mamba/test_transfer_mamba.py	test_mamba_kernel_full_indices (4 params)	NameError: name 'transfer_kv_mamba_lf_pf' is not defined	Log

test_mamba_kernel_empty_indices (4 params) PASSED — it returns early on src_indices.numel() == 0 before reaching the undefined symbol.

Failure Summary

The kernel-benchmark job's runner (test_transfer_mamba.py) reports 20/21 passed; the single failing file raises NameError: name 'transfer_kv_mamba_lf_pf' is not defined in MambaPoolHost._copy_tensor_all_layers_lf_pf (python/sglang/srt/mem_cache/memory_pool_host.py:407). 12 of the file's parametrized cases fail with this same error; the whole job then exits 255. The failure only touches the mamba host-pool io_backend="kernel" path.

Regression Status

Never-passed on ROCm (AMD port gap, NOT a fresh regression).

Recent history of test_transfer_mamba.py (test_mamba_kernel_*) in job nightly-test-1-gpu-kernel-rocm720 (most recent comparable completed scheduled runs: branch=main, event=schedule, completed only):

Date	Run	Test File Status	Failed Function	Error
Jul 24	30115085856	❌ Failed	test_mamba_kernel_* (12)	NameError: transfer_kv_mamba_lf_pf
Jul 23	30031529424	❌ Failed	test_mamba_kernel_* (12)	same
Jul 22	29944173150	❌ Failed (path jit/)	test_mamba_kernel_* (12)	same
Jul 21	29854963573	❌ Failed (path jit/)	test_mamba_kernel_* (12)	same
Jul 19	29697452589	❌ Failed (path jit/)	test_mamba_kernel_* (12)	same
Jul 17	29601455852	❌ Failed (path jit/)	test_mamba_kernel_* (12)	same — first appearance
Jul 14	29355338821	✅ N/A	—	test did not exist (no transfer_mamba in log)

First observed failure: 2026-07-17 (run 29601455852). No passing run for this test on ROCm exists — it has failed on every scheduled run since it was introduced (between Jul 14 and Jul 17). The test file was later relocated from test/registered/jit/ to test/registered/kernels/ops/mamba/, but the test functions and error are unchanged. (Jul 20 and earlier runs like Jul 12–09 were cancelled and excluded.)

Failure Cluster

NameError: transfer_kv_mamba_lf_pf undefined on ROCm (CUDA-only import gate)

Facts (verified observations)

• Error verbatim (12×): NameError: name 'transfer_kv_mamba_lf_pf' is not defined, raised at python/sglang/srt/mem_cache/memory_pool_host.py:407 inside MambaPoolHost._copy_tensor_all_layers_lf_pf, reached via backup_from_device_all_layer(..., io_backend="kernel") (memory_pool_host.py:481).
• The symbol is imported only under a CUDA-only gate — memory_pool_host.py:41 if _is_cuda: → from sglang.kernels.ops.mamba.transfer_mamba import transfer_kv_mamba_lf_pf, transfer_kv_mamba_pf_lf. The sibling sgl_kernel.kvcacheio import above it (line 32) uses if _is_cuda or _is_hip:. _is_cuda = is_cuda() returns False on ROCm/HIP, so on AMD these two mamba symbols are never bound.
• Reproducibility: same 12 cases fail in every completed scheduled run Jul 17 → Jul 24 (7 runs verified); test_mamba_kernel_empty_indices passes because of the early numel()==0 return.
• Runner: linux-mi325-1gpu-sglang (AMD MI325, ROCm 7.2). NVIDIA runs this path with the symbol defined, so the failure is AMD-only.
• The local sglang checkout is a shallow clone (fetch-depth: 1); git log -S blame therefore only surfaces 0a6d1930 (#30540, Jul 22), which is a squash-merge artifact and post-dates the first failure by 5 days — so it is NOT the origin.

Hypothesised Causes (with confidence)

• [HIGH] The mamba transfer-kernel import is gated if _is_cuda: (memory_pool_host.py:41) instead of if _is_cuda or _is_hip:, so on ROCm transfer_kv_mamba_lf_pf / transfer_kv_mamba_pf_lf are never imported; the io_backend="kernel" code path then hits an undefined name.
  • Supporting: the NameError line and the import gate are both directly readable in the current source; the sibling kvcacheio import one block above correctly includes _is_hip; the test unconditionally calls the io_backend="kernel" path (test_transfer_mamba.py:217-219), and the same error appears verbatim on every AMD run since the test was added.
  • Disconfirming: (none found) — the error is structural and reproduces on 100% of ROCm runs, independent of any specific recent commit.
  • How to verify: on an AMD MI325 ROCm container, python3 -c "import sglang.srt.mem_cache.memory_pool_host as m; print(hasattr(m,'transfer_kv_mamba_lf_pf'))" → expect False; changing the gate to if _is_cuda or _is_hip: (or otherwise providing a ROCm mamba kernel) should bind the symbol.

In-flight Fix Check

• In-flight fix: ✅ #31741 "[AMD] Enable mamba JIT transfer kernel on ROCm (fix transfer_kv_mamba NameError)" — open since 2026-07-20, not draft, mergeable_state: blocked (last updated 2026-07-23). This directly targets the failing symbol; chase reviewers / unblock rather than opening a duplicate.
• Search run: search/issues?q=repo:sgl-project/sglang+is:pr+is:open+transfer_kv_mamba → 1 hit (#31741). Related but broader: #31780 "Fix mamba mem_cache tests on non-CUDA (XPU) devices".

Suggested Triage Steps

• Verify #31741 resolves the ROCm import gate (memory_pool_host.py:41) and re-run test/registered/kernels/ops/mamba/test_transfer_mamba.py on an AMD MI325 ROCm 7.2 runner; it is currently blocked — identify and clear the blocking check/review.
• If #31741 needs more work, confirm a ROCm-capable transfer_kv_mamba_lf_pf kernel actually exists before merely widening the gate to if _is_cuda or _is_hip: — otherwise the NameError becomes a different kernel-launch failure.
• Interim option: quarantine/skip the io_backend="kernel" mamba cases on ROCm until #31741 lands, so the kernel nightly stops red-flagging on a known unimplemented path.

Status

• Persistence: Failing every completed scheduled run since 2026-07-17 (7+ consecutive runs; ~7 days). No green run for this test on ROCm.
• Scope: nightly-test-1-gpu-kernel-rocm720 job of nightly-test-amd-rocm720.yml; AMD-only (NVIDIA unaffected). 12 parametrized cases in one test file; fails the whole kernel-benchmark job (exit 255).
• Blocked work: mamba host-pool kernel-backend D2H/H2D transfer path unverified on ROCm.
• In-flight fix: ✅ #31741 (open since 2026-07-20, blocked).

nightly-accuracy-2-gpu-vlm-rocm720 — failed step(s): Nightly Accuracy Test ROCm 7.2 (2-GPU VLM MMMU)

Commit Info

• sglang: 2428f561
• aiter: 9127c94a (v0.1.17.dev110+g9127c94a1, source: dev image preserved — Dev/patched version detected → skipping rebuild)

Failed Tests

All failures are sub-cases of the single test method test_mmmu_vlm_models in TestNightlyVLMMmmuEvalAMD. The unittest run ends Ran 1 test … FAILED (errors=1) — the errors=1 is the InternVL server crash; the accuracy failures never reach their assertion (see Failure Summary).

Test File	Test Function (sub-case)	Error	Log
test/registered/amd/accuracy/mi30x/test_vlms_mmmu_eval_amd.py	test_mmmu_vlm_models (model=OpenGVLab/InternVL2_5-2B)	AttributeError: 'CLIPImageProcessor' object has no attribute 'tokenizer' → Exception: Server process exited with code 1 (uncaught, aborts test)	Log
test/registered/amd/accuracy/mi30x/test_vlms_mmmu_eval_amd.py	test_mmmu_vlm_models (model=Qwen/Qwen3-VL-30B-A3B-Instruct)	MMMU score 0.260 < threshold 0.29	Log
test/registered/amd/accuracy/mi30x/test_vlms_mmmu_eval_amd.py	test_mmmu_vlm_models (model=Qwen/Qwen2.5-VL-7B-Instruct)	MMMU score 0.280 < threshold 0.33	Log
test/registered/amd/accuracy/mi30x/test_vlms_mmmu_eval_amd.py	test_mmmu_vlm_models (model=Qwen/Qwen2-VL-7B-Instruct)	MMMU score 0.290 < threshold 0.30	Log

Failure Summary

The nightly-accuracy-2-gpu-vlm-rocm720 job runs test_mmmu_vlm_models, which iterates 11 VLMs. The job fails for two independent reasons. First, three Qwen VLMs score below their configured MMMU thresholds (Qwen2-VL-7B 0.290<0.30, Qwen2.5-VL-7B 0.280<0.33, Qwen3-VL-30B 0.260<0.29). Second, the InternVL2_5-2B server fails to launch with AttributeError: 'CLIPImageProcessor' object has no attribute 'tokenizer'; because popen_launch_server is called outside the model loop's try/except (test line 216), this uncaught exception aborts the entire test after only 4 of 11 models run — before the accuracy AssertionError in _check_results can even fire. The reported errors=1 is therefore the InternVL crash, not the accuracy failures.

Regression Status

Known recurring / Never-passed (no green run found in the queryable window).

Recent history of test_vlms_mmmu_eval_amd.py (test_mmmu_vlm_models) in job nightly-accuracy-2-gpu-vlm-rocm720 (most recent comparable completed scheduled runs: branch=main, event=schedule, completed only; cancelled runs Jul 20/15 skipped):

Date	Run	Test File Status	InternVL crash?	Qwen accuracy fails
Jul 24	30115085856	❌ Failed	✅ yes	Qwen2-VL 0.29, Qwen2.5-VL 0.28, Qwen3-VL 0.26
Jul 23	30031529424	❌ Failed	✅ yes	0.29 / 0.28 / 0.26
Jul 22	29944173150	❌ Failed	✅ yes	0.29 / 0.28 / 0.26
Jul 21	29854963573	❌ Failed	✅ yes	0.28 / 0.29 / 0.28
Jul 19	29697452589	❌ Failed	✅ yes	Qwen2-VL 0.30 ✅, 0.28, 0.26
Jul 18	29654521083	❌ Failed	✅ yes	0.29 / 0.29 / 0.26
Jul 17	29601455852	❌ Failed	✅ yes	Qwen2-VL 0.30 ✅, 0.28, 0.26
Jul 16	29521474931	❌ Failed	✅ yes	0.28 / 0.29 / 0.26
Jul 14	29355338821	❌ Failed	✅ yes	Qwen2-VL 0.30 ✅, 0.28, 0.26

The test file has failed in every queryable completed scheduled run (Jul 14 → Jul 24). The InternVL2_5-2B CLIPImageProcessor crash is present in all of them. No passing run exists in the window, so this is not a regression introduced by commit 2428f561. Qwen accuracy scores are stable-but-below-threshold; Qwen2-VL-7B oscillates right at its 0.30 boundary (0.28–0.30).

Failure Cluster

Two distinct clusters in one job: (1) InternVL processor missing .tokenizer — server launch crash; (2) Qwen VLM MMMU score below AMD threshold.

Facts (verified observations)

• Verbatim crash (all runs): AttributeError: 'CLIPImageProcessor' object has no attribute 'tokenizer' at python/sglang/srt/utils/hf_transformers/common.py:529 (get_tokenizer_from_processor, called from tokenizer_manager.py:3217 → processor.py:271).
• python/sglang/srt/utils/hf_transformers/common.py:524-529: get_tokenizer_from_processor does return processor.tokenizer for any non-PreTrainedTokenizerBase processor; its own comment reads "Some models don't have an available processor, e.g.: InternVL" — the image-only CLIPImageProcessor returned for InternVL2_5-2B has no .tokenizer.
• Test-harness fact: popen_launch_server(...) at test_vlms_mmmu_eval_amd.py:216 is outside the per-model try/except (which begins at line 224), so a launch failure propagates uncaught and aborts the whole method. Only 4 of 11 (Models to test: 11) models ran.
• InternVL2_5-2B is not in AMD_FAILING_VLM_MODELS (only zai-org/GLM-4.1V-9B-Thinking is excluded, test_vlms_mmmu_eval_amd.py:143-147), so it is still launched despite crashing every run.
• Accuracy (this run): Qwen2-VL-7B 0.290/0.30, Qwen2.5-VL-7B 0.280/0.33, Qwen3-VL-30B 0.260/0.29. All 100-sample MMMU evals.
• aiter unchanged across runs: dev image v0.1.17.dev110+g9127c94a1 preserved ([CI-AITER-CHECK] markers); no override.

Hypothesised Causes (with confidence)

• [MEDIUM] InternVL crash: get_tokenizer_from_processor (common.py:529) unconditionally returns processor.tokenizer, but InternVL2_5-2B's get_processor yields a CLIPImageProcessor (image-only, no tokenizer).
  • Supporting: stack trace terminates exactly at common.py:529 return processor.tokenizer; the function comment explicitly names InternVL as a model "without an available processor"; reproduces identically in every run.
  • Disconfirming: the guard is longstanding and the failure predates the queryable window, so I cannot pin it to a specific commit — it may be a transformers-version interaction (InternVL uses trust_remote_code + a downloaded tokenization_internlm2.py) rather than a pure sglang bug. Cannot A/B without a repro.
  • How to verify: locally launch sglang serve --model-path OpenGVLab/InternVL2_5-2B --trust-remote-code in the ROCm CI image; inspect the type returned by get_processor; check whether processor should be built via the InternVL-specific processor path rather than AutoProcessor falling back to CLIPImageProcessor.
• [LOW] Qwen accuracy below threshold is a threshold/hardware-calibration issue, not a model regression.
  • Supporting: scores are stable across 9 runs (Qwen3-VL flat at 0.26, Qwen2.5-VL flat at 0.28–0.29); Qwen2-VL-7B sits exactly on its 0.30 boundary and flips pass/fail run-to-run — classic threshold-too-tight signature for a noisy 100-sample eval on ROCm.
  • Disconfirming: a genuine numerical regression in AMD attention/RoPE (aiter fused RoPE warns of "lower precision") could also depress scores; cannot rule out without an NVIDIA-vs-AMD score comparison.
  • How to verify: compare these MMMU scores against the NVIDIA nightly for the same models/commit; if AMD is consistently ~0.02–0.05 lower, investigate ROCm attention/RoPE precision; otherwise recalibrate thresholds.

In-flight Fix Check

• InternVL crash: In-flight fix: ❌ none found — searched open PRs for InternVL+processor, get_tokenizer_from_processor, CLIPImageProcessor+tokenizer; no PR addresses the missing-.tokenizer path.
• Qwen VLM accuracy: In-flight fix: ❌ none found — searched VLM+mmmu+accuracy; no PR adjusts these AMD thresholds or the score regression.

Suggested Triage Steps

• Make the launch failure non-fatal to the suite: move popen_launch_server at test_vlms_mmmu_eval_amd.py:216 inside the per-model try/except, so one model's crash doesn't mask the other 7 unrun models. (Test-infra improvement.)
• For InternVL: either harden get_tokenizer_from_processor (common.py:524-529) to handle image-only processors (e.g. load the tokenizer separately when processor lacks .tokenizer), or add OpenGVLab/InternVL2_5-2B to AMD_FAILING_VLM_MODELS to unblock the rest of the suite while the processor path is fixed.
• For Qwen accuracy: compare MMMU scores against the NVIDIA nightly at the same commit; if AMD is systematically lower, bisect ROCm attention/RoPE precision; if scores are simply noisy near the boundary, recalibrate the three thresholds (esp. Qwen2-VL-7B 0.30, which flips run-to-run).

Status

• Persistence: Failing in every completed scheduled run in the queryable window (Jul 14 → Jul 24, 2026); no green run found. InternVL crash present in all 9 runs.
• Scope: nightly-accuracy-2-gpu-vlm-rocm720 job of nightly-test-amd-rocm720.yml. Two independent failure modes (InternVL launch crash + 3 Qwen accuracy shortfalls) within one test method; the crash aborts 7 of 11 models before they run.
• Blocked work: Full AMD ROCm 7.2 VLM MMMU accuracy signal — 7 of 11 configured models (MiniCPM, deepseek-vl2, Janus-Pro, Kimi-VL, MiMo-VL, DeepSeek-OCR-2, and InternVL itself) never execute due to the early abort.
• In-flight fix: ❌ none found for either failure mode.

nightly-accuracy-2-gpu-rocm720 — failed step(s): N/A

Commit Info

• sglang: 2428f561 — "[Bugfix] Fix Kimi-Linear state transfer across heterogeneous TP (#32262)"
• aiter: v0.1.17.dev110+g9127c94a1 (source: dev image; AITER_COMMIT_OVERRIDE empty, [CI-AITER-CHECK] reports "Dev/patched version detected → skipping rebuild"; Dockerfile default is aiter 9127c94a)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi30x/test_gsm8k_eval_amd.py	TestNightlyGsm8KEval.test_gsm8k_all_models (subTest model=mistralai/Mixtral-8x7B-Instruct-v0.1)	AssertionError: The following models failed: — Mixtral gsm8k score ≈0.31 < threshold 0.69	Job

Note: Qwen/Qwen3-4B prints Status: ❌ FAILED (threshold: N/A) in the per-model loop, but this is cosmetic — it is missing from MODEL_SCORE_THRESHOLDS, so check_model_scores classifies it as ⚠️ NO THRESHOLD and does not add it to failed_models. It is not the job-failing cause.

Failure Summary

The nightly-accuracy-2-gpu-rocm720 job runs test_gsm8k_all_models, which serves a list of instruction models and gates each on a gsm8k score threshold. Models with defined thresholds that were visible in the log passed (Llama-3.1-8B 0.828≥0.80, Qwen3-8B 0.916≥0.76, Llama-3.1-70B 0.930≥0.89). The job fails because mistralai/Mixtral-8x7B-Instruct-v0.1 scores ≈0.31 against its 0.69 threshold, causing check_model_scores (test file line 199-201) to raise AssertionError.

Regression Status

Known recurring / persistent failure (not introduced by 2428f561). The gsm8k accuracy test file has failed in every recent completed scheduled run with the identical Mixtral ≈0.31 signature.

Recent history of test_gsm8k_eval_amd.py (test_gsm8k_all_models) in job nightly-accuracy-2-gpu-rocm720 (most recent comparable completed scheduled runs; branch=main, event=schedule):

Date	Run	Test File Status	Failing model (score / threshold)
Jul 24	run	❌ Failed	Mixtral-8x7B-v0.1 (score beyond 4.2MB log cap; job=failure)
Jul 23	run	❌ Failed	Mixtral-8x7B-v0.1 (0.311 / 0.69)
Jul 22	run	❌ Failed	Mixtral-8x7B-v0.1 (0.323 / 0.69)
Jul 21	run	❌ Failed	Mixtral (score past log cap; job=failure)
Jul 19	run	❌ Failed	Mixtral-8x7B-v0.1 (0.316 / 0.69)
Jul 18	run	❌ Failed	Mixtral (score past log cap; job=failure)

No passing run for this test file exists in the queryable window (Jul 14–24; Jul 15/16/20 runs were cancelled, not completed). Per PR #29933, this has been red since the mgsm→gsm8k eval migration (#21931, 2026-04-07) — effectively never-passed for run-on models.

Failure Cluster

GSM8K accuracy gate failure on run-on models (Mixtral/Mistral) — eval answer-extraction picks wrong number.

Facts (verified observations)

• simple_eval_mixed_prefix_gsm8k.py:37-41 — get_answer_value() does numbers = re.findall(r"-?\d+\.?\d*", answer_str) then return ast.literal_eval(numbers[-1]), i.e. takes the last number in the model response (verified in current tree at 2428f561).
• Threshold table MODEL_SCORE_THRESHOLDS in test_gsm8k_eval_amd.py:46 sets Mixtral-8x7B-v0.1 = 0.69; test_gsm8k_eval_amd.py:199-201 raises AssertionError for any model below its threshold.
• Mixtral score is stable and deterministic across nights: 0.311/0.318 (Jul 23, 3 retry attempts), 0.323 (Jul 22), 0.316 (Jul 19) — all ≈0.31, i.e. not a flake. The 3× retry loop (test_gsm8k_eval_amd.py:275) re-ran and got ≈0.31 every attempt.
• Other models scored normally with the same harness (Llama-3.1-8B 0.82–0.83, Qwen3-8B 0.91–0.92, Llama-3.1-70B 0.92–0.93), so the low score is specific to run-on-style output, not a global harness break.
• Qwen/Qwen3-4B is in DEFAULT_MODEL_NAME_FOR_NIGHTLY_EVAL_TP1 (test_utils.py:154) but absent from MODEL_SCORE_THRESHOLDS → ⚠️ NO THRESHOLD (not a real failure).
• Log limitation: GitHub's job-log endpoint caps at ~4.2MB; today's log (and every historical log) truncates during the Mixtral eval, so today's exact Mixtral score and the final failing-models list are not directly captured. Additional models later in the run (Qwen2-57B-A14B thr 0.76, FP8 variants, Qwen3-30B thr 0.86) are beyond the cap and unverified.

Hypothesised Causes (with confidence)

• [HIGH] GSM8K eval answer-extraction bug: get_answer_value() returns the last number in the response. Instruction-tuned models that don't stop at the #### <answer> marker (Mixtral/Mistral especially) run on and emit further Question:/Answer:/#### N pairs, so the last number is not the answer → artificially low score (≈0.31), below the 0.69 gate.
  • Supporting: extraction code verified (numbers[-1]); Mixtral score stable ≈0.31 across all nights and all 3 retries; well-behaved models unaffected; PR #29933 documents the identical root cause and the identical Mixtral ≈0.31 / Mistral ≈0.41 signature on the sister NVIDIA job nightly-test-text-accuracy-2-gpu-h100.
  • Disconfirming: today's exact Mixtral score was not captured (log truncation), so the causal chain for today rests on the persistent historical pattern rather than a fresh in-log measurement. A genuine AMD MoE-kernel accuracy issue cannot be 100% excluded from the truncated tail, but the cross-vendor (H100) reproduction argues strongly against a hardware/kernel cause.
  • How to verify: apply PR #29933's extractor change locally and re-run gsm8k on Mixtral-8x7B-v0.1; expect score to jump to ≈0.74. Or dump a few Mixtral raw responses and confirm text after the first ####.
• [LOW] Additional models in the truncated tail (Qwen2-57B-A14B, FP8 variants, Qwen3-30B) also below threshold.
  • Supporting: they run after the 4.2MB cap and are unverified.
  • Disconfirming: none of these appeared as failures in the visible portion of any run; the Mixtral bf16 failure alone is sufficient to fail the job.
  • How to verify: fetch the github_summary.md step-summary artifact (final table) once the log/artifact is retained, or re-run with reduced aiter log verbosity.

In-flight Fix Check

• In-flight fix: ✅ #29933 "fix(eval): extract GSM8K answer from first #### marker, not last number" — open, DRAFT since 2026-07-02. Directly targets get_answer_value(); its PR body documents the same Mixtral ≈0.31 signature. Action: un-draft, review, and merge — do NOT open a duplicate. (Searched repo:sgl-project/sglang is:pr is:open Mixtral+gsm8k / Mixtral+accuracy.)

Suggested Triage Steps

• Chase #29933 out of draft and through review — it is the single fix for this cluster across both AMD (nightly-test-amd-rocm720) and NVIDIA (nightly-test-text-accuracy-2-gpu-h100) nightlies.
• Before/while merging, A/B verify: run gsm8k on mistralai/Mixtral-8x7B-Instruct-v0.1 (TP=2) with the current extractor vs the PR extractor; confirm score rises from ≈0.31 to ≈0.74.
• Separately, add a Qwen/Qwen3-4B entry to MODEL_SCORE_THRESHOLDS (or drop it from DEFAULT_MODEL_NAME_FOR_NIGHTLY_EVAL_TP1) so it stops printing a misleading ❌ FAILED (threshold: N/A).
• To recover full failure signal, reduce aiter GEMM log spam ([aiter] ... using torch solution:0, printed per-GEMM) or upload the final github_summary.md as an artifact — the ~4.2MB log cap is currently hiding the second half of every accuracy run.

Status (factual)

• Persistence: Test file red in every completed scheduled run in the queryable window (Jul 14–24); Mixtral ≈0.31 signature confirmed on Jul 19/22/23. Upstream this dates to the 2026-04-07 gsm8k migration (#21931).
• Scope: nightly-accuracy-2-gpu-rocm720 job in nightly-test-amd-rocm720.yml. Same root cause reported on NVIDIA nightly-test-text-accuracy-2-gpu-h100 per #29933.
• Blocked work: gsm8k accuracy signal on AMD ROCm 7.2 is red, masking any genuine accuracy regressions on the models that share this run.
• In-flight fix: ✅ #29933 (open draft since 2026-07-02) — needs un-drafting + merge.

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-07-25 22:47 UTC)

[amd-bot]

nightly-test-amd-rocm720.yml — Per-Job Analysis (continued 2/4)

nightly-2-gpu-mi35x-glm51-mxfp4-rocm720 — failed step(s): N/A

Commit Info

• sglang: 2428f561
• aiter: N/A — the job log blob is BlobNotFound (runner died before uploading logs), so no [CI-AITER-CHECK] markers are recoverable.

Failed Tests

Test File	Test Function	Error	Log
N/A (non-test infra failure)	N/A	The self-hosted runner lost communication with the server — runner process died mid-step	Job

This is not a test-level failure. The runner (linux-mi35x-gpu-2) lost communication with GitHub during step 6 (Accuracy Test MI35x ROCm 7.2 (2-GPU GLM-5.1-MXFP4 GSM8K)) and never uploaded its job log (Azure blob returns BlobNotFound). The suite that was executing is nightly-amd-2-gpu-mi35x-glm51-mxfp4, whose only test is test/registered/amd/accuracy/mi35x/test_glm51_mxfp4_tp2_gsm8k_mi35x.py → TestGLM51MXFP4TP2GSM8KMI35x.test_gsm8k, but no test result was reached — the runner was killed before pass/fail could be recorded.

Failure Summary

The job failed because the self-hosted MI35x runner lost communication with the GitHub server while step 6 (GLM-5.1-MXFP4 TP=2 GSM8K accuracy test) was still in_progress. The step ran for ~27 min (17:58:22 → 18:25:23 UTC), far short of its 120-min timeout-minutes, so this was not a step timeout — the runner agent itself died (host starvation, GPU/driver hang, or network loss). No server or test logs survive.

Regression Status

Known recurring / Never-passed infrastructure failure — no passing run of this job exists in the queryable window. The 4 most recent completed runs I inspected carry the byte-identical runner lost communication annotation with BlobNotFound logs, and every completed comparable run failed this job.

Recent history of the nightly-2-gpu-mi35x-glm51-mxfp4-rocm720 job (most recent comparable completed scheduled runs: branch=main, event=schedule, status=completed). Because no log is retrievable in any run, the verdict is at the runner-communication-failure level for this job's single suite:

Date	Run	Job	Job Status	Failure Signature
Jul 23	30031529424	job	❌ Failed	runner lost communication (verified annotation, BlobNotFound)
Jul 22	29944173150	job	❌ Failed	runner lost communication (verified annotation)
Jul 21	29854963573	job	❌ Failed	runner lost communication (verified annotation)
Jul 20	29766667696	job	❌ Failed	(not re-verified)
Jul 19	29697452589	job	❌ Failed	runner lost communication (verified annotation)
Jul 18	29654521083	job	❌ Failed	(not re-verified)
Jul 17	29601455852	job	❌ Failed	(not re-verified)
Jul 16	29521474931	job	❌ Failed	(not re-verified)
Jul 15	29438224738	job	⚪ Cancelled	(excluded from verdict)
Jul 14	29355338821	job	❌ Failed	(not re-verified)
Jul 13	29273096587	job	❌ Failed	(not re-verified)
Jul 12	29202600186	job	❌ Failed	(not re-verified)

Lookback range: 2026-07-12 → 2026-07-23. No green run found for this job in the window; first/last passing date is unknown (none observed).

Failure Cluster

MI35x self-hosted runner lost communication during GLM-5.1-MXFP4 GSM8K accuracy step (log never uploaded, BlobNotFound)

Facts (verified observations, no interpretation)

• Check-run annotation (current job, annotation_level: failure): "The self-hosted runner lost communication with the server. Verify the machine is running and has a healthy network connection. Anything in your workflow that terminates the runner process, starves it for CPU/Memory, or blocks its network access can cause this error." (FACT)
• The job's log endpoint 303-redirects to an Azure blob that returns <Code>BlobNotFound</Code> — no job log exists. Same for yesterday's job 89288985793. (FACT)
• Job step state at completion: step 6 Accuracy Test … GLM-5.1-MXFP4 GSM8K = in_progress, step 12 Post Checkout code = pending; steps 1–5 succeeded. The job was killed mid-step. (FACT)
• Duration: started 2026-07-24T17:58:22Z, completed 18:25:23Z (~27 min); step timeout-minutes: 120. Not a step timeout. (FACT — job API)
• Runner: runs-on: linux-mi35x-gpu-2 (workflow line 270). Step command: run_suite.py --hw amd --suite nightly-amd-2-gpu-mi35x-glm51-mxfp4 --nightly --timeout-per-file 7200 --continue-on-error (line 298; --continue-on-error is present for schedule). (FACT)
• The 4 most-recent completed comparable runs (Jul 19–23) all carry the byte-identical runner lost communication annotation. (FACT)

Hypothesised Causes (with confidence, with disconfirming evidence)

• [LOW] The MI35x runner host is being starved/hung by the GLM-5.1-MXFP4 TP=2 GSM8K workload (server launch + GSM8K at 512 threads), killing the runner agent.
  • Supporting: death occurs consistently ~27–32 min into the accuracy step; --mem-fraction-static 0.85, --num-threads 512, and the test itself raises RLIMIT_NOFILE noting "GSM8K with high parallelism can exceed the default"; --continue-on-error means an ordinary test failure would NOT fail the job — so the job-level failure implies the runner process itself died, not an assertion.
  • Disconfirming: with BlobNotFound there is zero server/host log to confirm OOM vs GPU hang vs network; the exact terminating cause is unverifiable from CI alone. Cannot distinguish workload-starvation from a pre-existing unhealthy runner.
  • How to verify: retrieve runner-side _diag/dmesg/rocm-smi logs from linux-mi35x-gpu-2 around 18:25 UTC; re-run the suite interactively on that host watching host RAM and GPU state.
• [LOW] The MI35x runner hardware/driver is intermittently unhealthy (hung GPU), independent of this specific workload.
  • Supporting: an open PR (#31068) exists specifically to "detect and recover hung GPUs in MI35x pre-flight", indicating MI35x GPU hangs are a known operational problem.
  • Disconfirming: the failure is specific to this GLM-5.1-MXFP4 job at a consistent point in the accuracy step, which points more at the workload than at random hardware flakes; other MI35x jobs in the same nightly would need checking to confirm host-wide unhealth.
  • How to verify: check whether sibling MI35x jobs (nightly-2-gpu-mi35x-deepseek-r1-mxfp4-tp2-rocm720, etc.) on the same runner also lose communication in the same runs.

No hypothesis above LOW confidence — the missing runner-side log blocks code-level attribution. Because the runner dies (not an assertion) and --continue-on-error is set, this is an infrastructure/runner-health failure, not a suspected sglang code regression; no commit bisect is warranted until runner logs are recovered.

In-flight Fix Check

• In-flight fix: ✅ [#31068](https://github.com/sgl-project/sglang/pull/31068) (open since 2026-07-13) — "[AMD] ci: detect and recover hung GPUs in MI35x pre-flight" — directly targets MI35x runner-health; chase reviewers rather than opening a duplicate. (Searched: mi35x runner.)
• Related runner-routing candidates: #32003 "route scheduled jobs to MI300 with model cache" and #30431 "self-heal driver-level VRAM leaks + graceful GPU cleanup" would move/harden this job off the failing MI35x runner.
• In-flight fix (accuracy side): ❌ none found — no open PR addresses a GLM-5.1-MXFP4 GSM8K accuracy failure (search GLM-5.1-MXFP4, glm51 mxfp4 returned only feature/recipe PRs), consistent with this being an infra failure, not an accuracy regression.

Suggested Triage Steps

• Recover runner-side diagnostics from linux-mi35x-gpu-2 around 2026-07-24 18:25 UTC (_diag logs, dmesg, rocm-smi, host memory) — the GitHub-side log is permanently unavailable (BlobNotFound), so root-causing requires the runner host.
• Confirm whether sibling MI35x jobs on the same runner in runs 30115085856 / 30031529424 also lost communication — distinguishes host-wide unhealth from this workload.
• Evaluate whether #31068 (MI35x hung-GPU pre-flight recovery) or #32003 (route to MI300) would restore signal for this job; verify one merges cleanly and re-run nightly-2-gpu-mi35x-glm51-mxfp4-rocm720.
• If workload-starvation is suspected, run the suite interactively on the MI35x host and watch host RAM / GPU state through the accuracy step to see whether the server launch (--mem-fraction-static 0.85, --tp 2) or the 512-thread GSM8K eval triggers the death.

Status (factual, no priority assignment)

• Persistence: This job has failed every completed comparable scheduled run in the queryable window (2026-07-12 → 2026-07-24, ≥11 consecutive failures excluding one cancelled run); the 4 most-recent verified share the identical runner lost communication annotation. No green run observed.
• Scope: The single job nightly-2-gpu-mi35x-glm51-mxfp4-rocm720 on linux-mi35x-gpu-2 in nightly-test-amd-rocm720.yml. Blocks all nightly GSM8K accuracy signal for amd/GLM-5.1-MXFP4 (TP=2) on MI35x/ROCm 7.2.
• Signal quality: No server, test, or accuracy data is recoverable (BlobNotFound); classification is infrastructure/runner-health, not a confirmed sglang code regression.
• In-flight fix: ✅ #31068 (open since 2026-07-13) targets MI35x runner health; accuracy-side fix ❌ none (none needed unless a real accuracy failure is confirmed after the runner is stabilized).

nightly-8-gpu-mi35x-deepseek-v4-pro-mtp-rocm720 — failed step(s): Accuracy + MTP Accept Length Test MI35x ROCm 7.2 (8-GPU DeepSeek-V4-Pro FP8 MTP, triton)

Commit Info

• sglang: 2428f561
• aiter: 9127c94a — v0.1.17.dev110+g9127c94a1 (source: dev image detected; [CI-AITER-CHECK] Dev/patched version detected → skipping rebuild, Dockerfile default for gfx950 was the same SHA)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/test_deepseek_v4_pro_fp4_mtp.py	test_a_gsm8k	requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=11000) … Connection refused	Log
test/registered/amd/test_deepseek_v4_pro_fp4_mtp.py	test_b_bs_1_speed	Exception: retry() exceed maximum number of retries. (server never ready)	Log

The sister file test_deepseek_v4_pro_fp4_tbo_mtp.py in the same job passed (Test Summary: 1/2 passed).

Failure Summary

The DeepSeek-V4-Pro server (TP8, EAGLE MTP / DeepseekV4ForCausalLMNextN) never reached ready state, so both tests in test_deepseek_v4_pro_fp4_mtp.py failed with Connection refused on port 11000. During NextN weight loading the process is stuck in FP8 dequantization running on CPU (libtorch_cpu.so mul_kernel AVX2), which trips the 300 s soft scheduler watchdog repeatedly across all 8 TP ranks. The job ran out of budget at 5003 s (the MTP test alone consumed 2453 s before giving up).

Regression Status

Regressed for this test file (Jul 23), but on a marginal timing boundary — the identical slow-CPU-dequant + watchdog signature is present in passing runs too.

Recent history of test_deepseek_v4_pro_fp4_mtp.py in job nightly-8-gpu-mi35x-deepseek-v4-pro-mtp-rocm720 (most recent comparable completed scheduled runs: branch=main, event=schedule, completed only):

Date	Run	Job	Test File Status	Failed Function	Error
Jul 24	run	job	❌ Failed	test_a_gsm8k, test_b_bs_1_speed	Connection refused (server not ready)
Jul 23	run	job	❌ Failed	mtp file (exit code 1)	same dequant/watchdog signature
Jul 22	run	job	✅ Passed	— (tbo_mtp file failed instead)	—
Jul 21	run	job	✅ Passed	—	— (2/2, but still had 224 watchdog dumps)
Jul 20	run	—	⏭️ Cancelled	—	excluded
Jul 18	run	job	✅ Passed	—	—

First observed failure for this test file: Jul 23 2026. Last known passing: Jul 22 2026. Note: Jul 21 passed despite 224 soft-watchdog timeouts during the same CPU dequant — evidence this is a slow-load boundary, not a hard break.

Failure Cluster

Server launch timeout from slow CPU FP8 wo_a dequant during DeepSeek-V4 NextN/MTP weight loading

Facts (verified observations)

• [FACT] py-spy dump of sglang::scheduler_DP6_TP6 MainThread is active in libtorch_cpu.so → at::native::mul_kernel (AVX2) → _dequant_fp8 (deepseek_v4.py:3143) → _dequant_fp8_wo_a (deepseek_v4.py:3163) → load_weights (deepseek_v4.py:2822) → load_weights (deepseek_v4_nextn.py:277). The multiply weight_f32 * scale.float() at deepseek_v4.py:3143 executes on CPU tensors.
• [FACT] All 8 TP ranks log Execute dequant fp8 wo_a at 18:07:16; the first Scheduler watchdog timeout (watchdog_timeout=300, soft=True) fires at 18:10:19 and recurs (~224 times) through 20:40.
• [FACT] Engine main thread is blocked in _wait_for_scheduler_ready (engine.py:1381) the entire time; tests then get Connection refused at 20:04:34.
• [FACT] Fail. Time elapsed: 5003.77s; MTP test elapsed 2453 s and failed, test_deepseek_v4_pro_fp4_tbo_mtp.py elapsed 2551 s and passed.
• [FACT] The passing Jul 21 run (job 88716968944) shows the identical 64× Execute dequant fp8 wo_a + 224× Scheduler watchdog timeout yet finished 2/2.
• [FACT] The 32-commit regression window 0bdd4730af..d0b9689805 contains no commit touching the dequant / NextN / DeepSeek-V4 weight-load path (only an unrelated mamba fix #30986 matched the keyword grep).

Hypothesised Causes (with confidence)

• [MEDIUM] Slow CPU FP8 wo_a dequant makes MTP server launch time exceed the test's launch budget. _dequant_fp8 runs weight.float() and the scale multiply on CPU (deepseek_v4.py:3143), confirmed by the native py-spy stack. On a marginal run this pushes NextN weight loading past the launch timeout → Connection refused.
  • Supporting: native stack is unambiguously in libtorch_cpu.so mul_kernel inside _dequant_fp8; open PR #28729 exists specifically to "Offload fp8 wo_a dequant to GPU to speed up weight loading".
  • Disconfirming: the same CPU-dequant path and 224 watchdog dumps appear in the passing Jul 21 run, so the code path alone is not sufficient to fail — it's a timing boundary, and something else (runner load, HF download speed, GPU contention) determines pass/fail on a given day.
  • How to verify: re-run this job 3–5× to measure flake rate; time NextN weight loading; or apply #28729 on a branch and re-run test_deepseek_v4_pro_fp4_mtp.py on MI35x.
• [LOW] A commit in 0bdd4730af..d0b9689805 slowed startup enough to cross the budget.
  • Supporting: the test file first failed Jul 23 (head d0b9689805).
  • Disconfirming: none of the 32 commits touch the dequant/NextN/weight-load path; the boundary nature (pass Jul 21/22 with same signature) argues against a specific code regression.
  • How to verify: bisect the window with repeated launches, watching NextN load duration rather than pass/fail alone.

No hypothesis reaches HIGH; the pass/fail split on an unchanged code path indicates a marginal performance/timeout boundary rather than a clean regression.

In-flight Fix Check

• In-flight fix: ✅ #28729 "[DeepSeek-V4] Offload fp8 wo_a dequant to GPU to speed up weight loading" (open since 2026-06-19, non-draft, last updated 2026-06-23) — directly removes the slow CPU dequant that is the root of this launch timeout. Chase reviewers; do NOT open a duplicate. (Searched is:pr is:open dequant fp8 and deepseek v4 nextn.)

Suggested Triage Steps

• Verify #28729 still rebases cleanly, then apply it on a branch and re-run test_deepseek_v4_pro_fp4_mtp.py on an MI35x runner — if launch time drops well under the timeout, this closes the failure.
• Re-run the current job 3–5× to quantify flake rate and confirm the pass/fail boundary (expected, given the passing Jul 21 run had the identical dequant/watchdog signature).
• Instrument or read NextN load time: compare Execute dequant fp8 wo_a timestamp to server-ready across a passing vs failing run to see how much headroom exists against the launch timeout.
• If #28729 is not landing soon, consider raising the MTP server-launch timeout for this specific AMD test as an interim measure (maintainer decision).

Status

• Persistence: test_deepseek_v4_pro_fp4_mtp.py failing on the last 2 completed scheduled runs (Jul 23–24); passed Jul 21–22 with the same underlying slow-dequant signature.
• Scope: single job nightly-8-gpu-mi35x-deepseek-v4-pro-mtp-rocm720 (nightly-test-amd-rocm720.yml); sister file test_deepseek_v4_pro_fp4_tbo_mtp.py passed this run.
• Blocked work: DeepSeek-V4-Pro MTP (EAGLE NextN) FP4 nightly signal on MI35x ROCm 7.2.
• In-flight fix: ✅ #28729 (open since 2026-06-19) — offloads wo_a dequant to GPU.

nightly-8-gpu-mi35x-deepseek-v4-flash-rocm720 — failed step(s): Accuracy + Performance Test MI35x ROCm 7.2 (8-GPU DeepSeek-V4-Flash FP8 + FP4, triton)

The report is complete. The background task was just the historical log download, which I already used in my analysis.

The analysis is done. Summary of findings:

Two GSM8K accuracy-gate failures in the DeepSeek-V4-Flash suite on MI35x/ROCm 7.2:

• test_deepseek_v4_flash_fp8.py::test_a_gsm8k — 0.9098/0.9090 < 0.91
• test_deepseek_v4_flash_fp8_tbo.py::test_gsm8k_tbo — 0.9030 < 0.91

Root cause (MEDIUM confidence): too-tight 0.91 accuracy gate. The tests are only 2 days old (added Jul 22 in PR #30540) and the suite's accuracy band (0.903–0.926) straddles the gate — a different member fails each day. The failing values are discrete correct-counts (1191/1200 out of 1319; gate needs ≥1201), and identical counts recur across passing and failing days, indicating near-deterministic output with small launch-to-launch drift rather than a code regression. A small numeric regression in the 44-commit window can't be fully excluded (LOW), so I recommended repeated same-SHA reruns to distinguish variance from regression before touching the threshold. No in-flight threshold fix exists; aiter was unchanged across passing/failing runs.

nightly-accuracy-8-gpu-mi35x-rocm720 — failed step(s): Accuracy Test MI35x ROCm 7.2 (8-GPU GPT-OSS)

Commit Info

• sglang: 2428f561
• aiter: 9127c94a — v0.1.17.dev110+g9127c94a1 (source: dev image, preserved — [CI-AITER-CHECK] Dev/patched version detected … → skipping rebuild)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi35x/test_qwen3_moe_eval_mi35x.py	TestQwen3MoeEvalMI35x.test_qwen3_moe_accuracy	GPU Memory access fault … Reason: Unknown → Fatal Python error: Aborted → scheduler_0 crashed exit code -6 → AssertionError: Failed models: ['Qwen/Qwen3-30B-A3B']	Log

The other 3 accuracy tests in this job passed: test_gpt_oss_eval_mi35x.py (0.510/0.875), test_gpt_oss_w4a8_mxfp4_eval_mi35x.py (0.835), test_qwen3_coder_next_eval_mi35x.py (0.955/0.950). Test Summary: 3/4 passed.

Failure Summary

The nightly-accuracy-8-gpu-mi35x-rocm720 job ran 4 accuracy suites; 3 passed and test_qwen3_moe_eval_mi35x.py failed. Serving Qwen/Qwen3-30B-A3B (bf16, --tp 8, MI355X/gfx950, ROCm 7.2), all 8 GPU ranks hit a Memory access fault … Reason: Unknown roughly 168 s in — immediately after CUDA-graph capture completed and the first GSM8K prefill/decode requests were served. Every rank aborted (exit code −6), and the test harness raised AssertionError: Failed models: ['Qwen/Qwen3-30B-A3B'].

Regression Status

Known recurring failure (not a new regression at this commit).

Recent history of test_qwen3_moe_eval_mi35x.py (test_qwen3_moe_accuracy) in job nightly-accuracy-8-gpu-mi35x-rocm720 (most recent comparable completed scheduled runs: branch=main, event=schedule):

Date	Run	Test File Status	Failed Function	Error
Jul 24	30115085856	❌ Failed	test_qwen3_moe_accuracy	GPU memory access fault → exit −6
Jul 23	30031529424	❌ Failed	same	same fault
Jul 22	29944173150	❌ Failed	same	same fault
Jul 21	29854963573	❌ Failed	same	same fault
Jul 19	29697452589	❌ Failed	same	same fault
Jul 18	29654521083	❌ Failed	same	same fault
Jul 17	29601455852	❌ Failed	same	same fault

Failing every completed run in the queryable window — no passing run found for this test file. First observed (in this 7-run window) 2026-07-17; likely older. Jul 20 (29766667696) and Jul 16 (29521474931) were cancelled (excluded). Jul 14 (29355338821) failed before reaching this suite.

Failure Cluster

GPU memory access fault during Qwen3-30B-A3B decode on MI355X (gfx950, ROCm 7.2)

Facts (verified observations)

• Memory access fault by GPU node-{2,3,6,7,8} … on address 0x…009000. Reason: Unknown. at 18:22:01 across all ranks (job log, line 21151). Fault addresses consistently end in 009000 (page-aligned).
• Fault occurs after Capture target decode CUDA graph end (18:21:58) and after the first two Prefill batch requests (18:21:59, 18:22:00) — i.e. during actual inference, not warmup/capture.
• Subprocess scheduler_0 (pid=34544) crashed with exit code -6 (line 21454); the printed Python tracebacks are only the watchdog / inductor subproc_pool reader threads (line 21454/21719–21720) — the GPU fault is asynchronous, so no faulting-kernel stack was captured.
• Launch config: sglang serve --model-path Qwen/Qwen3-30B-A3B --max-running-requests 128 --mem-fraction-static 0.8 --trust-remote-code --tp 8 --device cuda.
• AssertionError: Failed models: ['Qwen/Qwen3-30B-A3B'] (line 24291); Ran 1 test … FAILED (failures=1); job exit code 255.
• Reproducibility: identical symptom (same model, same fault address suffix 009000, same exit −6) in 7 consecutive completed runs Jul 17–24.
• Same aiter dev build v0.1.17.dev110+g9127c94a1 was used by the 3 passing suites in the same job — including MoE model test_qwen3_coder_next which passed (accuracy 0.955/0.950).
• Benign noise in log (present in passing suites too, not the cause): -amdgpu-coerce-illegal-types=1 is not supported by hipcc and Using default MoE kernel config … E=128,N=96,device_name=AMD_Instinct_MI355X (missing tuned config → default kernel).

Hypothesised Causes (with confidence)

• [LOW] Out-of-bounds access in a Qwen3-30B-A3B MoE/attention kernel path on gfx950 during decode.
  • Supporting: fault addresses are page-aligned (…009000), consistent with a kernel reading/writing past a buffer boundary; triggers only once real decode traffic starts, after CUDA-graph capture; reproducible byte-for-byte across 7 runs.
  • Disconfirming: the async GPU fault produced no faulting-kernel stack trace, so the exact kernel is unverified; another MoE model (qwen3_coder_next) passed on the same GPU/aiter, so it is not a blanket MoE-on-gfx950 break.
  • How to verify: relaunch with AMD_SERIALIZE_KERNEL=3 (or HIP_LAUNCH_BLOCKING=1) and --disable-cuda-graph to get a synchronous, attributable fault location; enable a GPU core dump.
• [LOW] aiter dev-build MoE/paged-attention kernel (module_moe_asm / topksoftmax_4x128x8_bf16 / pa_ragged) on gfx950.
  • Supporting: only AMD MI355X is affected; aiter loads these kernels immediately before the fault; job runs a non-default dev aiter build.
  • Disconfirming: the identical aiter build served 3 passing suites in this same job (log-line proximity ≠ causation); failure predates the current aiter dev image, so it is not obviously an aiter-version change.
  • How to verify: A/B the same test with the Dockerfile-default aiter vs this dev build; bisect aiter around the MoE/pa kernels.

No hypothesis reaches MEDIUM: the fault is async with no captured kernel frame, and the failure is long-standing (predates this commit), so needs an instrumented repro to localize.

In-flight Fix Check

• In-flight fix: ❌ none found matching this test. Searched open PRs for Qwen3+MoE+MI355X, qwen3+memory+access+fault, mi35x+accuracy. Matches like #32039, #31793, #31794, #29009 all target Qwen3.5 MoE / MoRI, a different model than the failing Qwen/Qwen3-30B-A3B bf16 suite. No open PR addresses this specific accuracy failure.

Suggested Triage Steps

• Reproduce on an MI355X (gfx950, ROCm 7.2) container with Qwen/Qwen3-30B-A3B --tp 8, adding AMD_SERIALIZE_KERNEL=3 + --disable-cuda-graph + GPU core dump enabled, to capture the exact faulting kernel and stack.
• If the synchronous repro points at an aiter kernel, A/B against the Dockerfile-default aiter to confirm/exonerate the v0.1.17.dev110 dev build.
• Extend history before 2026-07-17 to find the first failing run (pass_sha..fail_sha); this test never passes in the current window, so the regression window is older than 7 days.
• If not quickly root-caused, consider quarantining test_qwen3_moe_eval_mi35x.py on MI355X so the accuracy job's other 3 suites keep producing signal.

Status

• Persistence: Failing every completed scheduled run for ≥7 days (since at least 2026-07-17); 7/7 consecutive completed runs.
• Scope: 1 test file (test_qwen3_moe_eval_mi35x.py / Qwen/Qwen3-30B-A3B) in job nightly-accuracy-8-gpu-mi35x-rocm720; the job's other 3 accuracy suites pass.
• Blocked work: Blocks Qwen3-30B-A3B bf16 accuracy signal on MI355X ROCm 7.2 nightly.
• In-flight fix: ❌ none found (open Qwen3.5-MoE PRs target a different model).

nightly-8-gpu-mi35x-deepseek-r1-mxfp4-tp4-rocm720 — failed step(s): Accuracy Test MI35x ROCm 7.2 (DeepSeek-R1-MXFP4 TP4 MTP)

Commit Info

• sglang: 2428f561
• aiter: 9127c94a (v0.1.17.dev110+g9127c94a1, source: dev image preserved — [CI-AITER-CHECK] Dev/patched version detected → skipping rebuild; AITER_COMMIT_OVERRIDE empty)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi35x/test_deepseek_r1_mxfp4_tp4_mtp_mi35x.py	test_gsm8k (TestDeepSeekR1MXFP4TP4MTPMI35x)	AssertionError: 0.8483699772554966 not greater than or equal to 0.944	Log (step 7)

Failure Summary

The MI35x DeepSeek-R1-MXFP4 TP=4 MTP (EAGLE speculative decoding) GSM8K accuracy test measured accuracy=0.848 (1319 questions, temperature=0), below its GSM8K_MTP_ACCURACY_THRESHOLD = 0.944 (test line 197). In the same job, the sibling non-MTP test (step 6, test_deepseek_r1_mxfp4_tp4_mi35x.py, identical model/TP/config but no speculative decoding) passed at accuracy=0.949. The failing job step is step 7 "Accuracy Test MI35x ROCm 7.2 (DeepSeek-R1-MXFP4 TP4 MTP)".

Regression Status

Never-passed (no green run found in queryable window). This test has failed in every comparable completed scheduled run since it first appears in the queryable history (2026-07-18). The MTP accuracy is consistently ~0.85 while the sibling non-MTP test is consistently ~0.95. Lookback range: 2026-07-18 → 2026-07-24 (07-20 run was cancelled, excluded).

Recent history of test_deepseek_r1_mxfp4_tp4_mtp_mi35x.py (test_gsm8k) in job nightly-8-gpu-mi35x-deepseek-r1-mxfp4-tp4-rocm720 (most recent comparable completed scheduled runs: branch=main, event=schedule):

Date	Run	Test File Status	MTP acc	Non-MTP sibling acc	Error
Jul 24	30115085856	❌ Failed	0.848	0.949 ✅	0.848 < 0.944
Jul 23	30031529424	❌ Failed	0.864	0.948 ✅	0.864 < 0.944
Jul 22	29944173150	❌ Failed	0.849	0.954 ✅	0.849 < 0.944
Jul 21	29854963573	❌ Failed	0.858	0.945 ✅	0.858 < 0.944
Jul 19	29697452589	❌ Failed	0.851	0.952 ✅	0.851 < 0.944
Jul 18	29601455852	❌ Failed	0.854	0.945 ✅	0.854 < 0.944

First observed failure: 2026-07-18 (earliest queryable run containing the test). No passing run for this test file found in the window.

Failure Cluster

MTP/EAGLE GSM8K accuracy ~10pts below non-speculative baseline on DeepSeek-R1-MXFP4 (MI35x/ROCm7.2)

Facts (verified observations, no interpretation)

• accuracy=0.848 invalid=0.000 latency=162.3s for the MTP test; AssertionError: 0.8483699772554966 not greater than or equal to 0.944 (verbatim, log line 26518–26522).
• Same job, step 6 non-MTP sibling: accuracy=0.949 invalid=0.000 latency=146.0s, result OK (log line 13793–13796).
• Both tests use the same model (amd/DeepSeek-R1-MXFP4-Preview), TP=4, 1319 GSM8K questions, temperature=0 greedy (benchmark line 101). The only functional difference is the MTP test adds EAGLE spec decoding (--speculative-algorithm EAGLE --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4, lines 154-161).
• Spec-decode telemetry in the failing run: accept len: ~2.75, accept rate: ~0.58, cuda graph: True (decode-batch log lines).
• Reproducibility: 6/6 comparable completed scheduled runs failed at MTP acc 0.848–0.864; the non-MTP sibling passed at 0.945–0.954 in all 6.
• aiter is a preserved dev build, unchanged across these runs (not a scout run; AITER_COMMIT_OVERRIDE empty).

Hypothesised Causes (with confidence, with disconfirming evidence)

• [MEDIUM] Accuracy defect in the EAGLE speculative-decode verify path for DeepSeek-R1-MXFP4 on MI35x/ROCm. At temperature=0, correct EAGLE greedy verification is output-lossless vs non-spec greedy, yet MTP accuracy is ~10pts lower than the identical-config non-MTP sibling in every run.
  • Supporting: sibling non-MTP test on same model/config passes at ~0.95 every run; the only delta is spec decoding; gap is stable (~0.85 vs ~0.95) across 6 runs → deterministic, not flaky variance.
  • Disconfirming: no code-level evidence isolating a specific commit/kernel; could instead be a low-quality/mismatched EAGLE draft checkpoint or a deliberately optimistic threshold (see below), neither of which is a code regression.
  • How to verify: on an MI35x/ROCm7.2 box, run the MTP launch recipe and diff greedy token streams between spec-on and spec-off for a handful of GSM8K prompts; any divergence at temperature=0 confirms a verify bug.
• [LOW] Threshold 0.944 is miscalibrated — copied from the non-MTP baseline while the MTP path legitimately runs lower on this config.
  • Supporting: threshold equals the non-MTP passing band (~0.945–0.954); test has literally never passed since introduction.
  • Disconfirming: at temperature=0 a correct EAGLE path should ~match non-spec accuracy, so a "legitimate" 0.85 is itself a symptom of a real defect rather than expected behavior — lowering the threshold would mask that.
  • How to verify: check PR #30540 / the test author's intent for the 0.944 value; confirm whether MTP was ever green in pre-merge CI.

In-flight Fix Check

• In-flight fix: ⚠️ [#31214](https://github.com/sgl-project/sglang/pull/31214) "[ROCm] Fix EAGLE spec-decode verify silently sampling greedy on HIP" (open since 2026-07-14) — related area (EAGLE verify on HIP) but likely does NOT fix this: #31214 addresses temperature>0 silently decoding greedy, whereas this test runs temperature=0 where greedy is already the intended path. Treat as adjacent, not a fix.
• In-flight fix: ❌ none found that targets a temperature=0 MTP accuracy gap for DeepSeek-R1-MXFP4. (Searched open PRs for mxfp4 mtp, eagle mxfp4, DeepSeek-R1-MXFP4; other hits — #30407, #30105 — concern GPU faults, not accuracy.)

Suggested Triage Steps

• Diff greedy token output (spec-on vs spec-off) at temperature=0 on MI35x for a few GSM8K prompts; any mismatch localizes an EAGLE verify bug on the MXFP4/aiter path.
• If confirmed, inspect the HIP EAGLE verify/reject-sampling path (srt/speculative/eagle_utils.py, srt/speculative/reject_sampling.py) referenced by #31214 for correctness at temperature=0, not just temperature>0.
• Confirm the EAGLE draft checkpoint used for amd/DeepSeek-R1-MXFP4-Preview is the intended/matched one (a mismatched draft head would depress acceptance/accuracy without a code bug).
• Verify with the test author (PR #30540) whether 0.944 was ever met pre-merge; if the MTP path is expected to run lower, recalibrate the threshold — but only after ruling out a verify defect.

Status (factual, no priority assignment)

• Persistence: Never-passed since first queryable run 2026-07-18; failed in all 6 comparable completed scheduled runs (07-18 → 07-24).
• Scope: 1 job / 1 test file (nightly-8-gpu-mi35x-deepseek-r1-mxfp4-tp4-rocm720, MTP step only). Non-MTP sibling in the same job is green.
• Blocked work: Blocks the MI35x DeepSeek-R1-MXFP4 MTP/EAGLE accuracy signal on ROCm 7.2 nightly.
• In-flight fix: ⚠️ #31214 is adjacent (EAGLE verify on HIP) but targets temperature>0; no open PR clearly addresses this temperature=0 MTP accuracy gap.

nightly-8-gpu-mi35x-qwen3-235b-mxfp4-rocm720 — failed step(s): Accuracy Test + Performance Test MI35x ROCm 7.2 (8-GPU Qwen3-235B-MXFP4)

Commit Info

• sglang: 2428f561
• aiter: 9127c94a — v0.1.17.dev110+g9127c94a1 (source: dev/patched image preserved; Dockerfile default was AITER_COMMIT=9127c94a..., rebuild skipped per [CI-AITER-CHECK] markers)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/test_qwen3_instruct_mxfp4.py	TestQwen3Instruct2507MXFP4.setUpClass	Server process exited (code 1); Rank 0 scheduler died during init (exit code -6, SIGABRT)	Log

Failure Summary

The single registered test test_qwen3_instruct_mxfp4.py (amd/Qwen3-235B-A22B-Instruct-2507-mxfp4, --tp 4 --ep 2, --attention-backend aiter) failed in setUpClass: popen_launch_server never came up because rank 0's scheduler process aborted (SIGABRT, exit -6) during engine initialization. Weights loaded successfully (26/26 shards); the abort occurred later during decode CUDA-graph capture. Ran 0 tests in 470.475s — the server never became ready, so no assertion-level test ran.

Regression Status

Never-passed (no green run found in queryable window). Lookback: the ~20 most recent completed event=schedule runs on main (2026-07-05 → 2026-07-24). This job (nightly-8-gpu-mi35x-qwen3-235b-mxfp4-rocm720) is failure in every completed run in the window; no passing run for this test file exists. The identical hipErrorCapturedEvent graph-capture crash is verified in the four most recent test-reaching runs; the 2026-07-05 run was blocked earlier by a VRAM pre-check (test never ran).

Recent history of test_qwen3_instruct_mxfp4.py (TestQwen3Instruct2507MXFP4.setUpClass) in job nightly-8-gpu-mi35x-qwen3-235b-mxfp4-rocm720 (completed event=schedule runs, main):

Date	Run	Job	Test File Status	Failed Function	Error
Jul 24	run	job	❌ Failed	setUpClass	hipErrorCapturedEvent → SIGABRT (-6)
Jul 23	run	job	❌ Failed (verified)	setUpClass	hipErrorCapturedEvent → SIGABRT
Jul 22	run	job	❌ Failed (verified)	setUpClass	hipErrorCapturedEvent → SIGABRT
Jul 21	run	job	❌ Failed (verified)	setUpClass	hipErrorCapturedEvent → SIGABRT
Jul 19	run	job	❌ Failed (job-level)	—	not test-verified
Jul 18	run	job	❌ Failed (job-level)	—	not test-verified
Jul 17	run	job	❌ Failed (job-level)	—	not test-verified
Jul 14	run	job	❌ Failed (verified)	setUpClass	Fatal Python error: Aborted → exit 134 (SIGABRT), during graph capture
Jul 05	run	job	⚠️ Blocked	—	VRAM pre-check failed; test never ran (infra)

(Job column is for human reference. Verdict is at test-file level. Runs on Jul 20/16/15/13/12–06 were cancelled or not test-verified and are omitted/annotated. No passing run found in the window → Never-passed.)

Failure Cluster

HIP hipErrorCapturedEvent / RCCL watchdog abort during decode CUDA-graph capture

Facts (verified observations)

• Verbatim error (rank 0 & rank 1): terminate called after throwing an instance of 'c10::DistBackendError' … Process group watchdog thread terminated with exception: HIP error: operation not permitted on an event last recorded in a capturing stream → hipErrorCapturedEvent → Fatal Python error: Aborted.
• Launcher-side: RuntimeError: Rank 0 scheduler died during initialization (exit code: -6); Server process exited with code 1; Ran 0 tests in 470.475s / FAILED (errors=1).
• C++ abort stack (both ranks): c10d::ProcessGroupNCCL::Watchdog::runLoop() → WorkNCCL::isCompleted() → WorkNCCL::finishedGPUExecutionInternal() → c10_hip_check_implementation (ProcessGroupNCCL.cpp:2057/2063, HIPException.cpp:45). The RCCL watchdog polled a work event that was recorded inside a HIP-graph capturing stream.
• Python capture stack (rank 0): scheduler.py:859 init_model_worker → init_all_cuda_graphs → tp_worker.py:396 → model_runner.py:904 init_cuda_graphs → cuda_graph_setup.py:106/333 capture_decode_graph → decode_cuda_graph_runner.py:857 capture → model forward qwen3_moe.py:994 → qwen2_moe.py:891 → vocab_parallel_embedding.py:574 forward → tensor_model_parallel_all_reduce → parallel_state.py:932 _all_reduce_out_place → aiter custom_all_reduce.py:589 all_reduce. Verified vocab_parallel_embedding.py:574 is the tensor_model_parallel_all_reduce(output_parallel) call at commit 2428f561.
• Config facts: tp_size=4, ep_size=2, attention_backend='aiter', full decode CUDA graph enabled, mem_fraction_static=0.7735; runner GPU_ARCH=mi35x (gfx950), ROCm 7.2, nccl==2.27.7, image rocm/sgl-dev:v0.5.15.post1-rocm720-mi35x-20260724.
• During capture the AR path logged [AR] All-reduce call path: NCCL (custom AR disabled) — i.e. the embedding all-reduce ran on RCCL, whose watchdog is the crashing thread.
• Reproducibility: same hipErrorCapturedEvent signature (8 occurrences/run across ranks) verified in 4 test-reaching runs (Jul 21–24) + a same-test SIGABRT on Jul 14. No green run in window.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [MEDIUM] The torch/RCCL ProcessGroupNCCL watchdog thread polls WorkNCCL::isCompleted() on an event recorded inside the HIP-graph capturing stream during decode-graph capture; on ROCm this returns hipErrorCapturedEvent, which is fatal (SIGABRT). The embedding-layer tensor_model_parallel_all_reduce enqueues an RCCL work item whose event is queried mid-capture.
  • Supporting: both crashing threads are ProcessGroupNCCL::Watchdog::runLoop → isCompleted; error string is exactly operation not permitted on an event last recorded in a capturing stream; capture Python stack goes through the embedding all-reduce.
  • Disconfirming: cannot confirm which side "owns" the fix (RCCL/torch runtime vs sglang capture guard) from the log alone; no A/B run with the watchdog disabled available.
  • How to verify: re-run with TORCH_NCCL_ENABLE_MONITORING=0 / RCCL watchdog disabled during capture, or set a longer/nonexistent watchdog poll during init_cuda_graphs; alternatively force the embedding AR onto the custom-AR path during capture and re-run the test in the rocm720-mi35x image.
• [LOW] AMD/gfx950 + ROCm 7.2 + nccl==2.27.7 environment-specific interaction (custom AR disabled → RCCL fallback) that this large MoE + full decode graph config triggers.
  • Supporting: AMD-only nightly; custom AR disabled on the crashing AR; failure is 100% reproducible on this runner class.
  • Disconfirming: no NVIDIA comparison run in scope; cannot isolate the exact ROCm/RCCL component.
  • How to verify: try --disable-cuda-graph (or --disable-decode-cuda-graph) on this model — if it comes up, capture is confirmed as the trigger; compare against a gfx950 image with a newer RCCL (e.g. the ROCm 7.2.4 upgrade in #30984).
• No commit-level regression attribution is possible: the test has never passed in the queryable window, so there is no pass_sha..fail_sha to bisect.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for hipErrorCapturedEvent (0 hits), capturing+stream+watchdog (3 unrelated: #30967, #17356, #13023), ProcessGroupNCCL+graph+capture (#13886, unrelated LMCache), and qwen3+235b+mxfp4 (no matching fix; #30984 is a ROCm 7.2.4 image upgrade that may change the RCCL version but does not target this crash). Needs a new investigation/PR.

Suggested Triage Steps

• Confirm the trigger: re-run test_qwen3_instruct_mxfp4.py with --disable-decode-cuda-graph (or --disable-cuda-graph) in image rocm/sgl-dev:v0.5.15.post1-rocm720-mi35x-*. If the server comes up, the RCCL-watchdog-during-capture path is confirmed.
• Try disabling the RCCL watchdog / event monitoring during init_cuda_graphs (e.g. TORCH_NCCL_ENABLE_MONITORING=0) and re-run; a clean startup points the fix at guarding watchdog event polls during HIP-graph capture.
• Test whether routing the vocab_parallel_embedding all-reduce through the custom AR (instead of the RCCL fallback that logged custom AR disabled) during capture avoids the captured-event poll.
• Evaluate whether the ROCm 7.2.4 / newer-RCCL image in #30984 changes the behavior; if so, the crash may be an RCCL-runtime issue fixed upstream.

Status (factual)

• Persistence: Never-passed in the queryable window; the hipErrorCapturedEvent graph-capture crash is verified every test-reaching run from 2026-07-14 through 2026-07-24 (Jul 21–24 identical; Jul 14 same-test SIGABRT).
• Scope: This job/test only in this workflow (nightly-8-gpu-mi35x-qwen3-235b-mxfp4-rocm720, gfx950/MI35x, ROCm 7.2). Blocks all CI signal for the Qwen3-235B MXFP4 model on AMD ROCm 7.2.
• In-flight fix: ❌ none found — needs new PR or investigation.

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-07-25 22:47 UTC)

[amd-bot]

pr-test-amd.yml — 20 failure(s)

Run started (UTC): 2026-07-24 01:57 UTC
Last scanned (UTC): 2026-07-25 13:12 UTC
Commits: sglang a678a42

Summary

Counts: 20 failures · 8 clusters · 15 carrying over (≥3 days) · 5 new today

#	Cluster	Job	Test File	Test Function	Status	Hypothesis (confidence)
1	Runner lost comms	multimodal-gen-1gpu (…,1)	N/A (runner death)	N/A	Never-passed ≥8d	[HIGH] MI325 host/runner-agent death; no logs
2	Runner lost comms	multimodal-gen-1gpu (…,1)	N/A (runner death)	N/A	Never-passed ≥8d	[HIGH] runner lost communication
3	Runner lost comms	multimodal-gen-2gpu (…,0)	N/A (runner death)	N/A	Red every run ≥8d	[HIGH] runner lost mid 2-GPU suite
4	Runner lost comms	multimodal-gen-1gpu (…,0)	N/A (runner death)	N/A	Never-passed ≥8d	[HIGH] runner lost communication
5	Runner lost comms	multimodal-gen-2gpu (…,0)	N/A (runner death)	N/A	Recurring	[HIGH] runner lost communication
6	Runner lost comms	multimodal-gen-1gpu (…,1)	N/A (runner death)	N/A	Never-passed ≥10d	[HIGH] runner lost communication
7	update_weights transformer 400	multimodal-gen-1gpu (…,3)	test_update_weights_from_disk.py	test_update_weights_from_disk_default[Qwen-Image]	Never-passed	[HIGH] diffusers ckpt shape mismatch linear.py:410
8	update_weights transformer 400	multimodal-gen-1gpu (…,3)	test_update_weights_from_disk.py	[Qwen-Image] (4 failed/4 err)	Persistent	[HIGH] transformer reload abort → 400
9	update_weights transformer 400	multimodal-gen-1gpu (…,3)	test_update_weights_from_disk.py	[Qwen-Image] (4 failed/4 err)	Never-passed	[HIGH] shape mismatch → HTTP 400
10	PD+PP disagg KV-transfer	stage-b-8gpu-mi35x-disagg	test_disaggregation_pp.py	test_gsm8k	Recurring/flaky	[MEDIUM] mooncake/RDMA KV timeout on MI35x
11	PD+PP disagg KV-transfer	stage-b-8gpu-mi35x-disagg	test_disaggregation_pp.py	TestDisaggregationDecodePPAccuracy.test_gsm8k	~10/16 flaky	[MEDIUM] KV-transfer → decode crash (exit 255)
12	PD+PP disagg KV-transfer	stage-b-8gpu-mi35x-disagg	test_disaggregation_pp.py	test_gsm8k (analysis incomplete)	Recurring	[MEDIUM] KVTransferError → Internal Server Error
13	Diffusion 1-GPU part-0 hang	multimodal-gen-1gpu (…,0)	N/A (hang→kill ~80m)	N/A	Red ~16d (since 07-08)	[LOW] part-0 hang; infra/host
14	Diffusion 1-GPU part-0 hang	multimodal-gen-1gpu (…,0)	N/A (90-min timeout)	N/A	Never-passed in window	[LOW] step timeout / cold-cache
15	GPU Hang HW exc → exit 134	stage-b-1gpu-small (…,8)	N/A (build, node-2)	N/A	New today	[LOW] GPU Hang during sgl-kernel build (SIGABRT)
16	GPU Hang HW exc → exit 134	stage-b-2gpu-large (…,1)	test_unified_radix_cache_kl_full.py	test_multiturn_decode_cache_hit_branching	New today	[LOW] GPU Hang node-3 → watchdog SIGABRT
17	GH Actions internal error	check-changes	N/A (gate, 0 steps)	N/A	New today (1 run)	[HIGH] GitHub platform internal error
18	GH Actions internal error	call-gate / pr-gate	N/A (gate, 0 steps)	N/A	New today (1 run)	[FACT] cascaded from check-changes
19	FLUX.2 FP8 HIPBLAS on gfx942	multimodal-gen-2gpu (…,0)	test_server_2_gpu.py	test_diffusion_generation[flux2_modelopt_fp8_tp2_t2i]	Red all 16 runs	[HIGH] modelopt-FP8 QKV GEMM unsupported flux_2.py:77
20	Kimi-K2.5 TP barrier timeout	stage-c-8gpu-mi35x (…,1)	test_kimi_k25_mxfp4_bcg_mi35x.py	test_bcg_gsm8k	New regression (07-24)	[MEDIUM] TP3 stalled >480s monitored_barrier → exit -9

Cluster: Runner lost communication (6 jobs — largest)

• Affected: rows 1-6 — all multimodal-gen-test-*-gpu-amd shards across runs 30093299057, 30056142455, 30136388246.
• Shared facts: identical The self-hosted runner lost communication with the server annotation; no log/artifact uploaded (BlobNotFound, 0 artifacts); job killed mid diffusion server step after 80–130 min. [FACT]
• Top hypothesis: [HIGH] MI325 self-hosted runner/host death (resource starvation / GPU reset / agent crash). Disconfirming: no code-level trace survives, so a slow-model-download hang that trips the runner cannot be fully separated from a pure host fault.
• In-flight fix: ❌ none found (searched runner lost communication, diffusion runner AMD).
• Suggested triage: pull MI325 host telemetry (dmesg/OOM-killer, GPU/XGMI reset) for the affected runners in the failure windows; enable early log flushing so a partial log survives a runner drop.

Cluster: update_weights_from_disk transformer 400 (3 jobs)

• Affected: rows 7-9 — multimodal-gen-test-1-gpu-amd (…,3) on all three runs, test_update_weights_from_disk.py (4 failed, 2 passed, 4 errors).
• Shared facts: [Qwen-Image] cases assert 400 == 200 — transformer weight update aborts/rolls back (shape mismatch at linear.py:410); [FLUX.2-klein-base-4B] cases ERROR at fixture setup (no perturbed transformer/vae weights dir).
• Top hypothesis: [HIGH] diffusers checkpoint layout not reconciled with sglang params on reload → deterministic ROCm defect. Disconfirming: (none — reproduces every run).
• In-flight fix: ✅ #31925 (skip PR, awaiting merge); tracker #31924. Do NOT duplicate.
• Suggested triage: chase reviewers on #31925; confirm it covers both the Qwen-Image 400 and the FLUX fixture-setup error.

Cluster: PD+PP disaggregation KV-transfer (3 jobs)

• Affected: rows 10-12 — stage-b-test-large-8-gpu-mi35x-disaggregation-amd, test_disaggregation_pp.py::…test_gsm8k. Row 12's per-job analysis did not complete (manual triage).
• Shared facts: GSM8K decodes then decode server crashes → 500 Internal Server Error / KeyError:'answer', retry hits ConnectionRefusedError on port 11200; mooncake KV receiver stuck in Transferring; watchdog SIGQUIT (exit 255).
• Top hypothesis: [MEDIUM] KV-cache transfer failure over MI35x RDMA NIC → decode PP event-loop hang. Disconfirming: intermittent (~10/16), passes on unrelated SHAs → not a fresh code regression.
• In-flight fix: ⚠️ several candidates — #29127 (isolate faulty rdma5 NIC), #31800 (time out stuck KV receivers), #31589, #31869 — none confirmed to fix this flake.
• Suggested triage: complete row 12's analysis; verify whether #31800 merges cleanly and re-run the disagg shard; check rdma5 NIC health per #29127.

Cluster: Diffusion 1-GPU partition-0 hang/timeout (2 jobs)

• Affected: rows 13-14 — multimodal-gen-test-1-gpu-amd (…,0). Last green 2026-07-08; evolved from clean pytest fail → hang/kill.
• Top hypothesis: [LOW] partition-0 diffusion step hang (cold model download/inference exceeding 90-min limit). Disconfirming: no surviving log → could overlap the runner-lost cluster.
• In-flight fix: ❌ none found.
• Suggested triage: instrument partition-0 for per-model download/inference timing; confirm whether it's the same host fault as the runner-lost cluster or a genuine step timeout.

Cluster: GPU Hang HW exception → exit 134 (2 jobs, new today)

• Affected: row 15 (stage-b-1gpu-small (…,8), hang during sgl-kernel build, node-2) and row 16 (stage-b-2gpu-large (…,1), test_unified_radix_cache_kl_full.py, node-3).
• Shared facts: HW Exception by GPU node-N reason: GPU Hang → exit code 134 (SIGABRT); both single-occurrence, prior runs green.
• Top hypothesis: [LOW] transient GPU/hardware hang (build-time and test-time). Disconfirming: two different nodes/contexts → likely independent hardware events, not a code cause.
• In-flight fix: ❌ none (hardware — re-run candidate).
• Suggested triage: re-run both shards; if the hang recurs on the same runners, quarantine those nodes.

Cluster: GitHub Actions internal error (2 jobs, new today)

• Affected: rows 17-18 — check-changes + pr-gate in run 30157771919, 0 steps, no runner, identical 44 s window. Cascaded to skip all 22 downstream AMD test jobs → this run produced no AMD test signal.
• Top hypothesis: [FACT] transient GitHub platform internal error before any step. Disconfirming: (none).
• In-flight fix: ❌ none needed.
• Suggested triage: "Re-run failed jobs" on 30157771919; if it reproduces, check githubstatus.com.

Cluster: FLUX.2 modelopt-FP8 HIPBLAS on gfx942 (1 job)

• Affected: row 19 — flux2_modelopt_fp8_tp2_t2i crashes at first denoising step, HIPBLAS_STATUS_NOT_SUPPORTED from QKV GEMM flux_2.py:77. (Same job's joy_echo is a separate HF-download flake.)
• Top hypothesis: [HIGH] gfx942 lacks the FP8 GEMM path used by modelopt-FP8 FLUX.2.
• In-flight fix: ✅ #28889 (native e4m3fnuz FP8 GEMM for FLUX.2 on gfx942) — ping reviewers.
• Suggested triage: chase #28889; this is the recurring test-level cause behind the 2-GPU shard on days the runner survives.

Cluster: Kimi-K2.5-MXFP4 TP barrier timeout (1 job, new regression)

• Affected: row 20 — test_bcg_gsm8k; TP3 never emits Load weight end, TP0 = 1303 s vs TP1/TP2 ~400 s → exceeds fixed 480 s monitored_barrier → server exit -9.
• Top hypothesis: [MEDIUM] load-time stall of one TP rank on the 133 GB/rank checkpoint (I/O contention on shared /sgl-data/hf-cache). Disconfirming: no OOM line for TP3; 3 prior runs green; none of the 24 commits in 433429b16a..3d91a569ce touch the weight-load path.
• In-flight fix: ❌ none for the stall; dormant timeout-config PRs #15637, #9305 only mask it.
• Suggested triage: re-run 2-3×; if it passes, treat as load-time flake/quarantine; capture per-rank load timings + dmesg to distinguish slow-I/O from OOM-kill.

Suggested triage order

Rank	Triage step	Targets	Why this first
1	Re-run gate jobs on run 30157771919	rows 17-18 (cluster H)	Zero AMD signal this run; cheapest recovery, unblocks 22 skipped jobs
2	Chase merge of skip PR #31925	rows 7-9 (cluster D)	Root-caused, in-flight fix ready; clears 3 never-passing shards
3	Pull MI325 host telemetry for runner-lost windows	rows 1-6 (cluster A)	Largest cluster (6 jobs); no code fix — needs host-level evidence
4	Verify #31800/#29127 + complete row 12 analysis	rows 10-12 (cluster E)	In-flight candidates exist; row 12 needs manual triage
5	Re-run GPU-Hang + Kimi shards 2-3×	rows 15-16, 20	New single-occurrence infra/load flakes; re-run disambiguates before any bisect

---

Job	Failed Steps	Started
multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 1)	N/A	2026-07-24T15:18
multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 0)	N/A	2026-07-24T15:26
multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 0)	N/A	2026-07-24T15:57
stage-b-test-large-8-gpu-mi35x-disaggregation-amd (linux-mi35x-gpu-8.fabric)	Run test_disaggregation — ⚠️ analysis failed	2026-07-24T12:30
multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 3)	Run diffusion server tests (1-GPU)	2026-07-24T15:31
stage-b-test-1-gpu-small-amd (linux-mi325-1gpu-sglang, 8)	Install dependencies	2026-07-24T16:16
stage-b-test-2-gpu-large-amd (linux-mi325-2gpu-sglang, 1)	Run test	2026-07-24T16:05
stage-c-test-large-8-gpu-amd-mi35x (linux-mi35x-gpu-8, 1)	Run test	2026-07-24T12:31
multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 0)	Run diffusion server tests (2-GPU)	2026-07-24T02:28
multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 1)	N/A	2026-07-24T02:56
multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 3)	Run diffusion server tests (1-GPU)	2026-07-24T02:58
stage-b-test-large-8-gpu-mi35x-disaggregation-amd (linux-mi35x-gpu-8.fabric)	Run test_disaggregation	2026-07-24T01:57
multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 0)	N/A	2026-07-24T02:59
stage-b-test-large-8-gpu-mi35x-disaggregation-amd (linux-mi35x-gpu-8.fabric)	Run test_disaggregation	2026-07-25T00:29
multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 3)	Run diffusion server tests (1-GPU)	2026-07-25T00:56
multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 0)	N/A	2026-07-25T00:47
multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 0)	N/A	2026-07-25T00:38
multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 1)	N/A	2026-07-25T01:14
call-pr-test-amd-extra / call-gate / pr-gate	N/A	2026-07-25T12:22
check-changes	N/A	2026-07-25T12:22

Per-Job Analysis

multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 1) — failed step(s): N/A

Commit Info

• sglang: 3d91a569
• aiter: 9127c94a (v0.1.17.dev110+g9127c94a1, source: dev image preserved — [CI-AITER-CHECK] from sibling shard 3 of the same run; AITER_COMMIT_OVERRIDE empty)

Failed Tests

Test File	Test Function	Error	Log
N/A — runner crash, no test-level log	N/A	The self-hosted runner lost communication with the server (runner agent died ~84 min into the 90-min "Run diffusion server tests (1-GPU)" step)	Job

This is not a test-assertion failure. The job is partition-id 1 (partition 2/4) of the multimodal_gen 1-GPU suite, which runs the parametrized diffusion-server generation cases (T2I/I2V/LoRA). The per-job log blob returns BlobNotFound and no diffusion-failures artifact was uploaded, because the runner process was killed before log/artifact upload. No specific test file/function can be pinned from this run.

Failure Summary

The self-hosted MI325 runner executing this shard lost communication with GitHub during step 9 ("Run diffusion server tests (1-GPU)"), ~84 minutes into a 90-minute-timeout step. The job's own log and failure artifacts were never uploaded (BlobNotFound, total_count: 0 artifacts), so the failure is visible only via the check-run annotation. This is an infrastructure/runner-death failure, not a captured test failure.

Regression Status

Known recurring failure (infrastructure) — never passed in the queryable window. This exact shard has failed in all 15 most recent comparable scheduled runs, and every sampled run carries the identical "runner lost communication" annotation.

Recent history of multimodal-gen-test-1-gpu-amd (…, 1) — partition 2/4, job-level (no test-level log exists for a runner-death; annotation is the only signal). Most recent 15 comparable completed scheduled runs (branch=main, event=schedule, completed only):

Date (UTC)	Run	Job Status	Failure Annotation
Jul 24 12:29	30093299057	❌ Failed	runner lost communication
Jul 24 00:25	30056142455	❌ Failed	runner lost communication
Jul 23 12:32	30007380837	❌ Failed	runner lost communication
Jul 23 00:27	29969145884	❌ Failed	runner lost communication
Jul 22 12:32	29920027108	❌ Failed	runner lost communication
Jul 22 00:26	29880220845	❌ Failed	runner lost communication
Jul 21 12:32	29830494225	❌ Failed	runner lost communication
Jul 21 00:27	29790347494	❌ Failed	runner lost communication
Jul 20 12:42	29743214297	❌ Failed	runner lost communication
Jul 17–20 (6 more)	29709549202 … 29580198489	❌ Failed ×6	not individually re-sampled; all job-level ❌

First observed failure in window: Jul 17 (earliest queried). No passing run found in the 15-run window → effectively Never-passed on this shard within the lookback. Because there is no green run and no test-level log, no commit-range bisection is meaningful — the signal is a runner crash, not a code assertion.

Failure Cluster

Self-hosted MI325 runner lost communication during long 1-GPU diffusion generation (runner-agent death / suspected resource starvation). Same-run siblings today: 1-GPU part 0 89481337911 and 2-GPU part 0 89481337959 died with the identical annotation.

Facts (verified observations, no interpretation)

• Check-run annotation for job 89481337945 (verbatim): "The self-hosted runner lost communication with the server. Verify the machine is running and has a healthy network connection. Anything in your workflow that terminates the runner process, starves it for CPU/Memory, or blocks its network access can cause this error." (annotation_level: failure).
• Job timing: started 15:18:48Z, completed 16:42:52Z (~84 min). Step 9 "Run diffusion server tests (1-GPU)" timeout-minutes: 90 and shows status: in_progress at job end — i.e. the runner died before the step's own timeout fired. A GitHub step timeout would instead surface "The operation was canceled."
• Per-job log unavailable: GET actions/jobs/89481337945/logs → BlobNotFound; the run-archive zip contains no shard-1 file. Artifacts: total_count: 0 (upload step never ran).
• Same run, same annotation on 2 sibling shards: 1-GPU part 0 (89481337911, ran ~97 min) and 2-GPU part 0 (89481337959, ran ~134 min). Longer-running shards died; shorter/passing shards did not.
• Sibling shard 3 (partition 4/4, 89481337980) failed differently — a real exit code 1: test/single_test_file/test_update_weights_from_disk.py had 4 failed / 4 errored [Qwen-Image]/[FLUX.2-klein-base-4B] cases due to missing HF-cache model dir (models--Qwen--Qwen-Image/.../transformer not found). This is a separate cluster, not this shard's issue.
• History: this shard failed 15/15 recent scheduled runs; 8/8 sampled carry the identical "runner lost communication" annotation (Jul 20→24).

Hypothesised Causes (with confidence, with disconfirming evidence)

• [LOW→MEDIUM] Resource starvation (host-RAM / VRAM OOM) during the long diffusion generation run kills the runner agent.
  • Supporting: annotation text explicitly lists "starves it for CPU/Memory" as a cause; the pattern is deterministic on the longest-running shards (1-GPU part 0 ~97 min, part 1 ~84 min, 2-GPU part 0 ~134 min) while short/passing shards survive; partition 2/4 runs the heavy parametrized diffusion cases (real T2I/I2V/LoRA model generation on Qwen-Image / FLUX-class models).
  • Disconfirming: no dmesg/OOM-killer evidence is available (the crashed runner's host log is not in the GitHub artifacts); could equally be a network drop or a hardware fault on the runner pool. Cannot confirm OOM without host-side logs.
  • How to verify: capture host dmesg/cgroup memory stats on the MI325 1-GPU runner during this step; or lower SGLANG_TEST_NUM_INFERENCE_STEPS/case count on part 1 and observe whether the crash disappears.
• [LOW] Flaky/unhealthy specific runner or network in the MI325 1-GPU pool.
  • Supporting: "lost communication" is a generic control-plane disconnect; recurs across many days.
  • Disconfirming: it correlates with shard workload length (only the long shards die, same day, different physical runners), which points more at workload-induced resource pressure than random per-machine network flakiness.
  • How to verify: correlate runner hostnames across the 8 failing runs; if the same machine recurs → machine health; if different machines all die on the long shards → workload/resource pressure.
• [SPECULATION] aiter/model code regression — not supported: the failure predates and spans many distinct sglang SHAs (Jul 17→24) with no green baseline, and produces no stack trace. No code-level evidence.

In-flight Fix Check

• In-flight fix: ❌ none found specifically targeting "multimodal-gen 1-GPU runner lost communication". Searched repo:sgl-project/sglang is:pr is:open with multimodal_gen+runner+communication (0 hits) and multimodal-gen+amd+runner (20 unrelated hits). Adjacent-but-not-a-fix: #28584 "WIP ci: test AMD PR workflow on new MI300 runners" (runner-pool migration, could incidentally affect this) and #29584 "ci(amd): fix three pre-existing MI325 CI failures". Neither is confirmed to address this runner death — do not assume.

Suggested Triage Steps

• Pull host-side logs for the crashed MI325 runner (dmesg / cgroup OOM / GPU reset) during 15:18–16:43Z on 2026-07-24 — the GitHub job log alone cannot diagnose a runner-agent death; this is the decisive next step.
• Correlate runner hostnames across the 8 failing runs (this shard) to separate "one sick machine" from "workload-induced resource pressure on any machine".
• If OOM is confirmed, trial reducing the partition-2/4 workload (fewer parametrized cases per shard, or split 1-GPU suite into more partitions) and/or lower SGLANG_TEST_NUM_INFERENCE_STEPS, then re-run and watch whether the crash clears.
• Track the sibling shard-3 failure (test_update_weights_from_disk.py, missing Qwen-Image/FLUX.2 HF-cache model dir) as a separate issue — it is a real assertion failure, not part of this runner-death cluster.

Status (factual, no priority assignment)

• Persistence: Failing every completed scheduled run in the 15-run window (Jul 17→24, 2026); all sampled runs show the identical "runner lost communication" annotation. No green run on this shard in the window.
• Scope: At least 3 shards in today's run died identically (1-GPU part 0, 1-GPU part 1, 2-GPU part 0) — the longest-running multimodal_gen shards. Recurs across days on the MI325 1-GPU/2-GPU pool.
• Blocked work: Blocks reliable multimodal_gen 1-GPU diffusion CI signal on AMD for partition 2/4 (parametrized T2I/I2V/LoRA generation coverage).
• In-flight fix: ❌ none found targeting this runner death; #28584 (runner migration) is adjacent but unconfirmed.

multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 0) — failed step(s): N/A

Commit Info

• sglang: 3d91a569 — [MoE Backend] Add HPC-Ops FP8 MoE runner backend (#30541)
• aiter: 9127c94a (v0.1.17.dev110+g9127c94a1, source: Dockerfile default — AITER_COMMIT_OVERRIDE empty). aiter markers read from sister shard 3 log of the same run; shard 0's own log expired.

Failed Tests

Test File	Test Function	Error	Log
python/sglang/multimodal_gen/test/run_suite.py --suite 1-gpu --partition-id 0 (partition 0 of 4)	N/A — step-level timeout, individual test not resolvable	Step Run diffusion server tests (1-GPU) cancelled at its timeout-minutes: 90 limit; job killed before log/artifact upload	Job (log returns 404 — never uploaded)

⚠️ Evidence limitation (FACT): shard 0's job log blob returns HTTP 404 (BlobNotFound) and the job is absent from the run-logs archive. No diffusion-failures artifact was uploaded (0 artifacts on the run). The runner was killed when step 9 hit its 90-min timeout, before the log-upload step (step 10) could run. Therefore the exact failing test file + function inside partition 0 cannot be pinned — the partitioning is by dynamic runtime estimate (partitioning.py), so partition 0's contents are computed at runtime, not statically. The failure is established at the step/job level, not the pytest level.

Failure Summary

The multimodal-gen-test-1-gpu-amd shard 0 job ran 97 min (2026-07-24 15:26:12 → 17:03:17 UTC) and its Run diffusion server tests (1-GPU) step (timeout-minutes: 90, pr-test-amd.yml:750-751) was cancelled at its 90-minute limit (step conclusion None). Partition 0 of the 1-GPU diffusion suite did not complete inference across its assigned large models (Wan2.1-T2V-1.3B, HunyuanVideo, Qwen-Image, FLUX.1-dev) within the window. This is a wall-clock timeout, not an assertion failure.

Regression Status

Known recurring failure (Never-passed in queryable window). Shard 0 has failed with the same signature (step 9 = None, i.e. cancelled/timed-out, duration clustered at/above the 90-min step limit) in every one of the last 15 completed scheduled runs. No green run exists in the lookback window 2026-07-17 → 2026-07-24.

Recent history of multimodal-gen-test-1-gpu-amd (…, 0) partition-0 diffusion step in job multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 0) (most recent 15 comparable completed scheduled runs: branch=main, event=schedule, completed only):

Date (UTC)	Run	Step 9 (Run diffusion…) status	Duration	Verdict
Jul 24 12:29	30093299057	❌ Cancelled (None)	97m	Failed (this run)
Jul 24 00:25	30056142455	❌ Cancelled (None)	82m	Failed
Jul 23 12:32	30007380837	❌ Cancelled (None)	83m	Failed
Jul 23 00:27	29969145884	❌ Cancelled (None)	90m	Failed
Jul 22 12:32	29920027108	❌ Cancelled (None)	85m	Failed
Jul 22 00:26	29880220845	❌ Cancelled (None)	88m	Failed
Jul 21 12:32	29830494225	❌ Cancelled (None)	122m	Failed
Jul 21 00:27	29790347494	❌ Cancelled (None)	99m	Failed
… Jul 17–20 (7 more)	29743214297 … 29580198489	❌ Cancelled (None) all 7	74–107m	Failed

All 15/15 failed at the step level; none passed. (Historical logs are expired, so verdict is based on step-9 conclusion + duration, the only durable signals for timed-out jobs.)

Failure Cluster

Diffusion 1-GPU partition-0 step timeout (90-min limit exceeded during model download/inference)

Facts (verified observations)

• FACT: Job 89481337911 ran 15:26:12→17:03:17 UTC (97m). Step Run diffusion server tests (1-GPU) conclusion = None (cancelled); steps Upload diffusion failure artifacts and Post Checkout code also None.
• FACT: The step carries timeout-minutes: 90 (pr-test-amd.yml:751).
• FACT: Shard 0 log = HTTP 404 (BlobNotFound); 0 artifacts on the run. Shard 1 (…, 1, also long-running 84m) log = 404 too; short-running shards 2 (success, 14m) and 3 (failed, 35m) have intact logs.
• FACT (same-run runner cache state, from shard 3 diagnose step): WARNING: /sgl-data is not a mount point - models will be cleaned up after each test; Found 0 MIOpen cache files; all 5 diffusion models reported ✗ NOT CACHED (Wan2.1-T2V-1.3B, HunyuanVideo, Qwen-Image, FLUX.1-dev, FLUX.2-dev).
• FACT: Reproducibility — 15/15 last scheduled runs show shard 0 step-9 = None; the two long-running shards (0 and 1) are the ones whose logs never upload.
• FACT: aiter unchanged (Dockerfile default 9127c94a, no override), so aiter is not a variable here.

Hypothesised Causes (with confidence)

• [MEDIUM] Cold model/kernel cache forces multi-GB downloads + MIOpen recompilation that blow past the 90-min step budget for partition 0.
  • Supporting: Same-run diagnose output shows /sgl-data not mounted, 0 MIOpen cache files, and all 5 diffusion models NOT CACHED. Partition 0 is assigned the heaviest models (FLUX.1-dev / HunyuanVideo / Qwen-Image are tens of GB each); a full cold download+load plausibly exceeds 90 min, while lighter shards (2: 14m) finish. The failure is persistent across 15 runs, consistent with a systemic runner-caching problem rather than a code change.
  • Disconfirming: The diagnose output is from shard 3's runner, not shard 0's — each matrix job lands on a different linux-mi325-1gpu-sglang runner, so shard 0's exact cache state is unconfirmed (its log expired). Duration varies 74–122m across history (not always pinned to exactly 90m), which a pure fixed-timeout story doesn't fully explain — a 74m run means the step didn't reach 90m yet still ended as None, suggesting an additional cancel path (e.g. hang/OOM or sibling fast-fail) may sometimes contribute.
  • How to verify: Re-run shard 0 with a warm /sgl-data mount confirmed (check the diagnose step prints ✓ CACHED for its assigned models and MIOpen files > 0); or add per-model download timing to the step to see whether downloads alone consume the 90 min.
• [LOW] A specific model load/inference hang in partition 0 (not download-bound).
  • Supporting: Some historical durations (74m) end before 90m with None, hinting at a non-timeout cancel.
  • Disconfirming: No stack trace or log is available for any shard-0 run to confirm a hang; artifacts never uploaded.
  • How to verify: Add py-spy dump / watchdog on the diffusion server process before the 90-min kill so a stack is captured on timeout.

(Not a scout run → no Failure Origin / aiter attribution section. aiter is the Dockerfile default and unchanged across all 15 runs.)

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for multimodal-gen+timeout, diffusion+timeout, diffusion+cache, multimodal_gen+cache. Nearest hits (#29584 "fix three pre-existing MI325 CI failures", #31483 "run vetted nested multimodal_gen unit tests on AMD") do not target the shard-0 diffusion step timeout / cold-cache issue.

Suggested Triage Steps

• Confirm whether the linux-mi325-1gpu-sglang runner pool actually has /sgl-data mounted as a persistent volume — the same-run diagnose printed /sgl-data is not a mount point, which defeats the HF/MIOpen cache the step relies on. If the mount is missing/misconfigured, warm-caching the 1-GPU diffusion models on the runner pool is the highest-ROL fix.
• Re-run only shard 0 with the diagnose step's cache output captured; verify whether its assigned models show ✓ CACHED. If not cached, the timeout is download-bound (infra), not a code regression.
• Instrument the diffusion step to log per-model download + load wall-time (and add a py-spy dump on timeout) so a future timeout produces a stack/timeline instead of a lost log — this is currently a blind spot because the 90-min kill drops the log and artifacts.
• If partition 0 is simply over-weighted, consider re-balancing the runtime-estimate partition plan (partitioning.py / the bot/update-est-time-* estimates) or raising timeout-minutes for the 1-GPU diffusion step — but only after confirming the cache-mount question, since a fresh mount would likely bring the step well under 90 min on its own.

Status

• Persistence: Failing every completed scheduled run for ≥8 days (since at least 2026-07-17; 15/15 runs in the queryable window, no green run found).
• Scope: Shard 0 of multimodal-gen-test-1-gpu-amd; shard 1 shows the same long-runtime/log-loss signature. Shards 2 (pass) and 3 (fast test_update_weights_from_disk failure) differ — this Status covers the shard-0 partition-0 timeout only.
• Blocked work: 1-GPU diffusion (multimodal_gen) CI signal for partition 0 on AMD MI325 is effectively unobservable (no log/artifact on timeout).
• In-flight fix: ❌ none found in open PRs.

multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 0) — failed step(s): N/A

Commit Info

• sglang: 3d91a569
• aiter: unable to determine — this job's log blob was never uploaded (runner lost, see below). The nearest recoverable run 30056142455 (2026-07-24 00:25, same image) ran aiter v0.1.17.dev110+g9127c94a1 (dev image; Dockerfile default 9127c94a, preserved — no rebuild).

Failed Tests

Test File	Test Function	Error	Log
N/A (non-test failure)	N/A	Runner lost: "The self-hosted runner lost communication with the server." Job killed mid–step 9 (Run diffusion server tests (2-GPU)) after ~2h. No test log uploaded.	Job

⚠️ This run's per-test result is unknown — the runner died while the test step was still in_progress, so GitHub has no log blob for it (BlobNotFound). The rows below are the persistent test-level failure recovered from the two nearest runs whose logs survived (part 0 runs all 25 parametrized 2-GPU cases on test_server_2_gpu.py):

Test File	Test Function	Error	Log
test/.../server/test_server_2_gpu.py	TestDiffusionServerTwoGpu::test_diffusion_generation[flux2_modelopt_fp8_tp2_t2i]	RuntimeError: Model generation returned no output … CUDA error: HIPBLAS_STATUS_NOT_SUPPORTED	prev run
test/.../server/test_server_2_gpu.py	TestDiffusionServerTwoGpu::test_diffusion_generation[joy_echo_t2v_2gpu]	ValueError: Could not find model at jdopensource/JoyAI-Echo … CAS Client Error … error decoding response body (HF download)	prev run

Failure Summary

Job multimodal-gen-test-2-gpu-amd (…, 0) failed because the self-hosted MI325 runner lost communication with GitHub during step 9 (the 2-GPU diffusion test step), roughly 2 hours into the step. The runner death prevented any log/artifact upload, so the individual test outcomes for this specific run cannot be verified. Part 0 is the heaviest 2-GPU shard: with 2 standalone files, parametrized_partitions = 3 − 2 = 1, so part 0 runs all 25 parametrized cases on test_server_2_gpu.py (Wan2.2-A14B, Wan2.1-14B, LTX-2.3, Qwen-Image, FLUX.2, etc.), which is why it consistently runs 85–155 min.

Regression Status

Known recurring failure (job red every run ≥8 days) + this run is an infra/runner-lost instance. The part-0 job has failed in all 15 most-recent completed scheduled runs. In the two runs whose logs survived, the recurring test-level cause is flux2_modelopt_fp8_tp2_t2i (HIPBLAS_STATUS_NOT_SUPPORTED). No green run for this test file appears anywhere in the 15-run window.

Recent history of test_server_2_gpu.py job multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 0) (most recent 15 comparable completed scheduled runs: branch=main, event=schedule, completed only):

Date (UTC)	Run	Dur	Test File Status	Detail
07-24 12:29	30093299057	134m	⚠️ Unknown (no log)	This run — runner lost communication mid-step
07-24 00:25	30056142455	155m	❌ Failed	flux2_modelopt_fp8_tp2_t2i HIPBLAS_STATUS_NOT_SUPPORTED; joy_echo HF-download error
07-23 12:32	30007380837	100m	❌ Failed	log unavailable (404) — job failed
07-23 00:27	29969145884	96m	❌ Failed	log unavailable (404)
07-22 12:32	29920027108	106m	❌ Failed	log unavailable
07-22 00:26	29880220845	85m	❌ Failed	log unavailable
07-21 12:32	29830494225	100m	❌ Failed	log unavailable
07-21 00:27	29790347494	106m	❌ Failed	log unavailable
07-20 12:42	29743214297	109m	❌ Failed	log unavailable
07-20 00:28	29709549202	103m	❌ Failed	log unavailable
07-19 12:17	29686709871	91m	❌ Failed	log unavailable
07-19 00:27	29666971143	98m	❌ Failed	log unavailable
07-18 12:17	29644059344	10m	❌ Failed	log unavailable — short (early infra failure?)
07-18 00:25	29623067212	116m	❌ Failed	flux2_modelopt_fp8_tp2_t2i HIPBLAS_STATUS_NOT_SUPPORTED; joy_echo HF-download error (1 failed, 22 passed, 1 skipped, 1 error)
07-17 12:25	29580198489	90m	❌ Failed	log unavailable

First observed failing: at least 2026-07-17 (window boundary; likely older). No passing run for this test file in the window.

Failure Cluster

Two clusters live in this job: (1) "Self-hosted runner lost communication during 2-GPU diffusion suite" — the symptom of this run; (2) "FLUX.2 modelopt-FP8 GEMM HIPBLAS_STATUS_NOT_SUPPORTED on gfx942" — the persistent test-level failure in the recoverable runs.

Facts (verified observations)

• This run: GitHub check-run annotation on job 89481337959 = "The self-hosted runner lost communication with the server. Verify the machine is running and has a healthy network connection…" Step timing: step 9 started 16:10:29Z, never completed; job force-closed 18:11:16Z (~121 min in step 9). Steps 10/20 never ran. Job-log endpoint and run-logs zip both return no blob for this shard (BlobNotFound) — consistent with runner death before log upload. [FACT]
• Sibling shards passed: part 1 (test_disagg_server.py, job 89481337920) ✅ 22m; part 2 (test_ar_models.py, job 89481337881) ✅ 15m. Only the heavy parametrized part 0 failed. [FACT]
• Persistent test failure (recovered logs, runs 07-24 00:25 and 07-18 00:25): identical flux2_modelopt_fp8_tp2_t2i → CUDA error: HIPBLAS_STATUS_NOT_SUPPORTED when calling HIPBLAS_STATUS_NOT_SUPPORTED, raised from image_api.py:305 → utils.py:355 (Model generation returned no output). Case uses FP8 transformer lmsys/flux2-dev-modelopt-fp8-sglang-transformer (testcase_configs.py:709). [FACT]
• Runner arch: GPU_ARCH=mi30x, gfx942 (MI325 = CDNA3/gfx942) — hipBLAS(Lt) has no supported kernel for this ModelOpt-FP8 GEMM config on gfx942. [FACT]
• joy_echo_t2v_2gpu fails at server startup with an HF-Hub CAS download error for jdopensource/JoyAI-Echo — a network/download problem, not a code fault. Present in both recovered runs. [FACT]

Hypothesised Causes (with confidence)

• [FACT] This run's job failure = self-hosted runner lost communication (infra), not a test assertion.
  • Supporting: the check-run annotation states it verbatim; step 9 left in_progress; no log blob uploaded.
  • Disconfirming: (none) — the annotation is authoritative for the job's terminal cause. What it does not tell us is whether a specific case hung/OOM'd the runner first.
  • How to verify: check the MI325 runner host's kernel logs / GPU health around 2026-07-24 16:10–18:11 UTC (dmesg, rocm-smi, XGMI/PCIe resets); check whether the box OOM'd or the GPU faulted during a heavy case.
• [HIGH] The underlying persistent red on this shard is flux2_modelopt_fp8_tp2_t2i → HIPBLAS_STATUS_NOT_SUPPORTED (ModelOpt-FP8 GEMM unsupported on gfx942).
  • Supporting: identical error in the two runs whose logs survived (07-18, 07-24), same stack (image_api.py:305); open fix PR #28889 titled "[AMD] Use native e4m3fnuz fp8 GEMM for FLUX.2 modelopt-FP8 on gfx942" targets exactly this.
  • Disconfirming: cannot confirm this same case failed in this specific run (no log); the ~85–155 min spread across runs shows the suite reaches the end in most runs (fast-fail of one case), whereas this run was killed at ~2h — so this run's proximate cause is the runner loss, layered on top of the standing flux2 red.
  • How to verify: once a run's log survives, confirm flux2_modelopt_fp8_tp2_t2i HIPBLAS line; validate PR #28889 removes it on a gfx942 box.
• [LOW] joy_echo_t2v_2gpu HF-download flake contributes a second red (ERROR at setup).
  • Supporting: CAS "error decoding response body" for jdopensource/JoyAI-Echo in both recovered runs.
  • Disconfirming: it's a transient network/Hub error, unrelated to the runner-lost event and to flux2; may pass on retry / once cached.
  • How to verify: pre-cache jdopensource/JoyAI-Echo on /sgl-data/hf-cache or retry the case standalone.

In-flight Fix Check

• flux2 FP8 GEMM: ✅ #28889 "[AMD] Use native e4m3fnuz fp8 GEMM for FLUX.2 modelopt-FP8 on gfx942" — open (non-draft) since 2026-06-22, last updated 2026-07-21. Directly addresses HIPBLAS_STATUS_NOT_SUPPORTED for this case; chase reviewers rather than opening a duplicate. (Searched is:pr is:open HIPBLAS and flux2 fp8.)
• Runner-lost / joy_echo download: ❌ none found — infra items, not code PRs.

Suggested Triage Steps

• Runner health (this run's cause): inspect the MI325/gfx942 self-hosted runner that took job 89481337959 for host-level failure between 16:10–18:11 UTC on 2026-07-24 (dmesg/OOM-killer, GPU reset/XGMI fault, agent crash, disk/network). Runner-lost with no log ≈ hardware/host event, not a code regression.
• Verify the standing red: land/validate PR #28889 on a gfx942 runner and re-run test_server_2_gpu.py -k flux2_modelopt_fp8_tp2_t2i; if green, the persistent part-0 red is resolved.
• De-flake joy_echo: pre-cache jdopensource/JoyAI-Echo into the persistent HF cache (mirrors the workflow's existing model-cache diagnostics) so a Hub CAS hiccup doesn't ERROR the shard.
• Optional shard rebalance: part 0 carrying all 25 parametrized cases (~2h) makes it the most exposed to a mid-run runner loss; consider splitting the parametrized set across ≥2 partitions to shorten exposure. (Investigation suggestion, not a directive.)

Status

• Persistence: part-0 job red in all 15 most-recent completed scheduled runs (≥ since 2026-07-17). Recoverable-log runs show a consistent flux2_modelopt_fp8_tp2_t2i HIPBLAS failure; this run is additionally an infra/runner-lost instance.
• Scope: only the 2-GPU part-0 shard on mi325; sibling shards (part 1, part 2) pass. Note the 1-GPU multimodal-gen shards also show failures in this same run (separate analysis).
• Blocked work: full AMD 2-GPU diffusion signal on test_server_2_gpu.py (all 25 parametrized cases gated behind one red shard).
• In-flight fix: ✅ #28889 for the flux2 HIPBLAS failure; ❌ none for the runner-lost/host event.

stage-b-test-large-8-gpu-mi35x-disaggregation-amd (linux-mi35x-gpu-8.fabric) — failed step(s): Run test_disaggregation — ⚠️ analysis failed

Analysis did not complete. The per-job agent produced no usable output (likely a timeout, log download failure, or subprocess crash). Manual triage required.

• Run: https://github.com/sgl-project/sglang/actions/runs/30093299057
• Job log: https://github.com/sgl-project/sglang/actions/runs/30093299057/job/89481338316
• Job ID: 89481338316
• Failed step(s): Run test_disaggregation
• Started (UTC): 2026-07-24 12:30

multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 3) — failed step(s): Run diffusion server tests (1-GPU)

The analysis is complete. The report was delivered in my previous turn. The background download task (historical job logs) has finished — those logs were already used to verify the test-file-level regression history across all 15 comparable runs.

Summary of findings: The multimodal-gen-test-1-gpu-amd shard 3 failure is a deterministic, never-passed ROCm defect in test_update_weights_from_disk.py — the diffusion update_weights_from_disk transformer reload trips a shape-mismatch assertion at linear.py:410 (HTTP 400) because the diffusers checkpoint layout isn't reconciled with sglang params. It's already root-caused in tracker #31924 with an in-flight skip PR #31925 awaiting merge. Not an aiter/attention issue.

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-07-25 13:12 UTC)

[amd-bot]

pr-test-amd.yml — Per-Job Analysis (continued 2/5)

stage-b-test-1-gpu-small-amd (linux-mi325-1gpu-sglang, 8) — failed step(s): Install dependencies

Commit Info

• sglang: 3d91a569
• aiter: N/A (no [CI-AITER-CHECK] markers in log — this job builds sgl-kernel inside the prebuilt rocm/sgl-dev image; not an AITER Scout job)

Failed Tests

Test File	Test Function	Error	Log
N/A (build/setup failure — no tests ran)	N/A	HW Exception by GPU node-2 … reason: GPU Hang → Process completed with exit code 134 (SIGABRT)	Job log

This is not a test failure. The job aborted during the bash scripts/ci/amd/amd_ci_install_dependency.sh step while building the sgl_kernel.common_ops HIP extension. The test suite for shard 8 never started.

Failure Summary

During the dependency-install/build step, the runner emitted HW Exception by GPU node-2 (Agent handle: 0x556696a7a3c0) reason :GPU Hang (log line 2008) while hipcc was compiling sgl-kernel sources, and the process aborted with exit code 134 (128+6 = SIGABRT) (log line 2015). No pytest/unittest ever executed for this shard.

Regression Status

New failure — single occurrence (candidate infrastructure/hardware issue).

Recent history of job stage-b-test-1-gpu-small-amd (linux-mi325-1gpu-sglang, 8) (most recent 10 comparable completed scheduled runs: same branch, event=schedule, completed only). Since no test ran, status is tracked at the build-step / job level:

Date	Run	Job Status	Signature
Jul 24 12:29	30093299057	❌ Failed	GPU Hang during sgl-kernel build (exit 134)
Jul 24 00:25	30056142455	✅ Passed	—
Jul 23 12:32	30007380837	✅ Passed	—
Jul 23 00:27	29969145884	✅ Passed	—
Jul 22 12:32	29920027108	✅ Passed	—
Jul 22 00:26	29880220845	✅ Passed	—
Jul 21 12:32	29830494225	✅ Passed	—
Jul 21 00:27	29790347494	✅ Passed	—
Jul 20 12:42	29743214297	✅ Passed	—
Jul 20 00:28	29709549202	✅ Passed	—

First observed: 2026-07-24 12:29 UTC (this run). Last passing: 2026-07-24 00:25 UTC. Shard 8 was green in all 9 preceding scheduled runs.

Failure Cluster

GPU Hang HW exception during sgl-kernel build (exit 134 / SIGABRT)

Facts (verified observations, no interpretation)

• Log line 2008: HW Exception by GPU node-2 (Agent handle: 0x556696a7a3c0) reason :GPU Hang (verbatim).
• Log line 2015: ##[error]Process completed with exit code 134. — 134 = 128 + SIGABRT(6).
• Failing step is Run bash scripts/ci/amd/amd_ci_install_dependency.sh (log line 1824); the crash occurs while ninja/hipcc is compiling sgl_kernel.common_ops (log lines 2000-2014, building 'sgl_kernel.common_ops' extension).
• No test framework output precedes the abort — the job never reached the test phase.
• Transient infra noise earlier in the same job: Local docker registry pull failed; falling back to public registry (log lines 1572-1573) — recovered via public-registry fallback, non-fatal.
• Runner: linux-mi325-1gpu-sglang-ng8rq-runner-p8ftv; GPU arch parse warning defaulting to mi30x (log line 1567); image rocm/sgl-dev:v0.5.15.post1-rocm700-mi30x-20260724.
• No [CI-AITER-CHECK] markers present → aiter version not applicable to this job.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [MEDIUM] Runner-side GPU hardware hang on node-2, unrelated to the sglang commit.
  • Supporting: the error is a hardware-level GPU Hang HW exception raised by the GPU agent, not a compiler or Python error; it surfaced during a CPU-side hipcc compile where GPU compute is not exercised, indicating an asynchronous pre-existing hang on the device. The same shard passed in the 9 prior scheduled runs on the identical build path; there is no code change to the build in this window that would newly hang a GPU.
  • Disconfirming: (none found) — no sglang source path in a stack trace (there is no stack trace), and no comparable historical GPU-Hang occurrence for this shard in the queried window.
  • How to verify: re-run the job on a different runner node; check node-2 dmesg/rocm-smi for XGMI/RAS/hang events at 2026-07-24 16:24 UTC; if a clean node passes, the hang is confirmed infra.
• [LOW] sglang commit 3d91a569 (HPC-Ops FP8 MoE runner backend) introduced the failure.
  • Supporting: it is the head commit of the failing run.
  • Disconfirming: the crash is a GPU hardware hang during compilation, not a build/compile error in any changed file; the failure mode (HW GPU Hang) is orthogonal to a Python/MoE backend change; only 1 of 10 runs failed. No evidence the commit is reachable at build time.
  • How to verify: not actionable until the infra hypothesis is ruled out; a single re-run should clear it if infra-caused.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs repo:sgl-project/sglang is:pr is:open GPU Hang in:title (0 results). No code fix is expected for a hardware hang; this is a re-run candidate.

Suggested Triage Steps

• Re-run the failed job; a single green re-run on a healthy node confirms a transient GPU hang and closes this out.
• If it re-fails on the same runner (...-runner-p8ftv / node-2), cordon/inspect that node — check dmesg, rocm-smi, and RAS logs around 2026-07-24 16:24 UTC for a device hang.
• Track at the build-step level, not the test level: no test file is implicated, so this should not be attributed to any test regression.
• Note the earlier local-registry pull timeout on the same runner as corroborating (but non-fatal) evidence of node-side flakiness.

Status (factual, no priority assignment)

• Persistence: Single occurrence — failed once (2026-07-24 12:29 UTC run); shard 8 passed in the 9 prior scheduled runs.
• Scope: 1 job (stage-b-test-1-gpu-small-amd (linux-mi325-1gpu-sglang, 8)); crash in the build/install step, so no tests were exercised.
• Blocked work: shard 8 of stage-b small-model tests produced no signal this run.
• In-flight fix: ❌ none found — re-run candidate; not a code defect.

stage-b-test-2-gpu-large-amd (linux-mi325-2gpu-sglang, 1) — failed step(s): Run test

Commit Info

• sglang: 3d91a569ce480f5548ca836799e84b7ee7fe97c4
• aiter: 9127c94a18e4398e1eba91f6639e910f0994ad02 (v0.1.17.dev110+g9127c94a1) — source: dev image preserved ([CI-AITER-CHECK] Dev/patched version detected → skipping rebuild; AITER_COMMIT_OVERRIDE empty, so this is a normal pr-test-amd.yml run, not a scout run).

Failed Tests

Test File	Test Function	Error	Log
test/registered/radix_cache/unified_radix_tree/test_unified_radix_cache_kl_full.py	TestUnifiedFullRadixCache.test_gsm8k (retry) / test_multiturn_decode_cache_hit_branching (1st attempt)	Fatal Python error: HW Exception by GPU node-3 reason :GPU Hang → watchdog Aborted → exit 134	Log

Non-test / secondary observation (did NOT fail the job — passed on retry):

Test File	Test Function	Observation
test_unified_radix_cache_kl_full.py	TestUnifiedFullRadixCache.test_mmlu (Qwen/Qwen3-32B)	MMLU 0.796875 < 0.8 on 1st eval, retry passed 0.875 — borderline accuracy flake

Failure Summary

The job failed at step 6 "Run test" with exit code 134 (SIGABRT). Both the first attempt and the HF_HUB_OFFLINE=1 retry crashed inside test_unified_radix_cache_kl_full.py (TestUnifiedFullRadixCache) with HW Exception by GPU node-3 reason :GPU Hang; the sglang watchdog (watchdog.py:147 _watchdog_once) then aborted the process. The two attempts hung at different test methods (1st: test_multiturn_decode_cache_hit_branching at 16:34:45; retry: test_gsm8k at ~136/200 generate requests, 16:42:53), i.e. the hang is not tied to a single deterministic code path.

Regression Status

New today (flaky/infra GPU Hang) for this test file.

Recent history of test_unified_radix_cache_kl_full.py in job stage-b-test-2-gpu-large-amd (linux-mi325-2gpu-sglang, 1) (most recent comparable completed scheduled runs: branch=main, event=schedule, completed only):

Date (UTC)	Run	Test File Status	Note
Jul 24 12:29	30093299057	❌ Failed	GPU Hang → exit 134 (both attempts)
Jul 24 00:25	30056142455	✅ Passed	job success
Jul 23 12:32	30007380837	✅ Passed (elapsed 669s)	job failed on unrelated 45-min action timeout
Jul 23 00:27	29969145884	✅ Passed	job success
Jul 22 12:32	29920027108	✅ Passed	job success
Jul 22 00:26	29880220845	✅ Passed	job success
Jul 21 12:32	29830494225	✅ Passed	job success
Jul 21 00:27	29790347494	✅ Passed	job success
Jul 20 12:42	29743214297	✅ Passed	job success
Jul 20 00:28	29709549202	✅ Passed (elapsed 375s)	job failed on unrelated test_load_weights_from_remote_instance.py
Jul 19 12:17	29686709871	✅ Passed	job success

First observed failure for this test file: Jul 24 12:29 UTC (today). Last known passing: Jul 24 00:25 UTC (immediately preceding run). The test file passed in all 10 prior comparable completed runs.

Failure Cluster

GPU Hang (HW Exception, node-3) during unified-radix-cache 2-GPU test → watchdog SIGABRT (exit 134)

Facts (verified observations)

• Step 6 "Run test" concluded failure; final line ##[error]Process completed with exit code 134. (raw log L14239).
• First attempt: HW Exception by GPU node-3 (Agent handle: …) reason :GPU Hang repeated across TP ranks, then Fatal Python error: Aborted with top frame watchdog.py:147 _watchdog_once (raw log L9615–9621). This fired right after TestUnifiedFullRadixCache.test_multiturn_decode_cache_hit_branching began (L9539) and after test_mmlu had already passed on retry (0.875, L9538).
• Retry attempt (HF_HUB_OFFLINE=1, L9748): tests 0–2 passed (bench_one_batch, hicache, dp_attention gsm8k 0.92); crashed at test 3 test_unified_radix_cache_kl_full.py during TestUnifiedFullRadixCache.test_gsm8k (L13494), at 136/200 generate requests → Fatal Python error: HW Exception by GPU node-3 … reason :GPU Hang (raw log L14144).
• The two crashes occur at different test methods, so the hang is non-deterministic.
• Secondary: test_mmlu (Qwen3-32B) first eval 0.796875 < 0.8 threshold (L9172), retry 0.875 passed (L9538) — did not fail the job.
• resource_tracker: There appear to be 2 leaked semaphore objects … 1 leaked shared_memory at shutdown (L9824) — consistent with an abrupt abort, not a clean exit.
• No AITER_COMMIT_OVERRIDE; aiter unchanged (dev image v0.1.17.dev110+g9127c94a1) — this test file passed with the same aiter across the prior 10 runs.

Hypothesised Causes (with confidence)

• [LOW] Flaky / infrastructure GPU hang on the MI325 2-GPU runner, not a code regression.
  • Supporting: symptom is a raw HW Exception by GPU node-3 reason :GPU Hang (hardware/driver-level, not a Python assertion); the two retry attempts hung at different test methods; the same test file + same sglang commit family passed cleanly in the immediately preceding run and all 10 prior comparable runs; open AMD CI PRs #31068 and #30431 exist specifically to detect/recover hung AMD GPUs.
  • Disconfirming: only one occurrence so far, so a genuine intermittent regression introduced in the 433429b16a..3d91a569ce window cannot be fully excluded from a single failing run.
  • How to verify: re-run the job; if the test file passes again on a clean runner, confirms flake. If it hangs repeatedly, bisect the window below.
• [LOW] Regression in the sglang commit window 433429b16a..3d91a569ce (last-pass → first-fail HEAD).
  • Supporting: today's HEAD is the first run to hang; temporal correlation only.
  • Disconfirming: no code-level evidence — the crash is a bare GPU Hang with no sglang stack frame at fault (watchdog is the reporter, not the culprit); hang location differs between attempts.
  • How to verify: git bisect over the window, re-running test_unified_radix_cache_kl_full.py on an MI325 2-GPU runner.

No hypothesis above LOW confidence — single non-deterministic GPU-hang occurrence; needs a re-run before any bisect.

In-flight Fix Check

• In-flight fix (infra): ✅ #31068 "[AMD] ci: detect and recover hung GPUs in MI35x pre-flight" and #30431 "[AMD] ci: self-heal driver-level VRAM leaks + graceful GPU cleanup on shutdown" — both address AMD GPU-hang recovery in CI (targeted at MI35x, but same failure class). Chase reviewers; do not duplicate.
• In-flight fix (code): ❌ none found matching test_unified_radix_cache_kl_full GPU hang. (Searched open PRs for GPU+Hang, unified+radix+cache, HW+Exception.)

Suggested Triage Steps

• Re-run job stage-b-test-2-gpu-large-amd (linux-mi325-2gpu-sglang, 1) on run 30093299057 once. If test_unified_radix_cache_kl_full.py passes, classify as a runner GPU-hang flake and drop.
• If it hangs again, capture dmesg / rocm-smi on the MI325 host at hang time to distinguish driver wedge vs kernel; check whether other 2-GPU MI325 jobs on the same runner also hung (runner-specific hardware).
• Only if repeatable across runners: git bisect 433429b16a..3d91a569ce re-running the single test file.
• Separately, watch TestUnifiedFullRadixCache.test_mmlu (Qwen3-32B) — 0.797 vs 0.8 is a tight margin; if it recurs, the threshold or eval sample size may need revisiting (independent of the GPU hang).

Status

• Persistence: First failure of this test file today (Jul 24 12:29 UTC); passed in the immediately preceding run and all 10 prior comparable scheduled runs. Non-deterministic within the run (both attempts hung, at different methods).
• Scope: This one job/shard (stage-b-test-2-gpu-large-amd (linux-mi325-2gpu-sglang, 1)), test file test_unified_radix_cache_kl_full.py. Not observed in adjacent runs of the same job.
• Blocked work: Blocks the 2-GPU large AMD stage-b signal for this run only; likely transient.
• In-flight fix: infra recovery PRs #31068 / #30431 open; no code fix needed unless the hang proves repeatable.

stage-c-test-large-8-gpu-amd-mi35x (linux-mi35x-gpu-8, 1) — failed step(s): Run test

Commit Info

• sglang: 3d91a569ce480f5548ca836799e84b7ee7fe97c4
• aiter: 9127c94a18e4398e1eba91f6639e910f0994ad02 (v0.1.17.dev110+g9127c94a1, source: dev image preserved — [CI-AITER-CHECK] Dev/patched version detected … → skipping rebuild; matches Dockerfile default)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/test_kimi_k25_mxfp4_bcg_mi35x.py	test_bcg_gsm8k (variant=bcg)	Exception: Server process exited with code -9 (server killed after model-load barrier failure)	Log

Failure Summary

The Kimi-K2.5-MXFP4 TP4 server launched by test_bcg_gsm8k never became healthy: it was killed with exit code -9 during startup. The kill was triggered by dist_barrier_after_load — TP ranks finished weight loading at wildly different times (TP2 396.95s, TP1 405.25s, TP0 1303.29s) and TP3 never printed Load weight end at all, so the post-load monitored_barrier (480s) timed out and raised ValueError, which the parent turned into a kill_process_tree. The test ran for 1821.8s before failing (FAILED (errors=1)), and the job exited 255.

Regression Status

New regression / candidate flake — first observed in this run (2026-07-24 12:29 UTC); the previous 3 comparable scheduled runs passed this shard. The test file was added on 2026-07-22 22:06 UTC in #30540 (0a6d1930), so no scheduled run before 2026-07-23 00:27 exercised it.

Recent history of test_kimi_k25_mxfp4_bcg_mi35x.py (test_bcg_gsm8k) in job stage-c-test-large-8-gpu-amd-mi35x (linux-mi35x-gpu-8, 1) (comparable completed event=schedule, branch=main runs; shard-1 job pass/fail reflects this test since a load failure fails the whole job):

Date	Run	Job	Test File Status	Failed Function	Error
Jul 24 12:29	30093299057	shard 1	❌ Failed	test_bcg_gsm8k	Server process exited with code -9
Jul 24 00:25	30056142455	shard 1	✅ Passed	—	—
Jul 23 12:32	30007380837	shard 1	✅ Passed	—	—
Jul 23 00:27	29969145884	shard 1	✅ Passed	—	—
≤ Jul 22 12:32	—	—	⬜ N/A	—	test file not yet present (added Jul 22 22:06)

First observed failure: 2026-07-24 12:29 UTC. Last known passing: 2026-07-24 00:25 UTC.

Failure Cluster

Unbalanced/stalled TP weight loading → monitored_barrier timeout during Kimi-K2.5-MXFP4 model load (server killed, exit -9).

Facts (verified observations, no interpretation)

• test_bcg_gsm8k raised Exception: Server process exited with code -9. Check server logs for errors. at test_kimi_k25_mxfp4_bcg_mi35x.py:158 → :126 → test_utils.py:1045 (job log ~line 16554-16563).
• Per-rank Load weight timings (all Load weight begin at 12:38:54): TP2 = 396.95s (12:45:31), TP1 = 405.25s (12:45:39), TP0 = 1303.29s (13:00:38), TP3 = no Load weight end emitted — py-spy at 13:03:49 shows TP3 still inside do_load_weights (deepseek_weight_loader.py:436) ← load_weights (kimi_k25.py:888) loading MoE experts (fused_moe_triton/layer.py).
• Scheduler exception (job log line 16527): ValueError: TP rank 2 could finish the model loading, but there are other ranks that didn't finish loading. It is likely due to unexpected failures (e.g., OOM) or a slow node. raised from load_model_utils.py:270 (dist_barrier_after_load).
• Barrier is dist.monitored_barrier(..., timeout=UNBALANCED_MODEL_LOADING_TIMEOUT_S) with UNBALANCED_MODEL_LOADING_TIMEOUT_S = 480 (load_model_utils.py:40,266). Followed by Received sigquit from a child process + kill_process_tree(parent_pid=4261).
• Repeated soft watchdog timeouts (watchdog_timeout=300, soft=True) fired for TokenizerManager and all 4 TP schedulers at 12:43:42–12:48:43 and again 13:03:49 — i.e. loading exceeded 300s while ranks were still in do_load_weights.
• Model: amd/Kimi-K2.5-MXFP4, TP4, quant=quark, per-rank weight mem usage=133.35 GB (avail mem after load ~145.88 GB). Launch flags include --mem-fraction-static 0.765, --model-loader-extra-config '{"enable_multithread_load": true}', --kv-cache-dtype fp8_e4m3.
• The second test in the same job, test_breakable_cuda_graph.py (small Qwen3ForCausalLM, load 16.18s), passed — the failure is specific to the large Kimi load, not the runner/GPU generally.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [MEDIUM] Infra/performance flake: one TP rank (TP3) stalled and another (TP0, 1303s) was ~3× slower than TP1/TP2 (~400s) loading the ~133 GB/rank MoE checkpoint, exceeding the fixed 480s post-load monitored_barrier, so the load was aborted.
  • Supporting: huge per-rank spread in Load weight end times; TP3 never completed; error text explicitly names "slow node"; the shared /sgl-data/hf-cache checkpoint read + multithread loader is I/O-bound; 3 immediately-prior scheduled runs passed with the identical head-shard.
  • Disconfirming: no explicit OOM / hipErrorOutOfMemory / memory-access-fault line was found in the log for TP3 (it simply stopped emitting), so an OOM-kill of TP3 cannot be fully ruled out vs a pure slowdown.
  • How to verify: re-run the job; if it passes, it's a load-time flake. Check dmesg/oom-killer and per-rank Load weight end timings on re-run; try raising the barrier timeout / lowering weight_loader_prefetch_num_threads to test I/O contention.
• [LOW] Code regression in the 24-commit window 433429b16a..3d91a569ce.
  • Supporting: failure appeared exactly on the first run of this SHA after 3 green runs.
  • Disconfirming: none of the 24 commits touch the Kimi/DeepSeek weight-load path or dist_barrier_after_load; the symptom is a timing/stall, not a deterministic error; a code regression would be expected to reproduce, whereas 07-24 00:25 (SHA 433429b16a, only 24 commits earlier) passed cleanly.
  • How to verify: re-run current SHA 2-3×; only bisect if the failure reproduces deterministically.
• [LOW] aiter dev build v0.1.17.dev110+g9127c94a1.
  • Supporting: AMD-only path.
  • Disconfirming: same aiter dev build was present in the 3 prior passing runs (unchanged Dockerfile-default dev image); failure is in Python weight-load/barrier code, not an aiter kernel.
  • How to verify: not actionable until the perf/stall hypothesis is ruled out.

No hypothesis reaches HIGH; the load-time stall (MEDIUM) is the leading candidate.

In-flight Fix Check

• In-flight fix: ❌ none found targeting the stalled-rank root cause. Older open PRs exist that would mitigate (make the fixed 480s barrier configurable), not fix the stall: #15637 "Add unbalanced_model_loading_timeout_s Configuration" and #9305 "Support setting UNBALANCED_MODEL_LOADING_TIMEOUT_S through environment variables" — both long-dormant (low PR numbers). Searched: search/issues q=repo:sgl-project/sglang is:pr is:open with keywords monitored_barrier model loading, unbalanced model loading, Kimi-K2.5 MXFP4.

Suggested Triage Steps

• Re-run job stage-c-test-large-8-gpu-amd-mi35x (linux-mi35x-gpu-8, 1) on SHA 3d91a569 2-3×. If it passes, log as a load-time flake and consider quarantine/retry rather than a code bisect.
• On re-run, capture per-rank Load weight durations and check the runner's oom-killer/dmesg for TP3 (PID under sglang::scheduler_TP3) to distinguish "slow I/O" from "OOM-killed rank".
• If the imbalance persists, investigate loader I/O contention on the shared /sgl-data/hf-cache mount and the enable_multithread_load / weight_loader_prefetch_num_threads=4 behavior for the 133 GB/rank Kimi checkpoint; a longer/ configurable UNBALANCED_MODEL_LOADING_TIMEOUT_S (revive #15637) would prevent the abort but only masks slow loads.
• Only bisect 433429b16a..3d91a569ce if the failure reproduces deterministically across re-runs.

Status (factual, no priority assignment)

• Persistence: 1 run (first failure 2026-07-24 12:29 UTC); the 3 prior comparable scheduled runs (back to test introduction on 2026-07-23 00:27) passed.
• Scope: 1 test function (test_bcg_gsm8k) in 1 job/shard (stage-c-test-large-8-gpu-amd-mi35x (linux-mi35x-gpu-8, 1)) of pr-test-amd.yml; sibling shards 0 and 2 passed; the co-located test_breakable_cuda_graph.py passed.
• Blocked work: Kimi-K2.5-MXFP4 BCG (breakable CUDA-graph) accuracy signal on AMD MI35x for this run only.
• In-flight fix: ❌ none for the root cause; only dormant timeout-configurability PRs #15637, #9305.

multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 0) — failed step(s): Run diffusion server tests (2-GPU)

Commit Info

• sglang: 433429b1
• aiter: v0.1.17.dev110+g9127c94a1 (9127c94a) — source: Dev/patched image version detected, rebuild skipped (AITER_COMMIT_OVERRIDE empty)

Failed Tests

Test File	Test Function	Error	Log
sglang/multimodal_gen/test/server/test_server_2_gpu.py	TestDiffusionServerTwoGpu::test_diffusion_generation[flux2_modelopt_fp8_tp2_t2i]	RuntimeError: CUDA error: HIPBLAS_STATUS_NOT_SUPPORTED (server → openai.InternalServerError)	Log
sglang/multimodal_gen/test/server/test_server_2_gpu.py	TestDiffusionServerTwoGpu::test_diffusion_generation[joy_echo_t2v_2gpu]	RuntimeError: Server exited early (code 1) — HF Hub download CAS Client Error: ... error decoding response body	Log

Shard result: 1 failed, 22 passed, 1 skipped, 1 error in 8467.36s.

Failure Summary

Two distinct, unrelated failures in the 2-GPU diffusion server shard. flux2_modelopt_fp8_tp2_t2i reached warmup then crashed at the first denoising step with HIPBLAS_STATUS_NOT_SUPPORTED raised from a QKV projection GEMM in flux_2.py:77, surfacing to the client as an Internal Server Error. joy_echo_t2v_2gpu never started: the server exited during model load because the jdopensource/JoyAI-Echo snapshot download from HF Hub failed with a Xet/CAS response-decoding I/O error.

Regression Status

Known recurring job failure — this job (multimodal-gen-test-2-gpu-amd (…, 0)) has conclusion=failure in all 16 most-recent completed scheduled runs (2026-07-17 → 2026-07-24).

⚠️ Test-function-level history could not be verified: the GitHub Actions logs for every prior run (job IDs 89481337959, 89206599876, … back to 88354736970) now return Azure BlobNotFound (logs GC'd), so I cannot confirm which specific test failed in each historical run. The job-level failure streak is FACT; the per-test streak is unverified.

Date	Run	Job	Test File Status	Note
Jul 24 00:25	30056142455	…2-gpu-amd (…,0)	❌ Failed	flux2 FAILED + joy_echo ERROR (this run)
Jul 24 12:29	30093299057	…2-gpu-amd (…,0)	❌ Failed	log expired — test unknown
Jul 23 → Jul 17	30007380837 … 29544840981	…2-gpu-amd (…,0)	❌ Failed	logs expired — tests unknown

Failure Cluster

Two clusters: (1) HIPBLAS_STATUS_NOT_SUPPORTED in FLUX.2 modelopt-FP8 QKV GEMM on gfx942; (2) HF Hub Xet/CAS model-download I/O error during server warmup.

Facts (verified observations)

• flux2_modelopt_fp8_tp2_t2i: [denoising_step_0] Error ... CUDA error: HIPBLAS_STATUS_NOT_SUPPORTED when calling HIPBLAS_STATUS_NOT_SUPPORTED, stack top flux_2.py:77 in _get_qkv_projections → attn.to_q(hidden_states), called from flux_2.py:321 → 759 → 1136 forward (log lines 4708-4805). Client saw openai.InternalServerError: Internal Server Error (line 16246).
• flux_2.py:77 executes query, _ = attn.to_q(hidden_states) — a Linear projection; for the modelopt-FP8 model this is an FP8 GEMM. HIPBLAS_STATUS_NOT_SUPPORTED is hipBLAS rejecting the requested FP8 GEMM configuration.
• joy_echo_t2v_2gpu: worker died during snapshot_download → xet_get → RuntimeError: Task error: File reconstruction error: CAS Client Error: Format error: I/O error: error decoding response body, wrapped as ValueError: Could not find model at jdopensource/JoyAI-Echo and failed to download from HF Hub (lines 16116-16198). Server then Exit code: 1, test framework raised Server exited early.
• Runner arch mi30x / gfx942 (MI325); aiter dev build preserved ([CI-AITER-CHECK] lines 4308-4314); not an AITER override run.
• Many benign MIOpen(HIP): Error [Init] Not found conv-init lines appear throughout but coincide with PASSED cases — not correlated with either failure.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [MEDIUM] FLUX.2 modelopt-FP8 path uses an FP8 GEMM dtype/layout that hipBLAS does not support on gfx942 (MI325), so attn.to_q raises HIPBLAS_STATUS_NOT_SUPPORTED.
  • Supporting: crash is exactly at the FP8 projection GEMM (flux_2.py:77); open PR #28889 "[AMD] Use native e4m3fnuz fp8 GEMM for FLUX.2 modelopt-FP8 on gfx942" targets this precise model+arch+op combination, indicating a recognized gfx942 FP8-GEMM gap.
  • Disconfirming: historical logs are GC'd, so I cannot confirm whether flux2_modelopt_fp8_tp2_t2i ever passed on this job — it may be a never-passed/unsupported-path case rather than a fresh regression. PR #28889 has been open since 2026-06-22, consistent with a long-standing gap, not a new break.
  • How to verify: re-run test_server_2_gpu.py::...[flux2_modelopt_fp8_tp2_t2i] on an MI325 container with PR #28889 applied; confirm the to_q FP8 GEMM no longer hits hipBLAS.
• [FACT] joy_echo_t2v_2gpu failed due to an HF Hub Xet/CAS download transport error, not sglang code.
  • Supporting: error is error decoding response body inside huggingface_hub xet_get; independent of sglang model/server logic.
  • Disconfirming (as an sglang-caused failure): none — this is an infrastructure/network flake in model fetch; retrying the download would likely succeed.
  • How to verify: re-run the shard; if it passes or fails on a different network step, confirms transient. Consider pre-caching jdopensource/JoyAI-Echo on the runner.

In-flight Fix Check

• flux2 FP8: In-flight fix: ✅ #28889 (open since 2026-06-22) "[AMD] Use native e4m3fnuz fp8 GEMM for FLUX.2 modelopt-FP8 on gfx942" — directly targets this failing path; ping reviewers, do NOT open a duplicate. (Searched search/issues?q=…+flux2+fp8.)
• joy_echo download: In-flight fix: ❌ none found (searched …+JoyAI-Echo) — infra flake, no code fix expected.

Suggested Triage Steps

• flux2: verify PR #28889 still applies cleanly on 433429b1 and re-run test_server_2_gpu.py::...[flux2_modelopt_fp8_tp2_t2i] on an MI325 runner; if green, chase merge.
• flux2: if #28889 does not fully resolve it, add a temporary skip/xfail for flux2_modelopt_fp8_tp2_t2i on gfx942 until native FP8 GEMM lands, so the shard isolates real regressions.
• joy_echo: rerun the shard to confirm the download error is transient; if it recurs, pre-stage/cache jdopensource/JoyAI-Echo on the AMD runner image or pin an HF mirror to remove the Xet/CAS dependency.
• To recover per-test history (blocked here by log GC), pull the uploaded junit_results_2-gpu_0.xml artifacts from recent runs instead of job logs.

Status

• Persistence: job multimodal-gen-test-2-gpu-amd (…,0) has failed in all 16 most-recent completed scheduled runs (since ≥2026-07-17); per-test persistence unverified (historical logs expired).
• Scope: this job/shard on pr-test-amd.yml (MI325, 2-GPU); two independent failing cases (flux2_modelopt_fp8_tp2_t2i, joy_echo_t2v_2gpu) out of 25 in the shard.
• Blocked work: FLUX.2 modelopt-FP8 2-GPU CI signal on gfx942; JoyAI-Echo 2-GPU coverage.
• In-flight fix: flux2 → ✅ #28889; joy_echo → ❌ none (infra retry).

multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 1) — failed step(s): N/A

Commit Info

• sglang: 433429b1
• aiter: N/A — the job log blob has expired (BlobNotFound from the logs API), so no [CI-AITER-CHECK] markers are recoverable for this run.

Failed Tests

Test File	Test Function	Error	Log
python/sglang/multimodal_gen/test/server/test_server_1_gpu.py (partition 1 of 4, -k "not flux_2") — exact case unknown	N/A (job never reported a test result)	Runner-level failure: "The self-hosted runner lost communication with the server" — step 9 Run diffusion server tests (1-GPU) hung ~80 min and never concluded	Job

This is not a test-assertion failure. Step 9 (Run diffusion server tests (1-GPU)) received a null conclusion — it never finished. The runner died at 80 min (job window 02:56:35→04:16:39 UTC), before the step's 90-min timeout-minutes fired, so this is a runner/host loss, not a step timeout. Because the log blob is gone, the specific parametrized 1-GPU case (from ONE_GPU_CASES in python/sglang/multimodal_gen/test/server/gpu_cases.py) that was executing when the runner died cannot be pinned.

Failure Summary

The multimodal-gen-test-1-gpu-amd partition 1 job failed because the self-hosted MI325 runner (linux-mi325-1gpu-sglang-ng8rq-runner-npkjn) lost communication with GitHub while the diffusion 1-GPU server test suite was running. The test step never produced a pass/fail result or exit code; GitHub marked the job failed via the check-run annotation only. No test-level error is available.

Regression Status

Known recurring failure (persistent). Partition 1 of this job has failed with the identical runner-communication-loss signature in every one of the most recent 10 comparable completed scheduled runs I inspected (and the job appears as failure in all 15 runs sampled). No passing run for partition 1 exists in the query window → effectively Never-passed within the lookback.

Recent history of multimodal-gen-test-1-gpu-amd (part 1) (event=schedule, branch=main, completed runs only; signature verified from step conclusions + check-run annotations):

Date (UTC)	Run	Job	Test Step Status	Signature
Jul 24 12:29	run	job	❌ hung (null, 1:24)	runner lost communication
Jul 24 00:25	run	job	❌ hung (null, 1:20)	runner lost communication (this run)
Jul 23 12:32	run	job	❌ hung (null, 1:19)	runner lost communication
Jul 23 00:27	run	job	❌ hung (null, 1:12)	runner lost communication
Jul 22 12:32	run	job	❌ hung (null, 1:17)	runner lost communication
Jul 22 00:26	run	job	❌ hung (null, 1:25)	runner lost communication
Jul 21 12:32	run	job	❌ hung (null, 1:51)	runner lost communication
Jul 21 00:27	run	job	❌ hung (null, 1:30)	runner lost communication
Jul 20 12:42	run	job	❌ hung (null, 1:24)	runner lost communication
Jul 20 00:28	run	job	❌ hung (null, 1:11)	runner lost communication

First observed failure (this signature) in the query window: Jul 20 00:28 UTC or earlier; no green run found for partition 1 in the last 15 scheduled runs.

Failure Cluster

Self-hosted AMD runner lost communication during diffusion 1-GPU suite (partition hang → runner death)

Facts (verified observations, no interpretation)

• Check-run annotation (verbatim): "The self-hosted runner lost communication with the server. Verify the machine is running and has a healthy network connection. Anything in your workflow that terminates the runner process, starves it for CPU/Memory, or blocks its network access can cause this error." [FACT]
• Step 9 Run diffusion server tests (1-GPU) conclusion = null (never completed); steps 10 and 20 also null. No exit code emitted. [FACT]
• Job wall-clock = 1:20:04 (02:56:35→04:16:39 UTC); step timeout is 90 min, so the step timeout did not fire. [FACT]
• The job log blob is expired: logs API returns <Code>BlobNotFound</Code>, so no stdout/stack trace is available for this run. [FACT]
• Same signature (null step conclusion + this annotation) reproduced in 10/10 most-recently-inspected completed scheduled runs of partition 1. [FACT]
• Same-run scope: in run 30056142455, partition 0 job had the same runner-comms-loss hang (1:22, null step); partition 3 job failed differently — a real test failure (step 9 failure, exit code 1, fast 0:27); partition 2 passed. So the hang hits the longer-running partitions (0, 1), not all of them. [FACT]
• The 1-GPU suite loads large diffusion models (FLUX.1-dev, HunyuanVideo, Qwen-Image, Wan2.1-T2V-1.3B per the workflow's cache-diagnostic step). [FACT]

Hypothesised Causes (with confidence, with disconfirming evidence)

• [LOW] Host resource exhaustion (system-RAM OOM or GPU driver/HIP crash) during a heavy 1-GPU diffusion case makes the runner host unresponsive, so the agent loses contact with GitHub.
  • Supporting: annotation explicitly lists "starves it for CPU/Memory" as a cause; failure only hits the longer partitions (0, 1) that pack the heaviest cases; diffusion models are large; the hang consistently occurs ~70–110 min in, consistent with a specific heavy case rather than immediate infra failure.
  • Disconfirming: no log survives to confirm an OOM-killer message or HIP fault — this is inferred from the annotation and partition pattern only, not observed. Partition 3 fails with a normal exit-1 test error on the same runner class, showing the runner class itself can report cleanly.
  • How to verify: re-run partition 1 with log retention captured; watch dmesg/OOM-killer and rocm-smi on the host; the workflow already prints free -h and rocm-smi pre-test (step 8) — add a post-mortem or per-case memory dump, or bisect which ONE_GPU_CASES entry in partition 1 precedes the hang.
• [LOW] Pure infrastructure/network flake on the MI325 runner pool (unrelated to sglang code).
  • Supporting: the message is a generic GitHub runner-agent error; sglang commit changes across the 10-run window did not alter the signature.
  • Disconfirming: a random network flake would not so reliably hit the same two partitions (0, 1) at a consistent ~80 min mark across 10 consecutive runs — the consistency argues against a random flake and toward a workload-triggered condition.
  • How to verify: correlate with other jobs on the same physical runner at the same timestamps; check runner-pool health dashboards.

No hypothesis reaches MEDIUM confidence — the surviving evidence (annotation + step metadata) cannot distinguish a workload-induced host crash from a pool infra fault without the run log. Root-cause work needs a fresh run with retained logs and host-side memory/GPU telemetry.

In-flight Fix Check

• Searched open PRs for diffusion+runner and multimodal-gen+amd. No PR targets the "runner lost communication" hang on the 1-GPU diffusion suite. Closest infra-adjacent items are #28584 ("WIP ci: test AMD PR workflow on new MI300 runners") and several diffusion CI-enablement PRs (#31899, #31846, #31483), none of which address this hang.
• In-flight fix: ❌ none found — needs new investigation (host telemetry + log retention first).

Suggested Triage Steps

• Re-run multimodal-gen-test-1-gpu-amd partition 1 and capture the log before retention expires; grep for OOM-killer / HIP fault / hang location.
• Instrument the runner host: capture dmesg -T, free -h, and rocm-smi at intervals during step 9 (the workflow only samples them once pre-test) to catch RAM/VRAM exhaustion.
• Bisect within partition 1: run the ONE_GPU_CASES entries assigned to partition 1 one at a time (test_server_1_gpu.py) to find which case precedes the runner death — the dynamic runtime-based partitioner concentrates the heaviest cases in partitions 0/1.
• Compare against partition 3's real exit-1 failure (a different signature) to confirm the runner class can report cleanly — this isolates the hang as workload/partition-specific rather than a blanket runner outage.

Status (factual, no priority assignment)

• Persistence: Failing with this exact signature in ≥10 consecutive completed scheduled runs (verified Jul 20 00:28 UTC → Jul 24 12:29 UTC); no green run for partition 1 in the last 15 scheduled runs.
• Scope: Within a run, partitions 0 and 1 of multimodal-gen-test-1-gpu-amd hang the runner; partition 2 passes and partition 3 fails with an unrelated real test error. So this cluster is 2 jobs/run in the 1-GPU diffusion suite on the MI325 pool.
• Blocked work: Blocks CI signal for the heavier 1-GPU diffusion cases on AMD MI325 — those cases are never actually verified because the runner dies before reporting.
• In-flight fix: ❌ none found — needs new investigation (retain logs + host telemetry first).

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-07-25 13:12 UTC)

[amd-bot]

pr-test-amd.yml — Per-Job Analysis (continued 3/5)

multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 3) — failed step(s): Run diffusion server tests (1-GPU)

Commit Info

• sglang: 433429b1
• aiter: 9127c94a (v0.1.17.dev110+g9127c94a1) — source: dev image preserved ([CI-AITER-CHECK] Dev/patched version detected → skipping rebuild)

Failed Tests

Job: multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 3). Summary: 4 failed, 2 passed, 11 warnings, 4 errors.

Test File	Test Function	Error	Log
test/single_test_file/test_update_weights_from_disk.py	TestUpdateWeightsFromDisk::test_update_weights_from_disk_default[Qwen-Image]	assert 400 == 200 (line 432)	Log
test/single_test_file/test_update_weights_from_disk.py	TestUpdateWeightsFromDisk::test_update_weights_specific_modules[Qwen-Image]	AssertionError: Update failed: {...'Failed to update module transformer'...} (line 454)	Log
test/single_test_file/test_update_weights_from_disk.py	TestUpdateWeightsFromDisk::test_corrupted_weights_rollback[Qwen-Image]	assert (400 == 200) (line 535)	Log
test/single_test_file/test_update_weights_from_disk.py	TestUpdateWeightsFromDiskWithOffload::test_update_weights_with_offload_enabled[Qwen-Image]	AssertionError: Expected 200, got 400 (line 638)	Log
test/single_test_file/test_update_weights_from_disk.py	TestUpdateWeightsFromDisk::*[FLUX.2-klein-base-4B] (4 setup errors)	AssertionError: No weights dir for transformer in /tmp/sglang_perturbed_vae_* (fixture diffusion_server_no_offload, line 410 → _compute_checksum_from_disk line 220)	Log

Failure Summary

All failures are in one test file, test_update_weights_from_disk.py. For [Qwen-Image], the server rejects POST /update_weights_from_disk with 400 Bad Request because the transformer module weight update aborts and rolls back. For [FLUX.2-klein-base-4B], the four cases error at fixture setup because the perturbed model dir has no transformer weights subdir. Two unrelated cases in the file pass.

Regression Status

Known recurring failure (persistent, identical signature). Not a new regression at this SHA — the exact 4 failed, 2 passed, 4 errors pattern with the same 4 [Qwen-Image] failures and 4 [FLUX.2-klein-base-4B] setup errors appears in every completed scheduled run checked back to 2026-07-20.

Recent history of test_update_weights_from_disk.py in job multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 3) (comparable completed runs: branch=main, event=schedule):

Date (UTC)	Run	Test File Status	Failed Functions	Error
Jul 24 12:29	30093299057	❌ Failed	4 [Qwen-Image] + 4 [FLUX.2] setup	400 / No weights dir
Jul 24 00:25	30056142455 (this)	❌ Failed	same	same
Jul 23 12:32	30007380837	❌ Failed	same	same
Jul 23 00:27	29969145884	❌ Failed	same	same
Jul 22 12:32	29920027108	❌ Failed	same	same
Jul 22 00:26	29880220845	❌ Failed	same	same
Jul 21 12:32	29830494225	❌ Failed	same	same
Jul 21 00:27	29790347494	❌ Failed	same	same
Jul 20 12:42	29743214297	❌ Failed	same	same

No passing run for this test file exists in the queryable window → effectively Never-passed in-window. Lookback range: 2026-07-20 12:42 → 2026-07-24 12:29 UTC (9 consecutive failing runs verified at the test-file level). The failure origin predates the local clone horizon (shallow, depth 56); the culprit commit is not in the queryable history.

Failure Cluster

update_weights_from_disk: transformer weight update aborts (400 / shape-mismatch assert) + missing perturbed-weights dir

Facts (verified observations)

• Qwen-Image root symptom (verbatim, log lines 5544 / 12053): Weight update failed for module 'transformer': . Rolling back … ['text_encoder', 'transformer'] followed by:

  File ".../post_training/weights_updater.py", line 295, in load_weights_into_model
    weight_loader(param, loaded_weight.to(param.dtype))
  File ".../runtime/layers/linear.py", line 410, in weight_loader
    assert param_data.shape == loaded_weight.shape
  AssertionError
  

  This aborts the update → POST /update_weights_from_disk … 400 Bad Request (log line 12127) → the 4 [Qwen-Image] test asserts status_code == 200 fail.
• FLUX.2 root symptom (setup errors, log lines 12074–12108): fixture diffusion_server_no_offload (test_update_weights_from_disk.py:410) calls _compute_checksum_from_disk(perturbed_vae_model_dir, module) (:220) which asserts os.path.exists('/tmp/sglang_perturbed_vae_*/transformer') → AssertionError: No weights dir for transformer. Distinct symptom, same file.
• Related warnings preceding the Qwen/Flux aborts: Skipping weight update for Flux2Transformer2DModel: parameter 'img_in.bias' not found after mapping to 'x_embedder.bias' (log line 5553) — a name-mapping miss in _load_weights_into_module (weights_updater.py:276).
• Infra note (not the failure cause): [server-test] ROCm GPU memory not fully cleared after 60s (used: 17.89 GB). Proceeding anyway (log line 12139) appears at setup of the failing Qwen-Image cases.
• linear.py, weights_updater.py, and the test file are all unchanged since 0a6d1930 (#30540, 2026-07-22); git log 0a6d1930..433429b1 on these paths is empty. Failures pre-date that commit (first observed 2026-07-20), so #30540 is not the cause.
• Reproducibility: 9/9 completed scheduled runs (2026-07-20 → 2026-07-24), identical counts and functions.

Hypothesised Causes (with confidence)

• [MEDIUM] The Qwen-Image transformer weight update hits a genuine tensor-shape mismatch in ColumnParallelLinear.weight_loader (linear.py:410) — the checkpoint tensor loaded off disk does not match the model parameter shape for at least one transformer param, so the hard assert aborts the whole module update.
  • Supporting: stack trace terminates exactly at assert param_data.shape == loaded_weight.shape; the immediately preceding img_in.bias → x_embedder.bias "not found after mapping" warning shows the on-disk→model name/shape mapping for these DiTs is imperfect. The failure is 100% deterministic across 9 runs, consistent with a real code/model mismatch rather than a flake or resource issue.
  • Disconfirming: I did not enumerate which specific param mismatches (would need a repro / added logging), and the code is unchanged in-window so this cannot be pinned to a specific in-window commit; the true regression point is beyond the shallow-clone horizon.
  • How to verify: run test/single_test_file/test_update_weights_from_disk.py::TestUpdateWeightsFromDisk::test_update_weights_specific_modules[Qwen-Image] on an MI325 ROCm container with param.shape / loaded_weight.shape logged at linear.py:410; identify the mismatching param name.
• [LOW] FLUX.2 [FLUX.2-klein-base-4B] setup errors are a separate test-harness issue: the perturbed-VAE fixture never materializes a transformer/ weights subdir it later checksums.
  • Supporting: AssertionError: No weights dir for transformer in /tmp/sglang_perturbed_vae_* originates in the fixture (:220), not in server/model code.
  • Disconfirming: could instead be a downstream effect of the same weight-update abort; not separated yet.
  • How to verify: inspect the fixture perturbed_vae_model_dir construction for FLUX.2 vs Qwen-Image and confirm whether a transformer subdir is expected to be written.
• [LOW] aiter / GPU-memory environment contribution.
  • Supporting: AMD-only lane; a ROCm GPU memory not fully cleared warning precedes the failing setups.
  • Disconfirming: aiter version is a stable dev build unchanged across passing sibling tests in the same job; the failure is a Python-level shape assert, not a kernel/GPU fault; the memory warning is non-fatal ("Proceeding anyway").
  • How to verify: not actionable until the shape mismatch above is characterized.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for update_weights_from_disk (2 hits: #30761/#30749, both [RL] LLM-runtime, not diffusion) and multimodal_gen weight (no matching fix). #31483 "[AMD] ci: run vetted nested multimodal_gen unit tests on AMD" only expands AMD coverage of the unit/ suite; it does not touch or quarantine single_test_file/test_update_weights_from_disk.py.

Suggested Triage Steps

• Reproduce test_update_weights_specific_modules[Qwen-Image] on an MI325 ROCm container with the SHA at linear.py:410 patched to log param name, param_data.shape, loaded_weight.shape; identify which transformer param mismatches on the disk→model mapping.
• Cross-check the Qwen-Image DiT weight-name mapper (_build_module_weight_name_mapper, weights_updater.py:263) against the img_in↔x_embedder regex in configs/models/dits/flux.py:62 and the Qwen-Image config — the "not found after mapping" warning suggests a stale/incomplete rename map.
• Separately triage the [FLUX.2-klein-base-4B] fixture: confirm whether perturbed_vae_model_dir is expected to contain a transformer/ subdir; if the harness only perturbs the VAE, the _compute_checksum_from_disk(..., "transformer") assertion at :220 may be a test bug independent of server code.
• Because the failure predates the clone horizon and code is unchanged in-window, git-bisect must run on a full clone across the window before 2026-07-20 to locate the introducing commit; alternatively git log --all on weights_updater.py/linear.py/qwen_image DiT on a full checkout.

Status

• Persistence: Failing for ≥5 days — every completed event=schedule run from 2026-07-20 12:42 UTC through 2026-07-24 12:29 UTC (9 consecutive, identical 4 failed / 2 passed / 4 errors). No in-window green.
• Scope: This one job/shard multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 3); single test file test_update_weights_from_disk.py; 4 [Qwen-Image] failures + 4 [FLUX.2-klein-base-4B] setup errors. Other tests in the shard (2) pass.
• Blocked work: Blocks AMD CI signal for diffusion update_weights_from_disk (post-training / RL weight refresh) on Qwen-Image and FLUX.2.
• In-flight fix: ❌ none found — needs new PR or investigation.

stage-b-test-large-8-gpu-mi35x-disaggregation-amd (linux-mi35x-gpu-8.fabric) — failed step(s): Run test_disaggregation

Commit Info

• sglang: 433429b16a3454b3fbe46d80b67f58b7116694f7
• aiter: 9127c94a18e4398e1eba91f6639e910f0994ad02 (v0.1.17.dev110+g9127c94a1, source: Dockerfile default — AITER_COMMIT_OVERRIDE empty, image version matched)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/disaggregation/test_disaggregation_basic.py	TestDisaggregationAccuracy.setUpClass	RuntimeError: Server process exited with code -9 (prefill server killed after HF Hub 429 Too Many Requests fetching Qwen/Qwen3-8B)	Log
test/registered/amd/disaggregation/test_disaggregation_pp.py	TestDisaggregationPrefillPPAccuracy.test_gsm8k	RuntimeError: … KVTransferError: Failed to get kvcache from prefill instance, it might be dead (mooncake Sync batch data transfer timeout after 30000001630ns); retry() exceed maximum number of retries	Log

Job overall: Test Summary: 2/4 passed (the two *_transfer_engine_e2e.py files passed).

Failure Summary

The stage-b-test-large-8-gpu-mi35x-disaggregation-amd job runs 4 disaggregation test files; 2 failed. test_disaggregation_basic.py aborted in setUpClass because the prefill server was SIGKILL'd (-9) right after HuggingFace Hub returned 429 Too Many Requests while resolving Qwen/Qwen3-8B. test_disaggregation_pp.py::test_gsm8k failed because KV-cache transfer from the prefill to the decode instance timed out (mooncake Sync batch data transfer timeout after 30s → KVTransferError), returning HTTP 500 for a fraction of the 200 GSM8K requests and exhausting the test's retry budget.

Regression Status

Two independent failures with different characters:

• test_disaggregation_pp.py (test_gsm8k) — Flaky / recurring (intermittent KV-transfer timeout). Failed in 6 of the last 9 comparable completed scheduled runs; last green 2026-07-22 12:32 UTC.
• test_disaggregation_basic.py (setUpClass) — New today / Infrastructure (external) — HF Hub API rate limit. Passed in every other run in the window.

Recent history in job stage-b-test-large-8-gpu-mi35x-disaggregation-amd (most recent comparable completed scheduled runs: branch=main, event=schedule):

Date (UTC)	Run	test_disaggregation_pp.py	test_disaggregation_basic.py	pp failed function / error
07-24 12:29	30093299057	❌ Failed	✅ Passed	test_gsm8k — KVTransferError
07-24 00:25	30056142455 (this run)	❌ Failed	❌ Failed (HF 429)	test_gsm8k — KVTransferError
07-23 12:32	30007380837	❌ Failed	✅ Passed	test_gsm8k — KVTransferError
07-23 00:27	29969145884	❌ Failed	✅ Passed	test_gsm8k — KVTransferError
07-22 12:32	29920027108	✅ Passed	✅ Passed	— (4/4 passed)
07-22 00:26	29880220845	❌ Failed	✅ Passed	test_gsm8k — KVTransferError
07-21 12:32	29830494225	❌ Failed	✅ Passed	test_gsm8k — KVTransferError
07-21 00:27	29790347494	✅ Passed	✅ Passed	—
07-20 12:42	29743214297	✅ Passed	✅ Passed	—

The test_disaggregation_basic.py HF-429 failure is unique to this run within the window (it passed in all 8 other runs). The test_disaggregation_pp.py timeout is intermittent and predates this commit.

Failure Cluster

• Cluster A: PD disaggregation mooncake KV-transfer timeout on MI35x runner (test_disaggregation_pp.py).
• Cluster B: HuggingFace Hub 429 rate limit during model warmup (test_disaggregation_basic.py).

Facts (verified observations)

• test_disaggregation_basic.py: log shows verbatim 429 Too Many Requests: you have reached your 'api' rate limit … Url: https://huggingface.co/api/models/Qwen/Qwen3-8B, immediately followed by Received sigquit from a child process, then setUpClass raising RuntimeError: Server process exited with code -9 at test_disaggregation_basic.py:46 (wait_server_ready on the prefill /health).
• test_disaggregation_pp.py: log shows transfer_engine_py.cpp:614] Sync batch data transfer timeout after 30000001630ns → KVTransferError … Failed to get kvcache from prefill instance, it might be dead → HTTP 500. First attempt also raised KeyError: 'answer' (a request returned no completion); the retry hit the transfer timeout, then retry() exceed maximum number of retries.
• The pp failure reproduces with the same symptom across 6 runs on different head SHAs (433429b16a, b98a577fbe, 24da0e51b6, b13abfdedf, e4eea7ce2f, 3d91a569ce), and passes intermittently on other SHAs — behaviour consistent with a non-deterministic runtime/hardware issue, not a single-commit regression.
• aiter version is unchanged (Dockerfile default 9127c94a1) across passing and failing runs.
• A standing debug PR #29127 exists specifically to isolate a faulty RDMA NIC (rdma5) on the MI35x disaggregation runner — corroborating a runner-side RDMA transport hypothesis for cluster A.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [FACT] Cluster B (test_disaggregation_basic.py) is caused by an external HuggingFace Hub API rate limit (429) blocking model resolution, killing the prefill server (-9). Not a code defect in this PR/commit.
  • Supporting: explicit 429 … 'api' rate limit … quota of 3000 api requests per 5 minutes in the log; basic passes in every other run in the window.
  • Disconfirming: (none — the error message is unambiguous and external).
  • How to verify: re-run the job; enable HF token/HF_HUB caching or a local model mirror to avoid live HF calls.
• [MEDIUM] Cluster A (test_disaggregation_pp.py) is a runner/transport-level KV-cache transfer stall (mooncake 30s sync-transfer timeout), likely tied to a flaky RDMA NIC / fabric on the MI35x 8-GPU disaggregation runner rather than a specific sglang commit.
  • Supporting: identical timeout symptom across 6 different head SHAs; intermittent green runs on the same code; standing PR #29127 targets a faulty NIC (rdma5) on exactly this runner; error originates in transfer_engine_py.cpp:614 (transport layer), not model/attention code.
  • Disconfirming: no per-run pin to a NIC was captured in this log to prove rdma5 specifically was in use; a software race in decode-receiver cleanup (see in-flight PRs below) could produce the same symptom, so hardware is not conclusively isolated.
  • How to verify: correlate failing vs passing runs against the RDMA device selected (SGLANG_TEST_RDMA_DEVICE/--disaggregation-ib-device); run the pp test pinned to a known-good NIC; A/B test candidate PRs #31800 / #31589.
• [LOW] The first-try KeyError: 'answer' indicates some GSM8K requests returned empty completions independent of the transfer timeout.
  • Supporting: appears once before the transfer-timeout retry.
  • Disconfirming: only observed on the first attempt; the decisive, repeated failure is the KV-transfer timeout.
  • How to verify: inspect decode-server logs for the specific rid that produced no answer.

In-flight Fix Check

Cluster A (pp KV-transfer timeout / MI35x runner):

• In-flight fix: ✅ #29127 — [DEBUG] Isolate faulty RDMA NIC (rdma5) on MI35x disaggregation runner (directly targets this runner's transport).
• In-flight fix: ✅ #31800 — disaggregation/mooncake: time out KV receivers stuck in Transferring.
• In-flight fix: ✅ #31589 — [PD][NIXL] Clean up decode receiver state after failed transfers.
• Also related: #31798 [PD+PP] Make decode queue consensus authoritative, #31144 [PD] Prevent decode scheduler from blocking on ZMQ sends to a stalled prefill peer.

Cluster B (HF 429):

• In-flight fix: ❌ none found — external rate-limit; mitigate via CI HF caching/mirror, not a code PR. (Search: search/issues?q=…is:pr+is:open+huggingface+429 returned no matching CI-caching PR.)

Suggested Triage Steps

• Cluster A (pp): Correlate the RDMA device in passing vs failing runs; if rdma5 (or a specific NIC) is implicated, coordinate with the #29127 effort to quarantine it on the MI35x runner. If software-side, A/B #31800 then re-run test_disaggregation_pp.py on the MI35x 8-GPU runner.
• Cluster A: Since it is intermittent across SHAs (last green 29920027108, 07-22 12:32), do not bisect for a single culprit commit; treat as flake/infra until a NIC or receiver-cleanup fix lands.
• Cluster B (basic): Re-run the job — HF 429 is transient. Longer term, pre-cache Qwen/Qwen3-8B on the runner or route through a local HF mirror / authenticated token to avoid live-quota exhaustion during warmup.

Status

• Persistence: test_disaggregation_pp.py::test_gsm8k intermittently failing since ≥2026-07-21 (6 of last 9 completed scheduled runs; last green 2026-07-22 12:32 UTC). test_disaggregation_basic.py HF-429 failure is a one-off this run.
• Scope: Single job stage-b-test-large-8-gpu-mi35x-disaggregation-amd in pr-test-amd.yml; both failures are MI35x-disaggregation-runner-specific (transport / external HF), not attention/kernel/aiter code.
• Blocked work: PD+PP disaggregation CI signal on AMD MI35x is unreliable while cluster A persists.
• In-flight fix: cluster A — candidate PRs #29127, #31800, #31589; cluster B — none (external rate limit, mitigate in CI infra).

multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 0) — failed step(s): N/A

Commit Info

• sglang: 433429b1
• aiter: N/A — the target job's log blob has been purged (run is >24 h old) and no run artifacts survived, so [CI-AITER-CHECK] markers are unrecoverable.

Failed Tests

Test File	Test Function	Error	Log
python/sglang/multimodal_gen/test/run_suite.py (1-gpu suite, partition 0/4)	N/A — hang, not a test assertion	Step 9 "Run diffusion server tests (1-GPU)" never recorded a conclusion; post-job steps (20/21) were stripped and the job was killed after 82 min (02:59:01→04:21:05 UTC). Individual hanging test unrecoverable (log purged).	Job

Limitation (FACT): GitHub purged this job's log blob and all run artifacts (total_count: 0). The run-level log archive preserved only shards 2 and 3, not the target shard 0. The exact hanging model/test in partition 0 therefore cannot be extracted from this run. Classification below is derived from step-conclusion metadata + sibling-shard logs + job-history metadata.

Failure Summary

The multimodal-gen-test-1-gpu-amd shard 0 (partition 0 of 4 of the 1-GPU diffusion server suite) hung: its test step ran ~82 min without recording a conclusion, then the runner killed the job before the post-job steps. This is a hang/timeout, distinct from the clean pytest failure in sibling shard 3 (partition 4/4), which failed with test_update_weights_from_disk.py FLUX.2/Qwen-Image weight-dir assertions. Shard 1 shows the identical hang signature to shard 0; shard 2 passed in 12 min.

Regression Status

Known recurring failure (persistent hang). Partition 0 of this job has failed in every completed scheduled run for ~16 days. It evolved through two modes: clean pytest failures starting 2026-07-08 12:37, then hangs/kills starting ~2026-07-12. Last green: 2026-07-08 00:37.

Recent history of multimodal-gen-test-1-gpu-amd partition 0 (most recent 15 comparable completed scheduled runs: branch=main, event=schedule, completed only). Status is at the step-9 level (hang vs clean-fail vs pass), NOT per-test — per-test detail is unrecoverable because logs for these older runs are also purged:

Date (UTC)	Run	Step-9 Status	Mode	Job runtime
Jul 24 12:29	run	❌	killed/hang	97 min
Jul 24 00:25	run (this)	❌	killed/hang	82 min
Jul 23 12:32	run	❌	killed/hang	83 min
Jul 23 00:27	run	❌	killed/hang	90 min
Jul 22 12:32	run	❌	killed/hang	85 min
Jul 22 00:26	run	❌	killed/hang	88 min
Jul 21 12:32	run	❌	killed/hang	122 min
Jul 21 00:27	run	❌	killed/hang	99 min
Jul 20 12:42	run	❌	killed/hang	107 min
Jul 20 00:28	run	❌	killed/hang	89 min
Jul 19 12:17	run	❌	killed/hang	82 min
Jul 19 00:27	run	❌	killed/hang	80 min
Jul 18 12:17	run	❌	killed/hang	74 min
Jul 18 00:25	run	❌	killed/hang	91 min
Jul 17 12:25	run	❌	killed/hang	85 min

Earlier transition points (for boundary evidence):

• Jul 16 00:25 run — part-0 skipped (not run that cycle).
• Jul 12 00:27 run — first hang/kill observed (mode switch from clean-fail → hang).
• Jul 10 00:32 run — clean failure (step9=failure).
• Jul 08 12:37 run (sha 108a183f) — first failure (clean).
• Jul 08 00:37 run (sha b3632494) — last green (part-0 success, 42 min).

Failure Cluster

"Diffusion 1-GPU suite partition-0 hang → job killed after ~80 min (AMD MI325)"

Facts (verified observations)

• Target job 89368158643: steps 1–8 succeeded; step 9 conclusion = None; steps 10/20/21 absent; job conclusion = failure; runtime 82 min. (FACT, from jobs API metadata.)
• Sibling shard 1 89368158583 has the identical signature (step 9 None, killed, 80 min). Shard 2 89368158627 passed (12 min). Shard 3 89368158618 failed cleanly (step 9 failure, post-steps present, 27 min).
• Shard 3's clean failure is a different cluster: test_update_weights_from_disk.py::TestUpdateWeightsFromDisk — AssertionError: No weights dir for vae/transformer in .../models--black-forest-labs--FLUX.2-klein-base-4B/... and --Qwen--Qwen-Image/... → 4 failed, 2 passed, 4 errors in 1157s. (FACT, shard3 log lines 12175/12215/12256.) This is a missing-model-weights/cache issue, NOT the partition-0 hang.
• Step 9 has timeout-minutes: 90 (.github/workflows/pr-test-amd.yml:751); the kills at 74–122 min job-wall-time straddle that boundary, consistent with the step's own 90-min timeout firing (or a harness-internal kill). (FACT.)
• Partition 0 membership is dynamic (LPT by estimated runtime, run_suite.py:117-131); the 1-GPU suite spans Qwen-Image, FLUX.1/FLUX.2, Wan2.1-T2V, HunyuanVideo, Cosmos, etc. (FACT.) Which of these lands in partition 0 for this SHA is not recoverable without the log.
• Regression window last-green→first-fail = b3632494..108a183f = 27 commits, many touching python/sglang/multimodal_gen/. (FACT, from GitHub compare API.)

Hypothesised Causes (with confidence, with disconfirming evidence)

• [LOW→MEDIUM] Breakable CUDA Graph (BCG) enablement for diffusion DiTs — 33c3dfd7e0 "[diffusion] Enable breakable CUDA graph (BCG) for diffusion DiTs" (#27436), merged inside the July-8 regression window.
  • Supporting: adds a whole new multimodal_gen/runtime/breakable_cuda_graph/ runner + model padders and modifies pipelines_core/stages/denoising.py and runtime/layers/attention/layer.py — i.e. the diffusion forward/denoise hot path. CUDA-graph capture is a well-known hang source on ROCm/MI325. Timing aligns with the first failure (Jul 8) and the progression to full hangs (Jul 12) is consistent with a capture/graph-replay stall that only manifests on the heavier models that LPT concentrates into partitions 0/1.
  • Disconfirming: partition 2 of the same suite passes cleanly, so BCG is not wholesale broken — the hang is model/partition-specific. No partition-0 stack trace exists to confirm the hang is inside BCG. Multiple other diffusion commits (LingBot-World 2.0 db40fd83, Krea-2 SP 68901ba3, z-Image accuracy fa185ed8) landed in the same window and cannot be excluded.
  • How to verify: reproduce the 1-GPU suite partition 0 on an MI325 runner with BCG disabled (find its server-arg/env toggle) vs enabled, on SHA 433429b1; if the hang clears with BCG off, this is confirmed.
• [LOW] Genuine per-model regression among the July-8 diffusion changes (LingBot-World 2.0 / Krea-2 SP / z-Image) causing a server-start or denoise stall on the specific model(s) in partition 0.
  • Supporting: the clean-failure phase (Jul 8–11) preceded the hangs, suggesting a real functional break that later degraded into a hang as more follow-ups landed.
  • Disconfirming: no per-test evidence; purely temporal.
  • How to verify: once a fresh run's log is available (not purged), read which model in partition 0 stalls, then bisect within the 27-commit window filtered to that model's config/DiT.
• [SPECULATION] Runner/infra resource exhaustion (VRAM not cleared, host-mem OOM) causing a stall.
  • Supporting: shard 3 log warns "ROCm GPU memory not fully cleared after 60s (used 17.14 GB)... may cause OOM" (test_server_utils.py:360).
  • Disconfirming: the failure is deterministic across 16 days and specific to partitions 0/1 (2 always pass), which argues against random infra flake.
  • How to verify: not actionable until code hypotheses are ruled out.

In-flight Fix Check

• In-flight fix: ❌ none found targeting the AMD diffusion 1-GPU partition-0 hang. Searched open PRs for diffusion+cuda+graph+hang (2 hits, unrelated), breakable+cuda+graph (37 hits — all feature/extension work e.g. #31852, #30988, #31951, none an AMD hang fix), diffusion+1-gpu+timeout (12 hits, none matching).

Suggested Triage Steps

• Re-trigger this job on a fresh SHA and capture the log before it expires — the single biggest blocker here is that every historical log for this job is already purged. Add/keep the "Upload diffusion failure artifacts" output and consider a py-spy dump on the hanging server so the partition-0 stall point is captured.
• A/B the BCG hypothesis: run run_suite.py --suite 1-gpu --partition-id 0 --total-partitions 4 -k "not flux_2" on MI325 with breakable CUDA graph disabled vs enabled on 433429b1.
• Bisect the 27-commit window b3632494..108a183f, prioritizing the diffusion commits (#27436 BCG first, then #30361, #29777, #29742).
• Track shard 3's FLUX.2/Qwen-Image weight-dir assertion failures separately — it is a distinct missing-weights/cache cluster, not the partition-0 hang.

Status (factual)

• Persistence: partition 0 has failed in every completed scheduled run since 2026-07-08 12:37 (~16 days); hang mode since ~2026-07-12. Last green 2026-07-08 00:37.
• Scope: partitions 0 and 1 of multimodal-gen-test-1-gpu-amd hang identically each run; partition 2 passes, partition 3 fails on a separate weight-dir cluster. Blocks 1-GPU diffusion CI signal on AMD MI325.
• Blocked work: AMD 1-GPU diffusion server test coverage (partitions 0/1) is effectively dark.
• In-flight fix: ❌ none found in open PRs.

stage-b-test-large-8-gpu-mi35x-disaggregation-amd (linux-mi35x-gpu-8.fabric) — failed step(s): Run test_disaggregation

Commit Info

• sglang: 1ef2b85e (chore: bump docs install version to 0.5.16 #32347 — docs-only)
• aiter: 9127c94a — v0.1.17.dev110+g9127c94a1 (source: dev image preserved; [CI-AITER-CHECK] reported Dev/patched version detected → skipping rebuild; AITER_COMMIT_OVERRIDE empty ⇒ not a scout run)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/disaggregation/test_disaggregation_pp.py	TestDisaggregationDecodePPAccuracy.test_gsm8k (line 293)	RuntimeError: Internal Server Error → ConnectionRefusedError: [Errno 111] Connect call failed ('127.0.0.1', 11200); retry exhausted	Log

Other 3 files in this job passed (test_disaggregation_basic.py, test_mori_transfer_engine_e2e.py, test_nixl_transfer_engine_e2e.py). Test Summary: 3/4 passed.

Failure Summary

In the PP disaggregation accuracy test (Qwen3-8B, --tp-size 2 --pp-size 2, --disaggregation-transfer-backend mooncake), the GSM8K benchmark started decoding normally, then the decode server crashed mid-run. The GSM8K harness saw a burst of 500 Internal Server Error on /generate (yielding KeyError: 'answer'), and on the automatic retry the decode server at port 11200 was already gone (ConnectionRefusedError), so the test errored out. The job exited with code 255.

Regression Status

Known recurring / flaky failure (intermittent). test_disaggregation_pp.py fails in roughly 10 of the last 16 comparable scheduled runs and passes in the other 6, on unrelated head SHAs. The HEAD commit is a docs version bump and cannot affect the disaggregation runtime, so this is not a fresh regression tied to 1ef2b85e.

Recent history of test_disaggregation_pp.py in job stage-b-...-mi35x-disaggregation-amd (16 most recent comparable completed scheduled runs: branch=main, event=schedule):

Date (UTC)	Run	Test File Status	Failed Function	Error
07-25 00:29	30136388246	❌ Failed	TestDisaggregationDecodePPAccuracy.test_gsm8k	KV-transfer → decode crash
07-24 12:29	30093299057	❌ Failed	test_gsm8k (l.198)	Internal Server Error
07-24 00:25	30056142455	❌ Failed	test_gsm8k (l.104)	Internal Server Error
07-23 12:32	30007380837	❌ Failed	test_gsm8k (l.104)	Internal Server Error
07-23 00:27	29969145884	❌ Failed	test_gsm8k (l.198)	Internal Server Error
07-22 12:32	29920027108	✅ Passed	—	— (4/4 passed)
07-22 00:26	29880220845	❌ Failed	test_gsm8k (l.198)	Internal Server Error
07-21 12:32	29830494225	❌ Failed	test_gsm8k (l.293)	Internal Server Error
07-21 00:27	29790347494	✅ Passed	—	—
07-20 12:42	29743214297	✅ Passed	—	—
07-20 00:28	29709549202	✅ Passed	—	—
07-19 12:17	29686709871	✅ Passed	—	—
07-19 00:27	29666971143	❌ Failed	test_gsm8k (l.104)	Internal Server Error
07-18 12:17	29644059344	✅ Passed	—	—
07-18 00:25	29623067212	❌ Failed	test_gsm8k (l.293)	Internal Server Error
07-17 12:25	29580198489	❌ Failed	test_gsm8k (l.104)	Internal Server Error

First failure in window: 2026-07-17 12:25. Last known passing: 2026-07-22 12:32 (run 29920027108, 4/4 passed). Pattern is intermittent, not monotonic — 10 failed / 6 passed.

Failure Cluster

KV-cache transfer failure → decode scheduler PP event-loop hang → watchdog timeout + SIGQUIT (PD+PP disaggregation, mooncake backend)

Facts (verified observations)

• 00:41:36 [PP1 TP0] Decode transfer failed ... KVTransferError: Failed to get kvcache from prefill instance, it might be dead — the single KV-transfer failure that begins the cascade (1 occurrence in the run).
• Decode scheduler MainThread py-spy dumps (00:48:01–00:48:32) are all stuck at event_loop_pp_disagg_decode (scheduler_pp_mixin.py:437) → dispatch_event_loop (scheduler.py:4586) → run_event_loop (scheduler.py:1526). Verified scheduler_pp_mixin.py:437 is inside the PP send/launch section of the decode loop.
• 00:48:01 [PP0 TP1] Scheduler watchdog timeout (watchdog_timeout=300, soft=True); escalates to soft=False at 00:48:21; 00:48:26 SIGQUIT received ... It usually means one child failed.
• After teardown, decode /generate returns 500 Internal Server Error in bulk (00:49:33), producing KeyError: 'answer' in few_shot_gsm8k.py:123; the harness retry then hits ConnectionRefusedError [Errno 111] to 127.0.0.1:11200 (decode dead) via sglang_router/mini_lb.py:141.
• Config (from launch cmd): Qwen3-8B, prefill+decode each --tp-size 2 --pp-size 2, --attention-backend aiter, --disaggregation-transfer-backend mooncake, --disaggregation-ib-device rdma0; decode --base-gpu-id 4.
• Test assertion is accuracy > 0.70 (test line 293); it never reached the assertion — it errored on server unavailability.

Hypothesised Causes (with confidence)

• [MEDIUM] KV-cache transfer instability in the PD+PP mooncake path: the decode side fails to fetch KV from prefill (KVTransferError, "prefill instance might be dead"), and the decode PP decode event loop (scheduler_pp_mixin.py:437) then blocks long enough to trip the 300s watchdog and SIGQUIT the server.
  • Supporting: single KVTransferError at 00:41:36 directly precedes a ~6.5-min stall with every MainThread py-spy frame pinned in event_loop_pp_disagg_decode; watchdog + SIGQUIT follow; the failure only affects the PP disaggregation file, while the non-PP basic/mori/nixl files in the same job pass.
  • Disconfirming: root of the KVTransferError (why prefill couldn't serve the KV block) is not captured in this log — no explicit prefill-side crash/traceback was found, so "prefill dead" is the decode's inference, not directly observed. Symptom-level clustering only.
  • How to verify: re-run test_disaggregation_pp.py on linux-mi35x-gpu-8.fabric several times to measure the flake rate; enable prefill-side verbose logging / KV-transfer tracing to capture the prefill state at the moment of the failed transfer.
• [LOW] Intermittent hardware/RDMA transport hiccup on the 8-GPU mi35x fabric runner (mooncake over rdma0). Repeated UCX ERROR failed to get interface index for tw-eth* lines appear (later, in other test files) indicating a noisy RDMA environment.
  • Supporting: failure is intermittent across identical config; some runs pass with the same code.
  • Disconfirming: those UCX errors appear in the mori/nixl sections that still PASS, so they are not obviously fatal here.
  • How to verify: correlate failures with runner/host identity across the 10 failing runs.

In-flight Fix Check

• In-flight fix: ⚠️ candidates — two open PD+PP disaggregation PRs may be relevant but are not confirmed to target this flake:
  • #31869 [PD+PP] Honor PP consensus for bootstrap and prealloc
  • #31798 [PD+PP] Make decode queue consensus authoritative
  • Also potentially related: #19810 [PD] Add graceful RDMA shutdown for Mooncake/NIXL disaggregation connectors.
  • Searched: is:pr is:open disaggregation PP, KVTransfer decode, disaggregation watchdog. No PR explicitly names this test or the mi35x flake.

Suggested Triage Steps

• Re-run test_disaggregation_pp.py on linux-mi35x-gpu-8.fabric 5–10× at HEAD to quantify the flake rate (history suggests ~60%).
• Capture prefill-side logs at the moment of the KVTransferError (bootstrap_room 1965797220740909719) to confirm whether the prefill scheduler actually died or merely stalled — this decides whether the bug is in prefill KV send or decode KV receive.
• Check whether #31869 / #31798 touch the scheduler_pp_mixin.py decode consensus/prealloc path; if so, cherry-pick onto a branch and re-run this job to A/B the flake.
• If the transfer hang cannot be root-caused quickly, consider lowering/segmenting the GSM8K load or quarantining TestDisaggregationDecodePPAccuracy on mi35x until the PD+PP mooncake path is stabilized.

Status (factual)

• Persistence: Intermittent since at least 2026-07-17 12:25; 10 of the last 16 completed scheduled runs failed on this test file (last green 2026-07-22 12:32).
• Scope: 1 test file (test_disaggregation_pp.py, TestDisaggregationDecodePPAccuracy.test_gsm8k) in the mi35x 8-GPU disaggregation job of pr-test-amd.yml; the other 3 disaggregation files pass. No other backend affected.
• Blocked work: PD+PP disaggregation accuracy signal on AMD mi35x is unreliable.
• In-flight fix: ⚠️ candidate PRs #31869, #31798 — not confirmed; verify before assuming coverage.

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-07-25 13:12 UTC)

[amd-bot]

pr-test-amd.yml — Cross-Run Pattern Summary (4 runs in lookback window)

Last scanned (UTC): 2026-07-25 13:15 UTC

Agent assessment

Persistent clusters (top by job count)

1 · MI325 1-GPU diffusion runner-lost / partition-0 hang — jobs mm-gen-1gpu (…,0) and (…,1). Step-9 stuck in_progress, runner "lost communication", killed ~80 min. Failed 3/3 signal runs; never-passed for ≥8 days (part 0 since 2026-07-08). Top hypothesis [LOW] host resource exhaustion on the heaviest partitions; disconfirming: no host logs survive (all blobs BlobNotFound), so OOM vs runner-agent death is unproven. In-flight fix ❌ none.

2 · MI325 2-GPU diffusion — FLUX.2 modelopt-FP8 HIPBLAS_STATUS_NOT_SUPPORTED — job mm-gen-2gpu (…,0), test_server_2_gpu.py::…[flux2_modelopt_fp8_tp2_t2i]. Failed 3/3 signal runs (one masked by a runner-loss instance); persistent ≥8 days. [MEDIUM] hipBLAS lacks an FP8 GEMM kernel for this model on gfx942; disconfirming: historical logs GC'd, may be never-supported rather than a regression. In-flight fix ✅ #28889.

3 · MI325 1-GPU shard 3 — update_weights_from_disk transformer shape mismatch → HTTP 400 — job mm-gen-1gpu (…,3), test_update_weights_from_disk.py (4 Qwen-Image fails + 4 FLUX.2 setup errors). Never-passed ≥8 days, deterministic. [HIGH] diffusers→sglang checkpoint layout not reconciled, assert param_data.shape == loaded_weight.shape at linear.py:410; root cause tracked in #31924. In-flight fix ✅ #31925 (skip on AMD, open since 2026-07-21).

4 · MI35x 8-GPU PD+PP disaggregation mooncake KV-transfer timeout — job disaggregation-amd, test_disaggregation_pp.py::…test_gsm8k. Intermittent: 10/16 window runs (failed 3/3 signal runs here); last green 2026-07-22. [MEDIUM] KV-transfer stall trips 300s decode-PP watchdog → SIGQUIT; disconfirming: prefill-side crash never directly observed (decode's inference only). In-flight fix ⚠️ candidates #31869, #31798, NIC-isolation #29127 — none confirmed. (A one-off HF-Hub 429 also failed test_disaggregation_basic.py in run 30056142455.)

Regression candidates (latest-run-only)

The bucket flags pr-gate and check-changes as latest-run regressions, but both are a GitHub Actions internal platform error, not a code regression: check-changes and pr-gate failed with 0 steps / no runner / BlobNotFound in an identical 44s window ([FACT]), after passing 15/15 prior runs. HEAD a678a420 runs no code in these gate jobs. No genuine code regression candidate exists in this window — the run 30093299057 spike jobs are all single-occurrence infra/flake (below). Triage: re-run the failed jobs; a green re-run confirms transient platform error.

Flakes / one-off (run 30093299057 only)

• stage-b-1gpu-small (…,8) — GPU Hang during sgl-kernel build → exit 134. 1/10 runs; passed 9 prior. Re-run candidate (hardware hang, no test ran).
• stage-b-2gpu-large (…,1) — test_unified_radix_cache_kl_full.py GPU Hang node-3 → exit 134, both attempts hung at different methods. 1/11 runs. Re-run; quarantine only if repeatable.
• stage-c-mi35x (…,1) — test_kimi_k25_mxfp4_bcg_mi35x.py::test_bcg_gsm8k, TP3 stalled → monitored_barrier 480s timeout → exit -9. First fail after 3 green (test added 2026-07-22). Re-run 2-3×; likely load-time flake, not a code regression.

Suggested next actions

• Re-run 30157771919's failed gate jobs to recover AMD signal for a678a420 — this run currently has none (cluster on regression candidates).
• Chase in-flight fixes for the two clusters that have them: #31925 (cluster 3, greens shard 3) and #28889 (cluster 2, FLUX.2 FP8).
• Cluster 1 (highest ROI investigation): add an in-step free -h/rocm-smi heartbeat + py-spy to the 1-GPU diffusion step so the next runner-drop captures host state — every historical log is BlobNotFound, blocking root-cause.
• Cluster 4: verify whether #31869/#31798 touch scheduler_pp_mixin.py; A/B on the MI35x runner. Do not bisect — intermittent across many SHAs.
• Re-run the three run-30093299057 one-offs (GPU hangs, Kimi load timeout) before any bisect; treat as infra/flake unless they recur.

---

Runs analysed

Run	Started (UTC)	Failed jobs
30056142455	2026-07-24 01:57	5
30093299057	2026-07-24 12:30	8
30136388246	2026-07-25 00:29	5
30157771919	2026-07-25 12:22	2

Persistent failures (in every run)

(none)

Latest-only failures (potential regressions)

• call-pr-test-amd-extra / call-gate / pr-gate — in 1/4 runs (7771919)
• check-changes — in 1/4 runs (7771919)

Flaky / intermittent failures

• multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 0) — in 3/4 runs (6142455, 3299057, 6388246)
• multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 1) — in 3/4 runs (6142455, 3299057, 6388246)
• multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 3) — in 3/4 runs (6142455, 3299057, 6388246)
• multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 0) — in 3/4 runs (6142455, 3299057, 6388246)
• stage-b-test-1-gpu-small-amd (linux-mi325-1gpu-sglang, 8) — in 1/4 runs (3299057)
• stage-b-test-2-gpu-large-amd (linux-mi325-2gpu-sglang, 1) — in 1/4 runs (3299057)
• stage-b-test-large-8-gpu-mi35x-disaggregation-amd (linux-mi35x-gpu-8.fabric) — in 3/4 runs (6142455, 3299057, 6388246)
• stage-c-test-large-8-gpu-amd-mi35x (linux-mi35x-gpu-8, 1) — in 1/4 runs (3299057)

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-07-25 13:15 UTC)

[amd-bot]

pr-test-amd.yml — Per-Job Analysis (continued 4/5)

multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 3) — failed step(s): Run diffusion server tests (1-GPU)

Commit Info

• sglang: 1ef2b85ef7d4 (chore: bump docs install version to 0.5.16 #32347)
• aiter: 9127c94a1 — v0.1.17.dev110+g9127c94a1 (source: dev image preserved; Dockerfile default, no override)

Failed Tests

Test file: python/sglang/multimodal_gen/test/single_test_file/test_update_weights_from_disk.py — 4 failed, 2 passed, 4 errors (Log)

Test File	Test Function	Error	Log
test_update_weights_from_disk.py	TestUpdateWeightsFromDisk::test_update_weights_from_disk_default[Qwen-Image]	AssertionError: assert 400 == 200 (server returned HTTP 400)	Log
test_update_weights_from_disk.py	TestUpdateWeightsFromDisk::test_update_weights_specific_modules[Qwen-Image]	Update failed: {'success': False, 'message': "Failed to update module 'transformer': ..."} → 400 == 200	Log
test_update_weights_from_disk.py	TestUpdateWeightsFromDisk::test_corrupted_weights_rollback[Qwen-Image]	assert (400 == 200)	Log
test_update_weights_from_disk.py	TestUpdateWeightsFromDiskWithOffload::test_update_weights_with_offload_enabled[Qwen-Image]	AssertionError: Expected 200, got 400	Log
test_update_weights_from_disk.py	TestUpdateWeightsFromDisk::test_update_weights_from_disk_default[FLUX.2-klein-base-4B] (setup ERROR)	AssertionError: No weights dir for transformer/vae in <snapshot> (fixture diffusion_server_no_offload)	Log
test_update_weights_from_disk.py	TestUpdateWeightsFromDisk::test_update_weights_specific_modules[FLUX.2-klein-base-4B] (setup ERROR)	same fixture assertion (/tmp/sglang_perturbed_vae_*)	Log
test_update_weights_from_disk.py	TestUpdateWeightsFromDisk::test_update_weights_rejects_invalid_requests[FLUX.2-klein-base-4B] (setup ERROR)	same fixture assertion	Log
test_update_weights_from_disk.py	TestUpdateWeightsFromDisk::test_corrupted_weights_rollback[FLUX.2-klein-base-4B] (setup ERROR)	same fixture assertion	Log

Failure Summary

All 8 failures/errors are in the single standalone test file test_update_weights_from_disk.py. For Qwen-Image, POST /update_weights_from_disk returns HTTP 400 while the tests expect 200: the server-side cause is a bare AssertionError in the diffusion weight-loader when updating the transformer module. For FLUX.2-klein-base-4B, the four items error out earlier in the diffusion_server_no_offload fixture setup (_compute_checksum_from_disk cannot find a transformer/vae weights dir). 2 unrelated items in the shard passed.

Regression Status

Never-passed (no green run found) — this test file has failed with the identical signature in every comparable scheduled run in the queryable window.

Recent history of test_update_weights_from_disk.py in job multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 3) (most recent comparable completed scheduled runs: branch=main, event=schedule, completed only):

Date	Run	Job (shard 3)	Test File Status	Failed Functions	Error
Jul 25	30136388246	shard 3	❌ Failed	4 Qwen fail + 4 FLUX setup err	shape mismatch → 400
Jul 24	30093299057	shard 3	❌ Failed	same	same
Jul 24	30056142455	shard 3	❌ Failed	same	same
Jul 23	30007380837	shard 3	❌ Failed	same	same
Jul 23	29969145884	shard 3	❌ Failed	same	same
Jul 22	29920027108	shard 3	❌ Failed	same	same
Jul 22	29880220845	shard 3	❌ Failed	same	same
Jul 21	29830494225	shard 3	❌ Failed	same	same
Jul 21	29790347494	shard 3	❌ Failed	same	same
Jul 17	29544840981	shard 3	❌ Failed	5 fail + 4 err (variant)	same shape mismatch + no-weights-dir

No passing run exists in the lookback range 2026-07-17 → 2026-07-25 (verified at test-file level in shard-3 logs). NVIDIA CI does not run this file (standalone test excluded from the diffusion partition plan), so AMD is the only lane exercising it. Not an aiter-override run (AITER_COMMIT_OVERRIDE empty).

Failure Cluster

Diffusion update_weights_from_disk transformer reload shape mismatch → HTTP 400 (+ FLUX perturbed-dir fixture setup error).

Facts (verified observations)

• Server-side stack trace (verbatim, e.g. log ~line 5561):

  weights_updater.py:434 in _apply_weights → _load_weights_into_module
  weights_updater.py:203 → load_weights_into_model
  weights_updater.py:295 → weight_loader(param, loaded_weight.to(param.dtype))
  linear.py:410 in weight_loader → assert param_data.shape == loaded_weight.shape
  AssertionError
  

  Confirmed in source at python/sglang/multimodal_gen/runtime/layers/linear.py:410 and .../runtime/post_training/weights_updater.py:295.
• Server logs Failed to update module 'transformer': . All modules rolled back to original weights. (empty message = bare AssertionError), then POST /update_weights_from_disk HTTP/1.1 400 Bad Request (log lines 5974–5975, 7125, 8279).
• FLUX setup errors originate in fixture diffusion_server_no_offload (test_update_weights_from_disk.py:410) calling _compute_checksum_from_disk (:220), which asserts os.path.exists(<local_path>/transformer) — the perturbed VAE dir /tmp/sglang_perturbed_vae_* has no transformer subdir.
• Many benign Skipping weight update for FSDPQwen3ForCausalLM: parameter '…' not found after mapping warnings precede the fault (name-mapping mismatches; not the failing assertion itself).
• Reproducibility: identical signature in ≥10 consecutive scheduled runs (7/17→7/25); same in shard-3 baseline logs each day.

Hypothesised Causes

• [HIGH] Diffusion update_weights_from_disk transformer reload does not reconcile the diffusers-format checkpoint layout with sglang's transformer parameter shapes, so a weight trips assert param_data.shape == loaded_weight.shape in linear.py:410, the update aborts, all modules roll back, and the endpoint returns HTTP 400.
  • Supporting: stack trace lands exactly on linear.py:410; server message Failed to update module 'transformer' with empty detail matches a bare AssertionError; documented as the confirmed root cause in tracking issue #31924 (Qwen-Image and FLUX.2-klein-base-4B); explicitly distinguished from the InferenceMode-500 issue fixed by #31263.
  • Disconfirming: this run's FLUX items fail earlier at fixture setup (perturbed-dir missing transformer) rather than reaching the 400 path, so #31924's "both models hit the shape mismatch" is only directly observed for Qwen-Image here; the FLUX errors may be a separate perturbed-checkpoint-construction gap in the same file.
  • How to verify: on AMD MI325 container, run only test_update_weights_specific_modules[Qwen-Image], add traceback.print_exc() at the _apply_weights except site to print the param_data.shape vs loaded_weight.shape values for the failing transformer param.
• [MEDIUM] FLUX setup errors are a distinct test-fixture defect: the perturbed model directory (/tmp/sglang_perturbed_vae_*) only contains a perturbed vae, but the fixture requests a transformer checksum, so _compute_checksum_from_disk asserts before the server is even exercised.
  • Supporting: error is No weights dir for transformer in /tmp/sglang_perturbed_vae_...; path name implies a VAE-only perturbation dir; failure is at :410/:220 (setup), not at the request assertion (:432/:454/...).
  • Disconfirming: could also be a symptom of the same reload issue if the fixture reuses server-produced dirs; not fully separable from the log alone.
  • How to verify: inspect fixture construction of perturbed_vae_model_dir in the test file; check whether it is expected to contain a transformer subdir.

In-flight Fix Check

• In-flight fix: ✅ #31925 "[AMD] Skip test_update_weights_from_disk on ROCm pending reload fix" — open since 2026-07-21, last updated 2026-07-23. Skips this file on the AMD lane (aligning with NVIDIA, which already doesn't run it) pending the real fix. Root cause tracked in #31924 (open). Chase reviewers on #31925; do NOT open a duplicate. (Searched: search/issues q=repo:sgl-project/sglang is:pr is:open update_weights_from_disk multimodal.)

Suggested Triage Steps

• Land the in-flight skip #31925 to green the AMD lane, and keep #31924 open for the underlying reload fix.
• For the real fix: reproduce test_update_weights_specific_modules[Qwen-Image] on AMD MI325 and dump the mismatched transformer param shape vs the diffusers-checkpoint tensor shape at linear.py:410; reconcile the diffusers→sglang name/shape mapping in weights_updater.py (the weight_loader path around :295).
• Separately confirm whether the FLUX No weights dir for transformer fixture setup error is a test-side perturbed-dir construction bug (expects a transformer subdir that the VAE-only perturbation never creates) vs the same reload defect.
• If the reload fix is deferred, the skip in #31925 is the interim path; re-enable once #31924 is resolved.

Status

• Persistence: Failing every completed scheduled run in the queryable window since ≥2026-07-17 (never-passed on AMD; identical signature).
• Scope: multimodal-gen-test-1-gpu-amd shard 3 on pr-test-amd.yml; single test file test_update_weights_from_disk.py (4 Qwen failures + 4 FLUX setup errors). NVIDIA does not run this file.
• Blocked work: AMD diffusion update_weights_from_disk CI signal for this shard.
• In-flight fix: ✅ #31925 (skip on AMD, open since 2026-07-21) tracking root-cause issue #31924.

multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 0) — failed step(s): N/A

Commit Info

• sglang: 1ef2b85ef
• aiter: N/A — the job log blob is unavailable (BlobNotFound), so no [CI-AITER-CHECK] markers can be read.

Failed Tests

Test File	Test Function	Error	Log
N/A (non-test infra failure)	N/A	The self-hosted runner lost communication with the server	Job

This is not a test-level failure. The job died mid-way through step 9 ("Run diffusion server tests (1-GPU)"). Because the runner was severed before uploading logs, the specific diffusion test file/function that was executing when the runner dropped cannot be identified — the log blob returns BlobNotFound.

Failure Summary

Job multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 0) failed during step 9, "Run diffusion server tests (1-GPU)" (suite 1-gpu, partition 0 of 4). The step never completed — it remains in_progress — and the only failure annotation on the check-run is "The self-hosted runner lost communication with the server." No test logs were uploaded (BlobNotFound), and step 10 (artifact upload) never ran. The job ran 00:47:58 → 02:04:01 UTC (~76 min), below the 90-min step timeout, so this is a mid-run runner drop, not a step timeout.

Regression Status

Known recurring failure / Never-passed in the query window — this exact job (shard 0) has failed with the identical "runner lost communication" signature and step 9 stuck in_progress in every one of the last 16 comparable completed scheduled runs. No green run exists in the lookback window (back to 2026-07-17).

Recent history of step 9 "Run diffusion server tests (1-GPU)" in job multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 0) (most recent 16 comparable completed scheduled runs: branch=main, event=schedule, completed only). Signature verified via check-run annotation on this run + 5 historical runs (all identical); the remaining rows share the same step-9-in_progress + job-failure shape:

Date (UTC)	Run / Job	Step 9 Status	Failure Signature
Jul 25 00:29	job	❌ in_progress (never completed)	runner lost communication ✅verified
Jul 24 12:29	job	❌ in_progress	runner lost communication ✅verified
Jul 24 00:25	job	❌ in_progress	runner lost communication
Jul 23 12:32	job	❌ in_progress	runner lost communication
Jul 23 00:27	job	❌ in_progress	runner lost communication ✅verified
Jul 22 12:32	job	❌ in_progress	runner lost communication
Jul 22 00:26	job	❌ in_progress	runner lost communication ✅verified
Jul 21 12:32	job	❌ in_progress	runner lost communication
Jul 21 00:27	job	❌ in_progress	runner lost communication
Jul 20 12:42	job	❌ in_progress	runner lost communication
Jul 20 00:28	job	❌ in_progress	runner lost communication ✅verified
Jul 19 12:17	job	❌ in_progress	runner lost communication
Jul 19 00:27	job	❌ in_progress	runner lost communication
Jul 18 12:17	job	❌ in_progress	runner lost communication ✅verified
Jul 18 00:25	job	❌ in_progress	runner lost communication
Jul 17 12:25	job	❌ in_progress	runner lost communication

First observed failure in window: 2026-07-17 12:25 UTC (likely earlier — window edge). Last known passing run: none found in the 16-run window.

Note the sibling shard 3 (job 89621024259) is a different failure — its step 9 completed with exit code 1 and uploaded artifacts (a real test failure), while shard 2 passed. Only shards 0 (and often 1, whose step 9 also shows in_progress) exhibit this runner-drop signature.

Failure Cluster

Self-hosted MI325 runner loses communication during 1-GPU diffusion server tests (partition 0)

Facts (verified observations, no interpretation)

• Check-run annotation on this job (verbatim): "The self-hosted runner lost communication with the server. Verify the machine is running and has a healthy network connection. Anything in your workflow that terminates the runner process, starves it for CPU/Memory, or blocks its network access can cause this error." (annotation_level: failure).
• Step 9 "Run diffusion server tests (1-GPU)" status is in_progress and never reached a conclusion; steps 10 ("Upload diffusion failure artifacts") and 20/21 (post-job) never ran.
• Job log blob is unavailable: GET .../jobs/89621024236/logs → BlobNotFound. No stack trace or test output is recoverable.
• Runner name: linux-mi325-1gpu-sglang-ng8rq-runner-thdj5, label linux-mi325-1gpu-sglang.
• Timing: started 2026-07-25T00:47:58Z, completed 2026-07-25T02:04:01Z (~76 min); step-9 timeout is 90 min — not exceeded.
• This job runs sglang/multimodal_gen/test/run_suite.py --suite 1-gpu --partition-id 0 --total-partitions 4 -k "not flux_2" (models Wan2.1-T2V-1.3B, HunyuanVideo, Qwen-Image, FLUX.1). Partitions are assigned dynamically by estimated runtime, so the exact case in partition 0 is not statically determinable.
• The identical annotation string was directly verified on 6 jobs across the window (89621024236, 89481337911, 89087124781, 88799241374, 88251525270, 88079191158); all 16 window runs share the step-9-in_progress + job-failure shape.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [LOW] Runner resource exhaustion (OOM / VRAM starvation / process crash) during a heavy 1-GPU diffusion model run in partition 0 terminates or hangs the runner agent, tripping the "lost communication" watchdog.
  • Supporting: the annotation itself lists "starves it for CPU/Memory, or blocks its network access" as causes; diffusion e2e loads large models (FLUX.1-dev / HunyuanVideo) that are memory-heavy; the failure is reproducible on the same partition (0) every run, suggesting a workload-linked trigger rather than random infra noise.
  • Disconfirming: no logs exist to confirm an OOM/crash; a purely workload-driven OOM would more likely fail the test step (exit code) as shard 3 does, rather than silently drop the runner. The consistency across 8 days on the same runner pool could equally be a hardware/agent-health issue on specific MI325 nodes.
  • How to verify: inspect the host dmesg/journal on runner linux-mi325-1gpu-sglang-* for OOM-killer or GPU Xid/HIP faults around the failure timestamps; add heartbeat/rocm-smi logging inside step 9; reduce partition-0 batch size or SGLANG_TEST_NUM_INFERENCE_STEPS and re-run.
• [LOW] Infrastructure/agent health issue on the MI325 1-GPU runner pool (network partition or agent process death) independent of the workload.
  • Supporting: "lost communication with the server" is GitHub's generic runner-agent-death signal; step 9 is simply the longest-running step, so a drop is most likely to land there.
  • Disconfirming: the failure hits the same partition (0) consistently while partition 2 passes — a purely random infra fault would spread across partitions/jobs more evenly, which weakens the "workload-independent" framing.
  • How to verify: correlate failures with specific runner hostnames; check whether shard 1 (also frequently in_progress) and shard 0 land on the same physical nodes.
• No hypothesis reaches MEDIUM confidence: with no logs, both causes rest on the annotation text and the partition-consistency pattern only. Root cause needs host-side runner logs, which are outside the queryable GitHub API surface.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs with keywords multimodal-gen runner communication (0 hits), diffusion runner AMD (14 hits, none addressing the runner-drop/hang on shard 0 — e.g. #31483, #27587 are unrelated diffusion CI enablement/config), and multimodal_gen mi325 hang (2 unrelated kernel PRs). No PR targets this runner-communication failure.

Suggested Triage Steps

• Pull host-side logs (dmesg, systemd journal, GitHub runner _diag/ logs) for runner linux-mi325-1gpu-sglang-ng8rq-runner-thdj5 and its pool around 2026-07-25 01:00–02:04 UTC to see whether the agent was OOM-killed, hit a GPU HIP/Xid fault, or lost network.
• Add an in-step heartbeat (periodic free -h + rocm-smi echo) to step 9 so the last-logged state before the drop is captured even when the runner dies — this is the single highest-ROI step given logs are currently BlobNotFound.
• Determine which diffusion case lands in partition 0 (dump the partition plan from run_suite.py --suite 1-gpu --total-partitions 4); if it is a memory-heavy model (FLUX.1-dev / HunyuanVideo), try lowering its resource footprint or moving it off partition 0 and re-run to see if the drop follows the workload.
• Since shard 2 passes while shards 0/1 drop, check whether shards 0/1 are being scheduled onto specific unhealthy MI325 nodes; if so, drain/replace those runners.

Status (factual, no priority assignment)

• Persistence: Failing in every completed scheduled run for ≥8 days (since at least 2026-07-17 12:25 UTC); 16/16 runs in the query window, no green run found.
• Scope: Isolated to multimodal-gen-test-1-gpu-amd shard 0 (and frequently shard 1) on the linux-mi325-1gpu-sglang runner pool; shard 2 passes and shard 3 fails with a different (real test exit-1) signature.
• Blocked work: Blocks 1-GPU diffusion/multimodal-gen CI signal for partition 0 on AMD MI325 — that partition's diffusion cases have had no green coverage for over a week.
• In-flight fix: ❌ none found — needs new investigation (host-side runner logs) or a new PR.

multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 0) — failed step(s): N/A

Commit Info

• sglang: 1ef2b85e (chore: bump docs install version to 0.5.16)
• aiter: v0.1.17.dev110+g9127c94a1 (source: Dockerfile default AITER_COMMIT=9127c94a, dev image preserved). ⚠️ Best-effort proxy — the current run's own log/[CI-AITER-CHECK]markers are unrecoverable (blob expired); value read from the sister part-0 run [89368158479](https://github.com/sgl-project/sglang/actions/runs/30056142455/job/89368158479) which used the samediffusion` install path.

Failed Tests

Test File	Test Function	Error	Log
N/A (non-test failure)	N/A	The self-hosted runner lost communication with the server	Job

This is not a test failure. No specific test file/function can be pinned: the runner died mid-suite before writing any pytest summary, the job log blob returns BlobNotFound (expired/never-flushed), and the Upload diffusion failure artifacts step never ran (0 artifacts on the run).

Failure Summary

The Run diffusion server tests (2-GPU) step (step 9) started at 00:48:13 UTC and ran for 1:37:05 before the self-hosted MI325 runner lost communication with the GitHub Actions service, killing the job at 02:25:18 UTC (conclusion: failure, step 9 left in_progress with no completion, steps 10 & 20 never started). The GitHub check-run annotation is verbatim: "The self-hosted runner lost communication with the server … Anything in your workflow that terminates the runner process, starves it for CPU/Memory, or blocks its network access can cause this error." The step's own timeout-minutes: 150 was not reached (died at ~97 min), so this is a runner/host loss, not a step timeout.

Regression Status

Known recurring infrastructure failure — this is the dominant failure mode for this job, not a new regression.

Recent history of multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 0) — most recent 15 comparable completed scheduled runs (same branch main, event=schedule, completed only). Classification is by check-run annotation since no per-run test log survives for the runner-lost runs:

Date (UTC)	Run	Part-0 Job	Failure Mode	Detail
07-25 00:29	run	job	🔌 Runner lost comms	died ~97 min into step 9 (this run)
07-24 12:29	run	job	🔌 Runner lost comms	log expired
07-24 00:25	run	job	❌ Test failure (exit 1)	test_server_2_gpu.py::…[flux2_modelopt_fp8_tp2_t2i] FAILED + [joy_echo_t2v_2gpu] ERROR (HF download)
07-23 12:32	run	job	🔌 Runner lost comms	log expired
07-23 00:27	run	job	🔌 Runner lost comms	log expired
07-22 12:32	run	job	🔌 Runner lost comms	log expired
07-22 00:26	run	job	🔌 Runner lost comms	log expired
07-21 12:32	run	job	🔌 Runner lost comms	log expired
07-21 00:27	run	job	🔌 Runner lost comms	died ~1:46 in
07-20 12:42	run	job	🔌 Runner lost comms	died ~1:49 in
07-20 00:28	run	job	🔌 Runner lost comms	died ~1:43 in
07-19 12:17	run	job	❔ No annotation	ran ~91 min, cause unknown
07-19 00:27	run	job	🔌 Runner lost comms	died ~1:38 in
07-18 12:17	run	job	🔌 Runner lost comms	died early (~10 min)
07-18 00:25	run	job	❌ Test failure (exit 1)	Process completed with exit code 1

Verdict: 12/15 runs = "runner lost communication" (this run's exact mode), 2/15 = genuine test failures, 1/15 unknown. No green run for this part-0 job in the queryable 15-run window. The part-0 job has been red every scheduled run since at least 07-18 00:25 UTC.

Failure Cluster

Self-hosted MI325 runner lost communication during the long-running 2-GPU diffusion suite (symptom-level grouping; infra/host loss, not a pinned test).

Facts (verified observations)

• Check-run annotation for job 89621024271: "The self-hosted runner lost communication with the server…" (annotation_level: failure) — FACT.
• Step 9 Run diffusion server tests (2-GPU): status: in_progress, conclusion: null, started 00:48:13Z, no completed_at; steps 10/20 pending. Job conclusion: failure, completed 02:25:18Z → step ran 1:37:05 before host loss — FACT.
• Step 9 timeout is timeout-minutes: 150; the runner died at ~97 min, so the step timeout was not hit — FACT (workflow pr-test-amd.yml:890).
• Current job log blob = BlobNotFound; run has 0 artifacts; failure-artifact upload step never executed — FACT (no test-level evidence recoverable for this run).
• 12 of the last 15 scheduled part-0 runs carry the identical "runner lost communication" annotation; runner-loss clusters at ~1h38m–1h49m into the job across those runs — FACT.
• When the suite does complete (run 30056142455) it takes 8467.36s (2:21:07) — only ~9 min under the 150-min step budget — FACT.

Hypothesised Causes (with confidence)

• [MEDIUM] Host/infra loss under sustained 2-GPU diffusion load — the runner process is terminated/starved (CPU/memory), or the GPU driver hangs and takes the runner down.
  • Supporting: the annotation itself names process termination / CPU-memory starvation / network block as causes; 12 runs die with the identical annotation at a consistent ~90–110 min point deep into the heavy suite; the completing run runs 2:21:07 (very memory/GPU intensive, near budget).
  • Disconfirming: no host-side metrics (CPU/mem/GPU/dmesg) are available and no runner-lost run produced a log, so resource exhaustion vs. GPU-driver-hang vs. transient network loss cannot be distinguished; one runner-lost run died at only ~10 min (29644059344), which doesn't fit a "late-load" story and suggests at least some losses are flaky infra.
  • How to verify: on the MI325 2-gpu runner, capture dmesg/rocm-smi/host memory during the run; check whether a specific model stage (suite deep, ~90 min mark) reproducibly precedes the loss; inspect runner agent logs for the disconnect timestamp.
• [LOW] A specific diffusion model/case in the part-0 slice crashes/hangs the GPU, cascading into runner loss.
  • Supporting: runner-loss timing is fairly consistent (~1h38m–1h49m), hinting at a specific point in the suite.
  • Disconfirming: no surviving log for any runner-lost run to identify the offending case; the one completing run failed instead on flux2_modelopt_fp8_tp2_t2i (assertion) and a joy_echo HF-download error — different symptoms, so no single case is implicated.
  • How to verify: run the 2-gpu part-0 slice case-by-case on the runner with rocm-smi monitoring to find any case that reproducibly wedges the GPU.
• [LOW] sglang commit-window regression. Disconfirming: this is a runner-comms/infra failure, not a code assertion; the pattern predates 1ef2b85e (docs-only) and spans ≥7 days of unrelated head SHAs. No sglang commit is implicated for the runner-lost mode.

No hypothesis above MEDIUM. This is primarily an infrastructure/runner-stability issue; without host metrics or a surviving runner-lost log, root cause cannot be pinned.

In-flight Fix Check

• In-flight fix: ❌ none found — search repo:sgl-project/sglang is:pr is:open "runner lost communication" → 0 results; is:open diffusion runner → 28 PRs, none targeting runner-loss/host-stability on the 2-GPU diffusion job (closest tangential: #28584 "WIP ci: test AMD PR workflow on new MI300 runners", not a fix for this).

Suggested Triage Steps

• Instrument the linux-mi325-2gpu-sglang runner: capture dmesg, rocm-smi --showmeminfo vram, and host free -h continuously during a scheduled run, and correlate the disconnect timestamp with the runner agent's _diag logs to distinguish OOM-kill vs. GPU hang vs. network loss.
• Since runner-lost runs never upload the failure artifacts (step 10 runs only if: always() but the runner is already dead), consider streaming the pytest suite progress to an external sink / per-case heartbeat so the offending case at the ~90-min mark is identifiable even when the runner dies.
• Bisect the 2-gpu part-0 case list on the runner (run each case standalone with GPU monitoring) to check whether one model reproducibly wedges the GPU near the observed death window.
• Note the suite completing at 2:21:07 is only ~9 min under the 150-min step budget — even absent runner loss this job is one slowdown away from a step timeout; consider re-partitioning/trimming the 2-gpu suite regardless of the runner-comms fix.

Status

• Persistence: Part-0 job red every scheduled run in the queryable window (since ≥ 2026-07-18 00:25 UTC, ~7 days); the specific "runner lost communication" mode in 12 of the last 15 runs including this one.
• Scope: multimodal-gen-test-2-gpu-amd part 0 on linux-mi325-2gpu-sglang; the runner-loss pattern is host/infra-level and may affect sibling 2-gpu diffusion shards (parts 1–2) similarly — not verified here.
• Blocked work: No usable 2-GPU diffusion CI signal on AMD for part 0 — genuine test regressions (like the flux2_modelopt_fp8_tp2_t2i failure seen on 07-24) are masked by the recurring runner loss.
• In-flight fix: ❌ none found — needs infra/runner-stability investigation (new issue/PR).

multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 1) — failed step(s): N/A

Commit Info

• sglang: 1ef2b85ef — chore: bump docs install version to 0.5.16 (#32347). Docs-only; unrelated to this failure.
• aiter: N/A — job log is unavailable (BlobNotFound; the runner died before uploading logs), so no [CI-AITER-CHECK] markers can be read.

Failed Tests

Test File	Test Function	Error	Log
N/A (non-test failure)	N/A	The self-hosted runner lost communication with the server during step 9 Run diffusion server tests (1-GPU)	Job (log BlobNotFound)

This is not a test-assertion failure. The runner process was lost mid-step, so no per-test result was ever recorded. The step runs python3 sglang/multimodal_gen/test/run_suite.py --suite 1-gpu --partition-id 1 --total-partitions 4 -k "not flux_2" (partition 1/4 of the T2V/T2I/I2V/LoRA 1-GPU diffusion suite) — the exact failing test within that partition cannot be pinned because logs were never uploaded.

Failure Summary

The job multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 1) failed because the self-hosted MI325 runner lost communication with GitHub during step 9 (Run diffusion server tests (1-GPU)). Step 9 started 01:27:09Z and never completed; the job was marked failure ~70 min later (02:37:28Z), with steps 9/10/20 left in in_progress/pending. GitHub attached the check-run annotation "The self-hosted runner lost communication with the server… Anything in your workflow that terminates the runner process, starves it for CPU/Memory, or blocks its network access can cause this error."

Regression Status

Known recurring infrastructure failure — same signature every completed scheduled run in the queryable window; Never-passed in the last 15 comparable runs (lookback 2026-07-17 → 2026-07-25). The identical annotation also appears in older runs back to 2026-07-15.

Recent history of step 9 Run diffusion server tests (1-GPU) in job multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 1) (most recent 15 comparable completed scheduled runs: branch=main, event=schedule, status=completed):

Date (UTC)	Run	Job	Test File Status	Failure	Signature
07-25 01:14	30136388246	job	❌ Killed (no result)	step 9 killed	runner lost communication
07-24 15:26	30093299057	job	❌ Killed	step 9 killed	runner lost communication
07-24 03:05	30056142455	job	❌ Killed	step 9 killed	runner lost communication
07-23 14:32	30007380837	job	❌ Killed	step 9 killed	runner lost communication
07-23 04:43	29969145884	job	❌ Killed	step 9 killed	runner lost communication
07-22 14:09	29920027108	job	❌ Killed	step 9 killed	runner lost communication
07-22 02:32	29880220845	job	❌ Killed	step 9 killed	runner lost communication
07-21 12:32	29830494225	job	❌ Killed	step 9 killed	(same job pattern)
07-21 00:27	29790347494	job	❌ Killed	step 9 killed	(same job pattern)
07-20 12:42	29743214297	job	❌ Killed	step 9 killed	runner lost communication
07-20 00:28	29709549202	job	❌ Killed	step 9 killed	(same job pattern)
07-19 12:17	29686709871	job	❌ Killed	step 9 killed	(same job pattern)
07-19 00:27	29666971143	job	❌ Killed	step 9 killed	(same job pattern)
07-18 12:17	29644059344	job	❌ Killed	step 9 killed	(same job pattern)
07-18 00:25	29623067212	job	❌ Killed	step 9 killed	runner lost communication

First observed (this signature) ≤ 2026-07-15; no passing run for this job/step in the query window → Never-passed. Verdict is based on step-9 kill + annotation, not the job's overall pass/fail. Today's run 30136388246 is still in_progress at the workflow level but this job is completed=failure.

Failure Cluster

Self-hosted MI325 1-GPU runner loses communication during diffusion server tests (runner death mid-step, no logs uploaded)

Facts (verified observations)

• Check-run annotation (verbatim): "The self-hosted runner lost communication with the server. Verify the machine is running and has a healthy network connection. Anything in your workflow that terminates the runner process, starves it for CPU/Memory, or blocks its network access can cause this error."
• Step timeline: steps 1–8 succeeded; step 9 (Run diffusion server tests (1-GPU)) started 2026-07-25T01:27:09Z, status=in_progress, conclusion=null (never completed); steps 10 & 20 left pending. Job conclusion=failure, completed 02:37:28Z (~70 min into step 9).
• Job log is unavailable: GET .../jobs/89621024244/logs → HTTP 404 BlobNotFound. Consistent with the runner dying before log upload; prevents test-level and aiter-version extraction.
• Step config: timeout-minutes: 90; runner never hit the timeout (died at ~70 min), so this is a runner loss, not a step timeout.
• Reproducibility: identical annotation confirmed on 8 sampled historical jobs (89481337945, 89368158583, 89206599976, 89087124739, 88922971082, 88799241355, 88354737052, 88021852955); this job failed in 15/15 comparable runs.
• Head commit 1ef2b85ef is a docs version bump — cannot influence GPU diffusion tests.

Hypothesised Causes (with confidence)

• [MEDIUM] Host resource starvation (CPU/RAM OOM) during the 1-GPU diffusion suite starves the runner agent, killing its heartbeat.
  • Supporting: GitHub's own annotation names "starves it for CPU/Memory" as a cause; the diffusion suite loads large T2V/T2I/I2V/LoRA models and runs inference; the step accumulates ~70 min before death (workload builds up), and the following step is a free -h memory diagnostic (author already suspects memory pressure).
  • Disconfirming: no log/free -h output survived to confirm an OOM; could equally be a GPU hang or a bad runner node rather than host memory.
  • How to verify: re-run with host memory monitoring (dmesg/OOM-killer check, free -h sampled during the step); inspect the diffusion-failures artifact if any future run uploads it.
• [LOW] Runner-node / infrastructure fault (network drop or the linux-mi325-1gpu-sglang node itself is unhealthy), independent of the test workload.
  • Supporting: "lost communication with the server" is a classic node/network symptom; failure is confined to one runner label.
  • Disconfirming: the failure is deterministic across 15 runs on the same step at similar elapsed time, which points more to the workload than to random network flakes.
  • How to verify: check whether the same physical MI325 node backs all these runs; try the partition on a different node.
• [LOW] GPU hang / driver reset triggered by a specific diffusion test in partition 1, wedging the container and the runner.
  • Supporting: diffusion inference on ROCm can hang the GPU; partition 1 specifically fails while other partitions may not.
  • Disconfirming: unproven without logs; other -1gpu partitions' status not yet compared here.
  • How to verify: compare partitions 2–4 of the same job; if only partition 1 dies, bisect its test list.

No hypothesis reaches HIGH — root cause needs a surviving log / memory trace, which this failure mode destroys.

In-flight Fix Check

• In-flight fix: ❌ none found specific to this signature. Searched open PRs for diffusion+runner+lost+communication (0 results) and multimodal-gen+amd+runner / mi325+diffusion+timeout (only broadly-related PRs, e.g. #29584 "ci(amd): fix three pre-existing MI325 CI failures" and #31483 — none targets this 1-GPU runner-death pattern). Needs a new investigation/PR.

Suggested Triage Steps

• Add in-step resource monitoring to step 9 (background free -h/rocm-smi/dmesg -w capture, or lower --total-partitions granularity) so the next failure leaves evidence, since the current failure mode uploads no logs.
• Confirm whether all these runs land on the same physical linux-mi325-1gpu-sglang node; if so, cordon/health-check that node to rule out a bad host.
• Compare partitions 2–4 of multimodal-gen-test-1-gpu-amd and the 2-GPU job on the same schedule — if only partition 1 dies, bisect partition 1's diffusion test list (run_suite.py --suite 1-gpu --partition-id 1) to find the model/test that wedges the runner.
• If host OOM is confirmed, reduce concurrent model loads or cap memory in the diffusion suite for the 1-GPU AMD path.

Status

• Persistence: Failing every completed scheduled run for ≥10 days (since ≤ 2026-07-15; Never-passed in the 2026-07-17→2026-07-25 window), same runner-lost-communication signature.
• Scope: Confirmed on this job multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 1); cross-partition/2-GPU scope not yet confirmed here.
• Blocked work: Removes all PR CI signal for the 1-GPU AMD diffusion (multimodal-gen) partition-1 suite — those tests are effectively untested on AMD.
• In-flight fix: ❌ none found — needs new investigation/PR.

call-pr-test-amd-extra / call-gate / pr-gate — failed step(s): N/A

Commit Info

• sglang: a678a420
• aiter: N/A — the failing jobs never provisioned a runner (check-changes runs on ubuntu-latest, not the AMD image), so no [CI-AITER-CHECK] markers were emitted.

Failed Tests

Test File	Test Function	Error	Log
N/A (non-test)	N/A	GitHub Actions has encountered an internal error when running your job.	job

This is not a test failure. The job call-pr-test-amd-extra / call-gate / pr-gate is a gate/aggregation job in the pr-test-amd-extra.yml reusable workflow. It failed at the GitHub Actions platform level before running any steps.

Failure Summary

The pr-gate gate job failed with a GitHub Actions internal platform error, recording 0 steps and no assigned runner (job window 12:22:47→12:23:31 UTC, 44 s). The same internal error simultaneously hit check-changes (89678557327), call-pr-test-amd-extra / pr-test-amd-extra-finish (89678649799), and pr-test-amd-finish (89678662770) — all in the identical 44 s window. Because check-changes never emitted its change-filter outputs, every downstream AMD test job was skipped.

Regression Status

Infrastructure issue (GitHub Actions internal error). Not a code regression.

Recent history of the check-changes and call-pr-test-amd-extra / call-gate / pr-gate jobs (most recent comparable completed scheduled runs: branch=main, event=schedule):

Date	Run	Job	Status	Steps	Note
Jul 25 12:19	30157771919	pr-gate / check-changes	❌ Failed	0	internal error, no runner
Jul 25 00:29	30136388246	pr-gate / check-changes	✅ Passed	8 / 7	normal
Jul 24 12:29	30093299057	pr-gate / check-changes	✅ Passed	8 / 7	normal
Jul 24 00:25	30056142455	pr-gate / check-changes	✅ Passed	8 / 7	normal
Jul 23 12:32	30007380837	pr-gate / check-changes	✅ Passed	8 / 7	normal
Jul 23 00:27	29969145884	pr-gate / check-changes	✅ Passed	8 / 7	normal

First observed failure: today's run only (30157771919, 2026-07-25 12:22 UTC). Last known passing: previous scheduled run 30136388246 (2026-07-25 00:29 UTC). The failure is isolated to this single run.

Failure Cluster

GitHub Actions internal platform error (gate job, 0 steps, no runner)

Facts (verified observations, no interpretation)

• All four failing jobs carry the identical check-run annotation: GitHub Actions has encountered an internal error when running your job. (verified via /check-runs/<id>/annotations for jobs 89678557327, 89678557370, 89678649799, 89678662770).
• The pr-gate job (this job) recorded num steps: 0 and runner_name: '' — it never provisioned a runner. Same for check-changes (normally 7 steps on ubuntu-latest).
• All four failures share the exact same timestamps: started 12:22:47Z, completed 12:23:31Z (44 s).
• The job log blob returns BlobNotFound for all these jobs — consistent with jobs that never executed steps.
• In the 5 prior scheduled runs, both check-changes (7 steps) and pr-gate (8 steps) completed with conclusion success.
• call-gate (the pr-gate.yml caller) is correctly skipped on schedule events (if: github.event_name != 'schedule', pr-test-amd.yml:123); this is expected, not the failure. The failing pr-gate is the one nested under call-pr-test-amd-extra.
• All 22 downstream AMD test jobs were skipped because check-changes did not emit its main_package/sgl_kernel/etc. outputs.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [FACT] GitHub Actions platform internal error caused the gate/aggregator jobs to fail before running any step.
  • Supporting: identical GitHub Actions has encountered an internal error when running your job. annotation on all four jobs; 0 steps; no runner assigned; identical 44 s window; check-changes runs on GitHub-hosted ubuntu-latest (not the AMD self-hosted image), so this is not an AMD runner issue.
  • Disconfirming: (none found) — no evidence of any code, test, or dependency involvement; the immediately-prior scheduled run on a different SHA passed these jobs normally.
  • How to verify: re-run the failed jobs (or the full workflow). A clean re-run passing check-changes/pr-gate confirms a transient platform error.
• [SPECULATION] Commit a678a420 involvement — effectively ruled out.
  • Supporting: it is the head SHA of the failing run.
  • Disconfirming: the job that failed (check-changes) does a bare checkout + dorny/paths-filter and never reached any step; the failure is a GitHub-side internal error, not a checkout or filter error. No code path in the commit is exercised.
  • How to verify: not actionable; re-run suffices.

In-flight Fix Check

In-flight fix: ❌ none needed — this is a transient GitHub Actions platform error, not a code defect. No PR search performed because there is no code path to fix. Action is a workflow re-run.

Suggested Triage Steps

• Re-run the failed jobs for run 30157771919 ("Re-run failed jobs"). A green re-run confirms the transient internal error.
• If the re-run reproduces the internal error at the same check-changes/gate stage, check the GitHub Actions status page for an ongoing incident and, if present, wait it out before re-running.
• No code investigation, bisect, or aiter analysis is warranted — the failure never entered sglang or AMD test code.

Status (factual, no priority assignment)

• Persistence: single occurrence — run 30157771919 (2026-07-25 12:22 UTC). Prior scheduled run 12 h earlier passed these gate jobs.
• Scope: 4 gate/aggregator jobs in one run (check-changes, pr-gate, pr-test-amd-extra-finish, pr-test-amd-finish); cascaded to skip all 22 downstream AMD test jobs, so this run produced no AMD test signal.
• Blocked work: this scheduled run yielded zero AMD test coverage; a re-run is required to obtain signal for SHA a678a420.
• In-flight fix: none needed — transient GitHub Actions platform error; resolve by re-running.

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-07-25 13:12 UTC)

[amd-bot]

nightly-amd-mi355x-disagg.yml — 3 failure(s)

Run started (UTC): 2026-07-25 08:41 UTC
Last scanned (UTC): 2026-07-25 13:11 UTC
Commits: sglang ebcb74a

Summary

Counts: 3 failures · 2 clusters · 0 carrying over (≥3 days) · gate miss recurs intermittently (3/13 runs), 1 infra cancel new today

#	Cluster	Job	Test File	Test Function	Status	Hypothesis (confidence)
1	GSM8K accuracy gate miss (DSV4-Flash-FP8 1P1D DP8EP8)	nightly-mi355x-benchmark / dsv4flash-fp8-1k1k-1p1d-dp8ep8	GSM8K gate (few_shot_gsm8k.py, 1319q/8-shot)	N/A (accuracy gate, not pytest)	Flaky 3/13 completed runs (Jul 12, 18, 25); last pass Jul 24 @0.914	[HIGH] 0.91 threshold too tight for a config that sits at ~0.905–0.917
2	GSM8K accuracy gate miss (DSV4-Flash-FP8 1P1D DP8EP8)	collect-results	N/A (rollup of the dsv4flash leg)	N/A	Rollup red solely because of row 1	[HIGH] inherits row 1; no failure of its own
3	Per-leg job cancellation during runner setup	nightly-mi355x-benchmark / dsv4pro-fp4-1k1k-1p1d	N/A (benchmark never ran)	N/A	1 occurrence; prior 5 runs passed (Jul 20–24)	[LOW] transient infra/runner cancel on mia1-p01-g20 before launch

Cluster: GSM8K accuracy gate miss (DSV4-Flash-FP8 1P1D DP8EP8)

• Affected: dsv4flash-fp8-1k1k-1p1d-dp8ep8 (GSM8K gate) and its rollup collect-results. collect-results has no test of its own — it fails whenever any matrix leg fails, and the only failing leg here is the dsv4flash gate. These are one underlying failure, not two.
• Shared facts: bench.log [gsm8k] accuracy=0.905 threshold=0.91 → failing before sweep; eval ran clean over all 1319 questions (0.1% invalid, all 200s). Prefill rc=137 / decode rc=143 are teardown signals after the gate, not a crash. No agg_*.json sweep results produced. [FACT]
• Top hypothesis: [HIGH] the 0.91 gate is tighter than this config's real accuracy band (0.905/0.906/0.914/0.917 across runs). Disconfirming: the two per-job analyses disagree on the trend — the leg analysis sees interleaved pass/fail (→ noise, 3/13 flaky), the collect-results analysis sees a 3-day monotonic slide 0.917→0.914→0.905 (→ possible small real FP8 drift). Neither has code-level evidence. [LOW] alt: an FP8/MoE/DSV4 numerical shift in the Jul 24→25 48-commit window.
• In-flight fix: ❌ none directly retunes the gate. ⚠️ Watch #31727 (DSV4-Flash FP8 fused-RMS scale metadata on gfx950) — could nudge DSV4 FP8 accuracy; not a confirmed fix for this 0.005 dip.
• Suggested triage: re-run dsv4flash-fp8-1k1k-1p1d-dp8ep8 3–5× at HEAD ebcb74ab; oscillation ≥0.91 ⇒ retune gate (raise num_questions or lower to ~0.89). If reruns stay <0.91, bisect 99b29bf1..ebcb74ab on FP8/MoE/DSV4 commits (skip CUDA-only ones).

Cluster: Per-leg job cancellation during runner setup

• Affected: dsv4pro-fp4-1k1k-1p1d — cancelled ~7s in (09:34:34→41Z); Checkout/Launch steps skipped, so the benchmark never ran. Not a benchmark or code regression.
• Shared facts: fail-fast:false + cancel-in-progress:false rule out auto sibling/newer-run cancellation; adjacent legs on the same shared runner mia1-p01-g20 succeeded before and after. ⚠️ Stale-artifact trap: this leg's artifact holds only the sibling -dp8ep8 bundle (GSM8K=0.932 logs are the sibling's, timestamped ~23s before this job started) — not evidence this leg ran. [FACT]
• Top hypothesis: [LOW] transient infra cancel (runner disconnect/drain) on host mia1-p01-g20, which is both the sole orchestrating runner and an explicitly SLURM_EXCLUDEd degraded node. Disconfirming: neighboring legs on the same runner passed; no auto-cancel path exists.
• In-flight fix: ❌ none found (searched open PRs mi355x runner).
• Suggested triage: pull _diag runner-agent logs for mia1-p01-g20-di-shared around 09:34:34–41Z; re-run only this leg (5 prior passes ⇒ likely green). Consider hardening the Upload results glob to the per-leg RESULT_FILENAME so a no-op leg stops uploading a sibling's bundle.

Suggested triage order

Rank	Triage step	Targets	Why this first
1	Re-run dsv4flash-fp8-1k1k-1p1d-dp8ep8 3–5× at HEAD, record accuracy spread	Cluster A (rows 1–2)	Largest cluster (blocks the rollup); cheaply separates gate-flakiness from real FP8 drift before any bisect
2	Re-run the dsv4pro-fp4-1k1k-1p1d leg; pull mia1-p01-g20 agent logs	Cluster B (row 3)	5 prior passes ⇒ almost certainly a one-off infra cancel; single rerun confirms
3	Track #31727 and, only if reruns stay <0.91, bisect 99b29bf1..ebcb74ab on FP8/DSV4 commits	Cluster A	Higher cost; only worth it if step 1 shows a persistent sub-0.91 result

---

Job	Failed Steps	Started
nightly-mi355x-benchmark (dsv4flash-fp8-1k1k-1p1d-dp8ep8, dsv4flash-fp8-mi355x-dp8ep8-sglang, sgl...	Launch MI355X 2N 1P1D benchmark	2026-07-25T08:41
nightly-mi355x-benchmark (dsv4pro-fp4-1k1k-1p1d, dsv4pro-fp4-mi355x-sglang, deepseek-ai/DeepSeek-...	N/A	2026-07-25T09:34
collect-results	N/A	2026-07-25T12:29

Per-Job Analysis

nightly-mi355x-benchmark (dsv4flash-fp8-1k1k-1p1d-dp8ep8, dsv4flash-fp8-mi355x-dp8ep8-sglang, sgl... — failed step(s): Launch MI355X 2N 1P1D benchmark

Commit Info

• sglang: ebcb74abd477afbcb940c8ba30d45f983597e38e
• aiter: N/A — this SLURM disagg benchmark workflow emits no [CI-AITER-CHECK] markers in its job log; aiter version is baked into the runner image and not reported here.

Failed Tests

Test File	Test Function	Error	Log
GSM8K accuracy gate (few_shot_gsm8k.py, num_questions=1319 shots=8) — config dsv4flash-fp8-1k1k-1p1d-dp8ep8	N/A (accuracy gate, not a pytest)	accuracy=0.905 threshold=0.91 → below threshold, failing before sweep	step 5 · artifact:bench.log

This is not a pytest failure — it is the pre-benchmark GSM8K accuracy gate in step 5 ("Launch MI355X 2N 1P1D benchmark").

Failure Summary

The DeepSeek-V4-Flash-FP8 1P1D DP8/EP8 disagg servers started successfully and served the full 1319-question GSM8K gate, scoring accuracy 0.905 against a 0.91 threshold. Because 0.905 < 0.91, the harness printed accuracy below threshold -- failing before sweep (artifact:bench.log, and job log lines 4131–4132) and aborted before the throughput sweep ran — which is why this config's artifact contains only a _logs.tar.gz and no agg_*.json result files, unlike every other config in the run. The prefill server exit rc=137 (artifact:server_exit_prefill_mia1-p02-g09) and decode rc=143 (artifact:server_exit_decode_mia1-p02-g29) are teardown SIGKILL/SIGTERM after the gate failure, not an OOM or crash.

Regression Status

Flaky test — marginal accuracy oscillating around a too-tight threshold. Same failure signature (accuracy≈0.905–0.906 < 0.91) recurs intermittently; it is NOT a code regression (each failing run has a different head_sha, and passing runs interleave).

Recent history of GSM8K accuracy gate (config dsv4flash-fp8-1k1k-1p1d-dp8ep8) in job nightly-mi355x-benchmark (most recent comparable completed scheduled runs: branch=main, event=schedule, completed only):

Date	Run	Job	Gate Status	Accuracy	Error
Jul 25	30143908399	dsv4flash-fp8-1k1k-1p1d-dp8ep8	❌ Failed	0.905	below 0.91 threshold
Jul 24	30066681283	same	✅ Passed	0.914	—
Jul 23	29979446261	same	✅ Passed	—	—
Jul 22	29890971156	same	✅ Passed	—	—
Jul 21	29801120084	same	✅ Passed	—	—
Jul 20	29716984185	same	✅ Passed	—	—
Jul 19	29673297044	same	✅ Passed	—	—
Jul 18	29630291750	same	❌ Failed	0.906	below 0.91 threshold
Jul 17	29554632780	same	✅ Passed	—	(run had other failing configs)
Jul 16	29471180267	same	✅ Passed	—	(run had other failing configs)
Jul 15	29388634922	same	✅ Passed	—	(run had other failing configs)
Jul 13	29223989193	same	✅ Passed	—	(run had other failing configs)
Jul 12	29179694296	same	❌ Failed	0.905	below 0.91 threshold
Jul 11	29139507423	same	✅ Passed	—	—

First observed (this signature, in window): Jul 12 2026. Last known passing for this config: Jul 24 2026 (0.914). Failure ratio for this exact signature: 3/13 completed runs. (The Jul 14 run was cancelled and excluded. The current run 30143908399 is still queued at the run level for other configs, but this job itself completed=failure.)

Failure Cluster

GSM8K accuracy gate marginally below threshold (DSV4-Flash-FP8 1P1D) — pre-benchmark gate failure, no sweep results.

Facts (verified observations)

• artifact:bench.log and job log 4131–4132: [gsm8k] accuracy=0.905 threshold=0.91 → accuracy below threshold -- failing before sweep. [FACT]
• The gate ran the full dataset: Accuracy: 0.905 Invalid: 0.001 Latency: 38.827 s Output throughput: 3389.885 token/s (artifact:gsm8k.log). Servers were healthy; this is a quality miss, not a crash. [FACT]
• This config's artifact has no agg_*.json result files (MANIFEST lines 9–12), confirming the sweep never ran — consistent with "failing before sweep". [FACT]
• Server exit codes are teardown, not crash: prefill rc=137 (SIGKILL), decode rc=143 (SIGTERM) (artifact:server_exit_*). srun: error: mia1-p02-g09: task 0: Exited with exit code 1 is the gate's non-zero return propagating through srun. [FACT]
• Accuracy vs 0.91 threshold across runs: Jul 12 = 0.905 ❌, Jul 18 = 0.906 ❌, Jul 24 = 0.914 ✅ (+0.004 margin), Jul 25 = 0.905 ❌ (verified from each run's job log). Passing runs clear the bar by only ~0.4pp. [FACT]
• The three failing runs have distinct head SHAs (592c04381d21 Jul 12, 44e4999ab2f1 Jul 18, ebcb74abd477 Jul 25) with passing runs in between → no single-commit correlation. [FACT]

Hypothesised Causes (with confidence)

• [HIGH] The 0.91 GSM8K accuracy threshold is set too tight for this DSV4-Flash-FP8 1P1D DP8/EP8 config, whose real accuracy sits at 0.905–0.914 with run-to-run sampling variance of ±0.005. Runs land on either side of the bar depending on noise.
  • Supporting: measured accuracy clusters tightly at 0.905/0.906/0.914 across runs; passing margin is only +0.004; failing margin only −0.005; no code-commit correlates with the fail/pass pattern.
  • Disconfirming: I did not compute the formal standard deviation of GSM8K@1319 for this model; a genuine slow accuracy drift (e.g. a kernel/quantization change on gfx950) could also sit at this boundary. Weakening that alternative: passes and fails interleave rather than showing a monotonic drop.
  • How to verify: re-run this exact config 3–5× at the current HEAD without code changes; if accuracy oscillates 0.900–0.915, the threshold is the problem. Optionally raise num_questions or lower the gate to ~0.89 to absorb variance.
• [LOW] A DSV4 FP8 numerics issue on gfx950 (MI355X) slightly depresses accuracy.
  • Supporting: FP8 quantization can shave a few tenths of a point; open PR #31727 touches DSV4 fused-RMS FP8 scale metadata on gfx950.
  • Disconfirming: the config passed 6 consecutive runs (Jul 19–24) on the same image/kernels immediately before today; a persistent numerics bug would fail consistently, not intermittently.
  • How to verify: check whether accuracy improves after #31727 merges; compare accuracy on a non-gfx950 backend for the same model.

In-flight Fix Check

• In-flight fix: ❌ none found — no open PR relaxes or retunes the MI355X dsv4flash GSM8K threshold (searched mi355x+threshold, gsm8k+threshold, dsv4flash, mi355x+gsm8k).
• Tangentially related accuracy PR (not a threshold fix): #31727 "[AMD] Fix DeepSeek-V4 fused-RMS FP8 scale metadata on gfx950" — could nudge DSV4 FP8 accuracy; worth watching but does not directly address the marginal-gate flake.

Suggested Triage Steps

• Re-run config dsv4flash-fp8-1k1k-1p1d-dp8ep8 3–5× at HEAD ebcb74ab with no code change; record accuracy spread to confirm it straddles 0.91.
• If spread confirms variance, retune the gate in the recipe/gate script (raise num_questions beyond 1319 to shrink variance, or lower the threshold to ~0.89) — locate the threshold=0.91 and failing before sweep logic in the MI355X gate script (scripts/ci/slurm/... per the CONFIG_FILE=scripts/ci/slurm/recipes/mi355x-fp8/dsv4flash/1k1k/1p1d-dp8ep8.yaml env).
• Cross-check against #31727: if it lands and lifts DSV4 FP8 accuracy comfortably above 0.91, the flake may self-resolve — re-evaluate before changing the threshold.
• If accuracy is genuinely regressing (monotonic, not oscillating), bisect DSV4 FP8 / gfx950 changes in 44e4999ab2f1..ebcb74abd477 focusing on quantization/MoE/attention paths — but current evidence (interleaved passes) argues against this.

Status

• Persistence: Intermittent — failed 3/13 completed scheduled runs in the queryable window (Jul 12, Jul 18, Jul 25); passed the 6 runs immediately prior to today (Jul 19–24, last pass 0.914).
• Scope: One matrix config (dsv4flash-fp8-1k1k-1p1d-dp8ep8) in the nightly-amd-mi355x-disagg.yml workflow. Gate fails before the benchmark sweep, so this config publishes no throughput/latency results on failing days.
• Blocked work: No throughput/latency data point for DSV4-Flash-FP8 1P1D DP8/EP8 on failing nights; does not affect other configs in the run.
• In-flight fix: ❌ none found — needs a threshold retune or accuracy investigation (see #31727 as a watch item).

nightly-mi355x-benchmark (dsv4pro-fp4-1k1k-1p1d, dsv4pro-fp4-mi355x-sglang, deepseek-ai/DeepSeek-... — failed step(s): N/A

Commit Info

• sglang: ebcb74ab
• aiter: N/A — this leg's GitHub job log contains no [CI-AITER-CHECK] markers (only teardown/upload steps ran; the benchmark step that would install/log aiter never executed).

Failed Tests

Test File	Test Function	Error	Log
N/A (no test ran)	N/A	Job cancelled ~7s after start; benchmark step skipped	Job

This is not a test/benchmark failure. Job 89642143821 concluded cancelled (verified via API), running only 09:34:34Z→09:34:41Z. Steps 1–2 (Set up job, Set up runner) succeeded; steps 3–5 (Checkout code, Clean up prior Slurm jobs, Launch MI355X 2N 1P1D benchmark) were skipped; steps 6–11 (if: always() teardown) ran. Checkout and Launch have no if: guard (workflow lines 119, 142), so their skip is a FACT that the job was cancelled during setup, before the benchmark launched.

Failure Summary

The dsv4pro-fp4-1k1k-1p1d matrix leg was cancelled ~7 seconds in, before its prefill/decode servers or benchmark were ever launched. No 1P1D benchmark, GSM8K gate, or result JSON was produced by this leg. The surrounding legs on the same run completed normally, so this is a per-leg cancellation, not a workflow-wide abort.

[!CAUTION]
Stale-artifact trap — the "passing" logs in this leg's artifact are NOT from this leg. Artifact mi355x-dsv4pro-fp4-1k1k-1p1d-30143908399 (id 8618189572) contains only -dp8ep8-suffixed files (mi355x-dsv4pro-fp4-1k1k-1p1d-dp8ep8_logs.tar.gz, ..._dp8ep8_..._conc*.json); it has zero files for this leg's own RESULT_FILENAME=mi355x-dsv4pro-fp4-1k1k-1p1d. The Pack logs step logged WARN: log directory not found: …/.mi355x_ci/dsv4pro-fp4-1k1k-1p1d (job log line 62), and the Upload results step globs the shared workspace (*.json, *_logs.tar.gz) rather than the per-leg filename, so it swept up the sibling dp8ep8 leg's bundle. The bench.log GSM8K=0.932 / 1024-successful-request results (timestamps ending 09:34:11Z, ~23s before this job started) belong to the dp8ep8 sibling, not to dsv4pro-fp4-1k1k-1p1d.

Regression Status

Infrastructure cancellation (benchmark never executed) — not a benchmark regression.

Recent history of the dsv4pro-fp4-1k1k-1p1d benchmark leg (nightly-amd-mi355x-disagg.yml, event=schedule, branch=main, completed runs only):

Date	Run	Leg Status	Detail
Jul 25	30143908399	⚠️ Cancelled	Benchmark step skipped; run still queued/in-flight at snapshot
Jul 24	30066681283	✅ Passed	job
Jul 23	29979446261	✅ Passed	job
Jul 22	29890971156	✅ Passed	job
Jul 21	29801120084	✅ Passed	job
Jul 20	29716984185	✅ Passed	job

The leg passed the last 5 completed scheduled runs; today's outcome is a cancellation with no benchmark execution, so there is no code-level regression signal for this leg.

Failure Cluster

Per-leg job cancellation during runner setup (benchmark never launched) on the shared MI355X disagg runner.

Facts (verified observations)

• Job conclusion cancelled; wall time 09:34:34Z→09:34:41Z (~7s) — API jobs/89642143821.
• Steps 3 Checkout code, 4 Clean up prior Slurm jobs, 5 Launch MI355X 2N 1P1D benchmark = skipped; steps 1–2 success; steps 6–11 success (if: always() teardown). Checkout/Launch carry no if: guard → skip ⟹ cancellation.
• Workflow guards make auto-cancel unlikely: fail-fast: false (line 96) and cancel-in-progress: false (line 23). A sibling leg dsv4flash-fp8-1k1k-1p1d-dp8ep8 (89642143827) is failure, and legs after ours (e.g. 89642143852, 89642143881) are success — so the run was not wholesale-cancelled.
• All 19 matrix legs execute on one shared self-hosted runner mia1-p01-g20-di-shared / machine mia1-p01-g20 (verified by grepping Machine name across jobs 89642143821, 89642143881, 89642143812, 89642143827). The actual servers run on separate SLURM nodes (artifact: prefill mia1-p02-g09, decode mia1-p02-g29).
• The orchestrating runner host mia1-p01-g20 is the same node the workflow deliberately SLURM_EXCLUDEs (line 151) as broken ("ionic RDMA driver ABI mismatches the container, so MORI reports 'no active RDMA device' and the disagg server dies on init").
• This leg produced no own result JSON; its artifact holds only sibling -dp8ep8 files (MANIFEST lines 18–25).

Hypothesised Causes (with confidence)

• [LOW] Infrastructure/runner-level cancellation of this single leg before the benchmark launched (runner agent disconnect / node drain / job-started hook, or an individual manual cancel).
  • Supporting: steps 1–2 succeed then 3–5 skip within 7s (classic cancel-during-setup shape); single shared serial runner; the runner host mia1-p01-g20 is a known-degraded node (RDMA ABI mismatch per line 148–151); job-started hook cleanup-workspace.sh emitted chown: invalid group: '300611:docker' (job log line 21), indicating an imperfect runner environment.
  • Disconfirming: adjacent legs on the same runner succeeded immediately before and after; fail-fast:false + cancel-in-progress:false rule out automatic sibling/newer-run cancellation; the run continued to completion for other legs.
  • How to verify: inspect the self-hosted runner agent logs on mia1-p01-g20 around 09:34:34–41Z for a disconnect/reconnect or service restart; check the run's audit/timeline for a manual cancel event on this specific leg.
• [SPECULATION] Benchmark regression on dsv4pro-fp4-1k1k-1p1d. Rejected: the benchmark never ran (step 5 skipped) and the leg passed the prior 5 completed runs — there is no benchmark-level or code-level evidence, and the passing artifact logs are the dp8ep8 sibling's.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs repo:sgl-project/sglang is:pr is:open mi355x runner (22 results, all recipe/feature/tuning PRs; none address per-leg cancellation or mia1-p01-g20 runner health).

Suggested Triage Steps

• Pull the self-hosted runner agent (_diag) logs for mia1-p01-g20-di-shared around 09:34:34–09:34:41Z to confirm whether the runner dropped the job (disconnect/drain) vs an external cancel.
• Re-run only the dsv4pro-fp4-1k1k-1p1d leg once the run finishes; given 5 consecutive prior passes, a clean re-run will most likely go green, confirming a transient infra cancel.
• Consider hardening the Upload results glob to the per-leg RESULT_FILENAME prefix so a cancelled/no-op leg does not upload a sibling's *_logs.tar.gz / *.json and masquerade as having run (this stale-artifact trap misleads triage). Investigation note only — maintainer decides.
• Track mia1-p01-g20 runner-host health separately: it is both the sole orchestrating runner and an explicitly SLURM-excluded bad node, making it a single point of failure for all 19 legs.

Status

• Persistence: single occurrence — this leg passed the previous 5 completed scheduled runs (Jul 20–24); first cancellation observed 2026-07-25 09:34:34Z.
• Scope: 1 matrix leg (dsv4pro-fp4-1k1k-1p1d) of run 30143908399; run still queued/in-flight at snapshot. A separate sibling leg dsv4flash-fp8-1k1k-1p1d-dp8ep8 is failure (distinct issue, out of scope here).
• Blocked work: no 1P1D fp4 perf/accuracy signal for dsv4pro-fp4-1k1k-1p1d in this run; the benchmark did not execute.
• In-flight fix: ❌ none found in open PRs.

collect-results — failed step(s): N/A

Commit Info

• sglang: ebcb74abd477afbcb940c8ba30d45f983597e38e
• aiter: N/A (no [CI-AITER-CHECK] markers — this is a Slurm multi-node benchmark workflow; the job-page log blob for collect-results is expired/BlobNotFound, and the artifact bench logs do not carry aiter version markers)

Failed Tests

Test File	Test Function	Error	Log
N/A (GSM8K accuracy gate, config dsv4flash-fp8-1k1k-1p1d-dp8ep8)	GSM8K correctness gate on DeepSeek-V4-Flash-FP8 (1P1D DP8EP8)	accuracy=0.905 threshold=0.91 — accuracy below threshold, failing before sweep	job · artifact:mi355x-dsv4flash-fp8-1k1k-1p1d-dp8ep8_logs/bench.log

collect-results (job 89678875223) is a rollup job — it has no tests of its own and fails whenever any nightly-mi355x-benchmark matrix leg fails. The only failing leg in this run is dsv4flash-fp8-1k1k-1p1d-dp8ep8 (job 89642143827), step 5 "Launch MI355X 2N 1P1D benchmark". (One other leg, dsv4pro-fp4-1k1k-1p1d job 89642143821, was cancelled with Checkout skipped — a separate runner/infra cancellation, unrelated to this accuracy miss.)

Failure Summary

The DeepSeek-V4-Flash-FP8 disaggregated (1P1D, DP8EP8) benchmark leg failed its GSM8K accuracy gate: measured accuracy 0.905 against a 0.91 threshold (1319 questions, 8-shot). The harness prints accuracy below threshold -- failing before sweep and the prefill Slurm task exits rc=1, which propagates to collect-results. The prefill/decode servers were then torn down (prefill rc=137/SIGKILL, decode rc=143/SIGTERM — teardown artifacts, not the root cause). This is purely a correctness-gate miss, not a crash or hang: all /generate requests returned 200 and the eval completed.

Regression Status

New failure today (borderline gate). The config sits right on top of the 0.91 gate and dipped 0.5% below it today.

Recent history of the GSM8K accuracy gate for dsv4flash-fp8-1k1k-1p1d-dp8ep8 in workflow nightly-amd-mi355x-disagg.yml (most recent comparable completed scheduled runs, same branch main, event=schedule; accuracy read from each run's mi355x-dsv4flash-fp8-1k1k-1p1d-dp8ep8 artifact bench.log):

Date	Run	Config Status	GSM8K accuracy	Threshold
Jul 25	30143908399	❌ Failed	0.905	0.91
Jul 24	30066681283	✅ Passed	0.914	0.91
Jul 23	29979446261	✅ Passed	0.917	0.91
Jul 22	29890971156	✅ Passed (run all-green)	not extracted	0.91
Jul 21	29801120084	✅ Passed (run all-green)	not extracted	0.91
Jul 20	29716984185	✅ Passed (run all-green)	not extracted	0.91

First observed gate failure for this config: Jul 25 2026 (this run). Last passing: Jul 24 2026 at 0.914. The 3-day accuracy trend is a monotonic slide 0.917 → 0.914 → 0.905, with the historical margin already thin (only +0.007 / +0.004 above gate). Threshold has been stable at 0.91 across all three runs (not a newly-raised gate). Runs Jul 11–18 show workflow-level failures but were not gate-checked at this config level here; the immediately preceding 5 runs (Jul 20–24) were all-green, so this config passed each.

Failure Cluster

GSM8K accuracy gate miss on DeepSeek-V4-Flash-FP8 disagg (0.905 vs 0.91)

Facts (verified observations, no interpretation)

• artifact:.../bench.log (verbatim): [gsm8k] accuracy=0.905 threshold=0.91 then [gsm8k] accuracy below threshold -- failing before sweep then srun: error: mia1-p02-g09: task 0: Exited with exit code 1.
• artifact:.../gsm8k.log: Accuracy: 0.905, Invalid: 0.001, Latency: 38.827 s, Output throughput: 3389.885 token/s — the eval ran to completion over all 1319 questions.
• Exit markers: bench_exit=1; server_exit_prefill_mia1-p02-g09 = prefill@mia1-p02-g09 rc=137; server_exit_decode_mia1-p02-g29 = decode@mia1-p02-g29 rc=143. rc=137/143 are teardown signals after the gate already failed ("failing before sweep").
• Historical accuracy for the identical config (verified by extracting each run's artifact bench.log): Jul 23 = 0.917, Jul 24 = 0.914, Jul 25 = 0.905. Threshold = 0.91 in all three.
• Non-fatal WARN in bench.log:778: router reports conflicting load_balance_method: prefill=follow_bootstrap_room, decode=round_robin for the DeepSeek-V4-Flash-FP8 model — present but does not abort the run.
• Commit window Jul 24 → Jul 25 (99b29bf1889c..ebcb74abd477) contains 48 sglang commits.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [LOW] Run-to-run GSM8K sampling variance on a borderline gate — the config has been hovering just above 0.91 (0.917, 0.914) and dipped 0.005 below today.
  • Supporting: eval completed cleanly (0.1% invalid, all 200s); no crash/NaN/error in the eval path; a 0.5–1.2% swing on 1319-question GSM8K under FP8 quant is within typical sampling noise.
  • Disconfirming: the 3-day trend is monotonically decreasing (0.917→0.914→0.905) rather than oscillating, which is weakly more consistent with a small real drift than pure noise.
  • How to verify: re-run the dsv4flash-fp8-1k1k-1p1d-dp8ep8 leg 2–3× at HEAD; if accuracy lands ≥0.91 on reruns, it is gate flakiness.
• [LOW] An FP8/MoE/DSV4 numerical change in the 48-commit window slightly shifted DeepSeek-V4-Flash-FP8 outputs. Candidate commits touching plausibly-reachable numerics: 4d5917e74 (DeepSeek 1e-20 epsilon in top-k renorm, #31017), 9402012f0 (per_token_group_quant non-finite sanitization, #32296), 3d91a569c (HPC-Ops FP8 MoE runner backend, #30541), 0a212c611/f7986c860 (DSV4 SWA-KV-quant / indexer dispatch, #31086/#31087).
  • Supporting: several commits directly touch FP8/MoE/DSV4 code paths this config exercises.
  • Disconfirming: no code-level trace tying any commit to the output shift; a large fraction of the window is CUDA/NVIDIA-only (SM120 1e69765ba, A100 7de879275, flashinfer/TRT-LLM) and does not run on the AMD ROCm/gfx950 path; the miss is only 0.005, well within gate noise. No single commit is implicated.
  • How to verify: bisect 99b29bf1889c..ebcb74abd477 re-running this leg's GSM8K gate; only pursue if reruns at HEAD reproduce <0.91 consistently.

No hypothesis above LOW confidence — the failure is a marginal accuracy-gate miss with no code-level evidence; needs a rerun to separate gate flakiness from a real drift before any bisect.

In-flight Fix Check

• In-flight fix: ⚠️ candidate #31727 "[AMD] Fix DeepSeek-V4 fused-RMS FP8 scale metadata on gfx950" — directly targets DSV4-Flash FP8 numerics on gfx950 (MI355X); could plausibly move this exact gate. Also related open PRs: #30248 (A/B disable DSV4 Flash C128 state PD transfer), #30900 (fp8 per-token-group quant on gfx95x). None is a confirmed fix for a 0.5% GSM8K dip.
• Search ran: search/issues?q=repo:sgl-project/sglang+is:pr+is:open+dsv4flash+gsm8k and +mi355x+accuracy.

Suggested Triage Steps

• Re-run the dsv4flash-fp8-1k1k-1p1d-dp8ep8 leg 2–3× at HEAD ebcb74abd477 on MI355X; if accuracy returns ≥0.91, classify as borderline gate flakiness and consider a small gate margin (e.g. 0.90) or averaging over runs.
• If reruns stay <0.91, bisect 99b29bf1889c..ebcb74abd477 focusing on the FP8/MoE/DSV4 commits listed above (skip CUDA-only ones), re-running the GSM8K gate at each step.
• Check whether in-flight #31727 changes DSV4-Flash FP8 gfx950 numerics; if so, cherry-pick onto a branch and re-run this leg to see if accuracy recovers.
• Verify the router conflicting load_balance_method WARN (bench.log:778) is benign for this config and not silently degrading routing/accuracy.

Status (factual, no priority assignment)

• Persistence: First and only gate failure on 2026-07-25; passed the prior 5 scheduled runs (Jul 20–24), last at 0.914 on Jul 24.
• Scope: 1 benchmark leg (dsv4flash-fp8-1k1k-1p1d-dp8ep8) in 1 workflow (nightly-amd-mi355x-disagg.yml); surfaced via the collect-results rollup. Other legs in this run passed (one unrelated cancellation).
• Blocked work: Marks the MI355X disagg nightly red; blocks clean accuracy signal for DeepSeek-V4-Flash-FP8 DP8EP8 disaggregation.
• In-flight fix: ⚠️ candidate #31727 (DSV4-Flash FP8 gfx950) — unconfirmed for this specific 0.905-vs-0.91 dip.

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-07-25 13:11 UTC)

[amd-bot]

pr-test-amd.yml — Per-Job Analysis (continued 5/5)

check-changes — failed step(s): N/A

Commit Info

• sglang: a678a420
• aiter: N/A (check-changes is an ubuntu-latest gate job — no container, no [CI-AITER-CHECK] markers)

Failed Tests

Test File	Test Function	Error	Log
N/A (non-test)	N/A	Runner/infra failure — job concluded failure in 44s with 0 steps recorded and no log uploaded (BlobNotFound)	Job

This is not a test failure. check-changes is the lightweight gating job (.github/workflows/pr-test-amd.yml:126) that only checks out code, runs two bash echo steps (run-mode + continue-on-error), and a dorny/paths-filter step. It failed before any step executed.

Failure Summary

The check-changes gate job on ubuntu-latest failed after 44s without recording any of its 7 steps and without uploading a log (GitHub returns BlobNotFound for the log blob). Because every downstream AMD job depends on check-changes outputs, all 23 test jobs were skipped and the 3 aggregation jobs (pr-test-amd-finish, call-pr-test-amd-extra / pr-test-amd-extra-finish, call-pr-test-amd-extra / call-gate / pr-gate) failed as a cascade.

Regression Status

Infrastructure issue (isolated, first occurrence in the queried window).

Recent history of check-changes job in pr-test-amd.yml (most recent 15 comparable completed scheduled runs: branch=main, event=schedule, completed only):

Date (UTC)	Run	Job	Job Status	Duration	Steps
07-25 12:19	run	check-changes	❌ Failed	44s	0
07-25 00:29	run	check-changes	✅ Success	7s	7
07-24 12:29	run	check-changes	✅ Success	9s	7
07-24 00:25	run	check-changes	✅ Success	11s	7
07-23 12:32	run	check-changes	✅ Success	8s	7
07-23 00:27	run	check-changes	✅ Success	9s	7
07-22 12:32	run	check-changes	✅ Success	11s	7
07-22 00:26	run	check-changes	✅ Success	7s	7
07-21 12:32	run	check-changes	✅ Success	10s	7
07-21 00:27	run	check-changes	✅ Success	7s	7
07-20 12:42	run	check-changes	✅ Success	10s	7
07-20 00:28	run	check-changes	✅ Success	7s	7
07-19 12:17	run	check-changes	✅ Success	10s	7
07-19 00:27	run	check-changes	✅ Success	9s	7
07-18 12:17	run	check-changes	✅ Success	8s	7
07-18 00:25	run	check-changes	✅ Success	7s	7

First (and only) observed failure: 2026-07-25 12:19 UTC. Last passing: 2026-07-25 00:29 UTC (12h earlier). The job passed 15/15 immediately-prior scheduled runs on adjacent commits.

Failure Cluster

Runner infrastructure failure on ubuntu-latest gate job (no steps executed, no log uploaded)

Facts (verified observations)

• Job check-changes conclusion=failure, started_at=12:22:47Z, completed_at=12:23:31Z → 44s wall time (job API).
• The job's steps array is empty (steps=0) in the GitHub job API, whereas all 15 passing historical runs recorded 7 steps.
• Log download returns <Error><Code>BlobNotFound</Code> — no log was ever uploaded for this job.
• Normal successful runs complete in 7-11s; this failed run took 44s (≈4-6× longer), consistent with a hang/stall before the runner dropped.
• Run-level fanout: of 27 jobs, 4 failure + 23 skipped. The 3 non-check-changes failures are all aggregation jobs that needs: [... check-changes ...] and use if: always() — pr-test-amd-finish (pr-test-amd.yml:1296), pr-test-amd-extra-finish (pr-test-amd-extra.yml:220), and call-gate / pr-gate. They report failure because their upstream gate failed.
• Commit a678a420 is [KDA] Add target_verify support for speculative decoding (#26888) — touches speculative decoding code, none of which is exercised by check-changes (checkout + echo + paths-filter only).

Hypothesised Causes (with confidence, with disconfirming evidence)

• [HIGH] GitHub Actions runner infrastructure failure — the ubuntu-latest runner failed to execute/record any step and never uploaded a log.
  • Supporting: 0 steps recorded (vs 7 in every passing run); log blob missing (BlobNotFound); 44s duration vs typical 7-11s; the job body is trivial and deterministic (checkout + 2 echoes + paths-filter).
  • Disconfirming: (none found) — no log evidence exists to contradict; the isolated single-occurrence pattern against a 15/15 green history is consistent with transient infra.
  • How to verify: re-run the check-changes job (or the whole workflow) on the same SHA; expect it to pass in ~10s with 7 steps. If it reproduces, inspect GitHub Actions runner/service status for ubuntu-latest around 12:22-12:23 UTC 2026-07-25.
• [SPECULATION] Code regression from HEAD commit a678a420.
  • Supporting: it is the HEAD SHA of the failing run.
  • Disconfirming: check-changes runs no project code (only checkout + shell echoes + a paths filter); the commit only modifies speculative-decoding source; and a code fault would still produce steps + a log, not 0 steps and a missing blob.
  • How to verify: not actionable — no code path links this commit to the gate job.

In-flight Fix Check

Not applicable — this is a transient runner/infra failure, not a code defect, so no source fix is warranted. No PR search performed (nothing to pin/revert). If it recurs, it is a CI-infra reliability item, not an sglang code issue.

• In-flight fix: ❌ none needed — transient infrastructure failure

Suggested Triage Steps

• Re-run the failed jobs on run 30157771919 (or re-trigger the whole scheduled workflow) — a green check-changes in ~10s with 7 steps confirms transient infra.
• If it does not reproduce (expected), close as a one-off GitHub Actions runner blip; no code action.
• If it does reproduce, check GitHub Actions status/incidents for ubuntu-latest at 2026-07-25 ~12:22 UTC and inspect whether the actions/checkout@v4 step or paths-filter is stalling on network.

Status (factual)

• Persistence: Single occurrence — first and only failure at 2026-07-25 12:19 UTC; passed the prior 15/15 comparable scheduled runs (last green 12h earlier).
• Scope: 1 gate job (check-changes) in pr-test-amd.yml. Blast radius this run: 23 downstream jobs skipped + 3 aggregation jobs failed (all cascade, no independent failures).
• Blocked work: This scheduled run produced no AMD test signal — all stage-a/b/c and kernel/multimodal suites were skipped due to the gate failure. Re-run required to obtain CI signal for SHA a678a420.
• In-flight fix: ❌ none needed — transient infrastructure failure.

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-07-25 13:12 UTC)

[amd-bot]

pr-test-amd-rocm720.yml — Cross-Run Pattern Summary (2 runs in lookback window)

Last scanned (UTC): 2026-07-25 23:14 UTC

Agent assessment

Persistent clusters (top 3 by blast radius)

1 · Diffusion 1-GPU suite: 60-min step timeout + runner loss (3 of 4 shards)

• Jobs (run 30115153295): 1gpu part 0 (timeout), 1gpu part 1 (timeout), 1gpu part 3 (runner lost / BlobNotFound). Shard 2 passes.
• Persistence: parts 0/1 failed ≥8 consecutive scheduled runs (since ≥Jul 17); part 3 runner-loss ≥12 runs (since ≥Jul 13). Never-passed in window.
• Top hypothesis: [HIGH] the partition workload exceeds the 60-min step budget on MI325 (.github/workflows/pr-test-amd-rocm720.yml:728); wall-clock, not a per-case hang. Disconfirming: part 3 fast-fails in 25 min (runner-loss, a different infra signature — do not conflate). Empty HF cache (✗ NOT CACHED) is a [MEDIUM] contributor.
• In-flight fix: ❌ none.

2 · 2-GPU diffusion part-0: runner termination, no log uploaded (persistent both runs)

• Jobs: Jul 24 · Jul 25. Step 9 stuck in_progress, steps 10/20 pending, BlobNotFound. Parts 1 & 2 pass.
• Persistence: part-0 red in all ~9–15 comparable runs (since ≥Jul 10); never-passed. Historically (retrievable Jul 13/15/16 logs) = flux2_modelopt_fp8_tp2_t2i HIPBLAS_STATUS_NOT_SUPPORTED + joy_echo_t2v_2gpu startup hang leaking port 5555 → strict-ports cascade; recent runs degraded to a mid-run hang/lost-runner.
• Top hypothesis: [MEDIUM] MI325 2-GPU runner lost mid-test. Disconfirming: no log/diagnostics, so GPU-hang vs OOM vs infra unproven.
• In-flight fix: flux2 FP8 → ✅ #28889; joy_echo/port-cascade & runner-loss → ❌ none.

3 · GSM8K MXFP4 accuracy-gate flake (stage-b-1gpu-small-mi35x, persistent both runs)

• Jobs: Jul 24 (0.914/0.914) · Jul 25 (0.900/0.912). Test test/registered/quant/test_quark_mxfp4.py::TestFP8ToMXFP4MOETP1.test_gsm8k.
• Persistence: flaky since Jul 20 — failed Jul 22/24/25, passed Jul 20/21/23 (~3/6). Accuracy band {0.900–0.936} straddles the hard-coded 0.92 gate (:251).
• Top hypothesis: [HIGH] gate too tight vs GSM8K sampling variance. Disconfirming: none found (no comfortably-green baseline).
• In-flight fix: ❌ none.

Regression candidates (fresh, not flake)

Mechanical bucket = (none) because the "latest run" 30168318210 is in-flight and its heavy jobs have not run. But one genuine fresh regression appeared in the Jul 24 run and MUST be tracked:

• stage-c-test-large-8-gpu-amd-rocm720 (…, 1) — test_deepseek_v32_basic.py::TestDeepseekV32TP.test_a_gsm8k. HSA_STATUS_ERROR_OUT_OF_RESOURCES (0 MB free) mid-eval (810/1319) → server abort. Fresh: this test file was green the prior 7 runs (Jul 17–23, 0.957 on Jul 23); Jul 14–16 failures were a different setUpClass startup signature.
• Commit window: d0b96898..2428f561 (44 commits); config uses aggressive mem_fraction_static=0.896 + bf16 KV. No hypothesis > [LOW].
• Triage: re-run 3–5× on MI325 ROCm 7.2 to classify flaky vs deterministic before bisecting; probe lower mem_fraction_static (0.85). In-flight fix ❌.

Other persistent (single-shard, carrying over)

• AMD Triton offline throughput < 3500 gate — stage-b-1gpu-large, test_bench_serving_1gpu_part1.py::test_offline_throughput_with_triton_attention_backend. Never-passed ≥9 runs (band 3087–3244 t/s); [MEDIUM] mis-calibrated gate vs long-standing regression — indistinguishable without a pre-Jul-16 green. In-flight ❌.
• msgpack ModuleNotFoundError — multimodal-gen-unit-test, test_msgpack_roundtrip_preserves_string_keys_and_numpy_payloads. Never-passed since Jul 11 (dep missing in ROCm 7.2 image). In-flight fix ✅ #31899.

Flakes / intermittent

The "flaky" bucket (6 jobs in run 1, absent in run 2) is a window artifact — run 2 is in-flight and those jobs have not executed. Per their own histories they are persistent/never-passed, not intermittent. Only genuinely intermittent test: quark_mxfp4 MXFP4 gate (~3/6 runs, cluster 3 above).

Suggested next actions

1. Chase in-flight fixes: merge ✅ #31899 (msgpack) and ✅ #28889 (flux2 FP8 HIPBLAS) — each clears a persistent cluster with no duplicate needed.
2. DeepSeek-V3.2 OOM (cluster/regression): re-run stage-c 8-gpu 3–5× to classify flaky vs deterministic; bisect d0b96898..2428f561 only if deterministic — the sole fresh signal, highest ROI.
3. Diffusion 1-GPU timeouts (3 shards): decide capacity fix — raise timeout-minutes:60 or increase --total-partitions, and confirm /sgl-data/hf-cache is a live mount (empty cache forces multi-GB re-download inside the timed step).
4. 2-GPU part-0 & 1-GPU part-3 runner terminations: pull runner-side _diag/dmesg/rocm-smi around the failure windows — needs infra investigation, not a code PR.
5. Perf/accuracy gates (throughput 3500, mxfp4 0.92): maintainer decision on re-baselining vs quarantine; query scheduled history before Jul 16 to settle "gate too tight" vs "old regression" for the throughput test.

---

Runs analysed

Run	Started (UTC)	Failed jobs
30115153295	2026-07-24 17:59	8
30168318210	2026-07-25 17:47	2

Persistent failures (in every run)

• multimodal-gen-test-2-gpu-amd-rocm720 (linux-mi325-2gpu-sglang, 0) — in 2/2 runs (5153295, 8318210)
• stage-b-test-1-gpu-small-amd-mi35x-rocm720 (linux-mi35x-gpu-1) — in 2/2 runs (5153295, 8318210)

Latest-only failures (potential regressions)

(none)

Flaky / intermittent failures

• multimodal-gen-test-1-gpu-amd-rocm720 (linux-mi325-1gpu-sglang, 0) — in 1/2 runs (5153295)
• multimodal-gen-test-1-gpu-amd-rocm720 (linux-mi325-1gpu-sglang, 1) — in 1/2 runs (5153295)
• multimodal-gen-test-1-gpu-amd-rocm720 (linux-mi325-1gpu-sglang, 3) — in 1/2 runs (5153295)
• multimodal-gen-unit-test-amd-rocm720 — in 1/2 runs (5153295)
• stage-b-test-1-gpu-large-amd-rocm720 (linux-mi325-1gpu-sglang, 0) — in 1/2 runs (5153295)
• stage-c-test-large-8-gpu-amd-rocm720 (linux-mi325-8gpu-sglang, 1) — in 1/2 runs (5153295)

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-07-25 23:14 UTC)