[CI Monitor] Daily Report - 2026-07-23

[amd-bot]

Daily Cross-Workflow Summary — 2026-07-23

Snapshot: 2026-07-23 23:37 UTC · Only completed runs/jobs counted in trends · Auto-updated every 30 min

TL;DR

🟡 YELLOW · 2 active clusters (both carry-over: R270, R257) · 0 🆕 today · 3 RESOLVED today (R285, R286, R287) · in-flight fixes: R270→#31741 (open, blocked/needs rebase), R257→#31068/#32003.
👉 Today's ask: No new triage. Yesterday's three #31816/#28671 deterministic breaks are gone — revert #32100 (f5dcbe8f1, merged Jul-22 ~18:52 UTC) rolled back #31816. Remaining work is chasing the two long-standing carry-overs: (1) rebase/merge #31741 (R270 mamba NameError, now 8th day, never-passed); (2) chase #31068/#32003 or reboot the MI35x pool for R257 (15th day).

Caution

Jul-23 nightly-test-amd (30031602981), nightly-test-amd-rocm720 (30031529424), and pr-test-amd-rocm720 (30031591211) are all queued (IN-FLIGHT) at snapshot (head d0b968980). Trends below use the latest completed runs (Jul-22 29944275726 / 29944173150 / 29944235234). The already-concluded kernel/qwen3-235b legs of the IN-FLIGHT Jul-23 nightlies still show R270 + R257, so expect no nightly recovery.

Note

Good news: pr-test-amd Jul-23 run 30007380837 (completed, head b98a577fb) shows stage-c-large-8gpu (3) ✅, stage-c-large-8gpu-mi35x (1) ✅, and stage-b-small-nondeterministic ✅ — i.e. R285, R286, R287 all green. release-docker-amd-nightly, release-docker-amd-rocm720-nightly, and nightly-amd-mi355x-disagg all ✅ green (Jul-23). No AITER Scout failures today.

Workflow status

Workflow	Latest completed run	❌ jobs	Today's status	Δ vs yesterday
nightly-test-amd	Jul-22 29944275726 (failure)	8	Jul-23 30031602981 (IN-FLIGHT)	R270 + R257 persist
nightly-test-amd-rocm720	Jul-22 29944173150 (failure)	9	Jul-23 30031529424 (IN-FLIGHT)	R270/R257/R262/R273 carry
release-docker-amd-nightly	Jul-23 30007869689 ✅	0	green	✅ still green
release-docker-amd-rocm720-nightly	Jul-23 30007492164 ✅	0	green	✅ still green
nightly-amd-mi355x-disagg	Jul-23 29979446261 ✅	0	green	✅ still green
amd-aiter-scout	(no failures today)	—	n/a	—
pr-test-amd	Jul-23 30007380837 (failure)	7	🟢 R285/R286/R287 RESOLVED	✅ regressions reverted
pr-test-amd-rocm720	Jul-22 29944235234 (failure)	11	Jul-23 30031591211 (IN-FLIGHT)	#31816 breaks reverted

7d run-conclusion trend (older→newer, completed scheduled runs; ✅=green, ❌=failure, ⊘=cancelled): nightly-test-amd ❌·❌·❌·⊘·❌·❌ · nightly-rocm720 ❌·❌·❌·⊘·❌·❌ · mi355x ❌·❌·✅·✅·✅·✅·✅ · pr-test-amd ❌·❌·❌·❌·❌·❌ · pr-test-amd-rocm720 ❌·❌·❌·❌·❌·❌ · release-docker-amd ✅·✅·✅·✅·✅·✅·✅ · release-docker-rocm720 ✅·✅·✅·✅·✅·✅·✅.

Failure clusters (deduplicated across all workflows)

R270 · NameError: transfer_kv_mamba_lf_pf — mamba JIT transfer kernel CUDA-only import guard on AMD — nightly-amd + nightly-rocm720 kernel · ✅ #31741 (open, blocked/needs rebase)

• Status: carry-over, 8th day (first Jul-16), never-passed on AMD. 12 NameError subtests on the Jul-22 completed kernel legs; the already-concluded Jul-23 IN-FLIGHT kernel legs (89289230487, 89288985830) show the identical 12 NameErrors.
• Top hypothesis: [FACT] #30535/#31501 gated transfer_kv_mamba_* imports behind if _is_cuda: (memory_pool_host.py:41) while the call site (:407) is reachable on HIP. Disconfirming: (none — NameError directly observable; sibling sgl_kernel.kvcacheio import one line above uses if _is_cuda or _is_hip:).
• In-flight fix: ✅ #31741 (open since Jul-20; mergeable_state: blocked — still needs rebase/review). Suggested triage: rebase/merge #31741 (widens guard to _is_cuda or _is_hip + HIP .cuh compat), re-run test_transfer_mamba.py; verify the JIT kernel actually compiles under ROCm, not just that the import resolves.

Workflow	Job (shard)	Test File	Test Function	Error	Log
nightly-test-amd	nightly-test-1-gpu-kernel	test/registered/jit/test_transfer_mamba.py	12 subtests (roundtrip/single_item/full_indices)	NameError: transfer_kv_mamba_lf_pf → exit 255	Log
nightly-test-amd-rocm720	nightly-test-1-gpu-kernel-rocm720	test_transfer_mamba.py	same 12 subtests	same	Log

R257 · MI35x (gfx950) GPU hang / HW-exception family — hipErrorCapturedEvent · GPU memory access fault · watchdog abort · warmup hang — nightly-amd + nightly-rocm720 MI35x · ⚠️ #31068/#32003

• Status: carry-over, 15th day (first Jul-09/10). Model-/workflow-independent; alternates pass/fail on the same physical MI35x nodes → environment/HW over a bisectable regression.
• Top hypothesis: [MEDIUM] non-capturable MoE TP all-reduce (aiter custom_all_reduce) inside the decode HIP-graph capture stream → RCCL watchdog stream-capture violation (hipErrorCapturedEvent); compounded by MI35x node/driver hang. Disconfirming: alternates on identical SHAs → not a clean code regression.
• In-flight fix: ⚠️ #31068 (hung-GPU pre-flight recovery, open), #32003 (route scheduled jobs to MI300, open). Suggested triage: make MoE TP all-reduce capture-safe on ROCm; chase #31068/#32003; else reboot the MI35x pool.

Workflow	Job (shard)	Test File	Test Function	Error	Log
nightly-test-amd	nightly-8-gpu-mi35x-qwen3-235b-mxfp4	test_qwen3_instruct_mxfp4.py	setUpClass	hipErrorCapturedEvent (NCCL/RCCL watchdog, capturing stream) → -6	Log
nightly-test-amd	nightly-2-gpu-mi35x-glm51-mxfp4	test_glm51_mxfp4_tp2_gsm8k_mi35x.py	setUpClass	GPU Hang / watchdog timeout during model load/warmup	Log
nightly-test-amd-rocm720	nightly-8-gpu-mi35x-qwen3-235b-mxfp4-rocm720	test_qwen3_instruct_mxfp4.py	setUpClass	hipErrorCapturedEvent (aiter custom_all_reduce) → -6	Log
nightly-test-amd-rocm720	nightly-accuracy-8-gpu-mi35x-rocm720	test_qwen3_moe_eval_mi35x.py	test_qwen3_moe_accuracy (Qwen3-30B-A3B)	GPU memory access fault on first prefill → abort	Log

Resolved today ✅

ID	Cluster	Resolved by	Evidence
R285	AttributeError: communicator has no 'get_server_args' (test mock target removed by get_parallel() refactor)	Revert #32100 (f5dcbe8f1, merged) rolled back #31816	stage-c-large-8gpu (3) ✅ in 30007380837
R286	Dynamo graph-break capturing DeepSeek prefill under tc_piecewise (ParallelContext.__getattr__ untraceable)	same revert #32100	stage-c-large-8gpu-mi35x (1) ✅ in 30007380837
R287	reward-model AttributeError: '…SequenceClassification…' has no '_legacy_load_weights' (from #28671)	resolved (green on both Jul-23 pr-test-amd runs)	stage-b-small-nondeterministic ✅ in 29969145884 + 30007380837

Known stable / recurring clusters (carrying over, no new action today) · click to expand

Cluster	Workflow · Rep. job	Test / fn	Status	In-flight fix
R262 InternVL2_5-2B CLIPImageProcessor has no .tokenizer	nightly-rocm720 vlm	test_vlms_mmmu_eval_amd.py (InternVL2_5-2B)	recurring since Jul-14	❌
R273 accuracy near-misses: DeepSeek-V4-Flash FP8-TBO, DeepSeek-V3.2 basic (0.928/0.93), MiniMax-M2.5/M2.7 TP, DeepSeek-R1-MXFP4 MTP (~0.85), Mixtral gsm8k	nightly-rocm720 dsv4-flash · nightly-amd dsv32-mtp	test_deepseek_v4_flash_* etc.	marginal/boundary gates	⚠️ threshold recalib precedent #31702
Diffusion (multimodal-gen) family: 1-/2-GPU hang → 60/90-min step timeout, runner-lost/no-log (BlobNotFound), HIPBLAS_STATUS_NOT_SUPPORTED (FLUX.2 fp8), update_weights_from_disk shape-mismatch HTTP 400, Cosmos3-Nano GQA crash	pr-test-amd mm-gen 2gpu (0) · mm-gen 1gpu (3) ×many	test_server_1_gpu.py / test_server_2_gpu.py	never-passed/recurring	⚠️ msgpack #31844; skip #31925
Disaggregation family: R277 Mooncake 30s KV-transfer timeout, KeyError:'answer' PD+PP (flaky 7/13), Kimi-K2.5-MXFP4 load-barrier SIGKILL	pr-test-amd disagg mi35x · rocm720 mooncake	test_disaggregation_pp.py / test_kimi_k25_mxfp4_bcg_mi35x.py	recurring/flaky	⚠️ R277→#31800
R284 LoRA concurrent-adapter-update E2E latency 3-4× over 4000 ms gate	pr-test-amd stage-b-1gpu-large (0)	test_bench_serving_1gpu_part1.py test_online_lora_latency_with_concurrent_adapter_updates	flaky/borderline	❌
MXFP4 GSM8K accuracy just below strict gate (Qwen3-30B-A3B online FP8→MXFP4 requant)	pr-test-amd stage-b-small-mi35x	test_*_mxfp4_*.py	borderline eval flake	❌
Triton offline throughput below AMD 3500 tok/s gate; VLM MMMU timeout/near-chance; DSA topk_v2 JIT shortfall gfx950	pr-rocm720 stage-b-1gpu-large · nightly-rocm720 4-gpu	various	never-passed/recurring	mostly ❌; DSA #31291
ModuleNotFoundError: msgpack on AMD ROCm720 image	pr-rocm720 mm-gen-unit	multimodal-gen-unit-test-amd-rocm720	recurring	⚠️ #31844
Partition wall-time exceeds step cap (all files pass, killed by timeout); Runner-lost comms / infra	pr-test-amd stage-b-2gpu-large (1)	infra/timeout	recurring	⚠️ #31068, #32003

Workflow drill-down · click to expand

nightly-test-amd · latest completed 29944275726 (Jul-22) · 8 failures: R270 (kernel) · R257 (qwen3-235b-mxfp4, glm51-mxfp4) · R273 accuracy near-miss (dsv32-mtp) · GSM8K config gap (accuracy-2-gpu: Qwen3-4B missing threshold + Mixtral) · VLM MMMU (accuracy-2-gpu-vlm, 4-gpu encoder-DP timeout). Jul-23 30031602981 IN-FLIGHT shows R270 + R257 continuing.

nightly-test-amd-rocm720 · latest completed 29944173150 (Jul-22) · 9 failures: R270 (kernel) · R257 (qwen3-235b-mxfp4, qwen3-30b-a3b mem-fault) · R262 (InternVL2_5) · R273 (dsv4-flash fp8-tbo, dsv32 basic, minimax) · triton HSA-OOM (dsv4-pro-mtp) · Mixtral gsm8k. Jul-23 30031529424 IN-FLIGHT.

pr-test-amd · latest completed 30007380837 (Jul-23) · 7 failures, all known families: diffusion (mm-gen 2gpu-0 HIPBLAS FLUX.2, 1gpu-3 update_weights HTTP400, 1gpu-1 no-log, 1gpu-0 hang) · disagg (KeyError 'answer' PP) · MXFP4 accuracy (stage-b-small-mi35x) · wall-time timeout (stage-b-2gpu-large). R285/R286/R287 all ✅ green.

pr-test-amd-rocm720 · latest completed 29944235234 (Jul-22) · 11 failures: R277 Mooncake disagg · Kimi-K2.5 load hang · triton throughput · msgpack ModuleNotFound · diffusion family + DNS flakes · DSA piecewise (now reverted by #32100 on the IN-FLIGHT Jul-23 run). Jul-23 30031591211 IN-FLIGHT.

How this report is generated

• Only status == "completed" runs/jobs counted in trends. Jul-23 nightly-amd (30031602981), nightly-rocm720 (30031529424), and pr-rocm720 (30031591211) were queued at snapshot → (IN-FLIGHT), excluded; latest completed are the Jul-22 runs. pr-test-amd Jul-23 30007380837 was completed and is used directly.
• Resolved today: R285 + R286 via revert #32100 (f5dcbe8f1) of #31816; R287 reward-model _legacy_load_weights green on both Jul-23 pr-test-amd runs. No NEW clusters today.
• Carry-over: R270 mamba NameError (8th day, never-passed, #31741 blocked), R257 MI35x hang family (15th day, #31068/#32003).
• Confidence: FACT/HIGH/MEDIUM/LOW/SPECULATION. Bot does NOT assign Priority — engineers decide from cluster size + persistence + fix availability.

---

Generated by amd-bot · last updated 2026-07-23 23:37 UTC

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-07-23 23:37 UTC)

---

CI Monitor — 2026-07-23

Repo: sgl-project/sglang

Monitored Workflows:

• nightly-test-amd.yml
• nightly-test-amd-rocm720.yml
• release-docker-amd-nightly.yml
• release-docker-amd-rocm720-nightly.yml
• nightly-amd-mi355x-disagg.yml
• amd-aiter-scout.yml
• pr-test-amd.yml
• pr-test-amd-rocm720.yml

Per-workflow failure reports are appended as comments below; the Daily Cross-Workflow Summary is rendered above this section.

[amd-bot]

nightly-test-amd.yml — 13 failure(s)

Run started (UTC): 2026-07-22 17:53 UTC
Last scanned (UTC): 2026-07-23 23:33 UTC
Commits: sglang d0b9689

Summary

Counts: 13 failures · 9 clusters · 10 carrying over (≥3 days) · 3 new/borderline today

#	Cluster	Job	Test File	Test Function	Status	Hypothesis (confidence)
1	aiter all-reduce illegal in HIP graph capture	nightly-8-gpu-qwen3-235b-mxfp4	test/registered/amd/test_qwen3_instruct_mxfp4.py	TestQwen3Instruct2507MXFP4.setUpClass	~10d persistent (since Jul 13), no green	[MEDIUM] aiter custom all-reduce records/queries HIP event on capturing stream → hipErrorCapturedEvent
2	aiter all-reduce illegal in HIP graph capture	nightly-8-gpu-qwen3-235b-mxfp4	test/registered/amd/test_qwen3_instruct_mxfp4.py	TestQwen3Instruct2507MXFP4.setUpClass	6 consecutive runs (since Jul 17)	[MEDIUM] same aiter ca path during full-graph decode capture
3	GSM8K accuracy fail (Mixtral) + Qwen3-4B config gap	nightly-accuracy-2-gpu	test/registered/amd/accuracy/mi30x/test_gsm8k_eval_amd.py	TestNightlyGsm8KEval.test_gsm8k_all_models	~7d persistent (Jul 16→22), no green	[HIGH] Mixtral-8x7B 0.286≪0.69 real accuracy miss; [FACT] Qwen3-4B missing threshold prints cosmetic FAIL
4	GSM8K accuracy fail (Mixtral) + Qwen3-4B config gap	nightly-accuracy-2-gpu	test/registered/amd/accuracy/mi30x/test_gsm8k_eval_amd.py	test_gsm8k_all_models	~7d persistent, log tail truncated	[FACT] Qwen3-4B threshold N/A; terminating assertion in truncated tail (below-threshold model)
5	ROCm import guard NameError (mamba JIT transfer)	nightly-test-1-gpu-kernel	test/registered/jit/test_transfer_mamba.py	test_mamba_kernel_* (12/12)	7d, never passed on AMD (since Jul 16)	[FACT] transfer_kv_mamba_lf_pf imported only under if _is_cuda, called unguarded at memory_pool_host.py:407
6	ROCm import guard NameError (mamba JIT transfer)	nightly-test-1-gpu-kernel	test/registered/kernels/ops/mamba/test_transfer_mamba.py	test_mamba_kernel_* (12/12)	7d, never passed	[HIGH] same CUDA-only import; ROCm never binds the name
7	Borderline GSM8K accuracy gate (marginal miss)	nightly-8-gpu-deepseek-r1-mxfp4-tp4	test/registered/amd/accuracy/mi35x/test_deepseek_r1_mxfp4_tp4_mtp_mi35x.py	test_gsm8k	new today (passed Jul 21/22)	[MEDIUM] 0.9378<0.944; gate pinned to prior run values with ~0.001 margin
8	Borderline GSM8K accuracy gate (marginal miss)	nightly-4-gpu-minimax-m25	test/registered/amd/accuracy/mi35x/test_minimax_m25_tp4_eval_mi35x.py	test_minimax_m25_tp4_accuracy	flaky ~2/6 (0.929<0.93)	[MEDIUM] borderline threshold within run-to-run noise band
9	VLM MMMU accuracy below threshold (5 VLMs, MI30x)	nightly-accuracy-2-gpu-vlm	test/registered/amd/accuracy/mi30x/test_vlms_mmmu_eval_amd.py	test_mmmu_vlm_models (5 subtests)	persistent, no green	[MEDIUM] thresholds (0.29–0.33) sit above the stable 0.27–0.29 band for these 5 models
10	GLM-5.1-MXFP4 TP2 server launch GPU Hang	nightly-2-gpu-glm51-mxfp4	test/registered/amd/accuracy/mi35x/test_glm51_mxfp4_tp2_gsm8k_mi35x.py	TestGLM51MXFP4TP2GSM8KMI35x.setUpClass	~9d (last green Jul 5)	[LOW] a commit in Jul 5→13 broke GLM-5.1 TP2 load/warmup on gfx950
11	GLM-5.1-MXFP4 TP2 runner-lost communication	nightly-2-gpu-glm51-mxfp4	test/registered/amd/accuracy/mi35x/test_glm51_mxfp4_tp2_gsm8k_mi35x.py	TestGLM51MXFP4TP2GSM8KMI35x.test_gsm8k	6d, never passed	[LOW] MI35x runner/driver crash or OOM during heavy GLM-5.1 GSM8K run
12	VLM encoder-DP MMMU timeout	nightly-4-gpu	test/registered/vlm/test_encoder_dp.py	TestVLMEncoderDP.test_vlm_mmmu_benchmark	intermittent (model-dependent)	[MEDIUM] random.choice drew GLM-4.1V-9B-Thinking; long generations at ~8 tok/s exceed 3600s cap
13	DSA MTP/EAGLE draft-extend GPU-sync hang	nightly-accuracy-8-gpu-deepseek-v32-mtp	test/registered/amd/accuracy/mi35x/test_deepseek_v32_mtp_eval_mi35x.py	test_a_gsm8k	new (first Jul 22), test file intermittent	[LOW] DSA/dspark draft-extend commits in pass→fail window; TP1 stuck at dsa_backend.py:781 sync

Cluster: aiter all-reduce illegal in HIP graph capture (rows 1–2)

• Affected: nightly-8-gpu-mi35x-qwen3-235b-mxfp4 (both runs), test_qwen3_instruct_mxfp4.py::setUpClass.
• Shared facts: All 4 TP shards load; crash at start of decode CUDA-graph capture (bs=512, 0/52 captured); Rank 0 scheduler SIGABRT (exit -6, earlier runs exit 134). Root HIP error hipErrorCapturedEvent on all-reduce.
• Top hypothesis: [MEDIUM] aiter custom all-reduce (ca) queries a HIP event on a capturing stream (illegal on ROCm). Disconfirming: none found; signature stable across 6 runs.
• In-flight fix: ⚠️ partial — #29946 fixes the same bug class for symm-mem, not the aiter ca path; likely does not directly fix this.
• Suggested triage: relaunch with SGLANG_USE_AITER_AR=0 (per #23581) and/or TORCH_NCCL_ENABLE_MONITORING=0; if capture completes, aiter AR confirmed.

Cluster: GSM8K accuracy fail (Mixtral) + Qwen3-4B config gap (rows 3–4)

• Affected: nightly-accuracy-2-gpu (both runs), test_gsm8k_eval_amd.py::test_gsm8k_all_models.
• Shared facts: Qwen3-4B lacks a MODEL_SCORE_THRESHOLDS entry → cosmetic ❌ FAILED (threshold: N/A); the job-terminating AssertionError is driven by a genuine below-threshold model (Mixtral-8x7B 0.286≪0.69, verified Jul 22).
• Top hypothesis: [HIGH] Mixtral AMD-backend accuracy miss is the hard-failure contributor. Disconfirming: Jul 23 log truncated before the assertion, so that run's exact culprit unconfirmed.
• In-flight fix: ❌ none. #29933 touches GSM8K answer extraction but is not a targeted fix.
• Suggested triage: re-run Mixtral-8x7B (TP2) restricted on MI325 to confirm 0.286; add a Qwen/Qwen3-4B threshold entry to stop the cosmetic FAIL.

Cluster: ROCm import guard NameError — mamba JIT transfer (rows 5–6)

• Affected: nightly-test-1-gpu-kernel (both runs), all 12 subtests of test_transfer_mamba.py.
• Shared facts: transfer_kv_mamba_lf_pf imported only under if _is_cuda: (memory_pool_host.py:41) but called unguarded at line 407 for io_backend="kernel"; ROCm never binds the name. Other 20 kernel tests pass. AMD-only.
• Top hypothesis: [FACT] CUDA-only import guard; not aiter-related.
• In-flight fix: ✅ #31741 "[AMD] Enable mamba JIT transfer kernel on ROCm" — open since Jul 20, mergeable_state=blocked. Chase reviewers; do NOT duplicate.
• Suggested triage: unblock/merge #31741; verify with python -c "from sglang.srt.mem_cache import memory_pool_host as m; print(hasattr(m,'transfer_kv_mamba_lf_pf'))".

Cluster: Borderline GSM8K accuracy gate — marginal miss (rows 7–8)

• Affected: nightly-8-gpu-mi35x-deepseek-r1-mxfp4-tp4 (test_gsm8k, 0.9378<0.944) and nightly-4-gpu-mi35x-minimax-m25 (test_minimax_m25_tp4_accuracy, 0.929<0.93).
• Shared facts: Both are hard-coded thresholds set with ~0.001–0.002 margin; benchmarks ran to completion (invalid=0); misses are ~1–2 GSM8K questions. deepseek-r1 non-MTP sibling (thr 0.93) passed same commit.
• Top hypothesis: [MEDIUM] run-to-run accuracy jitter against razor-thin gates, not a code regression. Disconfirming: cannot fully exclude spec-decode commits (9b853e683, f35411ee8) for the MTP row.
• In-flight fix: ❌ none for either.
• Suggested triage: re-run each 3–5× at the same SHA; if oscillating around the line, widen thresholds (test_deepseek_r1_mxfp4_tp4_mtp_mi35x.py:38, test_minimax_m25_tp4_eval_mi35x.py:44/66) — maintainer decision.

Cluster: VLM MMMU accuracy below threshold — 5 VLMs on MI30x (row 9)

• Affected: nightly-accuracy-2-gpu-vlm, test_mmmu_vlm_models (Qwen2/2.5/3-VL, MiniCPM-o, deepseek-vl2).
• Top hypothesis: [MEDIUM] mis-calibrated thresholds (0.29–0.33) above the stable 0.27–0.29 band on the 100-sample MMMU subset. Disconfirming: could be genuine AMD VLM accuracy ([LOW]) — needs NVIDIA baseline.
• In-flight fix: ❌ none. Suggested triage: run Qwen2-VL-7B / Qwen3-VL-30B MMMU-100 on MI30x vs NVIDIA at 0bdd4730; recalibrate or raise sample count.

Cluster: GLM-5.1-MXFP4 TP2 launch instability (rows 10–11, related but distinct)

• Affected: nightly-2-gpu-mi35x-glm51-mxfp4 — row 10 GPU Hang/watchdog at setUpClass (server never launches); row 11 runner-lost during test_gsm8k. Same test file, different frames → kept as two clusters; likely a shared underlying GLM-5.1 TP2 launch/GPU-hang issue on gfx950.
• Top hypothesis: [LOW] regression in Jul 5→13 window broke GLM-5.1 TP2 load/warmup (row 10); [LOW] runner/driver crash or OOM (row 11).
• In-flight fix: row 10 ❌; row 11 ✅ #31068 (MI35x hung-GPU pre-flight recovery), also #30431.
• Suggested triage: bisect Jul 5→13 GLM-5.1 launch commits; verify runner-recovery #31068 covers this hang.

Cluster: VLM encoder-DP MMMU timeout (row 12)

• Top hypothesis: [MEDIUM] timeout depends on which model random.choice draws (GLM-4.1V-9B-Thinking). Disconfirming: co-located SIGKILL/OOM signature on same file on other dates — distinct symptom. Suggested triage: cap max_new_tokens for the thinking model or exclude it from the AMD random pool. In-flight ❌.

Cluster: DSA MTP/EAGLE draft-extend GPU-sync hang (row 13)

• Top hypothesis: [LOW] DSA draft-extend commits (08cb081e, 8ae0eb83, 024639a3, first nightly to include them). Facts: pure hang (no fault); TP1 blocked at dsa_backend.py:781 .item() sync, TP0 waits on request broadcast (TP desync). Disconfirming: test file is intermittently red with mixed signatures → could be pre-existing flakiness. In-flight ❌ (#31824 is same-area perf, not a hang fix).

Suggested triage order

Rank	Triage step	Targets	Why this first
1	Unblock/merge #31741 and re-run kernel job	rows 5–6	Ready in-flight fix for a [FACT] root cause; clears 2 jobs / 24 subtests
2	Relaunch qwen3-235b with SGLANG_USE_AITER_AR=0	rows 1–2	Cheap A/B confirms/refutes aiter ca hypothesis on largest persistent cluster
3	Re-run deepseek-r1-MTP + minimax 3–5× at same SHA	rows 7–8	Distinguishes borderline noise from regression before touching thresholds
4	Confirm Mixtral 0.286 repro; add Qwen3-4B threshold	rows 3–4	Separates real accuracy miss from cosmetic config gap
5	MMMU-100 NVIDIA-vs-MI30x baseline for 1–2 failing VLMs	row 9	Cheaply decides threshold-recalibration vs real AMD accuracy bug
6	Bisect Jul 5→13 GLM-5.1 launch window; verify #31068	rows 10–11	Two related GLM-5.1 TP2 failures; runner-recovery PR may cover row 11

---

Job	Failed Steps	Started
nightly-test-1-gpu-kernel	Nightly Kernel Benchmark (1-GPU)	2026-07-22T18:05
nightly-2-gpu-mi35x-glm51-mxfp4	N/A	2026-07-22T17:53
nightly-accuracy-2-gpu	N/A	2026-07-22T18:52
nightly-4-gpu	Nightly Test (4-GPU)	2026-07-22T17:53
nightly-accuracy-2-gpu-vlm	Nightly Accuracy Test (2-GPU VLM MMMU)	2026-07-22T18:39
nightly-8-gpu-mi35x-qwen3-235b-mxfp4	Accuracy Test + Performance Test MI35x (8-GPU Qwen3-235B-MXFP4)	2026-07-23T00:35
nightly-accuracy-8-gpu-mi35x-deepseek-v32-mtp	Accuracy Test MI35x (8-GPU DeepSeek-V3.2 TP+MTP)	2026-07-23T00:21
nightly-2-gpu-mi35x-glm51-mxfp4	N/A	2026-07-23T17:57
nightly-test-1-gpu-kernel	Nightly Kernel Benchmark (1-GPU)	2026-07-23T20:50
nightly-accuracy-2-gpu	N/A	2026-07-23T20:51
nightly-8-gpu-mi35x-deepseek-r1-mxfp4-tp4	Accuracy Test MI35x (DeepSeek-R1-MXFP4 TP4 MTP)	2026-07-23T22:03
nightly-4-gpu-mi35x-minimax-m25	Accuracy Test MI35x (4-GPU MiniMax-M2.5)	2026-07-23T22:17
nightly-8-gpu-mi35x-qwen3-235b-mxfp4	Accuracy Test + Performance Test MI35x (8-GPU Qwen3-235B-MXFP4)	2026-07-23T22:54

Per-Job Analysis

nightly-test-1-gpu-kernel — failed step(s): Nightly Kernel Benchmark (1-GPU)

Commit Info

• sglang: 0bdd4730a
• aiter: 9127c94a (v0.1.17.dev110+g9127c94a1, source: dev image preserved — Dev/patched version detected → skipping rebuild). Not relevant to this failure (pure Python import bug).

Failed Tests

All 12 subtests of test/registered/jit/test_transfer_mamba.py fail with the identical error NameError: name 'transfer_kv_mamba_lf_pf' is not defined at python/sglang/srt/mem_cache/memory_pool_host.py:407.

Test File	Test Function	Error	Log
test/registered/jit/test_transfer_mamba.py	test_mamba_kernel_backup_load_roundtrip (×4: page_first/page_first_direct × float16/bfloat16)	NameError: name 'transfer_kv_mamba_lf_pf' is not defined	Log
test/registered/jit/test_transfer_mamba.py	test_mamba_kernel_single_item (×4)	NameError: name 'transfer_kv_mamba_lf_pf' is not defined	Log
test/registered/jit/test_transfer_mamba.py	test_mamba_kernel_full_indices (×4)	NameError: name 'transfer_kv_mamba_lf_pf' is not defined	Log

Failure Summary

The nightly-test-1-gpu-kernel job runs 21 registered kernel tests on an AMD MI325 (ROCm) runner; all pass except test/registered/jit/test_transfer_mamba.py, which fails all 12 subtests. Every subtest raises NameError: name 'transfer_kv_mamba_lf_pf' is not defined from memory_pool_host.py:407 when the test exercises backup_from_device_all_layer(..., io_backend="kernel"). This is a name-resolution/import failure, not a numerical or kernel-runtime failure.

Regression Status

Never-passed on AMD (test has failed every completed scheduled run since it was first registered into this suite on 2026-07-16).

Recent history of test/registered/jit/test_transfer_mamba.py (test_mamba_kernel_*) in job nightly-test-1-gpu-kernel (most recent comparable completed scheduled runs, branch=main, event=schedule; verified at the test-file level by downloading each log):

Date	Run	Job	Test File Status	Failed Function	Error
Jul 22	29944275726	kernel	❌ Failed (12/12)	test_mamba_kernel_*	NameError: transfer_kv_mamba_lf_pf
Jul 21	29855050299	kernel	❌ Failed (12/12)	test_mamba_kernel_*	same
Jul 19	29697529176	kernel	❌ Failed (12/12)	test_mamba_kernel_*	same
Jul 18	29654601256	kernel	❌ Failed (12/12)	test_mamba_kernel_*	same
Jul 17	29601593052	kernel	❌ Failed (12/12)	test_mamba_kernel_*	same
Jul 16	29521595306	kernel	❌ Failed (12/12) — first observed	test_mamba_kernel_*	same
Jul 14	29355460805	kernel	➖ Not in suite	—	test file absent from kernel run
Jul 13	29273241962	kernel	➖ Not in suite	—	test file absent from kernel run

First observed failure: 2026-07-16. No passing run for this test file exists in the queryable window — the test file was not part of the kernel suite before Jul 16, and has been red on every run since. (Jul 20 run 29766761021 was cancelled, excluded.)

Failure Cluster

NameError: transfer_kv_mamba_lf_pf not imported on ROCm (CUDA-only import guard)

Facts (verified observations)

• Verbatim error, repeated 12× in the log: NameError: name 'transfer_kv_mamba_lf_pf' is not defined at ../python/sglang/srt/mem_cache/memory_pool_host.py:407 (log).
• Call chain from the traceback: test_transfer_mamba.py:218 → memory_pool_host.py:481 backup_from_device_all_layer → memory_pool_host.py:407 _copy_tensor_all_layers_lf_pf → transfer_kv_mamba_lf_pf(...). In-frame locals show io_backend = 'kernel', can_use_jit = False.
• memory_pool_host.py:41-45 gates the mamba import behind if _is_cuda: only:

  if _is_cuda:
      from sglang.kernels.ops.mamba.transfer_mamba import (
          transfer_kv_mamba_lf_pf, transfer_kv_mamba_pf_lf,
      )

  The sgl_kernel.kvcacheio import one block above (memory_pool_host.py:31) correctly uses if _is_cuda or _is_hip:.
• is_cuda() (python/sglang/srt/utils/common.py:145) returns torch.cuda.is_available() and torch.version.cuda is not None; on ROCm torch.version.cuda is None ⇒ _is_cuda = False, _is_hip = True. The if _is_cuda: block is therefore skipped on AMD, leaving both transfer_kv_mamba_lf_pf and transfer_kv_mamba_pf_lf undefined at module scope.
• Both call sites reach these names unconditionally on the io_backend == "kernel" branch: _copy_tensor_all_layers_lf_pf (line 407, backup) and _copy_tensor_all_layers_pf_lf (line 364, load). The test forces io_backend="kernel".
• The runner is AMD (linux-mi325-1gpu-sglang, GPU_ARCH=mi30x). The other 20 kernel tests in the same job passed (TIMINGS block), so this is isolated to the mamba transfer path, not an environment/runner problem.
• Test-file scope: on Jul 13-14 runs test_transfer_mamba.py was absent (0 mentions in log); it first appears Jul 16 with the identical 12 NameErrors. It is registered via register_amd_ci(est_time=10, suite="nightly-amd-kernel-1-gpu", nightly=True) in the test file.

Hypothesised Causes (with confidence)

• [FACT] The mamba JIT transfer symbols are never imported on ROCm because the import is gated behind if _is_cuda: (memory_pool_host.py:41), while the kernel code path (_copy_tensor_all_layers_lf_pf, line 407) calls them unconditionally for io_backend="kernel".
  • Supporting: import guard reads if _is_cuda:; is_cuda() is False on ROCm (source-verified); traceback shows the name is undefined exactly at the guarded call; the sibling sgl_kernel.kvcacheio import on the line above uses if _is_cuda or _is_hip: and its symbols resolve fine on AMD; all 12 failures are NameError on this one symbol.
  • Disconfirming: (none found) — the evidence is a directly observable Python name-resolution bug, not a hypothesis about numerics or timing.
  • How to verify: on an AMD ROCm box, python -c "import torch; from sglang.srt.mem_cache import memory_pool_host as m; print(hasattr(m,'transfer_kv_mamba_lf_pf'))" prints False; changing the guard to if _is_cuda or _is_hip: (or otherwise importing the JIT symbols on HIP) and re-running test/registered/jit/test_transfer_mamba.py on MI325 resolves it — provided the JIT-compiled transfer_mamba kernel builds under ROCm/hipify (the module uses load_jit(...) content-addressed JIT, so this second half must also be validated, not assumed).

In-flight Fix Check

• In-flight fix: ✅ #31741 "[AMD] Enable mamba JIT transfer kernel on ROCm (fix transfer_kv_mamba NameError)" by kangwangamd, open since 2026-07-20. Its description names the exact root cause (CUDA-only import guard on memory_pool_host.py) and the exact symptom (12 subtests, first seen Jul-16, never-passed on AMD). However mergeable_state: dirty (merge conflict) — it needs a rebase before it can land. Chase the author to rebase + reviewers to merge; do NOT open a duplicate.
• Search used: is:pr is:open transfer_kv_mamba and is:pr is:open transfer_mamba on sgl-project/sglang (both return only #31741).

Suggested Triage Steps

• Ask #31741 author to rebase onto current main (currently dirty/conflicting), then re-run nightly-test-amd.yml → nightly-test-1-gpu-kernel to confirm test_transfer_mamba.py goes green on MI325.
• Verify the fix covers BOTH branches: the import guard at memory_pool_host.py:41 (if _is_cuda: → include HIP) AND that the JIT transfer_mamba kernel (load_jit("transfer_mamba", ...)) actually compiles/runs under ROCm — the NameError is only the import half; a green run must prove the kernel executes on gfx942.
• If the ROCm JIT build is not yet ready, an interim option is to skip/guard the io_backend="kernel" mamba path on HIP so the suite reflects an intentional gap rather than a NameError — a maintainer decision, not a directive.

Status

• Persistence: Failing every completed scheduled run since 2026-07-16 (7 days); never passed on AMD.
• Scope: test/registered/jit/test_transfer_mamba.py (12 subtests) in nightly-test-1-gpu-kernel on AMD ROCm. AMD-only — the CUDA path imports the symbols normally. The other 20 kernel tests in the job pass.
• Blocked work: Mamba KV-cache HiCache io_backend="kernel" transfer (backup + load) is non-functional on ROCm; blocks AMD kernel-suite CI signal for this test file.
• In-flight fix: ✅ #31741 (open since 2026-07-20, currently conflicting — needs rebase).

nightly-2-gpu-mi35x-glm51-mxfp4 — failed step(s): N/A

Commit Info

• sglang: 0bdd4730af8c215545821049ea9f794519c13fa8
• aiter: N/A — today's job log is unavailable (see Facts), so [CI-AITER-CHECK] markers could not be read. This is nightly-test-amd.yml (no AITER_COMMIT_OVERRIDE), so aiter = Dockerfile default.

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi35x/test_glm51_mxfp4_tp2_gsm8k_mi35x.py	TestGLM51MXFP4TP2GSM8KMI35x.setUpClass (server launch)	Server fails to launch on MI355X — HW Exception … GPU Hang (exit 134) / Scheduler watchdog timeout → Server failed to start (exit 255). Test body test_gsm8k never runs.	Job

⚠️ Today's (Jul 22) job log returned BlobNotFound (run 29944275726 is still in progress; the failed job's log blob is not yet finalized). The test-file identity above is inferred from this job's fixed suite (nightly-amd-2-gpu-mi35x-glm51-mxfp4, a single test file) plus the identical failure on Jul 13/14/16. Today's exact error string is unverified.

Failure Summary

The nightly-2-gpu-mi35x-glm51-mxfp4 job runs one test file: a GSM8K accuracy gate that first launches amd/GLM-5.1-MXFP4 at TP=2 on a 2-GPU MI355X (gfx950) runner. Since Jul 13 the server never becomes healthy — it dies during model load/warmup with either a GPU-Hang HW exception (exit 134) or a scheduler watchdog timeout (exit 255) — so setUpClass fails and the GSM8K eval never executes. This is a server-launch failure, not the accuracy drop this test was originally written to guard.

Regression Status

New regression — began Jul 13, 2026 (last verified green Jul 5). Persistent for ~9 days.

Recent history of test_glm51_mxfp4_tp2_gsm8k_mi35x.py in job nightly-2-gpu-mi35x-glm51-mxfp4 (comparable runs: branch=main, event=schedule, completed only):

Date	Run	Test File Status	Signature	Verified?
Jul 22	run	❌ Failed (job-level)	log unavailable (BlobNotFound)	job-level only
Jul 21	run	❌ Failed (job-level)	log 404 (expired)	job-level only
Jul 20	run	⏭️ run cancelled	—	not comparable
Jul 19	run	❌ Failed (job-level)	log 404 (expired)	job-level only
Jul 18	run	❌ Failed (job-level)	log 404 (expired)	job-level only
Jul 17	run	❌ Failed (job-level)	log 404 (expired)	job-level only
Jul 16	run	❌ Failed	watchdog timeout → Server failed to start (exit 255)	✅ verified
Jul 15	run	⏭️ run cancelled	—	not comparable
Jul 14	run	❌ Failed	GPU Hang during load_model (exit 134)	✅ verified
Jul 13	run	❌ Failed	GPU Hang (exit 134)	✅ verified
Jul 6–12	(7 runs)	⏭️ all cancelled	—	not comparable
Jul 5	run	✅ Passed	score 0.937 ≥ 0.92	✅ verified
Jul 4	run	✅ Passed	score 0.943 ≥ 0.92	✅ verified

• First verified failure: Jul 13 2026. Last verified pass: Jul 5 2026 (score 0.937).
• The Jul 6–12 scheduled runs were all cancelled, so the true regression window cannot be narrowed below Jul 5 → Jul 13 via CI history.

Failure Cluster

GLM-5.1-MXFP4 TP=2 server fails to launch on MI355X (GPU Hang / watchdog timeout during model load/warmup) — a single symptom cluster with two surface manifestations (exit 134 GPU Hang, exit 255 watchdog/server-failed-to-start).

Facts (verified observations)

• Today's failed-job log is unavailable: curl …/jobs/89005587693/logs → HTTP 404 <Error><Code>BlobNotFound>. The parent run is still status=queued (42 jobs, several queued/in_progress at snapshot).
• Jul 13 log (86895900332): HW Exception by GPU node-2 … reason :GPU Hang, First attempt failed with exit code 134, retried under HF_HUB_OFFLINE=1, hung again → exit code 134.
• Jul 14 log (87161842082): watchdog thread stack in model_runner.py:841 load_model → :485 initialize, then GPU Hang, exit 134.
• Jul 16 log (87843850927): [TP1] Scheduler watchdog timeout (watchdog_timeout=300, soft=True) → TimeoutError: Server failed to start within the timeout period, FAILED (errors=1), exit 255.
• Jul 4/5 passed with GSM8K scores 0.943 / 0.937 (threshold 0.92) — same test file, same launch command.
• Reproduces across 7 distinct runners (Jul 13→22: 98ncc-r7wsc, 98ncc-c2r6s, b9rh2-g9dlh, and 4× glhqm-* pods). Jul 4/5 green runs used 98ncc pods too.
• GLM-5.1-MXFP4-specific: sister 2-GPU MI35x job nightly-2-gpu-mi35x-deepseek-r1-mxfp4-tp2 (89005587730) succeeded in the same run today; nightly-1-gpu-mi35x also passed.
• Launch command (verified from Jul 14/16 logs): --tp 2 --kv-cache-dtype fp8_e4m3 --disable-radix-cache --dsa-prefill-backend tilelang --dsa-decode-backend tilelang --model-loader-extra-config '{"enable_multithread_load": true, "num_threads": 8}' --tokenizer-worker-num 4, attention_backend='dsa'.

Hypothesised Causes (with confidence)

• [FACT] Not a single-runner hardware fault. The hang reproduces on 7 different runner pods across 4 runner pools. How to verify: already verified via runner_name per job.
• [LOW] A commit in the Jul 5 → Jul 13 window broke GLM-5.1-MXFP4 TP=2 launch on gfx950 (model-load/warmup path — weight loading, DSA/tilelang backend init, or fp8 KV cache init).
  • Supporting: clean green→red transition (0.937 on Jul 5 → GPU Hang on Jul 13); reproduces fleet-wide (software, not hardware); isolated to this model+config (deepseek-r1-mxfp4 TP=2 passes).
  • Disconfirming: the window is 283 commits (Jul 5→13, verified via compare API) and cannot be narrowed because Jul 6–12 runs were all cancelled; no single commit implicated; today's log is unavailable so the exact Jul 22 signature is unconfirmed.
  • How to verify: bisect 48ba79c11e..b44ac5d49a launching amd/GLM-5.1-MXFP4 --tp 2 on an MI355X box; grep the window for changes to model loading, DSA tilelang backend, fp8 KV-cache init, or GLM5 model code.
• [SPECULATION] A model-side / recipe change (weights, tokenizer, or required launch flags for GLM-5.1-MXFP4 on gfx950). Open PRs #29873, #31324, #30808 all adjust GLM-5.1 MXFP4 flags/paths on MI355X, hinting the launch recipe is in active flux.
  • Disconfirming: no evidence the CI launch command changed between Jul 5 and Jul 13; purely associative.

In-flight Fix Check

• In-flight fix: ❌ none found for this GPU-Hang / server-launch failure. Searched open PRs for GLM-5.1-MXFP4, GPU+Hang+mi35, glm51+mxfp4 (via search/issues). Matches (#29873, #31324, #30808, #32120) are GLM-5.1/5.2 MXFP4 feature/recipe/doc PRs, not fixes targeting this launch hang.

Suggested Triage Steps

• Re-run this job on a fresh MI355X runner to rule out a transient fleet-wide GPU state issue; if it still hangs, the software-regression hypothesis stands.
• Bisect the 48ba79c11e..b44ac5d49a window (283 commits) by launching sglang serve --model-path amd/GLM-5.1-MXFP4 --tp 2 --kv-cache-dtype fp8_e4m3 --disable-radix-cache --dsa-prefill-backend tilelang --dsa-decode-backend tilelang on gfx950 and checking for the load/warmup hang. Prioritize commits touching model loading, DSA/tilelang backends, fp8 KV cache, or GLM5 model code.
• To decouple variables, try launching with --attention-backend other than dsa and without enable_multithread_load to see whether the hang is DSA/tilelang- or multithread-loader-specific.
• Once the run completes, re-fetch today's job log (89005587693) to confirm the Jul 22 signature matches the cluster (exit 134 vs 255).

Status

• Persistence: Failing every verified completed run since Jul 13 2026 (~9 days); last green Jul 5 (score 0.937). Jul 6–12 and Jul 15/20 runs were cancelled.
• Scope: 1 job (nightly-2-gpu-mi35x-glm51-mxfp4) in nightly-test-amd.yml. GLM-5.1-MXFP4-specific — other MI35x 1-/2-GPU jobs in the same run pass. Reproduces across 7 runners → software regression, not hardware.
• Blocked work: No AMD MI355X CI signal for GLM-5.1-MXFP4 TP=2 accuracy since Jul 13.
• In-flight fix: ❌ none found in open PRs.

nightly-accuracy-2-gpu — failed step(s): N/A

Commit Info

• sglang: 0bdd4730a
• aiter: 9127c94a1 (v0.1.17.dev110+g9127c94a1) — source: dev/patched image preserved (Dockerfile default, no override; [CI-AITER-CHECK] "Dev/patched version detected → skipping rebuild")

Failed Tests

The job runs a single accuracy test file (test_gsm8k_eval_amd.py) plus test_dumper.py. The accuracy test is one function (test_gsm8k_all_models) that loops over many models; it raises one AssertionError at the end listing every model whose score fell below its threshold.

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi30x/test_gsm8k_eval_amd.py	TestNightlyGsm8KEval.test_gsm8k_all_models (subtest mistralai/Mixtral-8x7B-Instruct-v0.1)	Score 0.286 < threshold 0.69 (accuracy failure, TP=2)	Log
test/registered/amd/accuracy/mi30x/test_gsm8k_eval_amd.py	TestNightlyGsm8KEval.test_gsm8k_all_models (subtest Qwen/Qwen3-4B)	Status: ❌ FAILED printed with threshold: N/A (score 0.912) — config gap, see note	Log

Failure Summary

The nightly-accuracy-2-gpu job runs GSM8K accuracy across a model matrix. In this run, Llama-3.1-8B (0.820), Qwen3-8B (0.917) and Llama-3.1-70B (0.932) passed; Mixtral-8x7B-Instruct-v0.1 scored 0.286 against its 0.69 threshold (TP=2) and entered the 3-attempt retry loop; Qwen/Qwen3-4B scored 0.912 but printed ❌ FAILED because no threshold is defined for it. The retrievable job log is truncated (~18.2k lines) partway through Mixtral's retries — flooded by [aiter] shape is … GEMM logging — so the final check_model_scores summary and the AssertionError body are not present in the downloadable log. The job ran ~55 min (18:52→19:47 UTC), well under the default step timeout, and exited failure.

Regression Status

Known recurring failure. The nightly-accuracy-2-gpu job has concluded failure on every completed scheduled run in the queryable window. I verified the GSM8K test-file signature (Qwen3-4B threshold: N/A → ❌ FAILED; Mixtral eval region) directly on the two most recent completed runs; earlier runs are shown at job-conclusion level because their logs truncate identically at Mixtral.

Recent history of test_gsm8k_eval_amd.py (test_gsm8k_all_models) in job nightly-accuracy-2-gpu (scheduled, main, completed runs only):

Date	Run	Test File Status	Detail
Jul 22	29944275726	❌ Failed	Qwen3-4B threshold N/A; Mixtral 0.286<0.69 (verified)
Jul 21	29855050299	❌ Failed	Qwen3-4B threshold N/A (verified); Mixtral region truncated
Jul 19	29697529176	❌ Failed (job-level)	log truncates at Mixtral
Jul 18	29654601256	❌ Failed (job-level)	not test-file-verified
Jul 17	29601593052	❌ Failed (job-level)	not test-file-verified
Jul 16	29521595306	❌ Failed (job-level)	not test-file-verified

Note: Jul 20 (5ab3d90b81) and Jul 15 were cancelled (excluded). No green run for this job exists in the window → treat as persistently failing.

Failure Cluster

GSM8K nightly accuracy assertion — Mixtral-8x7B below threshold + Qwen3-4B missing-threshold misconfiguration.

Facts (verified observations)

• test_gsm8k_eval_amd.py registers one file into the nightly-amd suite (register_amd_ci(est_time=3600, suite="nightly-amd", nightly=True)); the job's suite listing shows only this file + test_dumper.py (log lines ~1803-1804).
• Verified scores this run: Llama-3.1-8B 0.820 (thr 0.80) ✅; Qwen3-8B 0.917 (thr 0.76) ✅; Qwen3-4B 0.912 (thr N/A) ❌; Llama-3.1-70B 0.932 (thr 0.89) ✅; Mixtral-8x7B 0.286 then retrying (log lines 5946, 8244, 12193, 15101, 17520).
• Code fact: MODEL_SCORE_THRESHOLDS (test_gsm8k_eval_amd.py:37-67) contains Qwen/Qwen3-8B and Qwen/Qwen3-30B-A3B-Thinking-2507 but no Qwen/Qwen3-4B, while Qwen/Qwen3-4B IS in DEFAULT_MODEL_NAME_FOR_NIGHTLY_EVAL_TP1 (test_utils.py:154). Pickaxe confirms Qwen/Qwen3-4B never had a threshold entry.
• Code fact: check_model_scores (test_gsm8k_eval_amd.py:165-177) maps a missing threshold to ⚠️ NO THRESHOLD and does not append it to failed_models; only score < threshold models trigger the AssertionError (line 199-201). The per-model ❌ FAILED print (line 289-301) uses passed = threshold and score >= threshold, so threshold=None prints FAILED cosmetically — a separate code path from the job-failing assertion.
• aiter is a dev build v0.1.17.dev110+g9127c94a1, unchanged (no override).
• Transient Health check failed. Server couldn't get a response from detokenizer for last 20 seconds at 19:03:48 during the first model's startup; that model (Llama-3.1-8B) still passed.
• Log truncation is consistent across runs (current ~18256 lines, Jul-21 ~18275 lines), both ending inside Mixtral's [aiter] shape is … spam.

Hypothesised Causes (with confidence)

• [HIGH] The job-failing AssertionError is driven by real accuracy failures, with Mixtral-8x7B (0.286 ≪ 0.69, TP=2) as a confirmed contributor.
  • Supporting: Mixtral's first attempt scored 0.286, far below 0.69; per code, this appends to failed_models → AssertionError. Job exited failure under timeout.
  • Disconfirming: the log truncates before check_model_scores, so I cannot see the final failed-model list; other untruncated models (Qwen2-57B-A14B, FP8 groups, AMD-specific groups) may also contribute or Mixtral could conceivably recover on a retry (unlikely from 0.286).
  • How to verify: re-run the job and capture the full step log (or github_summary.md artifact / step summary), or run test_gsm8k_eval_amd.py locally on MI325 restricted to Mixtral.
• [FACT] Qwen/Qwen3-4B has no entry in MODEL_SCORE_THRESHOLDS, producing a misleading ❌ FAILED (threshold: N/A) per-model line.
  • Supporting: direct code read + pickaxe (never had a threshold); identical print on Jul-21.
  • Disconfirming (important): by code (check_model_scores line 166-168) this does not contribute to the AssertionError, so it is a reporting/config bug, not the cause of the job failure.
  • How to verify: add "Qwen/Qwen3-4B": <thr> to MODEL_SCORE_THRESHOLDS; the per-model line flips to PASS while job pass/fail is unchanged.
• [LOW] Mixtral low score is an AMD-backend correctness issue rather than a threshold-too-strict issue.
  • Supporting: 0.286 is far below the reference (~0.744) — not a marginal miss.
  • Disconfirming: no code-level evidence tying it to a specific commit; aiter unchanged; decode batches ran normally in the log.
  • How to verify: bisect Mixtral GSM8K score across recent sglang commits on MI325; inspect MoE/aiter path for Mixtral bf16.

In-flight Fix Check

• Qwen3-4B threshold gap: In-flight fix: ❌ none found — searched open PRs Qwen3-4B threshold (4 unrelated hits) and gsm8k threshold (no matching fix).
• Mixtral accuracy: In-flight fix: ❌ none found — searched open PRs Mixtral accuracy (12 unrelated hits). Note #29933 "fix(eval): extract GSM8K answer from first #### marker" is open and touches GSM8K scoring extraction but is not a targeted fix for either symptom.

Suggested Triage Steps

• Capture the full GSM8K summary: re-run nightly-accuracy-2-gpu and read github_summary.md / the GitHub step summary (the check_model_scores table lists exactly which models failed) — the downloadable job log is truncated by aiter GEMM logging and hides the final assertion.
• Confirm the Mixtral failure: run test_gsm8k_eval_amd.py on linux-mi325-2gpu-sglang restricted to mistralai/Mixtral-8x7B-Instruct-v0.1 (TP=2); if 0.286 reproduces, investigate the AMD MoE/aiter path (or reassess whether 0.69 is appropriate for AMD).
• Close the config gap: add a Qwen/Qwen3-4B entry to MODEL_SCORE_THRESHOLDS in test_gsm8k_eval_amd.py:37 so its result stops printing a misleading ❌ FAILED.
• Consider suppressing the per-forward [aiter] shape is … not found tuned config … using torch solution:0 spam (or tuning a config CSV) so future job logs aren't truncated before the summary.

Status

• Persistence: nightly-accuracy-2-gpu has failed on every completed scheduled run in the window (Jul 16→22); no green run found.
• Scope: this job (test_gsm8k_eval_amd.py) on linux-mi325-2gpu-sglang. Confirmed hard-failure contributor: Mixtral-8x7B (0.286<0.69). Separate cosmetic config bug: Qwen3-4B missing threshold.
• Blocked work: AMD nightly GSM8K accuracy signal for the TP=2 model matrix.
• In-flight fix: ❌ none found for either the Mixtral accuracy failure or the Qwen3-4B threshold gap.

nightly-4-gpu — failed step(s): Nightly Test (4-GPU)

Commit Info

• sglang: 0bdd4730af8c215545821049ea9f794519c13fa8 (#32091 "[CI] Fix failures on main")
• aiter: 9127c94a18e4398e1eba91f6639e910f0994ad02 — v0.1.17.dev110+g9127c94a1 (source: dev image detected, rebuild skipped; matches Dockerfile default)

Failed Tests

Test File	Test Function	Error	Log
test/registered/vlm/test_encoder_dp.py	TestVLMEncoderDP.test_vlm_mmmu_benchmark	✗ TIMEOUT after 3600s (model drawn: zai-org/GLM-4.1V-9B-Thinking; est_time=500s)	Log

Failure Summary

The single test in job nightly-4-gpu, test_encoder_dp.py, exceeded the harness's 3600s wall-clock cap and was killed (exit code 255). The server launched successfully (ready at 18:06:52) and served requests continuously — 545 /v1/chat/completions calls all returned 200 OK right up to the kill at 19:03:32 — so this is a slow-progress timeout, not a crash or assertion. The run randomly selected zai-org/GLM-4.1V-9B-Thinking for the MMMU benchmark and decode throughput stayed abnormally low (peak ~8 tok/s, dipping to 1.28).

Regression Status

Known recurring / flaky performance-driven failure (not a fresh regression). This test file has failed intermittently across the queried window with two distinct signatures; the current TIMEOUT signature also occurred on 7-19.

Recent history of test/registered/vlm/test_encoder_dp.py (test_vlm_mmmu_benchmark) in job nightly-4-gpu (most recent comparable completed scheduled runs: branch=main, event=schedule; cancelled 7-20 & 7-15 excluded):

Date	Run / Job	Model drawn (random.choice)	Test File Status	Elapsed	Error
Jul 22	job	GLM-4.1V-9B-Thinking	❌ Failed	3600	TIMEOUT
Jul 21	job	Qwen3-VL-32B-Instruct	✅ Passed	632	—
Jul 19	job	GLM-4.1V-9B-Thinking	❌ Failed	3600	TIMEOUT
Jul 18	job	InternVL2_5-8B	✅ Passed	398	—
Jul 17	job	(n/a)	✅ Passed	453	—
Jul 16	job	(n/a)	✅ Passed	454	—
Jul 14	job	Qwen2.5-VL-72B-Instruct	✅ Passed	1483	—
Jul 13	job	GLM-4.1V-9B / Qwen3-VL-32B	❌ Failed	1340	exit 1 (server exit -9)
Jul 05	job	—	❌ Failed	1797	exit 1 (server exit -9)
Jul 04	job	—	❌ Failed	1540	exit 1 (server exit -9)
Jul 03	job	—	✅ Passed	397	—
Jul 02	job	—	✅ Passed	412	—

First observed failure for this test file in window: Jul 04 (exit-1/SIGKILL cluster). TIMEOUT signature specifically: Jul 19 and Jul 22. Last known passing: Jul 21 (632s, Qwen3-VL-32B).

Failure Cluster

VLM encoder-DP MMMU benchmark timeout (GLM-4.1V-9B-Thinking, slow decode). Note a separate co-located cluster on the same test file (Jul 04/05/13): server process SIGKILL (exit -9) — likely OOM, distinct symptom, not the current failure.

Facts (verified observations)

• ✗ TIMEOUT: .../test_encoder_dp.py after 3600 seconds; Test Summary: 0/1 passed; wrapper exit code 255 (log tail, run 29944275726).
• Server reached ready state (The server is fired up and ready to roll! at 18:06:52) and served 545 chat/completions requests, all 200 OK, through 19:03:14 — continuous forward progress, no crash/stack trace.
• Model drawn this run: zai-org/GLM-4.1V-9B-Thinking (verified from served_model_name). The test runs random.choice(MODELS) in CI (test_encoder_dp.py:33-34); MODELS = {Qwen2.5-VL-72B, Qwen3-VL-32B, InternVL2_5-8B, GLM-4.1V-9B-Thinking}.
• Both TIMEOUT runs (Jul 19, Jul 22) drew GLM-4.1V-9B-Thinking; all passing runs drew non-thinking models (398s InternVL-8B, 453/454s, 632s Qwen3-32B, 1483s Qwen2.5-72B).
• Decode throughput this run: peak 8.14 tok/s, min 1.28 tok/s. Fast passing run (Jul 18, 398s) peaked at 22.34 tok/s — ~3x faster.
• aiter bf16 GEMM fell back to using torch solution:0 (no tuned config in /tmp/aiter_configs/bf16_tuned_gemm.csv): 3137 occurrences this run vs 2016–4860 in passing runs — fallback is a normal CI state, present regardless of pass/fail.
• Config: tp=4, --mm-enable-dp-encoder, --cuda-graph-max-bs-decode 32, mem_fraction_static=0.68, attention_backend='aiter'.
• One transient Health check failed. ... detokenizer for last 20 seconds at 18:07:12, self-recovered (200 OK by 18:07:25).

Hypothesised Causes (with confidence)

• [MEDIUM] The timeout is driven by which model random.choice draws — specifically GLM-4.1V-9B-Thinking, a reasoning model whose long "thinking" generations, over the full MMMU set at ~8 tok/s AMD decode, exceed the 3600s harness cap.
  • Supporting: random.choice(MODELS) is in the test source (test_encoder_dp.py:34); 2/2 timeout runs drew GLM-thinking while every non-thinking draw passed (even the 72B at 1483s); the run served requests continuously (slow, not stuck).
  • Disconfirming: only 2 timeout data points; no observed run where GLM-thinking was drawn and finished under budget, so "GLM-thinking always times out" is not yet proven — the low ~8 tok/s throughput may be the dominant factor and GLM-thinking merely the most decode-heavy draw.
  • How to verify: pin models_to_test=[GLM-4.1V-9B-Thinking] and run test_encoder_dp.py on an MI30x runner; time it. Compare against a pinned InternVL2_5-8B run.
• [LOW] Runner-level performance variance (GPU contention / thermal) causing the ~3x decode slowdown (22→8 tok/s) independent of model choice.
  • Supporting: absolute throughput is low even in passing runs; head SHA differs each day yet fast/slow does not track any single commit.
  • Disconfirming: the pass/fail split correlates cleanly with model drawn, not obviously with runner identity (runner not captured here).
  • How to verify: repeat-run the same model on the same runner to measure throughput spread.
• [LOW] aiter bf16 GEMM using torch solution:0 fallback slowing decode.
  • Supporting: thousands of fallback lines; GEMMs run the slow default torch path.
  • Disconfirming: fallback count is equal-or-higher in passing runs (2016–4860) than timeouts (3137) — not the differentiator.
  • How to verify: supply a tuned bf16_tuned_gemm.csv and re-measure; not expected to change the outcome.

In-flight Fix Check

• In-flight fix: ❌ none found targeting this test. Searches: encoder_dp (0 open PRs), GLM-4.1V+timeout (0). Related but not a fix: #31735 "chore: update CI test est_time values" (open since 2026-07-20) adjusts est_time scheduling hints broadly but does not touch test_encoder_dp.py in its first 100 files, and est_time is a scheduler hint — it does not change the 3600s hard wall-clock cap. #32118 is a nightly-CI fix but unrelated to this VLM timeout.

Suggested Triage Steps

• Confirm the model-draw hypothesis: run test_encoder_dp.py on MI30x with models_to_test pinned to GLM-4.1V-9B-Thinking vs InternVL2_5-8B, comparing elapsed time against the 3600s cap.
• If GLM-thinking is the culprit, consider bounding generation for the thinking model in the MMMU kit (e.g. cap max_new_tokens/thinking length) or excluding GLM-4.1V-9B-Thinking from the random pool for the AMD nightly-4-gpu suite until decode throughput improves.
• Investigate the ~3x decode-throughput variance (22 vs 8 tok/s) on the MI30x runner: capture runner ID and check for co-scheduled jobs / thermal throttling across the fast vs slow runs.
• Track the separate exit-9 (SIGKILL/OOM) cluster on the same file (Jul 04/05/13) independently — likely a distinct memory issue, not addressed by any timeout fix.

Status (factual)

• Persistence: Test file intermittently failing since Jul 04 in the queried window; the current TIMEOUT signature seen Jul 19 and Jul 22. Interleaved with passes (last green Jul 21, 632s).
• Scope: Job nightly-4-gpu in nightly-test-amd.yml; single test file. Not observed spilling into other jobs from this data.
• Blocked work: VLM encoder-DP MMMU nightly signal on AMD is unreliable — a green result depends on which model random.choice happens to draw.
• In-flight fix: ❌ none found — needs new investigation/PR.

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-07-23 23:33 UTC)

[amd-bot]

nightly-test-amd.yml — Per-Job Analysis (continued 2/3)

nightly-accuracy-2-gpu-vlm — failed step(s): Nightly Accuracy Test (2-GPU VLM MMMU)

Commit Info

• sglang: 0bdd4730
• aiter: 9127c94a (v0.1.17.dev110+g9127c94a1; source: dev/patched CI image, matches Dockerfile default — no override)

Failed Tests

The job runs one test function (test_mmmu_vlm_models) that loops over 11 VLMs as subtests; 5 subtests scored below their configured MMMU accuracy thresholds, so the aggregate test raised an AssertionError.

Test File	Test Function (subtest model)	Error	Log
test/registered/amd/accuracy/mi30x/test_vlms_mmmu_eval_amd.py	test_mmmu_vlm_models — Qwen/Qwen2-VL-7B-Instruct	score=0.2799 < threshold=0.30	Log
test/registered/amd/accuracy/mi30x/test_vlms_mmmu_eval_amd.py	test_mmmu_vlm_models — Qwen/Qwen2.5-VL-7B-Instruct	score=0.2901 < threshold=0.33	Log
test/registered/amd/accuracy/mi30x/test_vlms_mmmu_eval_amd.py	test_mmmu_vlm_models — Qwen/Qwen3-VL-30B-A3B-Instruct	score=0.2700 < threshold=0.29	Log
test/registered/amd/accuracy/mi30x/test_vlms_mmmu_eval_amd.py	test_mmmu_vlm_models — openbmb/MiniCPM-o-2_6	score=0.2802 < threshold=0.32	Log
test/registered/amd/accuracy/mi30x/test_vlms_mmmu_eval_amd.py	test_mmmu_vlm_models — deepseek-ai/deepseek-vl2-small	score=0.2802 < threshold=0.31	Log

Job exited with code 255 (Test Summary: 0/1 passed). The other 6 models passed (InternVL2_5-2B, MiniCPM-v-2_6, Janus-Pro-7B, Kimi-VL-A3B-Instruct, MiMo-VL-7B-RL, DeepSeek-OCR-2); GLM-4.1V-9B-Thinking is excluded via AMD_FAILING_VLM_MODELS.

Failure Summary

test_mmmu_vlm_models failed because 5 of 11 VLMs produced MMMU accuracy below their per-model thresholds set in AMD_VLM_MODELS. The failing assertion is raised at test_vlms_mmmu_eval_amd.py:314 → _check_results at line 380. The eval itself ran to completion for every model (scores were produced normally, no server crash or eval error); this is purely an accuracy-below-threshold failure.

Regression Status

Known recurring failure (not new today, not flaky). The exact same 5 models fail with virtually identical scores in every completed scheduled run of this job in the queryable window. No passing run for these 5 models was found — effectively Never-passed for these specific subtests.

Recent history of test_vlms_mmmu_eval_amd.py (test_mmmu_vlm_models) in job nightly-accuracy-2-gpu-vlm (completed event=schedule, branch=main):

Date	Run	Test File Status	Failed Models (score/threshold)
Jul 22	1002	❌ Failed	Qwen2-VL 0.28/0.30 · Qwen2.5-VL 0.29/0.33 · Qwen3-VL-30B 0.27/0.29 · MiniCPM-o 0.28/0.32 · deepseek-vl2 0.28/0.31
Jul 21	999	❌ Failed	same 5, same scores
Jul 20	997	⏭️ Cancelled	(run cancelled — excluded)
Jul 19	978	❌ Failed	same 5, same scores
Jul 18	976	❌ Failed	same 5, same scores

First observed failure for these models: at least 2026-07-18 (earliest completed run in window; runs before this were cancelled). No passing run found in the queryable window. Scores are stable across all 4 runs (e.g. Qwen2-VL always ~0.28, Qwen3-VL-30B always 0.26–0.27), confirming this is deterministic, not flaky.

Failure Cluster

VLM MMMU accuracy below configured threshold on MI30x (5 specific models consistently score ~0.27–0.29)

Facts (verified observations)

• Aggregate assertion: AssertionError: The following models failed: listing the 5 models with score/threshold, raised at test_vlms_mmmu_eval_amd.py:380 (stack trace verbatim in log lines 15247–15257).
• Thresholds for the 5 failing models (git blame, verified): Qwen2-VL 0.30, Qwen2.5-VL 0.33, Qwen3-VL-30B 0.29, MiniCPM-o 0.32, deepseek-vl2-small 0.31 (test_vlms_mmmu_eval_amd.py:50,56,62,82,89).
• Observed scores this run: 0.2799 / 0.2901 / 0.2700 / 0.2802 / 0.2802 respectively — all just below threshold.
• Score stability across 4 completed runs (07-18, 07-19, 07-21, 07-22): the same 5 models fail with the same scores each time; the same 6 models pass each time.
• git blame shows all thresholds carry boundary commit e856eae9; this is the shallow-clone history boundary (^ prefix), NOT proof the values were introduced there — run 976 (07-18) already used the identical thresholds, predating that commit's tree in-window.
• aiter version identical (9127c94a) in the passing-model and failing-model portions of the same run — no aiter variable in play.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [MEDIUM] Mis-calibrated (too-high) per-model thresholds for these 5 VLMs on MI30x with the 100-sample MMMU subset. The 5 models consistently land ~0.27–0.29 while their thresholds (0.29–0.33) sit above that band; the 6 passing models clear their (lower or better-matched) thresholds comfortably.
  • Supporting: deterministic scores across 4 runs; several models miss by a razor-thin margin (Qwen2-VL 0.2799 vs 0.30; Qwen3-VL 0.27 vs 0.29); no eval/server error — the pipeline works, only the numeric gate fails.
  • Disconfirming: cannot distinguish "threshold too high" from "genuine AMD accuracy deficit" from CI alone; if these thresholds were chosen from a known-good NVIDIA/AMD baseline, the low scores could be a real model-quality regression rather than a bad threshold.
  • How to verify: run the same models on MI30x locally (or an NVIDIA baseline) and compare MMMU-100 scores; if NVIDIA also scores ~0.28, the threshold is simply too high for 100 samples; if NVIDIA scores higher, investigate an AMD-specific accuracy path.
• [LOW] Genuine AMD/ROCm VLM accuracy issue for these 5 architectures (Qwen2/2.5/3-VL, MiniCPM-o, deepseek-vl2) — e.g. vision-encoder or attention-backend numerics on MI30x pulling MMMU toward random chance (~0.25).
  • Supporting: the failing set skews toward Qwen-VL family + two others, hinting at a shared code path; scores cluster tightly near random-chance.
  • Disconfirming: 6 other VLMs (including MoE Kimi-VL, MiMo-VL, deepseek-vl2's sibling Janus-Pro-7B) pass, so no blanket VLM breakage; scores are stable, not degrading, which argues against a recent code regression.
  • How to verify: bisect is not useful (no green run in window); instead compare MMMU-100 scores for one failing model (e.g. Qwen2-VL-7B) on NVIDIA vs AMD at this SHA.

No hypothesis reaches HIGH: there is no in-window green run to bisect against, and the shallow clone hides the pre-07-18 threshold/model-list history that would date the regression.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs with keywords VLM+MMMU+AMD, mmmu+threshold (GET /search/issues?q=repo:sgl-project/sglang+is:pr+is:open+...); no PR targets these VLM MMMU thresholds or an AMD VLM accuracy fix.

Suggested Triage Steps

• Confirm the baseline: run Qwen/Qwen2-VL-7B-Instruct and Qwen/Qwen3-VL-30B-A3B-Instruct MMMU-100 on both MI30x and an NVIDIA runner at 0bdd4730. If NVIDIA also ~0.28, the thresholds in test_vlms_mmmu_eval_amd.py:50,56,62,82,89 are simply too tight for a 100-sample eval → recalibrate on a branch.
• If NVIDIA scores materially higher, treat as an AMD VLM accuracy path issue: compare vision-encoder / attention-backend output for one failing Qwen-VL model on AMD vs NVIDIA (note TRITON_ATTENTION_MODELS handling in the test).
• Trace pre-07-18 history in a full (non-shallow) clone to date when these thresholds were set and whether the job ever passed — this determines whether the fix is a threshold recalibration or a code regression hunt.
• Consider raising MMMU sample count (currently 100) to reduce near-threshold noise for models that miss by <0.02.

Status

• Persistence: Failing every completed scheduled run in window — 07-18, 07-19, 07-21, 07-22 (07-20 run was cancelled). Same 5 models, stable scores; no green run found for these subtests.
• Scope: 1 job (nightly-accuracy-2-gpu-vlm) in nightly-test-amd.yml; 5 of 11 VLM subtests. Isolated to MI30x AMD VLM accuracy gating; other VLMs and the 2-GPU non-VLM accuracy job are separate.
• Blocked work: Blocks the AMD nightly VLM MMMU accuracy signal (job is red every night).
• In-flight fix: ❌ none found — needs a new PR (threshold recalibration or accuracy investigation).

nightly-8-gpu-mi35x-qwen3-235b-mxfp4 — failed step(s): Accuracy Test + Performance Test MI35x (8-GPU Qwen3-235B-MXFP4)

Commit Info

• sglang: 0bdd4730 — [CI] Fix failures on main (#32091)
• aiter: 9127c94a (v0.1.17.dev110+g9127c94a1, source: dev/patched image preserved — rebuild skipped, no AITER_COMMIT_OVERRIDE)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/test_qwen3_instruct_mxfp4.py	TestQwen3Instruct2507MXFP4.setUpClass	Exception: Server process exited with code 1 (Rank 0 scheduler died during init, exit code -6 / SIGABRT)	Log

Ran 0 tests in 410.399s — the server never came up, so no accuracy/perf test executed.

Failure Summary

The amd/Qwen3-235B-A22B-Instruct-2507-mxfp4 server (--tp 4 --ep 2 --attention-backend aiter, SGLANG_USE_AITER=1) loaded all 4 TP shards successfully but crashed at the very start of decode CUDA-graph capture (bs=512, 0/52 batches captured). The Rank 0 scheduler process aborted with SIGABRT (exit -6), so setUpClass failed with Server process exited with code 1 and 0 tests ran.

Regression Status

Known recurring / persistent failure (no green run found in the queryable window).

Recent history of test/registered/amd/test_qwen3_instruct_mxfp4.py (TestQwen3Instruct2507MXFP4.setUpClass) in job nightly-8-gpu-mi35x-qwen3-235b-mxfp4 (most recent comparable completed scheduled runs on main, event=schedule):

Date	Run	Test File Status	Failed Function	Error
Jul 23	29944275726	❌ Failed	setUpClass	HIP capture watchdog crash, exit -6
Jul 21	29855050299	❌ Failed	setUpClass	same signature (exit -6)
Jul 19	29697529176	❌ Failed	setUpClass	same signature (exit -6)
Jul 18	29654601256	❌ Failed	setUpClass	same signature (exit -6)
Jul 17	29601593052	❌ Failed	setUpClass	same signature (exit -6)
Jul 16	29521595306	⚠️ Cancelled	—	run cancelled (excluded)
Jul 14	29355460805	❌ Failed	setUpClass	same HIP watchdog crash, exit 134 (128+6)
Jul 13	29273241962	❌ Failed	setUpClass	same HIP watchdog crash, exit 134

First observed failure in this window: at/before Jul 13 2026. No passing run found in the queryable window (Jul 5 – Jul 23; the Jul 5 run 28749559519 failed earlier with exit 1 before reaching this test). Verdict is based on Test File Status; cancelled runs excluded.

Failure Cluster

HIP graph-capture watchdog abort (NCCL/RCCL watchdog HIP event query on a capturing stream)

Facts (verified observations)

• Weights loaded fine on all 4 ranks (Load weight end ... quant=quark, avail mem=249 GB), so this is not a load/OOM failure (log lines ~5334–5337).
• Crash occurs at capture start: Capturing batches (bs=512 avail_mem=25.14 GB): 0%|...| 0/52 immediately before the error (log line ~5356).
• Crashing frame is the torch ProcessGroupNCCL watchdog thread: c10d::ProcessGroupNCCL::WorkNCCL::finishedGPUExecutionInternal() → c10_hip_check_implementation throwing HIP error: operation not permitted when stream is capturing (hipErrorStreamCaptureUnsupported) on rank 0, and operation not permitted on an event last recorded in a capturing stream (hipErrorCapturedEvent) on rank 1 (log lines ~5410–5445). The uncaught c10::DistBackendError → terminate → Fatal Python error: Aborted (SIGABRT).
• The main thread at crash time is inside aiter custom all-reduce during graph capture: custom_all_reduce.py:503 capture → :303 flush_graph_buffers → :336 _gather_ipc_meta, called from parallel_state.py:1901/581 graph_capture → decode_cuda_graph_runner.py:854 capture (log lines ~5464–5473).
• Identical stack + error signature reproduced in 4/4 most recent completed runs and in the Jul 13/14 runs (exit 134 = 128+6). No sglang code change to the test file is implicated: last edit to the test file was a82ead53 (#31667), unrelated to the crash path.

Hypothesised Causes (with confidence)

• [MEDIUM] The aiter custom all-reduce path launches IPC/collective work during HIP graph capture (_gather_ipc_meta / flush_graph_buffers), and the torch NCCL watchdog thread concurrently polls a HIP event for that in-flight collective; on ROCm, querying an event tied to a capturing stream is illegal → hipErrorStreamCaptureUnsupported → watchdog throws → process aborts.
  • Supporting: main-thread stack is entirely inside aiter/dist/device_communicators/custom_all_reduce.py during graph_capture; the crash is a HIP-capture-specific event error; an open PR #23581 titled "Default SGLANG_USE_AITER_AR to false to avoid HIP graph capture invalidation" describes exactly this interaction; the test forces SGLANG_USE_AITER=1 with the aiter attention backend.
  • Disconfirming: the actually-crashing frame is the torch NCCL watchdog thread, not aiter code directly — aiter may only be the trigger, not the faulting caller; and because the job has no green baseline in the window, it is unproven that disabling AITER AR alone resolves it (could be a broader ROCm/torch-watchdog vs graph-capture incompatibility on this gfx950/mi35x image).
  • How to verify: on a branch, launch the same command with SGLANG_USE_AITER_AR=0 (per #23581) and/or TORCH_NCCL_ENABLE_MONITORING=0 and re-run test/registered/amd/test_qwen3_instruct_mxfp4.py on MI35x; if capture completes, aiter AR is confirmed as the trigger.
• [LOW] A regression from the aiter dev build shipped in the CI image (v0.1.17.dev110+g9127c94a1) changed the graph-capture IPC path.
  • Supporting: the image ships a non-release dev aiter; the crash is inside aiter custom all-reduce.
  • Disconfirming: the failure has persisted across many days and multiple sglang SHAs; aiter version was not shown to change between the failing runs, so no A/B evidence points at a specific aiter delta.
  • How to verify: compare the aiter SHA in an older (pre-Jul 13) passing image vs current; bisect aiter if a passing image is found.

In-flight Fix Check

• In-flight fix: ✅ #23581 "[AMD] Default SGLANG_USE_AITER_AR to false to avoid HIP graph capture invalidation" (open since 2026-04-23, last updated 2026-05-04 — stale, non-draft) — directly targets this cluster; ping reviewers rather than opening a duplicate. Possibly related: #31135 "Fix custom all-reduce deadlock when one communicator is issued from two CUDA streams" (open 2026-07-14). (Search: is:pr is:open with watchdog+capturing+stream, hipErrorStreamCaptureUnsupported, graph+capture+watchdog.)

Suggested Triage Steps

• Reproduce with AITER AR disabled: re-run the failing command with SGLANG_USE_AITER_AR=0 on an MI35x container; if the server launches, chase #23581 to merge.
• If AR-disable does not fix it, test TORCH_NCCL_ENABLE_MONITORING=0 / raising the watchdog timeout to isolate whether the watchdog poll during capture is the sole trigger.
• Look for a pre-Jul-13 CI image where this job last passed; if found, diff the aiter dev SHA to bisect the aiter side; if none exists, treat as a Never-passed MI35x port and consider quarantining the job until #23581 lands.

Status

• Persistence: Failing every completed run in the window (Jul 13 → Jul 23, 6/6 non-cancelled runs), no green run found.
• Scope: nightly-8-gpu-mi35x-qwen3-235b-mxfp4 job in nightly-test-amd.yml (MI35x / gfx950, aiter attention backend + AITER AR).
• Blocked work: Blocks the Qwen3-235B-MXFP4 accuracy + performance signal on AMD MI35x (0 tests run).
• In-flight fix: ✅ #23581 (open since 2026-04-23) — awaiting merge.

nightly-accuracy-8-gpu-mi35x-deepseek-v32-mtp — failed step(s): Accuracy Test MI35x (8-GPU DeepSeek-V3.2 TP+MTP)

Commit Info

• sglang: 0bdd4730
• aiter: 9127c94a (v0.1.17.dev110+g9127c94a1; source: dev image detected & preserved, matches Dockerfile default per [CI-AITER-CHECK])

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi35x/test_deepseek_v32_mtp_eval_mi35x.py	test_a_gsm8k	Server hang → Scheduler watchdog timeout (watchdog_timeout=1200.0, soft=False) → SIGQUIT → test then dies with requests.exceptions.ConnectionError / KeyError: 'answer' on /flush_cache (port 11000 refused)	Log (step 6)

The underlying failure is a server-side hang, not an accuracy/assertion failure. The KeyError: 'answer' and ConnectionRefused are downstream symptoms after the scheduler was killed.

Failure Summary

DeepSeek-V3.2 TP=8 + EAGLE/MTP server ran the GSM8K eval to 136/200 questions (00:53:09), then a single in-flight request stuck the decode loop. The scheduler dumped Scheduler debug info: batch_size()=1 at 01:00:33 and made no further progress; the hard scheduler watchdog (1200 s) fired on TP1 at 01:22:06, raising SIGQUIT and tearing down the server. The eval client then failed with connection-refused/KeyError because the server was dead.

Regression Status

Known recurring / flaky test file, with a distinct NEW hang signature today. The test file fails intermittently with different signatures across the window; today's signature (mid-decode GPU-sync hang in the MTP draft-extend path → hard 1200 s watchdog) is not seen in the other failing runs in the window. The last 4 completed nightlies (07-17 → 07-21) all passed this job.

Recent history of test_deepseek_v32_mtp_eval_mi35x.py (test_a_gsm8k) in job nightly-accuracy-8-gpu-mi35x-deepseek-v32-mtp (most recent comparable completed scheduled runs on main, event=schedule; this job runs only this one test file, so job status = test-file status):

Date	Run	Test File Status	Signature
Jul 22	29944275726	❌ Failed	hang → 1200 s hard watchdog → SIGQUIT (MTP draft-extend, this run)
Jul 21	29855050299	✅ Passed	—
Jul 19	29697529176	✅ Passed	—
Jul 18	29654601256	✅ Passed	—
Jul 17	29601593052	✅ Passed	—
Jul 17	29521595306	⏭️ Cancelled	(excluded)
Jul 14	29355460805	❌ Failed	different (no 1200 s watchdog; file retried)
Jul 13	29273241962	❌ Failed	different (no 1200 s watchdog; file retried)
Jul 05	28749559519	✅ Passed	—
Jul 04	28714602438	❌ Failed	different (no 1200 s watchdog)
Jul 03	28676359018	✅ Passed	—
Jul 02	28611376093	❌ Failed	startup failure at setUpClass:75 (AssertionError: All of them must not be None)

First observed for this hang signature: 2026-07-22. Last known passing: 2026-07-21 (29855050299).

Failure Cluster

GPU-sync hang in MTP/EAGLE draft-extend (DSA backend) → hard scheduler watchdog → SIGQUIT

Facts (verified observations, no interpretation)

• [FACT] GSM8K reached 136/200 at 00:53:09, then no server-side batch progress until the watchdog fired ~29 min later (log lines 16274–16279).
• [FACT] [2026-07-23 01:22:06 TP1] Scheduler watchdog timeout (self.watchdog_timeout=1200.0, self.soft=False) followed by [01:22:10] SIGQUIT received … It usually means one child failed. (log lines 20298–20299).
• [FACT] Stuck batch was size 1: scheduler.cur_batch_for_debug.batch_size()=1 dumped by TP3 at 01:00:33 (log line 16280).
• [FACT] py-spy dump of TP1 (active): memcpy_and_sync ← _local_scalar_dense_cuda/.item() ← init_forward_metadata (dsa_backend.py:781) ← prepare_for_draft_extend (eagle_worker_common.py:197) ← _draft_extend_for_decode (eagle_worker_v2.py:904) ← forward_batch_generation (eagle_worker_v2.py:1211). TP1 is blocked in a HIP GPU→CPU sync.
• [FACT] py-spy dump of TP0 (idle): c10d::Work::wait ← broadcast_pyobj (utils/common.py:2175) ← _broadcast_reqs_across_ranks (request_receiver.py:186) ← recv_requests. TP0 has advanced to the next iteration and waits on the request broadcast collective — a TP desync where TP1 never completes its forward.
• [FACT] dsa_backend.py:781 is max_seqlen_k = int(forward_batch.seq_lens.max().item()) + draft_token_num — a full-stream synchronization point; it blocks when previously enqueued GPU work on the stream never completes.
• [FACT] No HIP error / "Memory access fault" / device-side assert appears in the log — this is a pure hang, not a crash/fault.
• [FACT] Server args include --dsa-decode-backend tilelang --dsa-prefill-backend tilelang --speculative-algorithm EAGLE (test file lines 54–71).
• [FACT] Today's run (29944275726) is the first scheduled nightly to include commits 08cb081e, 8ae0eb83, 024639a3 (verified NOT ancestors of the 07-21 pass SHA 9a6c96083f).

Hypothesised Causes (with confidence, with disconfirming evidence)

• [LOW] DSA/dspark draft-extend perf commits landing in the pass→fail window introduced a rare hang in the MTP draft-extend forward.
  • Candidates (all new this run, all in the exact code area of the hang): 08cb081e (#31981, "Skip page-table columns past kv length in DSA draft-extend metadata kernel"), 8ae0eb83 (#31985), 024639a3 (#31986).
  • Supporting: the hang is in init_forward_metadata/prepare_for_draft_extend (DSA draft-extend); these three commits are the only DSA-draft-extend changes in 9a6c96083f..0bdd4730 and this is their first nightly.
  • Disconfirming: (a) the test file is historically flaky with unrelated signatures (startup failure 07-02, retried failures 07-13/14), so a nondeterministic hang independent of these commits is plausible; (b) only 1 of ~200 requests hung — a hard regression would be expected to fail more reliably; (c) 08cb081e's early return is per-program in an independent-grid Triton kernel with no block-wide barrier after it (verified in python/sglang/kernels/ops/attention/dsa_metadata.py lines ~533–575), so a barrier-deadlock mechanism is not supported — it is a plausible correctness risk, not an obvious hang mechanism; (d) .item() is a generic sync point and the log does not identify which GPU kernel actually hung.
  • How to verify: re-run this job pinned to the 07-21 pass SHA 9a6c96083f several times to establish a hang baseline; if green, bisect 9a6c96083f..0bdd4730, reverting #31981/#31985/#31986 individually on a branch and re-running test_deepseek_v32_mtp_eval_mi35x.py on MI35x.
• [LOW] Nondeterministic GPU hang in the tilelang DSA decode kernel or draft model forward, unrelated to any commit (pre-existing flakiness).
  • Supporting: intermittent single-request hang; historically flaky test; pure hang with no fault; only manifests deep into a long eval.
  • Disconfirming: last 4 nightlies passed and this specific signature is new today, which is at least consistent with a fresh change.
  • How to verify: repeat the run at fixed SHA 9a6c96083f to measure baseline hang rate.

No hypothesis above LOW confidence; needs a repeat-at-pass-SHA baseline then git bisect across 9a6c96083f..0bdd4730.

In-flight Fix Check

• In-flight fix: ❌ none found targeting this hang. Searched open PRs for DSA draft, dsa metadata, deepseek mtp hang, watchdog draft extend (GET /search/issues?q=repo:sgl-project/sglang+is:pr+is:open+...). Many DSA/EAGLE PRs are open but none addresses a draft-extend hang/watchdog on this test; #31824 ("[Spec] perf: fuse draft extend metadata prep") touches the same area but is a perf change, not a hang fix.

Suggested Triage Steps

• Re-run nightly-accuracy-8-gpu-mi35x-deepseek-v32-mtp at the last-pass SHA 9a6c96083f 3–5× to determine whether the hang is a fresh regression or pre-existing flakiness.
• If 9a6c96083f is green, bisect 9a6c96083f..0bdd4730, prioritizing reverts of 08cb081e (#31981), 8ae0eb83 (#31985), 024639a3 (#31986) one at a time, re-running the test on MI35x.
• When reproducing, capture a py-spy --native dump of all 8 TP ranks at hang time to identify which GPU kernel behind the TP1 .item() sync is actually stuck (the current dump only shows the Python sync frame).
• If the hang proves nondeterministic and commit-independent, treat as a flaky DSA/tilelang MTP hang and track separately; consider lowering per-request generation risk (it currently blocks the whole 8-GPU accuracy signal for DeepSeek-V3.2 MTP).

Status (factual, no priority assignment)

• Persistence: This hang signature first seen 2026-07-22; 1/1 runs since the last green (07-21). The test file itself is intermittently red (❌ on 07-02, 07-04, 07-13, 07-14, 07-22; ✅ on 07-03, 07-05, 07-17, 07-18, 07-19, 07-21) with mixed signatures.
• Scope: Single job / single test file (nightly-accuracy-8-gpu-mi35x-deepseek-v32-mtp, DeepSeek-V3.2 TP=8 + MTP on MI35x). No other jobs share this hang in this run.
• Blocked work: Blocks the DeepSeek-V3.2 TP+MTP GSM8K accuracy + spec-accept-length signal on MI35x.
• In-flight fix: ❌ none found in open PRs.

nightly-2-gpu-mi35x-glm51-mxfp4 — failed step(s): N/A

Commit Info

• sglang: d0b96898 ("[Fix] Include disagg prefill waiting queue in FPM (#32122)")
• aiter: N/A — the job log blob was never uploaded (runner lost communication before log finalization), so no [CI-AITER-CHECK] markers are readable.

Failed Tests

This is not a test-assertion failure. The runner died mid-test, so no pytest result was produced.

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi35x/test_glm51_mxfp4_tp2_gsm8k_mi35x.py	TestGLM51MXFP4TP2GSM8KMI35x.test_gsm8k	Runner-lost during step 6 ("Accuracy Test MI35x (2-GPU GLM-5.1-MXFP4 GSM8K)"); no test result emitted	Job

Failure Summary

Job nightly-2-gpu-mi35x-glm51-mxfp4 on runner linux-mi35x-gpu-2-glhqm-runner-7xwcq concluded failure at 18:29:20 UTC (~32 min after start), with step 6 (the GSM8K accuracy test) still marked in_progress and post-steps pending. The check-run annotation reports: "The self-hosted runner lost communication with the server." The per-job log blob returns BlobNotFound (never uploaded), because the runner process died before the log was finalized. This is an infrastructure / runner-lost event, not a reported accuracy or assertion failure.

Regression Status

Known recurring failure (runner-lost signature, ~6 days). This exact signature — job failure + accuracy step stuck in_progress + "runner lost communication" annotation — has occurred in every completed scheduled run since Jul 17. Notably the signature changed on Jul 17: on Jul 16 and earlier the accuracy step ran to completed/failure (a genuine, logged test/accuracy failure), whereas from Jul 17 onward the runner is lost mid-step. No passing run for this test exists in the 15-run window (Never-passed in queryable history).

Recent history of test_glm51_mxfp4_tp2_gsm8k_mi35x.py (test_gsm8k) in job nightly-2-gpu-mi35x-glm51-mxfp4 (scheduled, event=schedule, branch=main, completed runs only):

Date	Run	Acc-step Status	Signature
Jul 23	30031602981	❌ in_progress (killed)	Runner lost communication
Jul 22	29944275726	❌ in_progress (killed)	Runner lost communication
Jul 21	29855050299	❌ in_progress (killed)	Runner lost communication
Jul 20	29766761021	—	Run cancelled (workflow-level)
Jul 19	29697529176	❌ in_progress (killed)	Runner lost communication
Jul 18	29654601256	❌ in_progress (killed)	Runner lost communication
Jul 17	29601593052	❌ in_progress (killed)	Runner lost communication
Jul 16	29521595306	❌ completed/failure	Logged test/accuracy failure (different signature)
Jul 15	29438351311	—	Run cancelled (workflow-level)
Jul 14	29355460805	❌ completed/failure	Logged test/accuracy failure (different signature)
Jul 13	29273241962	❌ completed/failure	Logged test/accuracy failure (different signature)

(Runner-lost annotation confirmed present on all six Jul 17–23 jobs. The Jul 13/14/16 jobs reached completed/failure on the accuracy step, so their logs exist and show a different, logged failure mode.)

Failure Cluster

Runner lost communication during GLM-5.1-MXFP4 GSM8K accuracy test (MI35x 2-GPU).

Facts (verified observations, no interpretation)

• Check-run annotation (verbatim): "The self-hosted runner lost communication with the server. Verify the machine is running and has a healthy network connection. Anything in your workflow that terminates the runner process, starves it for CPU/Memory, or blocks its network access can cause this error." — [check-runs/89289230550/annotations].
• Job conclusion=failure, status=completed; step 6 "Accuracy Test MI35x (2-GPU GLM-5.1-MXFP4 GSM8K)" status=in_progress, step 12 "Post Checkout code" status=pending. Post-steps never ran.
• Duration: started 17:57:18 → completed 18:29:20 UTC (~32 min). The runner died ~32 min in, well before the test's SERVER_LAUNCH_TIMEOUT of 5400 s (90 min), so this is not a plain server-launch timeout.
• Per-job log endpoint 302-redirects to an Azure blob that returns BlobNotFound — the log was never uploaded (consistent with the runner dying before log finalization).
• Runner: linux-mi35x-gpu-2-glhqm-runner-7xwcq, labels ['linux-mi35x-gpu-2'].
• The same runner-lost annotation is present on the 5 prior scheduled runs I inspected (Jul 17–22; job IDs 87954757713, 88106575932, 88220738842, 88717264923, 89005587693).
• The test config (test_glm51_mxfp4_tp2_gsm8k_mi35x.py:52-80) launches a large amd/GLM-5.1-MXFP4 server: --tp 2, --mem-fraction-static 0.85, --kv-cache-dtype fp8_e4m3, tilelang DSA prefill+decode backends, GSM8K with num_threads=512 full dataset — a heavy GPU/CPU load on a 2-GPU MI35x node.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [LOW] MI35x runner hardware/driver crash or CPU/memory starvation during the heavy GLM-5.1-MXFP4 GSM8K run (GPU hang, driver reset, or OOM killing the runner agent).
  • Supporting: annotation explicitly lists "terminates the runner process, starves it for CPU/Memory" as causes; death occurs mid-accuracy-step under a large-model + 512-thread GSM8K load; identical signature recurs only on the linux-mi35x-gpu-2 label; multiple open MI35x-runner-health PRs exist (see below), indicating a known-fragile runner class.
  • Disconfirming: the log was never uploaded, so there is no direct evidence of OOM / GPU fault / hang vs a genuine network drop — this is inference from the annotation and timing only. A pure network/infra blip on the runner host would produce the same annotation without any GPU issue.
  • How to verify: check MI35x host dmesg/GPU-driver logs and node metrics (VRAM, host RAM, CPU) around 18:29 UTC Jul 23; re-run the job on a healthy MI35x node (or an MI300 node per the routing PRs) and see if it survives to produce a logged result.
• [LOW] Underlying GLM-5.1-MXFP4 accuracy/server failure predating the runner-lost issue — the Jul 13/14/16 runs of this same test failed at completed/failure (logs exist).
  • Supporting: the signature changed on Jul 17; before then the job failed with the accuracy step completing, implying a genuine test/server-side failure that the runner-lost problem now masks.
  • Disconfirming: this is a separate (earlier) failure mode; it does not explain today's runner-lost event. Requires reading the Jul 13/14/16 logs (available) to characterize.
  • How to verify: download the Jul 16 job log (87843850927) to see whether the earlier failure was accuracy-below-threshold, a server crash, or a launch timeout.

No hypothesis reaches MEDIUM confidence — the authoritative log is missing. Root cause is not attributable to any sglang commit from the available evidence.

In-flight Fix Check

Several open PRs target MI35x runner-health / job-routing (the likely mitigation surface, not a code fix for this test):

• In-flight fix: ✅ #31068 — "[AMD] ci: detect and recover hung GPUs in MI35x pre-flight" (most directly relevant to hung-GPU/runner-lost on MI35x).
• In-flight fix: ✅ #32003 — "ci(amd): route scheduled jobs to MI300 with model cache".
• In-flight fix: ✅ #31409 — "[AMD] Route PR CI to MI300 runners".
• In-flight fix: ✅ #30431 — "[AMD][Not-Merge] ci: self-heal driver-level VRAM leaks + graceful GPU cleanup on shutdown".
• (Search: is:pr is:open mi35x runner → 39 hits; is:pr is:open GLM-5.1-MXFP4 → 9 hits. No PR specifically fixes the GSM8K test's runner-lost.)

Suggested Triage Steps

• Inspect MI35x node linux-mi35x-gpu-2-glhqm-runner-7xwcq host logs (dmesg, GPU driver/SMI, OOM-killer) around 18:29 UTC Jul 23 to confirm whether the runner died from GPU hang / OOM vs a network drop.
• Re-run the job once on a healthy runner to obtain an actual log; if it survives, the earlier failure mode (accuracy/server) will surface and can be triaged separately.
• Chase reviewers on #31068 (MI35x hung-GPU pre-flight detection) and consider the MI300-routing PRs (#32003/#31409) if MI35x runners remain unstable — do NOT open a duplicate.
• Read the Jul 16 logged-failure log (87843850927) to characterize the pre-Jul-17 accuracy/server failure that the runner-lost issue now masks.

Status

• Persistence: Runner-lost signature failing every completed scheduled run for ~6 days (since Jul 17, 6 consecutive completed runs; Jul 20 was workflow-cancelled). This test has never passed in the 15-run queryable window.
• Scope: Job nightly-2-gpu-mi35x-glm51-mxfp4 on the linux-mi35x-gpu-2 runner class. (Whether sibling MI35x jobs share the runner-lost signature was not cross-checked here.)
• Blocked work: No accuracy signal for GLM-5.1-MXFP4 TP=2 GSM8K on MI35x — the nightly regression gate for the gfx950 aiter BF16-GEMM accuracy drop this test was written to guard is currently blind.
• In-flight fix: MI35x runner-health/routing PRs open (#31068, #32003, #31409, #30431); none is a direct fix for this test.

nightly-test-1-gpu-kernel — failed step(s): Nightly Kernel Benchmark (1-GPU)

Commit Info

• sglang: d0b96898
• aiter: 9127c94a (v0.1.17.dev110+g9127c94a1) — source: dev image ([CI-AITER-CHECK] reports "Dev/patched version detected → skipping rebuild"; Dockerfile default matched the same SHA)

Failed Tests

Test File	Test Function	Error	Log
test/registered/kernels/ops/mamba/test_transfer_mamba.py	test_mamba_kernel_backup_load_roundtrip (page_first, page_first_direct × fp16, bf16)	NameError: name 'transfer_kv_mamba_lf_pf' is not defined	Log
test/registered/kernels/ops/mamba/test_transfer_mamba.py	test_mamba_kernel_single_item (page_first, page_first_direct × fp16, bf16)	NameError: name 'transfer_kv_mamba_lf_pf' is not defined	Log
test/registered/kernels/ops/mamba/test_transfer_mamba.py	test_mamba_kernel_full_indices (page_first, page_first_direct × fp16, bf16)	NameError: name 'transfer_kv_mamba_lf_pf' is not defined	Log

Result: 12 failed, 4 passed (0.95s). The 4 passing are test_mamba_kernel_empty_indices, which hit the if src_indices.numel() == 0: return early-exit at memory_pool_host.py:398 and never reach the missing symbol.

Failure Summary

The kernel-suite test test_transfer_mamba.py exercises MambaPoolHost with io_backend="kernel". On this AMD ROCm runner (GPU_ARCH=mi30x, gfx942), every non-empty-index case raises NameError: name 'transfer_kv_mamba_lf_pf' is not defined at python/sglang/srt/mem_cache/memory_pool_host.py:407. All 12 non-empty cases fail identically; only the 3 empty-index variants (early return) pass.

Regression Status

Never-passed (no green run found on AMD). The test was not present in the AMD nightly kernel job on 2026-07-14; it first appeared on 2026-07-16 and has failed with this exact NameError in every completed run since.

Recent history of test/registered/kernels/ops/mamba/test_transfer_mamba.py in job nightly-test-1-gpu-kernel (most recent comparable completed scheduled runs: nightly-test-amd.yml, branch main, event=schedule):

Date	Run	Test File Status	Failed Function	Error
Jul 23	30031602981	❌ Failed	roundtrip / single_item / full_indices	NameError: transfer_kv_mamba_lf_pf
Jul 22	29944275726	❌ Failed	same	NameError: transfer_kv_mamba_lf_pf
Jul 21	29855050299	❌ Failed	same	NameError: transfer_kv_mamba_lf_pf
Jul 20	29766761021	⏭️ Run cancelled	—	—
Jul 19	29697529176	❌ Failed	same	NameError: transfer_kv_mamba_lf_pf
Jul 17	29601593052	❌ Failed	same	NameError: transfer_kv_mamba_lf_pf
Jul 16	29521595306	❌ Failed (first appearance)	same	NameError: transfer_kv_mamba_lf_pf
Jul 15	29438351311	⏭️ Run cancelled	—	—
Jul 14	29355460805	➖ Test not in job (unregistered)	—	—

First observed failing: 2026-07-16. No passing run exists in the queryable window (lookback: 2026-07-14 … 2026-07-23). The current run 30031602981 is overall queued at snapshot time, but the nightly-test-1-gpu-kernel job itself completed (failure).

Failure Cluster

NameError: transfer_kv_mamba_lf_pf undefined on ROCm (CUDA-gated import)

Facts (verified observations)

• Log (verbatim): E NameError: name 'transfer_kv_mamba_lf_pf' is not defined at ../python/sglang/srt/mem_cache/memory_pool_host.py:407; test summary 12 failed, 4 passed, 1 warning in 0.95s.
• Runner is AMD ROCm: [CI-AITER-CHECK] Runner GPU_ARCH=mi30x, Using gfx942 block ⇒ is_hip()=True, is_cuda()=False.
• Source (memory_pool_host.py:41-45): the mamba transfer import is gated if _is_cuda: — so transfer_kv_mamba_lf_pf / transfer_kv_mamba_pf_lf are not imported on ROCm. (The sibling sgl_kernel.kvcacheio import at line 31 is gated if _is_cuda or _is_hip:, i.e. ROCm-enabled.)
• Source (memory_pool_host.py:399-415): the io_backend == "kernel" branch calls transfer_kv_mamba_lf_pf(...) unconditionally, with no platform guard — so on ROCm the name is undefined at call time.
• The test hard-codes io_backend="kernel" (test_transfer_mamba.py:219,237,...), so it always drives the CUDA-only path. The 4 passing cases are test_mamba_kernel_empty_indices (empty src_indices ⇒ early return at line 398 before the missing symbol).
• grep confirms transfer_kv_mamba_lf_pf is defined only in python/sglang/kernels/ops/mamba/transfer_mamba.py:64 and referenced at memory_pool_host.py:43 (import) and :407 (call).
• Historical logs for Jul 22, Jul 21, Jul 19, Jul 17, Jul 16 each contain 24 occurrences of the same NameError for this test file (path was registered/jit/test_transfer_mamba.py before the 2d1a7be8 reclassification; identical test functions).

Hypothesised Causes

• [HIGH] The io_backend="kernel" mamba transfer path is CUDA-only: transfer_kv_mamba_lf_pf is imported solely under if _is_cuda: (memory_pool_host.py:41) but called with no platform guard (memory_pool_host.py:407). On ROCm the name is never bound, so any non-empty transfer raises NameError.
  • Supporting: runner is gfx942/ROCm (is_cuda()=False); the missing symbol lives in a _is_cuda-gated block; the failing test forces io_backend="kernel"; only empty-index cases (which early-return) pass.
  • Disconfirming: (none found) — the code path is unambiguous and reproduces identically across 6+ completed runs.
  • How to verify: on an MI3xx container, python -c "from sglang.srt.mem_cache import memory_pool_host as m; print(hasattr(m,'transfer_kv_mamba_lf_pf'))" → expect False; run test/registered/kernels/ops/mamba/test_transfer_mamba.py.
• [LOW] (ruled out) #31919 (8260ade6, Jul 21) touches this import block.
  • Supporting: it restructured memory_pool_host.py and shows the if _is_cuda: guard.
  • Disconfirming: the identical NameError was already present in the Jul 16 and Jul 19 runs — before 8260ade6 landed. This commit only preserved a pre-existing gap, it did not introduce it. Not a regression from this commit.

In-flight Fix Check

• In-flight fix: ✅ #31741 "[AMD] Enable mamba JIT transfer kernel on ROCm (fix transfer_kv_mamba NameError)" — open since 2026-07-20, not draft, mergeable_state=blocked (last updated 2026-07-23). Directly targets this exact NameError. Chase reviewers / unblock; do NOT open a duplicate.
• Search run: search/issues?q=repo:sgl-project/sglang+is:pr+is:open+transfer_kv_mamba → 1 hit (#31741).

Suggested Triage Steps

• Prioritize landing #31741; confirm it either enables the mamba JIT transfer import on ROCm or gates the io_backend="kernel" call site by platform. It is currently blocked — check what is blocking merge (required CI/reviews).
• If a fix cannot land soon, consider skipping/xfail-ing the io_backend="kernel" cases in test_transfer_mamba.py on non-CUDA (is_cuda()) to restore AMD nightly kernel-suite signal, tracked against #31741.
• Verify parity for the sibling load path (transfer_kv_mamba_pf_lf at memory_pool_host.py:364), which is gated the same if _is_cuda: way and would fail identically once exercised on ROCm.

Status

• Persistence: Failing every completed AMD nightly kernel run since 2026-07-16 (first appearance in this job); never observed passing on AMD.
• Scope: nightly-test-1-gpu-kernel job in nightly-test-amd.yml; 12 of 16 parametrizations in one test file. Blocks the mamba KV-transfer kernel signal on AMD; does not affect other kernel test files (rest of shard: 20/21 passed).
• In-flight fix: ✅ #31741 (open since 2026-07-20, blocked).

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-07-23 23:33 UTC)

[amd-bot]

nightly-test-amd-rocm720.yml — 21 failure(s)

Run started (UTC): 2026-07-22 17:51 UTC
Last scanned (UTC): 2026-07-23 23:02 UTC
Commits: sglang d0b9689

Summary

Counts: 21 failures · 13 clusters · 9 carrying over (≥3 days) · 3 new today (+1 intermittent since Jul 22)

Two scheduled runs are covered: 29944173150 (Jul 22, sglang 0bdd4730) and 30031529424 (Jul 23, sglang d0b96898). aiter is the unchanged dev image 9127c94a in every job — not a variable. Most clusters recur across both runs.

#	Cluster	Job	Test File	Test Function	Status	Hypothesis (confidence)
1	Mixtral GSM8K stuck ~0.31	nightly-accuracy-2-gpu-rocm720	test_gsm8k_eval_amd.py	test_gsm8k_all_models (Mixtral-8x7B)	Never-passed; underlying eval bug since ~Apr 7	[HIGH] answer-extraction takes last number #29933
2	Mixtral GSM8K stuck ~0.31	nightly-accuracy-2-gpu-rocm720	test_gsm8k_eval_amd.py	test_gsm8k_all_models (Mixtral-8x7B)	same, Jul 22	[MEDIUM] MoE degraded / extraction
3	GPU mem fault — Qwen3-30B-A3B prefill	nightly-accuracy-8-gpu-mi35x-rocm720	test_qwen3_moe_eval_mi35x.py	test_qwen3_moe_accuracy	Never-passed (since ≥Jul 3)	[LOW] mem fault TP8 bf16 MoE (768//8 shape)
4	GPU mem fault — Qwen3-30B-A3B prefill	nightly-accuracy-8-gpu-mi35x-rocm720	test_qwen3_moe_eval_mi35x.py	test_qwen3_moe_accuracy	same, Jul 22	[LOW] mem fault; #31405 partial
5	EAGLE/MTP accuracy drop — DeepSeek-R1-MXFP4	nightly-8-gpu-mi35x-deepseek-r1-mxfp4-tp4-rocm720	test_deepseek_r1_mxfp4_tp4_mtp_mi35x.py	test_gsm8k	Never-passed since introduction ~Jul 3 (0.86 vs 0.944)	[MEDIUM] spec-decode correctness or mis-set gate
6	EAGLE/MTP accuracy drop — DeepSeek-R1-MXFP4	nightly-8-gpu-mi35x-deepseek-r1-mxfp4-tp4-rocm720	test_deepseek_r1_mxfp4_tp4_mtp_mi35x.py	test_gsm8k	same, Jul 22 (0.849)	[MEDIUM] spec-decode correctness
7	GPU Hang — GLM-5.1-MXFP4 2-GPU	nightly-2-gpu-mi35x-glm51-mxfp4-rocm720	test_glm51_mxfp4_tp2_gsm8k_mi35x.py	setUpClass	Never-passed since ≥Jul 8 (≥15 days)	[MEDIUM] GPU hang on runner; mitigations #31068/#32003
8	GPU Hang — GLM-5.1-MXFP4 2-GPU	nightly-2-gpu-mi35x-glm51-mxfp4-rocm720	test_glm51_mxfp4_tp2_gsm8k_mi35x.py	setUpClass	same, Jul 22 (log missing)	[MEDIUM] hang; [LOW] dsv4 cooperative_groups.h JIT
9	DeepSeek-V3.2 basic GSM8K < 0.93	nightly-accuracy-8-gpu-mi35x-deepseek-v32-rocm720	test_deepseek_v32_eval_mi35x.py	test_deepseek_v32_accuracy	Never-passed (0.855–0.905 band)	[MEDIUM] threshold too tight vs tilelang DSA
10	DeepSeek-V3.2 basic GSM8K < 0.93	nightly-accuracy-8-gpu-mi35x-deepseek-v32-rocm720	test_deepseek_v32_eval_mi35x.py	test_deepseek_v32_accuracy	same, Jul 22 (0.880)	[MEDIUM] threshold too tight
11	NameError transfer_kv_mamba_lf_pf on ROCm	nightly-test-1-gpu-kernel-rocm720	test_transfer_mamba.py	test_mamba_kernel_* (12×)	Never-passed on ROCm since ~Jul 16	[FACT] CUDA-only import gate; #31741 ✅
12	NameError transfer_kv_mamba_lf_pf on ROCm	nightly-test-1-gpu-kernel-rocm720	test_transfer_mamba.py	test_mamba_kernel_* (12×)	same, Jul 22	[FACT] CUDA-only import gate; #31741 ✅
13	hipErrorCapturedEvent — Qwen3-235B capture	nightly-8-gpu-mi35x-qwen3-235b-mxfp4-rocm720	test_qwen3_instruct_mxfp4.py	setUpClass	Never-passed; signature since Jul 17	[MEDIUM] RCCL watchdog polls event during graph capture
14	hipErrorCapturedEvent — Qwen3-235B capture	nightly-8-gpu-mi35x-qwen3-235b-mxfp4-rocm720	test_qwen3_instruct_mxfp4.py	setUpClass	same, Jul 22	[MEDIUM] aiter/RCCL capture-unsafe; #27650 related
15	GPU fault / HSA-OOR — DeepSeek-V4-Pro triton	nightly-8-gpu-mi35x-deepseek-v4-pro-mtp-rocm720	test_deepseek_v4_pro_fp4_mtp.py / _tbo_mtp.py	test_a_gsm8k, test_b_bs_1_speed, test_gsm8k_tbo_mtp	Intermittent since Jul 22	[LOW-MED] triton MLA spec fault #30407
16	GPU fault / HSA-OOR — DeepSeek-V4-Pro triton	nightly-8-gpu-mi35x-deepseek-v4-pro-mtp-rocm720	test_deepseek_v4_pro_fp4_tbo_mtp.py	test_gsm8k_tbo_mtp	New Jul 22 (triton cfg only)	[MEDIUM] prefill-delayer OOM #31835
17	InternVL processor crash + Qwen VLM < thr	nightly-accuracy-2-gpu-vlm-rocm720	test_vlms_mmmu_eval_amd.py	test_mmmu_vlm_models	Never-passed since ≥Jul 14	[FACT] CLIPImageProcessor has no .tokenizer
18	VLM MMMU timeout — 4-GPU	nightly-4-gpu-rocm720	test_encoder_dp.py	test_vlm_mmmu_benchmark	Flaky (model random; timed out Jul 13/14/19/22)	[MEDIUM] 3600s budget too tight
19	Marginal GSM8K < 0.91 — DeepSeek-V4-Flash FP4	nightly-8-gpu-mi35x-deepseek-v4-flash-rocm720	test_deepseek_v4_flash_fp4.py	test_a_gsm8k	🆕 new Jul 23 (triton cfg 0.904)	[LOW] variance at 0.91 threshold
20	GSM8K < 0.93 — MiniMax-M2.7	nightly-8-gpu-minimax-m27-rocm720	test_minimax_m27_eval_amd.py	test_minimax_m27_accuracy	🆕 new Jul 23 (0.915, missed by 3 Q)	[LOW] threshold-margin variance
21	Scheduler watchdog hang — Qwen3.5 triton DCP	nightly-8-gpu-mi35x-qwen35-triton-dcp-rocm720	test_qwen3p5_triton_dcp.py	test_a_gsm8k	🆕 new Jul 23 (passed prior 5 runs)	[LOW] aiter-JIT stall → DCP all_reduce hang

Cluster: NameError transfer_kv_mamba_lf_pf on ROCm (rows 11–12)

• Shared facts: memory_pool_host.py:41 imports the mamba JIT symbols under if _is_cuda: but the io_backend="kernel" call site (:407) runs unconditionally on ROCm. 12/16 params fail; the 4 direct/empty params pass. Pure Python NameError, aiter-independent.
• In-flight fix: ✅ #31741 (open since Jul 20, was mergeable_state: dirty — needs rebase).
• Suggested triage: chase #31741 rebase & merge, then re-run the kernel job; interim option is is_hip() xfail on the kernel params.

Cluster: Mixtral/Mistral GSM8K stuck ~0.31 (rows 1–2)

• Shared facts: Mixtral scores 0.30–0.32 vs 0.69 across every run and on the NVIDIA H100 sister nightly too → eval-harness bug, not ROCm/aiter. get_answer_value() returns the last number (simple_eval_mixed_prefix_gsm8k.py:41); chatty run-on models mis-parse.
• In-flight fix: ✅ #29933 (open since Jul 2) targets exactly this.
• Suggested triage: verify #29933 rebases on d0b96898, re-run; if still <0.69, capture Mixtral transcript for a real MoE delta.

Cluster: hipErrorCapturedEvent — Qwen3-235B MXFP4 capture (rows 13–14)

• Shared facts: crash inside decode CUDA graph capture (bs=512); ProcessGroupNCCL watchdog polls a WorkNCCL event on a capturing stream → hipErrorCapturedEvent → SIGABRT on all 4 TP ranks; setUpClass never completes. Never-passed in window.
• In-flight fix: ❌ none exact; #27650 (aiter AR capture-safety) is thematically related, different config.
• Suggested triage: re-run with TORCH_NCCL_ENABLE_MONITORING=0 or --disable-cuda-graph to confirm the watchdog-during-capture race.

Cluster: GPU fault / HSA-OOR — DeepSeek-V4-Pro triton backend (rows 15–16)

• Shared facts: crashes exclusive to the triton FlashMLA sub-run (unified_kv_triton passes same run); Memory access fault + HSA_STATUS_ERROR_OUT_OF_RESOURCES under 250+ running / 300+ queued. Intermittent (passed Jul 18/21). Row 16 (Jul 22) implicates prefill-delayer commit #31835; row 15 (Jul 23) adds an MLA-spec mem fault.
• In-flight fix: ✅ candidate #30407 (triton spec GPU fault on MLA) — verify it covers this.
• Suggested triage: cherry-pick #30407 and separately A/B-revert d03c8cee8 (#31835); fix /coredumps mount to capture the faulting kernel.

Cluster: GPU Hang — GLM-5.1-MXFP4 2-GPU (rows 7–8)

• Shared facts: HW Exception … GPU Hang → exit 134 (Jul 10/14 verified), surfacing as "runner lost communication" when the node wedges. Confined to runner class linux-mi35x-gpu-2-*; never-passed ≥15 days. Not the HEAD commit.
• In-flight fix: ⚠️ mitigations #31068 (hung-GPU recover), #32003 (route to MI300); root-cause ❌ none.

Cluster: GPU mem fault — Qwen3-30B-A3B prefill (rows 3–4)

• Shared facts: server ready + graph capture done, then Memory access fault by GPU node-* on all 8 ranks at first real request; bf16 TP8 MoE 768//8=96 shape flagged in the test's own source comment. Other 3 models in the job pass. Never-passed, no green baseline to bisect.
• In-flight fix: ⚠️ #31405 (same model, but TP2/FP8/ROCm7.0) — likely doesn't cover TP8/bf16/ROCm7.2.

Cluster: EAGLE/MTP accuracy drop — DeepSeek-R1-MXFP4 (rows 5–6)

• Shared facts: MTP variant scores a stable 0.849–0.864 vs 0.944 while the non-MTP sister step in the same job passes ~0.95. Only delta is EAGLE spec-decode args. Greedy spec-decode should be near-lossless → either a correctness bug in the aiter verify/draft path or a gate copied from the non-MTP target.
• In-flight fix: ❌ none confirmed; candidates #30105, #29876, #30407.

Cluster: DeepSeek-V3.2 basic GSM8K < 0.93 (rows 9–10)

• Shared facts: stable 0.855–0.905 band across ≥6 runs/multiple SHAs, server healthy. Signature of a mis-calibrated gate or a persistent tilelang-DSA numeric gap; no green baseline exists to distinguish. Not the HEAD commit.
• In-flight fix: ❌ none; WIP #25559 (NSA preshuffle) tangential.

Cluster: InternVL processor crash + Qwen VLM < threshold — 2-GPU (row 17)

• Facts: AutoProcessor returns a bare CLIPImageProcessor; get_tokenizer_from_processor (common.py:529) unconditionally reads .tokenizer → AttributeError, aborting the whole 12-model suite (launch is outside the try/except). Qwen2.5-VL/Qwen3-VL also chronically sub-threshold. In-flight fix ❌ none.
• Suggested triage: move launch inside try or add InternVL to AMD_FAILING_VLM_MODELS to restore signal for the 8 masked models.

Cluster: VLM MMMU timeout — 4-GPU (row 18)

• Facts: single test picks a random model per run; timed out (3600s) when it drew GLM-4.1V (Jul 14/22) and once on Qwen3-VL-32B (Jul 19) yet the same Qwen3-VL passed in 684s Jul 21 → flakiness/budget, not a code regression. In-flight fix ❌ none.

Cluster: DeepSeek-V4-Flash FP4 marginal < 0.91 (row 19) 🆕

• Facts: triton cfg scored 0.9098/0.9037; unified_kv_triton cfg passed 0.9196 same commit/image. First file-level fail after 5 green runs. Most likely threshold-margin variance; only in-window MoE/quant commit #32160 is a no-op move. In-flight fix ❌ none.

Cluster: MiniMax-M2.7 GSM8K < 0.93 (row 20) 🆕

• Facts: 0.915 vs 0.93 (missed by 3 Q); passed prior 3 measured runs at 0.945–0.950. aiter byte-identical to last pass; the only MoE/quant commit in the 32-commit window is a no-op refactor targeting inkling. Likely variance. In-flight fix ❌ none.

Cluster: Scheduler watchdog hang — Qwen3.5 triton DCP (row 21) 🆕

• Facts: hard watchdog (300s) killed all 8 ranks → GSM8K scored 0.0; soft-watchdog stack shows TP1 stuck in aiter ck_moe JIT build_module/file-baton while others park in DCP all_reduce. Passed prior 5 runs. Window 0bdd4730af8c..d0b968980 (32 commits); dcp/comm.py unchanged. In-flight fix ❌ none.

Suggested triage order

Rank	Triage step	Targets	Why this first
1	Chase #31741 rebase+merge, re-run kernel job	Cluster A (rows 11–12)	[FACT] root cause, fix already open — cheapest green
2	Verify #29933 rebases on HEAD, re-run accuracy-2gpu	Cluster F (rows 1–2)	[HIGH] eval-bug, fix open; longest-standing (cross-vendor since Apr)
3	Re-run new failures once; if persistent, bisect 0bdd4730af8c..d0b968980	Clusters K, L, M (rows 19–21)	3 NEW-today regressions; separate flake from real regression cheaply
4	Cherry-pick #30407 / A/B-revert d03c8cee8; fix /coredumps mount	Cluster H (rows 15–16)	Intermittent GPU fault w/ candidate fix; coredump needed to confirm
5	Confirm gate calibration with owners; re-run at full GSM8K set	Clusters D, E (rows 5–6, 9–10)	Never-passed accuracy gates — decide threshold vs kernel before deep work
6	Repro GPU faults w/ AMD_LOG_LEVEL=3 / disable-cuda-graph; track #31068/#32003	Clusters B, C, G (rows 3–4, 7–8, 13–14)	Hardest, no in-flight fix; needs kernel-level capture + infra work

---

Job	Failed Steps	Started
nightly-test-1-gpu-kernel-rocm720	Nightly Kernel Benchmark ROCm 7.2 (1-GPU)	2026-07-22T18:04
nightly-2-gpu-mi35x-glm51-mxfp4-rocm720	N/A	2026-07-22T17:52
nightly-accuracy-2-gpu-rocm720	N/A	2026-07-22T18:51
nightly-4-gpu-rocm720	Nightly Test ROCm 7.2 (4-GPU)	2026-07-22T17:51
nightly-accuracy-8-gpu-mi35x-rocm720	Accuracy Test MI35x ROCm 7.2 (8-GPU GPT-OSS)	2026-07-22T21:10
nightly-accuracy-8-gpu-mi35x-deepseek-v32-rocm720	Accuracy Test MI35x ROCm 7.2 (8-GPU DeepSeek-V3.2)	2026-07-22T23:47
nightly-accuracy-2-gpu-vlm-rocm720	Nightly Accuracy Test ROCm 7.2 (2-GPU VLM MMMU)	2026-07-22T19:59
nightly-8-gpu-mi35x-qwen3-235b-mxfp4-rocm720	Accuracy Test + Performance Test MI35x ROCm 7.2 (8-GPU Qwen3-235B-MXFP4)	2026-07-23T00:18
nightly-8-gpu-mi35x-deepseek-v4-pro-mtp-rocm720	Accuracy + MTP Accept Length Test MI35x ROCm 7.2 (8-GPU DeepSeek-V4-Pro FP8 MTP, triton)	2026-07-22T21:53
nightly-8-gpu-mi35x-deepseek-r1-mxfp4-tp4-rocm720	Accuracy Test MI35x ROCm 7.2 (DeepSeek-R1-MXFP4 TP4 MTP)	2026-07-23T00:38
nightly-accuracy-8-gpu-mi35x-rocm720	Accuracy Test MI35x ROCm 7.2 (8-GPU GPT-OSS)	2026-07-23T17:56
nightly-2-gpu-mi35x-glm51-mxfp4-rocm720	N/A	2026-07-23T17:56
nightly-8-gpu-mi35x-deepseek-r1-mxfp4-tp4-rocm720	Accuracy Test MI35x ROCm 7.2 (DeepSeek-R1-MXFP4 TP4 MTP)	2026-07-23T18:30
nightly-8-gpu-mi35x-qwen3-235b-mxfp4-rocm720	Accuracy Test + Performance Test MI35x ROCm 7.2 (8-GPU Qwen3-235B-MXFP4)	2026-07-23T19:41
nightly-accuracy-2-gpu-rocm720	N/A	2026-07-23T18:40
nightly-8-gpu-mi35x-qwen35-triton-dcp-rocm720	Accuracy Test MI35x ROCm 7.2 (8-GPU Qwen 3.5 Triton DCP)	2026-07-23T18:35
nightly-test-1-gpu-kernel-rocm720	Nightly Kernel Benchmark ROCm 7.2 (1-GPU)	2026-07-23T20:40
nightly-8-gpu-mi35x-deepseek-v4-flash-rocm720	Accuracy + Performance Test MI35x ROCm 7.2 (8-GPU DeepSeek-V4-Flash FP8 + FP4, triton)	2026-07-23T20:49
nightly-8-gpu-minimax-m27-rocm720	Accuracy Test ROCm 7.2 (8-GPU MiniMax-M2.7)	2026-07-23T21:44
nightly-8-gpu-mi35x-deepseek-v4-pro-mtp-rocm720	Accuracy + MTP Accept Length Test MI35x ROCm 7.2 (8-GPU DeepSeek-V4-Pro FP8 MTP, triton)	2026-07-23T19:09
nightly-accuracy-8-gpu-mi35x-deepseek-v32-rocm720	Accuracy Test MI35x ROCm 7.2 (8-GPU DeepSeek-V3.2)	2026-07-23T21:58

Per-Job Analysis

nightly-test-1-gpu-kernel-rocm720 — failed step(s): Nightly Kernel Benchmark ROCm 7.2 (1-GPU)

Commit Info

• sglang: 0bdd4730
• aiter: 9127c94a (source: dev image — [CI-AITER-CHECK] reports v0.1.17.dev110+g9127c94a1, "Dev/patched version detected → skipping rebuild")

Failed Tests

Job: nightly-test-1-gpu-kernel-rocm720 — 12 failed (test file test/registered/jit/test_transfer_mamba.py); overall job Test Summary: 20/21 files passed, this file exited 1.

Test File	Test Function	Error	Log
test/registered/jit/test_transfer_mamba.py	test_mamba_kernel_backup_load_roundtrip (4 params)	NameError: name 'transfer_kv_mamba_lf_pf' is not defined	Log
test/registered/jit/test_transfer_mamba.py	test_mamba_kernel_single_item (4 params)	NameError: name 'transfer_kv_mamba_lf_pf' is not defined	Log
test/registered/jit/test_transfer_mamba.py	test_mamba_kernel_full_indices (4 params)	NameError: name 'transfer_kv_mamba_lf_pf' is not defined	Log

(test_mamba_kernel_empty_indices (4 params) PASSED — it returns early on src_indices.numel() == 0 before reaching the undefined name.)

Failure Summary

test/registered/jit/test_transfer_mamba.py failed 12 of its 16 subtests on the ROCm 7.2 kernel job. Every failure is the same NameError: name 'transfer_kv_mamba_lf_pf' is not defined raised at python/sglang/srt/mem_cache/memory_pool_host.py:407 when the test drives backup_from_device_all_layer(..., io_backend="kernel") (memory_pool_host.py:481 → _copy_tensor_all_layers_lf_pf).

Regression Status

Known recurring failure (never-passed on ROCm in the queryable window). The identical NameError is present in every comparable completed scheduled run back through 2026-07-17; no green run for this test file was found in the 15-run history.

Recent history of test/registered/jit/test_transfer_mamba.py (test_mamba_kernel_backup_load_roundtrip et al.) in job nightly-test-1-gpu-kernel-rocm720 (most recent comparable completed scheduled runs: branch=main, event=schedule, completed only):

Date	Run	Job	Test File Status	Failed Function	Error
Jul 22	29944173150	nightly-test-1-gpu-kernel-rocm720	❌ Failed	test_mamba_kernel_backup_load_roundtrip +2	NameError transfer_kv_mamba_lf_pf
Jul 21	29854963573	nightly-test-1-gpu-kernel-rocm720	❌ Failed	same	same NameError (12×)
Jul 20	29766667696	nightly-test-1-gpu-kernel-rocm720	⏭️ run cancelled	—	(excluded)
Jul 19	29697452589	nightly-test-1-gpu-kernel-rocm720	❌ Failed	same	same NameError (12×)
Jul 18	29654521083	nightly-test-1-gpu-kernel-rocm720	❌ Failed	same	same NameError (12×)
Jul 17	29601455852	nightly-test-1-gpu-kernel-rocm720	❌ Failed	same	same NameError (12×)

First observed failure (this window): 2026-07-17 or earlier. No passing run found for this test file in the queryable history → treat as Never-passed on ROCm.

Failure Cluster

NameError: transfer_kv_mamba_lf_pf undefined on ROCm (CUDA-only import gate)

Facts (verified observations, no interpretation)

• E NameError: name 'transfer_kv_mamba_lf_pf' is not defined at python/sglang/srt/mem_cache/memory_pool_host.py:407, reached via memory_pool_host.py:481 in backup_from_device_all_layer → _copy_tensor_all_layers_lf_pf with io_backend='kernel', can_use_jit = False (verbatim from job log line 4685).
• 12 subtests fail with this exact NameError; the 4 empty_indices subtests pass (early return on numel()==0 at memory_pool_host.py before line 407).
• In source python/sglang/srt/mem_cache/memory_pool_host.py, transfer_kv_mamba_lf_pf / transfer_kv_mamba_pf_lf are imported only inside if _is_cuda: (lines 41-45). The transfer_kv_all_layer_* group above it is gated on if _is_cuda or _is_hip: (line 31). This job runs on mi30x (GPU_ARCH=mi30x, ROCm/HIP), so _is_cuda is False and the two mamba names are never bound.
• The same NameError with the same 12× count appears in the completed scheduled runs of 2026-07-17, 07-18, 07-19 and 07-21 (verified by downloading each job log and grepping — 12 matches each).
• aiter is unchanged across all these runs (AITER_COMMIT_OVERRIDE empty in all; Dockerfile-default dev image preserved). NVIDIA is unaffected because _is_cuda binds the imports there.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [FACT] The mamba JIT-transfer symbols are import-gated behind if _is_cuda: (memory_pool_host.py:41), but _copy_tensor_all_layers_lf_pf calls transfer_kv_mamba_lf_pf unconditionally on the io_backend=="kernel" path (memory_pool_host.py:407), so on ROCm the name is undefined.
  • Supporting: import block reads if _is_cuda: (not if _is_cuda or _is_hip:); the sibling transfer_kv_all_layer_* block one line above uses _is_cuda or _is_hip; job runs on mi30x; NameError points at exactly line 407.
  • Disconfirming: (none found) — the failure reproduces identically across ≥4 days with unchanged aiter, and NVIDIA (where _is_cuda is True) does not fail.
  • How to verify: on an MI3xx container, python3 -c "from sglang.srt.mem_cache import memory_pool_host as m; print(hasattr(m,'transfer_kv_mamba_lf_pf'))" → expect False.
• [FACT — ruled out] PR #32015 (977ea336, merged 2026-07-22) is not the cause. It only changed the import path (sglang.jit_kernel.transfer_mamba → sglang.kernels.ops.mamba.transfer_mamba) inside the same if _is_cuda: block; the NameError predates it (present 2026-07-17). The commit that first introduced the _is_cuda-gated call site is older than the queryable window.

In-flight Fix Check

• In-flight fix: ✅ #31741 "[AMD] Enable mamba JIT transfer kernel on ROCm (fix transfer_kv_mamba NameError)" — open (non-draft) since 2026-07-20, last updated 2026-07-21, but mergeable_state: dirty (merge conflict / needs rebase). Chase the author to rebase & land; do NOT open a duplicate.
  • Search run: GET /search/issues?q=repo:sgl-project/sglang+is:pr+is:open+transfer_kv_mamba → 1 hit (#31741).

Suggested Triage Steps

• Ping the author of #31741 to resolve the merge conflict (mergeable_state: dirty) and rebase onto current main so it can land — it directly targets this NameError.
• After #31741 merges, re-run test/registered/jit/test_transfer_mamba.py in the nightly-test-1-gpu-kernel-rocm720 job to confirm all 16 subtests pass on ROCm.
• If a faster interim signal is wanted: verify whether the ROCm/HIP build actually ships the mamba JIT kernel; if it does, widening the import gate at memory_pool_host.py:41 from if _is_cuda: to if _is_cuda or _is_hip: (mirroring the transfer_kv_all_layer_* block above) is the minimal candidate change — confirm on a branch, do not treat as a directive.

Status

• Persistence: Failing on ROCm 7.2 for ≥5 days (identical NameError in every completed scheduled run since 2026-07-17); never observed passing in the queryable window.
• Scope: nightly-test-1-gpu-kernel-rocm720 (this job); ROCm/HIP-specific — NVIDIA kernel jobs unaffected because the imports bind under _is_cuda. Likely also affects any other AMD job exercising MambaPoolHost with io_backend="kernel".
• Blocked work: Mamba host-KV-cache kernel-backend transfer coverage on AMD.
• In-flight fix: ✅ #31741 open since 2026-07-20 but mergeable_state: dirty — needs rebase before it can merge.

nightly-2-gpu-mi35x-glm51-mxfp4-rocm720 — failed step(s): N/A

Commit Info

• sglang: 0bdd4730
• aiter: N/A — the current job's GitHub log blob is unavailable (BlobNotFound on both the per-job log endpoint and the run-logs archive, which 404s because run 29944173150 is still partly in progress). No [CI-AITER-CHECK] markers are readable for this run. (For context only, the Jul 16 sibling run used the Dockerfile default AITER_COMMIT=9127c94a…, no override — but that is a different run.)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi35x/test_glm51_mxfp4_tp2_gsm8k_mi35x.py	TestGLM51MXFP4TP2GSM8KMI35x.setUpClass (server launch)	Job concluded failure; exact error not recoverable — log blob missing	Job

The job runs a single test file (run_suite.py --suite nightly-amd-2-gpu-mi35x-glm51-mxfp4). The Accuracy Test MI35x ROCm 7.2 (2-GPU GLM-5.1-MXFP4 GSM8K) step is stuck at in_progress (never concluded) and the Post step is pending, yet the job is marked completed/failure — the runner was terminated abruptly before the log was uploaded.

Failure Summary

The nightly-2-gpu-mi35x-glm51-mxfp4-rocm720 job failed launching the GLM-5.1-MXFP4 GSM8K accuracy server on MI35x/ROCm 7.2. The current run's log is unavailable (BlobNotFound), so the specific error for this run cannot be verified. This job has never passed in the queryable window (Jul 13–22); it has exhibited at least two distinct verified failure signatures across that window (see below), plus a recurring abrupt-termination pattern in the last five runs where logs are consistently missing.

Regression Status

Known recurring failure / Never-passed (no green run found in the queryable window).

Recent history of test/registered/amd/accuracy/mi35x/test_glm51_mxfp4_tp2_gsm8k_mi35x.py (setUpClass) in job nightly-2-gpu-mi35x-glm51-mxfp4-rocm720 (most recent comparable completed scheduled runs: branch=main, event=schedule, status=completed):

Date	Run	Job	Test File Status	Failed Function	Error / signature
Jul 22	29944173150	job	❌ Failed	setUpClass	Log missing (BlobNotFound); step stuck in_progress (~22 min)
Jul 21	29854963573	job	❌ Failed	setUpClass	Log missing (BlobNotFound); step in_progress (~25 min)
Jul 19	29697452589	job	❌ Failed	setUpClass	Log missing (BlobNotFound); step in_progress
Jul 19	29654521083	job	❌ Failed	setUpClass	Log missing (BlobNotFound); step in_progress
Jul 17	29601455852	job	❌ Failed	setUpClass	Log missing (BlobNotFound); step in_progress
Jul 16	29521474931	job	❌ Failed	setUpClass	JIT compile fail: cooperative_groups.h not found compiling dsv4 topk_v2 during CUDA graph capture (exit 255)
Jul 14	29355338821	job	❌ Failed	setUpClass	GPU Hang → Fatal Python error: Aborted (exit 134)
Jul 13	29273096587	job	❌ Failed	setUpClass	GPU Hang (exit 134)

Cancelled scheduled runs (Jul 15, Jul 20) excluded per methodology. First observed failure: earliest queryable run (Jul 13); no passing run found in the window — treat as Never-passed.

Failure Cluster

Two verified clusters in the window, plus one unresolved:

1. JIT kernel compile failure — cooperative_groups.h not found (dsv4 topk_v2, gfx950/ROCm 7.2) during decode CUDA graph capture (Jul 16, verified).
2. GPU Hang during server launch (HW Exception … reason: GPU Hang, exit 134) (Jul 13, Jul 14, verified).
3. Abrupt runner termination / log unavailable (Jul 17–22, current run) — symptom-level only; underlying cause not verifiable because logs are missing.

Facts (verified observations)

• Current job 89005250224: conclusion=failure, Accuracy step status=in_progress/conclusion=None, Post step pending; started 17:52:04Z, completed 18:14:25Z (~22 min). [FACT]
• Current per-job log endpoint and run-logs archive both return no usable log (BlobNotFound / HTTP 404). [FACT]
• Jul 16 log (job 87843000141): during Capturing batches (bs=512), the DSA attention backend JIT-compiles the DeepSeek-V4 topk_v2 kernel; hipcc/ninja fail with topk_impl.cuh:27:10: fatal error: 'cooperative_groups.h' file not found compiling for gfx950, raising Exception: Capture cuda graph failed: ninja exited with status 1, then server exits and popen_launch_server raises in setUpClass. [FACT]
• Source confirms python/sglang/jit_kernel/include/sgl_kernel/deepseek_v4/topk_impl.cuh:27 unconditionally #include <cooperative_groups.h> with namespace cg = cooperative_groups; at line 33 — no USE_ROCM/__HIP__ guard. [FACT]
• Jul 13 & Jul 14 logs: HW Exception by GPU node-2 … reason: GPU Hang, exit code 134, failing at test begin (before server ready). [FACT]
• The last five runs (Jul 17–22) share the identical shape: Accuracy step never concludes, log BlobNotFound, ~22–26 min duration — distinct from the Jul 13/14/16 runs which did produce logs. [FACT]

Hypothesised Causes (with confidence)

• [LOW] The current (Jul 22) failure is the same dsv4 topk_v2 cooperative_groups.h JIT compile failure seen Jul 16.
  • Supporting: same job/model/runner; the JIT compile is deterministic per code state and the offending unguarded include still exists in the tree at 0bdd4730.
  • Disconfirming: the current log is unavailable so this is unconfirmed; the ~22 min runtime and stuck-step/BlobNotFound pattern more closely resembles the Jul 13/14 GPU Hang runs (which killed the runner) than the Jul 16 JIT run (which produced a full log and ran ~75 min).
  • How to verify: re-run the job and capture the log before the runner dies; grep for cooperative_groups.h vs GPU Hang.
• [MEDIUM] dsv4 topk_v2 JIT kernel is not HIP-portable on gfx950/ROCm 7.2 — topk_impl.cuh includes the CUDA-only <cooperative_groups.h> with no ROCm guard, so any GLM-5.1-MXFP4 run that reaches DSA-backend CUDA graph capture will fail to compile it on AMD.
  • Supporting: verified compile error in the Jul 16 log + verified unguarded include in source; error is arch-specific (compiling for gfx950).
  • Disconfirming: does not explain the Jul 13/14 GPU-Hang failures, which occur earlier (before graph capture).
  • How to verify: on an MI35x/ROCm 7.2 container, launch sglang serve --model-path amd/GLM-5.1-MXFP4 --tp 2 … and confirm the JIT compile of dsp_v4_topk_v2 fails; test a HIP-guarded include locally.
• [LOW] Runner/GPU instability on linux-mi35x-gpu-2 causing hangs and abrupt log loss.
  • Supporting: Jul 13/14 explicit GPU Hang exit 134; Jul 17–22 abrupt terminations with missing logs.
  • Disconfirming: not separable from a real server crash that itself wedges the GPU; no independent infra signal gathered.
  • How to verify: check runner health / dmesg / VRAM-clear step on the affected runner around the failing runs.

No hypothesis reaches HIGH for the current run because its log is unavailable; the strongest code-level evidence (MEDIUM) is the unguarded CUDA header, verified against Jul 16.

In-flight Fix Check

• In-flight fix: ❌ none found specifically for the ROCm cooperative_groups.h / dsv4 topk_v2 gfx950 compile issue. Searched open PRs: cooperative_groups → #31839, #25575 (neither addresses the missing-header-on-ROCm path); topk_v2 rocm → 0 results; glm51 mxfp4 → #30984, #27209, #20894 (Docker/CPU, not this failure).

Suggested Triage Steps

• Re-run the job and capture the log before the runner is killed — the single biggest gap is that the last five runs have no readable log. Confirm whether the current failure is the JIT compile error or a GPU hang.
• If JIT compile: inspect python/sglang/jit_kernel/include/sgl_kernel/deepseek_v4/topk_impl.cuh:27 — add a ROCm/HIP guard or provide a HIP-compatible cooperative-groups path so the dsv4 topk_v2 kernel compiles on gfx950. Reproduce locally in an MI35x/ROCm 7.2 container by launching the GLM-5.1-MXFP4 server.
• If GPU Hang recurs: raise with AMD infra — check linux-mi35x-gpu-2 runner health and the ensure_vram_clear.sh step around Jul 13/14/17–22.
• File a tracking issue for the GLM-5.1-MXFP4 MI35x job's persistent red status (never-passed since at least Jul 13); no in-flight fix exists.

Status

• Persistence: Failing in every completed scheduled run in the queryable window (Jul 13 → Jul 22); no green run found. Current run 29944173150 is the latest.
• Scope: This single job nightly-2-gpu-mi35x-glm51-mxfp4-rocm720 (MI35x, 2-GPU, ROCm 7.2). Log unavailability affects the last 5 runs' diagnosability.
• Blocked work: GLM-5.1-MXFP4 GSM8K accuracy signal on MI35x/ROCm 7.2 is unavailable.
• In-flight fix: ❌ none found for the ROCm dsv4 topk_v2 / cooperative_groups.h issue.

nightly-accuracy-2-gpu-rocm720 — failed step(s): N/A

Commit Info

• sglang: 0bdd4730
• aiter: 9127c94a1 (v0.1.17.dev110+g9127c94a1, source: dev image preserved — [CI-AITER-CHECK] reported "Dev/patched version detected → skipping rebuild"; matches Dockerfile default 9127c94a)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi30x/test_gsm8k_eval_amd.py	TestNightlyGsm8KEval.test_gsm8k_all_models (model mistralai/Mixtral-8x7B-Instruct-v0.1)	gsm8k Score: 0.323 vs threshold: 0.69 → Status: ❌ FAILED; ran retry loop, then step terminated mid-run	Log

Non-test-level failure: the "Nightly Test ROCm 7.2 (2-GPU)" step never completed (job API shows step 6 in_progress/None while job conclusion is failure). The job was terminated ~58 min in (18:51:15 → 19:49:17 UTC); the downloaded log ends abruptly at 19:30:59 mid-Mixtral decode with no traceback, exit code, or check_model_scores summary.

Failure Summary

The 2-GPU ROCm 7.2 nightly gsm8k accuracy suite fails on mistralai/Mixtral-8x7B-Instruct-v0.1, which scored 0.323 against its 0.69 threshold (all other models passed: Llama-3.1-8B 0.818, Qwen3-8B 0.922, Llama-3.1-70B 0.924). Mixtral entered the harness's 3-attempt retry loop, and the test step was then terminated before completing, so the job never emitted its final summary/assertion — leaving step 6 in_progress and the job marked failure.

Regression Status

Known recurring failure / Never-passed in window. The test_gsm8k_all_models suite (Mixtral row) has failed in every comparable completed scheduled run of nightly-accuracy-2-gpu-rocm720 in the queryable window; no green run found (lookback Jul 4 → Jul 22 2026).

Recent history of test_gsm8k_eval_amd.py (test_gsm8k_all_models, Mixtral-8x7B) in job nightly-accuracy-2-gpu-rocm720 (completed scheduled runs, branch=main, event=schedule):

Date	Run	Test File Status	Signature
Jul 22	run	❌ Failed	Mixtral 0.323; step terminated mid-Mixtral (in_progress)
Jul 21	run	❌ Failed	ends at "Testing: Mixtral-8x7B"; step in_progress
Jul 19	run	❌ Failed	step in_progress
Jul 19	run	❌ Failed	step in_progress
Jul 17	run	❌ Failed	ends at "Testing: Mixtral-8x7B"; step in_progress
Jul 15	run	❌ Failed	failed before test step (no step 6)
Jul 13	run	❌ Failed	clean completion; Mixtral 0.304/0.298/0.295/0.295 → gate AssertionError
Jul 5	run	❌ Cancelled	step6 cancelled
Jul 4	run	❌ Failed	step6 completed/failure

First failure in window: on/before Jul 4 2026. No passing run observed. The Jul 13 clean run confirms the Mixtral score sits at ~0.29–0.32 (consistently ~half the 0.69 threshold).

Failure Cluster

Mixtral-8x7B gsm8k accuracy far below threshold on AMD MI325 2-GPU ROCm 7.2 (with recurring mid-Mixtral step termination).

Facts (verified observations)

• [METRIC] gsm8k_score=0.3234398782343988 labels={"model": "mistralai/Mixtral-8x7B-Instruct-v0.1", "eval": "gsm8k"} (job.log:17701); printed Score: 0.323 (threshold: 0.69) is below threshold, triggering the harness's up-to-3-attempt retry loop (test_gsm8k_eval_amd.py:275-283).
• Other models in the same run passed: Llama-3.1-8B 0.818 (✅, thr 0.80), Qwen3-8B 0.922 (✅, thr 0.76), Llama-3.1-70B 0.924 (✅, thr 0.89). So the eval harness, server, and aiter GEMM/attention paths work for dense models.
• Job API: step 6 "Nightly Test ROCm 7.2 (2-GPU)" = in_progress/None, job conclusion = failure. Log ends at 19:30:59 mid-Mixtral decode (Decode batch, #running-req: 651 …, job.log:18465) with no error/exit code; job completed_at 19:49:17. → step terminated externally, not a clean pytest exit.
• Red herring 1 — Qwen3-4B "❌ FAILED (threshold: N/A)" (job.log:12335,12342) is cosmetic. Qwen/Qwen3-4B has no entry in MODEL_SCORE_THRESHOLDS; the per-model print computes passed = threshold and score >= threshold (test_gsm8k_eval_amd.py:288-289) → falsy when threshold is None → prints FAILED. But the actual gate check_model_scores treats a missing threshold as ⚠️ NO THRESHOLD and does not add it to failed_models (test_gsm8k_eval_amd.py:165-168, 199-201). Qwen3-4B scored 0.909 — healthy.
• Red herring 2 — the single Pyspy failed (py-spy dump … Failed to get stack traces) (job.log:11643, 19:17:12 TP1) occurred during Llama-3.1-70B startup, which subsequently PASSED (0.924). Not the fatal event.
• Job has no timeout-minutes (workflow line 259-266) → GitHub default 360 min; job ran only ~58 min, so this is not a GitHub step timeout.

Hypothesised Causes (with confidence + disconfirming evidence)

• [MEDIUM] Mixtral-8x7B MoE produces numerically degraded outputs on MI325 ROCm 7.2 (MoE path runs with SGLANG_MOE_PADDING=1 + aiter; test_gsm8k_eval_amd.py:245-253).
  • Supporting: score is consistently ~0.29–0.32 across many runs and stable across 4 retries; Mixtral is the only MoE model exercised in this run while every dense model passes; failure long-standing across the whole window.
  • Disconfirming: no A/B against a Triton-MoE / no-padding config in the logs; cannot yet isolate MoE vs Mixtral-specific chat formatting from CI logs alone.
  • How to verify: run Mixtral-8x7B gsm8k on MI325 ROCm 7.2 and inspect raw generations — garbage math ⇒ numerical/kernel; correct-but-misparsed ⇒ extraction/template. Then A/B with SGLANG_USE_AITER=0 (Triton MoE) and SGLANG_MOE_PADDING=0.
• [LOW] Answer-extraction / chat-template mismatch for Mixtral (candidate PR #29933 "extract GSM8K answer from first #### marker").
  • Supporting: an open eval-scoring fix exists.
  • Disconfirming: all other models score correctly with the same extractor, so a general extractor bug cannot explain a Mixtral-only 0.30 score.
  • How to verify: apply #29933 and re-score Mixtral responses offline.
• [LOW] Recent mid-Mixtral step terminations (step in_progress) are a hang during Mixtral's retry loop or runner/container instability, distinct from the accuracy fail.
  • Supporting: log ends abruptly with no error; job killed ~18 min after last log line; recurring Jul 17–22.
  • Disconfirming: Jul 13 completed cleanly at ~68 min (longer than today's 58 min), so it is not a fixed timeout; intermittent infra is plausible.
  • How to verify: check runner/container/GPU logs for the terminated runs; add explicit timeout-minutes so failures exit cleanly with the gate AssertionError.

In-flight Fix Check

• In-flight fix: ❌ none found for Mixtral accuracy on AMD (search q=repo:sgl-project/sglang is:pr is:open Mixtral ROCm in:title → 0; Mixtral accuracy → only unrelated PRs). Nearest open PR is #29933 (general gsm8k answer-extraction), a weak candidate per the disconfirming evidence above.

Suggested Triage Steps

• Reproduce Mixtral-8x7B gsm8k on MI325 2-GPU ROCm 7.2 (sglang serve --model-path mistralai/Mixtral-8x7B-Instruct-v0.1 --tp 2 --trust-remote-code) and inspect sample generations to classify: numerical/kernel degradation vs answer-extraction/template.
• A/B the Mixtral MoE path: rerun with SGLANG_USE_AITER=0 (Triton MoE) and separately SGLANG_MOE_PADDING=0; if score recovers, the aiter/padded-MoE path is implicated.
• Test whether #29933 changes Mixtral's parsed score offline (rescore existing responses); if it doesn't, this is a real model/backend accuracy issue, not an eval bug.
• Separately investigate the mid-Mixtral step terminations (runner/container/GPU state for the in_progress runs); consider adding timeout-minutes to nightly-accuracy-2-gpu-rocm720 so the job exits cleanly and surfaces the gate AssertionError instead of dangling in_progress.
• Minor test-config cleanup (not the failure cause): add a MODEL_SCORE_THRESHOLDS entry for Qwen/Qwen3-4B, otherwise it always prints a misleading per-model "❌ FAILED" and has no real accuracy gate.

Status

• Persistence: Failing every completed scheduled run in the queryable window (since ≥ 2026-07-04; ~19 days). Mixtral gsm8k ≈0.29–0.32 vs 0.69 threshold consistently.
• Scope: Job nightly-accuracy-2-gpu-rocm720 (2-GPU ROCm 7.2 accuracy). Mixtral-8x7B is the sole failing model; all dense models pass. Recent runs additionally terminate mid-test (step in_progress) rather than failing cleanly.
• Blocked work: Blocks the AMD ROCm 7.2 2-GPU nightly accuracy signal for Mixtral/MoE; masks any other regression in later models in the suite (Mixtral kills the run before they complete).
• In-flight fix: ❌ none found for Mixtral accuracy; #29933 is a low-confidence candidate only.

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-07-23 23:02 UTC)

[amd-bot]

nightly-test-amd-rocm720.yml — Per-Job Analysis (continued 2/5)

nightly-4-gpu-rocm720 — failed step(s): Nightly Test ROCm 7.2 (4-GPU)

Commit Info

• sglang: 0bdd4730
• aiter: 9127c94a (v0.1.17.dev110+g9127c94a1, source: dev image preserved — AITER_COMMIT_OVERRIDE empty, image already carried the Dockerfile-expected dev build)

Failed Tests

Test File	Test Function	Error	Log
test/registered/vlm/test_encoder_dp.py	TestVLMEncoderDP.test_vlm_mmmu_benchmark (model zai-org/GLM-4.1V-9B-Thinking)	TIMEOUT after 3600s (est_time=500s)	Log

Failure Summary

The only test in the nightly-amd-4-gpu suite, test/registered/vlm/test_encoder_dp.py, timed out after the 3600 s per-file cap. The test picks one random model per run (random.choice(MODELS) under is_in_ci()); this run drew zai-org/GLM-4.1V-9B-Thinking. The server started successfully (The server is fired up and ready to roll! at 18:05:34) and served /v1/chat/completions with 200 OK continuously right up to the kill at 19:02:28 — there was no crash, hang, or error. The MMMU benchmark simply did not finish inside 3600 s.

Regression Status

Flaky / chronically-slow test (not a clean new regression).

Recent history of test/registered/vlm/test_encoder_dp.py (test_vlm_mmmu_benchmark) in job nightly-4-gpu-rocm720 (most recent comparable completed scheduled runs: branch=main, event=schedule; cancelled runs on Jul 15 & Jul 20 excluded):

Date	Run	Model drawn	Test File Status	Elapsed	Error
Jul 22	29944173150	GLM-4.1V-9B-Thinking	❌ Timeout	3600s	TIMEOUT
Jul 21	29854963573	Qwen3-VL-32B	✅ Passed	684s	—
Jul 19	29697452589	Qwen3-VL-32B	❌ Timeout	3600s	TIMEOUT
Jul 18	29654521083	Qwen2.5-VL-72B	✅ Passed	2758s	—
Jul 17	29601455852	Qwen2.5-VL-72B	✅ Passed	2892s	—
Jul 16	29521474931	Qwen2.5-VL-72B	✅ Passed	2974s	—
Jul 14	29355338821	GLM-4.1V-9B-Thinking	❌ Timeout	3600s	TIMEOUT
Jul 13	29273096587	InternVL2_5-8B	❌ Failed	2450s	exit code 1 (not a timeout)

First observed timeout for this test file: Jul 14 (GLM-4.1V-9B-Thinking). Last passing: Jul 21 (Qwen3-VL-32B, 684s). Passing runs range 684 s–2974 s vs an est_time=500 s, i.e. the test is chronically running 1.4×–6× its estimate and frequently brushes the 3600 s cap.

Failure Cluster

VLM MMMU benchmark timeout on AMD 4-GPU (test_encoder_dp.py exceeds 3600 s)

Facts (verified observations)

• ✗ TIMEOUT: .../test_encoder_dp.py after 3600 seconds / Test Summary: 0/1 passed (job log line 8402–8398).
• Server reached ready state and returned 200 OK on /v1/chat/completions from 18:05:34 through 19:01:53, ending only when kill_process_tree fired at 19:02:28 (log lines 4644, 8350–8395). No stack trace, no CUDA/HIP fault, no server crash in the log.
• The one transient Health check failed ... detokenizer for last 20 seconds at 18:05:58 self-recovered (GET /health_generate ... 200 OK at 18:06:11, line 4645–4610) and is not the failure cause.
• Model selection is random per run: models_to_test = [random.choice(MODELS)] (test/registered/vlm/test_encoder_dp.py:32). This run drew GLM-4.1V-9B-Thinking (server_args model_path=..., line 4258).
• Same model, opposite outcomes across runs: Qwen3-VL-32B passed in 684 s on Jul 21 but timed out on Jul 19 — a >5× swing for the identical model (verified from both logs). GLM-4.1V-9B-Thinking timed out both times it was drawn (Jul 22, Jul 14).
• Essentially every GEMM logs not found tuned config in /tmp/aiter_configs/bf16_tuned_gemm.csv, will use default config! using torch solution:0 (e.g. lines 4653, 8350+) — aiter is falling back to the untuned torch GEMM path for these shapes on gfx942/ROCm 7.2.
• aiter JIT-compiled mha_batch_prefill* kernels on the fly during the run (finish build ... cost 35.1s, cost 32.4s, lines 5531/6010), adding minutes of one-off compile latency inside the timed window.

Hypothesised Causes (with confidence)

• [MEDIUM] The test's 3600 s budget is too tight for the actual VLM MMMU workload on the AMD 4-GPU runner; the test is chronically slow and any run-to-run variance tips it over the cap.
  • Supporting: even passing runs take 684 s–2974 s vs est_time=500 s; the largest model (Qwen2.5-VL-72B) consistently finishes at 2758–2974 s, i.e. within ~600 s of the cap. The failure mode is a timeout with the server healthy and actively serving, never a crash.
  • Disconfirming: Qwen3-VL-32B passed in only 684 s on Jul 21, so the workload can finish an order of magnitude under the cap — pure "budget too small" doesn't explain the 684 s vs 3600 s swing for the same model.
  • How to verify: re-run the job pinned to GLM-4.1V-9B-Thinking (remove the random.choice) several times and record wall-clock; if it clusters near/over 3600 s, raise --timeout-per-file or lower per-model work for this suite.
• [LOW] Untuned aiter GEMM fallback (using torch solution:0) for GLM/Qwen VL shapes on ROCm 7.2 inflates decode throughput enough to blow the budget.
  • Supporting: every GEMM in the run uses the torch fallback; no tuned config exists for these bf16 shapes.
  • Disconfirming: the same fallback path is present in the passing runs' logs too (Qwen2.5-VL-72B passed with identical torch solution:0 lines), so the fallback alone is not sufficient to cause a timeout.
  • How to verify: compare per-request decode latency between a passing (Qwen3-VL-32B, 684 s) and timing-out run of the same model to see whether GEMM time or something else (model download, warmup, runner contention) dominates.
• [LOW] Runner-level contention / variance (model download, JIT compile, shared-host load) pushes a normally-passing run over the cap.
  • Supporting: the 684 s vs 3600 s swing for identical model+code strongly implies an environmental variable, not a code regression; ~1–2 min of in-window aiter JIT builds observed.
  • Disconfirming: cannot confirm host load from the job log alone.
  • How to verify: check runner labels/host and concurrent jobs across the passing vs failing runs.

No hypothesis reaches HIGH; the swing across identical model+code points to flakiness/budget rather than a specific culprit commit, so a pass_sha..fail_sha bisect is unlikely to be productive here.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for test_encoder_dp (0), encoder_dp timeout (0), GLM-4.1V timeout (0); the two mmmu timeout amd hits (#29202, #27667) are unrelated keyword matches.

Suggested Triage Steps

• Reproduce deterministically: temporarily pin models_to_test to GLM-4.1V-9B-Thinking (bypass random.choice) and run test/registered/vlm/test_encoder_dp.py on an MI325 4-GPU ROCm 7.2 runner 3–5× to measure wall-clock spread.
• Instrument the MMMU harness to log per-model wall-clock and per-request decode latency, so a passing (Qwen3-VL-32B, 684 s) vs timing-out run can be A/B'd to see whether GEMM/decode, model load, or warmup dominates.
• If the workload genuinely needs >3600 s for the heavier models, raise --timeout-per-file for the nightly-amd-4-gpu suite (or split per-model), rather than treating this as a code regression.
• Because model choice is random, consider making the model deterministic per run (e.g. rotate by date) so this job's pass/fail is reproducible and its history comparable.

Status

• Persistence: Intermittent since at least Jul 13 — failed Jul 13 (exit 1), Jul 14 (timeout), Jul 19 (timeout), Jul 22 (timeout); passed Jul 16/17/18/21. Correlates with the randomly-drawn model (GLM-4.1V-9B-Thinking timed out both times it was drawn).
• Scope: nightly-4-gpu-rocm720 job, nightly-test-amd-rocm720.yml; single test file (test_encoder_dp.py) — it is the only enabled test in the nightly-amd-4-gpu suite for this run.
• Blocked work: VLM encoder-DP MMMU nightly signal on AMD 4-GPU ROCm 7.2 is unreliable.
• In-flight fix: ❌ none found — needs new investigation/PR.

nightly-accuracy-8-gpu-mi35x-rocm720 — failed step(s): Accuracy Test MI35x ROCm 7.2 (8-GPU GPT-OSS)

Commit Info

• sglang: 0bdd4730 ([CI] Fix failures on main (#32091))
• aiter: 9127c94a — v0.1.17.dev110+g9127c94a1 (source: Dockerfile default; dev/patched image preserved, no AITER_COMMIT_OVERRIDE)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi35x/test_qwen3_moe_eval_mi35x.py	test_qwen3_moe_accuracy (TestQwen3MoeEvalMI35x)	AssertionError: Failed models: ['Qwen/Qwen3-30B-A3B'] — server crashed with Memory access fault by GPU on first prefill (scheduler exit code -6/SIGABRT); benchmark hit except → ❌ ERROR, no accuracy= line was ever printed	Job log

The other 3 test files in this job passed: test_gpt_oss_eval_mi35x.py, test_gpt_oss_w4a8_mxfp4_eval_mi35x.py, test_qwen3_coder_next_eval_mi35x.py (Test Summary: 3/4 passed).

Failure Summary

The MI35x 8-GPU accuracy job failed only on test_qwen3_moe_eval_mi35x.py. The Qwen/Qwen3-30B-A3B (bf16, TP=8, attention_backend=aiter) server loaded successfully (KV cache allocated, CUDA-graph capture completed, The server is fired up and ready to roll! at 21:33:51, health check 200 OK at 21:34:01), then crashed on the first real forward pass with Memory access fault by GPU node-{2,5,7,9} → GPU core dump failed → Fatal Python error: Aborted; scheduler_0 (pid=34558) crashed with exit code -6 at 21:34:06. The test finished in ~175 s (vs ~1200 s estimated), confirming a crash rather than a genuine low-accuracy score.

Regression Status

Known recurring failure (not new today). No green run for this test file was found in the queryable window (Jul 13–22), so it is effectively Never-passed in the 15-run window.

Recent history of test_qwen3_moe_eval_mi35x.py (test_qwen3_moe_accuracy) in job nightly-accuracy-8-gpu-mi35x-rocm720 (most recent comparable completed scheduled runs: branch=main, event=schedule):

Date	Run	Test File Status	Failed Function	Error
Jul 22	29944173150	❌ Failed	test_qwen3_moe_accuracy	Memory access fault → scheduler exit -6
Jul 21	29854963573	❌ Failed	test_qwen3_moe_accuracy	Failed models: ['Qwen/Qwen3-30B-A3B'] (3 × Memory access fault)
Jul 19	29697452589	❌ Failed	test_qwen3_moe_accuracy	Failed models: ['Qwen/Qwen3-30B-A3B']
Jul 18	29654521083	❌ Failed	test_qwen3_moe_accuracy	Failed models: ['Qwen/Qwen3-30B-A3B']
Jul 17	29601455852	❌ Failed	test_qwen3_moe_accuracy	Failed models: ['Qwen/Qwen3-30B-A3B']
Jul 16	29521474931	⏭️ Cancelled	—	job cancelled (not comparable)
Jul 14	29355338821	❌ Failed (job crashed)	indeterminate	HW Exception by GPU node-3 reason: GPU Hang → suite aborted before Test Summary
Jul 13	29273096587	❌ Failed (job crashed)	indeterminate	suite aborted before Test Summary

First confirmed same-signature failure for this test file: Jul 17 (4 consecutive scheduled runs + today, identical Memory access fault + Failed models: ['Qwen/Qwen3-30B-A3B'], elapsed ~168–175 s each). Jul 13/14 show related GPU crashes but the suite aborted before recording per-file results.

Failure Cluster

GPU memory access fault on Qwen3-30B-A3B first prefill (MI35x, ROCm 7.2, aiter backend)

Facts (verified observations)

• The server is fired up and ready to roll! at 21:33:51; GET /health_generate ... 200 OK at 21:34:01; then Memory access fault by GPU node-2/5/7/9 ... Reason: Unknown. (job log).
• GPU core dump failed → Fatal Python error: Aborted; Subprocess scheduler_0 (pid=34558) crashed with exit code -6. Triggering SIGQUIT for cleanup... at 21:34:06.
• py-spy main-thread stack at crash: synchronize (torch/cuda/streams.py:231) ← process_batch_result_prefill (scheduler_components/batch_result_processor.py:183) ← process_batch_result (scheduler.py:3607) ← event_loop_overlap. Line 183 is result.copy_done.synchronize() — the sync point where the asynchronous GPU fault surfaces; the actual fault occurred earlier in the prefill forward pass.
• Config: model=Qwen/Qwen3-30B-A3B, dtype=auto(bf16), tp_size=8, attention_backend='aiter', mem_fraction_static=0.68, device=cuda; threshold 0.75 (per test file).
• No accuracy= line was emitted for this model (contrast the passing models: accuracy=0.620/0.815/0.845/0.965), confirming the except Exception → ❌ ERROR path, not a threshold miss.
• During Qwen3-30B-A3B warmup an aiter JIT emitted: -mllvm -amdgpu-coerce-illegal-types=1 is not supported by hipcc and failed to execute: .../clang++ ... -x hip -c /dev/null while compiling a pa_ragged (paged-attention) kernel.
• Same-signature failure reproduced in 5 completed runs (Jul 17, 18, 19, 21, 22); each ~168–175 s; each with Memory access fault by GPU.
• Sibling models on the same job/runner (gpt-oss-20b/120b, amd/gpt-oss-120b-w-mxfp4-a-fp8, Qwen3-Coder-Next) also use aiter and passed.

Hypothesised Causes (with confidence)

• [LOW] GPU memory access fault in the Qwen3-30B-A3B bf16 prefill forward pass (MoE/attention) on MI35x + ROCm 7.2 + aiter attention backend.
  • Supporting: fault fires on the very first prefill after the server is ready; model uses attention_backend='aiter'; a pa_ragged (aiter paged-attention) kernel JIT failed to execute during warmup with an unsupported -mllvm flag on this hipcc; crash is model-specific and highly reproducible (5/5 completed runs).
  • Disconfirming: other models on the identical job/runner using aiter passed, so the aiter backend is not broadly broken; the pa_ragged/amdgpu-coerce-illegal-types warning may be a benign fallback (it printed during CUDA-graph capture which itself completed); the fault surfaces at a generic stream.synchronize(), so log proximity to aiter lines is not proof of aiter causation.
  • How to verify: re-run test_qwen3_moe_eval_mi35x.py on MI35x/ROCm 7.2 with --attention-backend triton vs aiter to A/B the backend; if aiter-only, capture a GPU core dump / AMD_LOG_LEVEL=3 to pin the faulting kernel.
• [LOW] Standing AMD port/kernel issue for Qwen3-30B-A3B on MI35x ROCm 7.2 rather than a fresh sglang regression.
  • Supporting: failing continuously since at least Jul 17 with no green run in the 15-run window; Jul 13/14 also crashed on GPU (GPU Hang).
  • Disconfirming: no green baseline is available in the queryable window, so a precise regression commit cannot be bisected; the pre–Jul 17 crashes (GPU Hang) differ slightly from the Memory access fault signature and may be a distinct root cause.

No hypothesis reaches MEDIUM confidence; a definitive attribution needs a backend A/B run and a faulting-kernel capture, plus a green baseline (none exists in Jul 13–22) to bisect any sglang regression.

In-flight Fix Check

• In-flight fix: ❌ none found matching this exact signature (MI35x / ROCm 7.2 / TP8 / bf16 memory access fault). Searched open PRs for Qwen3-30B-A3B, qwen3 moe mi35x, memory access fault mi35x.
• Related-but-not-matching: #31405 [AMD] Gate Qwen3-30B-A3B TP2 qkv/o_proj CK block-FP8 GEMM shapes to Triton (ROCm 7.0) targets Qwen3-30B-A3B on AMD but for TP2 / block-FP8 / ROCm 7.0, whereas this failure is TP8 / bf16 / ROCm 7.2 — unlikely to cover this case, but the same model family and GEMM-shape gating idea may be relevant [LOW].

Suggested Triage Steps

• A/B the attention backend: re-run test_qwen3_moe_eval_mi35x.py on MI35x/ROCm 7.2 with --attention-backend triton vs the default aiter; if it passes on triton, the fault is in the aiter paged-attention path for this model shape.
• Capture the faulting kernel: run the Qwen3-30B-A3B server with AMD_LOG_LEVEL=3 / enable GPU core dump on MI35x and send one prefill request to identify the exact kernel behind the Memory access fault.
• Investigate the pa_ragged JIT warning (-mllvm -amdgpu-coerce-illegal-types=1 is not supported by hipcc) on the ROCm 7.2 image — confirm whether aiter silently falls back to a broken/wrong kernel for this GQA/head-size config (gqa_ratio=4, head_size=128, block_size=1).
• Check whether #31405's GEMM-shape→Triton gating approach extends to TP8/bf16/ROCm 7.2; if so, gate the offending Qwen3-30B-A3B shapes similarly on a branch and re-run.
• Since no green baseline exists in Jul 13–22, look further back than the queryable scheduled-run window (or in older nightly artifacts) for the last passing date to scope any bisect.

Status

• Persistence: Failing every comparable completed run since at least 2026-07-17 (5 completed runs: Jul 17, 18, 19, 21, 22), same Memory access fault signature; related GPU crashes on Jul 13/14. No green run in the 15-run query window.
• Scope: 1 test file (test_qwen3_moe_eval_mi35x.py) in job nightly-accuracy-8-gpu-mi35x-rocm720; the other 3 accuracy test files in this job pass. Not observed to be a scout run (no aiter override).
• Blocked work: Qwen3-30B-A3B (unquantized bf16, TP8) accuracy signal on MI35x / ROCm 7.2 nightly.
• In-flight fix: ❌ none found for this exact signature; #31405 is related (different TP/quant/ROCm) but unlikely to cover it.

nightly-accuracy-8-gpu-mi35x-deepseek-v32-rocm720 — failed step(s): Accuracy Test MI35x ROCm 7.2 (8-GPU DeepSeek-V3.2)

Commit Info

• sglang: 0bdd4730
• aiter: 9127c94a — v0.1.17.dev110+g9127c94a1 (source: Dockerfile default; dev image detected, rebuild skipped; AITER_COMMIT_OVERRIDE empty)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi35x/test_deepseek_v32_eval_mi35x.py	test_deepseek_v32_accuracy	AssertionError: Failed models: ['deepseek-ai/DeepSeek-V3.2 (basic)'] — GSM8K accuracy=0.880 threshold=0.93 ❌	Log

Failure Summary

The single registered test test_deepseek_v32_accuracy (DeepSeek-V3.2 "basic", TP=8, DSA tilelang prefill+decode backend) ran the GSM8K completion benchmark over 200 questions and scored 0.880, below the hard-coded accuracy_threshold=0.93 (test_deepseek_v32_eval_mi35x.py:71). The server started, served all 200 /generate requests with HTTP 200, and completed cleanly — this is a pure accuracy shortfall, not a crash/hang/OOM. [FACT]

Regression Status

Never-passed (no green run found in queryable window) — persistent accuracy shortfall.

Recent history of test_deepseek_v32_eval_mi35x.py (test_deepseek_v32_accuracy) in job nightly-accuracy-8-gpu-mi35x-deepseek-v32-rocm720 (most recent comparable completed scheduled runs: branch=main, event=schedule, completed only):

Date (run created)	Run	Test File Status	Measured accuracy	Note
Jul 22	29944173150	❌ Failed	0.880	this run
Jul 21	29854963573	❌ Failed	0.880
Jul 19	29697452589	❌ Failed	0.895
Jul 18	29654521083	❌ Failed	0.855
Jul 17	29601455852	❌ Failed	0.885
Jul 14	29355338821	❌ Failed	—	server crash during weight loading (fused_moe_triton/layer.py weight_loader); did not reach accuracy stage — different failure
Jul 13	29273096587	❌ Failed	—	same weight-loading crash; did not reach accuracy stage

First observed accuracy-shortfall failure: Jul 17 (earliest run in window that reached the accuracy stage). No passing run exists anywhere in the queried window; every run that reached the benchmark scored 0.855–0.895, i.e. 0.035–0.075 below the 0.93 gate. Earlier runs (Jul 13–14) failed even earlier during weight loading — a separate, now-resolved failure mode.

Failure Cluster

GSM8K accuracy below threshold — DeepSeek-V3.2 basic (DSA tilelang) on MI35x

Facts (verified observations)

• Verbatim log: accuracy=0.880 threshold=0.93 ❌ FAIL followed by AssertionError: Failed models: ['deepseek-ai/DeepSeek-V3.2 (basic)'] (test_deepseek_v32_eval_mi35x.py:245). [FACT]
• The run served all 200 GSM8K requests with HTTP 200 and normal decode throughput; no CUDA error, memory fault, watchdog kill, or OOM in the benchmark phase. [FACT]
• Threshold 0.93 and the full arg set (--dsa-prefill-backend tilelang --dsa-decode-backend tilelang --mem-fraction-static 0.85 --tp 8) are hard-coded in test_deepseek_v32_eval_mi35x.py:69-87. [FACT]
• Accuracy across the 5 runs that reached the benchmark: 0.855 / 0.885 / 0.855→0.895 / 0.880 / 0.880 — mean ≈ 0.879, all below 0.93, run-to-run spread ≈ 0.04. [FACT]
• aiter version is identical (v0.1.17.dev110+g9127c94a1, Dockerfile default) in this run; AITER_COMMIT_OVERRIDE is empty (not a scout run). [FACT]
• num_questions=200, so 0.880 = 176/200; the gate 0.93 = 186/200 — the test needs ~10 more correct answers. [FACT]

Hypothesised Causes (with confidence)

• [MEDIUM] The 0.93 threshold is set too tight for what the current DeepSeek-V3.2 basic + DSA tilelang config actually achieves on MI35x (~0.88 mean).
  • Supporting: 5 independent runs cluster tightly at 0.855–0.895 with no green run; the spread (~0.04) sits entirely below the gate, so this reads as a stable operating point ~0.88, not a flake around 0.93. The test file was authored with 0.93 and appears to have never gone green in the window.
  • Disconfirming: no confirmed green baseline exists to prove 0.93 was ever achievable on this hardware/config, so a genuine accuracy regression (below) cannot be excluded; a real quality bug and a too-tight threshold produce the same signal here.
  • How to verify: check whether any historical run of this suite (before the query window) ever passed at 0.93; if none, the gate was aspirational. Compare against the sister CUDA/MI300 DeepSeek-V3.2 GSM8K number for the same commit.
• [LOW] Accuracy regression in the DSA (DeepSeek Sparse Attention) tilelang prefill/decode path on gfx950/MI35x.
  • Supporting: DSA tilelang backend is MI35x-specific and relatively new; a numerical/kernel issue would depress GSM8K accuracy without crashing.
  • Disconfirming: no green baseline to anchor a pass_sha..fail_sha window; cannot attribute to any commit. aiter is unchanged across the failing runs, so the aiter delta is not implicated.
  • How to verify: run the same GSM8K benchmark with a non-DSA / alternative attention backend on MI35x and compare accuracy; A/B the DSA tilelang kernels.
• No hypothesis reaches HIGH; the two are indistinguishable from CI signal alone because there is no green baseline to bisect from.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for DeepSeek-V3.2+accuracy+mi35x, deepseek+v32+mi35x, dsv32+accuracy; matches (e.g. #31221, #30342, #30546) are DSA/AITER feature work, none targets this accuracy gate or this test. Two related branches exist upstream (amd_fix_dpskv32_basic_0518, bingxche/debug-dsv32-mi35x-mtp-mem-access-fault) but neither is an open PR.

Suggested Triage Steps

• Confirm whether this suite ever passed at 0.93 on MI35x (check runs older than the 7-run window). If it never has, treat the gate as mis-set and decide a realistic threshold (mean ≈ 0.88, so e.g. 0.85) rather than chasing a phantom regression.
• Compare this commit's DeepSeek-V3.2 GSM8K accuracy on a non-MI35x backend (CUDA / MI300) to distinguish a model-quality regression from an MI35x-DSA-specific one.
• If a quality regression is suspected, A/B the DSA tilelang prefill/decode backend against an alternative attention backend on MI35x for the same commit; if accuracy recovers, investigate the DSA tilelang kernels.
• Raise GSM8K_NUM_QUESTIONS (currently 200) for a run or two to shrink sampling variance before making a threshold decision — the ~0.04 run-to-run spread on 200 questions is largely sampling noise.

Status

• Persistence: Accuracy-shortfall failing on every completed scheduled run that reached the benchmark since Jul 17 (5 consecutive runs, 0.855–0.895 vs 0.93); Jul 13–14 failed earlier during weight loading (separate mode). No green run in the queryable window.
• Scope: Single job / single test — nightly-accuracy-8-gpu-mi35x-deepseek-v32-rocm720 (nightly-test-amd-rocm720.yml). MI35x DSA tilelang path only; no evidence of impact on other jobs.
• Blocked work: DeepSeek-V3.2 accuracy CI signal on MI35x ROCm 7.2 is permanently red, so this job cannot catch a real DSV3.2 accuracy regression until the gate/backend issue is resolved.
• In-flight fix: ❌ none found in open PRs.

nightly-accuracy-2-gpu-vlm-rocm720 — failed step(s): Nightly Accuracy Test ROCm 7.2 (2-GPU VLM MMMU)

Commit Info

• sglang: 0bdd4730
• aiter: 9127c94a — v0.1.17.dev110+g9127c94a1 (source: dev image preserved; [CI-AITER-CHECK] reports "Dev/patched version detected → skipping rebuild", AITER_COMMIT_OVERRIDE empty)

Failed Tests

All in test/registered/amd/accuracy/mi30x/test_vlms_mmmu_eval_amd.py, driver function test_mmmu_vlm_models (parametrized per-model via self.subTest). The unittest reports FAILED (errors=1), exit code 255.

Test File	Test Function (model subtest)	Error	Log
test_vlms_mmmu_eval_amd.py	test_mmmu_vlm_models [model=OpenGVLab/InternVL2_5-2B]	AttributeError: 'CLIPImageProcessor' object has no attribute 'tokenizer' → server exits code 1 (the unittest ERROR)	Log
test_vlms_mmmu_eval_amd.py	test_mmmu_vlm_models [model=Qwen/Qwen2-VL-7B-Instruct]	MMMU score 0.290 < threshold 0.30	Log
test_vlms_mmmu_eval_amd.py	test_mmmu_vlm_models [model=Qwen/Qwen2.5-VL-7B-Instruct]	MMMU score 0.280 < threshold 0.33	Log
test_vlms_mmmu_eval_amd.py	test_mmmu_vlm_models [model=Qwen/Qwen3-VL-30B-A3B-Instruct]	MMMU score 0.260 < threshold 0.29	Log

Failure Summary

The 2-GPU VLM MMMU accuracy suite runs a single unittest (test_mmmu_vlm_models) that loops over 12 VLMs. Three Qwen VLMs scored below their MMMU thresholds, and the 4th model (OpenGVLab/InternVL2_5-2B) crashed at server launch with AttributeError: 'CLIPImageProcessor' object has no attribute 'tokenizer'. Because popen_launch_server is called at test line 216 outside the try/except, the InternVL crash propagates as the unittest ERROR and aborts the whole run before the remaining 8 models (MiniCPM/DeepSeek/Kimi/MiMo/DeepSeek-OCR) ever run.

Regression Status

Known recurring failure (both signatures present in every completed run checked, spanning many head SHAs — not a new regression).

Recent history of test_vlms_mmmu_eval_amd.py (test_mmmu_vlm_models) in job nightly-accuracy-2-gpu-vlm-rocm720 (comparable completed scheduled runs, branch=main, event=schedule):

Date	Run	Test File Status	Failing signatures
Jul 22	29944173150	❌ Failed	InternVL crash (errors=1) + Qwen2-VL 0.290/0.30, Qwen2.5-VL 0.280/0.33, Qwen3-VL 0.260/0.29
Jul 21	29854963573	❌ Failed	InternVL crash (errors=1) + 3 Qwen sub-threshold
Jul 19	29697452589	❌ Failed	InternVL crash (errors=1)
Jul 18	29654521083	❌ Failed	InternVL crash (errors=1)
Jul 17	29601455852	❌ Failed	InternVL crash (errors=1) + Qwen2.5-VL 0.280/0.33, Qwen3-VL 0.260/0.29 (Qwen2-VL passed at 0.300)
Jul 16	29521474931	❌ Failed	(job failure)
Jul 15	29438224738	⚪ cancelled	not counted
Jul 14	29355338821	❌ Failed	InternVL crash (errors=1) + Qwen2.5-VL 0.280/0.33, Qwen3-VL 0.260/0.29 (Qwen2-VL passed at 0.300)

• InternVL2_5-2B crash: appears identically in all 6 completed runs checked (Jul 14–22) → effectively Never-passed for this model in the queryable window.
• Qwen2.5-VL / Qwen3-VL sub-threshold: consistently failing every run.
• Qwen2-VL-7B: borderline/flaky right at threshold — passed at exactly 0.300 on Jul 14 & 17, failed at 0.290 on Jul 21 & 22.

Failure Cluster

Two distinct clusters:

• CLIPImageProcessor has no attribute 'tokenizer' — InternVL processor load crash (the errors=1 abort).
• Qwen VLM MMMU score below threshold (accuracy shortfall, ~random-level scores).

Facts (verified observations)

• InternVL stack trace (verbatim, log ~line 8435–8460): get_processor (processor.py:271) → get_tokenizer_from_processor (common.py:529) → return processor.tokenizer → AttributeError: 'CLIPImageProcessor' object has no attribute 'tokenizer'. The server process then exits code 1; popen_launch_server raises Exception: Server process exited with code 1 at test_utils.py:1045.
• The source comment at common.py (just above get_tokenizer_from_processor) already states: "Some models don't have an available processor, e.g.: InternVL" — the function only special-cases PreTrainedTokenizerBase, then unconditionally reads .tokenizer.
• Qwen scores this run: Qwen2-VL-7B 0.290 (thr 0.30), Qwen2.5-VL-7B 0.280 (thr 0.33), Qwen3-VL-30B-A3B 0.260 (thr 0.29). All 4-choice-random-adjacent (random ≈ 0.25).
• Warning present for all Qwen runs: Aiter backend is selected for fused RoPE. This has lower precision. To disable aiter, export USE_ROCM_AITER_ROPE_BACKEND=0 (apex/.../fused_rope.py:49).
• Environment: image rocm/sgl-dev:v0.5.15.post1-rocm720-mi30x-20260722, runner linux-mi325-2gpu, attention backend aiter (default).
• Test-abort mechanism: InternVL's popen_launch_server is at test line 216, outside the try/except/finally (lines 225–308), so its exception aborts test_mmmu_vlm_models entirely; models 5–12 are never exercised (no signal for them).

Hypothesised Causes (with confidence)

• [FACT] InternVL crash mechanism: AutoProcessor.from_pretrained("OpenGVLab/InternVL2_5-2B") returns a bare CLIPImageProcessor (no .tokenizer), and get_tokenizer_from_processor (common.py:529) unconditionally accesses processor.tokenizer. Directly observable in the stack trace.
  • Supporting: exact trace + the in-code comment acknowledging InternVL lacks a full processor; reproduces identically across 6+ runs and many head SHAs.
  • Disconfirming: not tied to any commit in a narrow regression window — it has failed identically since at least Jul 14, so this is a long-standing InternVL support/processor-loading gap on this transformers/ROCm image, not a fresh code regression. (Root cause of why AutoProcessor returns only the image processor — transformers version behavior vs. missing sglang InternVL processor override — is [LOW] and needs a local repro.)
  • How to verify: on the CI image, run from transformers import AutoProcessor; AutoProcessor.from_pretrained("OpenGVLab/InternVL2_5-2B", trust_remote_code=True) and inspect the returned type; check whether sglang registers an InternVL-specific processor path.
• [LOW] Qwen accuracy shortfall from aiter fused-RoPE precision / tight thresholds: MMMU scores hover ~0.26–0.29, just under thresholds; the aiter fused-RoPE "lower precision" warning is emitted.
  • Supporting: only AMD/aiter path; scores are stable and coherent per-category (not garbage), consistent with a small precision/threshold gap rather than breakage.
  • Disconfirming: Qwen2-VL-7B reached exactly 0.300 (passing) on Jul 14 & 17, so the shortfall is borderline/threshold-sensitive, not a hard functional break; no evidence linking to a specific commit.
  • How to verify: re-run Qwen models with USE_ROCM_AITER_ROPE_BACKEND=0 and compare MMMU scores against thresholds; if scores rise materially, precision is implicated; otherwise thresholds may be mis-calibrated for ROCm720.

In-flight Fix Check

• InternVL crash: In-flight fix: ❌ none found — searches is:pr is:open InternVL+CLIPImageProcessor (0), CLIPImageProcessor (0), get_tokenizer_from_processor (2, both unrelated: #28270 Qwen-VL event loop, #10866 EAGLE) returned nothing addressing this path.
• Qwen VLM accuracy on AMD: In-flight fix: ❌ none found — VLM+MMMU+AMD+accuracy returned only unrelated PRs.

Suggested Triage Steps

• Highest ROI — stop the suite-wide abort: the InternVL popen_launch_server at test line 216 is outside the try/except, so its crash masks all 8 later models. Consider moving launch inside the try block (records error + continues) or adding OpenGVLab/InternVL2_5-2B to AMD_FAILING_VLM_MODELS until the processor gap is fixed — this alone restores signal for MiniCPM/DeepSeek/Kimi/MiMo/DeepSeek-OCR.
• InternVL root cause: repro AutoProcessor.from_pretrained on the CI image (above); if transformers returns only CLIPImageProcessor, the fix direction is an InternVL-specific processor construction path in get_processor, not a patch to get_tokenizer_from_processor.
• Qwen thresholds/precision: A/B the three Qwen models with USE_ROCM_AITER_ROPE_BACKEND=0 vs default; decide whether to adjust thresholds (Qwen2.5-VL 0.33 and Qwen3-VL 0.29 have never passed in-window) or treat as a real aiter precision regression.

Status

• Persistence: Both signatures recurring for ≥8 days (InternVL crash in every completed run Jul 14–22; Qwen2.5-VL & Qwen3-VL sub-threshold every run).
• Scope: 1 job (nightly-accuracy-2-gpu-vlm-rocm720), 1 test file. The InternVL crash additionally suppresses signal for 8 downstream VLMs in the same suite.
• Blocked work: No green signal for the AMD 2-GPU VLM MMMU accuracy suite on ROCm 7.2.
• In-flight fix: ❌ none found for either cluster.

nightly-8-gpu-mi35x-qwen3-235b-mxfp4-rocm720 — failed step(s): Accuracy Test + Performance Test MI35x ROCm 7.2 (8-GPU Qwen3-235B-MXFP4)

Commit Info

• sglang: 0bdd4730a (#32091 "[CI] Fix failures on main")
• aiter: 9127c94a1 — v0.1.17.dev110+g9127c94a1 (source: dev/patched CI image; [CI-AITER-CHECK] reported "Dev/patched version detected → skipping rebuild", Dockerfile default was the same SHA)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/test_qwen3_instruct_mxfp4.py	TestQwen3Instruct2507MXFP4.setUpClass	Exception: Server process exited with code 1 — server never became ready; scheduler died with RuntimeError: Rank 0 scheduler died during initialization (exit code: -6)	Log

The single test in this suite never runs its assertions — the failure is in setUpClass (server launch), so test_a_gsm8k / accuracy tests were never reached. Final result: Test Summary: 0/1 passed.

Failure Summary

The nightly-8-gpu-mi35x-qwen3-235b-mxfp4-rocm720 job launches a Qwen3-235B-A22B-Instruct-2507-mxfp4 server (--tp 4 --ep 2 --attention-backend aiter, SGLANG_USE_AITER=1) on MI35x/gfx950. During decode full CUDA graph capture (first batch, bs=512), all TP ranks hit a HIP error operation not permitted on an event last recorded in a capturing stream (hipErrorCapturedEvent), the RCCL watchdog aborts each rank (SIGABRT, exit -6), and the scheduler dies during init so the server never comes up. This is a server-launch crash, not a test-assertion failure.

Regression Status

Known recurring failure (this cluster) — consistent for the last 5 completed scheduled runs (since 2026-07-17). Never-passed for this test file in the queryable window (no green run found for test_qwen3_instruct_mxfp4.py in the last 15 comparable runs).

Recent history of test/registered/amd/test_qwen3_instruct_mxfp4.py (TestQwen3Instruct2507MXFP4.setUpClass) in job nightly-8-gpu-mi35x-qwen3-235b-mxfp4-rocm720 (most recent comparable completed scheduled runs on main, event=schedule):

Date	Run	Test File Status	Failure Signature
Jul 23	29944173150	❌ Failed	hipErrorCapturedEvent during decode CUDA graph capture (exit -6)
Jul 21	29854963573	❌ Failed	same — hipErrorCapturedEvent (exit -6)
Jul 19	29697452589	❌ Failed	same — hipErrorCapturedEvent (exit -6)
Jul 18	29654521083	❌ Failed	same — hipErrorCapturedEvent (exit -6)
Jul 17	29601455852	❌ Failed	same — hipErrorCapturedEvent (exit -6)
Jul 16	29521474931	⏭️ Cancelled	run cancelled (excluded)
Jul 14	29355338821	❌ Failed (different cluster)	GPU Hang / HW Exception during weight load (exit 134)
Jul 13	29273096587	❌ Failed (different cluster)	GPU Hang / HW Exception during weight load (exit 134)

First observed (this signature): 2026-07-17. No passing run for this test file in the window; the Jul 13–14 failures are a distinct GPU-hang-during-weight-load cluster, not this capture crash.

Failure Cluster

hipErrorCapturedEvent in aiter custom_all_reduce during full CUDA graph capture

Facts (verified observations, no interpretation)

• Suite launched via run_suite.py --hw amd --suite nightly-8-gpu-mi35x-qwen3-235b-mxfp4 --nightly (log line ~3846); only file is test_qwen3_instruct_mxfp4.py.
• Capture began: Capture target decode CUDA graph begin. backend=full, num_tokens_per_req=1, bs=[1..512] on all of TP0–TP3 (log ~4781–4787), then Capturing batches (bs=512 ...) (log ~4790).
• Error (verbatim, all ranks): [ProcessGroupNCCL.cpp:2057] ... Process group watchdog thread terminated with exception: HIP error: operation not permitted on an event last recorded in a capturing stream (log ~4847, 4955, 5056), plus [FATAL ERROR]: HIP failure: 'operation failed due to a previous error during capture' / Application is trying to use an invalidated stream to launch RCCL kernel (log ~5057–5058).
• Crash stack (rank threads, log ~5019–5048): aiter/dist/device_communicators/custom_all_reduce.py:608 custom_all_reduce → :589 all_reduce → sglang parallel_state.py:870 _all_reduce_out_place → vocab_parallel_embedding.py:574 forward → qwen2_moe.py:893 / qwen3_moe.py:994 forward → decode_cuda_graph_runner.py capture path → cuda_graph_setup.py:333 capture_decode_graph.
• Terminal outcome: RuntimeError: Rank 0 scheduler died during initialization (exit code: -6) → Exception: Server process exited with code 1 in setUpClass → FAILED (errors=1) → Test Summary: 0/1 passed (log ~5303–5337).
• aiter is a dev/patched image build (9127c94a1); no AITER_COMMIT_OVERRIDE set (this is a regular nightly, not AITER Scout).

Hypothesised Causes (with confidence, with disconfirming evidence)

• [MEDIUM] aiter's custom all-reduce (custom_all_reduce.py:589/608) is not HIP-graph-capture-safe under backend=full decode CUDA graph for this Qwen3-235B MXFP4 config: it records/waits on a HIP event on a stream that is already being captured, which is exactly what hipErrorCapturedEvent ("operation not permitted on an event last recorded in a capturing stream") reports.
  • Supporting: the failing frame is aiter custom_all_reduce, called during graph capture (decode_cuda_graph_runner → capture_decode_graph); the HIP error semantics match an illegal event op during capture; reproducible identically across all 4 TP ranks and 5 consecutive nightlies.
  • Disconfirming: the exact aiter commit (9127c94a1) is unchanged across the passing-window search (dev image), and no green baseline exists for this test file, so this could be a config that has never worked on gfx950 rather than a regression. Cannot A/B without a known-good aiter/sglang pair.
  • How to verify: relaunch the server locally with --attention-backend aiter --tp 4 --ep 2 and (a) --cuda-graph-backend-decode disabled (or --disable-cuda-graph) to confirm capture is the trigger, then (b) toggle aiter custom all-reduce (--disable-custom-all-reduce / disable SGLANG_USE_AITER) to isolate the aiter comm path.
• [LOW] A general RCCL/graph-capture interaction on ROCm 7.2 / MI35x (rather than aiter-specific).
  • Supporting: RCCL fatal-errors on invalidated stream appear alongside the HIP error.
  • Disconfirming: the RCCL messages are downstream of the first hipErrorCapturedEvent raised inside aiter's all-reduce (the RCCL "previous error during capture" is a consequence, not the origin).
  • How to verify: check whether a non-aiter comm backend on the same runner captures cleanly.

In-flight Fix Check

• In-flight fix: ⚠️ partial/related — #27650 "[AMD] Fix PCG capture fail with aiter fused all-reduce+RMSNorm" (open since 2026-06-09) makes aiter all-reduce capture-safe, but targets the --enable-aiter-allreduce-fusion + --enforce-piecewise-cuda-graph (KimiK2.5) path, whereas this job crashes in plain ca_comm.custom_all_reduce under backend=full. May overlap but is not a confirmed fix for this config — verify before assuming coverage.
• In-flight fix: ❌ none found for an exact match (searched open PRs for hipErrorCapturedEvent → 0 results; qwen3 mxfp4 capture → no direct capture-crash fix).

Suggested Triage Steps

• Reproduce on MI35x/gfx950 in the nightly ROCm 7.2 image: launch amd/Qwen3-235B-A22B-Instruct-2507-mxfp4 with --tp 4 --ep 2 --attention-backend aiter SGLANG_USE_AITER=1, then A/B --disable-cuda-graph to confirm the capture path is the trigger.
• Isolate the comm path: rerun with aiter custom all-reduce disabled (--disable-custom-all-reduce) to confirm the crash originates in aiter/dist/device_communicators/custom_all_reduce.py.
• Confirm whether #27650 (or a follow-up) also covers non-fused custom_all_reduce capture safety; if it does, chase reviewers rather than opening a duplicate. If not, this needs a new aiter-side capture-safety fix.
• If capture cannot be made safe on gfx950 for this model soon, consider disabling decode CUDA graph for this suite on AMD until the aiter comm path is capture-safe (author/maintainer decision).

Status (factual, no priority assignment)

• Persistence: this signature failing for 6 days (every completed scheduled run since 2026-07-17 06:41 UTC: Jul 17, 18, 19, 21, 23; Jul 16 cancelled). Test file has no green run in the queryable window.
• Scope: 1 job (nightly-8-gpu-mi35x-qwen3-235b-mxfp4-rocm720) in nightly-test-amd-rocm720.yml; all 4 TP ranks affected. Blocks all Qwen3-235B MXFP4 accuracy/GSM8K signal on MI35x ROCm 7.2.
• In-flight fix: ⚠️ related-only #27650 (different config); no exact-match fix found.

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-07-23 23:02 UTC)

[amd-bot]

pr-test-amd-rocm720.yml — 19 failure(s)

Run started (UTC): 2026-07-22 17:53 UTC
Last scanned (UTC): 2026-07-23 23:34 UTC
Commits: sglang d0b9689

Summary

Counts: 21 failures · 12 clusters · 16 carrying over (≥3 days) · 5 new (≤2 days) — across 2 scheduled runs (29944235234 Jul 22, 30031591211 Jul 23)

#	Cluster	Job	Test File	Test Function	Status	Hypothesis (confidence)
1	Diffusion 1-GPU 60-min timeout	mmgen-1gpu (…,1)	test_server_1_gpu.py	test_diffusion_generation[ltx_2_3_hq_pipeline]	no green shard	[HIGH] suite runtime > 60-min step budget
2	Diffusion 1-GPU 60-min timeout	mmgen-1gpu (…,0)	test_server_1_gpu.py	test_diffusion_generation[longlive2_i2v]	Never-passed Jul8–22	[MEDIUM] suite over budget
3	Diffusion 1-GPU 60-min timeout	mmgen-1gpu (…,1)	test_server_1_gpu.py	test_diffusion_generation[wan2_1_t2v_1.3b_teacache_enabled]	recurring, no green	[HIGH] suite runtime > 60-min
4	Diffusion 1-GPU 60-min timeout	mmgen-1gpu (…,0)	test_server_1_gpu.py	test_diffusion_generation[sana_wm_ti2v]	timeout Jul20–23	[MEDIUM] suite over budget
5	Triton throughput < 3500 gate	stage-b-1gpu-large (…,0)	test_bench_serving_1gpu_part1.py	test_offline_throughput_with_triton_attention_backend	Never-passed ≥14d	[MEDIUM] 3500 AMD gate miscalibrated
6	Triton throughput < 3500 gate	stage-b-1gpu-large (…,0)	test_bench_serving_1gpu_part1.py	test_offline_throughput_with_triton_attention_backend	3226<3500, no green	[FACT] persistent threshold gap
7	Shard-3 runner lost comms	mmgen-1gpu (…,3)	test_update_weights_from_disk.py	(killed, no log)	Never-passed 15/15	[LOW] runner killed mid-step, no log
8	Shard-3 runner lost comms	mmgen-1gpu (…,3)	N/A (runner lost comms)	N/A	14 consec since Jul9	[LOW] self-hosted runner drop
9	msgpack ModuleNotFound (AMD)	mmgen-unit-test	test_pi05_action_api.py	test_msgpack_roundtrip_preserves_string_keys_and_numpy_payloads	recurring since Jul11	[HIGH] msgpack absent from pyproject_other.toml
10	msgpack ModuleNotFound (AMD)	mmgen-unit-test	test_pi05_action_api.py	test_msgpack_roundtrip_...	recurring since Jul14	[HIGH] msgpack not in ROCm image
11	Kimi-MXFP4 TP load hang → -9	stage-c-8gpu-mi35x (…,1)	test_kimi_k25_mxfp4_bcg_mi35x.py	test_bcg_gsm8k	6/7 runs since Jul16	[MEDIUM] multithread weight-load hang
12	Kimi-MXFP4 TP load hang → -9	stage-c-8gpu-mi35x (…,1)	test_kimi_k25_mxfp4_bcg_mi35x.py	test_bcg_gsm8k	Never-passed since Jul16	[MEDIUM] TP=4 HIP-mutex load hang
13	Mooncake KV-transfer timeout	stage-b-8gpu-mi35x-disagg	test_disaggregation_pp.py	TestDisaggregationPrefillPPDynamicChunkAccuracy.test_gsm8k	recurring since Jul17 (5/7)	[MEDIUM] RDMA/mooncake flake
14	Mooncake KV-transfer timeout	stage-b-8gpu-mi35x-disagg	test_disaggregation_pp.py	TestDisaggregationDecodePPAccuracy.test_gsm8k	recurring since Jul19	[LOW] no in-window code cause
15	HF DNS name-resolution flake	stage-b-1gpu-small (…,2)	test_radix_cache_hit.py	TestRadixCacheHit.setUpClass	new today, green ×5 prior	[HIGH] transient DNS to huggingface.co
16	HF DNS name-resolution flake	stage-b-1gpu-large (…,0)	test_awq.py	TestAWQ.test_mmlu	new today	[FACT] MMLU CSV download DNS fail
17	Cosmos3-Nano GQA/AITer crash	mmgen-1gpu (…,0)	test_server_1_gpu.py	test_diffusion_generation[cosmos3_nano_t2i]	Never-passed on AMD (Jul07+)	[FACT] AITer no-GQA + diffusers missing class
18	2-GPU diffusion hang (no log)	mmgen-2gpu (…,0)	test_server_2_gpu.py	TestDiffusionServerTwoGpu.test_diffusion_generation	Never-passed 15/15	[LOW] FP8 GEMM / hang (no log)
19	MXFP4 GSM8K < 0.92 gate	stage-b-1gpu-small-mi35x	test_quark_mxfp4.py	TestFP8ToMXFP4MOETP1.test_gsm8k	1st hard fail Jul22 (borderline)	[MEDIUM] 0.92 gate too tight
20	Dynamo "Unsupported method call"	stage-c-8gpu-mi35x (…,2)	test_deepseek_r1_mxfp4_8gpu.py	TestDeepseekR1MXFP4.setUpClass	new Jul22 (stale SHA)	[HIGH] #31816 get_parallel() in compiled forward
21	communicator missing get_server_args	stage-c-8gpu-rocm720 (…,3)	test_aiter_allreduce_fusion_amd.py	test_dense_tp_fuses	new Jul22	[HIGH] #31816 removed get_server_args

Cluster: Diffusion 1-GPU 60-min timeout (rows 1–4)

• Affected: shards 0 & 1 of multimodal-gen-test-1-gpu-amd-rocm720 in both runs; shard 2 passes. Failure point drifts by case (ltx_2_3, longlive2_i2v, wan2_1…teacache, sana_wm) but the step always dies at the 60-min GH-Actions cap.
• Shared facts: timeout-minutes: 60 at .github/workflows/pr-test-amd-rocm720.yml:728; ~11–12 sequential server launch/warmup/teardown cases + cold HF re-downloads (LTX-2.3 not pre-cached) exceed budget. No assertion failure.
• In-flight fix: ❌ none. Triage: sum per-case durations shard 0/1 vs shard 2; rebalance part:[0,1,2,3] or raise the step timeout; pre-seed HF cache (LTX-2.3/LongLive).

Cluster: Triton throughput < 3500 gate (rows 5–6)

• Affected: stage-b-test-1-gpu-large-amd-rocm720 (…,0) both runs. Measured 3104–3244 tok/s, stable ~7–11% under the hardcoded 3500 AMD floor (test_bench_serving_1gpu_part1.py:120); Never-passed in the queryable window.
• In-flight fix: ❌ none. Triage: git log -S '3500' on a full clone to see if the gate was ever met on MI325/ROCm-7.2 → recalibrate or bisect a pre-window green baseline.

Cluster: Shard-3 runner lost comms / killed (rows 7–8)

• Affected: multimodal-gen-test-1-gpu-amd-rocm720 (…,3) both runs; step 9 conclusion None, log blob BlobNotFound, 0 artifacts, killed ~20–33 min in. 14 consecutive since Jul 9; sibling shard 2 passes → workload-linked, not fleet-wide.
• In-flight fix: ❌ none. Triage: pull runner host dmesg/journal at drop time; add out-of-band resource logger; confirm same physical node vs rotating.

Cluster: msgpack ModuleNotFound on AMD (rows 9–10)

• Affected: multimodal-gen-unit-test-amd-rocm720 both runs (1 fail / 793 pass). msgpack is in pyproject.toml but absent from pyproject_other.toml (AMD manifest), so lazy import msgpack in vla/protocol.py:59 fails. Boundary aligns with Pi0.5 test addition (#30633).
• In-flight fix: ✅ #31899 (open since 2026-07-21) — chase reviewers, do NOT duplicate.

Cluster: Kimi-K2.5-MXFP4 TP load hang → exit -9 (rows 11–12)

• Affected: stage-c-test-large-8-gpu-amd-mi35x-rocm720 test_bcg_gsm8k (TP=4). All loader threads park in a HIP driver mutex in _load_w13 on the final weight shard → 300s watchdog → dist_barrier_after_load ValueError → SIGKILL. Sibling non-BCG Kimi at TP=8 passes with same model/flags.
• In-flight fix: ❌ none. Triage: rerun with enable_multithread_load:false and separately --tp 8 to isolate the multithread-copy-at-TP4 hypothesis.

Cluster: Mooncake KV-transfer timeout, PD disagg (rows 13–14)

• Affected: stage-b…mi35x-disaggregation test_disaggregation_pp.py::test_gsm8k. Sync batch data transfer timeout after 30s → KVTransferError: prefill … might be dead → decode PP scheduler stalls. Only PP + mooncake + Qwen3-8B fails; non-PP mooncake basic test passes. scheduler_pp_mixin.py unchanged in the regression window.
• In-flight fix: ⚠️ candidates #31800, #31589, #29127 — none confirmed. Triage: bisect b3a0185cab..d4801be447 plus diff mooncake/RDMA image contents Jul18→Jul19.

Cluster: HF DNS name-resolution flake (rows 15–16)

• Affected: test_radix_cache_hit.py::setUpClass (config HEAD) and test_awq.py::test_mmlu (dataset CSV), both [Errno -3] Temporary failure in name resolution with HF_HUB_OFFLINE=0. New today; both paths green in prior runs.
• In-flight fix: ❌ none. Triage: re-run to confirm transient; consider pre-seeded cache + offline mode so a blip can't hard-fail startup.

Cluster: RuntimeContext get_parallel() refactor fallout — #31816 (rows 20–21)

• Two distinct symptoms, same culprit commit 745b2ca4: row 20 = Dynamo Unsupported method call (ParallelContext.__getattr__→_config) during DeepSeek prefill piecewise-CUDA-graph compile; row 21 = mock.patch.object(comm,"get_server_args") AttributeError after the symbol was removed from communicator.py. Both passed on Jul-21 head 9a6c9608, fail on Jul-22 head 0bdd4730.
• In-flight fix: row 20 ✅ #32100 already merged as f5dcbe8f (this run predates it → transient). Row 21 ❌ none — the stale test mock is NOT reverted by #32100; needs a test-side fix.

Cluster: Cosmos3-Nano GQA/AITer crash (row 17)

• nvidia/Cosmos3-Nano is GQA (aiter.py:128 rejects), native diffusers fallback lacks Cosmos3OmniTransformer (diffusers 0.37.1). Never-passed on AMD since Jul 07. In-flight fix: ❌ none. Triage: skip on AMD/AITer or add GQA support.

Cluster: 2-GPU diffusion hang, no log (row 18)

• multimodal-gen-test-2-gpu (…,0) Never-passed 15/15; mode shifted Jul 17 from clean FLUX.2 FP8 HIPBLAS_STATUS_NOT_SUPPORTED failure to a logless hang. In-flight fix: ✅ #28889 for the FP8-GEMM cause; ❌ for the hang. Triage: add per-case watchdog/py-spy so the next hang leaves a trail.

Cluster: MXFP4 GSM8K < 0.92 gate (row 19)

• test_quark_mxfp4.py::test_gsm8k measured 0.918/0.91 vs 0.92 gate; accuracy band 0.906–0.928 straddles the threshold. In-flight fix: ❌ none. Triage: re-run 3–5× to confirm noise vs regression before touching the gate.

Suggested triage order

Rank	Triage step	Targets	Why this first
1	Confirm next scheduled run on SHA ≥ f5dcbe8f is green; open a test-side fix for the stale get_server_args mock	rows 20–21	Both are fresh #31816 regressions; row 20 fix already merged, row 21 still needs one
2	Chase/merge msgpack fix #31899	rows 9–10	Confirmed in-flight fix, HIGH confidence, cheap
3	Rebalance/raise timeout for diffusion 1-GPU shards; add shard-3 resource logger + 2-GPU per-case watchdog	rows 1–4, 7–8, 18	Largest cluster + two logless clusters share the same partition/infra levers
4	Re-run to confirm DNS flakes are transient	rows 15–16	Likely infra, cheap disconfirmation
5	Single-variable reruns: Kimi multithread_load:false/--tp 8; mooncake bisect + image diff	rows 11–14	Persistent MI35x hangs; need evidence before code blame
6	Threshold calibration decisions (perf 3500 floor; MXFP4 0.92 gate)	rows 5–6, 19	Maintainer judgement; low ROI until above cleared

---

Job	Failed Steps	Started
stage-b-test-1-gpu-small-amd-mi35x-rocm720 (linux-mi35x-gpu-1)	Run test	2026-07-22T17:53
stage-b-test-large-8-gpu-mi35x-disaggregation-amd-rocm720 (linux-mi35x-gpu-8.fabric)	Run test_disaggregation	2026-07-22T18:00
stage-c-test-large-8-gpu-amd-mi35x-rocm720 (linux-mi35x-gpu-8, 1)	Run test	2026-07-22T21:26
stage-c-test-large-8-gpu-amd-mi35x-rocm720 (linux-mi35x-gpu-8, 2)	Run test	2026-07-22T21:30
stage-b-test-1-gpu-large-amd-rocm720 (linux-mi325-1gpu-sglang, 0)	Run test	2026-07-22T18:04
multimodal-gen-test-1-gpu-amd-rocm720 (linux-mi325-1gpu-sglang, 1)	Run diffusion server tests (1-GPU)	2026-07-22T18:04
multimodal-gen-test-1-gpu-amd-rocm720 (linux-mi325-1gpu-sglang, 0)	Run diffusion server tests (1-GPU)	2026-07-22T18:04
multimodal-gen-test-2-gpu-amd-rocm720 (linux-mi325-2gpu-sglang, 0)	N/A	2026-07-22T20:30
multimodal-gen-unit-test-amd-rocm720	Run diffusion unit tests	2026-07-22T18:04
stage-c-test-large-8-gpu-amd-rocm720 (linux-mi325-8gpu-sglang, 3)	Run test	2026-07-22T18:42
multimodal-gen-test-1-gpu-amd-rocm720 (linux-mi325-1gpu-sglang, 3)	N/A	2026-07-22T18:05
stage-b-test-large-8-gpu-mi35x-disaggregation-amd-rocm720 (linux-mi35x-gpu-8.fabric)	Run test_disaggregation	2026-07-23T17:57
stage-c-test-large-8-gpu-amd-mi35x-rocm720 (linux-mi35x-gpu-8, 1)	Run test	2026-07-23T20:21
multimodal-gen-unit-test-amd-rocm720	Run diffusion unit tests	2026-07-23T21:40
multimodal-gen-test-1-gpu-amd-rocm720 (linux-mi325-1gpu-sglang, 3)	N/A	2026-07-23T21:48
multimodal-gen-test-1-gpu-amd-rocm720 (linux-mi325-1gpu-sglang, 0)	Run diffusion server tests (1-GPU)	2026-07-23T21:41
stage-b-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang, 2)	Run test	2026-07-23T22:33
multimodal-gen-test-1-gpu-amd-rocm720 (linux-mi325-1gpu-sglang, 1)	Run diffusion server tests (1-GPU)	2026-07-23T21:42
stage-b-test-1-gpu-large-amd-rocm720 (linux-mi325-1gpu-sglang, 0)	Run test	2026-07-23T22:20

Per-Job Analysis

stage-b-test-1-gpu-small-amd-mi35x-rocm720 (linux-mi35x-gpu-1) — failed step(s): Run test

Commit Info

• sglang: 0bdd4730
• aiter: 9127c94a (v0.1.17.dev110+g9127c94a1) — source: dev image detected ([CI-AITER-CHECK] Dev/patched version detected … → skipping rebuild), matches Dockerfile default for gfx950.

Failed Tests

Test File	Test Function	Error	Log
test/registered/quant/test_quark_mxfp4.py	TestFP8ToMXFP4MOETP1.test_gsm8k	AssertionError: 0.918 not greater than 0.92 (retry: 0.91 not greater than 0.92)	Log

The other 4 files in the suite (test_dsa_skip_logits_clean.py, test_verify_splitkv.py, test_mamba_state_scatter_triton.py, test_deepseek_mla_dispatch.py) all passed. Suite result: 4/5.

Failure Summary

The single failing test is TestFP8ToMXFP4MOETP1.test_gsm8k, a GSM8K accuracy gate for the Qwen3-30B-A3B-Instruct-2507-FP8 model after online FP8→MXFP4 requantization. It requires accuracy > 0.92. The run measured 0.918 on the first attempt and 0.91 on the single retry, so both attempts fell just below the threshold and the test errored out. All other tests in the shard passed.

Regression Status

Marginal-threshold failure (near-flaky) — first hard failure for this job today (2026-07-22).

The test file was relanded 2026-07-21 in PR #28291 (dcd9014f), so it only exists in scheduled runs from 07-21 onward. Recent history of test_quark_mxfp4.py (TestFP8ToMXFP4MOETP1.test_gsm8k) in job stage-b-test-1-gpu-small-amd-mi35x-rocm720 (comparable completed event=schedule, branch=main runs):

Date	Run	Job	Test File Status	Measured Accuracy (threshold 0.92)	Result
Jul 22	29944235234	stage-b…mi35x-rocm720	❌ Failed	0.918 → retry 0.91	both below → fail
Jul 21	29855003895	stage-b…mi35x-rocm720	✅ Passed	0.906 → retry 0.928	first below, retry above → pass
Jul 20	29766743590	stage-b…mi35x-rocm720	✅ Passed (test predates reland)	—	pre-reland form

First observed failure for this test: 2026-07-22. Last known passing: 2026-07-21 (but it passed only on retry, first attempt 0.906 was already below threshold). The measured accuracy has hovered in a 0.906–0.928 band across two days, straddling the 0.92 gate — i.e. the test was already borderline on its first passing day.

Failure Cluster

MXFP4 GSM8K accuracy just below threshold (0.92 gate too tight for Qwen3-30B MXFP4)

Facts (verified observations)

• Verbatim: AssertionError: 0.918 not greater than 0.92 (first attempt, log line 7744), then on retry AssertionError: 0.91 not greater than 0.92 (log line 9183), then Exception: retry() exceed maximum number of retries. — FAILED (errors=1, skipped=3).
• Test source: test/registered/quant/test_quark_mxfp4.py:251 calls _test_gsm8k(accuracy_threshold=0.92); comment states reference accuracy for this FP8 model is ~0.948, but the on-hardware MXFP4-requantized result is ~0.91–0.93.
• Model & path: Qwen/Qwen3-30B-A3B-Instruct-2507-FP8, TP=1, requantized FP8→MXFP4 (class TestFP8ToMXFP4MOETP1).
• The retry harness (python/sglang/srt/utils/common.py:3177 / test/test_utils.py:2330) allows exactly 1 retry; today both draws (0.918, 0.91) landed below 0.92.
• On the 07-21 passing run, the same test drew 0.906 (first, below) then 0.928 (retry, above) — verified from job 88717083448 log (Accuracy: 0.906 / 0.928).
• aiter version identical across 07-21 (pass) and 07-22 (fail) runs — this is a dev image with the Dockerfile-default aiter; not an aiter delta.
• All 4 other tests in the shard passed; no GPU fault, no import error, no server crash — the server served all 500 GSM8K requests (200 OK, output_throughput ~3100–3600 token/s).

Hypothesised Causes (with confidence)

• [MEDIUM] The 0.92 accuracy threshold is too tight for GSM8K on this MXFP4-requantized Qwen3-30B model on MI35x/ROCm 7.2; run-to-run accuracy variance (0.906–0.928 observed over two days) straddles the gate, so the test fails whenever both the initial draw and the single retry land below 0.92.
  • Supporting: measured accuracy band 0.906/0.910/0.916/0.918/0.928 across 07-21 and 07-22; 07-21 passed only via retry (0.906→0.928); reference for the un-quantized FP8 model is ~0.948 but MXFP4 requant lowers it near the gate.
  • Disconfirming: none strong. If a code regression had lowered accuracy, one would expect a larger, monotonic drop; instead values fluctuate around the threshold with no clear step-change, and no code in the diff between 07-21 (9a6c9608) and 07-22 (0bdd4730) obviously targets MXFP4 numerics (not yet exhaustively bisected — see triage).
  • How to verify: re-run the shard 5× on the same SHA; if it passes intermittently, the gate is too tight. Compare per-question output to confirm no systematic degradation.
• [LOW] A commit merged in the 07-21→07-22 window (git log 9a6c96083f..0bdd4730af) altered MXFP4/MoE numerics for this model, shaving ~1% off accuracy.
  • Supporting: measured band shifted slightly lower today (0.918/0.91 vs 0.906/0.928 — mixed, not clearly lower).
  • Disconfirming: today's first draw (0.918) is actually higher than 07-21's first draw (0.906); the shift is within observed noise, arguing against a real regression.
  • How to verify: git log --oneline 9a6c96083f..0bdd4730af and grep for mxfp4, quark, moe, quant, requant; A/B any suspect commit by reverting on a branch and re-running this test on MI35x.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for test_quark_mxfp4 (7 results, all add new checkpoints/features, none adjust the 0.92 gate), accuracy_threshold+mxfp4 (0 results), and mxfp4+gsm8k/quark+mxfp4 (none targeting this threshold). Queries: search/issues?q=repo:sgl-project/sglang+is:pr+is:open+test_quark_mxfp4, …+accuracy_threshold+mxfp4.

Suggested Triage Steps

• Re-run the stage-b-test-1-gpu-small-amd-mi35x-rocm720 shard 3–5× on SHA 0bdd4730 to measure the accuracy distribution and confirm whether this is threshold noise vs a consistent drop.
• If noise-bound: consider loosening the gate in test/registered/quant/test_quark_mxfp4.py:251 (e.g. accuracy_threshold=0.90) or increasing num_questions from 500 to tighten the estimate — decision for the AMD quant owners (PR #28291 authors).
• If a regression is suspected: bisect 9a6c9608..0bdd4730, focusing on commits touching MXFP4/Quark/MoE quant paths; revert any suspect on a branch and re-run this test on MI35x/ROCm 7.2.

Status

• Persistence: First hard failure on 2026-07-22; the same job passed 2026-07-21 (only via retry, 0.906→0.928) and 2026-07-20. Not yet persistent.
• Scope: 1 test (TestFP8ToMXFP4MOETP1.test_gsm8k) in 1 job/shard; the other 4 tests in the shard passed. No evidence of the same failure in sibling jobs.
• Blocked work: Blocks the stage-b-test-1-gpu-small-amd-mi35x ROCm 7.2 shard signal (job exits 255 on this one test).
• In-flight fix: ❌ none found — needs owner decision on threshold vs regression investigation.

stage-b-test-large-8-gpu-mi35x-disaggregation-amd-rocm720 (linux-mi35x-gpu-8.fabric) — failed step(s): Run test_disaggregation

Commit Info

• sglang: 0bdd4730
• aiter: 9127c94a — v0.1.17.dev110+g9127c94a1 (source: dev/patched image preserved; matches Dockerfile default, no rebuild)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/disaggregation/test_disaggregation_pp.py	TestDisaggregationPrefillPPDynamicChunkAccuracy.test_gsm8k	KeyError: 'answer' (in few_shot_gsm8k.run_eval), triggered by KVTransferError: Failed to get kvcache from prefill instance, it might be dead after Sync batch data transfer timeout after 30000001618ns	Log

The other 3 test files in this job passed (test_disaggregation_basic.py, test_mori_transfer_engine_e2e.py, test_nixl_transfer_engine_e2e.py). Job overall: 3/4 files passed, exit code 255.

Failure Summary

In the PD-disaggregation PP GSM8K accuracy test, 1–2 of the 200 concurrent (parallel=128) requests hit a 30-second mooncake KV-cache transfer timeout (transfer_engine_py.cpp:614), producing a 500 error on /generate. Because few_shot_gsm8k.run_eval (python/sglang/test/few_shot_gsm8k.py:123) unconditionally reads states[i]["answer"], one failed request raises KeyError: 'answer', which fails the whole test after retry() exhausts its single retry. The test never reached its accuracy assertion.

Regression Status

Known recurring / intermittent failure (same KV-transfer-timeout signature on Jul 17, 19, 20, 21, 22; passed Jul 18). This is not a fresh regression on today's commit.

Recent history of test_disaggregation_pp.py (TestDisaggregationPrefillPPDynamicChunkAccuracy.test_gsm8k) in job stage-b-test-large-8-gpu-mi35x-disaggregation-amd-rocm720 (comparable completed scheduled runs, branch=main, event=schedule):

Date	Run	Job	Test File Status	Failed Function	Error
Jul 22	29944235234	job	❌ Failed	test_gsm8k	KeyError: 'answer' ← KV transfer timeout
Jul 21	29855003895	job	❌ Failed	test_gsm8k	same (3× timeout)
Jul 20	29766743590	job	❌ Failed	test_gsm8k	same
Jul 19	29697501385	job	❌ Failed	test_gsm8k	same (2× timeout)
Jul 18	29654571916	job	✅ Passed (4/4)	—	—
Jul 17	29601526478	job	❌ Failed	test_gsm8k	same (2× timeout)
Jul 16	29521557976	job	⏭️ Cancelled	—	excluded
Jul 15	29438298954	job	⏭️ Cancelled	—	excluded
Jul 14	29355401408	job	❌ Failed (different)	N/A	Suite abort Fatal Python error: Aborted, exit 134 — pp results not produced

First observed (this signature, in window): Jul 17. Last known passing: Jul 18. Lookback: 15 comparable scheduled runs.

Failure Cluster

Mooncake RDMA KV-cache transfer 30s timeout in PD disaggregation (→ 500 → gsm8k KeyError: 'answer')

Facts (verified observations)

• Direct log evidence (this run): transfer_engine_py.cpp:614] Sync batch data transfer timeout after 30000001618ns then Decode transfer failed … KVTransferError … Failed to get kvcache from prefill instance, it might be dead (job log ~18:15:45 and ~18:16:18), followed by KeyError: 'answer' at few_shot_gsm8k.py:123 and Exception: retry() exceed maximum number of retries.
• Transfer backend: mooncake over RDMA — in CI PDDisaggregationServerBase sets --disaggregation-transfer-backend mooncake (python/sglang/test/server_fixtures/disaggregation_fixture.py:86); the 30s timeout is a mooncake sync-transfer timeout.
• Same signature reproduced across 5 of the last 7 non-cancelled scheduled runs (Jul 17,19,20,21,22); Jul 18 passed cleanly (4/4). Only 1–4 of 200 requests time out per failing run.
• Test fragility (FACT): run_eval reads states[i]["answer"] unconditionally, so a single failed request aborts the whole test rather than being scored as incorrect.
• aiter unchanged across passing (Jul 18) and failing runs (dev image v0.1.17.dev110+g9127c94a1 preserved throughout).

Hypothesised Causes (with confidence)

• [MEDIUM] Intermittent RDMA/mooncake transfer stall on the MI35x disaggregation runner (hardware/fabric flakiness), not a code regression.
  • Supporting: failure is intermittent (Jul 18 passed with same aiter and near-identical sglang); only a handful of 200 requests time out; a maintainer debug PR #29127 "Isolate faulty RDMA NIC (rdma5) on MI35x disaggregation runner" confirms known NIC issues on this exact runner class; UCX warning in log failed to get interface index for tw-eth5 (errno 19).
  • Disconfirming: cannot see per-NIC binding in this job log to confirm the same faulty NIC was in use; the timeout could also be a genuine mooncake receiver-stuck bug rather than pure hardware.
  • How to verify: re-run the job; check which rdma* device was bound; correlate failures with a specific NIC; test with the #29127 NIC isolation applied.
• [MEDIUM] Mooncake decode receiver gets stuck in "Transferring" when a transfer stalls and is not cleaned up, turning a transient slow transfer into a hard 30s timeout / dead-peer error.
  • Supporting: open PRs #31800 "time out KV receivers stuck in Transferring" and #31589 "[PD][NIXL] Clean up decode receiver state after failed transfers" target exactly this symptom; error text "it might be dead" matches receiver-side give-up.
  • Disconfirming: those PRs are not yet merged, so cannot be causally tied to Jul 18's pass; symptom may be upstream of receiver cleanup (i.e. transfer never starts).
  • How to verify: apply #31800 on a branch and re-run this job 5–10× to see if timeouts drop.
• [FACT] Test does not tolerate a single failed request — the KeyError (not an accuracy miss) is what fails the test.
  • How to verify: n/a (code-visible at few_shot_gsm8k.py:123).

In-flight Fix Check

• In-flight fix: ⚠️ candidates — #31800 (open 2026-07-20, "time out KV receivers stuck in Transferring") and #31589 (open 2026-07-17, "[PD][NIXL] Clean up decode receiver state after failed transfers") both address the KV-transfer-timeout symptom; #29127 (open 2026-06-24) addresses the faulty RDMA NIC on this runner. None is confirmed to fix this exact test. (Searched search/issues for disaggregation transfer timeout, KVTransferError.)

Suggested Triage Steps

• Re-run job 89005546196 several times to confirm intermittency and capture which rdma* NIC is bound on failing vs passing runs (test hardware/runner isolation per #29127).
• On a branch, apply #31800 and/or #31589 and re-run test_disaggregation_pp.py 5–10× on the MI35x disagg runner; compare timeout frequency.
• Consider making few_shot_gsm8k.run_eval tolerate missing answer (score as incorrect) so a single transient transfer failure degrades accuracy rather than throwing KeyError — decision for maintainers, would change what this test measures.

Status

• Persistence: Recurring/intermittent since at least Jul 17 (5 of last 7 non-cancelled scheduled runs failed with this signature; Jul 18 passed).
• Scope: stage-b-test-large-8-gpu-mi35x-disaggregation-amd-rocm720 job; single test file test_disaggregation_pp.py, function test_gsm8k. Mooncake/RDMA path on MI35x disaggregation runner.
• Blocked work: Blocks green signal for PD-disaggregation PP accuracy on AMD MI35x ROCm 7.2.
• In-flight fix: candidate PRs #31800, #31589, #29127 — none confirmed; do not open duplicates.

stage-c-test-large-8-gpu-amd-mi35x-rocm720 (linux-mi35x-gpu-8, 1) — failed step(s): Run test

Commit Info

• sglang: 0bdd4730
• aiter: 9127c94a (v0.1.17.dev110+g9127c94a1, source: Dockerfile default — dev image build detected & preserved, AITER_COMMIT_OVERRIDE empty)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/test_kimi_k25_mxfp4_bcg_mi35x.py	TestKimiK25MXFP4BcgMI35x.test_bcg_gsm8k (variant='bcg')	Exception: Server process exited with code -9 — server never became ready; ValueError: TP rank 2 could finish the model loading, but there are other ranks that didn't finish loading	Log

The other test in this job, test/registered/cuda_graph/breakable/test_breakable_cuda_graph.py, passed (Test Summary: 1/2 passed).

Failure Summary

The amd/Kimi-K2.5-MXFP4 server (TP=4) failed to start during the test_bcg_gsm8k variant. Two of the four TP ranks (TP0, TP3) never finished weight loading, while TP1/TP2 completed in ~380–395 s and then blocked at the post-load distributed barrier. After the 480 s barrier timeout (UNBALANCED_MODEL_LOADING_TIMEOUT_S), dist_barrier_after_load raised a ValueError, the parent received SIGQUIT, and kill_process_tree tore the server down (exit code -9). The popen_launch_server call therefore raised and the test errored.

Regression Status

Known recurring / intermittent failure — same signature on 6 of the 7 most recent comparable completed scheduled runs, with a single pass on 2026-07-19. NOT a fresh regression at 0bdd4730.

Recent history of test_kimi_k25_mxfp4_bcg_mi35x.py (test_bcg_gsm8k) in job stage-c-test-large-8-gpu-amd-mi35x-rocm720 (branch=main, event=schedule, completed only; the test floats between shard 0/1 due to time-balanced partitioning, so status is tracked at the test-file level):

Date	Run	Job (shard)	Test File Status	Failed Function	Error
Jul 22	29944235234	shard 1	❌ Failed	test_bcg_gsm8k	ranks didn't finish loading → exit -9
Jul 21	29855003895	shard 0	❌ Failed	test_bcg_gsm8k	ranks didn't finish loading → exit -9
Jul 20	29766743590	shard 0	❌ Failed	test_bcg_gsm8k	ranks didn't finish loading → exit -9
Jul 19	29697501385	shard 0	✅ Passed	—	—
Jul 18	29654571916	shard 0	❌ Failed	test_bcg_gsm8k	Server process exited with code -9
Jul 17	29601526478	shard 0	❌ Failed	test_bcg_gsm8k	ranks didn't finish loading → exit -9
Jul 16	29521557976	shard 0	❌ Failed	test_bcg_gsm8k	ranks didn't finish loading → exit -9

First observed failure in this lookback window: 2026-07-16. Last known pass: 2026-07-19. Lookback range: 2026-07-16 → 2026-07-22 (7 comparable completed runs examined; did not query beyond 07-16).

Failure Cluster

Kimi-K2.5-MXFP4 TP weight-loading hang → unbalanced-load barrier timeout → server exit -9 (MI35x)

Facts (verified observations)

• Server command (from log): sglang serve --model-path amd/Kimi-K2.5-MXFP4 --tensor-parallel-size 4 --mem-fraction-static 0.765 --kv-cache-dtype fp8_e4m3 --model-loader-extra-config '{"enable_multithread_load": true}' ... (quant=quark, MXFP4).
• Ranks that finished loading: TP2 Load weight end. elapsed=379.68 s (avail mem=145.02 GB, usage=133.35 GB); TP1 elapsed=395.06 s. No Load weight end line was ever emitted for TP0 or TP3.
• Repeated py-spy dumps at 21:43, 21:48, 21:53, 21:58 UTC consistently show TP0 (PID 4759) and TP3 (PID 4762) stuck in do_load_weights (deepseek_weight_loader.py:436) → load_weights (kimi_k25.py:888), while TP1/TP2 wait in monitored_barrier → dist_barrier_after_load (load_model_utils.py:264). The stuck ranks are still active (not dead) across the whole 15-minute window.
• deepseek_weight_loader.py:436 is future.result() inside a ThreadPoolExecutor loop — i.e. TP0/TP3 are blocked waiting on their concurrent weight-loading tasks.
• Final failure (22:02:31 UTC): ValueError: TP rank 2 could finish the model loading, but there are other ranks that didn't finish loading. It is likely due to unexpected failures (e.g., OOM) or a slow node. raised at load_model_utils.py:270, followed by Received sigquit from a child process, kill_process_tree ... pid=4581, and Exception: Server process exited with code -9.
• Exit code -9 (SIGKILL) is emitted by kill_process_tree after the barrier ValueError — it is the cleanup kill of the hung process tree, not necessarily an independent OOM-kill.
• Identical ValueError: ... didn't finish loading + exited with code -9 signature verified in the Jul 16/17/20/21 shard-0 logs; Jul 18 log shows the exited with code -9 half of the same signature.
• aiter version identical (v0.1.17.dev110+g9127c94a1) between failing and passing runs — not an aiter-version variable.

Hypothesised Causes (with confidence)

• [MEDIUM] Weight-loading hang / severe load imbalance on 2 of 4 TP ranks in the multithreaded Deepseek/Kimi weight loader.
  • Supporting: four consecutive py-spy dumps over 15 min pin TP0 & TP3 in do_load_weights (future.result()), never completing, while TP1/TP2 finished in ~380–395 s; the barrier failure message itself is the downstream symptom, not the cause.
  • Disconfirming: the test passed on 2026-07-19 with essentially the same code, so this is intermittent, not a hard deadlock — pointing to a contention/environmental factor rather than a pure code bug.
  • How to verify: re-run this test with enable_multithread_load=false (or a reduced loader thread count); if it stabilises, the concurrent loader is implicated.
• [LOW] Host-RAM pressure from enable_multithread_load=true (many shards decompressed concurrently for a large MXFP4 checkpoint) causing swap/thrash on the two slow ranks.
  • Supporting: the barrier error explicitly names OOM as a candidate; multithreaded load multiplies peak host memory.
  • Disconfirming: no OS OOM-killer / "Out of memory" line captured; the stuck processes remained alive in every py-spy dump (an OOM-killer would have terminated them, not left them executing).
  • How to verify: capture host dmesg / free-memory on the MI35x runner during load; check whether the two slow ranks share a NUMA node / host-memory domain.
• [LOW] Slow-node / disk-I/O variance on the specific MI35x runner (intermittent).
  • Supporting: loading already consumed ~380–395 s on the ranks that finished (close to the barrier budget); passes vs fails do not correlate with any code change in-window.
  • Disconfirming: none strong — consistent with the intermittent pattern; hard to distinguish from the MEDIUM hypothesis without host metrics.

No hypothesis reaches HIGH; distinguishing "loader hang" vs "host OOM/thrash" vs "slow node" requires host-level metrics not present in the job log.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for Kimi K2.5 loading, enable_multithread_load, dist_barrier_after_load, didn't finish loading. Matches (e.g. #28905 Kimi K2.6 w2-scale loading, #31744 Elastic EP recovery) address unrelated loading paths, not this MI35x TP-load hang / OOM. No open PR targets this failure.

Suggested Triage Steps

• Re-run test_kimi_k25_mxfp4_bcg_mi35x.py on MI35x with --model-loader-extra-config '{"enable_multithread_load": false}' (or a lower thread count) to test the loader-contention hypothesis.
• Capture host memory (free -g / dmesg -T | grep -i oom) and per-rank load timing on the MI35x runner during weight load to confirm/rule out host-RAM OOM vs slow-I/O.
• If confirmed as host-resource contention, consider raising UNBALANCED_MODEL_LOADING_TIMEOUT_S only as a mitigation after the root resource issue is understood — a longer timeout will not fix a genuine hang.
• Given the intermittent nature (1 pass in 7 runs), consider quarantining/marking this test as flaky on MI35x-rocm720 until the loader/host-resource issue is resolved, so it stops masking other regressions in this job.

Status

• Persistence: Failing in 6 of the 7 most recent comparable scheduled runs (2026-07-16 → 2026-07-22), single pass on 2026-07-19. Same signature throughout.
• Scope: stage-c-test-large-8-gpu-amd-mi35x-rocm720 job, test_bcg_gsm8k only; the sibling test_breakable_cuda_graph.py in the same job passes. Confined to the amd/Kimi-K2.5-MXFP4 TP=4 launch on MI35x.
• Blocked work: Blocks Kimi-K2.5-MXFP4 breakable-CUDA-graph accuracy signal on AMD MI35x (ROCm 7.2).
• In-flight fix: ❌ none found in open PRs.

stage-c-test-large-8-gpu-amd-mi35x-rocm720 (linux-mi35x-gpu-8, 2) — failed step(s): Run test

Commit Info

• sglang: 0bdd4730
• aiter: 9127c94a v0.1.17.dev110+g9127c94a1 (source: dev image preserved — [CI-AITER-CHECK] Dev/patched version detected → skipping rebuild; matches Dockerfile default)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/test_deepseek_r1_mxfp4_8gpu.py	TestDeepseekR1MXFP4.setUpClass	Exception: Server process exited with code -9 (root: RuntimeError: Capture prefill CUDA graph failed: torch._dynamo.exc.Unsupported: Unsupported method call)	Log

Suite result: 2/3 passed — test_kimi_k25_mxfp4.py ✅ and test_qwen3_coder_next_8gpu.py ✅ passed; only test_deepseek_r1_mxfp4_8gpu.py failed.

Failure Summary

The 8-GPU DeepSeek-R1 MXFP4 server failed to start: all TP ranks crashed during prefill CUDA-graph capture, and the test's setUpClass aborted with server exit code -9. The scheduler exception is a torch.compile/Dynamo tracing error (Unsupported method call) raised while compiling the DeepSeek forward pass in the TcPiecewiseCudaGraphBackend compile pass.

Regression Status

New regression (introduced 2026-07-22).

Recent history of test/registered/amd/test_deepseek_r1_mxfp4_8gpu.py in workflow pr-test-amd-rocm720.yml (event=schedule, branch=main, completed runs; the test's partition shard varies because --auto-partition-size changed from 2→3, so history is tracked at the test-file level, not the job shard):

Date	Run	Job	Test File Status	Failed Function	Error
Jul 22	29944235234	stage-c…mi35x-rocm720 (…, 2)	❌ Failed	TestDeepseekR1MXFP4.setUpClass	Unsupported method call in prefill graph capture
Jul 21	29855003895	stage-c…mi35x-rocm720 (…, 1)	✅ Passed	—	"passed": true, "elapsed": 802
≤ Jul 21	29766743590 …and earlier	—	✅ Passed	—	all predate culprit #31816 (merged Jul 22 08:18 UTC), so this Dynamo signature cannot occur

First observed failure: 2026-07-22. Last known pass: 2026-07-21 (run 29855003895, same test file, partition 1).

Failure Cluster

Dynamo "Unsupported method call" (type.__getattribute__) during prefill piecewise-CUDA-graph compile — get_parallel() config-leaf read inside compiled forward.

Facts (verified observations)

• Verbatim scheduler error (all 8 TP ranks): RuntimeError: Capture prefill CUDA graph failed: Unsupported method call → torch._dynamo.exc.Unsupported: Unsupported method call — Dynamo does not know how to trace method __getattribute__ of class type. Developer debug context: call_method BuiltinVariable(object) __getattribute__ [UserDefinedObjectVariable(ParallelContext), ConstantVariable(str: '_config')] (log lines 4736–4820).
• Stack trace (user code) reaches the fault via: deepseek_v2.py:2234 maybe_prefetch_next_full_attention_kv → communicator_dsa_cp.py:63 dsa_use_prefill_cp → dsa/utils.py:277 → dsa/utils.py:79 is_dsa_enable_prefill_cp → return get_parallel().enable_dsa_prefill_context_parallel → runtime_context.py:130 __getattr__ → config = object.__getattribute__(self, "_config"). Verified against local source: python/sglang/srt/layers/attention/dsa/utils.py:78-79 and python/sglang/srt/runtime_context.py:125-130.
• Compile-pass origin: tc_piecewise_cuda_graph_backend.py:192 _run_compile_pass → prefill_cuda_graph_runner.py:513 _run_forward → compilation/compile.py:233 trampoline (Dynamo trace of model.forward).
• Server exit code -9 (SIGKILL after sigquit cascade); setUpClass then raised Exception: Server process exited with code -9 (log line 5421–5430), suite FAILED (errors=1), file exit code 1 (log line 5446).
• The 2026-07-21 run passed this exact test file ("passed": true, "elapsed": 802, job 88717083140) on head_sha 9a6c96083f, which predates the culprit commit.
• aiter version is a dev/patched image build matching the Dockerfile default ([CI-AITER-CHECK]), unchanged relative to the passing run — not a variable here.

Hypothesised Causes (with confidence)

• [HIGH] Commit 745b2ca4 (#31816 "config: read parallel config leaves via get_parallel()") changed dsa/utils.py:79 from get_server_args().enable_dsa_prefill_context_parallel to get_parallel().enable_dsa_prefill_context_parallel. enable_dsa_prefill_context_parallel is not a slot/property on ParallelContext, so the read falls through to ParallelContext.__getattr__, whose object.__getattribute__(self, "_config") Dynamo cannot trace inside the compiled DeepSeek forward.
  • Supporting: (a) the exact modified line and its __getattr__→_config continuation appear verbatim in the failing stack trace; (b) the Dynamo debug context names UserDefinedObjectVariable(ParallelContext) accessing '_config'; (c) temporal fit — #31816 merged 2026-07-22 08:18 UTC, the test passed 2026-07-21 and failed on the first scheduled run after the merge; (d) #31816 is an ancestor of the tested SHA 0bdd4730, the revert is not.
  • Disconfirming: (none found) — the read is on the exact code path in the trace and the pre-#31816 form (get_server_args(), a dataclass attribute) traced cleanly.
  • How to verify: already effectively verified by the in-flight revert below; alternatively git revert 745b2ca4 (or apply #32100) on a branch and re-run test_deepseek_r1_mxfp4_8gpu.py on MI35x ROCm 7.2.

In-flight Fix Check

In-flight fix: ✅ #32100 "Revert RuntimeContext config-namespace reads/roles (#31813–#31817)" — already merged as f5dcbe8f at 2026-07-22 18:52 UTC (this scheduled run started 17:52 UTC on the pre-revert SHA 0bdd4730, so it did not include the fix). The revert restores dsa/utils.py to get_server_args() and rolls back runtime_context.py (~146 lines). API call: curl .../search/issues?q=repo:sgl-project/sglang+is:pr+32100.

Suggested Triage Steps

• Confirm the fix landed: re-run pr-test-amd-rocm720.yml on current main HEAD (post-f5dcbe8f) and verify test_deepseek_r1_mxfp4_8gpu.py passes — expected green since the revert is already merged.
• No new PR needed; this is a scheduled run against a stale pre-revert SHA. Ignore if the next scheduled run (on a SHA ≥ f5dcbe8f) is green.
• If the re-landing of #31813–#31817 is planned, add a Dynamo-safe accessor for enable_dsa_prefill_context_parallel / dsa_prefill_cp_mode (e.g. resolve outside the compiled region or make it a real property/slot on ParallelContext) before reintroducing the get_parallel() reads on the compiled forward path.

Status

• Persistence: 1 completed run (2026-07-22 17:52 UTC). Passed on 2026-07-21.
• Scope: test_deepseek_r1_mxfp4_8gpu.py (8-GPU DeepSeek-R1 MXFP4) on MI35x ROCm 7.2; the same get_parallel()-in-compiled-forward path can affect any DeepSeek-family model exercising DSA prefill-CP under piecewise CUDA graph. Other two tests in the shard passed.
• Blocked work: DeepSeek-R1 MXFP4 8-GPU CI signal on AMD MI35x (transient — tested SHA predates the fix).
• In-flight fix: ✅ #32100 merged as f5dcbe8f 2026-07-22 18:52 UTC.

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-07-23 23:34 UTC)

[amd-bot]

pr-test-amd-rocm720.yml — Per-Job Analysis (continued 2/4)

stage-b-test-1-gpu-large-amd-rocm720 (linux-mi325-1gpu-sglang, 0) — failed step(s): Run test

Commit Info

• sglang: 0bdd4730
• aiter: 9127c94a (v0.1.17.dev110+g9127c94a1, source: Dockerfile default / dev image preserved — AITER_COMMIT_OVERRIDE empty, no rebuild)

Failed Tests

Test File	Test Function	Error	Log
test/registered/perf/test_bench_serving_1gpu_part1.py	test_offline_throughput_with_triton_attention_backend	AssertionError: 3177.18 not greater than 3500	Log

Other 9 test files in this partition passed (Test Summary: 9/10 passed).

Failure Summary

The 1-GPU-large perf suite (partition 0) failed because test_offline_throughput_with_triton_attention_backend measured an output throughput of 3177.18 tok/s against the AMD-CI threshold of 3500 (test_bench_serving_1gpu_part1.py:120, self.assertGreater(res["output_throughput"], 3500) guarded by is_in_amd_ci()). The test's built-in retry() re-ran the benchmark (second attempt 3168.52 tok/s) and then raised Exception: retry() exceed maximum number of retries., so the job exited with code 255.

Regression Status

Never-passed (no green run found in the 15-run query window). Persistent perf-threshold shortfall.

Recent history of test_bench_serving_1gpu_part1.py (test_offline_throughput_with_triton_attention_backend) in job stage-b-test-1-gpu-large-amd-rocm720 (linux-mi325-1gpu-sglang, 0) (most recent comparable completed scheduled runs: branch=main, event=schedule, completed only):

Date	Run	Test File Status	Failed Function	Measured throughput
Jul 22	29944235234	❌ Failed	test_offline_throughput_with_triton_attention_backend	3177.18 (<3500)
Jul 21	29855003895	❌ Failed	same	3177.95
Jul 20	29766743590	❌ Failed	same	3159.69
Jul 19	29697501385	❌ Failed	same	3174.22
Jul 18	29654571916	❌ Failed	same	3155.41
Jul 17	29601526478	❌ Failed	same	3104.10
Jul 16	29521557976	❌ Failed	same	3157.62
Jul 15	29438298954	❌ Failed	same	3162.05
Jul 14	29355401408	❌ Failed	same	3167.52
Jul 13	29273187026	❌ Failed	same	3174.25
Jul 12	29202664092	❌ Failed	same	3197.36
Jul 11	29162225443	❌ Failed	same	3155.45
Jul 10	29113205616	❌ Failed	same	3175.84
Jul 9	29039527644	❌ Failed	server-startup crash (Ran 0 tests, test_offline_throughput_default errored)	n/a
Jul 8	28964460204	❌ Failed	test_offline_throughput_with_triton_attention_backend	3167.80

First observed failure: at or before the start of this window (Jul 8, the oldest queryable run). No passing run for this test file in the entire 15-run window. Measured throughput sits in a tight 3104–3197 tok/s band — a stable ~9–11% below the 3500 gate, not a one-off dip.

Failure Cluster

Perf-threshold shortfall — triton-backend offline throughput below AMD gate (test config / hardware).

Facts (verified observations)

• Verbatim error (current run): AssertionError: 3177.1755141208887 not greater than 3500, then Exception: retry() exceed maximum number of retries. (job log lines 9738, 9611).
• Assertion site: test/registered/perf/test_bench_serving_1gpu_part1.py:120 → self.assertGreater(res["output_throughput"], 3500) inside an if is_in_amd_ci(): branch (non-AMD gate is 3700).
• Benchmark result block for the run: 500 requests, Output token throughput 3177.18 tok/s, Total token throughput 9208.20 tok/s, Concurrency 351.93, benchmark duration 167.61 s.
• The 3500 threshold is not in the queryable git window: git blame shows only the shallow-clone boundary commit (e856eae9), i.e. the threshold predates the local history — it was not introduced by any commit in this regression window.
• Reproducibility: same test file + function failed in 14/14 comparable completed scheduled runs Jul 8–Jul 21; throughput consistently 3104–3197 tok/s (never above ~3200).
• aiter version unchanged across the window (dev image v0.1.17.dev110+g9127c94a1), so aiter is not a moving variable here.
• Runner: linux-mi325-1gpu-sglang (MI325, gfx942, ROCm 7.2), image rocm/sgl-dev:v0.5.15.post1-rocm720-mi30x-20260722.

Hypothesised Causes (with confidence)

• [MEDIUM] The 3500 AMD gate is set higher than this config's achievable throughput on MI325/ROCm-7.2 with --attention-backend triton --context-length 8192 (i.e. a mis-calibrated threshold, not a fresh regression).
  • Supporting: throughput is extremely stable (3104–3197 across 14 days, σ small) and never approaches 3500; no green run exists in the window; the gate is not modified by any commit in the queryable history.
  • Disconfirming: cannot see the commit that set 3500 (shallow clone) — if it was raised recently, this could instead be a genuine perf regression that the window doesn't capture. No known-good baseline throughput is available to confirm 3500 was ever met on this hardware.
  • How to verify: git log -S "output_throughput\"], 3500" --follow on a full clone to find when/why the gate was set to 3500 and whether it ever passed on MI325 ROCm-7.2.
• [LOW] A persistent throughput regression in the triton attention backend / serving path that predates Jul 8.
  • Supporting: absolute throughput (~3.2k tok/s) is well below the gate the author presumably calibrated to.
  • Disconfirming: no in-window commit correlates (throughput is flat, not step-changed); a regression would typically show a drop, but the whole window is uniformly ~3.1–3.2k.
  • How to verify: bisect on a full clone across a range predating Jul 8 to find the last run where this test measured >3500 tok/s (if any exists).

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for test_bench_serving+throughput (1 hit, #27650, unrelated PCG/aiter capture fix) and triton throughput amd (no PR targeting this perf gate). No open PR adjusts the 3500 threshold or the triton-backend throughput for this test.

Suggested Triage Steps

• On a full clone, run git log -S '3500' --follow -- test/registered/perf/test_bench_serving_1gpu_part1.py to find when the AMD gate was set to 3500 and whether a green MI325/ROCm-7.2 run ever existed; if the gate was never met on this hardware, it is a calibration issue rather than a regression.
• If calibration: propose lowering the AMD gate to a value consistent with the observed 3104–3197 band (with margin), on a branch, and re-run this suite in container rocm/sgl-dev:v0.5.15.post1-rocm720-mi30x-20260722 on MI325.
• If a prior green baseline (>3500) is found: bisect the range from that baseline to the first sub-3500 run to locate the throughput regression in the triton attention / serving path, then re-run test_offline_throughput_with_triton_attention_backend to confirm.
• Independently, note the Jul 9 run (29039527644) failed via a server-startup crash (Ran 0 tests) rather than the throughput gate — a separate, likely-transient infra symptom, not the same cluster.

Status

• Persistence: Failing continuously for ≥14 days (every completed scheduled run Jul 8–Jul 22); Never-passed in the queryable window.
• Scope: test_offline_throughput_with_triton_attention_backend in the stage-b-test-1-gpu-large-amd-rocm720 (…, 0) partition of pr-test-amd-rocm720.yml. This single test failure red-lines the whole partition (exit 255) despite the other 9 test files passing.
• Blocked work: Keeps the 1-GPU-large ROCm-7.2 scheduled perf signal permanently red, masking any genuine new regressions that would land in the same partition.
• In-flight fix: ❌ none found — needs a new PR (threshold recalibration) or a bisect to confirm/deny a perf regression.

multimodal-gen-test-1-gpu-amd-rocm720 (linux-mi325-1gpu-sglang, 1) — failed step(s): Run diffusion server tests (1-GPU)

Commit Info

• sglang: 0bdd4730
• aiter: 9127c94a (v0.1.17.dev110+g9127c94a1, source: Dockerfile default — AITER_COMMIT_OVERRIDE empty, image version matches Dockerfile)

Failed Tests

Test File	Test Function	Error	Log
python/sglang/multimodal_gen/test/server/test_server_1_gpu.py	TestDiffusionServerOneGpu::test_diffusion_generation[ltx_2_3_hq_pipeline]	Step timeout — The action 'Run diffusion server tests (1-GPU)' has timed out after 60 minutes (killed mid-run while downloading Lightricks/LTX-2.3 from HF Hub)	Log
python/sglang/multimodal_gen/test/server/test_server_1_gpu.py	TestDiffusionServerOneGpu::test_diffusion_generation[cosmos3_nano_t2v]	SKIPPED — server failed to start: AttributeError: module diffusers has no attribute Cosmos3OmniTransformer (native fallback after NotImplementedError: AITer backend does not support Grouped Query Attention yet)	Log

This job (partition 1 of 4) is NOT a pytest assertion failure — it is a GitHub Actions step timeout. 12 of the 14 testcases in this partition passed; cosmos3_nano_t2v was skipped; ltx_2_3_hq_pipeline (the last case in the partition) never finished because the 60-minute step budget expired while it was still downloading the LTX-2.3 model.

Failure Summary

The Run diffusion server tests (1-GPU) step (timeout-minutes: 60, .github/workflows/pr-test-amd-rocm720.yml:728) ran 14 diffusion server testcases sequentially on partition 1. Wall clock from the first testcase (qwen_image_t2i, 18:17:32Z) to the timeout (19:16:47Z) was ~59 minutes. The final testcase ltx_2_3_hq_pipeline began at 19:13:33Z and the step was killed at 19:16:47Z while Lightricks/LTX-2.3 was still downloading from HF Hub ("Model not found in cache, will download"; overlay fetch stuck at 2/40 files). The cumulative runtime of the sequential server-launch cases exhausted the 60-minute budget.

Regression Status

Known recurring failure (timeout at ltx_2_3_hq_pipeline on partition 1). The ,1 shard of multimodal-gen-test-1-gpu-amd-rocm720 has concluded failure in every completed scheduled run in the queryable window. The identical timeout-at-LTX pattern is verified in the two runs whose logs are still retained; older shard-1 logs have expired (BlobNotFound), so those are shown at job-level only.

Recent history of test_server_1_gpu.py (test_diffusion_generation[ltx_2_3_hq_pipeline]) in job multimodal-gen-test-1-gpu-amd-rocm720 (…, 1) (most recent comparable completed scheduled runs: branch=main, event=schedule):

Date	Run	Job (shard 1)	Test File Status	Detail
Jul 22	29944235234	job	❌ Timeout	60-min step timeout during ltx_2_3_hq_pipeline download (verified)
Jul 21	29855003895	job	❌ Failed	log expired (BlobNotFound) — testcase unverified
Jul 20	29766743590	job	❌ Timeout	60-min step timeout during ltx_2_3_hq_pipeline download (verified)
Jul 19	29697501385	job	❌ Failed	log expired (BlobNotFound) — testcase unverified
Jul 18	29654571916	job	❌ Failed	not downloaded (older)
Jul 17	29601526478	job	❌ Failed	not downloaded (older)
Jul 16	29521557976	job	❌ Failed	not downloaded (older)

No passing shard-1 run found in the query window (Jul 16–22); other partitions (,2) frequently pass, so this is partition-1-specific. First verified LTX timeout: Jul 20; no green shard-1 run observed.

Failure Cluster

60-minute step timeout on diffusion 1-GPU partition 1 (uncached LTX-2.3 download tips cumulative suite runtime over budget)

Facts (verified observations)

• Step Run diffusion server tests (1-GPU) has timeout-minutes: 60 (.github/workflows/pr-test-amd-rocm720.yml:728); the runner emitted ##[error]The action 'Run diffusion server tests (1-GPU)' has timed out after 60 minutes at 19:16:47Z (job log line 8387).
• Partitioning is deterministic: the step runs run_suite.py --suite 1-gpu --partition-id 1 --total-partitions 4 -k "not flux_2" (.github/workflows/pr-test-amd-rocm720.yml:750-755), so partition 1 always executes the same testcase set ending in ltx_2_3_hq_pipeline.
• Testcase outcomes in this run: 12 PASSED (qwen_image_t2i, qwen_image_t2i_cache_dit_enabled, qwen_image_t2i_cache_dit_scm_config_diffusers_1gpu, pi05_action_http, flux_image_t2i, zimage_image_t2i_fp8, zimage_image_t2i_multi_lora, qwen_image_edit_2509_ti2i, qwen_image_edit_2511_ti2i, joyai_image_edit_ti2i, longlive2_t2v, wan2_1_t2v_1.3b_teacache_enabled, wan2_1_t2v_1.3b_frame_interp_2x, wan2_1_t2v_1.3b_frame_interp_2x_upscaling_4x), 1 SKIPPED (cosmos3_nano_t2v), 1 incomplete/killed (ltx_2_3_hq_pipeline). (Note: 14 passed labels — the pass set includes all cases before LTX.)
• Lightricks/LTX-2.3 was a cold download: "Model not found in cache, will download from HF Hub" (line 8374); overlay fetch was at 2/40 files when killed. LTX-2.3 is not in the workflow's pre-cache check list (only Wan2.1, HunyuanVideo, Qwen-Image, FLUX.1, FLUX.2 — .github/workflows/pr-test-amd-rocm720.yml:714).
• Same 60-min timeout at ltx_2_3_hq_pipeline reproduced on Jul 20 (BEGIN ...ltx_2_3_hq_pipeline 02:46:46Z → timeout 02:52:49Z, job 88434814040).
• cosmos3_nano_t2v sub-failure (unrelated to the timeout): AITer backend rejects GQA (aiter.py:128), native fallback then fails because installed diffusers lacks Cosmos3OmniTransformer (log lines 7046, 7118).

Hypothesised Causes (with confidence)

• [HIGH] The suite's cumulative sequential runtime exceeds the 60-minute step budget, and the cold, un-pre-cached Lightricks/LTX-2.3 download is the tail case that pushes it over.
  • Supporting: measured wall clock 18:17:32→19:16:47 ≈ 59 min for 14 sequential server-launch cases; timeout fires mid-download of LTX-2.3 (2/40 overlay files); LTX-2.3 absent from the pre-cache list so it re-downloads each run; identical timeout at the same terminal testcase on Jul 20.
  • Disconfirming: (none found) — no earlier testcase failed; all pre-LTX cases passed within budget.
  • How to verify: pre-seed Lightricks/LTX-2.3 into /sgl-data/hf-cache (add to the pre-cache check at line 714) OR raise timeout-minutes OR move ltx_2_3_hq_pipeline to a lighter partition; re-run shard 1 and confirm it completes.
• [LOW] Slow HF Hub / hf_transfer throughput on the runner extended the LTX download.
  • Supporting: overlay fetch progressed only 2/40 files in ~3 min before the kill; an earlier snapshot fetch took 2m21s for 3 files (line 8378).
  • Disconfirming: even a fast download would still be additive to an already ~56-min-full budget, so throughput alone is not the primary driver.
  • How to verify: compare LTX-2.3 download duration when the model is cache-warm vs cold.
• [LOW] cosmos3_nano_t2v skip/error is a separate, pre-existing issue (AITer GQA unsupported + diffusers missing Cosmos3OmniTransformer), not the cause of the timeout.
  • Supporting: it resolved to SKIPPED and the suite continued past it.
  • Disconfirming: it consumed ~90s and does not explain the terminal LTX timeout.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for diffusion+timeout, LTX-2.3+cache, multimodal-gen+partition; no PR addresses the partition-1 60-min timeout or LTX-2.3 pre-caching. (GET /search/issues?q=repo:sgl-project/sglang+is:pr+is:open+<keyword>)

Suggested Triage Steps

• Confirm the budget hypothesis: re-run shard 1 with Lightricks/LTX-2.3 pre-seeded into /sgl-data/hf-cache/hub and check whether the partition finishes under 60 min.
• Consider rebalancing partitions (--total-partitions) or moving heavy video/LTX cases off partition 1, so no single partition's sequential runtime approaches the step budget.
• Add Lightricks/LTX-2.3 to the pre-cache check list (.github/workflows/pr-test-amd-rocm720.yml:714) so the model is warm before the timed step, or raise timeout-minutes for this step if a cold download is expected.
• Track cosmos3_nano_t2v separately: AITer backend lacks GQA support (aiter.py:128) and the native diffusers fallback lacks Cosmos3OmniTransformer — investigate/disable independently of the timeout.

Status

• Persistence: Partition-1 shard failing in every completed scheduled run Jul 16–22 (7/7); the LTX timeout signature is verified on Jul 20 and Jul 22 (older shard-1 logs expired). No green shard-1 run in the window.
• Scope: multimodal-gen-test-1-gpu-amd-rocm720 partition ,1 only; sibling partition ,2 frequently passes. Same testcase set/partitioning also runs on pr-test-amd.yml, so LTX may exhibit the same risk there.
• Blocked work: Blocks the 1-GPU diffusion server CI signal for partition 1 on ROCm 7.2; ltx_2_3_hq_pipeline is never actually exercised to completion.
• In-flight fix: ❌ none found — needs new PR or investigation.

multimodal-gen-test-1-gpu-amd-rocm720 (linux-mi325-1gpu-sglang, 0) — failed step(s): Run diffusion server tests (1-GPU)

Commit Info

• sglang: 0bdd4730
• aiter: 9127c94a (v0.1.17.dev110+g9127c94a1, source: dev/patched CI image — preserved, no rebuild)

Failed Tests

Test File	Test Function	Error	Log
sglang/multimodal_gen/test/server/test_server_1_gpu.py	TestDiffusionServerOneGpu::test_diffusion_generation[longlive2_i2v]	Step wall-clock timeout — The action 'Run diffusion server tests (1-GPU)' has timed out after 60 minutes (killed mid model-download)	Log

This is a job-step timeout, not a test assertion failure. No individual testcase failed an assertion.

Failure Summary

The Run diffusion server tests (1-GPU) step on shard 0 hit the GitHub Actions 60-minute step timeout. Eleven parametrized diffusion testcases ran and all PASSED (zimage_image_t2i, cosmos3_nano_t2i, qwen_image_edit_ti2i, qwen_image_layered_i2i, wan2_1_t2v_1.3b, wan2_1_t2v_1.3b_upscaling_4x, wan2_1_t2v_1_3b_lora_1gpu, fast_hunyuan_video, wan2_2_ti2v_5b, fastwan2_2_ti2v_5b, sana_wm_ti2v). The 12th and final case longlive2_i2v began at 19:15:51, started downloading Rabinovich/LongLive-2.0-5B-Diffusers (17 files), and the 60-min limit fired at 19:16:14 before it could finish. The shard's cumulative work simply exceeds the 60-minute wall clock.

Regression Status

Known recurring failure (persistent / shard-0 never-passed in window).

Recent history of test_server_1_gpu.py (shard 0 — linux-mi325-1gpu-sglang, 0), same branch=main, event=schedule, completed runs only:

Date	Run	Job	Test File Status	Signature
Jul 22	29944235234	shard 0	❌ Timeout	60-min timeout at longlive2_i2v download
Jul 21	29855003895	shard 0	❌ Timeout	60-min timeout at longlive2_i2v
Jul 20	29766743590	shard 0	❌ Timeout	60-min timeout after sana_wm_ti2v
Jul 19	29697501385	shard 0	❌ Timeout	60-min timeout at longlive2_i2v
Jul 18	29654571916	shard 0	❌ Failure	(shard 0 failed)
Jul 17	29601526478	shard 0	❌ Timeout	60-min timeout at longlive2_i2v
Jul 16	29521557976	shard 0	❌ Failure	(shard 0 failed)
… Jul 08	28964460204	shard 0	❌ Failure	(shard 0 failed)

Shard 0 failed in every completed scheduled run in the 15-run window (Jul 08 – Jul 22); no green shard-0 run found → Never-passed in the queryable window. Note shard 2 passes consistently; shards 0/1/3 fail — the failure is specific to how the testcase matrix is partitioned across shards, not a single broken test.

Failure Cluster

Diffusion 1-GPU server-test shard exceeds 60-minute step timeout (shard-0 wall-clock, aggravated by repeated HF model re-downloads).

Facts (verified observations)

• ##[error]The action 'Run diffusion server tests (1-GPU)' has timed out after 60 minutes at 19:16:14 (log line 7438). FACT.
• First testcase zimage_image_t2i BEGAN 18:17:07; timeout fired 19:16:14 ⇒ ~59 min of test wall-clock consumed. FACT.
• All 11 completed testcases printed [server-test] PASSED; only longlive2_i2v (last in shard) did not complete. FACT.
• Every testcase logs Model found in cache but incomplete, will download from HF Hub and re-fetches from HF Hub (e.g. sana_wm_ti2v fetch took ~2m33s; qwen_image_edit_ti2i ~1m55s), so a large fraction of wall-clock is model download, not compute. FACT.
• Timeout is the identical signature across historical shard-0 runs (Jul 17/19/21/22 all die at longlive2_i2v; Jul 20 dies just after sana_wm_ti2v). FACT.
• aiter unchanged (dev image v0.1.17.dev110+g9127c94a1, no override). FACT.

Hypothesised Causes (with confidence)

• [MEDIUM] Shard-0's assigned set of diffusion testcases takes longer than the 60-minute step budget; the suite is timing out, not failing functionally.
  • Supporting: 11 testcases pass then the step is killed mid-12th; ~59 min elapsed; the last case (longlive2_i2v, a 5B video model) is consistently the one that never starts/finishes; shard 2 (fewer/lighter cases) always passes.
  • Disconfirming: on Jul 20 the timeout fired right after sana_wm_ti2v (one case earlier), so exact per-run timing varies — total slack is thin and download-speed–dependent, meaning it's a borderline budget problem rather than one specific hanging case.
  • How to verify: sum per-testcase durations for shard 0 from this log; check whether re-balancing the shard matrix or raising the step timeout lets it complete.
• [MEDIUM] Repeated HF Hub model re-downloads (cache … incomplete, will download) inflate wall-clock and push the shard over budget.
  • Supporting: every testcase re-downloads its model snapshot; several downloads take 1–2.5 min each; cache is reported "incomplete" each time despite the runner presumably having prior copies.
  • Disconfirming: even with zero download time the compute for 12 video-diffusion models on one MI325 may still be tight — download reduction alone may not guarantee completion.
  • How to verify: check runner HF cache persistence / pre-seeding; measure suite time with a warm cache.
• [LOW] A specific hang in longlive2_i2v server startup/generation.
  • Supporting: it is the case in progress at timeout in most runs.
  • Disconfirming: it is always killed during model download at 0%, before the server even loads — no evidence of a compute hang; and Jul 20 timed out before longlive2_i2v even began. Strongly weakens a case-specific-hang theory.
  • How to verify: run longlive2_i2v in isolation on MI325 and time it end-to-end.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for multimodal-gen+timeout, diffusion+server+test+timeout, longlive2 (0 results), multimodal-gen-test. No open PR addresses the shard-0 60-minute diffusion server-test timeout. (Related open PRs exist for multimodal-gen deps/unit tests, e.g. #31899, #31483, but none target this timeout.)

Suggested Triage Steps

• Sum the per-testcase durations for shard 0 from this log to confirm the suite is over the 60-min budget, then rebalance the diffusion testcase matrix so shard 0 carries fewer/lighter cases (currently shards 0/1/3 fail, shard 2 passes).
• Investigate why the runner HF cache is reported incomplete every testcase — pre-seed or persist the HF Hub cache on MI325 runners to cut the repeated multi-minute downloads.
• If the workload is genuinely at the budget, consider raising the step timeout-minutes for the 1-GPU diffusion server tests, or splitting longlive2_i2v (5B) into its own shard.
• Run longlive2_i2v standalone on MI325 to get an isolated end-to-end time and rule out a true hang before assuming it's purely a budget issue.

Status

• Persistence: Failing in every completed scheduled run in the queryable window (Jul 08 – Jul 22, ≥15 days); no green shard-0 run found.
• Scope: multimodal-gen-test-1-gpu-amd-rocm720 shards 0, 1, 3 (shard 2 passes) on pr-test-amd-rocm720.yml; same 60-min-timeout signature.
• Blocked work: Blocks the 1-GPU diffusion server-test signal on AMD ROCm 7.2 for the affected shards.
• In-flight fix: ❌ none found in open PRs.

multimodal-gen-test-2-gpu-amd-rocm720 (linux-mi325-2gpu-sglang, 0) — failed step(s): N/A

Commit Info

• sglang: 0bdd4730
• aiter: 9127c94a (v0.1.17.dev110+g9127c94a1, source: dev image — Dev/patched version detected). Note: read from a same-run sibling shard (...2gpu-sglang, 2), because this job's own log was never uploaded (see below).

Failed Tests

Test File	Test Function	Error	Log
python/sglang/multimodal_gen/test/server/test_server_2_gpu.py	TestDiffusionServerTwoGpu::test_diffusion_generation[...] (exact case unknowable for this run — no log)	Step hung ~1h47m, hard-killed; no test log uploaded (BlobNotFound)	Job

This is not a clean test failure with a recoverable log — it is a hang/hard-kill. Step 9 Run diffusion server tests (2-GPU) is stuck in_progress and step 10 (artifact upload) never ran, so no diffusion-failures artifact and no job log blob exist for this run.

Failure Summary

Job shard 0 of multimodal-gen-test-2-gpu-amd-rocm720 ran the parametrized test_server_2_gpu.py::TestDiffusionServerTwoGpu::test_diffusion_generation[...] cases (siblings shard 1 = test_disagg_server.py, shard 2 = test_ar_models.py, both passed). Step 9 ran from 20:30:01→22:17:32 UTC (~1h47m) and was terminated before the 150-min step timeout, with the step still in_progress — the runner/container hung and no log or artifact was uploaded. The exact stuck parametrized case cannot be pinned from this run's evidence.

Regression Status

Never-passed (no green run found in window) — persistent daily failure.

Recent history of test_server_2_gpu.py (TestDiffusionServerTwoGpu::test_diffusion_generation) in job multimodal-gen-test-2-gpu-amd-rocm720 (linux-mi325-2gpu-sglang, 0) (15 most recent comparable completed scheduled runs: branch=main, event=schedule, completed only):

Date	Run	Test File Status	Failure mode	Notes
Jul 22	29944235234	❌ Failed	Hang (step9 in_progress, 1:47:31)	TARGET — no log
Jul 21	29855003895	❌ Failed	Hang (1:49:06)	no log
Jul 20	29766743590	❌ Failed	Hang (1:42:07)	no log
Jul 19	29697501385	❌ Failed	Hang (1:41:05)	no log
Jul 18	29654571916	❌ Failed	Early fail (0:10:00, step9 not reached)	infra/setup
Jul 17	29601526478	❌ Failed	Hang (1:26:22)	no log
Jul 16	29521557976	❌ Failed	Test failure (1:57:29)	HIPBLAS_STATUS_NOT_SUPPORTED
Jul 15	29438298954	❌ Failed	Test failure (1:56:07)	—
Jul 14	29355401408	❌ Failed	Test failure (1:30:06)	—
Jul 13	29273187026	❌ Failed	Test failure (1:59:46)	—
Jul 12	29202664092	❌ Failed	Test failure (2:09:02)	—
Jul 11	29162225443	❌ Failed	Test failure (2:00:15)	—
Jul 10	29113205616	❌ Failed	Test failure (1:58:00)	—
Jul 9	29039527644	❌ Failed	Test failure (1:55:44)	—
Jul 8	28964460204	❌ Failed	Test failure (1:03:00)	—

No passing run for this test file in the queried window (Jul 8–22, 15 runs). First-observed failure predates the window. Notable mode shift: through Jul 16 the step completed with a clean failure (logs exist); from Jul 17 onward the mode is a hang (step never returns, no logs) — Jul 18 is a distinct 10-min early failure.

Failure Cluster

2-GPU diffusion server test hang / hard-kill (no log) — with a prior mode of FP8 GEMM HIPBLAS_STATUS_NOT_SUPPORTED on FLUX.2 modelopt-FP8.

Facts (verified observations)

• This run has no recoverable log. Direct jobs/89005546294/logs returns BlobNotFound; the job is absent from the run-level log zip (siblings shard 1 & 2 are present). Step 9 status in_progress, step 10 pending (job API). [FACT]
• Job wall time 20:30:01→22:17:32 UTC = 1:47:31; step 9 timeout-minutes: 150 was not reached (.github/workflows/pr-test-amd-rocm720.yml:867). [FACT]
• Shard 0 runs the parametrized test_server_2_gpu.py::TestDiffusionServerTwoGpu::test_diffusion_generation[...] cases; sibling shard 1 = test_disagg_server.py (PASSED), shard 2 = test_ar_models.py (PASSED). Partition layout: gpu_cases.py:980 (PARAMETRIZED) + :994 (STANDALONE). [FACT]
• This shard 0 job failed in all 15 most-recent completed scheduled runs (Jul 8–22). [FACT]
• In the last run with a usable log (Jul 16, 87699749934): case test_diffusion_generation[flux2_modelopt_fp8_tp2_t2i] FAILED with RuntimeError: CUDA error: HIPBLAS_STATUS_NOT_SUPPORTED when calling HIPBLAS_STATUS_NOT_SUPPORTED (log line 4353 onward; Detected ModelOpt FP8 checkpoint), and all remaining parametrized cases ERRORed afterward (cascade). [FACT]
• aiter unchanged across the window in these runs (dev image v0.1.17.dev110+g9127c94a1, AITER_COMMIT_OVERRIDE empty). [FACT]

Hypothesised Causes (with confidence)

• [LOW] The Jul 22 hang is the same FP8-GEMM problem, now manifesting as a hang instead of a clean 500.
  • Supporting: same job, same shard, same test file has been red continuously; the FP8 case is the heaviest/first parametrized case; a server stuck retrying a failing FP8 GEMM (7-attempt retry harness) plus 5 min HF/model startup per case can plausibly consume ~1h47m.
  • Disconfirming: there is no log for this run, so the FP8 signature is not observed here — this is inference from the Jul 16 log only. The mode change (clean failure → hang) starting Jul 17 could be a different root cause (e.g. a hang in a specific case, driver/GPU wedge, or model-download stall), not the FP8 GEMM.
  • How to verify: re-run shard 0 with py-spy dump/SGLANG_... verbose and per-case timeouts; or run test_server_2_gpu.py -k flux2_modelopt_fp8_tp2_t2i locally on MI325/ROCm 7.2.
• [LOW] Runner/container-level hang (infra), not a specific test.
  • Supporting: step never returns and no log is uploaded — consistent with the runner losing heartbeat / GPU wedge; branch amd/ci-gpu-hang-probe exists upstream, suggesting a known hang investigation.
  • Disconfirming: siblings on the same runner class completed fine in the same run; the hang recurs on the same shard daily, which points at shard-0 content rather than random infra.
  • How to verify: check runner host logs / dmesg/rocm-smi at ~22:17 UTC; add a per-case watchdog to identify the stuck case.
• No hypothesis above LOW for THIS run — root cause needs a run that actually uploads a log or a watchdog dump. The FP8 HIPBLAS_STATUS_NOT_SUPPORTED cause is [MEDIUM] only for the pre-Jul-17 clean-failure runs.

In-flight Fix Check

• FP8 GEMM failure: In-flight fix: ✅ #28889 "[AMD] Use native e4m3fnuz fp8 GEMM for FLUX.2 modelopt-FP8 on gfx942" (open, non-draft, created 2026-06-22, last updated 2026-07-21) — directly targets the flux2_modelopt_fp8_tp2_t2i / HIPBLAS_STATUS_NOT_SUPPORTED signature. Chase reviewers; do not duplicate. (Searched search/issues?q=...HIPBLAS_STATUS_NOT_SUPPORTED.)
• Hang mode (Jul 17+): In-flight fix: ❌ none found matching the 2-GPU diffusion hang — needs new investigation.

Suggested Triage Steps

• Add a per-case watchdog / py-spy dump to Run diffusion server tests (2-GPU) so the next hang identifies the stuck parametrized case instead of hard-killing with no log — this is the blocking gap (no log ⇒ no test-level root cause for Jul 17+).
• Verify #28889 resolves the FP8 case, then re-run shard 0 and confirm whether the hang disappears (tests whether the hang is the FP8 case looping in the retry harness).
• Bisect the mode shift between Jul 16 87699749934 (clean failure) and Jul 17 87954595311 (first hang): git log --oneline <Jul16_sha>..<Jul17_sha> grepping for diffusion/2-gpu/FP8/retry-harness changes.
• If the hang is confirmed to be the FP8 case, consider quarantining flux2_modelopt_fp8_tp2_t2i on AMD MI325/ROCm 7.2 until #28889 lands, so shard 0 gives usable signal for the other cases.

Status

• Persistence: This shard 0 job has failed in all 15 most-recent completed scheduled runs (Jul 8–22, 2026); never green in the queryable window. Mode shifted from clean test-failure to hang on ~Jul 17.
• Scope: multimodal-gen-test-2-gpu-amd-rocm720 (…, 0) only within this run's 2-GPU matrix (shards 1 & 2 pass). The FP8 signature is shard-0-specific.
• Signal gap: For Jul 17+ runs (including this one) no job log/artifact is uploaded, so test-level attribution is blocked until a watchdog/timeout is added.
• In-flight fix: ✅ #28889 for the FP8 GEMM cause; ❌ none for the hang mode.

multimodal-gen-unit-test-amd-rocm720 — failed step(s): Run diffusion unit tests

Commit Info

• sglang: 0bdd4730
• aiter: 9127c94a (v0.1.17.dev110+g9127c94a1; source: dev image preserved — [CI-AITER-CHECK] Dev/patched version detected → skipping rebuild)

Failed Tests

Test File	Test Function	Error	Log
python/sglang/multimodal_gen/test/unit/test_pi05_action_api.py	test_msgpack_roundtrip_preserves_string_keys_and_numpy_payloads	ModuleNotFoundError: No module named 'msgpack'	Log

Result: 1 failed, 793 passed, 1 skipped, 5 deselected, 50 subtests passed in 149.84s.

Failure Summary

The multimodal-gen-unit-test-amd-rocm720 job's step 7 ("Run diffusion unit tests") had exactly one test failure. test_msgpack_roundtrip_preserves_string_keys_and_numpy_payloads calls pack_msgpack() in python/sglang/multimodal_gen/runtime/entrypoints/vla/protocol.py:59, which does import msgpack; the module is not present in the AMD ROCm CI image, raising ModuleNotFoundError. All 793 other tests passed.

Regression Status

Known recurring failure — persistent since 2026-07-11, every completed scheduled run since. Introduced by the Pi0.5 test addition (the test file did not exist before then).

Recent history of test_pi05_action_api.py (test_msgpack_roundtrip_preserves_string_keys_and_numpy_payloads) in job multimodal-gen-unit-test-amd-rocm720 (branch main, event=schedule, completed runs):

Date	Run	Test File Status	Failed Function	Error
Jul 22	29944235234	❌ Failed	test_msgpack_roundtrip...	ModuleNotFoundError: msgpack
Jul 21	29855003895	❌ Failed	test_msgpack_roundtrip...	ModuleNotFoundError: msgpack
Jul 11	29162225443	❌ Failed	test_msgpack_roundtrip...	ModuleNotFoundError: msgpack
Jul 10	29113205616	✅ Passed	—	— (test did not yet exist)

First observed failure: 2026-07-11. Last known passing: 2026-07-10 (before the test was introduced). Verified at the test-file level by downloading each job log and confirming the identical msgpack failure signature.

Failure Cluster

ModuleNotFoundError: msgpack missing on AMD ROCm image

Facts (verified observations)

• Verbatim error: sglang/multimodal_gen/runtime/entrypoints/vla/protocol.py:59: in pack_msgpack → import msgpack → E ModuleNotFoundError: No module named 'msgpack' (job log lines 5814-5816).
• pack_msgpack / unpack_msgpack in protocol.py both do a lazy import msgpack (verified by reading the source).
• The failing test and protocol.py were both added by commit 649ce5dd "model: support Pi0.5" (#30633), authored 2026-07-10T23:50Z. The test file has no earlier commits.
• msgpack is declared only in the diffusion optional-dependencies extra of python/pyproject.toml:110 (the NVIDIA/default file).
• The AMD install script swaps in pyproject_other.toml before installing: mv python/pyproject_other.toml python/pyproject.toml then pip install -e "python[${EXTRAS}]" (scripts/ci/amd/amd_ci_install_dependency.sh:132-134). EXTRAS="dev_hip,tracing" (line 29).
• pyproject_other.toml contains no msgpack entry anywhere (grep returned nothing), including in its diffusion_common / diffusion_hip extras. So msgpack is never installed on the AMD ROCm image.
• Reproducibility: same failure in every scheduled run Jul 11 → Jul 22 (1 test failed, ~793 passed each run). NVIDIA is unaffected because its pyproject.toml diffusion extra carries msgpack.

Hypothesised Causes

• [HIGH] The Pi0.5 test (#30633, 649ce5dd) added a msgpack-dependent unit test, but msgpack was never added to pyproject_other.toml (the AMD/ROCm dependency manifest), so it is not installed on the AMD image.
  • Supporting: error is a plain ModuleNotFoundError at the import msgpack line; msgpack present in pyproject.toml (NVIDIA) but absent from pyproject_other.toml (AMD); failure begins exactly at the run following the commit that added the test; Jul 10 run (before the test existed) passed; identical signature every run since.
  • Disconfirming: (none found) — the passing→failing boundary aligns precisely with the test's introduction and the dependency gap is directly observable in the two manifests.
  • How to verify: docker exec ci_sglang python -c "import msgpack" in the ROCm image → expect ModuleNotFoundError; or add msgpack to pyproject_other.toml diffusion deps and re-run the job.

In-flight Fix Check

In-flight fix: ✅ #31899 "[AMD] Add msgpack to ROCm diffusion deps (fix multimodal-gen unit test ModuleNotFoundError)" — open since 2026-07-21. This directly targets the root cause; ping reviewers, do NOT open a duplicate. (Search: repo:sgl-project/sglang is:pr is:open msgpack.) Related AMD CI PRs #31844 and #31483 also touch msgpack/nested-unit-test gating on AMD.

Suggested Triage Steps

• Verify #31899 adds msgpack to the AMD diffusion extra in pyproject_other.toml, merges cleanly, and re-run multimodal-gen-unit-test-amd-rocm720.
• As an alternative if the dep is not wanted on AMD: skip/guard the test behind an availability check (pytest.importorskip("msgpack")) — consistent with the LAPACK/msgpack env skips approach in #31844.
• Confirm the fix also covers the sister pr-test-amd.yml (mi30x/mi325) multimodal-gen job, which runs the same suite from the same manifest.

Status

• Persistence: Failing for 12 days (since 2026-07-11), every completed scheduled run since.
• Scope: 1 test in multimodal-gen-unit-test-amd-rocm720; the same manifest gap likely affects the equivalent job on pr-test-amd.yml. NVIDIA unaffected.
• Blocked work: Keeps the AMD ROCm 7.2 multimodal-gen unit suite red (1/794 tests); low blast radius but blocks a clean signal.
• In-flight fix: ✅ #31899 (open since 2026-07-21) — awaiting merge.

stage-c-test-large-8-gpu-amd-rocm720 (linux-mi325-8gpu-sglang, 3) — failed step(s): Run test

Commit Info

• sglang: 0bdd4730
• aiter: 9127c94a — v0.1.17.dev110+g9127c94a1 (source: dev/patched image, preserved; AITER_COMMIT_OVERRIDE empty)

Failed Tests

Test File	Test Function	Error	Log
test/registered/ops/test_aiter_allreduce_fusion_amd.py	TestAiterAllreduceFusionGate.test_dense_tp_fuses	AttributeError: module 'sglang.srt.layers.communicator' does not have the attribute 'get_server_args'	Log

Other files in this shard passed: test_deepseek_v3_mtp.py, test_deepseek_v3_basic.py, test_aiter_allgather_amd.py. Shard result: 3/4 files passed; the aiter-allreduce-fusion file reported Ran 4 tests … FAILED (errors=1).

Failure Summary

The gate unit test test_dense_tp_fuses (in TestAiterAllreduceFusionGate) errors during setup of its mock context: it calls mock.patch.object(comm, "get_server_args", …) at test/registered/ops/test_aiter_allreduce_fusion_amd.py:420, but sglang.srt.layers.communicator no longer exposes a get_server_args symbol, so mock raises AttributeError. The test's built-in retry then exhausts and the file exits non-zero, failing the job (exit 255).

Regression Status

New regression.

Recent history of test/registered/ops/test_aiter_allreduce_fusion_amd.py (test_dense_tp_fuses) in job stage-c-test-large-8-gpu-amd-rocm720 (linux-mi325-8gpu-sglang, 3) (scheduled, branch main, completed runs):

Date	Run	Job	Test File Status	Failed Function	Error
Jul 22	29944235234	shard 3	❌ Failed	test_dense_tp_fuses	AttributeError: …get_server_args
Jul 21	29855003895	shard 3	✅ Passed	—	—
Jul 20	29766743590	shard 3	✅ Passed	—	—

The job (which runs this test file) succeeded on Jul 20 and Jul 21 — the run-level "failure" conclusion on those days came from other shards/jobs, not this one. First observed failure: 2026-07-22. Last known passing: 2026-07-21 run (head 9a6c9608).

Failure Cluster

AttributeError: communicator module missing get_server_args (stale test mock target)

Facts (verified observations)

• Verbatim error (log line 15636): AttributeError: <module 'sglang.srt.layers.communicator' from '/sglang-checkout/python/sglang/srt/layers/communicator.py'> does not have the attribute 'get_server_args'.
• Failing frame: _evaluate_gate → stack.enter_context(mock.patch.object(comm, "get_server_args", lambda: server_args)) at test_aiter_allreduce_fusion_amd.py:420 (log lines 15626–15636).
• Current python/sglang/srt/layers/communicator.py contains zero references to get_server_args (grep returned nothing at HEAD 0bdd4730).
• Commit 745b2ca4 ("config: read parallel config leaves via get_parallel()", #31816) removed the get_server_args import from communicator.py and replaced all four usages with get_parallel().
• Merge window: 745b2ca4 is NOT an ancestor of the last-pass head 9a6c9608 but IS an ancestor of the failing head 0bdd4730 — verified via git merge-base --is-ancestor. Commit landed 2026-07-22 01:18 PDT, between the Jul-21 and Jul-22 scheduled runs.
• The test already patches get_parallel separately (line 415), so it was partially updated for the config refactor but the get_server_args patch target was left stale.
• aiter version is unchanged (dev image preserved, no override); NVIDIA/other backends unaffected by this Python-symbol issue.

Hypothesised Causes (with confidence)

• [HIGH] Commit 745b2ca4 (#31816) removed get_server_args from sglang.srt.layers.communicator, but the test at test_aiter_allreduce_fusion_amd.py:420 still mock.patch.objects that removed symbol. mock.patch.object requires the attribute to exist, hence the AttributeError.
  • Supporting: three independent chains converge — (a) the diff of 745b2ca4 deletes the get_server_args import and its usages (log/git confirmed); (b) HEAD communicator.py has no get_server_args; (c) the commit lands exactly in the 9a6c9608..0bdd4730 regression window, and the file passed in the immediately prior scheduled run.
  • Disconfirming: (none found). The only nuance is that this run reported errors=1 of Ran 4 tests while _evaluate_gate is shared by several gate tests — the other executed gate tests should hit the same line; this doesn't weaken the cause, only suggests the CI method selector ran a subset.
  • How to verify: on a branch, either re-add a thin get_server_args shim to communicator.py, OR update the test to patch get_parallel/get_server_args at their real new location, then re-run python3 test/registered/ops/test_aiter_allreduce_fusion_amd.py on AMD MI325.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for get_server_args in:title (0), aiter_allreduce_fusion (0), and communicator get_parallel (1 result, #27139 "[FEAT] Fast Recovery" — unrelated). No open PR addresses this stale mock target.

Suggested Triage Steps

• Confirm the culprit: git show 745b2ca4 -- python/sglang/srt/layers/communicator.py and note the removed get_server_args import; grep the test for its patch targets.
• Fix direction (test-side): update test_aiter_allreduce_fusion_amd.py:420 to stop patching the removed comm.get_server_args — the config fields it fed (e.g. moe_dense_tp_size, dwdp_size, enable_attn_tp_input_scattered) are now read via get_parallel(), which the test already patches at line 415; fold the needed fields into that SimpleNamespace.
• Alternatively (src-side), if get_server_args should remain part of the communicator public surface for mocking, re-export it; but prefer the test-side fix to match the #31816 refactor intent.
• After patching, run python3 test/registered/ops/test_aiter_allreduce_fusion_amd.py in the ROCm 7.2 CI container on an 8-GPU MI325 runner and confirm all gate tests pass.

Status

• Persistence: First failing 2026-07-22 (this run); passed in the 2026-07-21 and 2026-07-20 scheduled runs for this test file. 1 failing run so far.
• Scope: test_dense_tp_fuses in test_aiter_allreduce_fusion_amd.py. Root cause is a repo-wide symbol removal, so any other CI job/shard that runs this file (e.g. the CUDA/other-arch equivalents and pr-test-amd) is expected to hit the same AttributeError.
• Blocked work: aiter all-reduce fusion gate unit coverage on AMD ROCm 7.2.
• In-flight fix: ❌ none found — needs a new PR (test-side update to #31816's refactor).

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-07-23 23:34 UTC)

[amd-bot]

pr-test-amd-rocm720.yml — Per-Job Analysis (continued 3/4)

multimodal-gen-test-1-gpu-amd-rocm720 (linux-mi325-1gpu-sglang, 3) — failed step(s): N/A

Commit Info

• sglang: 0bdd4730
• aiter: N/A — the job log blob is unavailable (see below), so [CI-AITER-CHECK] markers could not be read.

Failed Tests

Test File	Test Function	Error	Log
python/sglang/multimodal_gen/test/single_test_file/test_update_weights_from_disk.py	(unrecoverable — job killed mid-step)	Step 9 Run diffusion server tests (1-GPU) ended with conclusion None (killed), not a clean pass/fail	Job

This is not a normal test-assertion failure — the shard was killed during execution and its log was never finalized. Partition 3 of the 1-gpu suite deterministically runs exactly one standalone file: with 4 partitions and 2 standalone files (STANDALONE_FILES["1-gpu"]), parametrized_partitions = 2, so part 3 → standalone_idx 1 → test_update_weights_from_disk.py (est. 1200 s). In CI this test is pinned to SGLANG_MMGEN_UPDATE_WEIGHTS_PAIR=FLUX.2-klein-base-4B (gpu_cases.py:1047).

Failure Summary

The multimodal-gen-test-1-gpu-amd-rocm720 (…, 3) shard ran for ~33 min (18:05:39→18:38:40 UTC) and then step 9 was killed (conclusion None) well before its 60-min timeout-minutes. The GitHub job-log blob returns BlobNotFound, the run-level log archive contains only partitions 0 and 2 (not 3), and no failure artifacts were uploaded (step 10 never ran). The concrete error/stack trace is therefore unrecoverable from all available sources.

Regression Status

Known recurring failure — effectively Never-passed in queryable history. No green run for this shard exists in the queried window (checked back to 2026-06-30; last 15 completed scheduled runs all failed).

Recent history of test_update_weights_from_disk.py (partition 3) in job multimodal-gen-test-1-gpu-amd-rocm720 (linux-mi325-1gpu-sglang, 3) (most recent 15 comparable completed scheduled runs: branch=main, event=schedule, completed only):

Date	Run	Test File Status	step9 conclusion	Duration
Jul 22	29944235234	❌ Failed (killed)	None	33 min
Jul 21	29855003895	❌ Failed (killed)	None	24 min
Jul 20	29766743590	❌ Failed (killed)	None	40 min
Jul 19	29697501385	❌ Failed (killed)	None	24 min
Jul 18	29654571916	❌ Failed (killed)	None	27 min
Jul 17	29601526478	❌ Failed (killed)	None	25 min
Jul 16	29521557976	❌ Failed (killed)	None	23 min
Jul 15	29438298954	❌ Failed	n/a	23 min
Jul 14	29355401408	❌ Failed	n/a	33 min
Jul 13	29273187026	❌ Failed	n/a	27 min
Jul 12	29202664092	❌ Failed	n/a	26 min
Jul 11	29162225443	❌ Failed	n/a	26 min
Jul 10	29113205616	❌ Failed	n/a	29 min
Jul 09	29039527644	❌ Failed	n/a	31 min
Jul 08	28964460204	❌ Failed	failure	21 min

Lookback range for the "no green run" claim: 2026-06-30 → 2026-07-22 (also spot-checked 28465906216 on 06-30 = ❌ Failed). First observed passing date: none found. Note the log blob is BlobNotFound for every one of these part-3 jobs — a consistent signature of a shard killed mid-step whose log was never uploaded.

Failure Cluster

Shard killed mid-run during test_update_weights_from_disk (log never finalized) — a symptom-level grouping (job dies at ~23-40 min < 60-min timeout, conclusion None, log/artifacts absent). This is distinct from the sibling parts 0/1 cluster in the same run, which is HF model-download 60-min step timeout.

Facts (verified observations)

• Partition math: STANDALONE_FILES["1-gpu"] = [test_generate_zimage_turbo_cli.py, test_update_weights_from_disk.py]; parametrized_partitions = 4 − 2 = 2; therefore part 3 = standalone_idx 1 = test_update_weights_from_disk.py (run_suite.py:622-711, est. 1200 s at gpu_cases.py:1008).
• Job 89005546467: steps 1-8 succeeded; step 9 Run diffusion server tests (1-GPU) conclusion None; steps 10 & 20 None; job conclusion failure; ran 18:05:39→18:38:40 UTC (~33 min); step 9 timeout-minutes: 60 was not reached.
• Log unrecoverable: GET /actions/jobs/89005546467/logs → BlobNotFound; run archive /actions/runs/29944235234/logs contains only …(…, 0) and …(…, 2) — no part 3; run has 0 artifacts (step 10 never executed).
• Sibling shards, same run: part 0 failed hitting the 60-min step timeout mid HF-download (Fetching 17 files: 0% for Rabinovich/LongLive-2.0-5B-Diffusers); part 1 failed the same way (~72 min); part 2 passed (standalone test_generate_zimage_turbo_cli.py, 15 min).
• Persistence: part 3 failed in all 15 most-recent completed scheduled runs; the last 7 (since 07-16) share the identical step9=None, killed-at-23-40 min, blob-missing signature.

Hypothesised Causes (with confidence)

• [LOW] Hard crash of the container/runner (GPU memory-access fault, segfault, or OOM-kill) during the heavy test_update_weights_from_disk.py run, which loads FLUX.2-klein-base-4B and performs a disk weight-update on a single MI325 GPU.
  • Supporting: conclusion None (not failure) + job dying well before the 60-min timeout + the log blob and artifacts never being finalized are all consistent with the runner agent's log streaming being taken down abruptly (a hang/timeout normally still finalizes the log, as seen for parts 0/1).
  • Disconfirming: no log/stack trace/exit code is available to confirm a fault vs. an infra reclaim; the "killed" vs "clean-timeout" distinction alone cannot separate a GPU crash from a runner-side kill.
  • How to verify: re-run this single shard with docker exec … python3 sglang/multimodal_gen/test/run_suite.py --suite 1-gpu --partition-id 3 --total-partitions 4 -k "not flux_2" on an MI325 rocm720 container and capture dmesg/rocm-smi/container exit code live; watch for Memory access fault / OOM-kill.
• [LOW] Self-hosted runner infra reclaim / lost connection.
  • Supporting: None conclusions + missing logs can also result from a runner disappearing.
  • Disconfirming: the failure is 100% reproducible across 15 runs on different head SHAs — an intermittent infra reclaim would not reproduce this deterministically. This weakens the pure-infra explanation and points back to something deterministic in this shard's workload.
• No hypothesis reaches MEDIUM: the log is unavailable, so root cause cannot be established from evidence. Needs a live re-run to capture the crash.

In-flight Fix Check

• In-flight fix: ❌ none found targeting the part-3 test_update_weights_from_disk.py shard crash on rocm720. Searched update_weights_from_disk (17 open PRs — all are srt/RL LLM weight-update or LoRA quant PRs, e.g. #30761, #31524; none touch multimodal_gen), and multimodal-gen rocm720 (related but different jobs: #31899 msgpack for the unit test, #31844/#31846 nested unit tests).

Suggested Triage Steps

• Recover a live log first — this is the blocker. Re-run the shard directly on an MI325 rocm720 container: docker exec -e SGLANG_MMGEN_UPDATE_WEIGHTS_PAIR=FLUX.2-klein-base-4B … python3 sglang/multimodal_gen/test/run_suite.py --suite 1-gpu --partition-id 3 --total-partitions 4 -k "not flux_2", capturing container exit code, dmesg, and rocm-smi — the CI upload path (step 10) never runs when the job is killed, so the log must be captured out-of-band.
• If a GPU memory-access fault / OOM is confirmed, investigate test_update_weights_from_disk.py VRAM footprint on a single MI325 with the FLUX.2-klein-base-4B pair (weight-update loads model twice) and whether it needs an AMD-specific skip/quarantine until fixed.
• If instead the runner is being reclaimed, escalate to CI infra — but the deterministic 15/15 reproduction argues against pure infra.
• Consider quarantining/disabling partition 3 (test_update_weights_from_disk.py) on pr-test-amd-rocm720 until root-caused, since it has never passed in the queryable window and currently only produces an unrecoverable red shard.

Status (factual)

• Persistence: Failing continuously for the entire queryable window — 15/15 most-recent completed scheduled runs (2026-07-08 → 2026-07-22) and back to at least 2026-06-30; no green run found.
• Scope: One shard (multimodal-gen-test-1-gpu-amd-rocm720, partition 3 = test_update_weights_from_disk.py) on pr-test-amd-rocm720.yml. Sibling parts 0/1 fail separately (HF-download 60-min timeout); part 2 passes.
• Evidence gap: Log blob and artifacts unrecoverable for every occurrence — root cause cannot be established without a live re-run.
• In-flight fix: ❌ none found for this specific shard failure.

stage-b-test-large-8-gpu-mi35x-disaggregation-amd-rocm720 (linux-mi35x-gpu-8.fabric) — failed step(s): Run test_disaggregation

Commit Info

• sglang: d0b96898
• aiter: 9127c94a (v0.1.17.dev110+g9127c94a1, source: Dockerfile default — dev image preserved, no override)
• Image: rocm/sgl-dev:v0.5.15.post1-rocm720-mi35x-20260723 (daily-rebuilt)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/disaggregation/test_disaggregation_pp.py	test_gsm8k (TestDisaggregationDecodePPAccuracy)	RuntimeError: Internal Server Error → Exception: retry() exceed maximum number of retries (underlying KVTransferError: Failed to get kvcache from prefill instance, it might be dead)	Log

The other 3 files in this job passed: test_disaggregation_basic.py (6 tests OK), test_mori_transfer_engine_e2e.py (2 OK), test_nixl_transfer_engine_e2e.py (2 OK).

Failure Summary

The PP disaggregation accuracy test test_gsm8k failed. During the gsm8k eval, the mooncake KV-cache transfer from the prefill instance to the decode instance timed out (Sync batch data transfer timeout after ~30s), the decode side raised KVTransferError: Failed to get kvcache from prefill instance, it might be dead, and the decode PP scheduler then hung in an inter-stage gloo send until the 300 s scheduler watchdog fired. The decode server (port 11200) became unreachable, so the eval's /generate calls returned HTTP 500 and the test exhausted its retries.

Regression Status

Known recurring failure — failing every completed scheduled run since 2026-07-19; last green 2026-07-18.

Recent history of test_disaggregation_pp.py (test_gsm8k) in job stage-b-test-large-8-gpu-mi35x-disaggregation-amd-rocm720 (comparable completed runs: branch=main, event=schedule):

Date	Run	Test File Status	Failed Function	Error
Jul 23	30031591211	❌ Failed	test_gsm8k	KVTransferError → watchdog timeout (soft)
Jul 22	29944235234	❌ Failed	test_gsm8k	KVTransferError "prefill dead"
Jul 21	29855003895	❌ Failed	test_gsm8k	KVTransferError → watchdog soft and hard (soft=False)
Jul 20	29766743590	❌ Failed	test_gsm8k	KVTransferError "prefill dead"
Jul 19	29697501385	❌ Failed	test_gsm8k	KVTransferError "prefill dead"
Jul 18	29654571916	✅ Passed	—	—

First observed failure: 2026-07-19. Last passing: 2026-07-18 (all 4 disagg files green). Verdict is at the test-file level (each historical log confirmed the same test_disaggregation_pp.py KVTransfer signature). The parent workflow has shown failure on every day in the window, but that reflects other jobs too — this specific test file has an exact 07-18 green / 07-19-onward red boundary.

Failure Cluster

Disaggregation PP KV-transfer timeout → "prefill dead" hang (mooncake backend)

Facts (verified observations)

• Failing call site: test_disaggregation_pp.py:293 → run_eval(args) → few_shot_gsm8k.run_batch / cache_prefix → _assert_success → RuntimeError: Internal Server Error; retry loop then raises retry() exceed maximum number of retries. (job log ~lines 25316–25376)
• Crash sequence (verbatim from log): transfer_engine_py.cpp:614] Sync batch data transfer timeout after 30000001732ns (18:10:22) → [PP1 TP1] Decode transfer failed ... KVTransferError(...): Failed to get kvcache from prefill instance, it might be dead → py-spy shows scheduler PP0 TP1 blocked in c10d::ProcessGroupGloo::SendWork::wait at _pp_commit_comm_work (scheduler_pp_mixin.py:924) called from event_loop_pp_disagg_decode (scheduler_pp_mixin.py:437) → [PP0 TP0] Scheduler watchdog timeout (self.watchdog_timeout=300, self.soft=True) (18:16:43).
• Downstream symptom: ConnectionRefusedError: [Errno 111] Connect call failed ('127.0.0.1', 11200) in the mini-lb; eval KeyError: 'answer' because /generate returned 500.
• Launch config: --model-path Qwen/Qwen3-8B --tp-size 2 --pp-size 2 --disable-overlap-schedule --attention-backend aiter --disaggregation-transfer-backend mooncake --disaggregation-ib-device rdma0.
• Differentiator: the same job passed test_disaggregation_basic.py (mooncake, tp1, no PP) and the mori/nixl e2e tests (Llama-3.2-1B, no PP). Only the PP + mooncake + Qwen3-8B combination fails.
• Same signature reproduced in all 5 failing runs (07-19 → 07-23); on 07-21 the watchdog escalated from soft to hard.
• Regression window: b3a0185cab..d4801be447 = 11 commits. scheduler_pp_mixin.py (where the hang lives) was not modified in this window.

Hypothesised Causes (with confidence + disconfirming evidence)

• [LOW] Commit cce5fe76 (#31687) "Move the WAR barrier to right after each run_batch launch" — the only window commit touching disaggregation/decode.py + disaggregation/prefill.py.
  • Supporting: lands in the regression window; edits disagg event loops.
  • Disconfirming (strong, effectively exonerates it): _war_barrier_enabled = is_cuda() or SGLANG_ENABLE_WAR_BARRIER (scheduler.py:1501), so on ROCm _apply_war_barrier() is a no-op regardless of call position; and it edits the non-PP event_loop_overlap_disagg_decode/prefill, while the hang is in event_loop_pp_disagg_decode (scheduler_pp_mixin.py), which the commit does not touch.
  • How to verify: git revert cce5fe7696 on a branch and re-run — expected to have no effect on AMD.
• [LOW] Some other sglang commit in b3a0185cab..d4801be447.
  • Supporting: clean 07-18 green / 07-19 red boundary.
  • Disconfirming: none of the 11 commits touches the AMD PP+mooncake KV-transfer path; the hanging file scheduler_pp_mixin.py is unchanged in the window.
  • How to verify: git bisect b3a0185cab..d4801be447 on MI35x 8-GPU with the exact pp launch.
• [LOW] Environment/image regression (mooncake-transfer-engine / RDMA / RCCL / ROCm libs in the daily-rebuilt image).
  • Supporting: the initiating failure is a transport-level Sync batch data transfer timeout (mooncake C++), and the image is rebuilt daily; a mooncake/RDMA change would fit a code-invisible 07-19 boundary.
  • Disconfirming: cannot diff image contents from the job log; non-PP mooncake (test_disaggregation_basic.py) still passes, so any transport regression would have to be PP-specific.
  • How to verify: compare mooncake/RDMA package versions between the 20260718 and 20260719 images; inspect the prefill server log for a crash/OOM around 18:10 that makes it "dead".

No hypothesis reaches MEDIUM. This needs a git bisect over the window plus an image-contents diff, since no in-window source change plausibly explains a PP-only mooncake transfer timeout on ROCm.

In-flight Fix Check

• In-flight fix: ❌ none found that directly targets this PP+mooncake transfer-timeout hang. Related open PP-disagg robustness work exists but is not a confirmed fix: #31798 / #31869 (PD+PP consensus authoritative), #31217 (staging-buffer failure handling), #32171 (mooncake host buffers). (Searched open PRs for disaggregation pp decode, mooncake kvcache transfer, KVTransferError prefill.)

Suggested Triage Steps

• git bisect over b3a0185cab..d4801be447 on MI35x 8-GPU, reproducing with the exact --tp-size 2 --pp-size 2 --disaggregation-transfer-backend mooncake command.
• Diff the docker image (mooncake-transfer-engine, RDMA/RCCL, ROCm) between ...-mi35x-20260718 and ...-mi35x-20260719; the initiating error is a mooncake transport timeout, so an infra/image bisect is as important as the code bisect.
• Pull the prefill server stdout/stderr for this run (~18:10) to confirm whether it actually crashed/OOM'd (making it "dead") vs. just stalled — this decides prefill-side vs. transport-side root cause.
• Use the passing test_disaggregation_basic.py (non-PP mooncake) as an A/B control to isolate the PP-specific factor.

Status

• Persistence: Failing every completed scheduled run since 2026-07-19 (5 consecutive days); last green 2026-07-18.
• Scope: 1 test file (test_disaggregation_pp.py::test_gsm8k) in job stage-b-test-large-8-gpu-mi35x-disaggregation-amd-rocm720. Same mooncake+PP signature likely also affects the non-720 pr-test-amd.yml disagg job (not verified in this analysis).
• Blocked work: PD + pipeline-parallel disaggregation CI signal on AMD MI35x ROCm 7.2.
• In-flight fix: ❌ none found directly; related PP-disagg PRs listed above are unconfirmed.

stage-c-test-large-8-gpu-amd-mi35x-rocm720 (linux-mi35x-gpu-8, 1) — failed step(s): Run test

Commit Info

• sglang: d0b9689
• aiter: 9127c94a1 (v0.1.17.dev110+g9127c94a1, source: dev image preserved — AITER_COMMIT_OVERRIDE empty, so this is a regular pr-test-amd-rocm720 run, not a scout run)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/test_kimi_k25_mxfp4_bcg_mi35x.py	TestKimiK25MXFP4BcgMI35x.test_bcg_gsm8k (variant=bcg)	Exception: Server process exited with code -9 (server never came up — model-weight loading hung during _load_w13)	Log

Failure Summary

The BCG GSM8K test for amd/Kimi-K2.5-MXFP4 (TP=4, MI35x) never reached the eval phase: its server process died during startup. Model-weight loading stalled at shard 63/64 (98%), all ThreadPoolExecutor loader threads blocked on a HIP driver mutex inside _load_w13 (moe/fused_moe_triton/layer.py:614), the 300 s soft watchdog fired repeatedly, one TP rank eventually reached dist_barrier_after_load while others were still stuck, and the launcher SIGKILLed the process tree (exit -9). The subsequent test in the same shard (test_breakable_cuda_graph.py) ran normally, so the shard/runner was otherwise healthy.

Regression Status

Never-passed (new test, red on every run where it has executed).

The test file test_kimi_k25_mxfp4_bcg_mi35x.py was added by #27833 (merged 2026-06-24). It is registered to suite stage-c-test-large-8-gpu-amd-mi35x and is distributed across the mi35x-gpu-8 stage-c partitions run-to-run, so history below is tracked at the test-file level across all partitions (0/1/2), not the job-matrix index.

Recent history of test_kimi_k25_mxfp4_bcg_mi35x.py (test_bcg_gsm8k) in stage-c-test-large-8-gpu-amd-mi35x-rocm720 (comparable completed event=schedule, branch=main runs):

Date (UTC)	Run	Partition (job)	Test File Status	Failed Function	Error
07-23	30031591211	p1 (89289299762)	❌ Failed	test_bcg_gsm8k	loading hang → exit -9
07-22	29944235234	p1 (89005546170)	❌ Failed	test_bcg_gsm8k	loading hang / watchdog
07-21	29855003895	p0 (88717083227)	❌ Failed	test_bcg_gsm8k	loading hang / watchdog
07-20	29766743590	p0 (88434813772)	❌ Failed	test_bcg_gsm8k	loading hang / watchdog
07-19	29697501385	— (only 2 partitions that run; test not scheduled)	➖ Not run	—	—
07-18	29654571916	p0 (88106525133)	❌ Failed	test_bcg_gsm8k	loading hang / watchdog
07-17	29601526478	p0 (87954595359)	❌ Failed	test_bcg_gsm8k	loading hang / watchdog
07-16	29521557976	p0 (87699749969)	❌ Failed	test_bcg_gsm8k	loading hang / watchdog

No passing run found in the queried window. First observed failure: 2026-07-16 (earliest run examined); last passing: never. Same failure signature every time.

Failure Cluster

Model-weight loading hang during _load_w13 (HIP driver mutex) → 300 s watchdog → dist_barrier_after_load failure → server SIGKILL (exit -9)

Facts (verified observations)

• Load began 20:28:40 (avail mem≈278 GB per rank, TP4). Multi-thread shard loading reached 98% Completed | 63/64 at 20:33:23, then stalled — the 64th shard never completed (log line 4535).
• py-spy dumps for all 4 TP PIDs show every ThreadPoolExecutor-1_* loader thread parked in pthread_mutex_lock inside libamdhip64.so.7.2 under c10::hip::memcpy_and_sync ← copy_ ← _load_w13 (moe/fused_moe_triton/layer.py:614) ← device_aware_wrapper (model_loader/utils.py:324) (log ~17540-17600).
• Soft watchdog (watchdog_timeout=300, self.soft=True) fired first at 20:33:23 (TokenizerManager) and repeatedly on TP0-3 thereafter; because soft=True (CI) it only dumped py-spy and continued.
• At 20:59:45, rank 2 passed monitoredBarrier but errored waiting on rank 0; dist_barrier_after_load (load_model_utils.py:270) raised ValueError: TP rank 2 could finish the model loading, but there are other ranks that didn't finish loading → SIGQUIT → kill_process_tree → server exit -9. Test ran 1901 s before failing.
• Launch config (from log): --tensor-parallel-size 4 --mem-fraction-static 0.765 --enable-aiter-allreduce-fusion --cuda-graph-backend-prefill breakable --model-loader-extra-config '{"enable_multithread_load": true}', env SGLANG_AITER_MLA_PERSIST=1.
• Sibling non-BCG test test_kimi_k25_mxfp4.py PASSED (560 s, run 29944235234 p2) using the same model, same enable_multithread_load: true, same SGLANG_AITER_MLA_PERSIST=1, but --tp 8 and --mem-fraction-static 0.8. So the model itself loads fine on this stack.
• Identical loading-hang/watchdog signature verified in the 07-16, 07-17, 07-21 logs (not accuracy failures).

Hypothesised Causes (with confidence)

• [MEDIUM] The BCG variant's TP=4 config (vs the passing sibling's TP=8) doubles per-rank weight volume and concurrent host→device copy pressure, hanging the HIP driver on the final weight shard during multithreaded _load_w13 on MI35x / ROCm 7.2.
  • Supporting: the only functional difference vs the green sibling that touches the loading path is TP count / per-rank weight size; all loader threads are blocked in the HIP driver mutex during memcpy_and_sync; the stall is deterministic at 63/64 shards across ≥6 runs.
  • Disconfirming: the BCG launch also differs in --enable-aiter-allreduce-fusion, --mem-fraction-static 0.765, --chunked-prefill-size 8192, --max-running-requests 1024 — none isolated to TP; symptom is a hang, not an OOM error, so a pure memory-capacity explanation is not directly evidenced.
  • How to verify: rerun the BCG config with enable_multithread_load: false (single-thread load) and separately with --tp 8; if either clears the hang, the multithreaded-copy-at-TP4 hypothesis is confirmed.
• [LOW] Memory headroom at TP=4 for the full MXFP4 checkpoint is marginal, causing driver-level stalls near the end of loading.
  • Supporting: TP=4 holds ~2× the per-rank weight of the green TP=8 run.
  • Disconfirming: no HIP OOM/allocation error is printed; 278 GB was free at load-begin; failure manifests as a mutex hang, not an allocation failure.
  • How to verify: log free HBM per rank just before the 64th shard; try --mem-fraction-static lower / weight_loader_disable_mmap.

No hypothesis reaches HIGH; a single-variable rerun (multithread-load off, or TP=8) is needed to promote confidence.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for Kimi+BCG, test_kimi_k25_mxfp4_bcg, breakable+cuda+graph+kimi, multithread_load+hang; only unrelated results returned.

Suggested Triage Steps

• Reproduce on an MI35x / ROCm 7.2 container with the exact BCG launch command; rerun once with --model-loader-extra-config '{"enable_multithread_load": false}' to test the HIP-multithreaded-copy-deadlock hypothesis.
• A/B the TP dimension: rerun the BCG config at --tp 8 (matching the green sibling). If it passes, the test's TP=4 assumption is the problem and the test config should be revisited.
• Since this test has never passed in CI (red since 2026-07-16) while its validating PR (#27833) reported GSM8K ~0.93, ping the PR author to confirm the validated environment vs the MI35x/ROCm 7.2 runner; consider quarantining/marking the BCG test on this runner until the loading hang is resolved so it stops masking the stage-c signal.

Status

• Persistence: Failing on every executed run since 2026-07-16 (7 of 7 runs where it was scheduled); never passed.
• Scope: Single test file test_kimi_k25_mxfp4_bcg_mi35x.py in stage-c-test-large-8-gpu-amd-mi35x-rocm720; migrates between partitions 0/1 run-to-run. Sibling non-BCG Kimi test is green, so blast radius is limited to the BCG capture-path signal.
• In-flight fix: ❌ none found in open PRs.

multimodal-gen-unit-test-amd-rocm720 — failed step(s): Run diffusion unit tests

Commit Info

• sglang: d0b96898
• aiter: 9127c94a (v0.1.17.dev110+g9127c94a1, source: dev image preserved — AITER_COMMIT_OVERRIDE empty, Dev/patched version detected → skipping rebuild)

Failed Tests

Test File	Test Function	Error	Log
python/sglang/multimodal_gen/test/unit/test_pi05_action_api.py	test_msgpack_roundtrip_preserves_string_keys_and_numpy_payloads	ModuleNotFoundError: No module named 'msgpack'	Log

Result line: 1 failed, 793 passed, 1 skipped, 5 deselected, 50 subtests passed.

Failure Summary

A single unit test failed. test_msgpack_roundtrip_preserves_string_keys_and_numpy_payloads calls pack_msgpack() in python/sglang/multimodal_gen/runtime/entrypoints/vla/protocol.py:59, which does a lazy import msgpack. The msgpack package is not installed in the ROCm 7.2 CI image (rocm/sgl-dev:v0.5.15.post1-rocm720-mi30x-20260723), so the import raises ModuleNotFoundError. All 793 other collected tests passed.

Regression Status

Known recurring failure (environment/dependency gap in the ROCm720 CI image). NOT caused by commit d0b96898.

Recent history of test_pi05_action_api.py (test_msgpack_roundtrip_preserves_string_keys_and_numpy_payloads) in job multimodal-gen-unit-test-amd-rocm720 (comparable completed event=schedule, branch=main runs):

Date	Run	Job	Test File Status	Failed Function	Error
Jul 23	30031591211	job	❌ Failed	test_msgpack_roundtrip...	ModuleNotFoundError: msgpack
Jul 22	29944235234	job	❌ Failed	test_msgpack_roundtrip...	ModuleNotFoundError: msgpack
Jul 22	29855003895	job	❌ Failed	test_msgpack_roundtrip...	ModuleNotFoundError: msgpack
Jul 21	29766743590	job	❌ Failed	test_msgpack_roundtrip...	ModuleNotFoundError: msgpack
Jul 18	29654571916	job	❌ Failed	test_msgpack_roundtrip...	ModuleNotFoundError: msgpack
Jul 14	29355401408	job	❌ Failed	test_msgpack_roundtrip...	ModuleNotFoundError: msgpack (781 passed)
Jul 10	29113205616	job	✅ Passed	—	test not collected (743 passed, no msgpack test)

First observed failure: run 29355401408 (Jul 14). Last known green: run 29113205616 (Jul 10), where the msgpack test was not yet part of the AMD-collected suite (743 vs 793 items). The failure was introduced when the nested multimodal_gen unit tests were enabled on AMD, not by a code regression.

Failure Cluster

ModuleNotFoundError: msgpack missing in ROCm720 CI image

Facts (verified observations)

• msgpack is a declared dependency: python/pyproject.toml:110 lists "msgpack".
• The failing import is a lazy import msgpack inside pack_msgpack at python/sglang/multimodal_gen/runtime/entrypoints/vla/protocol.py:59 (verified from stack trace and source).
• The error is identical (No module named 'msgpack') across 6 consecutive scheduled runs (Jul 14–23); it is deterministic, not flaky.
• The test count for this job rose from 743 (Jul 10, msgpack test absent, job green for this file) to 781/793 (Jul 14+, msgpack test present and failing), consistent with the nested multimodal_gen suite being newly enabled on AMD ROCm720.
• Only this one test fails; the environment gap is isolated to the msgpack package.

Hypothesised Causes

• [HIGH] The ROCm 7.2 CI docker image does not have msgpack installed, even though it is declared in python/pyproject.toml. The nested multimodal_gen unit suite was recently turned on for AMD (see PRs #31483, #31844) without adding msgpack to the ROCm image's installed deps.
  • Supporting: import fails only for msgpack; the package is in pyproject.toml but absent at runtime; failure begins exactly when the suite was enabled (Jul 14), test count jumps; an open fix PR #31899 titled "[AMD] Add msgpack to ROCm diffusion deps" targets precisely this.
  • Disconfirming: (none found) — this is an environment/dependency gap, well supported by the log and the existing fix PR.
  • How to verify: in the ROCm720 container, run python -c "import msgpack"; confirm it raises, then pip install msgpack and re-run the test.

In-flight Fix Check

• In-flight fix: ✅ [#31899](https://github.com/sgl-project/sglang/pull/31899) (open since 2026-07-21) — "[AMD] Add msgpack to ROCm diffusion deps (fix multimodal-gen unit test ModuleNotFoundError)" — this is the exact fix; chase reviewers, do NOT open a duplicate.
• Related enablement PRs: #31844 (gate nested unit tests behind LAPACK/msgpack env skips), #31483 (run nested multimodal_gen unit tests on AMD).
• Search: is:pr is:open msgpack → 17 results; #31899 is the direct match.

Suggested Triage Steps

• Verify #31899 adds msgpack to the ROCm diffusion deps / CI image and merges cleanly; ping reviewers to land it.
• Alternatively/until then, #31844 proposes skipping the test behind an msgpack env guard — decide whether to install the dep (preferred, since it's a declared dependency) or skip.
• Confirm the ROCm720 image (rocm/sgl-dev:...-rocm720-mi30x-*) rebuild picks up msgpack after the fix, then re-run multimodal-gen-unit-test-amd-rocm720.

Status

• Persistence: Failing every completed scheduled run since 2026-07-14 (run 29355401408); ~9 days, 6+ consecutive runs.
• Scope: 1 test in the multimodal-gen-unit-test-amd-rocm720 job. Environment gap specific to the ROCm720 CI image; not a code regression, not aiter-related.
• Blocked work: Blocks a green signal for the AMD ROCm720 multimodal-gen unit suite (1 red test masks the otherwise-passing 793).
• In-flight fix: ✅ #31899 (open since 2026-07-21) — awaiting merge.

multimodal-gen-test-1-gpu-amd-rocm720 (linux-mi325-1gpu-sglang, 3) — failed step(s): N/A

Commit Info

• sglang: d0b96898 (HEAD of main, verified in local git log)
• aiter: N/A — the job log blob was never uploaded (runner died mid-step), so [CI-AITER-CHECK] markers are unreadable.

Failed Tests

Test File	Test Function	Error	Log
N/A — no test-level result captured	N/A	Self-hosted runner lost communication with the server during step 9 (Run diffusion server tests (1-GPU))	Job

This is not a test failure. The runner died ~20 min into the diffusion test step before any pytest result, log, or failure artifact was produced. The specific test case in partition 3 that was executing when the runner dropped is unknown — the log blob returns BlobNotFound and step 10 (Upload diffusion failure artifacts) never ran.

Failure Summary

Job multimodal-gen-test-1-gpu-amd-rocm720 (linux-mi325-1gpu-sglang, 3) failed because the self-hosted MI325 runner (linux-mi325-1gpu-sglang-ng8rq-runner-5ddtr) lost communication with GitHub during step 9. Step 9 started 2026-07-23T22:04:48Z and the runner dropped at 2026-07-23T22:24:53Z (~20 min in, well under the 60-min step timeout). The job's log blob was never flushed and no failure artifacts were uploaded, so there is no captured test-level failure.

Regression Status

Infrastructure issue — persistent/recurring on shard 3. Never-passed in the queryable 15-run window.

Recent history of shard 3 (multimodal-gen-test-1-gpu-amd-rocm720 (…, 3)) — most recent comparable scheduled runs (branch=main, event=schedule, completed job only). "Status" here is the failure signature, verified per-job via check-run annotations + step state, not just the job's pass/fail flag:

Date	Run	Shard-3 Job	Signature
07-23	30031591211	job	❌ Runner lost communication (this job)
07-22	29944235234	job	❌ Runner lost communication
07-21	29855003895	job	❌ Runner lost communication
07-20	29766743590	job	❌ Runner lost communication
07-19	29697501385	job	❌ Runner lost communication
07-18	29654571916	job	❌ Runner lost communication
07-17	29601526478	job	❌ Runner lost communication
07-16	29521557976	job	❌ Runner lost communication
07-15	29438298954	job	❌ Runner lost communication
07-14	29355401408	job	❌ Runner lost communication
07-13	29273187026	job	❌ Runner lost communication
07-12	29202664092	job	❌ Runner lost communication
07-11	29162225443	job	❌ Runner lost communication
07-10	29113205616	job	❌ Runner lost communication
07-09	29039527644	job	❌ Runner lost communication (first observed)
07-08	28964460204	job	⚠️ Different — genuine pytest failure (step 9 completed/failure, exit 1, artifacts uploaded normally)

• First observed (runner-loss): 2026-07-09. 14 consecutive scheduled runs (07-09 → 07-23) show the identical runner-loss signature on shard 3.
• No passing run for shard 3 anywhere in the 15-run window. The 07-08 run failed too, but with a different signature (a real test failure that produced logs) — so it is not a "green" baseline and marks the boundary before the runner-loss pattern began.
• Current run 30031591211 is still IN-FLIGHT overall (shards 0,1 in_progress, shard 2 ✅ passed, shard 3 ❌ this failure).

Failure Cluster

Self-hosted MI325 runner lost communication during 1-GPU diffusion test step (shard 3).

Facts (verified observations)

• Check-run annotation on this job (FACT): "The self-hosted runner lost communication with the server. Verify the machine is running and has a healthy network connection. Anything in your workflow that terminates the runner process, starves it for CPU/Memory, or blocks its network access can cause this error."
• Job conclusion is failure, yet step 9 (Run diffusion server tests (1-GPU)) is frozen at in_progress and step 10 (Upload diffusion failure artifacts) is pending — the job was force-failed externally, not by a clean test exit.
• Job log endpoint returns BlobNotFound (blob never uploaded); run-level artifacts count = 0. No diffusion-failure artifacts exist.
• Timing: step 9 22:04:48Z → job completed_at 22:24:53Z ≈ 20 min; the 07-22 shard-3 job had the same signature ~18 min in (18:20:31Z → 18:38:40Z). Consistent mid-step drop.
• Scope: signature reproduced on shard 3 in 14 consecutive scheduled runs (07-09 → 07-23), all citing the identical "runner lost communication" annotation. The 07-08 run (85944006119) had a normal completed/failure step 9 (real test failure), confirming the runner-loss pattern is a newer, distinct regime.
• Step 5 (Start CI container) took ~11 min (21:49:09Z → 22:00:36Z) — unusually long but succeeded; not the failing step.
• Sibling shard 2 passed today; shards 0/1 failures on 07-22 were genuine test failures (they have logs / normal exit code 1 annotations), so the runner-loss pattern is specific to shard 3's workload, not all 1-GPU shards.

Hypothesised Causes (with confidence)

• [LOW] A specific case in LPT partition 3 of the 1-gpu suite starves the host of CPU/RAM or hangs the GPU ~18–20 min into the run, killing the runner agent. Partition 3 by LPT tends to carry the heaviest-estimated cases; the 1-GPU set includes memory-heavy models (HunyuanVideo, FLUX.1-dev, Qwen-Image, Wan2.1).
  • Supporting: failure is shard-3-specific and reproducible; sibling shard 2 passes; drop occurs at a consistent ~18–20 min mark mid-step; the GitHub annotation itself names "starves it for CPU/Memory" as a cause.
  • Disconfirming: zero log/artifact evidence — no OOM message, stack trace, or GPU fault was captured (the runner died before flushing). The specific partition-3 case list is not recoverable without logs. Equally consistent with a flaky physical runner / network / host-crash unrelated to the workload.
  • How to verify: re-run this shard alone with host dmesg/free -h/rocm-smi sampling to a persistent volume; or split partition 3 further and bisect which case precedes the drop; check the runner host's system journal around 22:24Z for OOM-killer or reboot.
• [LOW] Runner-fleet / host-health issue on the MI325 pool independent of test content (bad node, network flap, agent crash).
  • Supporting: "lost communication" is the classic self-hosted-runner infra symptom; identical across 14 runs.
  • Disconfirming: if it were random fleet health, we'd expect it to hit shards 0/1/2 too, but only shard 3 shows this exact signature consistently — pointing back toward workload rather than pure infra.
  • How to verify: correlate the physical runner names across the 14 failing shard-3 jobs; if always the same node → node-specific; if varied → workload-driven.

No hypothesis exceeds LOW confidence — the runner died before emitting any diagnostic, so root cause cannot be established from CI data alone.

In-flight Fix Check

• In-flight fix: ❌ none found targeting shard-3 runner-loss. Searches run: search/issues q=repo:sgl-project/sglang is:pr is:open with diffusion+runner+communication (0 hits), mi325+runner+lost (0 hits), multimodal-gen+runner (29 hits, none about runner-loss), multimodal-gen-test+partition (5 hits, none matching). Related-but-not-a-fix: #28584 (WIP: test AMD PR workflow on new MI300 runners), #29584 (fix pre-existing MI325 CI failures). Neither addresses this shard-3 runner-loss.

Suggested Triage Steps

• Pull the runner host's system journal / dmesg for linux-mi325-1gpu-sglang-ng8rq-runner-5ddtr around 2026-07-23T22:24Z (and the same node for prior failing runs) to confirm OOM-killer, kernel panic, GPU hang, or reboot.
• Re-run shard 3 in isolation with free -h / rocm-smi / df -h sampled to a persistent path (independent of the runner agent) so evidence survives a runner death.
• Temporarily raise --total-partitions or move the heaviest partition-3 case to a standalone shard to bisect which diffusion case precedes the ~18–20 min drop; if one case reliably triggers it, quarantine that case on AMD 1-GPU until fixed.
• Confirm whether the 14 failing shard-3 jobs share one physical runner node (node-specific health) vs. rotate across nodes (workload-driven) by comparing runner_name across them.
• Because logs are lost on death, add a lightweight background resource logger to the diffusion 1-GPU step so future runner-loss events leave a trail.

Status

• Persistence: Runner-loss on shard 3 since 2026-07-09 (~14 days), every completed scheduled run since; identical annotation each time. Prior boundary run 07-08 failed differently (real test failure).
• Scope: Isolated to multimodal-gen-test-1-gpu-amd-rocm720 shard 3 on the MI325 pool; sibling shard 2 passes and shards 0/1 fail with unrelated test-level errors.
• Blocked work: Removes all CI signal for whichever 1-gpu diffusion cases LPT assigns to partition 3 — those cases are effectively untested on ROCm 7.2 (no log, no result).
• In-flight fix: ❌ none found in open PRs — needs runner-host investigation (infra) and/or a new PR to split/quarantine partition 3.

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-07-23 23:34 UTC)

[amd-bot]

pr-test-amd.yml — 28 failure(s)

Run started (UTC): 2026-07-22 00:26 UTC
Last scanned (UTC): 2026-07-23 18:35 UTC
Commits: sglang b98a577

Summary

Counts: 28 failures · 10 clusters · 21 carrying over (≥3 days) · 7 new/recent (≤2 days) · all pr-test-amd.yml, spanning 4 completed scheduled runs (Jul 22–23)

#	Cluster	Job	Test File	Test Function	Status	Hypothesis (confidence)
1	1-GPU diffusion shard hang→kill (no log)	multimodal-gen-1gpu (…,0)	test_server_1_gpu.py	N/A (hang)	Never-passed ≥8d	[MEDIUM] GPU hang/runner reap; log lost
2	1-GPU diffusion shard hang→kill (no log)	multimodal-gen-1gpu (…,1)	test_server_1_gpu.py	N/A (hang)	Never-passed ≥8d	[MEDIUM] pre-existing infra hang
3	1-GPU diffusion shard hang→kill (no log)	multimodal-gen-1gpu (…,0)	test_server_1_gpu.py	N/A (90-min timeout)	Never-passed ≥8d	[MEDIUM] partition aggregate runtime > budget
4	1-GPU diffusion shard hang→kill (no log)	multimodal-gen-1gpu (…,1)	N/A (log expired)	N/A	Recurring ≥10d	[LOW] unidentified — log purged
5	1-GPU diffusion shard hang→kill (no log)	multimodal-gen-1gpu (…,0)	test_server_1_gpu.py	N/A (hang)	Never-passed ≥8d	[MEDIUM] GPU hang
6	1-GPU diffusion shard hang→kill (no log)	multimodal-gen-1gpu (…,1)	N/A (runner loss)	N/A	Recurring ≥5d	[MEDIUM] GPU hang/runner loss
7	1-GPU diffusion shard hang→kill (no log)	multimodal-gen-1gpu (…,0)	N/A (hang)	N/A	Never-passed ≥7d	[MEDIUM] hang, BlobNotFound
8	1-GPU diffusion shard hang→kill (no log)	multimodal-gen-1gpu (…,1)	test_server_1_gpu.py	N/A (hang)	Never-passed ≥8d	[LOW] hang, log uncapturable
9	Diffusion update_weights transformer reload 400	multimodal-gen-1gpu (…,3)	test_update_weights_from_disk.py	TestUpdateWeightsFromDisk (Qwen-Image/FLUX.2)	Never-passed ≥8d	[HIGH] diffusers layout not reconciled → shape mismatch (#31924)
10	Diffusion update_weights transformer reload 400	multimodal-gen-1gpu (…,3)	test_update_weights_from_disk.py	TestUpdateWeightsFromDisk	Never-passed ≥8d	[HIGH] shape mismatch at linear.py:410
11	Diffusion update_weights transformer reload 400	multimodal-gen-1gpu (…,3)	test_update_weights_from_disk.py	TestUpdateWeightsFromDisk	Never-passed ≥10d	[HIGH] #31924
12	Diffusion update_weights transformer reload 400	multimodal-gen-1gpu (…,3)	test_update_weights_from_disk.py	TestUpdateWeightsFromDisk	Never-passed ≥8d	[HIGH] #31924
13	FLUX.2 modelopt-FP8 HIPBLAS_STATUS_NOT_SUPPORTED (gfx942, 2-GPU)	multimodal-gen-2gpu (…,0)	test_server_2_gpu.py	test_diffusion_generation[flux2_modelopt_fp8_tp2_t2i]	Never-passed ≥8d	[HIGH] FP8 GEMM unsupported gfx942
14	FLUX.2 modelopt-FP8 HIPBLAS_STATUS_NOT_SUPPORTED (gfx942, 2-GPU)	multimodal-gen-2gpu (…,0)	test_server_2_gpu.py	test_diffusion_generation[flux2_modelopt_fp8_tp2_t2i]	Never-passed ≥8d	[MEDIUM] FP8 GEMM (+joy_echo dl flake)
15	FLUX.2 modelopt-FP8 HIPBLAS_STATUS_NOT_SUPPORTED (gfx942, 2-GPU)	multimodal-gen-2gpu (…,0)	test_server_2_gpu.py	test_diffusion_generation[flux2_modelopt_fp8_tp2_t2i]	Never-passed ≥8d	[FACT] torch._scaled_mm unsupported gfx942
16	FLUX.2 modelopt-FP8 HIPBLAS_STATUS_NOT_SUPPORTED (gfx942, 2-GPU)	multimodal-gen-2gpu (…,0)	test_server_2_gpu.py	test_diffusion_generation[flux2_modelopt_fp8_tp2_t2i]	Never-passed ≥7d	[MEDIUM] FP8 GEMM + hang cascade
17	KeyError: 'answer' PP-disagg GSM8K eval	stage-b-8gpu-mi35x-disagg	test_disaggregation_pp.py	TestDisaggregationPrefillPPDynamicChunkAccuracy.test_gsm8k	Flaky (cluster ≥Jul18)	[MEDIUM] Mooncake RDMA 30s KV-transfer timeout
18	KeyError: 'answer' PP-disagg GSM8K eval	stage-b-8gpu-mi35x-disagg	test_disaggregation_pp.py	TestDisaggregationPrefillPPDynamicChunkAccuracy.test_gsm8k	Flaky ~40%	[LOW] nondeterministic empty generation
19	KeyError: 'answer' PP-disagg GSM8K eval	stage-b-8gpu-mi35x-disagg	test_disaggregation_pp.py	test_gsm8k (PP classes)	Flaky 7/13	[LOW] dropped request in PP KV-transfer
20	MXFP4 GSM8K accuracy < strict 0.92	stage-b-1gpu-small-mi35x	test_quark_mxfp4.py	TestFP8ToMXFP4MOETP1.test_gsm8k	Flaky since Jul22	[HIGH] threshold too tight (0.912–0.928 band)
21	MXFP4 GSM8K accuracy < strict 0.92	stage-b-1gpu-small-mi35x	test_quark_mxfp4.py	TestFP8ToMXFP4MOETP1.test_gsm8k	Flaky 2/9	[HIGH] boundary threshold
22	MXFP4 GSM8K accuracy < strict 0.92	stage-b-1gpu-small-mi35x	test_quark_mxfp4.py	TestFP8ToMXFP4MOETP1.test_gsm8k	Flaky 3/6	[HIGH] gate inside noise band
23	LoRA concurrent-adapter perf > 4000 ms	stage-b-1gpu-large (…,0)	test_bench_serving_1gpu_part1.py	test_online_lora_latency_with_concurrent_adapter_updates	Flaky 2/15 hard	[HIGH] threshold too tight + latency spikes
24	LoRA concurrent-adapter perf > 4000 ms	stage-b-1gpu-large (…,0)	test_bench_serving_1gpu_part1.py	test_online_lora_latency_with_concurrent_adapter_updates	Flaky 3/15 hard	[HIGH] noise-sensitive perf gate
25	Dynamo graph break in prefill piecewise CUDA-graph capture	stage-c-8gpu-mi35x (…,1)	test_deepseek_r1_mxfp4_8gpu.py	TestDeepseekR1MXFP4.setUpClass	New Jul22, self-healed	[HIGH] 745b2ca4 (#31816) → reverted
26	AttributeError: communicator.get_server_args (mock target removed)	stage-c-8gpu (…,3)	test_aiter_allreduce_fusion_amd.py	TestAiterAllreduceFusionGate.test_dense_tp_fuses	New Jul22, self-healed	[HIGH] 745b2ca4 (#31816) → reverted
27	AttributeError: _legacy_load_weights on reward model load	stage-b-1gpu-small-nondeterministic	test_reward_models.py	model load (setUp, exit -9)	New Jul22 00:26	[HIGH] becf252e (#28671)
28	Partition wall-time > 45-min step cap	stage-b-2gpu-large (…,1)	N/A (7/7 passed)	N/A (step timeout)	New Jul23	[FACT] wall 2701s > 2700s cap

Cluster: 1-GPU diffusion shard hang→kill (no log) — 8 jobs (rows 1-8)

• Affected: multimodal-gen-test-1-gpu-amd parametrized shards parts 0 & 1 across all 4 runs. Same symptom every run: step 9 Run diffusion server tests (1-GPU) sticks in_progress, job killed at ~72–122 min (below the 90-min step cap), if: always() upload never runs, log blob = BlobNotFound.
• Shared facts: sibling standalone shards (parts 2/3) upload logs fine → loss is specific to the killed parametrized shards. Head commits vary (LLM n-gram, spec-decode, docs-only) and are excluded — failure predates them by ≥8 days.
• Top hypothesis: [MEDIUM] a heavy test_server_1_gpu.py case wedges the GPU/runner; the step timeout fails to reap it cleanly so no log flushes. Disconfirming: exact case unverifiable (no log ever); walls vary 72–122 min, and parts 0/1 sometimes die near-simultaneously → could be shared-node runner reap.
• In-flight fix: ❌ none for the hang; 🟡 mitigation-only #31068, #30431 (hung-GPU recovery). Biggest blast radius, zero observability — top priority is restoring logs, not root-causing.
• Suggested triage: add a per-case subprocess timeout in run_suite.py/pytest_runner.py so a stuck case SIGKILLs and flushes the log + diffusion-failures/ artifact before the runner is reaped; then bisect ONE_GPU_CASES case-by-case on MI325.

Cluster: Diffusion update_weights_from_disk transformer reload 400 — 4 jobs (rows 9-12)

• Affected: multimodal-gen-test-1-gpu-amd (…,3) standalone test_update_weights_from_disk.py in all 4 runs; deterministic 4 failed / 2 passed / 4 errors.
• Shared facts: Qwen-Image → server-side bare AssertionError at linear.py:410 (param_data.shape == loaded_weight.shape) via weights_updater.py:434→203→295, "not found after mapping" for img_in.bias→x_embedder.bias; endpoint returns 400. FLUX.2 → fixture ERROR at setup (No weights dir for transformer). AMD-only (NVIDIA skips this nightly=True file).
• Top hypothesis: [HIGH] diffusers checkpoint layout not reconciled with sglang transformer params. Disconfirming: (none) — log, timing, and maintainer issue #31924 converge.
• In-flight fix: ✅ mitigation #31925 (skip on ROCm, open since 2026-07-21); real fix tracked in #31924. Chase reviewers on #31925.

Cluster: FLUX.2 modelopt-FP8 HIPBLAS_STATUS_NOT_SUPPORTED (gfx942, 2-GPU) — 4 jobs (rows 13-16)

• Affected: multimodal-gen-test-2-gpu-amd (…,0) in all 4 runs; first case flux2_modelopt_fp8_tp2_t2i.
• Shared facts: torch._scaled_mm FP8 GEMM returns HIPBLAS_STATUS_NOT_SUPPORTED at denoising.py:1624 (verified in 3 recoverable logs Jul16–18); crash cascades into ~17 server-startup errors → shard hang/runner-kill (recent runs BlobNotFound). Parts 1/2 pass.
• Top hypothesis: [FACT] FP8 GEMM path unsupported on gfx942; [HIGH] that failure poisons GPU/server state → hang. Disconfirming: recent runs' own logs lost, so today-specific chain inferred.
• In-flight fix: ✅ #28889 "native e4m3fnuz fp8 GEMM for FLUX.2 modelopt-FP8 on gfx942" (open since 2026-06-22, blocked). Unblock/merge; do NOT duplicate. Also isolate the case so it stops taking down the other 24.

Cluster: KeyError: 'answer' PP-disagg GSM8K eval — 3 jobs (rows 17-19)

• Affected: stage-b-test-large-8-gpu-mi35x-disaggregation-amd, test_disaggregation_pp.py::test_gsm8k; failing sub-test rotates across all 3 PP classes with identical KeyError: 'answer' at few_shot_gsm8k.py:123.
• Top hypothesis: [MEDIUM] Mooncake RDMA KV-transfer 30s timeout / nondeterministic dropped request → empty generation. Disconfirming: no code-level isolation; passes/fails interleave on same SHAs (flake, not a bisectable regression).
• In-flight fix: candidates #29127 (draft, RDMA NIC), #29984 (ZMQ sender race), #30551 (Mooncake room cleanup). Low blast radius (2/200 requests).

Cluster: MXFP4 GSM8K accuracy < strict 0.92 — 3 jobs (rows 20-22)

• Affected: stage-b-test-1-gpu-small-amd-mi35x, test_quark_mxfp4.py::TestFP8ToMXFP4MOETP1.test_gsm8k (Qwen3-30B-A3B, online FP8→MXFP4, TP1).
• Top hypothesis: [HIGH] strict assertGreater(acc, 0.92) sits inside the observed 0.906–0.928 variance band (invalid=0.0, server healthy). Disconfirming: (none) — aiter unchanged, Dense sibling stable, non-monotonic pass/fail.
• In-flight fix: ❌ none. Triage: run ~10× to confirm spread; loosen threshold (~0.90, mirroring Dense gate) or raise num_questions. Test author fxmarty-amd (#28291/#29328).

Cluster: LoRA concurrent-adapter perf > 4000 ms — 2 jobs (rows 23-24)

• Affected: stage-b-test-1-gpu-large-amd (…,0), test_online_lora_latency_with_concurrent_adapter_updates.
• Top hypothesis: [HIGH] noise-sensitive perf gate — 10/15 runs breach 4000 ms on first try and are retry-rescued; hard fails swing 4666→20742 ms; both cluster on ~00:2x UTC (runner contention). Disconfirming: pass→fail window is a single unrelated commit; not a clean regression.
• In-flight fix: ❌ none. Triage: sample median_e2e_latency_ms over ~30 runs; relax threshold or profile adapter-swap contention.

Cluster: RuntimeContext migration regressions — 2 jobs, 2 symptoms (rows 25-26)

• Affected: row 25 Dynamo __getattribute__ graph break in prefill piecewise CUDA-graph capture (test_deepseek_r1_mxfp4_8gpu.py); row 26 mock.patch.object(comm, "get_server_args") AttributeError (test_aiter_allreduce_fusion_amd.py). Distinct symptoms/stack frames, same root commit.
• Top hypothesis: [HIGH] both caused by 745b2ca4 (#31816, config-namespace migration #31813–#31817) — merged 08:18 UTC between the last green and first red run. Disconfirming: (none).
• In-flight fix: ✅ MERGED #32100 (f5dcbe8f, revert, 18:52 UTC Jul22). Both self-resolve on 24da0e51+ — confirm next scheduled run is green.

Cluster: AttributeError: _legacy_load_weights on reward model load — 1 job (row 27)

• Affected: stage-b-test-1-gpu-small-amd-nondeterministic, test_reward_models.py (server exit -9). Backend-agnostic Python — likely reproduces on NVIDIA too.
• Top hypothesis: [HIGH] becf252e (#28671) refactored LlamaForCausalLM.load_weights to call subclass-only self._legacy_load_weights, but LlamaForSequenceClassification (which borrows it via unbound call at llama_reward.py:77) lacks the method. Disconfirming: (none) — 5 prior runs green, first red is first run after merge.
• In-flight fix: ❌ none — needs a new PR (make _legacy_load_weights/_load_weights_v2 reachable on the reward classes).

Cluster: Partition wall-time > 45-min step cap — 1 job (row 28)

• Affected: stage-b-test-2-gpu-large-amd (…,1) — 7/7 files passed but wall 2701.69s > 2700s cap. test_bench_one_batch_2gpu.py (1014s vs est 630s) + test_unified_radix_cache_kl_full.py (669s vs est 400s) drove the overrun.
• Top hypothesis: [MEDIUM] chronically over-budget partition (LPT est 2514s = 93% of cap); [LOW] genuine slowdown of test_bench_one_batch_2gpu.py (573→695→1014s). Disconfirming: prior 14 runs mostly fit; cold aiter-JIT builds inflate warmup.
• In-flight fix: ✅ #30225 (2→3 partition split, open but stalled since 2026-07-06). Revive it; check whether test_bench_one_batch_2gpu.py is truly regressing.

Suggested triage order

Rank	Triage step	Targets	Why this first
1	Confirm merged revert #32100 greens the next scheduled run	rows 25-26	Fix already on main; zero remaining work, just verify
2	Add per-case subprocess timeout so hung diffusion shards flush logs; then bisect test_server_1_gpu.py cases	rows 1-8 (cluster C)	Largest cluster (8 jobs), currently zero observability
3	Unblock/merge #28889 (native e4m3fnuz FP8 GEMM gfx942) + isolate the flux2 case	rows 13-16 (cluster B)	4 jobs, verified [FACT] root cause, in-flight fix exists
4	Chase reviewers on skip-mitigation #31925	rows 9-12 (cluster A)	4 jobs, deterministic, mitigation ready
5	Open a new PR for reward-model _legacy_load_weights regression from becf252e	row 27	New regression, backend-agnostic, no in-flight fix

Note: all 4 runs are completed scheduled runs; run 30007380837 was flagged [IN-FLIGHT] in some per-job analyses at snapshot time — its listed failures are from concluded shards only. Clusters D/E/F are flaky (retry/threshold-sensitive), not correctness regressions.

---

Job	Failed Steps	Started
multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 3)	Run diffusion server tests (1-GPU)	2026-07-22T13:55
multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 0)	N/A	2026-07-22T15:55
multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 0)	N/A	2026-07-22T13:53
stage-c-test-large-8-gpu-amd-mi35x (linux-mi35x-gpu-8, 1)	Run test	2026-07-22T14:54
stage-c-test-large-8-gpu-amd (linux-mi325-8gpu-sglang, 3)	Run test	2026-07-22T18:10
multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 1)	N/A	2026-07-22T14:01
multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 1)	N/A	2026-07-22T02:20
multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 3)	Run diffusion server tests (1-GPU)	2026-07-22T02:36
multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 0)	N/A	2026-07-22T02:34
multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 0)	N/A	2026-07-22T04:17
stage-b-test-1-gpu-small-amd-mi35x (linux-mi35x-gpu-1)	Run test	2026-07-22T00:26
stage-b-test-large-8-gpu-mi35x-disaggregation-amd (linux-mi35x-gpu-8.fabric)	Run test_disaggregation	2026-07-22T00:43
stage-b-test-1-gpu-small-amd-nondeterministic (linux-mi325-1gpu-sglang)	Run test	2026-07-22T04:26
stage-b-test-1-gpu-large-amd (linux-mi325-1gpu-sglang, 0)	Run test	2026-07-22T02:51
stage-b-test-1-gpu-small-amd-mi35x (linux-mi35x-gpu-1)	Run test	2026-07-23T00:28
stage-b-test-large-8-gpu-mi35x-disaggregation-amd (linux-mi35x-gpu-8.fabric)	Run test_disaggregation	2026-07-23T01:07
multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 3)	Run diffusion server tests (1-GPU)	2026-07-23T02:14
multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 0)	N/A	2026-07-23T01:38
stage-b-test-1-gpu-large-amd (linux-mi325-1gpu-sglang, 0)	Run test	2026-07-23T02:49
multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 0)	N/A	2026-07-23T02:34
multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 1)	N/A	2026-07-23T04:33
stage-b-test-1-gpu-small-amd-mi35x (linux-mi35x-gpu-1)	Run test	2026-07-23T12:32
stage-b-test-large-8-gpu-mi35x-disaggregation-amd (linux-mi35x-gpu-8.fabric)	Run test_disaggregation	2026-07-23T12:32
multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 3)	Run diffusion server tests (1-GPU)	2026-07-23T14:19
multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 0)	N/A	2026-07-23T14:20
multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 1)	N/A	2026-07-23T14:21
multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 0)	N/A	2026-07-23T15:57
stage-b-test-2-gpu-large-amd (linux-mi325-2gpu-sglang, 1)	Run test	2026-07-23T17:19

Per-Job Analysis

multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 3) — failed step(s): Run diffusion server tests (1-GPU)

Commit Info

• sglang: b13abfde
• aiter: 9127c94a (v0.1.17.dev110+g9127c94a1) — source: dev image preserved ([CI-AITER-CHECK] reported Dev/patched version detected … → skipping rebuild; Dockerfile default was the same SHA)

Failed Tests

Job: multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 3) — 4 failed, 2 passed, 4 errors. All in python/sglang/multimodal_gen/test/single_test_file/test_update_weights_from_disk.py.

Test File	Test Function	Error	Log
test_update_weights_from_disk.py	TestUpdateWeightsFromDisk::test_update_weights_from_disk_default[Qwen-Image]	assert 400 == 200 (server rejected update)	Log
test_update_weights_from_disk.py	TestUpdateWeightsFromDisk::test_update_weights_specific_modules[Qwen-Image]	AssertionError: Update failed: {'success': False, 'message': "Failed to update module 'transformer': …"}	Log
test_update_weights_from_disk.py	TestUpdateWeightsFromDisk::test_corrupted_weights_rollback[Qwen-Image]	assert (400 == 200)	Log
test_update_weights_from_disk.py	TestUpdateWeightsFromDiskWithOffload::test_update_weights_with_offload_enabled[Qwen-Image]	AssertionError: Expected 200, got 400	Log
test_update_weights_from_disk.py	TestUpdateWeightsFromDisk::test_update_weights_from_disk_default[FLUX.2-klein-base-4B]	ERROR at setup: AssertionError: No weights dir for transformer in /tmp/sglang_perturbed_vae_*	Log
test_update_weights_from_disk.py	TestUpdateWeightsFromDisk::test_update_weights_specific_modules[FLUX.2-klein-base-4B]	ERROR at setup: No weights dir for transformer …	Log
test_update_weights_from_disk.py	TestUpdateWeightsFromDisk::test_update_weights_rejects_invalid_requests[FLUX.2-klein-base-4B]	ERROR at setup: No weights dir for transformer …	Log
test_update_weights_from_disk.py	TestUpdateWeightsFromDisk::test_corrupted_weights_rollback[FLUX.2-klein-base-4B]	ERROR at setup: No weights dir for transformer …	Log

Passed: test_update_weights_rejects_invalid_requests[Qwen-Image], test_update_weights_with_offload_enabled[FLUX.2-klein-base-4B].

Failure Summary

The test_update_weights_from_disk.py diffusion suite failed on this AMD MI325 shard with two distinct symptoms on two model pairs. For Qwen-Image, the server rejects POST /update_weights_from_disk with 400 Bad Request and logs Failed to update module 'transformer': . (empty exception message), rolling all modules back → 4 client-side 400 != 200 assertion failures. For FLUX.2-klein-base-4B, the diffusion_server_no_offload fixture errors at setup because the perturbed-VAE model clone has no transformer/ weights directory (No weights dir for transformer in /tmp/sglang_perturbed_vae_*), erroring the 4 tests that use that fixture.

Regression Status

Never-passed within the queryable window for this test file (persistent, ≥8 days). Every one of the 15 most-recent comparable completed scheduled runs (same job, branch=main, event=schedule) failed on test_update_weights_from_disk.py. No green run exists for this test file in the window (lookback: 2026-07-15 → 2026-07-22).

Recent history of test_update_weights_from_disk.py in job multimodal-gen-test-1-gpu-amd (…, 3) (15 comparable completed scheduled runs):

Date	Run	Test File Status	Summary
Jul 22	29920027108	❌ Failed	4 failed / 2 passed / 4 errors
Jul 22	29880220845	❌ Failed	4 failed / 2 passed / 4 errors
Jul 21	29830494225	❌ Failed	4 failed / 2 passed / 4 errors
Jul 21	29790347494	❌ Failed	4 failed / 2 passed / 4 errors
Jul 20	29743214297	❌ Failed	4 failed / 2 passed / 4 errors
Jul 20	29709549202	❌ Failed	5 failed / 1 passed / 4 errors
Jul 19	29686709871	❌ Failed	5 failed / 1 passed / 4 errors
Jul 19	29666971143	❌ Failed	5 failed / 1 passed / 4 errors
Jul 18	29644059344	❌ Failed	5 failed / 1 passed / 4 errors
Jul 18	29623067212	❌ Failed	5 failed / 1 passed / 4 errors
Jul 17	29580198489	❌ Failed	5 failed / 1 passed / 4 errors
Jul 17	29544840981	❌ Failed	5 failed / 1 passed / 4 errors
Jul 16	29498496614	❌ Failed	5 failed / 1 passed / 4 errors
Jul 15	29415344552	❌ Failed	5 failed / 1 passed / 4 errors
Jul 15	29378873430	❌ Failed	5 failed / 1 passed / 4 errors

Note: the mix shifted from 5 failed / 1 passed to 4 failed / 2 passed around 2026-07-20 (one test — rejects_invalid_requests[Qwen-Image] — began passing), but the core transformer-update and FLUX.2-setup failures are unchanged throughout.

Failure Cluster

multimodal_gen update_weights_from_disk — transformer module failures (Qwen-Image server 400 + FLUX.2 missing transformer dir)

Facts (verified observations)

• Qwen-Image server-side ([FACT]): server logs Failed to update module 'transformer': . with an empty exception message, then returns 400 Bad Request on POST /update_weights_from_disk. Occurs on every Qwen update attempt (log lines 5933, 7084, 8237, 8258, 9430, 10583, 12047). All modules are rolled back, so the pipeline stays functional but the update fails.
• FLUX.2 fixture setup ([FACT]): diffusion_server_no_offload fails at test_update_weights_from_disk.py:410 → :220 with AssertionError: No weights dir for transformer in /tmp/sglang_perturbed_vae_*. The perturbed-VAE clone built by _clone_model_with_modified_module contains no transformer/ subdirectory.
• FLUX.2 model layout ([FACT]): earlier warning threads show the same on the real snapshot — No weights dir for transformer/vae in …/models--black-forest-labs--FLUX.2-klein-base-4B/snapshots/…; server load reported transformer: 0.0 GiB memory, i.e. the transformer weights are not in a transformer/ subdir with safetensors as the test's _compute_checksum_from_disk expects (test_update_weights_from_disk.py:219-227).
• Offload path differs ([FACT]): test_update_weights_with_offload_enabled[FLUX.2-klein-base-4B] passed (got 200 OK, log line 11137) while the same test on Qwen-Image failed → the failure is model-pair-specific, not a blanket update-path break.
• Reproducibility ([FACT]): identical failure set across all 15 comparable scheduled runs (Jul 15–22).
• Environment ([FACT]): aiter dev image preserved at 9127c94a; an AITER_USE_SYSTEM_TRITON=1 warning notes triton 3.4.0 < required 3.6.0 for aiter gluon kernels (warning only, not tied to these failures).

Hypothesised Causes (with confidence)

• [MEDIUM] The two symptoms are a model-layout / checkpoint-namespace mismatch in the multimodal_gen update_weights_from_disk path for these specific diffusion model pairs on AMD, not a generic regression.
  • Supporting: FLUX.2 checksum helper asserts on a missing transformer/ subdir both in the real snapshot and the perturbed clone; server reports transformer: 0.0 GiB at load; Qwen-Image transformer update fails server-side with an empty error while VAE/other modules roll back cleanly; offload-enabled FLUX.2 update succeeds (200), so the update machinery itself works. This points at how transformer weights are named/laid out for these checkpoints rather than a broken endpoint.
  • Disconfirming: the empty server exception message means the concrete server-side error type is not visible in the log, so the exact failing operation (weight-name mapping vs. file discovery vs. quant metadata) is unconfirmed; no server stack trace was emitted at --log-level=debug.
  • How to verify: on an AMD MI325 container, run sglang serve --model-path Qwen/Qwen-Image --num-gpus 1 --log-level=debug, POST /update_weights_from_disk for the transformer module, and capture the swallowed exception (patch/inspect the Failed to update module catch site in the multimodal_gen weight-update handler); separately ls the FLUX.2 and Qwen-Image snapshot dirs to confirm whether a transformer/ safetensors subdir exists.
• [LOW] Test-side assumption bug: _compute_checksum_from_disk / _clone_model_with_modified_module assume every _DIFFERING_MODULE (transformer, vae) lives in a same-named subdir, which may not hold for the FLUX.2-klein-base-4B checkpoint layout.
  • Supporting: assertion fires purely on os.path.exists(<dir>/transformer); clone only creates dirs that already exist in the source.
  • Disconfirming: the same helper works for the VAE module and for FLUX.2 offload test; if the layout were simply wrong the offload variant would also fail.
  • How to verify: inspect the on-disk layout of models--black-forest-labs--FLUX.2-klein-base-4B on the runner cache and compare against _DIFFERING_MODULES.
• [LOW] Runner data/cache issue (incomplete FLUX.2 download) rather than code.
  • Supporting: No weights dir is consistent with a missing subdirectory on /sgl-data/hf-cache.
  • Disconfirming: it reproduces identically across 15 runs on presumably different runners, and Qwen-Image fails via a different (server 400) mechanism — a stale single-runner cache would not explain both.
  • How to verify: re-pull both models fresh and re-run.

No hypothesis reaches HIGH — the swallowed empty server error blocks code-level attribution. Because these tests have never passed in the queryable window, this is most likely an AMD-enablement gap for the multimodal_gen weight-update suite rather than a fresh regression; a git bisect has no green anchor to bisect against.

In-flight Fix Check

• In-flight fix: ❌ none found targeting these specific failures. Related open PRs for context (none address the transformer-update 400 / missing-dir symptoms):
  • #31483 "[AMD] ci: run vetted nested multimodal_gen unit tests on AMD" (open since 2026-07-16) — the AMD-enablement PR for this suite; still open.
  • #31899 "[AMD] Add msgpack to ROCm diffusion deps (fix multimodal-gen unit test ModuleNotFound)" (open since 2026-07-21) — different symptom (ModuleNotFound), not this failure.
  • (Search: repo:sgl-project/sglang is:pr is:open update_weights_from_disk / multimodal_gen+update+weights — no PR matches the transformer 400 or missing-transformer/-dir symptoms.)

Suggested Triage Steps

• Reproduce Qwen-Image server-side: run the diffusion server on AMD MI325 and inspect the swallowed exception at the Failed to update module 'transformer': catch site in the multimodal_gen weight-update handler — the empty message is hiding the real error.
• Inspect on-disk layout of both FLUX.2-klein-base-4B and Qwen-Image snapshots vs. the test's _DIFFERING_MODULES (transformer, vae) assumption in test_update_weights_from_disk.py:219-227 / :230-268.
• Confirm whether this suite ever passed on AMD (check the enablement PR #31483 CI history); if it never has, treat as an AMD port gap and gate/quarantine the transformer-update assertions on AMD until the layout/handler issue is fixed rather than bisecting.
• If confirmed model-pair-specific, consider restricting the CI model pair (_CI_MODEL_PAIR_ENV / _CI_MODEL_PAIR_IDS) to the working configuration on AMD while the fix lands.

Status

• Persistence: Failing every completed scheduled run for ≥8 days (since at least 2026-07-15; no green run for this test file in the 15-run window).
• Scope: test_update_weights_from_disk.py in multimodal-gen-test-1-gpu-amd (…, 3) on pr-test-amd.yml; two model pairs (Qwen-Image 400 failures, FLUX.2 setup errors) — 4 failed + 4 errored, 2 passing.
• Blocked work: multimodal_gen (diffusion) update_weights_from_disk CI signal on AMD MI325.
• In-flight fix: ❌ none found targeting these symptoms (see In-flight Fix Check).

multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 0) — failed step(s): N/A

Commit Info

• sglang: b13abfde (schedule, branch main)
• aiter: N/A — the target job hung and its log blob was never uploaded (BlobNotFound), so [CI-AITER-CHECK] markers are unavailable for this run. This is pr-test-amd.yml (not a scout run), so aiter is the Dockerfile default; sibling completed runs show AITER_COMMIT_OVERRIDE= empty.

Failed Tests

Test File	Test Function	Error	Log
python/sglang/multimodal_gen/test/server/test_server_2_gpu.py	TestDiffusionServerTwoGpu::test_diffusion_generation[flux2_modelopt_fp8_tp2_t2i]	Step hung ~90 min → runner killed; no log for this run. Same shard's completed runs fail here with RuntimeError: CUDA error: HIPBLAS_STATUS_NOT_SUPPORTED	Job

Caveat: the target run (Jul 22) produced no log — step 9 "Run diffusion server tests (2-GPU)" started 16:11:34Z, never recorded completion (conclusion None, no end time), and the job was killed at 17:42:07Z (~90 min, under the 150-min step timeout). The failing test above is inferred from three completed runs of the same shard (Jul 16/17/18) whose logs still exist and consistently pin flux2_modelopt_fp8_tp2_t2i. See Regression Status + Facts.

Failure Summary

The multimodal-gen-test-2-gpu-amd part 0 shard failed. In this run it manifested as a hang: the diffusion-server test step ran ~90 minutes without completing and the runner was terminated before any job log was uploaded (BlobNotFound), so no per-test evidence exists for this specific run. Sibling parts 1 and 2 of the same 2-GPU suite passed quickly (~17 min / ~9 min) in the same run, so the problem is localized to part 0's case set. This shard has failed every comparable scheduled run in the queryable window.

Regression Status

Known recurring failure — Never-passed in the 15-run window (no green run found Jul 15–Jul 22).

Recent history of test_server_2_gpu.py shard multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 0) (15 most recent comparable completed scheduled runs: event=schedule, branch main, status completed):

Date	Run	Job	Test File Status	Mode	Error
Jul 22	run	part 0	❌ Failed (this run)	Hang → runner killed (no log)	step9 None, no end
Jul 22	run	part 0	❌ Failed	Hang → runner killed	step9 None, no end
Jul 21	run	part 0	❌ Failed	Hang → runner killed	step9 None, no end
Jul 21	run	part 0	❌ Failed	Hang → runner killed	step9 None, no end
Jul 20	run	part 0	❌ Failed	Hang → runner killed	step9 None, no end
Jul 20	run	part 0	❌ Failed	Hang → runner killed	step9 None, no end
Jul 19	run	part 0	❌ Failed	Hang → runner killed	step9 None
Jul 19	run	part 0	❌ Failed	Hang → runner killed	step9 None, no end
Jul 18	run	part 0	❌ Failed	Hang → runner killed	step9 None
Jul 18	run	part 0	❌ Failed	Assertion/runtime	flux2_modelopt_fp8_tp2_t2i HIPBLAS_STATUS_NOT_SUPPORTED (+ERROR joy_echo_t2v_2gpu)
Jul 17	run	part 0	❌ Failed	Hang → runner killed	step9 None, no end
Jul 17	run	part 0	❌ Failed	Assertion/runtime	flux2_modelopt_fp8_tp2_t2i HIPBLAS (1 failed, 6 passed, 1 skipped, 17 errors)
Jul 16	run	part 0	❌ Failed	Assertion/runtime	flux2_modelopt_fp8_tp2_t2i HIPBLAS_STATUS_NOT_SUPPORTED
Jul 16	run	part 0	⚪ Skipped	step9 skipped (run cancelled)	—
Jul 15	run	part 0	❌ Failed	Hang → runner killed	step9 None

• First observed failure: at or before Jul 15 2026 (oldest run in window; no green run found — failure predates the queryable window).
• Last known passing run for this shard: none in the 15-run window.
• The shard fails in two symptom sub-modes: (a) a clean runtime failure on flux2_modelopt_fp8_tp2_t2i (verified in the 3 runs whose logs survive), and (b) a hang that kills the runner before the log uploads (this run + 10 others). The hang mode leaves no log, so its exact failing case is unverified.

Failure Cluster

FLUX.2 modelopt-FP8 GEMM HIPBLAS_STATUS_NOT_SUPPORTED on gfx942 (2-GPU part 0) — with a secondary manifestation as a long hang / runner loss on the same shard.

Facts (verified observations)

• Target job 88922971070: step 9 "Run diffusion server tests (2-GPU)" started 2026-07-22T16:11:34Z, conclusion None, no completion timestamp; job killed 2026-07-22T17:42:07Z (~90 min, below the 150-min step timeout). Job log = BlobNotFound; no failure artifacts (step 10 never ran). [FACT]
• Same-run sibling shards passed: 2-GPU part 1 [job 88922971080] ✅ (~17 min), 2-GPU part 2 [job 88922971054] ✅ (~9 min). The failure is isolated to part 0's case set. [FACT]
• In three completed runs of this exact shard (Jul 16 [job 87621199565], Jul 17 [job 88102981818], Jul 18 [job 88021852948]), the first case flux2_modelopt_fp8_tp2_t2i fails during denoising warmup with RuntimeError: CUDA error: HIPBLAS_STATUS_NOT_SUPPORTED when calling HIPBLAS_STATUS_NOT_SUPPORTED. Jul 18 summary: FAILED ...[flux2_modelopt_fp8_tp2_t2i] + ERROR ...[joy_echo_t2v_2gpu]. Jul 17 summary: 1 failed, 6 passed, 1 skipped, 7 warnings, 17 errors in 5443.52s (1:30:43). [FACT]
• The case is defined at python/sglang/multimodal_gen/test/server/gpu_cases.py:595-607: FLUX.2-dev with a ModelOpt FP8 transformer (--transformer-path MODELOPT_FLUX2_FP8_TRANSFORMER), tp_size=2. The failing GEMM is the FP8 matmul path. [FACT]
• Container is rocm/sgl-dev:...-rocm700-mi30x, runner linux-mi325-2gpu-sglang (gfx942). [FACT]
• In the same Jul 22 run, 1-GPU part 0 [job 88922971025] and 1-GPU part 1 [job 88922971082] also hit the identical "step9 None, no end, no log" pattern; 1-GPU part 3 [job 88922971051] had a normal step failure. [FACT]

Hypothesised Causes (with confidence + disconfirming evidence)

• [HIGH] The shard's reproducible root failure is FP8 GEMM unsupported on gfx942 hipBLAS for the FLUX.2 modelopt-FP8 case (flux2_modelopt_fp8_tp2_t2i).
  • Supporting: three surviving logs (Jul 16/17/18) all pin this exact case to HIPBLAS_STATUS_NOT_SUPPORTED on the FP8 denoise matmul; the case explicitly loads an FP8 transformer; an open PR (#28889) titled "Use native e4m3fnuz fp8 GEMM for FLUX.2 modelopt-FP8 on gfx942" independently confirms the FP8-GEMM-on-gfx942 gap.
  • Disconfirming: the target Jul 22 run itself has no log, so I cannot prove this run failed on flux2 rather than hanging on a different case; and the parallel 1-GPU part 0/1 hangs in the same run suggest a broader runner/infra hang was also in play Jul 22.
  • How to verify: re-run part 0 with bypass-fastfail/rerun so a log uploads; grep for HIPBLAS_STATUS_NOT_SUPPORTED and flux2_modelopt_fp8_tp2_t2i.
• [MEDIUM] The hang → runner-kill sub-mode is the FP8 failure wedging the ROCm device (unrecoverable hipBLAS error mid-warmup) so subsequent cases block until the runner reclaims the job.
  • Supporting: hang mode is confined to the same part-0 shard that reproducibly errors on the FP8 case; the kill happens well before the 150-min step timeout, consistent with a device wedge rather than a slow test.
  • Disconfirming: 1-GPU parts 0/1 (different case sets, no FP8 flux2) also hung identically in the Jul 22 run — points to a run-wide runner/infra issue independent of the FP8 case; and the assertion-mode runs recovered and continued past flux2, showing the device is not always wedged.
  • How to verify: correlate hang runs against runner health / VRAM-clear logs; check whether 1-GPU hangs share the same physical runner as 2-GPU part 0.
• [LOW] Pure GitHub/runner infrastructure loss (job evicted), unrelated to the test.
  • Supporting: BlobNotFound + None step conclusion + kill before timeout is a classic runner-loss signature.
  • Disconfirming: the failure recurs on the same shard far more than random infra loss would predict (never-passed in 15 runs), and that shard has a known reproducible FP8 error.

In-flight Fix Check

• In-flight fix: ✅ #28889 "[AMD] Use native e4m3fnuz fp8 GEMM for FLUX.2 modelopt-FP8 on gfx942" — open since 2026-06-22, updated 2026-07-21, mergeable_state: blocked. Directly targets the FLUX.2 modelopt-FP8 GEMM path on gfx942. Chase reviewers / unblock; do not open a duplicate. (Searched flux2 fp8, HIPBLAS_STATUS_NOT_SUPPORTED, flux2 modelopt in open PRs.)

Suggested Triage Steps

• Prioritize landing / unblocking #28889; it addresses the verified flux2_modelopt_fp8_tp2_t2i HIPBLAS failure that anchors this shard.
• Re-run multimodal-gen-test-2-gpu-amd (part 0) on a healthy MI325 runner to capture a fresh log (the last several runs uploaded none) and confirm whether Jul 22's hang is the same FP8 case wedging the device or a separate runner-loss event.
• If #28889 cannot land soon, consider temporarily skipping/quarantining flux2_modelopt_fp8_tp2_t2i on AMD gfx942 so the rest of part 0 (which passes: ideogram4, wan2_2/wan2_1, mova, ltx_2, fsdp-inference) produces a usable green signal.
• Investigate the Jul 22 run-wide hang separately: 1-GPU parts 0/1 also hung with no log in the same run — check runner health/VRAM-clear and whether a wedged device is failing to release the job before the step timeout.

Status (factual)

• Persistence: Failing every completed scheduled run in the window (Jul 15–Jul 22, 8 days); never-passed. 14/15 runs failed, 1 skipped (cancelled run); 0 passing.
• Scope: multimodal-gen-test-2-gpu-amd part 0 in pr-test-amd.yml. Sibling parts 1/2 pass. The flux2_modelopt_fp8_tp2_t2i FP8-on-gfx942 signature is the reproducible anchor; the hang sub-mode also coincided with unrelated 1-GPU part 0/1 hangs in the Jul 22 run.
• Blocked work: Blocks the AMD 2-GPU diffusion (FLUX.2 modelopt-FP8) CI signal.
• In-flight fix: ✅ #28889 (open since 2026-06-22, blocked) — unblock/merge rather than duplicate.

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-07-23 18:35 UTC)

[amd-bot]

pr-test-amd.yml — Per-Job Analysis (continued 2/6)

multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 0) — failed step(s): N/A

Commit Info

• sglang: b13abfdedf — "Fix LongCat n-gram token-table crashes on padded batches (#31312)"
• aiter: 9127c94a1 — v0.1.17.dev110+g9127c94a1 (source: dev image preserved — Dev/patched version detected → skipping rebuild; AITER_COMMIT_OVERRIDE empty). Read from sibling shard 2's [CI-AITER-CHECK] markers — shard 0's own log was purged (see below).

Failed Tests

Test File	Test Function	Error	Log
N/A (whole shard timed out)	N/A	Step 9 "Run diffusion server tests (1-GPU)" started 14:01:07Z, never received a conclusion (None), job killed at 15:19:16Z (~78 min into the step) with job conclusion failure. No step-9 log was uploaded.	Job

This is not a test-assertion failure — it is a shard-level hang/timeout. The specific failing test function is not recoverable: timeout-killed shards on this job never upload their step-9 log (the raw blob returns BlobNotFound, and the run-logs zip contains only the completed shards 2 & 3, never the timed-out shards 0 & 1 — confirmed on both this run and the prior run 29880220845).

Failure Summary

Shard 0 of multimodal-gen-test-1-gpu-amd runs the LPT-partitioned parametrized diffusion cases from test/server/test_server_1_gpu.py (ONE_GPU_CASES, -k "not flux_2"). The shard hung for ~78 minutes and was externally killed before step 9 completed, so the job failed with no usable test-level log. Sibling shard 2 passed (20 min); sibling shard 3 completed with a separate, unrelated real failure in test_update_weights_from_disk.py; sibling shard 1 shows the same timeout signature as shard 0.

Regression Status

Known recurring failure / never-passed in the query window. Shard 0 has failed on every completed scheduled run in the last 15, all with the same "step 9 never completes" signature and long durations (74–122 min).

Recent history of multimodal-gen-test-1-gpu-amd (…, 0) shard (most recent 15 comparable completed scheduled runs: branch=main, event=schedule, completed only). Test-file granularity is unavailable because timeout-killed shards upload no step log — status shown at the step-9 level:

Date (UTC)	Run	Shard-0 Status	Step-9 concl.	Job dur
2026-07-22	29920027108	❌ Timed out (no log)	None	85.6 min
2026-07-22	29880220845	❌ Timed out (no log)	None	88.0 min
2026-07-21	29830494225	❌ Timed out (no log)	None	122.2 min
2026-07-21	29790347494	❌ Timed out (no log)	None	99.1 min
2026-07-20	29743214297	❌ Timed out (no log)	None	107.5 min
2026-07-20	29709549202	❌ Timed out (no log)	None	89.1 min
2026-07-19	29686709871	❌ Failed (no step-9 marker)	n/a	82.1 min
2026-07-19	29666971143	❌ Timed out (no log)	None	80.1 min
2026-07-18	29644059344	❌ Timed out (no log)	None	74.0 min
2026-07-18	29623067212	❌ Timed out (no log)	None	91.1 min
2026-07-17	29580198489	❌ Timed out (no log)	None	85.1 min
2026-07-17	29544840981	❌ Failed (no step-9 marker)	n/a	73.0 min
2026-07-16	29498496614	❌ Timed out (no log)	None	75.1 min
2026-07-15	29415344552	❌ Failed early (~10 min, infra)	n/a	10.0 min
2026-07-15	29378873430	❌ Failed (no step-9 marker)	n/a	79.0 min

First observed in window: 2026-07-15. No passing shard-0 run exists in the 15-run window (lookback 2026-07-15 → 2026-07-22).

Failure Cluster

multimodal-gen 1-GPU diffusion shard hang/timeout (step 9 never completes, no log uploaded)

Facts (verified observations)

• Step 9 "Run diffusion server tests (1-GPU)" started 2026-07-22T14:01:07Z; job ended 2026-07-22T15:19:16Z (~78 min later) with step-9 conclusion None and job conclusion failure. Setup steps 1–8 all succeeded (container start 4.8 min, deps 2.5 min).
• Shard 0's raw job-log blob returns BlobNotFound; the run-logs zip for both this run and the prior run contains only the completed shards (2 & 3), never the timed-out shards (0 & 1). Timeout-killed shards do not upload step logs.
• Same-run sibling shards: shard 2 ✅ passed (20.6 min, ran test_generate_zimage_turbo_cli.py); shard 1 ❌ same timeout signature (step9=None, 77.1 min); shard 3 ❌ different real failure — test/single_test_file/test_update_weights_from_disk.py::TestUpdateWeightsFromDisk (assert 400 == 200, "No weights dir for transformer") — NOT shard 0's failure.
• Shard 0 durations across the window are highly variable (74–122 min), inconsistent with a clean 90-min step-timeout cut.
• runner/pytest_runner.py:100 _run_pytest_attempt reads the pytest subprocess stdout in a blocking loop with no per-attempt timeout; run_pytest (line 286) retries "retryable" failures (OOM / TimeoutError / SafetensorError / FileNotFoundError) up to 7 attempts total. A hung subprocess blocks indefinitely, bounded only by the GH step timeout / external kill.
• Not a scout run — AITER_COMMIT_OVERRIDE empty; aiter is the Dockerfile/dev-image default, unchanged vs the passing shard 2.

Hypothesised Causes (with confidence)

• [LOW] A parametrized diffusion case assigned to shard 0 (ONE_GPU_CASES in test/server/test_server_1_gpu.py) hangs — or repeatedly trips a retryable failure — driving the pytest_runner retry loop until the shard is externally killed.
  • Supporting: shards 0 & 1 consistently run 74–122 min and die with step9=None while shards 2 (20 min) and 3 (32 min) complete; the LPT partitioner concentrates the slow/heavy cases into shards 0–1; the retry path has no per-attempt subprocess timeout.
  • Disconfirming: shard 0's log is unrecoverable, so the specific hanging case is unverified; variable duration (74–122 min) argues against a single fixed cause and points at external/runner kill rather than a deterministic per-case hang.
  • How to verify: re-run shard 0 with a shortened step timeout-minutes (so the log flushes/uploads before the kill), or run ONE_GPU_CASES individually / bisect via -k to find which case hangs.
• [LOW] Partition/time-budget imbalance: shard 0's assigned cases legitimately exceed the wall-clock budget, especially once multiplied by up to 7 retry attempts per case.
  • Supporting: heavy models in the 1-gpu suite (Qwen-Image, FLUX.1, cosmos3, z-image variants) plus the retry multiplier.
  • Disconfirming: same log-unavailability caveat; does not explain why shard 2 (also 1-gpu) finishes in 20 min unless LPT is badly skewed.

No hypothesis reaches MEDIUM confidence — the shard-0 step-9 log is not recoverable for any run in the window. Needs a re-run that captures the log before the kill.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for multimodal-gen+timeout, diffusion+1-gpu+timeout, multimodal_gen+partition, diffusion+hang. The closest match, #29584 ("ci(amd): fix three pre-existing MI325 CI failures"), targets stage-b bench-serving tests, not this diffusion shard timeout.

Suggested Triage Steps

• Re-run shard 0 with a reduced step timeout-minutes (e.g. 60) or an explicit per-case timeout so step-9 output flushes and the failure artifact uploads before the runner kill — this is the prerequisite to any real diagnosis, since every historical run lost its log.
• Bisect the shard-0 assignment: run ONE_GPU_CASES from test/server/test_server_1_gpu.py one case at a time on an MI325 runner to identify the hanging/OOM case.
• Add a per-attempt subprocess timeout to runner/pytest_runner.py:100 (_run_pytest_attempt) — the current blocking stdout.read() loop lets a single hung server stall the whole shard indefinitely, which both causes and masks this failure.
• Check MI325 runner health (GPU state / OOM-killer) during the 74–122 min windows; inspect the diffusion-failures-amd-1gpu-0-* upload artifact if present.
• Verify the LPT partition balance — if shards 0 & 1 systematically get the heavy cases, rebalance STANDALONE_FILE_EST_TIMES / case est-times so no shard exceeds the budget.

Status

• Persistence: Failing on every completed scheduled run in the 15-run window (since at least 2026-07-15, ~8 days), always the "step 9 never completes" timeout signature; no passing shard-0 run in that window.
• Scope: Shards 0 and 1 of multimodal-gen-test-1-gpu-amd both time out this run; shard 3 has a separate real failure (test_update_weights_from_disk.py); shard 2 passes. Blocks the AMD 1-GPU diffusion CI signal.
• In-flight fix: ❌ none found — needs a log-capturing re-run + investigation (and likely a per-attempt timeout in the pytest runner).

stage-c-test-large-8-gpu-amd-mi35x (linux-mi35x-gpu-8, 1) — failed step(s): Run test

Commit Info

• sglang: b13abfde
• aiter: N/A (no [CI-AITER-CHECK] markers in this pr-test-amd job log; AITER_COMMIT_OVERRIDE empty; image rocm/sgl-dev:v0.5.15.post1-rocm700-mi35x-20260721)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/test_deepseek_r1_mxfp4_8gpu.py	TestDeepseekR1MXFP4.setUpClass	Exception: Server process exited with code -9 → underlying RuntimeError: Capture prefill CUDA graph failed: Unsupported method call	Log

The other two files in this job (test_kimi_k25_mxfp4.py, test_qwen3_coder_next_8gpu.py) passed; the job failed solely on the DeepSeek-R1-MXFP4 8-GPU test.

Failure Summary

The DeepSeek-R1-MXFP4 8-GPU server (sglang serve --model-path amd/DeepSeek-R1-MXFP4-Preview --tp 8 ... --enforce-piecewise-cuda-graph --piecewise-cuda-graph-compiler eager) crashed during startup while capturing the prefill piecewise CUDA graph. TorchDynamo could not trace a __getattribute__ call, the scheduler raised RuntimeError: Capture prefill CUDA graph failed, the child process sent SIGQUIT and the server exited (-9), so setUpClass never brought the server up and Ran 0 tests.

Regression Status

New regression (first failing run today, 2026-07-22).

Recent history of test/registered/amd/test_deepseek_r1_mxfp4_8gpu.py in job stage-c-test-large-8-gpu-amd-mi35x (linux-mi35x-gpu-8, 1) (pr-test-amd.yml, event=schedule, branch=main, completed runs):

Date (UTC)	Run	Job	Test File Status	Failed Function	Error
Jul 22 12:32	29920027108	job	❌ Failed	setUpClass	prefill CUDA graph capture / Dynamo __getattribute__
Jul 22 00:26	29880220845	job	✅ Passed	—	—
Jul 21 12:32	29830494225	job	✅ Passed	—	—

Last passing run: 29880220845 (2026-07-22 00:26 UTC, 8ae0eb83). First failing: current run (12:32 UTC). Regression window 8ae0eb83...b13abfde = 34 commits.

Failure Cluster

Dynamo graph break (__getattribute__ untraceable) during prefill piecewise CUDA graph capture

Facts (verified observations)

• Verbatim error at prefill_cuda_graph_runner.py:277: RuntimeError: Capture prefill CUDA graph failed: Unsupported method call — Dynamo does not know how to trace method __getattribute__ of class type.
• The Dynamo from user code frame resolves to, in order: models/deepseek_v2.py:2231 → layers/communicator_dsa_cp.py:63 maybe_prefetch_next_full_attention_kv → layers/attention/dsa/utils.py:277 dsa_use_prefill_cp → layers/attention/dsa/utils.py:79 is_dsa_enable_prefill_cp → runtime_context.py:130 ParallelContext.__getattr__ → object.__getattribute__(self, "_config").
• runtime_context.py:126-140: ParallelContext.__getattr__ serves config leaves via object.__getattribute__(self, "_config") — the exact call Dynamo rejects.
• Failure occurs under --enforce-piecewise-cuda-graph --piecewise-cuda-graph-compiler eager (tc_piecewise prefill backend), i.e. only on the torch.compile prefill path.
• Job passed in the two prior scheduled runs; this is the first red run.
• Not a scout workflow; aiter override empty, so aiter is not implicated.

Hypothesised Causes

• [HIGH] Commit 745b2ca4 (#31816, "config: read parallel config leaves via get_parallel()") changed is_dsa_enable_prefill_cp() from get_server_args().enable_dsa_prefill_context_parallel to get_parallel().enable_dsa_prefill_context_parallel. get_parallel() returns a ParallelContext whose config leaves resolve through __getattr__ → object.__getattribute__(self, "_config") (runtime_context.py:130), which TorchDynamo cannot trace, breaking prefill CUDA graph compilation.
  • Supporting: modifies the EXACT failing line dsa/utils.py:79 present in the Dynamo stack trace; lands inside the regression window; commit merged 2026-07-22 08:18 UTC — after the last green run (00:26 UTC) and before the first red run (12:32 UTC); the stack trace terminates precisely at object.__getattribute__(self, "_config"); the old get_server_args() path was a plain attribute read (traceable), the new get_parallel() path routes through a custom __getattr__ (not traceable).
  • Disconfirming: (none found) — no green run exists after the commit landed; the code-path and timing evidence both point the same way.
  • How to verify: on a branch, revert 745b2ca4 (or locally restore is_dsa_enable_prefill_cp to read get_server_args()), re-run test/registered/amd/test_deepseek_r1_mxfp4_8gpu.py on MI35x 8-GPU; alternatively add the server flag --cuda-graph-backend-prefill=disabled to confirm the crash is compile-path-specific.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for get_parallel+dynamo, ParallelContext+compile, enable_dsa_prefill_context_parallel, runtime_context+compile, getattribute+dynamo, 31816 (all 0 hits; piecewise+cuda+graph+getattr returned only unrelated PRs).

Suggested Triage Steps

• Bisect / inspect the 34-commit window 8ae0eb83...b13abfde, starting with 745b2ca4 (#31816) — the only commit touching the failing dsa/utils.py:79 path.
• Revert 745b2ca4 on a branch and re-run test_deepseek_r1_mxfp4_8gpu.py on MI35x 8-GPU to A/B confirm.
• If confirmed, fix direction: make ParallelContext config-leaf access Dynamo-traceable (avoid object.__getattribute__ in the hot __getattr__ path — e.g. read the flag outside compile / hoist it to a plain attribute), rather than reverting the broader config refactor.
• Interim workaround to unblock CI signal only: launch this test with --cuda-graph-backend-prefill=disabled or --cuda-graph-backend-prefill=breakable (per the server's own suggestion) — not a fix.

Status

• Persistence: First failing run 2026-07-22 12:32 UTC (1 completed run); passed in the prior 2 scheduled runs.
• Scope: test_deepseek_r1_mxfp4_8gpu.py in stage-c-test-large-8-gpu-amd-mi35x (linux-mi35x-gpu-8, 1); other tests in the job pass. Likely also affects any DSA/DeepSeek-V2 model exercising the piecewise-CUDA-graph prefill path via get_parallel().
• Blocked work: DeepSeek-R1-MXFP4 8-GPU AMD CI signal.
• In-flight fix: ❌ none found — needs new PR or investigation.

stage-c-test-large-8-gpu-amd (linux-mi325-8gpu-sglang, 3) — failed step(s): Run test

Commit Info

• sglang: b13abfde
• aiter: 9127c94a v0.1.17.dev110+g9127c94a1 (source: dev image preserved — [CI-AITER-CHECK] Dev/patched version detected → skipping rebuild). Not implicated in this failure.

Failed Tests

Test File	Test Function	Error	Log
test/registered/ops/test_aiter_allreduce_fusion_amd.py	TestAiterAllreduceFusionGate.test_dense_tp_fuses	AttributeError: <module 'sglang.srt.layers.communicator'> does not have the attribute 'get_server_args'	Log

Module summary: Ran 4 tests … FAILED (errors=1) → 3/4 passed, 1 error. The other 3 tests in the run (test_deepseek_v3_mtp.py, test_deepseek_v3_basic.py, test_aiter_allgather_amd.py) passed.

Failure Summary

test_dense_tp_fuses calls _evaluate_gate, which does mock.patch.object(comm, "get_server_args", …) at test line 421. At run commit b13abfde, communicator.py imports get_exec, get_forward, get_parallel, get_spec from runtime_context — it no longer has a module-level get_server_args, so mock.patch.object raises AttributeError on __enter__. The CustomTestCase retry wrapper then reports retry() exceed maximum number of retries.

Regression Status

New regression. Introduced in the window 8ae0eb83..b13abfde.

Recent history of test_aiter_allreduce_fusion_amd.py (test_dense_tp_fuses) in job stage-c-test-large-8-gpu-amd (linux-mi325-8gpu-sglang, 3) (pr-test-amd.yml, event=schedule, branch=main, completed runs):

Date (UTC)	Run	head_sha	Test File Status	Failed Function	Error
2026-07-22 12:32	29920027108	b13abfde	❌ Failed (3/4)	test_dense_tp_fuses	AttributeError: get_server_args
2026-07-22 00:26	29880220845	8ae0eb83	✅ Passed (4/4)	—	—
(earlier runs)	—	pre-migration	✅ Passed	—	get_server_args present in communicator.py before 745b2ca4

First observed failing: 2026-07-22 (run 29920027108). Last known passing: 2026-07-22 00:26 UTC (run 29880220845, verified at test-file level — Test Summary: 4/4 passed). The next scheduled run 29969145884 is queued on 24da0e51, which includes the fix (see below), so it is expected to pass.

Failure Cluster

AttributeError: communicator has no get_server_args (mock.patch.object target removed by RuntimeContext migration)

Facts (verified observations)

• Log (verbatim): AttributeError: <module 'sglang.srt.layers.communicator' from '/sglang-checkout/python/sglang/srt/layers/communicator.py'> does not have the attribute 'get_server_args', raised inside mock.patch.object get_original(); wrapped by retry() exceed maximum number of retries. log.
• At b13abfde, communicator.py line 75 imports get_exec, get_forward, get_parallel, get_spec — no get_server_args (verified via git show b13abfde:…/communicator.py).
• The test still patches get_server_args at b13abfde (git show b13abfde:test/…/test_aiter_allreduce_fusion_amd.py line 421).
• Commit 745b2ca4 (#31816, 2026-07-22 08:18:50 UTC) removed the get_server_args import and all its usages from communicator.py; it is an ancestor of b13abfde but not of 8ae0eb83 (the last passing run) — verified via git merge-base --is-ancestor.
• Same job passed 4/4 in the immediately-prior scheduled run before the migration landed.

Hypothesised Causes

• [HIGH] Commit 745b2ca4 (#31816 "config: read parallel config leaves via get_parallel()") — part of the RuntimeContext config-namespace migration #31813–#31817 — removed get_server_args from communicator.py's namespace, but test_aiter_allreduce_fusion_amd.py was not updated and still mock.patch.object(comm, "get_server_args", …).
  • Supporting: import removed by exactly this commit (diff verified); commit is in the failing SHA but not the last-passing SHA; error message names the exact missing attribute; sister-run A/B shows the same job passed 4/4 immediately before the migration.
  • Disconfirming: (none found — direct code + timing + A/B all converge).
  • How to verify: already confirmed by the merged revert #32100 restoring the attribute (below).

In-flight Fix Check

• Fix already MERGED: #32100 (f5dcbe8f "Revert RuntimeContext config-namespace reads/roles (#31813–#31817)", 2026-07-22 18:52 UTC) restores from sglang.srt.runtime_context import get_forward, get_parallel, get_server_args in communicator.py. It is an ancestor of current main 24da0e51 but not of the run's b13abfde — so this run predates the fix. No new PR needed. (Search API for open PRs is unnecessary since the revert is already on main.)

Suggested Triage Steps

• Confirm the next scheduled pr-test-amd.yml run on 24da0e51+ (e.g. 29969145884) shows test_aiter_allreduce_fusion_amd.py back to 4/4 — closes this regression.
• If the RuntimeContext config-namespace migration (#31813–#31817) is re-landed later, update test_aiter_allreduce_fusion_amd.py::_evaluate_gate to patch the new accessor (get_exec / get_parallel) instead of get_server_args, so the test tracks the source API.

Status

• Persistence: Single completed run (2026-07-22 12:32 UTC scheduled run). Introduced by the migration ~08:18 UTC and already reverted ~18:52 UTC the same day.
• Scope: test_dense_tp_fuses in this 8-GPU AMD job on pr-test-amd.yml. Any job running this test on a commit in 745b2ca4..f5dcbe8f is affected.
• In-flight fix: ✅ Merged as #32100 — regression self-resolves on 24da0e51+.

multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 1) — failed step(s): N/A

Commit Info

• sglang: b13abfde — "Fix LongCat n-gram token-table crashes on padded batches" (#31312); touches only srt/layers/n_gram_embedding.py + ngram_embedding_manager.py — not multimodal_gen/diffusion.
• aiter: N/A — this job's log blob is expired (BlobNotFound) and it is absent from the run-logs zip, so [CI-AITER-CHECK] markers cannot be read. Regular pr-test-amd uses the Dockerfile-default aiter; sibling shards pulled image rocm/sgl-dev:v0.5.15.post1-rocm700-mi30x-20260721.

Failed Tests

Test File	Test Function	Error	Log
python/sglang/multimodal_gen/test/server/test_server_1_gpu.py	(parametrized diffusion cases — LPT subset for partition 1; specific case not recoverable)	Step hung ~69 min, force-killed (step 9 in_progress → None); no clean pytest result	Job

Note: this partition's own log is unavailable. The failing test file is established structurally: for the 1-gpu suite, --total-partitions 4 with 2 standalone files (test_generate_zimage_turbo_cli.py, test_update_weights_from_disk.py) → partitions 2 & 3 are standalone; partitions 0 and 1 run the parametrized cases from test_server_1_gpu.py (T2V/T2I/I2V/LoRA). Verified from the sibling part-2 log (Partition: 3/4 (standalone), 2 passed) and part-3 log (test_update_weights_from_disk.py).

Failure Summary

The multimodal-gen-test-1-gpu-amd partition-1 shard hangs inside step 9 ("Run diffusion server tests (1-GPU)") while running the parametrized test_server_1_gpu.py diffusion generation cases. Step 9 started 14:09:41 and the job was force-terminated at 15:18:33 (~68.8 min) with the step never reaching a conclusion. No diffusion-failure artifacts were produced because the job died before the upload step ran.

Regression Status

Never-passed (no green run found in the 15-run window; pre-existing persistent hang — NOT caused by b13abfde).

Recent history of test_server_1_gpu.py (partition-1 parametrized cases) in job multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 1) (most recent 15 comparable completed scheduled runs: branch=main, event=schedule, completed only):

Date (UTC)	Run	Job	Test File Status	Notes
Jul 22 12:32	29920027108	part 1	❌ Hang (killed ~69m)	current run
Jul 22 00:26	29880220845	part 1	❌ Hang (killed ~74m)	step9=None
Jul 21 12:32	29830494225	part 1	❌ Hang (killed ~83m)	step9=None
Jul 21 00:27	29790347494	part 1	❌ Hang (killed ~76m)	step9=None
Jul 20 12:42	29743214297	part 1	❌ Hang	step9=None
Jul 20 00:28	29709549202	part 1	❌ Hang	step9=None
Jul 19 12:17	29686709871	part 1	❌ Hang	step9=None
Jul 19 00:27	29666971143	part 1	❌ Hang	step9=None
Jul 18 12:17	29644059344	part 1	❌ Hang	step9=None
Jul 18 00:25	29623067212	part 1	❌ Hang	step9=None
Jul 17 12:25	29580198489	part 1	❌ Hang	step9=None
Jul 17 00:27	29544840981	part 1	❌ Hang	step9=None
Jul 16 12:31	29498496614	part 1	❌ Hang	step9=None
Jul 16 00:25	29461421122	part 1	⏭️ N/A	run cancelled, step9 skipped — not comparable
Jul 15 12:28	29415344552	part 1	❌ Hang	step9=None
Jul 15 00:22	29378873430	part 1	❌ Hang	step9=None

First observed failure in window: Jul 15 00:22 UTC (earliest queried). Last known passing run: none in the 15-run lookback (Jul 15–22). Sibling shard part 0 (the other parametrized shard) exhibits the identical hang the same runs; parametrized-shard part 2 and standalone-shard part 3 complete.

Failure Cluster

Diffusion 1-GPU server test hangs during parametrized generation (test_server_1_gpu.py), force-killed before step timeout.

Facts (verified observations)

• Step 9 "Run diffusion server tests (1-GPU)" started 2026-07-22T14:09:41Z; job force-terminated 2026-07-22T15:18:33Z (~68.8 min). Step 9 conclusion = None (never finalized) → abnormal kill, not a clean pytest exit. Upload diffusion failure artifacts (step 10) never ran; run has 0 artifacts.
• The 90-min step timeout (timeout-minutes: 90, workflow line 751) was not reached; historical hang durations vary 69–83 min, so the kill is not the step's own timeout.
• Calibration: sibling part-3 (which completed) shows step 9 conclusion=failure with a real pytest summary (4 failed, 2 passed … 4 errors in 1010.86s) — confirming None specifically means the shard was killed mid-run, not a normal failure.
• Sibling part-0 (other parametrized shard) hung identically: step 9 in_progress → None, killed 15:19:16 — i.e. parts 0 and 1 both died within ~43 s of each other despite starting step 9 ~8.5 min apart.
• Structural mapping verified: 1-gpu STANDALONE_FILES = test_generate_zimage_turbo_cli.py, test_update_weights_from_disk.py → partitions 2 (2 passed) & 3 (weights assertions). Partitions 0/1 = parametrized test_server_1_gpu.py.
• Head commit b13abfde modifies only LLM n-gram embedding code; git show --name-only shows no multimodal_gen/diffusion files.
• This job's per-job log blob is expired (BlobNotFound) and is not present in the run-logs zip; the exact parametrized case that hangs cannot be recovered for this run.

Hypothesised Causes (with confidence)

• [MEDIUM] The hang is a pre-existing, environment/infra-level issue on the MI325 1-GPU runners specific to the parametrized test_server_1_gpu.py shards, not a code regression.
  • Supporting: identical hang in every completed scheduled run for 8+ days across many unrelated head SHAs; both parametrized shards (0 & 1) hang while standalone shards (2 & 3) complete; kill happens at varying wall-times (69–83 min) below the step timeout, and parts 0/1 die near-simultaneously — consistent with an external watchdog/runner reclaim or a shared-node GPU stall rather than a per-test assertion.
  • Disconfirming: without part-0/1 logs the exact stall point (model download stall, server never healthy, GPU/kernel hang) is unconfirmed; a genuinely hanging single case inside test_server_1_gpu.py would also fit and would be a code/test issue, not infra.
  • How to verify: re-run multimodal-gen-test-1-gpu-amd part 1 with log capture; add a per-case timeout inside run_suite.py/pytest so a stuck case produces a stack dump instead of a silent kill; check the runner for a co-scheduling/GPU-reset event around the kill timestamps.
• [LOW] A specific parametrized model case (a T2V/I2V video model or LoRA case assigned by LPT to partitions 0/1) deadlocks during server startup or denoise on ROCm/MI325.
  • Supporting: only the parametrized shards hang; part-2's parametrized-adjacent standalone (test_generate_zimage_turbo_cli) passes in 302 s, so lighter cases work.
  • Disconfirming: cannot name the case without the shard log; LPT assignment isn't verified for this run.
  • How to verify: run test_server_1_gpu.py case-by-case on MI325 to find which parametrization stalls.
• Head commit b13abfde: excluded — unrelated code path (LLM n-gram), and the failure predates it by 8+ days. No hypothesis above MEDIUM; the shard log is required for a definitive root cause.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for multimodal-gen+amd (40 hits, all about unit tests / deps like #31899, #31846 — none address the test_server_1_gpu.py server-test hang), diffusion+hang+amd (3 unrelated), and test_server_1_gpu (0 hits).

Suggested Triage Steps

• Re-run part 1 with ACTIONS_STEP_DEBUG / preserved logs, since the current shard log expired — first priority is to capture where it hangs.
• Add a wall-clock per-case timeout + py-spy dump / stack trace hook inside the diffusion server test harness so a stuck case fails loudly instead of being silently killed at ~70 min (silent kills currently produce no artifacts, defeating diagnosis).
• Bisect the parametrized cases: run test_server_1_gpu.py case-by-case on an MI325 1-GPU container (rocm/sgl-dev:v0.5.15.post1-rocm700-mi30x-20260721) to isolate the hanging model/scenario in partitions 0/1.
• Investigate the near-simultaneous kill of parts 0 & 1: check whether these shards co-schedule on the same physical MI325 node and whether a GPU driver reset / runner reclaim event lines up with 15:18–15:19 UTC.
• Do not pursue b13abfde — it is unrelated and the failure predates it.

Status

• Persistence: Failing every completed scheduled run for ≥8 days (since at least 2026-07-15 00:22 UTC; no green run in the 15-run window). Never-passed in queryable history.
• Scope: Both parametrized 1-GPU shards (part 0 and part 1) of multimodal-gen-test-1-gpu-amd in pr-test-amd.yml. Standalone shards (part 2 pass, part 3 real assertion failures) are separate.
• Blocked work: No PR-CI signal for AMD 1-GPU diffusion server generation (test_server_1_gpu.py) — green is unattainable for these shards.
• In-flight fix: ❌ none found in open PRs.

multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 1) — failed step(s): N/A

Commit Info

• sglang: 8ae0eb83 — "[Perf] Fold dspark dense draft embedding into the draft graph via forward_embed" (#31985)
• aiter: N/A — the job log is purged (BlobNotFound), so the [CI-AITER-CHECK] markers cannot be read.

Failed Tests

Test File	Test Function	Error	Log
unavailable	unavailable	Non-test-level: step "Run diffusion server tests (1-GPU)" (partition 1/4) exited non-zero after ~74 min	Job

⚠️ Test-level granularity is not recoverable for this run. The GitHub Actions job log returns BlobNotFound (purged), and the run has 0 artifacts (the diffusion-failures-amd-1gpu-1-* upload has also expired). I confirmed the log is gone even for the 12 h-newer run 29920027108. Per CI-Bot rules I will not fabricate a test file/function I cannot verify.

Failure Summary

The multimodal-gen-test-1-gpu-amd (part 1/4) job failed at step 9, "Run diffusion server tests (1-GPU)" (sglang/multimodal_gen/test/run_suite.py --suite 1-gpu --partition-id 1 --total-partitions 4 -k "not flux_2"). The step started 2026-07-22 02:32:08 UTC and the job concluded failure at 03:45:55 UTC — ~74 min, below the 90-min timeout-minutes, so this is a test failure/crash exit, not a wall-clock timeout. The specific failing diffusion case cannot be pinned because the log and failure artifacts are purged.

Regression Status

Known recurring failure (persistent) — no passing run found in the queryable window.

Recent history of the multimodal-gen-test-1-gpu-amd (…, 1) job (most recent 15 comparable completed scheduled runs: branch=main, event=schedule, status=completed):

Date (UTC)	Run	Job Status	Test File Status
2026-07-22 12:32	29920027108	❌ Failed	⚠️ log expired
2026-07-22 00:26	29880220845	❌ Failed (this run)	⚠️ log expired
2026-07-21 12:32	29830494225	❌ Failed	⚠️ log expired
2026-07-21 00:27	29790347494	❌ Failed	⚠️ log expired
2026-07-20 12:42	29743214297	❌ Failed	⚠️ log expired
2026-07-20 00:28	29709549202	❌ Failed	⚠️ log expired
2026-07-19 12:17	29686709871	❌ Failed	⚠️ log expired
2026-07-19 00:27	29666971143	❌ Failed	⚠️ log expired
2026-07-18 12:17	29644059344	❌ Failed	⚠️ log expired
2026-07-18 00:25	29623067212	❌ Failed	⚠️ log expired
2026-07-17 12:25	29580198489	❌ Failed	⚠️ log expired
2026-07-17 00:27	29544840981	❌ Failed	⚠️ log expired
2026-07-16 12:31	29498496614	❌ Failed	⚠️ log expired
2026-07-15 12:28	29415344552	❌ Failed	⚠️ log expired
2026-07-15 00:22	29378873430	❌ Failed	⚠️ log expired

The job also failed in older runs I spot-checked (2026-07-14 and 2026-07-13, runs 29332511138, 29215394044). No passing run of this job exists anywhere in the queried window (2026-07-13 → 2026-07-22, 15+ runs). First-observed failure is earlier than the window; I cannot bound it precisely without older logs.

⚠️ Caveat on the verdict. Per CI-Bot rules the regression call should be at the test-file level, not the job level. Because every historical log is purged, I can only assert job-level persistence. It is possible (though there is no evidence either way) that the specific failing case in part 1 has rotated over these 10 days while the job stayed red. Treat "same failure for 10 days" as [LOW] confidence; "job red for 10 days" is [FACT].

Failure Cluster

Diffusion 1-GPU server test suite failure (partition 1) — symptom-level only; the underlying case is unidentified due to log expiry.

Facts (verified observations, no interpretation)

• [FACT] Job 88799241355 conclusion = failure; failing step = step 9 "Run diffusion server tests (1-GPU)".
• [FACT] Step 9 ran 02:32:08 → job end 03:45:55 UTC (~74 min) < 90-min timeout-minutes — not a wall-clock timeout.
• [FACT] Steps 1–8 (container start, dep install, cache setup, diagnostics) all succeeded; the failure is inside the pytest invocation.
• [FACT] This job failed in every completed scheduled pr-test-amd.yml run from 2026-07-13 through 2026-07-22 (15+ consecutive runs).
• [FACT] The run has 0 downloadable artifacts and the job log returns BlobNotFound; same for the newer run 29920027108. Test-level evidence is unrecoverable.
• [FACT] The 1-GPU suite is LPT-partitioned (run_suite.py:117 auto_partition); partition-1 content is dynamic, so the case set cannot be inferred statically with certainty.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [LOW] A specific diffusion 1-GPU case in partition 1 is broken on ROCm/MI325 and has been persistently red for ~10 days.
  • Supporting: job red every scheduled run over the window; failure is inside the pytest step, not infra (container/deps/caches all passed).
  • Disconfirming: no log/artifact to identify the case; cannot confirm the same case fails each run vs. a rotating failure. No commit-level evidence.
  • How to verify: re-run the job (or trigger pr-test-amd.yml on main) and capture the live log before it expires; read run_suite.py stdout for the "Running N cases … Test files:" block plus the pytest FAILED lines and the execution_report_1-gpu_1.json / diffusion-failures-amd-1gpu-1-* artifact.
• No hypothesis above LOW confidence. Root cause cannot be attributed to any commit without the failing case identity; a git bisect is not actionable until the test is identified.

In-flight Fix Check

Cannot map to a specific in-flight PR without the failing case. Candidate open PRs touching diffusion/multimodal_gen on AMD (verify against the live failure before assuming any of these applies):

• #31925 — "[AMD] Skip test_update_weights_from_disk on ROCm pending reload fix" (a 1-GPU standalone file; note part-1 is likely a parametrized-case partition, so this may not be the match).
• #25969 — "[AMD] Disable torch.compile for ROCm diffusion layernorm".
• #27587 — "[AMD][CI] Enable SGLANG_USE_ROCM_FLYDSL=1 in diffusion e2e tests".
• In-flight fix: ❌ none confirmed — none can be confirmed as the fix without the failing test name.

Suggested Triage Steps

• Re-trigger and capture the live log immediately — this workflow's logs expire fast (already gone < 24 h). Re-run job multimodal-gen-test-1-gpu-amd (…, 1) on main and download the log within the retention window, or read it from the running job.
• From the live log, extract: the run_suite.py "Running N cases … Test files: [...]" banner (identifies partition-1 cases), the pytest FAILED … lines, and the execution_report_1-gpu_1.json / diffusion-failures-amd-1gpu-1-* artifact.
• Once the failing case is known, bisect within the window bounded by the first failing scheduled run (earlier than 2026-07-13) using git log --oneline <pass_sha>..<fail_sha> filtered to python/sglang/multimodal_gen/**.
• If CI retention is the blocker, consider raising the Actions log/artifact retention for pr-test-amd.yml so persistent-failure triage is possible after the fact.

Status (factual, no priority assignment)

• Persistence: Job red in every completed scheduled run 2026-07-13 → 2026-07-22 (15+ runs, ≥10 days). [FACT] at job level; [LOW] that it's the identical test each time.
• Scope: multimodal-gen-test-1-gpu-amd partition 1/4 in pr-test-amd.yml. Other partitions/workflows not assessed in this per-job task.
• Blocked work: Diffusion (multimodal_gen) 1-GPU e2e signal on AMD MI325 for this shard.
• In-flight fix: ❌ none confirmed — failing case unidentified due to purged logs; candidate PRs listed above must be verified against a fresh log.

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-07-23 18:35 UTC)

[amd-bot]

pr-test-amd.yml — Per-Job Analysis (continued 3/6)

multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 3) — failed step(s): Run diffusion server tests (1-GPU)

Commit Info

• sglang: 8ae0eb83
• aiter: 9127c94a (v0.1.17.dev110+g9127c94a1, source: Dev/patched image — [CI-AITER-CHECK] detected a dev build and skipped rebuild; AITER_COMMIT_OVERRIDE empty)

Failed Tests

Job runs the standalone file test_update_weights_from_disk.py (diffusion 1-GPU suite, partition 4/4). Result: 4 failed, 2 passed, 4 errors.

Test File	Test Function	Error	Log
python/sglang/multimodal_gen/test/single_test_file/test_update_weights_from_disk.py	TestUpdateWeightsFromDisk::test_update_weights_from_disk_default[Qwen-Image]	assert 400 == 200 (line 432)	Log
same	TestUpdateWeightsFromDisk::test_update_weights_specific_modules[Qwen-Image]	AssertionError: Update failed: {...transformer... rolled back} assert 400 == 200 (line 454)	Log
same	TestUpdateWeightsFromDisk::test_corrupted_weights_rollback[Qwen-Image]	assert (400 == 200) (line 535)	Log
same	TestUpdateWeightsFromDiskWithOffload::test_update_weights_with_offload_enabled[Qwen-Image]	AssertionError: Expected 200, got 400 (line 638)	Log
same	TestUpdateWeightsFromDisk::test_update_weights_from_disk_default[FLUX.2-klein-base-4B]	ERROR — _compute_checksum_from_disk thread: No weights dir for transformer/vae in /tmp/sglang_perturbed_vae_... (line 220)	Log
same	TestUpdateWeightsFromDisk::test_update_weights_specific_modules[FLUX.2-klein-base-4B]	ERROR — same setup/teardown checksum thread	Log
same	TestUpdateWeightsFromDisk::test_update_weights_rejects_invalid_requests[FLUX.2-klein-base-4B]	ERROR — same	Log
same	TestUpdateWeightsFromDisk::test_corrupted_weights_rollback[FLUX.2-klein-base-4B]	ERROR — same	Log

Failure Summary

The multimodal_gen /update_weights_from_disk endpoint returns HTTP 400 for the transformer module of diffusion models loaded from diffusers-format checkpoints. Server-side, the update raises a bare AssertionError from a weight shape mismatch at linear.py:410, so the four Qwen-Image tests that assert status_code == 200 fail, and the four FLUX.2-klein-base-4B tests error out in their checksum setup/teardown threads. This is a genuine functional failure, not infra.

Regression Status

Known recurring failure / Never-passed (in queryable window). test_update_weights_from_disk.py has failed in every completed scheduled pr-test-amd.yml run in the 15-run window (Jul 15 → Jul 22); no green run found. Recent runs are slightly greener (4 failed vs 5) after #31263 fixed a separate InferenceMode-500 failure in the same file; the transformer-reload shape-mismatch failure documented here remains.

Recent history of test_update_weights_from_disk.py in job multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 3) (most recent comparable completed scheduled runs: branch=main, event=schedule):

Date	Run	Test File Status	Failed Functions	Summary
Jul 22	29880220845	❌ Failed	4 failed + 4 errors (this run)	4 failed, 2 passed, 4 errors
Jul 21 12:32	29830494225	❌ Failed	update_weights (Qwen-Image ×4)	4 failed, 2 passed, 4 errors
Jul 21 00:27	29790347494	❌ Failed	update_weights (Qwen-Image ×4)	4 failed, 2 passed, 4 errors
Jul 20 12:42	29743214297	❌ Failed	update_weights (Qwen-Image ×4)	4 failed, 2 passed, 4 errors
Jul 20 00:28	29709549202	❌ Failed	update_weights ×5	5 failed, 1 passed, 4 errors
Jul 19 00:27	29666971143	❌ Failed	update_weights ×5	5 failed, 1 passed, 4 errors
Jul 18 00:25	29623067212	❌ Failed	update_weights ×5	5 failed, 1 passed, 4 errors
Jul 17 00:27	29544840981	❌ Failed	update_weights ×5	5 failed, 1 passed, 4 errors
Jul 16 12:31	29498496614	❌ Failed	update_weights ×5	5 failed, 1 passed, 4 errors
Jul 15 12:28	29415344552	❌ Failed	update_weights ×5	5 failed, 1 passed, 4 errors

First observed failure: at or before Jul 15 (window boundary). No last-known-passing run within the query window.

Failure Cluster

HTTP 400 on diffusion update_weights_from_disk transformer reload (weight shape mismatch)

Facts (verified observations)

• Test bodies (verbatim): test_update_weights_from_disk.py:432 assert status_code == 200 → E assert 400 == 200; :454 ... AssertionError: Update failed: {'success': False, 'message': "Failed to update module 'transformer': . All modules rolled back to original weights."}; :638 AssertionError: Expected 200, got 400.
• Server-side stack trace (verbatim): weights_updater.py:434 _apply_weights → :203 _load_weights_into_module → :295 load_weights_into_model → linear.py:410 assert param_data.shape == loaded_weight.shape → AssertionError, preceded by Weight update failed for module 'transformer': . Rolling back ....
• Name-mapping evidence (verbatim): Skipping weight update for Flux2Transformer2DModel: parameter 'img_in.bias' not found after mapping to 'x_embedder.bias' — confirms the diffusers checkpoint layout is not reconciled with sglang transformer param names/shapes.
• The 4 FLUX ERRORs are PytestUnhandledThreadExceptionWarning → AssertionError raised inside the _compute_checksum_from_disk helper thread (test_update_weights_from_disk.py:220) against a perturbed temp dir /tmp/sglang_perturbed_vae_*, i.e. test-harness checksum setup, downstream of the same reload path.
• linear.py:410 in the checked-out source is exactly assert param_data.shape == loaded_weight.shape (confirmed by reading the file).
• aiter version identical across passing-vs-failing timeframe is not the variable — the failure predates and outlasts any aiter change in the window.
• Reproducibility: same test file, same 8 (test × model) rows, failing 10/10 completed scheduled runs Jul 15–22.

Hypothesised Causes (with confidence)

• [HIGH] The diffusion update_weights_from_disk reload path does not reconcile the diffusers checkpoint parameter layout (names/shapes) with the sglang transformer parameters, so a transformer weight fails param_data.shape == loaded_weight.shape and the endpoint returns HTTP 400.
  • Supporting: stack trace lands on linear.py:410 shape assert reached via weights_updater.py load path; log shows img_in.bias→x_embedder.bias mapping miss for the Flux transformer; maintainer issue #31924 documents the identical root cause ("diffusers checkpoint layout not reconciled with sglang params → shape mismatch 400").
  • Disconfirming: (none found) — the observed symptom, stack trace, and affected models match the tracked issue exactly.
  • How to verify: reproduce with test_update_weights_from_disk.py on Qwen-Image; inspect the offending transformer param name/shape at linear.py:410 vs the diffusers checkpoint tensor.
• [FACT] This is not the InferenceMode-500 issue: that distinct failure in the same file was already fixed by #31263 (why the run went 5→4 failures); the shape-mismatch 400 is separate and still open (per #31924).

In-flight Fix Check

• In-flight fix: ✅ #31925 (open since 2026-07-21) — "[AMD] Skip test_update_weights_from_disk on ROCm pending reload fix (#31924)"; a mitigation (skip on AMD to align with NVIDIA, which already doesn't run this standalone file), not a functional fix. Root cause tracked in #31924 (open). Chase reviewers on #31925; do NOT open a duplicate.
• API queried: search/issues?q=repo:sgl-project/sglang+is:pr+is:open+update_weights_from_disk.

Suggested Triage Steps

• Land/merge the mitigation #31925 to unblock the AMD pr-test-amd diffusion signal while the real fix is developed.
• For the real fix (per #31924): in weights_updater.py load path, reconcile the diffusers checkpoint layout with sglang transformer params (add the missing name mapping, e.g. img_in.*→x_embedder.*, and any shard/shape reshaping) before the linear.py:410 assert; replace the bare AssertionError with an actionable error message so the 400 is not empty.
• Confirm scope: NVIDIA does not run this standalone file, so this is an AMD-lane coverage gap, not an AMD-specific bug — validate the fix against Qwen-Image and FLUX.2-klein-base-4B locally.

Status

• Persistence: Failing every completed scheduled pr-test-amd.yml run in the 15-run window (Jul 15 → Jul 22 2026); never passed in that window.
• Scope: This job (multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 3)); 8 (test × model) rows in test_update_weights_from_disk.py. Deterministic on ROCm per #31924.
• Blocked work: Diffusion update_weights_from_disk CI signal on AMD.
• In-flight fix: ✅ mitigation #31925 (skip on AMD, open); root-cause fix still needed (#31924 open).

multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 0) — failed step(s): N/A

Commit Info

• sglang: 8ae0eb83
• aiter: N/A — this run's job log blob is purged (BlobNotFound), so no [CI-AITER-CHECK] markers are readable. Recovered sibling-date logs show aiter running from /sgl-data/aiter-kernels/module_aiter_core.so (Dockerfile default; no override in this non-scout workflow).

Failed Tests

This run's log is purged (GitHub 404 BlobNotFound), so the exact failing case for job 88799241370 cannot be pinned from its own log. The job partition runs a single test file; the recurring failing cases (verified from the two most recent recoverable part-0 logs, Jul 17 & Jul 18) are:

Test File	Test Function	Error	Log
test/server/test_server_2_gpu.py	TestDiffusionServerTwoGpu::test_diffusion_generation[flux2_modelopt_fp8_tp2_t2i]	CUDA error: HIPBLAS_STATUS_NOT_SUPPORTED during denoising step (FP8 GEMM)	Jul18 log
test/server/test_server_2_gpu.py	TestDiffusionServerTwoGpu::test_diffusion_generation[joy_echo_t2v_2gpu]	RuntimeError: Server exited early (code 1) → HF Hub CAS download I/O error for jdopensource/JoyAI-Echo	Jul18 log

Path prefix: python/sglang/multimodal_gen/.

Failure Summary

The multimodal-gen-test-2-gpu-amd part-0 job runs the parametrized test_server_2_gpu.py diffusion suite on 2×MI325. In this run, step 9 ("Run diffusion server tests (2-GPU)") started at 02:45:58Z and the job was hard-terminated at 03:59:57Z (~74 min) — the step never recorded completion (stuck in_progress) and the if: always() "Upload diffusion failure artifacts" step never ran, indicating a hard runner/container termination rather than a clean pytest exit. Sibling parts 1 and 2 in the same run passed. Because the log is purged, the exact terminal cause for this run is not directly observable.

Regression Status

Known recurring / Never-passed in window — part 0 failed in all 15 most recent comparable scheduled runs (same branch main, event=schedule, completed only). No green run exists in the lookback window (2026-07-15 → 2026-07-22). Note: part 0's parametrized partition runs only test_server_2_gpu.py, so the job's pass/fail equals this test file's status.

Recent history of test_server_2_gpu.py in job multimodal-gen-test-2-gpu-amd (…, 0) (most recent 15 comparable completed scheduled runs):

Date	Run	Test File Status	Step 9 terminal state	Notes
Jul 22	29920027108	❌ Failed	in_progress (hard-kill)	log purged
Jul 22	29880220845	❌ Failed	in_progress (hard-kill)	this run, log purged
Jul 21	29830494225	❌ Failed	in_progress (hard-kill)	log purged
Jul 21	29790347494	❌ Failed	in_progress (hard-kill)	log purged
Jul 21	29743214297	❌ Failed	in_progress (hard-kill)	log purged
Jul 20	29709549202	❌ Failed	in_progress (hard-kill)	log purged
Jul 19	29686709871	❌ Failed	failed pre-step9	log purged
Jul 19	29666971143	❌ Failed	in_progress (hard-kill)	log purged
Jul 18	29644059344	❌ Failed	died in early step (~10 min)	log purged
Jul 18	29623067212	❌ Failed	failure/completed (clean)	HIPBLAS FP8 + joy_echo download
Jul 17	29580198489	❌ Failed	in_progress (hard-kill)	log purged
Jul 17	29544840981	❌ Failed	failure/completed (clean)	HIPBLAS FP8 cascade (19 errors)
Jul 16	29498496614	❌ Failed	failure/completed (clean)	log recoverable
Jul 15	29415344552	❌ Failed	failed pre-step9	log purged
Jul 15	29378873430	❌ Failed	failed pre-step9	log purged

First observed failure in window: 2026-07-15 (earliest queried; likely older). No last-known-passing run found in the 15-run window.

Failure Cluster

HIPBLAS_STATUS_NOT_SUPPORTED during FP8 denoising on MI325 (gfx942) — the dominant, recurring symptom in recoverable part-0 logs (29 occurrences per log). A secondary cluster is HF Hub CAS download failure (joy_echo / JoyAI-Echo).

Facts (verified observations, no interpretation)

• This run's job log returns HTTP 404 BlobNotFound — logs expired/purged; direct per-run evidence unavailable.
• Step timings (from Jobs API): step 9 Run diffusion server tests (2-GPU) start 02:45:58Z, status in_progress, conclusion None; job completed 03:59:57Z with conclusion failure. Step 9 timeout-minutes: 150 (.github/workflows/pr-test-amd.yml:890) was not reached (~74 min elapsed).
• Step 10 Upload diffusion failure artifacts (if: always(), workflow line 922) never ran (status: pending) — no clean step exit occurred.
• Sibling matrix jobs in the same run passed: part 1 88799241351 ✅, part 2 88799241361 ✅. Only part 0 failed.
• Part 0 failed in all 15 recent comparable scheduled runs.
• Recovered Jul 18 log: short test summary = FAILED …[flux2_modelopt_fp8_tp2_t2i] (CUDA HIPBLAS_STATUS_NOT_SUPPORTED in denoising step) + ERROR …[joy_echo_t2v_2gpu] (ValueError: Could not find model at jdopensource/JoyAI-Echo … CAS Client Error: … I/O error).
• Recovered Jul 17 log: HIPBLAS_STATUS_NOT_SUPPORTED first appears at server warmup ([denoising_step_0] Error … CUDA error: HIPBLAS_STATUS_NOT_SUPPORTED), followed by a ~19-case ERROR cascade.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [MEDIUM] The job's recurring root failure is a hipBLAS FP8 GEMM path unsupported on gfx942 (MI325) for FLUX.2/ideogram modelopt-FP8 cases (HIPBLAS_STATUS_NOT_SUPPORTED raised inside the denoising step).
  • Supporting: 29 HIPBLAS_STATUS_NOT_SUPPORTED hits per recoverable part-0 log; the error fires at the FP8 denoising GEMM; open PR #28889 targets exactly this ("native e4m3fnuz fp8 GEMM for FLUX.2 modelopt-FP8 on gfx942").
  • Disconfirming: this specific run's log is purged, so I cannot confirm the FP8 crash for run 29880220845 itself; its hard-kill terminal state differs from the clean-failure runs where HIPBLAS was observed.
  • How to verify: re-run part 0 at head with logging retained; grep step 9 for HIPBLAS_STATUS_NOT_SUPPORTED; A/B by applying PR #28889 and re-running [flux2_modelopt_fp8_tp2_t2i].
• [LOW] This run's hard termination (in_progress step + skipped always() upload after ~74 min) is a hang / OOM / runner-crash distinct from a clean assertion failure — possibly the HIPBLAS error escalating into a wedged server, or an unrelated container/runner death.
  • Supporting: several part-0 runs (Jul 19–22) share the same in_progress/hard-kill signature; the always() upload never executing points to abrupt termination, not pytest exit.
  • Disconfirming: no log for any of these hard-kill runs, so the terminal cause is unverified; could be infra (runner eviction) rather than a code fault.
  • How to verify: inspect runner-side kernel/dmesg + rocm-smi around 03:59Z; enable core-dump/py-spy on the 2-GPU runner; retain step-9 logs on next run.
• [LOW] Secondary: joy_echo_t2v_2gpu fails on HF Hub Xet/CAS download I/O error for jdopensource/JoyAI-Echo — an environmental/download flake, independent of the HIPBLAS cluster.
  • Supporting: explicit CAS Client Error: … I/O error: error decoding response body at model setup.
  • Disconfirming: transient network signature; may not reproduce every run.
  • How to verify: pre-cache jdopensource/JoyAI-Echo into /sgl-data/hf-cache; retry download.

In-flight Fix Check

• HIPBLAS FP8 cluster: In-flight fix: ✅ #28889 "[AMD] Use native e4m3fnuz fp8 GEMM for FLUX.2 modelopt-FP8 on gfx942" (open since 2026-06-22, last updated 2026-07-21) — directly addresses the FP8 GEMM path on gfx942/MI325. Chase reviewers; do NOT open a duplicate. (search: repo:sgl-project/sglang is:pr is:open HIPBLAS / flux2 fp8.)
• joy_echo download: In-flight fix: ❌ none found (search: joy_echo → 0 open PRs).
• hard-kill/hang signature: In-flight fix: ❌ none found — needs the retained-log rerun above first.

Suggested Triage Steps

• Re-run multimodal-gen-test-2-gpu-amd (…, 0) at head with step-9 logs retained (or download the artifact before expiry) to capture the actual terminal cause for the hard-kill runs — current diagnosis relies on sibling-date logs because this run's log is purged.
• Verify #28889 still applies cleanly and re-run [flux2_modelopt_fp8_tp2_t2i]/[ideogram4_fp8_tp2_t2i] on MI325 to confirm it clears the HIPBLAS_STATUS_NOT_SUPPORTED cluster.
• If the hard-kill recurs without a HIPBLAS crash in the retained log, treat as a hang/OOM: enable py-spy + core-dump on linux-mi325-2gpu-sglang and inspect dmesg/rocm-smi around the termination timestamp.
• Pre-cache jdopensource/JoyAI-Echo on the runner to remove the joy_echo CAS download flake from the signal, isolating the HIPBLAS cluster.

Status (factual, no priority assignment)

• Persistence: part 0 has failed every completed scheduled run in the 2026-07-15 → 2026-07-22 window (15/15); no green run found.
• Scope: multimodal-gen-test-2-gpu-amd part 0 only; parts 1 & 2 of the same runs pass. Recurring HIPBLAS FP8 symptom confirmed in ≥2 recoverable part-0 logs; secondary joy_echo download error alongside.
• Blocked work: 2-GPU diffusion FP8 (FLUX.2/ideogram modelopt-FP8) CI signal on AMD MI325.
• In-flight fix: ✅ #28889 for the HIPBLAS FP8 cluster; ❌ none for the joy_echo download error or the hard-kill/hang signature.
• Caveat: this run's own log is purged; the exact terminal cause for run 29880220845 is not directly verifiable and the HIPBLAS attribution is inferred from same-job recoverable logs.

multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 0) — failed step(s): N/A

Commit Info

• sglang: 8ae0eb83 (#31985, spec-decode draft embedding — does not touch multimodal_gen/)
• aiter: v0.1.17.dev110+g9127c94a1 (9127c94a) — source: dev image, preserved (Dev/patched version detected → skipping rebuild). Read from the sibling shard-3 [CI-AITER-CHECK] markers in the same run; shard-0's own log is unavailable (see below).

Failed Tests

Test File	Test Function	Error	Log
test/.../multimodal_gen/test/server/test_server_1_gpu.py (partition 1/4)	cannot pin — step timed out	Step "Run diffusion server tests (1-GPU)" exceeded its timeout-minutes: 90 (job wall = 88.0 min)	Job

⚠️ Log caveat: shard-0's step log is not retrievable. The job-log API returns BlobNotFound, and shard 0 is absent from the run-level log archive (only shards 2 & 3 are present). GitHub does not finalize a step's log blob when that step is killed by its own timeout — so the exact hanging test function cannot be named from this run. Evidence below is from job metadata, the workflow config, sibling shards in the same run, and the suite source.

Failure Summary

The shard-0 job of multimodal-gen-test-1-gpu-amd ran 88.0 min and the test step hit its 90-minute limit (pr-test-amd.yml:751), failing the job. Shard 0 runs partition 1/4 of the 1-gpu suite — the heavy parametrized test_server_1_gpu.py server cases (Wan2.1-T2V, Qwen-Image, FLUX.1/2, Cosmos, etc.), whereas shards 2 & 3 each run a single standalone file and finished in 22/26 min. Both shards that carry the parametrized cases (0 and 1) time out; shards with one standalone file each do not.

Regression Status

Known recurring failure (not new; not caused by the head commit).

Recent history of multimodal-gen-test-1-gpu-amd (…, 0) in pr-test-amd.yml (most recent comparable completed scheduled runs: branch=main, event=schedule, completed only). Shard-0 logs time out and aren't archived, so status is tracked at the job level (job conclusion + wall-duration vs the 90-min step budget):

Date (UTC)	Run	Shard-0 Status	Wall (min)	Note
Jul 22 12:32	29920027108	❌ Failed	85.6	~timeout
Jul 22 00:26	29880220845	❌ Failed (this run)	88.0	hit 90-min timeout
Jul 21 12:32	29830494225	❌ Failed	122.2	over budget
Jul 21 00:27	29790347494	❌ Failed	99.1	over budget
Jul 20 12:42	29743214297	—	—	shard 0 did not run
Jul 20 00:28	29709549202	❌ Failed	89.1	~timeout
Jul 19 12:17	29686709871	❌ Failed	82.1	~timeout
Jul 19 00:27	29666971143	❌ Failed	80.1	~timeout
Jul 18 12:17	29644059344	❌ Failed	74.0	near budget
Jul 18 00:25	29623067212	❌ Failed	91.1	over budget
Jul 17 12:25	29580198489	❌ Failed	85.1	~timeout
Jul 17 00:27	29544840981	❌ Failed	73.0	near budget
Jul 16 12:31	29498496614	❌ Failed	75.1	near budget
Jul 15 12:28	29415344552	❌ Failed	10.0	fast fail (different mode)
Jul 15 00:22	29378873430	❌ Failed	79.0	near budget

Shard 0 has failed in 14/14 comparable runs where it ran (one run didn't schedule it). No green run in the queryable window → first-observed passing date is unknown; this failure predates the head commit and is not a regression introduced by 8ae0eb83.

Failure Cluster

1-GPU diffusion server suite (partition 1/4) exceeds the 90-min step timeout

Facts (verified observations)

• Job 88799241374: conclusion=failure, step 9 "Run diffusion server tests (1-GPU)" conclusion=null (killed mid-step), wall = 88.0 min (04:17:52Z→05:45:55Z).
• Step budget is timeout-minutes: 90 (.github/workflows/pr-test-amd.yml:751).
• Same-run sibling shards: shard 2 ✅ 22.2 min (standalone test_generate_zimage_turbo_cli.py), shard 3 ❌ 26.1 min (standalone test_update_weights_from_disk.py), shard 1 ❌ 85.1 min, 2-gpu shard 0 ❌ 85.1 min. The two shards carrying parametrized server cases (0, 1) both time out; single-standalone-file shards do not.
• Suite layout (python/sglang/multimodal_gen/test/server/gpu_cases.py): the 1-gpu suite = parametrized test_server_1_gpu.py (ONE_GPU_CASES, 30+ heavy model cases: Wan2.1-T2V, Qwen-Image, FLUX.1/2-dev, FLUX.2-klein, Cosmos, LongLive) + 2 standalone files; partitioned by LPT across 4 shards. Standalone test_update_weights_from_disk.py est = 1200 s (gpu_cases.py:1008).
• Shard-0 step log is not archived (BlobNotFound + absent from both this run's and the next run's log zip).

Hypothesised Causes (with confidence)

• [MEDIUM] Aggregate runtime of the parametrized server cases in partition 1/4 exceeds the 90-min step budget.
  • Supporting: 88-min wall ≈ 90-min step limit; both parametrized-carrying shards (0 & 1) time out while single-standalone shards finish in ≤26 min; persistent across 14 runs independent of head SHA; ONE_GPU_CASES is 30+ heavy image/video-model server tests (each pays model-load + server-start + denoise).
  • Disconfirming: shard-0's own log is unavailable, so I cannot rule out a single hung case consuming the budget rather than aggregate overrun; wall duration varies widely (73–122 min), consistent with variable per-run model-download time layered on top of a near-budget suite.
  • How to verify: temporarily lower the step timeout and enable per-test timing/artifact capture on shard 0, or run partition 1/4 locally and sum per-case durations against 90 min.
• [LOW] A single case hangs (server-startup deadlock or stalled HF model download).
  • Supporting: duration variance run-to-run; one run fast-failed at 10 min (a different mode).
  • Disconfirming: shard 1 (a different parametrized subset) also times out at ~85 min, which favors an aggregate-runtime problem over one specific hanging case.
  • How to verify: capture per-case start/finish timestamps on shard 0; check for a case that never emits Pixel data generated successfully.

Head commit 8ae0eb83 and the aiter dev build are excluded as causes: the failure is persistent across 14 runs with varying SHAs, and neither the commit nor aiter touches multimodal_gen/.

In-flight Fix Check

• Shard-0 timeout: In-flight fix: ❌ none found — searched open PRs for multimodal-gen-test+timeout, diffusion+test+partition, multimodal_gen+1-gpu+timeout; no PR addresses the partition-0 90-min overrun.
• Related (sibling shard 3, not this failure): #31925 (open since 2026-07-21) skips test_update_weights_from_disk on ROCm, tracked in #31924 — resolves the shard-3 Qwen-Image failure only.

Suggested Triage Steps

• Capture shard-0 evidence first: add per-case timing (or a step-level artifact flushed on timeout) so the hang-vs-aggregate question can be settled; the current timeout swallows the log.
• If aggregate overrun is confirmed: rebalance the LPT plan (the two standalone est-times exist, but ONE_GPU_CASES cases may lack accurate per-case estimates, pushing too many onto partitions 0/1), increase partition count beyond 4, or raise timeout-minutes for this step.
• Do not bisect main — this is not a regression tied to a commit window (no green run in history); a bisect will not find a culprit.

Status

• Persistence: shard-0 failing in 14/14 comparable scheduled runs it ran in (queryable window Jul 15–22 2026); predates the head commit; no green run found.
• Scope: this shard (partition 0). Same-run companions also over budget: 1-gpu shard 1 (85.1 min) and 2-gpu shard 0 (85.1 min) — a shared "parametrized-partition exceeds step budget" pattern. Sibling shard-3's test_update_weights_from_disk failure is a distinct issue with its own in-flight fix.
• Blocked work: 1-GPU diffusion server-test signal on AMD MI325 is not obtained for partition 0 (and 1); a green PR CI cannot verify these code paths.
• In-flight fix: ❌ none for the shard-0 timeout; #31925 covers only the shard-3 test_update_weights_from_disk failure.

stage-b-test-1-gpu-small-amd-mi35x (linux-mi35x-gpu-1) — failed step(s): Run test

Commit Info

• sglang: 8ae0eb83
• aiter: 9127c94a (v0.1.17.dev110+g9127c94a1, source: dev image preserved — matches Dockerfile default, no override)

Failed Tests

Test File	Test Function	Error	Log
test/registered/quant/test_quark_mxfp4.py	TestFP8ToMXFP4MOETP1.test_gsm8k	AssertionError: 0.92 not greater than 0.92	Log

Suite stage-b-test-1-gpu-small-amd-mi35x: 4/5 files passed; only test_quark_mxfp4.py failed (exit 255 → job fail). Other 4 files (test_dsa_skip_logits_clean.py, test_verify_splitkv.py, test_mamba_state_scatter_triton.py, test_deepseek_mla_dispatch.py) passed.

Failure Summary

The online FP8→MXFP4 requantization GSM8K accuracy test for Qwen/Qwen3-30B-A3B-Instruct-2507-FP8 (TP=1) missed its threshold. It requires assertGreater(accuracy, 0.92) (strict). First attempt scored 0.912, the single retry scored exactly 0.920 — both fail strict-greater, so the test errored out.

Regression Status

Flaky test — borderline accuracy threshold on a newly-landed test.

The test file was (re)landed by dcd9014f (#28291) on 2026-07-21 09:50 UTC, so only three completed scheduled runs of this job have ever contained it. Recent history of test_quark_mxfp4.py (TestFP8ToMXFP4MOETP1.test_gsm8k) in job stage-b-test-1-gpu-small-amd-mi35x (event=schedule, branch=main, completed only):

Date (UTC)	Run	Job	Test File Status	MOETP1 accuracy	Error
2026-07-22 12:32	29920027108	job	✅ Passed	~0.926–0.928	—
2026-07-22 00:26	29880220845	job	❌ Failed	0.912 → 0.920	AssertionError: 0.92 not greater than 0.92
2026-07-21 12:32	29830494225	job	✅ Passed	~0.920–0.922	—
2026-07-21 00:27	29790347494	job	⚪ N/A (pre-reland)	test not yet present	—

First observed failure of this test: 2026-07-22 00:26 UTC. Last known pass: 2026-07-22 12:32 UTC (the very next run). Failure ratio since landing: 1/3 runs. The failure is sandwiched between two passing runs, which rules out a persistent regression.

Failure Cluster

MXFP4 GSM8K accuracy just below strict threshold (borderline eval flake)

Facts (verified observations, no interpretation)

• Failing assertion verbatim: AssertionError: 0.92 not greater than 0.92 at test_quark_mxfp4.py:171 via test_gsm8k at line 251 (log step:6:9367).
• Two attempts in this run: Accuracy: 0.912 (first), Accuracy: 0.920 (retry). The retry harness (_callTestMethod → retry, max 1 retry) exhausted retries.
• Threshold is _test_gsm8k(accuracy_threshold=0.92) with assertGreater (strict), while the code comment states reference accuracy ~0.948 (test_quark_mxfp4.py:250-251).
• Same test in the next run 29920027108 passed with accuracy ~0.926; in the prior run 29830494225 passed at ~0.922. Observed MXFP4 accuracy across runs spans 0.912–0.928, straddling the 0.92 line.
• No CUDA fault, no crash, no import error; the server served all 500 GSM8K questions (Invalid: 0.000) in every attempt.
• aiter version identical (dev image 9127c94a1) in this run — no aiter delta involved (single-variable, non-scout workflow).

Hypothesised Causes (with confidence, with disconfirming evidence)

• [HIGH] Accuracy threshold is set too tight for the observed run-to-run variance of this eval. Observed MXFP4 GSM8K accuracy on 500 questions clusters at 0.912–0.928; a strict > 0.92 threshold has near-zero margin and GSM8K/500q sampling variance (~±0.01–0.02) crosses it.
  • Supporting: exact-boundary failure (0.920 retry), and the immediately-adjacent runs pass at 0.922 / 0.926 with the same code and same aiter — pure variance around the cutoff.
  • Disconfirming: the comment claims reference ~0.948, so 0.92 "should" have margin — but the actually observed MXFP4 accuracy is ~0.91–0.93, not 0.948, so the real margin is ~0; (none that supports a code regression).
  • How to verify: re-run test_quark_mxfp4.py 5–10× on the same SHA 8ae0eb83; expect intermittent pass/fail around 0.92. Consider lowering the threshold or increasing num_questions to reduce variance.
• [LOW] Genuine accuracy regression from a commit in e4eea7ce..8ae0eb83.
  • Supporting: this run scored lower (0.912) than the flanking runs.
  • Disconfirming: the very next run 29920027108 (head b13abfde, a superset of these commits) passed at 0.926 — a persistent regression would not self-heal one run later. This strongly argues for variance, not a regression.
  • How to verify: not warranted unless repeated re-runs on 8ae0eb83 are consistently below 0.92.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for quark_mxfp4 (0 results) and MXFP4 gsm8k (77 unrelated results, none targeting this threshold). No PR adjusting TestFP8ToMXFP4MOETP1 threshold or quarantining this test.

Suggested Triage Steps

• Re-run the stage-b-test-1-gpu-small-amd-mi35x suite on SHA 8ae0eb83 several times to confirm the ~0.92 boundary flakiness (expect intermittent pass/fail).
• If confirmed as boundary noise, loosen TestFP8ToMXFP4MOETP1._test_gsm8k(accuracy_threshold=0.92) in test/registered/quant/test_quark_mxfp4.py:251 to a value with real margin (e.g. 0.88–0.90, consistent with the other MXFP4 classes which use 0.85–0.89), and/or raise num_questions above 500 to shrink variance. Loop in the test author (fxmarty-amd, #28291).
• Only if repeated re-runs on 8ae0eb83 stay below 0.92, bisect e4eea7ce..8ae0eb83 for an MXFP4/MoE quantization change — but the passing next run makes this unlikely.

Status (factual, no priority assignment)

• Persistence: 1 failing run out of 3 completed scheduled runs since the test landed (2026-07-21 09:50 UTC); the immediately preceding and following runs both passed.
• Scope: single test file (test_quark_mxfp4.py) in one job (stage-b-test-1-gpu-small-amd-mi35x); other 4 files in the suite passed. Not seen in sibling jobs.
• Blocked work: fails the stage-b-test-1-gpu-small-amd-mi35x job (exit 255) whenever the MXFP4 MoE eval dips ≤0.92, producing intermittent red on pr-test-amd scheduled CI.
• In-flight fix: ❌ none found — needs a threshold adjustment PR or test author input.

stage-b-test-large-8-gpu-mi35x-disaggregation-amd (linux-mi35x-gpu-8.fabric) — failed step(s): Run test_disaggregation

Commit Info

• sglang: 8ae0eb83 (#31985 "Fold dspark dense draft embedding into the draft graph")
• aiter: 9127c94a — v0.1.17.dev110+g9127c94a1 (source: dev/patched CI image, rebuild skipped; no AITER_COMMIT_OVERRIDE)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/disaggregation/test_disaggregation_pp.py	TestDisaggregationPrefillPPDynamicChunkAccuracy.test_gsm8k	Exception: retry() exceed maximum number of retries. (underlying KeyError: 'answer' from KVTransferError)	Log

All other test suites in this job passed: TestDisaggregationAccuracy (6 tests OK), TestMori*/TestNixl* (OK). Only the test_disaggregation_pp.py suite errored (Ran 3 tests … FAILED (errors=1)), and the job exited 255.

Failure Summary

During the PD-disaggregation gsm8k eval (200 questions, Qwen3-8B, PP=2/TP=2, Mooncake RDMA backend), 2 of 200 decode requests failed to receive their KV cache from the prefill instance. The eval harness (few_shot_gsm8k.run_eval) then hit KeyError: 'answer' for the incomplete requests and, after exhausting its single retry, raised retry() exceed maximum number of retries. The prefill server did not crash — it kept logging prefill batches throughout; the transfer engine timed out.

Regression Status

Flaky test (recurring file-level cluster). The exact function TestDisaggregationPrefillPPDynamicChunkAccuracy.test_gsm8k failed today but passed in the prior 8 comparable runs. However, the underlying Mooncake KV-transfer-timeout cluster recurs across this job, striking a different PP gsm8k test each time it fires (DecodePP / PrefillPP / PrefillPPDynamicChunk).

Recent history of test_disaggregation_pp.py (TestDisaggregationPrefillPPDynamicChunkAccuracy.test_gsm8k) in job stage-b-test-large-8-gpu-mi35x-disaggregation-amd (most recent comparable completed scheduled runs: main, event=schedule):

Date	Run	Test Fn Status	File-level KV-timeout cluster	Which PP test errored
Jul 22 00:26	29880220845	❌ Failed	❌ present	PrefillPPDynamicChunk (this run)
Jul 21 12:32	29830494225	✅ Passed	❌ present	DecodePP
Jul 21 00:27	29790347494	✅ Passed	✅ clean	—
Jul 20 12:42	29743214297	✅ Passed	✅ clean	—
Jul 20 00:28	29709549202	✅ Passed	✅ clean	—
Jul 19 12:17	29686709871	✅ Passed	✅ clean	—
Jul 19 00:27	29666971143	✅ Passed	❌ present	PrefillPPAccuracy
Jul 18 12:17	29644059344	✅ Passed	✅ clean	—
Jul 18 00:25	29623067212	❌ (Decode)	❌ present	DecodePP

Exact function: 1/9 failing (flaky). File-level KV-timeout cluster: 4/9 runs. Cluster observed at least since Jul 18 (edge of query window — likely older).

Failure Cluster

Mooncake RDMA KV-transfer 30 s timeout in PD-disaggregation gsm8k eval.

Facts (verified observations)

• Transfer-engine timeout (verbatim, job log ~L23570): I0722 00:57:40.824287 33944 transfer_engine_py.cpp:614] Sync batch data transfer timeout after 30000001628ns (≈30 s).
• Immediately after: [TP0] Decode transfer failed … KVTransferError(bootstrap_room=549728668283250624): Failed to get kvcache from prefill instance, it might be dead → POST /generate returns 500. A second request (bootstrap_room=6211172381509252560) failed the same way at 00:58:15. Only 2 of 200 requests affected.
• Test-level stack (job log ~L24746): few_shot_gsm8k.py:123 → get_answer_value(states[i]["answer"]) → KeyError: 'answer', wrapped by retry() exceed maximum number of retries (test_utils.py:2330).
• Prefill server alive throughout: Prefill batch … lines continue at 00:57:11+ with normal throughput; no crash, watchdog, HIP error, or OOM in the log.
• Config (from server_args, L5429/L5430): disaggregation_transfer_backend='mooncake', disaggregation_ib_device='rdma0', RoCE fabric detected; set MC_GID_INDEX=1 (L5391), attention_backend='aiter'.
• Runner: linux-mi35x-gpu-8.fabric. UCX warnings failed to get interface index for tw-eth0..7 (errno 19) recur (L26576+) — noisy network-interface probing on this runner.
• No [CI-AITER-CHECK] override; aiter unchanged vs passing runs (same dev image).

Hypothesised Causes (with confidence)

• [MEDIUM] Flaky Mooncake RDMA KV-transfer stall on the MI35x disaggregation runner — a batch transfer exceeds the 30 s timeout, decode marks the prefill "dead" and 500s the request.
  • Supporting: explicit transfer_engine_py.cpp:614 … transfer timeout after 30000001628ns; prefill demonstrably alive; RoCE/RDMA path; recurring UCX interface-probe errors; the cluster recurs intermittently (4/9 runs) hitting random requests/tests — hallmark of an infra/transport flake, not a deterministic code bug. An open DEBUG PR #29127 explicitly targets a "faulty RDMA NIC (rdma5) on MI35x disaggregation runner."
  • Disconfirming: --disaggregation-ib-device rdma0 is used here (not rdma5); the timeout could be a software transfer hang rather than a hardware NIC fault, so #29127's specific NIC hypothesis may not cover it.
  • How to verify: re-run the job; correlate with which physical NIC/GID each PP rank binds; check Mooncake transfer-queue depth at failure time.
• [LOW] ZMQ socket race in Mooncake KV-manager senders causing a dropped/stalled transfer.
  • Supporting: open fix PR #29984 "Fix race on shared ZMQ sockets in mooncake KV manager senders" addresses a sender-side race that could manifest as a stalled transfer.
  • Disconfirming: no ZMQ error/traceback in the log; only a clean 30 s timeout — consistent with transport stall, not a socket race.
  • How to verify: apply #29984 on a branch, run this suite ~20× on the MI35x runner, compare cluster incidence.
• [LOW] sglang commit regression in the current window.
  • Supporting: none specific.
  • Disconfirming: intermittent, present across many head SHAs back to Jul 18; passing and failing runs share the same code paths — not a clean regression.
  • How to verify: not actionable; this is a flake, not a bisectable regression.

In-flight Fix Check

• In-flight fix: ✅ [#29127](https://github.com/sgl-project/sglang/pull/29127) (draft, open since 2026-06-24) — "[DEBUG] Isolate faulty RDMA NIC on MI35x disaggregation runner" — directly targets this runner/backend; still a draft, needs to be driven to a real fix.
• In-flight fix: ✅ [#29984](https://github.com/sgl-project/sglang/pull/29984) (open since 2026-07-02) — mooncake KV-manager ZMQ sender race — candidate, awaiting merge.
• In-flight fix: ✅ [#30551](https://github.com/sgl-project/sglang/pull/30551) (open since 2026-07-08) — propagate aborted requests + clean up Mooncake rooms — related PD/Mooncake robustness.
• Searched: search/issues?q=repo:sgl-project/sglang+is:pr+is:open+KVTransferError (and mooncake/kvcache variants). Do NOT open a duplicate.

Suggested Triage Steps

• Treat as a flaky infra failure, not a code regression — the exact function passed in 8/9 prior runs and the cluster hits random PP tests.
• Chase the RDMA-NIC investigation in draft #29127 — confirm whether the rdma0 device / GID binding on linux-mi35x-gpu-8.fabric intermittently stalls; the UCX tw-eth* errno 19 probes suggest a misconfigured/absent interface worth ruling out.
• Evaluate #29984 (ZMQ sender race) and #30551 (Mooncake room cleanup): run this suite ~20× per candidate on the MI35x runner and compare cluster incidence.
• Consider adding a bounded harness-level retry/tolerance so a single 30 s transfer timeout among 200 requests does not error the whole gsm8k eval (the accuracy gate is 0.70; 198/200 correct would still pass) — a mitigation, not a root-cause fix.

Status

• Persistence: Cluster intermittent since at least 2026-07-18 (4 of last 9 completed scheduled runs); this exact function failing 1/9 (flaky).
• Scope: stage-b-test-large-8-gpu-mi35x-disaggregation-amd job, test_disaggregation_pp.py PP suite; Mooncake RDMA KV-transfer path on the linux-mi35x-gpu-8.fabric runner. Other transfer backends (Nixl, Mori) in the same job passed.
• Blocked work: Intermittently red-flags PD-disaggregation CI signal on MI35x; low blast radius (flaky, single request out of 200).
• In-flight fix: draft #29127 + candidates #29984, #30551.

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-07-23 18:35 UTC)

[amd-bot]

pr-test-amd.yml — Per-Job Analysis (continued 4/6)

stage-b-test-1-gpu-small-amd-nondeterministic (linux-mi325-1gpu-sglang) — failed step(s): Run test

Commit Info

• sglang: 8ae0eb83
• aiter: 9127c94a — v0.1.17.dev110+g9127c94a1 (source: dev image detected, rebuild skipped; not relevant to this failure)

Failed Tests

Test File	Test Function	Error	Log
test/registered/prefill_only/test_reward_models.py	(server crash at setUp/model load — process exit -9)	AttributeError: 'LlamaForSequenceClassificationWithNormal_Weights' object has no attribute '_legacy_load_weights'	Log

test/registered/models/test_vlm_models.py (the other file in this suite) passed. Suite result: 1/2 passed.

Failure Summary

The reward-model server crashed during weight loading: the scheduler child process died with SIGKILL (exit -9) after raising AttributeError: 'LlamaForSequenceClassificationWithNormal_Weights' object has no attribute '_legacy_load_weights'. The crash originates in python/sglang/srt/models/llama.py:662, reached via llama_reward.py:77 → LlamaForCausalLM.load_weights(self, weights). This is the first scheduled run of this job to fail; the prior 5 all passed.

Regression Status

New regression.

Recent history of test/registered/prefill_only/test_reward_models.py in job stage-b-test-1-gpu-small-amd-nondeterministic (linux-mi325-1gpu-sglang) (most recent comparable completed scheduled runs: branch=main, event=schedule):

Date (UTC)	Run	Head SHA	Job Conclusion	Test File Status	Failed Function	Error
Jul 22 00:26	29880220845	8ae0eb83	failure	❌ Failed	model load	AttributeError: no attribute '_legacy_load_weights'
Jul 21 12:32	29830494225	e4eea7ce	success	✅ Passed	—	—
Jul 21 00:27	29790347494	8905cbd4	success	✅ Passed	—	—
Jul 20 12:42	29743214297	370f454e	success	✅ Passed	—	—
Jul 20 00:28	29709549202	b3570a45	success	✅ Passed	—	—
Jul 19 12:17	29686709871	a03ca46a	success	✅ Passed	—	—

First observed failure: 2026-07-22 00:26 UTC. Last known passing: 2026-07-21 12:32 UTC (29830494225). (Job overall was success in each passing run — the regression verdict is at the test-file level. Other scheduled runs marked failure at the workflow level fail in different jobs, not this one.)

Failure Cluster

AttributeError: '_legacy_load_weights' missing on LlamaForSequenceClassification during reward-model weight load

Facts (verified observations)

• Verbatim top stack frame (log lines 4383–4382):

  File ".../python/sglang/srt/models/llama.py", line 662, in load_weights
      return self._legacy_load_weights(weights)
  ...
  AttributeError: 'LlamaForSequenceClassificationWithNormal_Weights' object has no attribute '_legacy_load_weights'
  

  Call chain: llama_reward.py:140 super().load_weights → llama_reward.py:77 LlamaForCausalLM.load_weights(self, weights) → llama.py:662.
• The scheduler then received sigquit from a child process; the runner reported ✗ FAILED: test_reward_models.py returned exit code -9.
• Code fact: in python/sglang/srt/models/llama.py:657-662, LlamaForCausalLM.load_weights now dispatches: if envs.SGLANG_ENABLE_WEIGHT_LOADER_V2.get(): return self._load_weights_v2(weights) else return self._legacy_load_weights(weights).
• Code fact: python/sglang/srt/models/llama_reward.py:33 — class LlamaForSequenceClassification(nn.Module) does not inherit from LlamaForCausalLM; it borrows LlamaForCausalLM.load_weights via an explicit unbound call passing self (llama_reward.py:77). _legacy_load_weights is defined only on LlamaForCausalLM, so it is absent on the LlamaForSequenceClassification* instance.
• Regression window: last passing head e4eea7ce → first failing head 8ae0eb83. Commit becf252e is an ancestor of the failing SHA and not of the passing SHA (verified via git merge-base --is-ancestor).
• Scope: only test_reward_models.py in this job; test_vlm_models.py passed. aiter version unchanged (dev build preserved) — not a factor.

Hypothesised Causes (with confidence)

• [HIGH] Commit becf252e (PR #28671 "AutoWeightLoader support Sglang native models 1: demo", merged 2026-07-21 21:32 UTC) refactored LlamaForCausalLM.load_weights to call self._legacy_load_weights(weights), which does not exist on LlamaForSequenceClassification/...WithNormal_Weights.
  • Supporting: the commit diff adds the exact line + return self._legacy_load_weights(weights) in llama.py (the top stack frame at llama.py:662); it is the only commit in the e4eea7ce..8ae0eb83 window touching llama.py load_weights; timing is exact — this run is the first scheduled run after the merge, and all 5 prior runs passed; llama_reward.py was not updated in the same commit (git show becf252e --stat lists only environ.py, auto_loader.py, llama.py, qwen2.py, and two test files — not llama_reward.py).
  • Disconfirming: (none found) — the borrowing call LlamaForCausalLM.load_weights(self, ...) in llama_reward.py:77 predates the commit, but it only broke once load_weights began referencing the subclass-only method.
  • How to verify: on a branch, run python3 test/registered/prefill_only/test_reward_models.py on AMD MI325 at HEAD (reproduces), then git revert becf252e and re-run (should pass); or add _legacy_load_weights/_load_weights_v2 to the LlamaForSequenceClassification code path and re-run.

In-flight Fix Check

In-flight fix: ❌ none found — searched open PRs for _legacy_load_weights (1 unrelated hit, #31415 Ray metrics), llama_reward (0 hits), reward load_weights (3 unrelated hits). No open PR addresses this AttributeError; needs a new fix.

Suggested Triage Steps

• Reproduce: run python3 test/registered/prefill_only/test_reward_models.py on AMD MI325 (container linux-mi325-1gpu-sglang) at HEAD 8ae0eb83.
• A/B confirm: git revert becf252e on a branch and re-run the same test; if it passes, becf252e is confirmed.
• Fix direction (hypothetical, maintainer decides): the borrowing pattern in llama_reward.py:77 (LlamaForCausalLM.load_weights(self, weights)) needs the _legacy_load_weights/_load_weights_v2 methods reachable on LlamaForSequenceClassification, or load_weights should be called in a way that does not require subclass-only methods. Note this likely affects any non-LlamaForCausalLM class that borrows LlamaForCausalLM.load_weights the same way — worth grepping for other reuse sites.
• Not NVIDIA-specific: this is a pure-Python code path (no AMD-only code), so the same crash should reproduce on NVIDIA if test_reward_models.py runs there — check NVIDIA CI for the same signature.

Status

• Persistence: Failing since 2026-07-22 00:26 UTC (this is the first failing scheduled run; 5 prior scheduled runs of this job passed).
• Scope: test_reward_models.py in stage-b-test-1-gpu-small-amd-nondeterministic; root cause is backend-agnostic Python (llama.py/llama_reward.py), so it likely affects reward/sequence-classification model loading on all backends, not just AMD.
• Blocked work: reward-model (LlamaForSequenceClassification) serving/CI signal on this suite.
• In-flight fix: ❌ none found — needs a new PR.

stage-b-test-1-gpu-large-amd (linux-mi325-1gpu-sglang, 0) — failed step(s): Run test

Commit Info

• sglang: 8ae0eb83fc319e181f180b65a9f2587ca3969bde
• aiter: 9127c94a18e4398e1eba91f6639e910f0994ad02 (v0.1.17.dev110+g9127c94a1, source: dev image detected → rebuild skipped, per [CI-AITER-CHECK] markers)

Failed Tests

Test File	Test Function	Error	Log
test/registered/perf/test_bench_serving_1gpu_part1.py	TestBenchServing1GPUPart1.test_online_lora_latency_with_concurrent_adapter_updates	AssertionError: 20742.33 not less than 4000 (1st try), 9400.92 not less than 4000 (retry) → retry() exceed maximum number of retries	Log

Other 7 tests in the file passed (Ran 8 tests, FAILED (errors=1)). The remaining 9 test files in the partition passed (Test Summary: 9/10 passed).

Failure Summary

The perf test test_online_lora_latency_with_concurrent_adapter_updates measures median end-to-end serving latency while a background task repeatedly loads/unloads LoRA adapters, then asserts median_e2e_latency_ms < 4000 (test_bench_serving_1gpu_part1.py:169). Both the initial attempt (20742 ms) and the single retry (9400 ms) exceeded the 4000 ms threshold, so the built-in retry wrapper raised and the job failed. This is the only failure in the job.

Regression Status

Flaky test — marginal perf threshold. The measured median latency for this test routinely sits at or above the 4000 ms assertion boundary and is normally rescued by the 1-retry wrapper; it only fails hard when a large latency spike overwhelms both attempts. Both hard failures observed in the window landed on the 00:2x nightly slot.

Recent history of test_bench_serving_1gpu_part1.py (test_online_lora_latency_with_concurrent_adapter_updates) in job stage-b-test-1-gpu-large-amd (…, 0) (most recent 15 comparable completed scheduled runs: branch=main, event=schedule, completed, partition 0 where this test file runs):

Date (UTC)	Run	Test File Status	1st-try median	Note
Jul 22 12:32	29920027108	✅ Passed (retry)	4249	over on 1st try, recovered
Jul 22 00:26	29880220845 (current)	❌ Failed	20742 → 9400	both attempts >4000
Jul 21 12:32	29830494225	✅ Passed (retry)	4330	recovered
Jul 21 00:27	29790347494	❌ Failed	15641 → 11156	both attempts >4000
Jul 20 12:42	29743214297	✅ Passed (retry)	5092	recovered
Jul 20 00:28	29709549202	✅ Passed (retry)	5233	recovered
Jul 19 12:17	29686709871	✅ Passed clean	<4000	10/10
Jul 19 00:27	29666971143	✅ Passed clean	<4000	10/10
Jul 18 12:17	29644059344	✅ Passed clean	<4000	—
Jul 18 00:25	29623067212	✅ Passed clean	<4000	—
Jul 17 12:25	29580198489	✅ Passed (retry)	4023	recovered
Jul 17 00:27	29544840981	✅ Passed clean	<4000	—
Jul 16 12:31	29498496614	✅ Passed (retry)	4437	recovered
Jul 15 12:28	29415344552	✅ Passed (retry)	4059	recovered
Jul 15 00:22	29378873430	✅ Passed (retry)	4235	recovered

• Hard-fail ratio: 2 / 15 runs (~13%).
• First-attempt-exceeds-threshold ratio: 10 / 15 runs (~67%) — the retry masks the marginality on most runs.
• First observed hard failure for this test file: Jul 21 00:27 (29790347494). Last clean (first-try) pass: Jul 19 12:17 (29686709871). The marginal behavior (first-try >4000, retry-rescued) extends back to at least Jul 15, the start of the query window.

Failure Cluster

LoRA concurrent-adapter perf threshold exceeded (median_e2e_latency_ms < 4000)

Facts (verified observations, no interpretation)

• Verbatim error: AssertionError: 20742.32725193724 not less than 4000 then on retry AssertionError: 9400.922418106347 not less than 4000, followed by Exception: retry() exceed maximum number of retries. and FAILED (errors=1) (job log step 6).
• The retry attempt's serving benchmark reported: Median E2E Latency (ms): 9400.92, Mean E2E Latency (ms): 12639.14, P90: 30737.69, P99: 51064.33, Max ITL (ms): 18617.14, Concurrency: 50.13, Successful requests: 400, throughput 3.97 req/s — i.e. very high tail latency under the concurrent load/unload background task.
• The assertion is at test/registered/perf/test_bench_serving_1gpu_part1.py:169; the test runs _run_lora_latency_test(enable_background_task=True), which spawns a background task cycling load/unload of three LoRA adapters (pbevan11/llama-3.1-8b-ocr-correction, faridlazuarda/valadapt-llama-3.1-8B-it-chinese, philschmid/code-llama-3-1-8b-text-to-sql-lora) against /load_lora_adapter and /unload_lora_adapter during the benchmark.
• Server config: max_loras_per_batch=1, lora_backend='csgmv', enable_lora=True, mem_fraction_static=0.68, attention_backend='aiter' (from the server_args dump).
• Reproducibility: 2 hard failures in the last 15 scheduled runs; 8/15 runs had the first attempt over 4000 ms. Both hard failures were on the 00:2x nightly slot.
• aiter version is a preserved dev build (v0.1.17.dev110+g9127c94a1) — unchanged across passing and failing runs (same Dockerfile default 9127c94a…).
• Scope: this failure appears only in the stage-b-test-1-gpu-large-amd (…, 0) partition (where test_bench_serving_1gpu_part1.py is scheduled). Other failing jobs in this run (multimodal-gen, mi35x, nondeterministic, disaggregation) are unrelated test files.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [HIGH] The 4000 ms threshold is set too tight for this test's actual latency distribution on the MI325 runner; the test is inherently borderline and only survives via the 1-retry wrapper.
  • Supporting: 10/15 runs breach 4000 ms on the first attempt (4023–5233 ms typical); median latency clusters right at the boundary. The test is not "passing cleanly" — it is retry-rescued most of the time.
  • Disconfirming: On Jul 18–19 several runs passed cleanly (<4000 on first try), so raw latency does drift below threshold sometimes; a pure "threshold too low" framing doesn't explain the occasional clean passes vs. the 15641/20742 ms spikes.
  • How to verify: collect the first-try median_e2e_latency_ms across ~30 scheduled runs; if the distribution routinely straddles 4000 ms, the threshold needs raising or the retry count increasing.
• [MEDIUM] Large latency spikes (11000–20742 ms, P99 up to 51 s, Max ITL 18.6 s) are driven by contention between the concurrent LoRA load/unload background task and request serving (adapter swap stalls the batch), amplified on the busier 00:2x nightly slot.
  • Supporting: both hard failures show extreme tail latency and both occurred on the 00:2x slot; the mechanism (background load/unload during serving) directly perturbs decode latency; max_loras_per_batch=1 forces frequent adapter reloads.
  • Disconfirming: the 12:32 slot on the same days recovered on retry with far lower first-try values (4249/4330 ms), so the spike is intermittent rather than deterministic — consistent with runner-level contention rather than a code regression.
  • How to verify: correlate failure timestamps with runner utilization; re-run the test in isolation on an idle MI325 to see if the >10 s spikes reproduce.
• [LOW] A code change in the LoRA load/unload path or scheduler slowed adapter swapping in the last ~week.
  • Supporting: clean first-try passes were more common Jul 17–19 than Jul 20–22.
  • Disconfirming: first-try breaches go back to Jul 15 (4235 ms) and earlier, so no clean "green→red" regression boundary exists; the shallow CI clone prevents commit-range bisect here, and aiter is unchanged.
  • How to verify: bisect the LoRA serving path across the last ~week on an idle runner; not actionable until the contention/threshold hypotheses are ruled out.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for lora latency (43 results, all LoRA feature/perf work — e.g. #27181, #31305 — none targeting this test's threshold) and test_bench_serving (1 result, #27650, unrelated PCG/all-reduce fix). No PR addresses the median_e2e_latency_ms < 4000 marginality.

Suggested Triage Steps

• Sample the first-try median_e2e_latency_ms from write_github_step_summary across the last ~30 scheduled runs; if it routinely straddles 4000 ms, consider raising the AMD threshold or increasing retries for test_online_lora_latency_with_concurrent_adapter_updates (owner decision).
• Re-run test/registered/perf/test_bench_serving_1gpu_part1.py in isolation on an idle linux-mi325-1gpu-sglang runner to check whether the >10 s spikes (20742/15641 ms) reproduce absent runner contention.
• Instrument the LoRA load/unload background task to log per-swap stall time during the benchmark, to confirm/deny that adapter swapping is the tail-latency source.
• If confirmed as environmental/marginal, quarantine or relax the AMD threshold for this specific assertion rather than blocking the whole stage-b-test-1-gpu-large-amd partition on a flaky perf bound.

Status (factual, no priority assignment)

• Persistence: Flaky — 2 hard failures in the last 15 scheduled runs (Jul 21 00:27 and Jul 22 00:26 UTC); first-attempt threshold breaches in 10/15 runs back to Jul 15.
• Scope: 1 test function in 1 test file, in the stage-b-test-1-gpu-large-amd (…, 0) partition of pr-test-amd.yml. Not observed in other workflows.
• Blocked work: Fails the stage-b-test-1-gpu-large-amd partition-0 job on affected runs; blocks clean AMD serving-perf signal for LoRA concurrent-adapter updates on those runs.
• In-flight fix: ❌ none found in open PRs.

stage-b-test-1-gpu-small-amd-mi35x (linux-mi35x-gpu-1) — failed step(s): Run test

Commit Info

• sglang: 24da0e51 — "Warn on small Mamba chunked prefill size" (#30938)
• aiter: 9127c94a (v0.1.17.dev110+g9127c94a1, source: dev/patched CI image preserved — matches Dockerfile default, no override, AITER_COMMIT_OVERRIDE empty)

Failed Tests

Test File	Test Function	Error	Log
test/registered/quant/test_quark_mxfp4.py	TestFP8ToMXFP4MOETP1.test_gsm8k	AssertionError: 0.914 not greater than 0.92 (retry: 0.912 then 0.914)	Log

Other 4 files in the job passed (test_dsa_skip_logits_clean.py, test_verify_splitkv.py, test_mamba_state_scatter_triton.py, test_deepseek_mla_dispatch.py). Job summary: 4/5 passed, exit code 255.

Failure Summary

The single failing test is TestFP8ToMXFP4MOETP1.test_gsm8k (model Qwen/Qwen3-30B-A3B-Instruct-2507-FP8 requantized online to quark_mxfp4, TP=1). It runs a 500-question GSM8K eval and asserts accuracy > 0.92 (strict, assertGreater, test_quark_mxfp4.py:171 / :251). This run scored 0.912 then 0.914 on retry — both marginally below the 0.92 boundary. No crash, no invalid outputs (invalid: 0.000); the server served all 500 requests successfully.

Regression Status

Flaky test — marginal accuracy at a too-tight boundary threshold. The measured accuracy for this quark_mxfp4-requantized MOE model hovers right around the 0.92 cutoff (observed 0.912–0.928 across runs), so run-to-run eval variance (~±0.01) tips it over/under the strict > 0.92 line. This is not a regression introduced by 24da0e51 (#30938 only warns on small Mamba chunked prefill size — unrelated to quantization or GSM8K), and the immediately preceding run passed on a later commit while an earlier run failed.

Recent history of test/registered/quant/test_quark_mxfp4.py (TestFP8ToMXFP4MOETP1.test_gsm8k) in job stage-b-test-1-gpu-small-amd-mi35x (linux-mi35x-gpu-1) (most recent comparable completed scheduled runs: branch=main, event=schedule, completed only):

Date (UTC)	Run	Test File Status	Failed Function	Error / Accuracy
Jul 23 00:32	29969145884	❌ Failed	TestFP8ToMXFP4MOETP1.test_gsm8k	0.912 / 0.914 < 0.92
Jul 22 12:32	29920027108	✅ Passed	—	0.928
Jul 22 00:26	29880220845	❌ Failed	TestFP8ToMXFP4MOETP1.test_gsm8k	0.912 / 0.92 < 0.92
Jul 21 12:32	29830494225	✅ Passed	—	—
Jul 21 00:27	29790347494	✅ Passed	—	—
Jul 20 12:42	29743214297	✅ Passed	—	—
Jul 20 00:28	29709549202	✅ Passed	—	—
Jul 19 12:17	29686709871	✅ Passed	—	—
Jul 19 00:27	29666971143	✅ Passed	—	—

Failure ratio: 2 / 9 completed runs (≈22%). First observed (this window): Jul 22 00:26. Last passing: Jul 22 12:32 (0.928). The two "success" runs adjacent to failures ran and passed this exact test file (verified from logs: Jul 22 12:32 scored 0.928 > 0.92).

Failure Cluster

GSM8K accuracy marginally below strict threshold (quark_mxfp4 online-requant MOE)

Facts (verified observations)

• Error verbatim: AssertionError: 0.914 not greater than 0.92; first attempt 0.912 not greater than 0.92; retry() exceed maximum number of retries. → FAILED (errors=1, skipped=3); file returned exit code 1; job Process completed with exit code 255.
• The assertion is self.assertGreater(metrics["accuracy"], 0.92) at test_quark_mxfp4.py:171, called with accuracy_threshold=0.92 at :251 for class TestFP8ToMXFP4MOETP1 (Qwen/Qwen3-30B-A3B-Instruct-2507-FP8, --quantization quark_mxfp4, --context-length 3000, TP=1).
• invalid: 0.000 in both attempts — all 500 GSM8K responses parsed; the server served every request (200 OK), no crash/OOM/kernel fault.
• Cross-run accuracy for this test: 0.912/0.914 (Jul 23, fail), 0.928 (Jul 22 12:32, pass), 0.912/0.92 (Jul 22 00:26, fail). The distribution straddles the 0.92 cutoff.
• aiter is unchanged across passing and failing runs (v0.1.17.dev110+g9127c94a1, Dockerfile default preserved; [CI-AITER-CHECK] markers, no override).
• The comment at test_quark_mxfp4.py:250 notes reference (unquantized) accuracy is ~0.948; the quark_mxfp4-requantized model actually scores ~0.91–0.93, i.e. requantization loss puts it right at the threshold.

Hypothesised Causes (with confidence)

• [HIGH] Flaky boundary threshold: the quark_mxfp4-requantized MOE model's GSM8K accuracy naturally varies around 0.91–0.93, and the strict > 0.92 gate is inside that variance band.
  • Supporting: three data points 0.912/0.914 (fail), 0.928 (pass), 0.912/0.92 (fail) all cluster around 0.92; invalid: 0.000 and full request success rule out a functional/kernel regression; the exact-0.92 second attempt on Jul 22 shows the strict-greater comparison itself is the tripwire.
  • Disconfirming: (none found) — no correlated code change, crash, or version delta observed.
  • How to verify: re-run test/registered/quant/test_quark_mxfp4.py several times on an mi35x runner and record the accuracy spread; confirm it straddles 0.92.
• [LOW] A genuine small accuracy regression in the quark_mxfp4 requant / MOE path landing in the recent commit window.
  • Supporting: two of the last few runs failed.
  • Disconfirming: the failures are non-monotonic (pass on Jul 22 12:32 at a later commit than the Jul 22 00:26 failure), and #30938 (the head commit) touches only Mamba chunked-prefill warnings, unrelated to this code path.
  • How to verify: bisect only if the failure rate climbs; compare mean accuracy over ≥5 runs before vs after a suspected commit.

In-flight Fix Check

• In-flight fix: ❌ none found — no open PR targets the TestFP8ToMXFP4MOETP1.test_gsm8k threshold or its flakiness. (Searched is:pr is:open quark_mxfp4 → 2 unrelated PRs #29328, #28734; is:pr is:open mxfp4 gsm8k → 77, none addressing this threshold.)

Suggested Triage Steps

• Re-run the job (or test_quark_mxfp4.py alone) 5+ times on an mi35x runner to confirm the accuracy spread straddles 0.92 and quantify the flake rate.
• If confirmed flaky, consider lowering TestFP8ToMXFP4MOETP1's accuracy_threshold from 0.92 to a value below the observed floor (e.g. 0.90, consistent with the sibling TestOnlineQuantizationMemoryLoadMOE.test_gsm8k which uses 0.89) — or increase num_questions above 500 to shrink eval variance. (Maintainer decides; this is triage, not a directive.)
• If a follow-up shows accuracy trending down (mean well below 0.92 across many runs rather than straddling it), then bisect the quark_mxfp4 requant / AITER MOE MLP path instead of adjusting the threshold.

Status

• Persistence: Intermittent — 2 of last 9 completed scheduled runs failed (Jul 23 00:32 and Jul 22 00:26); passed in between (Jul 22 12:32, 0.928).
• Scope: One test function (TestFP8ToMXFP4MOETP1.test_gsm8k) in one job (stage-b-test-1-gpu-small-amd-mi35x, mi35x 1-GPU). Other 4 files in the job pass.
• Blocked work: Blocks the mi35x 1-gpu-small AMD stage-b signal when it flips red, but no functional/runtime defect is evident (accuracy is only ~0.6–0.8% under the gate).
• In-flight fix: ❌ none found — needs a threshold/variance adjustment PR or an accuracy investigation.

stage-b-test-large-8-gpu-mi35x-disaggregation-amd (linux-mi35x-gpu-8.fabric) — failed step(s): Run test_disaggregation

Commit Info

• sglang: 24da0e51
• aiter: 9127c94a (v0.1.17.dev110+g9127c94a1) — source: dev image detected, rebuild skipped (matches Dockerfile default, AITER_COMMIT_OVERRIDE empty)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/disaggregation/test_disaggregation_pp.py	TestDisaggregationPrefillPPDynamicChunkAccuracy.test_gsm8k	KeyError: 'answer' → Exception: retry() exceed maximum number of retries	Log (step 10)

Within the job, 3/4 test files passed; only test_disaggregation_pp.py failed (Ran 3 tests, FAILED (errors=1)). The other three disaggregation test files (test_disaggregation_basic.py, test_mori_transfer_engine_e2e.py, test_nixl_transfer_engine_e2e.py) passed.

Failure Summary

The 8-GPU MI35x disaggregation job failed because test_disaggregation_pp.py returned exit code 1. The test_gsm8k accuracy test of the TestDisaggregationPrefillPPDynamicChunkAccuracy class (Qwen3-8B, prefill tp2/pp2 --enable-dynamic-chunking --disable-overlap-schedule, decode tp2, aiter backend) raised KeyError: 'answer' inside few_shot_gsm8k.run_eval and exhausted its single retry. The KeyError occurs at few_shot_gsm8k.py:123 (get_answer_value(states[i]["answer"])), i.e. the SGLang-lang answer fork variable was never populated for at least one of the 200 GSM8K requests — a generation returned empty/no output rather than an accuracy-below-threshold assertion.

Regression Status

Flaky test (intermittent, ~40% failure rate; not tied to a green→red commit boundary).

Recent history of test_disaggregation_pp.py in job stage-b-test-large-8-gpu-mi35x-disaggregation-amd (most recent 10 comparable completed scheduled runs: branch=main, event=schedule, completed only):

Date	Run	Test File Status	Failed Function	Error
Jul 23 00:xx	29969145884	❌ Failed	TestDisaggregationPrefillPPDynamicChunkAccuracy.test_gsm8k	KeyError: 'answer'
Jul 22 12:32	29920027108	✅ Passed (4/4)	—	—
Jul 22 00:26	29880220845	❌ Failed	TestDisaggregationPrefillPPDynamicChunkAccuracy.test_gsm8k	KeyError: 'answer'
Jul 21 12:32	29830494225	❌ Failed	TestDisaggregationDecodePPAccuracy.test_gsm8k	KeyError: 'answer'
Jul 21 00:27	29790347494	✅ Passed (4/4)	—	—
Jul 20 12:42	29743214297	✅ Passed (4/4)	—	—
Jul 20 00:28	29709549202	✅ Passed (4/4)	—	—
Jul 19 12:17	29686709871	✅ Passed (4/4)	—	—
Jul 19 00:27	29666971143	❌ Failed	TestDisaggregationPrefillPPAccuracy.test_gsm8k	KeyError: 'answer'
Jul 18 12:17	29644059344	✅ Passed (4/4)	—	—

4 failures / 6 passes over 10 runs. Critically, the failing sub-test rotates across all three PP classes (DynamicChunk, DecodePP, PrefillPPAccuracy) while the symptom (KeyError: 'answer') is identical — the hallmark of a nondeterministic generation flake, not a class-specific bug. First observed failure in this window: Jul 19 00:27; most recent pass: Jul 22 12:32.

Failure Cluster

KeyError: 'answer' in GSM8K eval on PP disaggregation path (empty generation)

Facts (verified observations)

• Job overall: FAILED (errors=1), Ran 3 tests in 319.743s; job-summary Test Summary: 3/4 passed, test_disaggregation_pp.py (exit code 1) (job log lines 29211–29224).
• Verbatim stack (job log ~25596): ERROR: test_gsm8k (__main__.TestDisaggregationPrefillPPDynamicChunkAccuracy) → test_disaggregation_pp.py:198 metrics = run_eval(args) → few_shot_gsm8k.py:123 preds.append(get_answer_value(states[i]["answer"])) → interpreter.py:357 return self.variables[name] → KeyError: 'answer' → common.py:3192 raise Exception("retry() exceed maximum number of retries.").
• The same job's first PP class (TestDisaggregationPrefillPPAccuracy.test_gsm8k, line 104) hit the identical KeyError: 'answer' once but recovered on retry (job log ~19968: "retry() failed once … Will delay 1.92s and retry"), showing the failure is transient/probabilistic within a single run.
• aiter version identical (v0.1.17.dev110+g9127c94a1) across current and all historical runs checked — not a variable.
• The GSM8K progress bar reached 100%|██████████| 200/200 before the KeyError, i.e. requests completed but at least one response carried no answer fork variable (empty/failed generation), not a hang or crash.
• Reproducibility: same KeyError: 'answer' symptom in test_disaggregation_pp.py in runs on Jul 23, Jul 22, Jul 21, Jul 19 — spanning multiple sglang head SHAs and rotating across all 3 PP test classes.

Hypothesised Causes (with confidence)

• [LOW] Nondeterministic empty generation on the prefill/decode PP disaggregation path: an occasional GSM8K request returns no completion, so the SGLang-lang answer fork variable is never set and run_eval KeyErrors.
  • Supporting: identical symptom recurs across ≥4 runs and rotates across all 3 PP classes; the same run recovered the first class on retry, consistent with a per-request probabilistic drop; failure sits at states[i]["answer"] lookup, which is only unset when a request produced no text.
  • Disconfirming: no code-level evidence isolating the empty response; no green→red commit boundary (passes and fails interleave day-to-day on the same code); could alternatively be a test-harness robustness gap (only 1 retry, and run_eval doesn't tolerate a missing answer).
  • How to verify: rerun test_disaggregation_pp.py on MI35x 8-GPU in a loop (10×) to measure the flake rate; on failure, dump the offending states[i] / raw /generate response to confirm it is empty; check prefill/decode server logs for aborted or zero-length outputs at the failing request.
• [LOW] Test-harness fragility rather than a serving bug: few_shot_gsm8k.run_eval indexes states[i]["answer"] without guarding against a missing variable, and the class retries only once.
  • Supporting: the first PP class passed after one retry in this very run — one extra retry might have masked the failure.
  • Disconfirming: even if masked, an empty generation from the server is still a real (if rare) serving defect; hardening the test would hide, not fix, it.
  • How to verify: inspect whether increasing retries/parallel reduces recurrence; if it does, the underlying empty-generation issue still needs a separate fix.

No hypothesis reaches MEDIUM confidence — this is an intermittent flake with no code-level or temporal-commit evidence. A git bisect is not meaningful because passing and failing runs interleave on the same/adjacent SHAs.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for disaggregation pp gsm8k (17 results, all unrelated PP/disagg features) and few_shot_gsm8k answer (1 result, #14884, NPU Qwen3-AWQ, unrelated). No open PR targets this KeyError: 'answer' flake or the PP-disaggregation GSM8K path.

Suggested Triage Steps

• Loop-run test/registered/amd/disaggregation/test_disaggregation_pp.py (~10×) on an MI35x 8-GPU runner to quantify the flake rate and confirm it is not head-SHA-specific.
• On a reproduced failure, log the raw failing states[i] and the corresponding prefill/decode server /generate response to confirm the response is empty and locate where the completion was dropped (prefill→decode KV transfer, or decode early-exit).
• Compare failing vs passing runs' prefill/decode logs for the failing request to see if the empty output correlates with --enable-dynamic-chunking / --disable-overlap-schedule / bootstrap-transfer timing.
• If confirmed to be an occasional empty generation, fix on the serving side; if it is purely a harness gap, harden few_shot_gsm8k.run_eval to skip/flag missing answer states and consider raising the retry count — but do not use that to mask a real empty-generation defect.

Status

• Persistence: Intermittent since at least Jul 19 00:27 UTC; 4 failures in the last 10 completed scheduled runs (~40%), most recent pass Jul 22 12:32 UTC.
• Scope: One job (stage-b-test-large-8-gpu-mi35x-disaggregation-amd) in pr-test-amd.yml; one test file (test_disaggregation_pp.py), symptom rotating across its 3 PP GSM8K classes. Only the PP disaggregation accuracy tests are affected — basic/mori/nixl transfer tests pass.
• Blocked work: Flaky-fails the 8-GPU MI35x disaggregation CI signal roughly every other scheduled run.
• In-flight fix: ❌ none found — needs new investigation/PR.

multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 3) — failed step(s): Run diffusion server tests (1-GPU)

Commit Info

• sglang: 24da0e51
• aiter: 9127c94a (v0.1.17.dev110+g9127c94a1; source: dev image detected, rebuild skipped — Dockerfile default matches)

Failed Tests

Standalone file test/single_test_file/test_update_weights_from_disk.py — result 4 failed, 2 passed, 4 errors.

Test File	Test Function	Error	Log
.../test_update_weights_from_disk.py	TestUpdateWeightsFromDisk::test_update_weights_from_disk_default[Qwen-Image]	assert 400 == 200 (update returned HTTP 400)	Log
.../test_update_weights_from_disk.py	TestUpdateWeightsFromDisk::test_update_weights_specific_modules[Qwen-Image]	AssertionError: Update failed: {...'Failed to update module 'transformer'...'}; assert 400 == 200	Log
.../test_update_weights_from_disk.py	TestUpdateWeightsFromDisk::test_corrupted_weights_rollback[Qwen-Image]	assert (400 == 200)	Log
.../test_update_weights_from_disk.py	TestUpdateWeightsFromDiskWithOffload::test_update_weights_with_offload_enabled[Qwen-Image]	AssertionError: Expected 200, got 400	Log
.../test_update_weights_from_disk.py	TestUpdateWeightsFromDisk::test_update_weights_from_disk_default[FLUX.2-klein-base-4B]	ERROR at setup — AssertionError: No weights dir for transformer in /tmp/sglang_perturbed_vae_* (fixture sanity check)	Log
.../test_update_weights_from_disk.py	TestUpdateWeightsFromDisk::test_update_weights_specific_modules[FLUX.2-klein-base-4B]	ERROR at setup — same fixture assertion	Log
.../test_update_weights_from_disk.py	TestUpdateWeightsFromDisk::test_update_weights_rejects_invalid_requests[FLUX.2-klein-base-4B]	ERROR at setup — same fixture assertion	Log
.../test_update_weights_from_disk.py	TestUpdateWeightsFromDisk::test_corrupted_weights_rollback[FLUX.2-klein-base-4B]	ERROR at setup — same fixture assertion	Log

Failure Summary

The standalone diffusion test test_update_weights_from_disk.py (shard 3) failed with 4 failures + 4 setup errors. For Qwen-Image, POST /update_weights_from_disk returns HTTP 400 because the server-side weight loader raises a bare AssertionError on a transformer weight. For FLUX.2-klein-base-4B, all four tests error during the diffusion_server_no_offload fixture's checksum sanity check because the perturbed/clone model dir has no transformer weights directory.

Regression Status

Never-passed (no green run found) — this test file has failed in every comparable completed scheduled run in the query window (2026-07-13 → 2026-07-23, event=schedule, branch=main).

Recent history of test_update_weights_from_disk.py in job multimodal-gen-test-1-gpu-amd (…, 3) (comparable completed scheduled runs; partition is dynamic so historical logs were located by test-file content, not shard number):

Date	Run	Job	Test File Status	Signature
Jul 23	29969145884	job	❌ Failed	4 failed, 2 passed, 4 errors
Jul 22	29920027108	job	❌ Failed	4 failed, 2 passed, 4 errors
Jul 22	29880220845	job	❌ Failed	4 failed, 2 passed, 4 errors
Jul 21	29830494225	job	❌ Failed	4 failed, 2 passed, 4 errors
Jul 19	29666971143	job	❌ Failed	5 failed, 1 passed, 4 errors
Jul 15	29415344552	job	❌ Failed	5 failed, 1 passed, 4 errors
Jul 13	29215394044	job	❌ Failed	5 failed, 1 passed, 4 errors

No passing run for this test file exists in the query window. The signature improved from 5 failed / 1 passed to 4 failed / 2 passed between Jul 19 and Jul 21 (one previously-failing test began passing — consistent with the InferenceMode-500 fix #31263 referenced in the tracking issue). Test file was introduced 2026-06-27 by #28762.

Failure Cluster

Diffusion update_weights_from_disk transformer reload — shape-mismatch AssertionError → HTTP 400 (AMD/ROCm).

Facts (verified observations)

• Server-side stack trace (Qwen-Image), verbatim from log: weights_updater.py:434 _apply_weights → :203 _load_weights_into_module → :295 load_weights_into_model → linear.py:410 weight_loader → assert param_data.shape == loaded_weight.shape → AssertionError (log line 12041+).
• The bare AssertionError propagates as Failed to update module 'transformer': . All modules rolled back and the endpoint returns 400 Bad Request (log line 12053).
• linear.py:410 is exactly assert param_data.shape == loaded_weight.shape in ColumnParallelLinear.weight_loader (verified in source at python/sglang/multimodal_gen/runtime/layers/linear.py:410).
• FLUX.2 setup errors originate in the fixture sanity check test_update_weights_from_disk.py:408-413 calling _compute_checksum_from_disk (:220 assert os.path.exists(weights_dir)); the perturbed clone dir /tmp/sglang_perturbed_vae_* lacks a transformer subdir (verified in source and log).
• Reproducibility: identical 4 failed / 2 passed / 4 errors signature in the 4 most recent runs; broader 5→4 failed pattern across all 7 sampled runs back to Jul 13. Affects both Qwen-Image and FLUX.2-klein-base-4B.
• aiter is unchanged (Dockerfile-default dev build v0.1.17.dev110); AITER_COMMIT_OVERRIDE empty. This is a diffusion weight-loading path, not an attention/kernel path.

Hypothesised Causes (with confidence)

• [HIGH] The diffusion update_weights_from_disk reload path does not reconcile the diffusers-format checkpoint layout with sglang's transformer parameter shapes, so a transformer weight trips assert param_data.shape == loaded_weight.shape (linear.py:410) and the update returns HTTP 400.
  • Supporting: exact stack trace in the log matches; maintainer-authored tracking issue #31924 documents this precise root cause ("diffusers checkpoint layout not reconciled with sglang params … bare AssertionError … HTTP 400") and reproduces it on this same job/shard for both Qwen-Image and FLUX.2; failure is deterministic across every run.
  • Disconfirming: (none found) — the FLUX.2 rows surface as setup ERRORs rather than the 400 assertion, but the tracking issue attributes both models to the same underlying transformer-reload defect; the FLUX.2 fixture never reaches the update call because its pre-check trips first.
• [LOW] FLUX.2 setup errors could additionally reflect a model-layout/_DIFFERING_MODULES mismatch (perturbed clone missing a transformer dir) independent of the reload bug.
  • Supporting: the FLUX.2 error is a fixture os.path.exists assertion, not the weight-loader assertion.
  • Disconfirming: #31924 explicitly names FLUX.2-klein-base-4B as a repro of the same transformer-reload issue; the Qwen-Image path (same fixture) completes setup and fails later, so the fixture logic itself is not fundamentally broken.
  • How to verify: inspect the on-disk module layout of the FLUX.2 snapshot vs _DIFFERING_MODULES on an MI325 container.

In-flight Fix Check

• In-flight fix: ✅ #31925 (open since 2026-07-21) — [AMD] Skip test_update_weights_from_disk on ROCm pending reload fix (#31924); aligns AMD with NVIDIA (which already skips this standalone file). Merging this un-reds the shard while the real fix is developed. Chase reviewers; do NOT open a duplicate.
• Tracking issue: #31924 — full root-cause writeup for the real fix.
• Possibly related to the real fix: #30761 / #30749 [RL] update_weights_from_disk: load quantized checkpoints like initial loading.
• Search used: repo:sgl-project/sglang is:pr is:open update_weights_from_disk.

Suggested Triage Steps

• Merge the CI mitigation #31925 to stop the recurring red on multimodal-gen-test-1-gpu-amd shard 3, then track the real fix in #31924.
• For the real fix: reconcile the diffusers→sglang transformer parameter layout in the reload path (weights_updater.py load_weights_into_model / linear.py weight_loader) so shapes match, as they do on the initial load path. Compare initial-load weight mapping vs the reload mapping for the transformer module.
• Reproduce on an MI325 container (amd-bot-runner) with SGLANG_CI_MODEL_PAIR=Qwen-Image, calling POST /update_weights_from_disk on the transformer module; capture the mismatching param_data.shape vs loaded_weight.shape to pinpoint the offending parameter.
• Separately confirm whether the FLUX.2 fixture ERROR is fully explained by the same reload defect or needs a _DIFFERING_MODULES/layout adjustment.

Status

• Persistence: Failing every completed scheduled run in the query window — never-passed since at least 2026-07-13 (test file added 2026-06-27). Deterministic, not flaky.
• Scope: multimodal-gen-test-1-gpu-amd shard 3 (standalone test_update_weights_from_disk.py) on AMD/ROCm; both Qwen-Image and FLUX.2-klein-base-4B. NVIDIA does not run this standalone file. Not aiter-related.
• In-flight fix: ✅ #31925 (skip on ROCm) open since 2026-07-21; root cause tracked in #31924.

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-07-23 18:35 UTC)

[amd-bot]

pr-test-amd.yml — Cross-Run Pattern Summary (4 runs in lookback window)

Last scanned (UTC): 2026-07-23 18:38 UTC

Agent assessment

Persistent clusters (fail every run)

1. FLUX.2 ModelOpt-FP8 GEMM HIPBLAS_STATUS_NOT_SUPPORTED on gfx942 — multimodal-gen-test-2-gpu-amd (…,0): 29880 · 29920 · 29969 · 30007. Never-passed (≥8 days, since ≤2026-07-15).

• [FACT] torch._scaled_mm (FP8) at fp8_utils.py:1878 returns HIPBLAS_STATUS_NOT_SUPPORTED for the FLUX.2 transformer on gfx942; disconfirming: none — unambiguous code path + status code. The crash also poisons shard state → hang/runner-kill (BlobNotFound logs).
• In-flight fix: ✅ #28889 (native e4m3fnuz FP8 GEMM for gfx942, open since 2026-06-22) — names this exact case; chase reviewers.

2. Diffusion update_weights_from_disk transformer reload → HTTP 400 / shape mismatch — multimodal-gen-test-1-gpu-amd (…,3): 29880 · 29920 · 29969 · 30007. Never-passed (4 failed/2 passed/4 errors every run) in test_update_weights_from_disk.py.

• [MEDIUM] Model-layout/checkpoint-namespace mismatch for these diffusion pairs (Qwen-Image transformer update returns empty-message 400; FLUX.2 clone missing transformer/ dir); disconfirming: swallowed server exception blocks code-level attribution; FLUX.2 offload variant passes.
• In-flight fix: ❌ none targeting these symptoms (related-only: #31483, #31899).

3. Diffusion 1-GPU server suite hang → runner-kill, no log (BlobNotFound) — multimodal-gen-test-1-gpu-amd (…,0) 30007 + (…,1) 30007 (both shards, all 4 runs). Step 9 freezes in_progress, log blob never uploads → no per-test evidence.

• [LOW] 90-min step timeout vs hard hang in test_server_1_gpu.py; disconfirming: no capturable log — observability must be restored before root-causing. In-flight fix: ❌ none.

Regression candidates (latest run only)

• stage-b-test-2-gpu-large-amd (…,1) — 30007380837: Run test timed out after 45 min, but 7/7 passed at wall 2701.69s (~2s over the timeout-minutes: 45 cap). Prior 14 runs mostly fit (2184s on 07-20, 1973s on 07-18) → marginal capacity overrun, [FACT] not a test failure. Window [e4eea7ce..b98a577f](https://github.com/sgl-project/sglang/compare/e4eea7ce2f...b98a577fbe); check whether test_bench_one_batch_2gpu.py (573→695→1014s) is truly slowing. In-flight fix: ✅ #30225 (2→3 partitions, stalled since 2026-07-06).

Flakes / intermittent

• stage-b-test-1-gpu-small-amd-mi35x — GSM8K/MXFP4 accuracy just below strict threshold (Qwen3-30B-A3B online FP8→MXFP4 requant): 3/4 (29880, 29969, 30007). Borderline eval flake → investigate MXFP4 requant accuracy or relax AMD threshold.
• stage-b-test-large-8-gpu-mi35x-disaggregation-amd — PP-disagg GSM8K KeyError: 'answer' (empty generation): 3/4 (29880, 29969, 30007). [LOW] nondeterministic dropped request in PP KV-transfer; ~40% rate. In-flight fix: ❌ none → investigate.
• stage-b-test-1-gpu-large-amd (…,0) — LoRA concurrent-adapter perf (median_e2e_latency < 4000): 2/4 (29880, 29969); retry-rescued in 10/15 runs. [HIGH] threshold too tight for MI325 → raise threshold/retries or quarantine.
• stage-b-test-1-gpu-small-amd-nondeterministic — reward-model _legacy_load_weights AttributeError: 1/4 (29880 only). ⚠️ [HIGH] genuine backend-agnostic regression from becf252e (#28671, merged 07-21); did not recur in later runs (test rotation / possible fix) — verify it's actually resolved. In-flight fix: ❌ none.
• stage-c-test-large-8-gpu-amd (…,3) — communicator get_server_args AttributeError: 1/4 (29920). ✅ Already reverted by merged #32100 (07-22 18:52) — self-resolved.
• stage-c-test-large-8-gpu-amd-mi35x (…,1) — Dynamo graph break during prefill CUDA-graph capture: 1/4 (29920). Single occurrence; monitor.

Suggested next actions

• Chase reviewers on ✅ #28889 — resolves cluster 1 (2-GPU FLUX.2 FP8, largest persistent blast radius + log-destroying hangs).
• Revive stalled ✅ #30225 (or bump timeout-minutes short-term) for the latest-run 45-min timeout on stage-b-test-2-gpu-large-amd.
• Confirm the reward-model regression (becf252e) is truly gone — rerun test_reward_models.py; if it recurs, needs a new fix (affects all backends).
• Restore observability on the diffusion 1-GPU hang shards (ensure artifact upload survives step kill) before root-causing cluster 3.
• Triage the borderline threshold flakes (mi35x GSM8K, 1-gpu-large LoRA) as a batch: sample the metric distribution and raise/quarantine rather than chasing per-run reds.

---

Runs analysed

Run	Started (UTC)	Failed jobs
29880220845	2026-07-22 00:26	8
29920027108	2026-07-22 13:53	6
29969145884	2026-07-23 00:28	7
30007380837	2026-07-23 12:32	7

Persistent failures (in every run)

• multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 0) — in 4/4 runs (0220845, 0027108, 9145884, 7380837)
• multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 1) — in 4/4 runs (0220845, 0027108, 9145884, 7380837)
• multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 3) — in 4/4 runs (0220845, 0027108, 9145884, 7380837)
• multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 0) — in 4/4 runs (0220845, 0027108, 9145884, 7380837)

Latest-only failures (potential regressions)

• stage-b-test-2-gpu-large-amd (linux-mi325-2gpu-sglang, 1) — in 1/4 runs (7380837)

Flaky / intermittent failures

• stage-b-test-1-gpu-large-amd (linux-mi325-1gpu-sglang, 0) — in 2/4 runs (0220845, 9145884)
• stage-b-test-1-gpu-small-amd-mi35x (linux-mi35x-gpu-1) — in 3/4 runs (0220845, 9145884, 7380837)
• stage-b-test-1-gpu-small-amd-nondeterministic (linux-mi325-1gpu-sglang) — in 1/4 runs (0220845)
• stage-b-test-large-8-gpu-mi35x-disaggregation-amd (linux-mi35x-gpu-8.fabric) — in 3/4 runs (0220845, 9145884, 7380837)
• stage-c-test-large-8-gpu-amd (linux-mi325-8gpu-sglang, 3) — in 1/4 runs (0027108)
• stage-c-test-large-8-gpu-amd-mi35x (linux-mi35x-gpu-8, 1) — in 1/4 runs (0027108)

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-07-23 18:38 UTC)

[amd-bot]

nightly-test-amd-rocm720.yml — Per-Job Analysis (continued 3/5)

nightly-8-gpu-mi35x-deepseek-v4-pro-mtp-rocm720 — failed step(s): Accuracy + MTP Accept Length Test MI35x ROCm 7.2 (8-GPU DeepSeek-V4-Pro FP8 MTP, triton)

Commit Info

• sglang: 0bdd4730a ([CI] Fix failures on main, #32091)
• aiter: 9127c94a1 — v0.1.17.dev110+g9127c94a1 (source: dev image preserved; [CI-AITER-CHECK] reports "Dev/patched version detected → skipping rebuild", AITER_COMMIT_OVERRIDE empty)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/test_deepseek_v4_pro_fp4_tbo_mtp.py	test_gsm8k_tbo_mtp	Exception: retry() exceed maximum number of retries (server crashed mid-eval: HSA_STATUS_ERROR_OUT_OF_RESOURCES … Available Free mem : 0 MB, then ConnectionRefusedError: [Errno 111])	Log

This failure occurred only in the SGLANG_HACK_FLASHMLA_BACKEND=triton sub-run (step 7). The prior SGLANG_HACK_FLASHMLA_BACKEND=unified_kv_triton sub-run (step 6) passed 2/2, and test_deepseek_v4_pro_fp4_mtp.py passed under triton too — so the triton sub-run finished 1/2.

Failure Summary

During the triton FlashMLA config sub-run, the DeepSeek-V4-Pro FP4 DP+TBO+MTP server crashed while serving the GSM8K benchmark (parallel=512, 1319 questions) at ~55% completion. One TP/DP rank aborted with HSA_STATUS_ERROR_OUT_OF_RESOURCES at Available Free mem : 0 MB (GPU memory exhaustion). The surviving ranks then failed a prefill_delayer collective (all_gather_into_tensor → gloo Connection closed by peer), scheduler_0 crashed with exit code -3 (SIGQUIT), and the remaining GSM8K requests returned connection-refused, failing the test after retries.

Regression Status

New regression (as of 2026-07-22).

Recent history of test_deepseek_v4_pro_fp4_tbo_mtp.py (test_gsm8k_tbo_mtp, triton FlashMLA sub-run) in job nightly-8-gpu-mi35x-deepseek-v4-pro-mtp-rocm720 (comparable event=schedule, branch=main, completed runs):

Date	Run	Job	Test File Status (triton cfg)	Error
Jul 22	29944173150	job	❌ Failed	HSA out-of-resources → server crash → conn refused
Jul 21	29854963573	job	✅ Passed	—
Jul 19	29697452589	job	⚠️ Canceled (triton sub-run soft watchdog timeout 300s, all TP ranks)	operation canceled
Jul 19	29654521083	job	✅ Passed	—
Jul 18	29601455852	job	✅ Passed (tbo_mtp; fp4_mtp failed separately here)	—

First observed failure for this test: 2026-07-22. Last known pass: 2026-07-21. The workflow's overall "failure" conclusions on earlier dates were driven by other test files/jobs, not this test. Note the triton config has shown prior fragility at high concurrency (Jul 19 soft-watchdog timeout).

Failure Cluster

GPU out-of-resources (HSA free-mem 0 MB) crash during TBO GSM8K prefill under triton FlashMLA backend

Facts (verified observations)

• :0:rocdevice.cpp :3582: … HSA_STATUS_ERROR_OUT_OF_RESOURCES: The runtime failed to allocate the necessary resources … Available Free mem : 0 MB, followed by Fatal Python error: Aborted on multiple DP ranks (step:7:83252).
• Scheduler crash stack (DP2 TP2): schedule_policy.py:851 add_chunked_req → prefill_delayer.py:360/154/322 → torch.distributed.all_gather_into_tensor → RuntimeError: … Connection closed by peer (step:7:83569).
• Subprocess scheduler_0 (pid=25912) crashed with exit code -3. Triggering SIGQUIT (step:7:83612).
• Crash occurred under heavy prefill load: repeated Prefill batch … #running-req: 55–62, #queue-req: 5–10, #pending-token: ~10000 immediately before abort.
• Same test + same sglang commit passed under unified_kv_triton (step 6) in this run → failure is triton-FlashMLA-config-specific.
• Test launches with --enable-prefill-delayer --enable-two-batch-overlap --enable-dp-attention --tp 8 --dp 8 --attention-backend dsv4 --kv-cache-dtype fp8_e4m3 + GPU_MAX_HW_QUEUES=5 (test file, lines 47/71-90).
• Regression window 9a6c96083f..0bdd4730a = 58 commits (compare).

Hypothesised Causes

• [MEDIUM] #31835 (d03c8cee8 "Negotiate PrefillDelayer only after KV-budget admission checks") changed prefill admission ordering, increasing peak KV/prefill memory pressure.
  • Supporting: modifies exactly the two files in the crash stack (prefill_delayer.py + schedule_policy.py add_chunked_req); merged 2026-07-21 19:07 UTC — after the last passing scheduled run (started 17:54 UTC, head 9a6c96083f, confirmed NOT an ancestor) and before this failing run; the test explicitly runs --enable-prefill-delayer; crash happens during prefill admission at high concurrency where free mem hits 0 MB.
  • Disconfirming: the first observable error is a GPU OOM abort on a rank, and the all_gather in the stack is a collective wait whose peer had already died — proximity is not proof. The same code passed under unified_kv_triton in this run, so #31835 is at most an interaction with the higher-memory triton backend, not a standalone regression. Only 1 failing run so far; a borderline/flaky OOM is not excluded (the triton sub-run soft-watchdog-timed-out on Jul 19 too, before #31835).
  • How to verify: git revert d03c8cee8 on a branch and re-run the triton sub-run of nightly-amd-8-gpu-mi35x-deepseek-v4-pro-mtp; or run once with --enable-prefill-delayer removed while keeping SGLANG_HACK_FLASHMLA_BACKEND=triton.
• [LOW] triton FlashMLA backend has a larger memory footprint than unified_kv_triton, leaving no headroom at parallel=512.
  • Supporting: failure is exclusively in the triton sub-run; unified_kv_triton passed the identical workload.
  • Disconfirming: triton sub-run passed cleanly on Jul 21/19/18, so the backend alone was sufficient until this run.
  • How to verify: re-run at reduced parallel/mem_fraction_static, or compare peak HBM between the two backends.

In-flight Fix Check

• Suspected cause #31835 is already merged — no existing fix PR; a revert/adjustment would be new work.
• In-flight fix (mitigation candidate): #31450 "[AMD] Fix DeepSeek-V4 FP4 MoE expert memory bloat" (open since 2026-07-16) — reduces DeepSeek-V4 FP4 memory footprint on AMD; could relieve the OOM headroom but is not a targeted fix for this crash. (searched is:pr is:open flashmla triton and deepseek-v4-pro memory.)
• No open PR found specifically for prefill-delayer OOM / test_deepseek_v4_pro_fp4_tbo_mtp.

Suggested Triage Steps

• Re-run the failing job once to establish flaky-vs-deterministic (only 1 failure observed; the same config passed 3 of the last 4 completed runs).
• A/B test #31835: revert d03c8cee8 on a branch and re-run only the SGLANG_HACK_FLASHMLA_BACKEND=triton sub-run of nightly-amd-8-gpu-mi35x-deepseek-v4-pro-mtp on MI35x ROCm 7.2.
• If revert doesn't help, bisect the 58-commit window 9a6c96083f..0bdd4730a focusing on KV-cache/prefill/memory changes.
• If confirmed a real memory-headroom regression, consider lowering mem_fraction_static/parallel for the triton sub-run, or landing mitigation #31450.

Status

• Persistence: First failure 2026-07-22 for this test under triton; passed as recently as 2026-07-21. 1 failing completed run so far.
• Scope: 1 test file (test_deepseek_v4_pro_fp4_tbo_mtp.py), 1 job, triton FlashMLA sub-run only; unified_kv_triton sub-run and the sibling fp4_mtp test are green.
• Blocked work: DeepSeek-V4-Pro FP4 DP+TBO+MTP GSM8K accuracy signal under the triton FlashMLA backend on MI35x ROCm 7.2.
• In-flight fix: no direct fix; suspected cause #31835 already merged; mitigation candidate #31450 open since 2026-07-16.

nightly-8-gpu-mi35x-deepseek-r1-mxfp4-tp4-rocm720 — failed step(s): Accuracy Test MI35x ROCm 7.2 (DeepSeek-R1-MXFP4 TP4 MTP)

Commit Info

• sglang: 0bdd4730af8c215545821049ea9f794519c13fa8
• aiter: 9127c94a18e4398e1eba91f6639e910f0994ad02 (v0.1.17.dev110+g9127c94a1, source: dev image preserved — AITER_COMMIT_OVERRIDE empty, Dev/patched version detected → skip rebuild)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi35x/test_deepseek_r1_mxfp4_tp4_mtp_mi35x.py	test_gsm8k (TestDeepSeekR1MXFP4TP4MTPMI35x)	AssertionError: 0.849128127369219 not greater than or equal to 0.944	Log — step 7

Failure Summary

The MI35x DeepSeek-R1-MXFP4 TP4 EAGLE/MTP GSM8K accuracy test failed: measured accuracy=0.849 (invalid=0.002) against threshold 0.944. The sister non-MTP test in the same job (step 6, test_deepseek_r1_mxfp4_tp4_mi35x.py) passed with accuracy=0.954. The only configuration difference between the passing and failing test is the addition of EAGLE speculative decoding (--speculative-algorithm EAGLE --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4, --attention-backend aiter).

Regression Status

Known recurring failure / Never-passed in queryable window. The MTP step has failed in every completed comparable scheduled run in the lookback window (2026-07-13 → 2026-07-23) with the same ~0.85 accuracy symptom, while the non-MTP base step passes. No green run for this test found in the window.

Recent history of test_deepseek_r1_mxfp4_tp4_mtp_mi35x.py (test_gsm8k) in job nightly-8-gpu-mi35x-deepseek-r1-mxfp4-tp4-rocm720 (most recent comparable completed scheduled runs: branch=main, event=schedule, completed only):

Date	Run	Job	Test File Status	MTP Accuracy	Base (step 6)	Error
Jul 23	29944173150	job	❌ Failed	0.849	✅ 0.954	0.849 < 0.944
Jul 21	29854963573	job	❌ Failed	0.858	✅ 0.945	0.858 < 0.944
Jul 19	29697452589	job	❌ Failed	—	✅	assertion
Jul 18	29654521083	job	❌ Failed	—	✅	assertion
Jul 18	29601455852	job	❌ Failed	0.854	✅ 0.945	0.854 < 0.944
Jul 17	29521474931	job	⏭️ Cancelled	—	—	job cancelled
Jul 14	29355338821	job	❌ Failed	—	❌ (base also failed)	assertion
Jul 13	29273096587	job	❌ Failed	—	❌ (base also failed)	assertion

First observed MTP failure in window: at least 2026-07-13 (lookback range 07-13 → 07-23). No passing MTP run in window. Base step is green from 07-18 onward, so the base model quality is healthy; the MTP path specifically regresses accuracy.

Failure Cluster

EAGLE/MTP speculative-decoding accuracy drop on DeepSeek-R1-MXFP4 TP4 (MI35x / gfx950, aiter backend)

Facts (verified observations, no interpretation)

• MTP run (step 7): accuracy=0.849 invalid=0.002 latency=160.3s; assertion 0.849128127369219 not greater than or equal to 0.944 at test_deepseek_r1_mxfp4_tp4_mtp_mi35x.py:197 (job log ~01:26:57).
• Sister non-MTP run (step 6): accuracy=0.954 invalid=0.000 latency=145.2s → OK (job log ~01:05:54).
• Both tests use the same model (amd/DeepSeek-R1-MXFP4-Preview), TP=4, --attention-backend aiter, --kv-cache-dtype fp8_e4m3, greedy (temperature=0). The only delta is EAGLE speculative decoding args in the MTP test (test_deepseek_r1_mxfp4_tp4_mtp_mi35x.py:155-161).
• Runner GPU_ARCH=mi35x, Dockerfile gfx950 block; aiter v0.1.17.dev110+g9127c94a1 (dev image, not overridden).
• Failing MTP decode telemetry shows low/variable acceptance: accept len ~2.69 falling to ~1.6, accept rate ~0.56 falling to 0.19–0.27 (job log ~01:26:52–01:26:56).
• Same symptom reproduced across runs: 07-23 acc 0.849, 07-21 acc 0.858, 07-18 acc 0.854 — all with base step passing (~0.945–0.954). [FACT]
• aiter version unchanged (dev image) across passing base and failing MTP steps within the same job → not an aiter-version delta between the two steps. [FACT]

Hypothesised Causes (with confidence, with disconfirming evidence)

• [MEDIUM] EAGLE/MTP speculative-decoding correctness bug in the aiter-backend verify/draft path on gfx950. Since EAGLE with greedy decoding and target-model verification should be near-lossless, a ~10-point accuracy drop vs. the identical non-speculative config points to wrong tokens being emitted (verification/packing/routing), not merely a tight threshold.
  • Supporting: the sole config difference between the 0.954 passing step and the 0.849 failing step is EAGLE; same model, same aiter attention backend, same kv-cache dtype, greedy decoding. Low/erratic accept rate (down to 0.19) is consistent with a degraded draft/verify path.
  • Disconfirming: no code-level stack trace was captured (failure is a numerical accuracy assertion, not a crash), so the exact faulty kernel/function is not directly proven from this log. Threshold-too-high cannot be fully excluded without a known-good historical MTP baseline (none exists in the queryable window).
  • How to verify: re-run the MTP recipe with --attention-backend triton (non-aiter) and compare accuracy; A/B toggle EAGLE off vs on at fixed seed; bisect the aiter dev build vs. an older aiter on the same sglang SHA.
• [LOW] Accuracy threshold 0.944 (test_deepseek_r1_mxfp4_tp4_mtp_mi35x.py:38) set too high for the achievable MTP accuracy on this config.
  • Supporting: MTP accuracy is stable (~0.85) across runs, suggesting a consistent baseline rather than a flake.
  • Disconfirming: base non-MTP accuracy is ~0.95 and MTP is expected to match it (lossless verification), so 0.944 is a reasonable expectation — a 0.85 result more likely indicates a real defect than a mis-set threshold.
  • How to verify: confirm with model owners what MTP accuracy this recipe achieved when the threshold was introduced.
• [LOW] aiter v0.1.17.dev110+g9127c94a1 dev build regressed the spec-decode kernels on gfx950.
  • Supporting: only the MTP/aiter spec path is affected; base path (also aiter, no spec) is healthy.
  • Disconfirming: cannot attribute without an aiter A/B; the failure predates the queryable window so no aiter-version transition is pinned.
  • How to verify: pin an earlier aiter commit and re-run the MTP test on the same sglang SHA.

No hypothesis reaches HIGH; the base-vs-MTP isolation is strong (MEDIUM) but no direct code-level stack evidence exists. A/B disconfirmation (backend swap / EAGLE off) is the cheapest next step.

In-flight Fix Check

No open PR is a confirmed fix for this specific DeepSeek-R1-MXFP4 TP4 MTP accuracy failure. Candidate related (AMD spec-decode / gfx950) open PRs to check — treat as leads, not confirmed fixes:

• In-flight fix (candidate): #29876 (open since 2026-07-02) — "Fix EAGLE speculative decoding with DSA backend on gfx950 (MI355X)" — gfx950 EAGLE; check whether it covers the aiter backend path used here.
• In-flight fix (candidate): #30105 (open since 2026-07-04) — "Fix aiter GQA packing + split-KV routing in NEXTN spec attention (verify & draft_extend)" — directly touches aiter spec verify/draft path.
• In-flight fix (candidate): #30407 (open since 2026-07-07) — "Skip split-KV EAGLE-verify fast path for MLA" — EAGLE-verify on MLA (DeepSeek is MLA), though scoped to a GPU fault rather than accuracy.

(API queried: search/issues?q=repo:sgl-project/sglang+is:pr+is:open+<kw> with keywords MXFP4/MTP, DeepSeek-R1-MXFP4, EAGLE/MI35x, speculative/accuracy/ROCm.)

Suggested Triage Steps

• A/B disconfirmation: re-run test_deepseek_r1_mxfp4_tp4_mtp_mi35x.py with --attention-backend triton (and separately with EAGLE disabled) on MI35x; if triton restores ~0.95, the defect is in the aiter spec-decode path.
• Compare against the candidate PRs above — apply #30105 and/or #29876 on a branch at SHA 0bdd4730 and re-run the MTP test.
• aiter A/B: pin an earlier aiter commit vs. v0.1.17.dev110+g9127c94a1 on the same sglang SHA to rule the aiter dev build in or out.
• Confirm with model owners whether GSM8K_MTP_ACCURACY_THRESHOLD=0.944 (test_deepseek_r1_mxfp4_tp4_mtp_mi35x.py:38) matches a known-good historical MTP result; if it never achieved that, the test may be Never-passed rather than a fresh regression.

Status (factual, no priority assignment)

• Persistence: MTP step failing in every completed comparable scheduled run in the queryable window (2026-07-13 → 2026-07-23), same ~0.85 accuracy symptom; no green MTP run found in window.
• Scope: 1 test file / 1 test function in 1 job (nightly-8-gpu-mi35x-deepseek-r1-mxfp4-tp4-rocm720), MI35x/gfx950 ROCm 7.2 only. The non-MTP sister step is green — isolated to the EAGLE/MTP speculative-decoding path.
• Blocked work: DeepSeek-R1-MXFP4 EAGLE/MTP accuracy signal on MI35x (nightly).
• In-flight fix: no confirmed fix; candidate related PRs #29876, #30105, #30407 — verify before opening a duplicate.

nightly-accuracy-8-gpu-mi35x-rocm720 — failed step(s): Accuracy Test MI35x ROCm 7.2 (8-GPU GPT-OSS)

Commit Info

• sglang: d0b96898
• aiter: 9127c94a — v0.1.17.dev110+g9127c94a1 (source: dev/patched image preserved; AITER_COMMIT_OVERRIDE empty, no rebuild — per [CI-AITER-CHECK] markers)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi35x/test_qwen3_moe_eval_mi35x.py	test_qwen3_moe_accuracy (model Qwen/Qwen3-30B-A3B)	Memory access fault by GPU node-X … Reason: Unknown → GPU core dump failed → Fatal Python error: Aborted; test raises AssertionError: Failed models: ['Qwen/Qwen3-30B-A3B']	Log

The other 3 accuracy tests in this job passed: test_gpt_oss_eval_mi35x.py (0.480/0.47, 0.835/0.79), test_gpt_oss_w4a8_mxfp4_eval_mi35x.py (0.815/0.79), test_qwen3_coder_next_eval_mi35x.py (0.960/0.9). Overall job result: Test Summary: 3/4 passed, Process completed with exit code 255.

Failure Summary

The unquantized bf16 Qwen/Qwen3-30B-A3B MoE server (TP8, SGLANG_USE_AITER=1) started and completed CUDA graph capture successfully (20/20 batches), then hit a Memory access fault by GPU node-* on all 8 ranks while serving the first real GSM8K request (a batch of M:789 tokens). Every rank aborted, the server died (SIGQUIT received … one child failed), the accuracy harness recorded the model as N/A ERROR, and test_qwen3_moe_accuracy failed its assertion. This is a hard GPU crash, not an accuracy-threshold miss.

Regression Status

Known recurring failure / Never-passed in queryable window. No green run for this test file was found in the queryable scheduled history (2026-07-03 → 2026-07-23).

Recent history of test_qwen3_moe_eval_mi35x.py (test_qwen3_moe_accuracy) in job nightly-accuracy-8-gpu-mi35x-rocm720 (most recent comparable completed scheduled runs: branch=main, event=schedule; cancelled runs excluded as the job did not run to completion):

Date	Run	Test File Status	Failed Function	Error
Jul 22	29944173150	❌ Failed	test_qwen3_moe_accuracy	Memory access fault → AssertionError
Jul 21	29854963573	❌ Failed	test_qwen3_moe_accuracy	Memory access fault → AssertionError
Jul 19	29697452589	❌ Failed	test_qwen3_moe_accuracy	Memory access fault → AssertionError
Jul 18	29654521083	❌ Failed	test_qwen3_moe_accuracy	Memory access fault → AssertionError
Jul 17	29601455852	❌ Failed	test_qwen3_moe_accuracy	Memory access fault → AssertionError
Jul 14	29355338821	❌ Failed	test_qwen3_moe_accuracy	Fatal Python error: Aborted (crash)
Jul 13	29273096587	❌ Failed	test_qwen3_moe_accuracy	crash / abort
Jul 5	28749501468	❌ Failed	test_qwen3_moe_accuracy	AssertionError: Qwen/Qwen3-30B-A3B
Jul 4	28714556066	❌ Failed	test_qwen3_moe_accuracy	AssertionError: Qwen/Qwen3-30B-A3B (also gpt-oss-20b)

First observed failure for this test file: at/before 2026-07-03 (earliest queryable run). Last known passing date: none found in the 20-day window. Runs on Jul 20 (29766667696) and Jul 16 (29521474931) were cancelled and excluded.

Failure Cluster

GPU memory access fault on Qwen3-30B-A3B MoE forward pass (post-graph-capture, first inference request), MI35x/ROCm 7.2, TP8, aiter enabled.

Facts (verified observations, no interpretation)

• CUDA graph capture reached Capturing batches … 100%|██| 20/20 (job.log:20875-20876) before the fault — the crash is not during capture.
• Fault appears on all 8 ranks simultaneously: Memory access fault by GPU node-{2,4,5,6,7} … Reason: Unknown (job.log:20996-21000), immediately after a burst of M:789 bf16 GEMMs (N:768/2048/128, K:2048/512) routed to using torch solution:0 (job.log:20972-20995) and MoE topksoftmax_4x128x8_bf16 / moe_sum kernels.
• Followed by GPU core dump failed (no /coredumps dir), Fatal Python error: Aborted (job.log:21001-21003), then SIGQUIT received … one child failed (job.log:21326) and TCPStore Connection reset by peer teardown across ranks.
• Test config: Qwen/Qwen3-30B-A3B, bf16, tp_size=8, --mem-fraction-static 0.8, --max-running-requests 128, SGLANG_USE_AITER=1, threshold 0.75. Test source comment notes "aiter CK can't run the TP-sharded intermediate, e.g. 768 // 8" → runner routes unquantized Qwen3 MoE to Triton on ROCm.
• MoE default-config warnings for E=128,N=96 (= 768/8) MI355X configs are emitted just before the fault (job.log:20314-20351).
• Reproducibility: identical symptom in every completed scheduled run examined from Jul 3 → Jul 23 (9 runs above); the other 3 models in the same job pass every time, so this is isolated to Qwen3-30B-A3B, not a job-wide/runner-wide failure.
• aiter version is a preserved dev image (v0.1.17.dev110+g9127c94a1); no override was applied.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [LOW] Illegal memory access in a bf16 GEMM / MoE kernel on the TP8-sharded Qwen3-30B-A3B intermediate shape (N=768 → 768/8=96) during the first decode/prefill of a real request.
  • Supporting: fault fires right after M:789 GEMMs with N:768 and E=128,N=96 MoE configs; test source explicitly flags the 768 // 8 TP-sharded shape as problematic for aiter CK; the 3 non-MoE / differently-shaped models in the same job pass.
  • Disconfirming: Reason: Unknown gives no offending kernel; aiter log-line proximity is not causation (a GEMM prints immediately before the next op that may actually fault); no stack trace pinpoints the kernel; capture (which exercises the same kernels) succeeded, so the trigger is input-shape/runtime-state specific, not the kernel per se.
  • How to verify: re-run test_qwen3_moe_eval_mi35x.py on MI35x/ROCm7.2 with AMD_LOG_LEVEL=3 / HSA_ENABLE_SDMA=0 and --attention-backend/MoE-runner variations; bisect Triton-vs-aiter MoE path via env toggles to isolate the faulting op.
• [LOW] Persistent AMD-specific port issue for this model on this runner (never validated green).
  • Supporting: no passing run exists in the 20-day queryable window; failure is deterministic and model-isolated.
  • Disconfirming: cannot point to an introducing commit because there is no green baseline to bisect from.
• Not attributable to the aiter delta: this is not an AITER Scout run (no override), aiter is unchanged (preserved dev image) across all failing runs, so the aiter version does not explain a regression.

No hypothesis reaches MEDIUM: there is no green baseline in the window to bisect, and the fault reason is Unknown with no kernel-level stack trace.

In-flight Fix Check

• In-flight fix: ⚠️ partial — #31405 "[AMD] Gate Qwen3-30B-A3B TP2 qkv/o_proj CK block-FP8 GEMM shapes to Triton (ROCm 7.0)" targets the same model's AMD GEMM-shape gating, but scoped to TP2 / block-FP8 / ROCm 7.0, whereas this failure is bf16 / TP8 / ROCm 7.2 / mi35x — likely does not cover this path. Worth checking with its author whether the TP8 bf16 shape needs the same Triton gating.
• In-flight fix: ❌ none found for the exact "memory access fault on Qwen3-30B-A3B mi35x TP8" signature (searches: Qwen3-30B-A3B mi35x, qwen3_moe mi35x accuracy → 0 direct matches). Needs new PR or investigation.

Suggested Triage Steps

• Reproduce on MI35x/ROCm7.2 in the CI container with Qwen/Qwen3-30B-A3B TP8 SGLANG_USE_AITER=1 and capture AMD_LOG_LEVEL=3 output to identify the faulting kernel/dispatch (fault reason is currently Unknown).
• Toggle the MoE runner path (Triton vs aiter CK for the 768//8=96 intermediate) via env/flags and re-run to confirm whether the fault is in the TP-sharded MoE GEMM referenced in the test's own source comment.
• Ask the author of #31405 whether the TP8 bf16 Qwen3-30B-A3B shape needs the same "gate to Triton" treatment; if so, extend the gating to this config on a branch and re-run this job.
• Since no green run exists in the queryable window, consider quarantining/marking this model on MI35x until the crash is root-caused, to restore signal for the other 3 (passing) accuracy tests in the job.

Status

• Persistence: Failing on every completed scheduled run examined from 2026-07-03 through 2026-07-23 (≥9 runs); no green run in the queryable window (Never-passed here).
• Scope: 1 test file / 1 model (Qwen/Qwen3-30B-A3B) in job nightly-accuracy-8-gpu-mi35x-rocm720; the other 3 models in the same job pass consistently. Not observed to be a runner-wide crash.
• Blocked work: Blocks bf16 Qwen3-30B-A3B MoE accuracy signal on MI35x/ROCm 7.2; job stays red (exit 255) despite 3/4 models passing.
• In-flight fix: ⚠️ #31405 covers the same model but a different config (TP2/FP8/ROCm7.0) — likely not a direct fix; no exact-match PR found.

nightly-2-gpu-mi35x-glm51-mxfp4-rocm720 — failed step(s): N/A

Commit Info

• sglang: d0b96898 — "[Fix] Include disagg prefill waiting queue in FPM (#32122)"
• aiter: N/A — today's job log blob is unavailable (BlobNotFound; the parent run 30031529424 is still queued, so per-job logs are not yet downloadable), so [CI-AITER-CHECK] markers cannot be extracted for this run.

Failed Tests

Test File	Test Function	Error	Log
N/A — accuracy benchmark step (2-GPU GLM-5.1-MXFP4 GSM8K), not a pytest test	N/A	The self-hosted runner lost communication with the server (runner died mid-test)	Job

The failing unit is step 6 "Accuracy Test MI35x ROCm 7.2 (2-GPU GLM-5.1-MXFP4 GSM8K)" — a GSM8K accuracy benchmark, not a discrete pytest function. Step 6 shows in_progress (never completed) and step 12 pending on a completed/failed job, the classic runner-death signature.

Failure Summary

The nightly-2-gpu-mi35x-glm51-mxfp4-rocm720 job failed during step 6, the 2-GPU GLM-5.1-MXFP4 GSM8K accuracy benchmark. Today's failure surfaced as a self-hosted-runner disconnect (GitHub annotation: "The self-hosted runner lost communication with the server"). On historical runs the same step aborts with exit code 134 (SIGABRT) accompanied by HW Exception by GPU node-2 ... reason: GPU Hang — i.e. the GPU on the MI35x 2-GPU runner hangs during this test, and when the hang is severe enough it also takes the runner daemon offline (today's variant).

Regression Status

Known recurring failure / Never-passed in the queryable window. Every completed scheduled run in the last 15 comparable runs failed on step 6 (the GLM-5.1-MXFP4 accuracy test); no green run for this job exists in the window. This is not a regression introduced by today's HEAD d0b96898 (disagg-prefill FPM change — unrelated to GLM MXFP4 accuracy or GPU hang).

Recent history of step 6 (2-GPU GLM-5.1-MXFP4 GSM8K accuracy) in job nightly-2-gpu-mi35x-glm51-mxfp4-rocm720 (most recent 15 comparable completed scheduled runs: branch=main, event=schedule, completed only):

Date	Run	Step 6 Status	Symptom
Jul 23	30031529424	❌ Failed	Runner lost communication (GPU hang killed runner)
Jul 22	29944173150	❌ Failed	Runner lost communication (annotation verified)
Jul 21	29854963573	❌ Failed	Runner lost communication (annotation verified)
Jul 20	29766667696	❌ Failed	Step 6 killed mid-run
Jul 19	29697452589	❌ Failed	Step 6 killed mid-run
Jul 18	29654521083	❌ Failed	Step 6 killed mid-run
Jul 17	29601455852	❌ Failed	Step 6 killed mid-run
Jul 17	29521474931	❌ Failed	exit 134 (SIGABRT)
Jul 15	29438224738	⚪ Cancelled	(excluded — job cancelled)
Jul 14	29355338821	❌ Failed	GPU Hang → exit 134 (verified)
Jul 13	29273096587	❌ Failed	exit 134 (SIGABRT)
Jul 13	29202600186	❌ Failed	exit 134 (SIGABRT)
Jul 11	29162179641	❌ Failed	exit 134 (SIGABRT)
Jul 10	29113137016	❌ Failed	GPU Hang → exit 134 (verified)
Jul 09	29039471605	❌ Failed	exit 134 (SIGABRT)
Jul 08	28964405353	❌ Failed	Failed earlier at step 3 (VRAM clear)

First observed failure: at/before the start of the window (2026-07-08). No passing run found in the 15-run lookback (2026-07-08 → 2026-07-23). Spot-checks further back (Jul 5, 4, 3) are also failure, so this job appears to have never passed recently.

Failure Cluster

GPU Hang on MI35x 2-GPU runner during GLM-5.1-MXFP4 GSM8K accuracy test (surfaces as either exit code 134/HW Exception ... GPU Hang, or, when the runner daemon is also lost, self-hosted runner lost communication).

Facts (verified observations, no interpretation)

• Today's job 89288985793: step 6 in_progress, step 12 pending, job conclusion failure; GitHub annotation = "The self-hosted runner lost communication with the server." (verified via check-runs annotations API).
• Jul 22 89005250224 and Jul 21 88716968459: identical "runner lost communication" annotation (verified).
• Jul 14 87161431322 log tail (verified verbatim): First attempt failed with exit code 134 → Retrying with HF_HUB_OFFLINE=1 → HW Exception by GPU node-2 (Agent handle: ...) reason :GPU Hang → ##[error]Process completed with exit code 134.
• Jul 10 86430008361 log: multiple HW Exception by GPU node-2 ... reason :GPU Hang lines, then exit 134 (verified). Same signature.
• The crash/hang consistently involves GPU node-2 on runner class linux-mi35x-gpu-2-* (2-GPU MI35x); the faulthandler dump on Jul 14 was in the model-loading path (deepseek_v2.py:load_weights → download_weights_from_hf under a filelock), consistent with a hang during model bring-up before/into the accuracy run.
• All 15 comparable completed scheduled runs in the window failed on step 6 (Jul 08 failed even earlier, at step 3 VRAM-clear). Scope is this single job/runner class — no other jobs in this cluster observed here.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [MEDIUM] GPU-level hang on the MI35x 2-GPU runner during GLM-5.1-MXFP4 bring-up/inference, not a code regression from the HEAD commit.
  • Supporting: two independent historical logs (Jul 10, Jul 14) show HW Exception by GPU node-2 ... reason :GPU Hang → exit 134; today/Jul 21/Jul 22 show "runner lost communication", which is the expected outer symptom when a GPU hang wedges the whole node. Failure is confined to one runner class (linux-mi35x-gpu-2-*) and one test.
  • Disconfirming: the exact HIP/kernel frame that hangs is not captured (the faulthandler dump landed in the weight-download path, which may be a red herring vs. the actual compute hang); today's log blob is unavailable so today's precise symptom is inferred from the annotation only.
  • How to verify: reproduce on an MI35x 2-GPU box in the ROCm 7.2 container running the GLM-5.1-MXFP4 TP2 GSM8K command; capture dmesg/rocm-smi and a GPU-side backtrace at hang time.
• [LOW] Pre-existing / never-enabled-correctly test (test or model config is broken for this platform, so it has never passed).
  • Supporting: no green run in the 15-run window; a branch revert-26396-jacob/glm51-mxfp4-tp2-gsm8k-amd-ci exists upstream, suggesting churn around enabling this exact test.
  • Disconfirming: the observed symptom is a hardware GPU hang, which points at a runtime/GPU issue rather than a pure test-harness misconfiguration.
  • How to verify: check when this job was added to nightly-test-amd-rocm720.yml and whether it ever had a green run since introduction.
• [SPECULATION] Regression from today's HEAD d0b96898.
  • Supporting: none.
  • Disconfirming: the failure predates this commit by ~2 weeks (present since ≥ 2026-07-08); the commit touches disagg-prefill FPM, unrelated to GLM MXFP4 accuracy or GPU hang. Ruled out.

In-flight Fix Check

• In-flight fix: ⚠️ partial — #31068 "[AMD] ci: detect and recover hung GPUs in MI35x pre-flight" (open since 2026-07-13) targets exactly the MI35x GPU-hang/recovery symptom, but is a CI-resilience mitigation, not a fix for the underlying GLM-5.1-MXFP4 hang. #32003 "ci(amd): route scheduled jobs to MI300 with model cache" (open since 2026-07-22) could relocate the job off the flaky MI35x runner.
• In-flight fix (root cause): ❌ none found — no open PR directly fixes the GLM-5.1-MXFP4 2-GPU accuracy GPU hang. (Searched GLM+MXFP4 and mi35x open PRs.)

Suggested Triage Steps

• Reproduce the GLM-5.1-MXFP4 TP2 GSM8K accuracy command on an MI35x 2-GPU ROCm 7.2 box; capture dmesg, rocm-smi, and a GPU backtrace at the hang to pin whether the hang is in model load vs. a specific attention/MoE kernel.
• Confirm whether this job has ever had a green run since it was added to nightly-test-amd-rocm720.yml; if never-passed, treat as an enablement/porting bug and consider quarantining on AMD MI35x until fixed.
• Track #31068 (hung-GPU detect/recover) and #32003 (route to MI300) — either would stop the runner-death from masking the real signal; ping reviewers rather than opening duplicates.
• Re-pull today's job log once run 30031529424 reaches completed to confirm today's symptom is the same GPU Hang (currently inferred from the runner-disconnect annotation).

Status (factual, no priority assignment)

• Persistence: Failing on step 6 in every completed scheduled run for ≥15 days (since ≥ 2026-07-08); no green run in the queryable window.
• Scope: 1 job — nightly-2-gpu-mi35x-glm51-mxfp4-rocm720 on runner class linux-mi35x-gpu-2-* in nightly-test-amd-rocm720.yml. No cross-job spread observed in this analysis.
• Blocked work: No GLM-5.1-MXFP4 2-GPU GSM8K accuracy signal on AMD MI35x ROCm 7.2.
• In-flight fix: root-cause fix ❌ none; mitigations #31068 and #32003 open.

nightly-8-gpu-mi35x-deepseek-r1-mxfp4-tp4-rocm720 — failed step(s): Accuracy Test MI35x ROCm 7.2 (DeepSeek-R1-MXFP4 TP4 MTP)

Commit Info

• sglang: d0b968980
• aiter: 9127c94a1 (v0.1.17.dev110+g9127c94a1, source: dev image preserved — matches Dockerfile default for gfx950; verified via [CI-AITER-CHECK] markers at log line 3842-3848)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi35x/test_deepseek_r1_mxfp4_tp4_mtp_mi35x.py	TestDeepSeekR1MXFP4TP4MTPMI35x.test_gsm8k	AssertionError: 0.8635329795299469 not greater than or equal to 0.944	Log

Failure Summary

The GSM8K accuracy gate for the EAGLE/MTP speculative-decoding variant of DeepSeek-R1-MXFP4 (TP4) on MI35x scored accuracy=0.864 (invalid=0.005), below its GSM8K_MTP_ACCURACY_THRESHOLD = 0.944 (test file line 38; assertion at line 197). The non-MTP variant of the same job (step 6) passed the same run at accuracy=0.948. This is a numerical-accuracy assertion failure, not a crash — the server launched, ran all 1319 GSM8K questions, and returned answers.

Regression Status

Never-passed (no green run found). The MTP test file consistently scores in a tight 0.849–0.864 band and has failed every completed scheduled run since it was introduced on 2026-07-01 by #29290 ([AMD] Cover DeepSeek-R1 MXFP4 TP4 MTP nightly CI). The 0.944 threshold has been unchanged since that introducing commit (548f505cc, verified via GitHub contents API). Lookback confirmed failures back to the first post-introduction runs (Jul 3, Jul 5).

Recent history of test_deepseek_r1_mxfp4_tp4_mtp_mi35x.py (test_gsm8k) in job nightly-8-gpu-mi35x-deepseek-r1-mxfp4-tp4-rocm720 (most recent comparable completed scheduled runs: branch=main, event=schedule):

Date	Run	MTP Test Status	MTP acc	non-MTP acc (same job)
Jul 23	30031529424	❌ Failed	0.864	0.948 ✅
Jul 22	29944173150	❌ Failed	0.849	0.954 ✅
Jul 21	29854963573	❌ Failed	0.858	0.945 ✅
Jul 20	29766667696	⏭️ Run cancelled	—	—
Jul 19	29697452589	❌ Failed	0.851	0.952 ✅
Jul 18	29654521083	❌ Failed	0.854	0.946 ✅
Jul 17	29601455852	❌ Failed	0.854	0.945 ✅
Jul 16	29521474931	⏭️ Cancelled	—	—
Jul 14	29355338821	❌ Failed	(no acc — job failed earlier)	❌
Jul 13	29273096587	❌ Failed	(no acc — job failed earlier)	❌

First observed failure: 2026-07-03 (first run after introduction). Last known passing: none.

Failure Cluster

MTP/EAGLE GSM8K accuracy below threshold (DeepSeek-R1-MXFP4 MI35x) — never-passed gate

Facts (verified observations, no interpretation)

• accuracy=0.864 invalid=0.005 latency=164.5s at log line 26475; assertion 0.8635329795299469 not greater than or equal to 0.944 at line 26481-26483 (test_gsm8k, line 197).
• The non-MTP sister step in the same job passed: accuracy=0.948 invalid=0.000 (log line 13779).
• The MTP variant enables EAGLE spec decode: --speculative-algorithm EAGLE --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 (test file lines 154-161).
• GSM8K is run at temperature=0 (greedy; test file line 101).
• Speculative acceptance during the bulk of the run was healthy — the dominant accept len was ~2.70–2.77 (accept rate ~0.55–0.59); the single accept len: 1.30, accept rate: 0.10 line is the trailing final request, not representative.
• MTP acc is extremely stable across runs: 0.849 / 0.851 / 0.854 / 0.854 / 0.858 / 0.864 (Jul 17–23), never ≥ 0.944.
• Threshold 0.944 is unchanged since the file was introduced 2026-07-01 (#29290).
• aiter version identical (dev v0.1.17.dev110+g9127c94a1) across passing non-MTP and failing MTP — not an aiter-version delta.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [MEDIUM] The 0.944 gate was mis-calibrated at introduction — it appears copied from the non-MTP target (~0.95) but the EAGLE/MTP path on this MXFP4/MI35x stack has a real, stable accuracy loss to ~0.85, so the gate could never pass.
  • Supporting: never passed since day one; acc sits in a very tight 0.849–0.864 band across 6+ runs (stable ⇒ not a flake); non-MTP consistently ~0.95.
  • Disconfirming: at temperature=0, EAGLE verification is designed to be lossless (accepted draft tokens must match the target model's greedy token), so a ~10-point accuracy drop should NOT happen if the draft/verify path is correct — which argues the threshold may actually be right and the code is wrong (see next hypothesis).
  • How to verify: run the same server recipe on MI35x, compare per-question MTP vs non-MTP outputs; if outputs are token-identical, the loss is spurious/threshold issue; if they diverge under greedy, it is a correctness bug.
• [MEDIUM] Correctness bug in the EAGLE/MTP draft+verify path for MXFP4 on the aiter attention backend (MI35x), causing accepted tokens to diverge from the target model's greedy output.
  • Supporting: greedy spec decode should be lossless, yet MTP loses ~10 pts vs non-MTP on the identical model/build; loss is reproducible and stable; MXFP4 + SGLANG_ROCM_FUSED_DECODE_MLA + SGLANG_AITER_MLA_PERSIST + fp8 KV cache is a complex spec-decode path.
  • Disconfirming: acceptance rate (~0.57) looks normal, not zero/garbage; no crash or NaN; invalid is only 0.005 — outputs are well-formed, just less accurate.
  • How to verify: A/B the MTP run with --kv-cache-dtype auto (drop fp8 KV) and/or a different attention backend to isolate whether fp8-KV / aiter MLA fused decode degrades verified accuracy; diff greedy token streams MTP vs non-MTP.
• [LOW] fp8 KV cache (--kv-cache-dtype fp8_e4m3) interacting with the draft model amplifies quantization error under speculation.
  • Supporting: MTP uses the same fp8 KV as non-MTP but adds a draft model on top; quantized draft logits could shift acceptance toward slightly-wrong tokens.
  • Disconfirming: non-MTP with the same fp8 KV passes at 0.95, so fp8 KV alone is not the cause.
  • How to verify: rerun MTP with bf16/auto KV cache.

In-flight Fix Check

• In-flight fix: ❌ none found targeting this specific DeepSeek-R1-MXFP4 TP4 MTP accuracy gate. Searched open PRs for MXFP4 MTP (28 open) and EAGLE MXFP4 (36 open); the closest spec-decode-correctness PRs are #30105 ([AMD][Spec] Fix aiter GQA packing + split-KV routing in NEXTN spec attention) and #30407 (Skip split-KV EAGLE-verify fast path for MLA), but both target GPU faults / different symptoms, not this accuracy shortfall — do not assume they fix this.

Suggested Triage Steps

• Decide the gate's intent first: is 0.944 the expected MTP accuracy, or the non-MTP number copied in error? Ping the author of #29290 to confirm the calibrated MTP baseline for this stack.
• Diff greedy (temperature=0) token streams between the MTP and non-MTP variants on the identical build; if they diverge, this is a spec-decode correctness bug (not a threshold problem).
• If divergence confirmed, bisect the EAGLE/MXFP4/aiter-MLA path: A/B toggle --kv-cache-dtype fp8_e4m3 → auto, SGLANG_ROCM_FUSED_DECODE_MLA, and SGLANG_AITER_MLA_PERSIST to isolate which component degrades verified accuracy.
• If the gate is confirmed mis-calibrated (outputs are actually acceptable), the maintainer may lower/re-baseline GSM8K_MTP_ACCURACY_THRESHOLD; until resolved, this never-passed gate is producing permanent red on this nightly.

Status (factual, no priority assignment)

• Persistence: Failing every completed run since introduction on 2026-07-03 (~20 days); never passed. Current run acc 0.864 vs 0.944.
• Scope: One test file (test_deepseek_r1_mxfp4_tp4_mtp_mi35x.py, test_gsm8k) in job nightly-8-gpu-mi35x-deepseek-r1-mxfp4-tp4-rocm720 on nightly-test-amd-rocm720.yml. The non-MTP sibling gate in the same job is healthy (~0.95).
• Blocked work: Keeps this MI35x DeepSeek-R1-MXFP4 nightly job red; masks any future real MTP regression (a gate that is always red can't signal new breakage).
• In-flight fix: ❌ none found — needs author confirmation of intended MTP baseline and/or a spec-decode correctness investigation.

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-07-23 23:02 UTC)

[amd-bot]

pr-test-amd.yml — Per-Job Analysis (continued 5/6)

multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 0) — failed step(s): N/A

Commit Info

• sglang: 24da0e51
• aiter: unknown — today's job log is unavailable (BlobNotFound, see below). The most recent completed sibling run of this job (Jul 18) ran aiter v0.1.17.dev110+g9127c94a1 (dev image preserved; Dockerfile default 9127c94a). aiter is not implicated in this failure.

Failed Tests

Test File	Test Function	Error	Log
test/server/test_server_2_gpu.py	TestDiffusionServerTwoGpu::test_diffusion_generation[flux2_modelopt_fp8_tp2_t2i]	RuntimeError: CUDA error: HIPBLAS_STATUS_NOT_SUPPORTED in torch._scaled_mm (FP8 GEMM)	Job (log lost) · clean repro Jul 18

Note: today's job (part 0) hung and was killed before uploading logs, so the exact failing case cannot be read from today's log. The case above is the deterministic failure of this partition, confirmed in every completed sibling run (see Regression Status). Attribution of today's hang to this case is HIGH, not FACT.

Failure Summary

The multimodal-gen-test-2-gpu-amd part-0 shard runs 25 parametrized cases from test_server_2_gpu.py; the first case, flux2_modelopt_fp8_tp2_t2i (FLUX.2-dev ModelOpt-FP8, TP=2), crashes during warmup on the first FP8 linear. Today the step Run diffusion server tests (2-GPU) started at 01:47:50 UTC and never completed — the job was terminated at 03:14:59 UTC (~87 min) and its log blob is missing (BlobNotFound), the signature of an abnormally-killed runner. Sibling partitions (parts 1 & 2) passed.

Regression Status

Never-passed (no green run found in the 15-run window) — this shard has failed on every comparable scheduled run since at least Jul 15.

Recent history of test/server/test_server_2_gpu.py (test_diffusion_generation[flux2_modelopt_fp8_tp2_t2i]) in job multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 0) (15 most recent comparable completed scheduled runs: branch=main, event=schedule):

Date (UTC)	Run	Test File Status	Detail
Jul 23 00:27	29969145884	❌ Failed (hung, log lost)	step killed ~87m, BlobNotFound
Jul 22 12:32	29920027108	❌ Failed (hung)	stuck on diffusion step ~106m
Jul 22 00:26	29880220845	❌ Failed (hung)	~85m
Jul 21 12:32	29830494225	❌ Failed (hung)	~100m
Jul 21 00:27	29790347494	❌ Failed (hung)	~106m
Jul 20 12:42	29743214297	❌ Failed (hung)	~109m
Jul 20 00:28	29709549202	❌ Failed (hung)	~103m
Jul 19 12:17	29686709871	❌ Failed (killed)	log lost
Jul 19 00:27	29666971143	❌ Failed (hung)	~98m
Jul 18 12:17	29644059344	⚠️ Failed (infra)	died at Start CI container (~10m) — separate infra failure, not the flux2 case
Jul 18 00:25	29623067212	❌ Failed (clean)	flux2...t2i FAILED — 1 failed, 22 passed, 1 skipped, 1 error
Jul 17 12:25	29580198489	❌ Failed (hung)	~90m
Jul 17 00:27	29544840981	❌ Failed (clean)	flux2...t2i FAILED + 17 cascade server-startup errors
Jul 16 12:31	29498496614	❌ Failed (clean)	flux2...t2i FAILED + 17 cascade server-startup errors
Jul 15 12:28	29415344552	❌ Failed (killed)	log lost

First observed failure: ≥ Jul 15 (start of window). No passing run for this shard exists in the queryable window. Per PR #28889 (below), the FLUX.2 ModelOpt-FP8 checkpoint has never served on ROCm.

Failure Cluster

FP8 GEMM HIPBLAS_STATUS_NOT_SUPPORTED on FLUX.2 ModelOpt-FP8 (upstream-tracked as R192). Two observed manifestations, same root: (a) a clean single-case test failure, and (b) a shard hang/runner-kill when the crashed case leaves GPUs/servers in a bad state, cascading into repeated server-startup failures that inflate runtime and eventually kill the runner (today's manifestation).

Facts (verified observations)

• Today's job: step 9 Run diffusion server tests (2-GPU) in_progress, never completed; job conclusion=failure, started=01:38:55, completed=03:14:59 UTC; step 9 ran 01:47:50 → kill (~87 min, under the 150-min step budget). Log endpoint returns BlobNotFound on all retries. (verified via jobs API + 3× log fetch)
• Verbatim error (sibling run 88021852948): RuntimeError: CUDA error: HIPBLAS_STATUS_NOT_SUPPORTED when calling \HIPBLAS_STATUS_NOT_SUPPORTED`, raised at Warmup requests 0%duringdenoising_step_0`.
• Stack trace (verbatim, verified): models/dits/flux_2.py:77 _get_qkv_projections → attn.to_q() → layers/linear.py:426 → layers/quantization/modelopt_quant.py:432 apply → apply_fp8_linear → srt/layers/quantization/fp8_utils.py:1878 → torch._scaled_mm(...).
• Config (verified from server_args log line): model_path=black-forest-labs/FLUX.2-dev, component_paths.transformer=lmsys/flux2-dev-modelopt-fp8-sglang-transformer, tp_size=2, attention_backend=aiter. Runner GPU_ARCH mi30x / gfx942.
• Reproducibility: 15/15 comparable runs failed; the 3 that ran to completion all pinned the same case; ≥6 runs share today's hung/killed signature. Only parts 1 & 2 of this matrix passed.
• Cascade evidence: in 88102981818 and 87621199565, the flux2 crash was followed by 17 FAILED during server startup errors for later cases (1 failed, 6 passed, 17 errors).

Hypothesised Causes (with confidence, with disconfirming evidence)

• [FACT] torch._scaled_mm (FP8 scaled GEMM) for the FLUX.2 ModelOpt-FP8 transformer returns HIPBLAS_STATUS_NOT_SUPPORTED on gfx942 (MI325/CDNA3). Directly observed in the stack trace across 3 completed runs.
  • Supporting: the failing frame is fp8_utils.py:1878 output = torch._scaled_mm(...); only the FP8 flux2 case fails while non-FP8 cases in the same shard pass. AMD-only (hipBLAS-specific status code).
  • Disconfirming: (none found) — the error code and code path are unambiguous.
  • How to verify: run the single case in-container: python3 sglang/multimodal_gen/test/run_suite.py --suite 2-gpu -k flux2_modelopt_fp8_tp2_t2i on an MI325.
• [HIGH] Today's job hang/runner-kill is caused by the same flux2 FP8 crash leaving the 2-GPU servers/GPUs in a bad state, cascading into repeated server-startup hangs until the runner is lost.
  • Supporting: identical partition and case list; deterministic flux2 crash; ≥6 recent runs show the same in-progress-then-killed signature; observed cascade (17 startup errors) in the clean-completing runs.
  • Disconfirming: today's log is BlobNotFound, so the today-specific chain cannot be read directly — hence HIGH, not FACT. (The Jul 18 12:17 run failed at Start CI container for an unrelated infra reason, showing not every red here is the flux2 case.)
  • How to verify: re-run this shard with SGLANG_SKIP_CONSISTENCY/artifact upload and confirm the killed step ends on the flux2 case; inspect runner dmesg/rocm-smi for GPU reset around the kill time.

In-flight Fix Check

• In-flight fix: ✅ #28889 "[AMD] Use native e4m3fnuz fp8 GEMM for FLUX.2 modelopt-FP8 on gfx942" — open (non-draft) since 2026-06-22, last updated 2026-07-21. Its description names this exact job, test case, error, and cluster (R192). Chase reviewers / resolve mergeable_state; do NOT open a duplicate. (search: search/issues?q=...+is:open+HIPBLAS_STATUS_NOT_SUPPORTED and ...+flux2+fp8)

Suggested Triage Steps

• Prioritize landing #28889 (native e4m3fnuz FP8 GEMM for gfx942) — it directly targets the failing torch._scaled_mm path; verify it merges cleanly and re-run the multimodal-gen-test-2-gpu-amd (…, 0) shard.
• Until it lands, consider quarantining flux2_modelopt_fp8_tp2_t2i on AMD (skip/xfail in gpu_cases.py) so the crash stops cascading into shard hangs that kill the runner and destroy logs for the other 24 cases in part 0.
• Investigate why the crashed case takes down the whole shard: the cascade + runner-kill means one unsupported-op crash is masking the signal of every other 2-GPU case; a per-case server teardown/GPU-reset on failure would isolate it.
• If logs continue to be lost, ensure the Upload diffusion failure artifacts step can run even when the test step is killed (it currently never reaches step 10).

Status

• Persistence: Failing on every comparable scheduled run since ≥ 2026-07-15 (Never-passed on AMD in the queryable window; per #28889 the FLUX.2 FP8 checkpoint has never served on ROCm).
• Scope: multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 0) only; sibling parts 1 & 2 pass. When it hangs it also destroys the shard's logs and blocks the other 24 cases in the partition.
• Blocked work: FLUX.2 ModelOpt-FP8 2-GPU serving coverage on AMD MI325.
• In-flight fix: ✅ #28889 (open since 2026-06-22).

stage-b-test-1-gpu-large-amd (linux-mi325-1gpu-sglang, 0) — failed step(s): Run test

Commit Info

• sglang: 24da0e51
• aiter: 9127c94a (v0.1.17.dev110+g9127c94a1, source: dev image preserved — AITER_COMMIT_OVERRIDE empty)

Failed Tests

Test File	Test Function	Error	Log
test/registered/perf/test_bench_serving_1gpu_part1.py	TestBenchServing1GPUPart1.test_online_lora_latency_with_concurrent_adapter_updates	AssertionError: 4666.40 not less than 4000 (retry: 4741.71 not less than 4000)	Log

The other 7 tests in the file passed (Ran 8 tests … FAILED (errors=1)).

Failure Summary

The perf test test_online_lora_latency_with_concurrent_adapter_updates asserts the benchmark's median_e2e_latency_ms < 4000 (test/registered/perf/test_bench_serving_1gpu_part1.py:169). Both the initial attempt (4666.40 ms) and the built-in retry (4741.71 ms) exceeded the 4000 ms threshold, so the job's "Run test" step exited 255. This is a single latency-threshold assertion; no crash, kernel fault, or import error occurred.

Regression Status

Flaky test — intermittent threshold overshoot, not a clean regression.

Recent history of test_bench_serving_1gpu_part1.py (test_online_lora_latency_with_concurrent_adapter_updates) in job stage-b-test-1-gpu-large-amd (linux-mi325-1gpu-sglang, 0) (most recent 15 comparable completed scheduled runs: branch=main, event=schedule, completed only):

Date (UTC)	Run	Test Fn Status	median_e2e_latency_ms	Error
07-23 00:27	29969145884	❌ Failed	4666 → 4741 (retry)	not less than 4000
07-22 12:32	29920027108	✅ Passed	4249 first attempt, passed on retry	—
07-22 00:26	29880220845	❌ Failed	20742 → 9400 (retry)	not less than 4000
07-21 12:32	29830494225	✅ Passed	—	—
07-21 00:27	29790347494	❌ Failed	15641 → 11156 (retry)	not less than 4000
07-20 12:42	29743214297	✅ Passed	—	—
07-20 00:28	29709549202	✅ Passed	—	—
07-19 12:17	29686709871	✅ Passed	—	—
07-19 00:27	29666971143	✅ Passed	—	—
07-18 12:17	29644059344	✅ Passed	—	—
07-18 00:25	29623067212	✅ Passed	—	—
07-17 12:25	29580198489	✅ Passed	—	—
07-17 00:27	29544840981	✅ Passed	—	—
07-16 12:31	29498496614	✅ Passed	—	—
07-15 12:28	29415344552	✅ Passed	—	—

• First observed failure for this test function: 07-21 00:27 UTC (29790347494). All runs 07-15 → 07-20 passed.
• Last clean pass: 07-22 12:32 (but that run's first attempt was already over threshold at 4249 ms — it only passed because the retry landed under 4000).
• Ratio: failed both attempts in 3 of the last 15 completed runs; a 4th run (07-22 12:32) overshot on the first attempt. Failures cluster on the ~00:2x UTC runs; the ~12:3x UTC runs pass.

Failure Cluster

LoRA concurrent-adapter-update perf test exceeds median-e2e-latency threshold (4000 ms)

Facts (verified observations)

• Assertion site: test_bench_serving_1gpu_part1.py:169 → self.assertLess(res["median_e2e_latency_ms"], 4000) (verified in source and stack trace).
• Both attempts today over threshold: 4666.40 ms, then 4741.71 ms (verbatim from log lines 17258 / 19502). Ran 8 tests in 1476.738s … FAILED (errors=1).
• Historical failing values are highly variable: 4666, 4741, 9400, 11156, 15641, 20742 ms — a ~5× spread across failing runs.
• The 07-22 12:32 "passed" run recorded 4249 ms on the first attempt (over threshold) and only passed via the retry.
• The pass→first-fail commit window (370f454e..8905cbd4) contains exactly one commit — 8905cbd4 "Fix MiniMax-M3 crash on ROCm" (#31837) — unrelated to LoRA/latency.
• The test drives a background task that repeatedly loads/unloads LoRA adapters concurrently with the benchmark (_run_lora_latency_test(enable_background_task=True), test file lines 176-194).
• aiter unchanged across the window (dev image v0.1.17.dev110+g9127c94a1, no override).

Hypothesised Causes (with confidence)

• [HIGH] Flaky / noise-sensitive perf assertion, dominated by runtime variance rather than a code defect.
  • Supporting: failing values swing 4666→20742 ms (5× spread); the same test passes on ~half the runs and even passes on retry within a single run (07-22 12:32); the concurrent LoRA load/unload background task inherently produces latency spikes that periodically breach the 4000 ms median gate.
  • Disconfirming: failures only began 07-21 (clean 07-15→07-20), which could indicate a real slowdown rather than pure noise — but see next hypothesis's disconfirming evidence.
  • How to verify: re-run the job several times on linux-mi325-1gpu-sglang; record median_e2e_latency_ms distribution. If it straddles 4000 ms with high variance, it's a threshold-margin flake.
• [LOW] Code regression around 07-20/07-21 raising LoRA e2e latency.
  • Supporting: no failure of this test in the 07-15→07-20 window; first failure 07-21 00:27.
  • Disconfirming: the pass→fail commit window is a single unrelated commit (MiniMax-M3 ROCm fix); a real regression would produce consistently higher latency, not values ranging 4249→20742 ms; the noon runs continue to pass. The two LoRA commits in the broader 4-day window (0a3cd26b Ascend backend, 57e5846b TMA-down guard) target Ascend/NVIDIA-TMA paths, not AMD MI325.
  • How to verify: bisect 370f454e..8905cbd4 — trivial (1 commit); if it doesn't move latency, regression hypothesis is dead.

In-flight Fix Check

Searched open PRs with lora latency — matches are feature PRs (#27181 LoRA async pipelining, #31305 paged LoRA memory pool), none targeting this test's threshold or AMD LoRA latency flakiness.

• In-flight fix: ❌ none found — no open PR addresses this perf-threshold flake

Suggested Triage Steps

• Re-run stage-b-test-1-gpu-large-amd (linux-mi325-1gpu-sglang, 0) 3-5× at the current HEAD and collect the median_e2e_latency_ms distribution for this test to quantify variance vs the 4000 ms gate.
• Confirm the regression hypothesis is dead by checking the 1-commit window 370f454e..8905cbd4; if latency is unaffected, treat as a flake.
• If confirmed flaky, consider (maintainer's call) relaxing the 4000 ms median-e2e threshold for the AMD concurrent-adapter path, and/or check whether the background load/unload task is over-aggressive for the CI runner — the wildly high spikes (9-20 s) suggest occasional adapter-swap contention worth profiling.
• Check whether ~00:2x UTC runner contention correlates with the failures (all both-attempt failures are midnight-UTC runs; noon runs pass).

Status

• Persistence: intermittent since 07-21 00:27 UTC; failed both attempts in 3 of last 15 completed scheduled runs (plus 1 first-attempt overshoot that passed on retry). Clean 07-15→07-20.
• Scope: single test function in one job (stage-b-test-1-gpu-large-amd, linux-mi325-1gpu-sglang, 0) on pr-test-amd.yml; other 7 tests in the file pass.
• Blocked work: gates the AMD stage-b 1-GPU-large perf signal on the failing runs; no correctness impact.
• In-flight fix: ❌ none found.

multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 0) — failed step(s): N/A

Commit Info

• sglang: 24da0e51
• aiter: 9127c94a (v0.1.17.dev110+g9127c94a1) — source: Dockerfile default, dev/patched image preserved, AITER_COMMIT_OVERRIDE empty. Read from the sibling part-3 [CI-AITER-CHECK] markers in the same run (same image); the part-0 job log itself was not persisted.

Failed Tests

Test File	Test Function	Error	Log
N/A — log not persisted	N/A	Job killed while step 9 Run diffusion server tests (1-GPU) was still in_progress; runner terminated at ~90 min wall, no test summary emitted	Job

The specific failing test file/function for this shard cannot be pinned: the job log returns BlobNotFound (never uploaded — the if: always() artifact-upload step 10 never ran), and 1-gpu shards are assigned by dynamic LPT partitioning so the case set varies per run. Sibling shards in the same run give context (see Facts).

Failure Summary

multimodal-gen-test-1-gpu-amd (part 0) job 89087124781 ran for exactly ~90 min (02:34:14→04:04:19 UTC) and was marked failure with step 9 (Run diffusion server tests (1-GPU)) still in_progress and no downstream step executed. Its log was never persisted (BlobNotFound), consistent with an abrupt runner kill rather than a graceful test-assertion failure (the sibling part-3 job, which failed on assertions, uploaded its log normally). This shard has failed in every recent completed scheduled run.

Regression Status

Known recurring failure (persistent, ≥ 8 days). No passing run for this shard exists in the 15-run query window → effectively Never-passed within window for multimodal-gen-test-1-gpu-amd (part 0).

Recent history of multimodal-gen-test-1-gpu-amd (part 0) in pr-test-amd.yml (most recent 15 comparable completed scheduled runs: branch=main, event=schedule, completed only). Test-file status is shown as Unknown (hang) where the log was not persisted; the Job column is human reference only.

Date (UTC)	Run	Job	Job Result	Wall	Test File Status
Jul 22 12:32	29920027108	part 0	❌ Failed	85m	Unknown (log not persisted — hang)
Jul 22 00:26	29880220845	part 0	❌ Failed	88m	Unknown (log not persisted — hang)
Jul 21 12:32	29830494225	part 0	❌ Failed	122m	Unknown (log not persisted — hang)
Jul 21 00:27	29790347494	part 0	❌ Failed	99m	Unknown (log not persisted — hang)
Jul 20 12:42	29743214297	part 0	❌ Failed	107m	Unknown (log not persisted — hang)
Jul 20 00:28	29709549202	part 0	❌ Failed	89m	Unknown (log not persisted — hang)
Jul 19 00:27	29666971143	part 0	❌ Failed	80m	Unknown (log not persisted — hang)
Jul 18 12:17	29644059344	part 0	❌ Failed	74m	Unknown (log not persisted — hang)
Jul 18 00:25	29623067212	part 0	❌ Failed	91m	Unknown (log not persisted — hang)
Jul 17 12:25	29580198489	part 0	❌ Failed	85m	Unknown (log not persisted — hang)
Jul 16 00:25	29461421122	part 0	❌ Failed	8m	Distinct cluster — cmake --build . -j 256 exit 1 during dependency install (not the diffusion test)
Jul 15 12:28	29415344552	part 0	❌ Failed	10m	Unknown (short — likely install/build)

First observed failure: at or before the start of the query window (Jul 15). No green run for this shard in-window.

Failure Cluster

1-GPU diffusion server test hang → runner killed at ~90 min (no log persisted). (Symptom-level grouping. A secondary, distinct cluster of short ~8–10 min install/build failures — e.g. cmake --build exit 1 — also appears intermittently for this shard.)

Facts (verified observations)

• Job 89087124781: conclusion=failure, 02:34:14 → 04:04:19 UTC (~90m05s). Step 9 Run diffusion server tests (1-GPU) = in_progress (no conclusion); step 10 Upload diffusion failure artifacts (if: always()) and step 20 Post Checkout = pending/never ran. (GitHub jobs API.)
• Job log = HTTP 404 BlobNotFound (never persisted). Sibling part-3 89087124715 log downloaded fine (2.2 MB) — so log absence is job-specific, not a global outage.
• Step 9 has timeout-minutes: 90 (pr-test-amd.yml:751); it began at 02:46:28, so its own timeout would fire at ~04:16:28 — after the job actually died (04:04:19). Several historical runs ran the step well past 90 min (job walls 99/107/122 min), meaning the step-level timeout did not reliably terminate the hung process.
• Same run, 1-gpu diffusion suite health: part 2 ✅ passed; part 1 89087124739 still in_progress since 00:28:10 (~4 h — hung); part 3 89087124715 ❌ real failures in sglang/multimodal_gen/test/single_test_file/test_update_weights_from_disk.py — 4 failed [Qwen-Image] + 4 errored [FLUX.2-klein-base-4B] (4 failed, 2 passed, 4 errors). The 2-gpu part 0 also ❌ failed.
• Run 29969145884 is [IN-FLIGHT] at snapshot (status not completed; 8 jobs still None). Trend/counts here reflect partial data.
• aiter unchanged across window (v0.1.17.dev110+g9127c94a1, Dockerfile default, no override).

Hypothesised Causes (with confidence)

• [MEDIUM] A diffusion server test in shard 0 hard-hangs on MI325 (GPU/kernel hang during model load, warmup, or denoise), leaving the runner unresponsive; GitHub reaps the job at ~90 min and no log is flushed.
  • Supporting: step 9 stuck in_progress; the always() upload step never ran; log never persisted; consistent 73–122 min walls across 15/15 runs; a second shard (part 1) is simultaneously hung ~4 h in the same run; step-level timeout-minutes: 90 fails to clean up (job walls exceed 90 min historically → the timeout can't kill a wedged GPU process).
  • Disconfirming: the exact failing test/model is unverifiable (no log); the ~90-min wall does not line up cleanly with step 9's own timeout window (step started at job+12 min), so the reaper may be a runner heartbeat-loss rather than the step timeout; part-2 passing shows the hang is case-specific, not a blanket suite/runner failure.
  • How to verify: re-run shard 0 with -s/verbose and py-spy dump/rocm-smi sampling in the container; capture per-case timing to identify which model wedges; add a hard per-case subprocess kill so the log flushes.
• [LOW] Distinct minority mode: dependency build failure (cmake --build . -j 256 exit 1) — observed in the 8-min run 29461421122 (Jul 16). Unrelated to the hang; likely transient/build-env.
  • Disconfirming: only 1–2 short runs in the window show this; the dominant mode is the long hang.
• Not attributable to a specific commit: this is a ≥8-day persistent failure with no in-window green anchor, so a pass_sha..fail_sha bisect is not possible from the queryable history.

In-flight Fix Check

• In-flight fix: ❌ none found that directly targets the 1-gpu diffusion server-test hang. Searched open PRs with multimodal-gen+hang, diffusion+timeout+amd, multimodal_gen+amd+test. Related but non-matching diffusion/AMD PRs exist (e.g. #31899 msgpack dep for the unit test; #28889 FLUX.2 fp8) — none address this shard's hang.

Suggested Triage Steps

• Reproduce shard 0 locally/on MI325: python3 sglang/multimodal_gen/test/run_suite.py --suite 1-gpu --partition-id 0 --total-partitions 4 -k "not flux_2" inside the CI image, with per-case timing and py-spy dump + rocm-smi sampling to pin the hanging model/case.
• Add a hard per-case timeout that SIGKILLs the diffusion server subprocess so the pytest tail + diffusion-failures/ artifacts flush before the runner is reaped (today the log is lost, blocking all root-cause work). Investigate why step 9's timeout-minutes: 90 does not terminate the wedged process.
• Once a case is identified, check whether it's aiter-attention-related: the part-3 server ran attention_backend=aiter; try attention_backend fallback for that model to A/B the hang.
• Treat the short cmake --build failures as a separate transient issue; only investigate if it recurs.

Status

• Persistence: multimodal-gen-test-1-gpu-amd (part 0) failed in 15/15 recent completed scheduled runs (Jul 15 → Jul 22); dominant mode is a ~73–122 min hang.
• Scope: Affects multiple 1-gpu diffusion shards concurrently (part 0 killed, part 1 hung ~4 h, part 3 assertion failures; 2-gpu part 0 also failing) — blocks the AMD multimodal_gen 1-gpu CI signal.
• Data caveat: Today's run 29969145884 is [IN-FLIGHT]; target-job log unavailable (BlobNotFound); shard-to-test mapping is non-deterministic (LPT), so exact test-file attribution for part 0 is unverified.
• In-flight fix: ❌ none found targeting this hang.

multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 1) — failed step(s): N/A

Commit Info

• sglang: 24da0e51 ("Warn on small Mamba chunked prefill size" #30938)
• aiter: unknown — the job log is unretrievable (BlobNotFound), so [CI-AITER-CHECK] markers cannot be read for this job.

Failed Tests

Test File	Test Function	Error	Log
N/A (non-test failure)	N/A	Job terminated abnormally during step "Run diffusion server tests (1-GPU)"; step never recorded a conclusion and the job log was never uploaded	Job

The specific failing diffusion test file/function for this shard cannot be identified — see Failure Summary. The GitHub job log for 89087124739 returns BlobNotFound and no failure artifacts were uploaded.

Failure Summary

multimodal-gen-test-1-gpu-amd (…, 1) failed on the MI325 runner. Steps 9 (Run diffusion server tests (1-GPU)), 10 (Upload diffusion failure artifacts) and 20 (Post Checkout) all have conclusion: None, the job log is BlobNotFound, and 0 artifacts were uploaded for the run — i.e. the runner was lost / the job was force-terminated during the diffusion test step rather than producing a normal pytest failure. The job ran ~72 min (04:33:58→05:46:01 UTC), under the step's timeout-minutes: 90 (workflow pr-test-amd.yml:751), so this is not a clean step-timeout.

Regression Status

Known recurring failure (shard-level; test-file granularity unverifiable due to missing logs).

Recent history of multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 1) (most recent 8 comparable completed scheduled runs: branch=main, event=schedule, completed only). Test-file status is ❓ unknown because every one of these failing runs has an unretrievable log (same BlobNotFound pattern), so the failure cannot be confirmed at the test-function level:

Date	Run	Duration	Shard Conclusion	Test File Status
Jul 23	29969145884	72 min	❌ Failed	❓ log BlobNotFound
Jul 22	29920027108	77 min	❌ Failed	❓ log BlobNotFound
Jul 22	29880220845	85 min	❌ Failed	❓ log BlobNotFound
Jul 21	29830494225	111 min	❌ Failed	❓ log BlobNotFound
Jul 21	29790347494	90 min	❌ Failed	❓ log BlobNotFound
Jul 20	29743214297	84 min	❌ Failed	❓ log BlobNotFound
Jul 20	29709549202	71 min	❌ Failed	❓ log BlobNotFound
Jul 19	29686709871	91 min	❌ Failed	❓ log BlobNotFound

No green run for this shard was found in the queryable window; the shard has failed on every scheduled run for ≥5 days. First-observed date is not pinpointable at the test-file level because no log from any failing run is retrievable.

Failure Cluster

AMD diffusion 1-GPU shard terminated with no log (runner loss / GPU hang during long diffusion test)

Facts (verified observations)

• Job 89087124739: conclusion=failure; steps 9/10/20 conclusion=None (verified via jobs API). Ran 04:33:58→05:46:01 UTC (~72 min).
• Job log endpoint returns BlobNotFound on repeated attempts; the run has 0 artifacts (upload-artifact step never executed — conclusion=None).
• Log availability is not a global outage. In the same run: successful jobs return HTTP 200 (multimodal-gen-test-2-gpu (…,1) 89087124681 677 KB; multimodal-gen-unit-test-amd 89087124712 1.2 MB; multimodal-gen-test-1-gpu (…,2) 89087124736 618 KB). The short failed shard 3 (34 min) 89087124715 returns a full 2.2 MB log.
• The BlobNotFound pattern hits exactly the long-running failed shards: shard 0 89087124781 (~90 min, no log), shard 1 (this job, 72 min, no log), 2-GPU shard 0 89087124711 (~96 min, no log). Short/successful shards keep their logs.
• The step under which the kill happened invokes heavy video/image diffusion models (Wan2.1-T2V-1.3B, HunyuanVideo, Qwen-Image, FLUX.1) — pr-test-amd.yml:736-778.
• Separate symptom, different partition (context only, NOT this shard): shard 3's retrievable log shows a real pytest failure — AssertionError: param_data.shape == loaded_weight.shape at python/sglang/multimodal_gen/runtime/layers/linear.py:410 via weights_updater.py:295 in test/single_test_file/test_update_weights_from_disk.py (FLUX.2-klein-base-4B / Qwen-Image). This is a distinct weight-loading failure and is not evidence for shard 1's cause.

Hypothesised Causes (with confidence)

• [MEDIUM] GPU hang / runner loss during a long-running diffusion model test on the MI325 runner, terminating the job before step/artifact/log finalization.
  • Supporting: 72-min runtime with no step conclusion; log never uploaded (BlobNotFound) while sibling success and the short failed shard-3 logs upload normally; the no-log pattern maps precisely onto the long-running shards (0, 1, 2gpu-0) and not the short ones; heavy diffusion models run in this step; active branches on the remote (amd/ci-gpu-hang-probe, amd_fix_request_timeout, bingxche/amd-jit-kernel-timeout-30m, fix/add-job-timeout-pr-test) and open PRs #31068/#30431 indicate an ongoing AMD-CI hung-GPU/runner class of problem.
  • Disconfirming: without the log I cannot confirm which test hung, nor rule out a non-hang kill (e.g. OOM-killer taking down the runner agent, or a slow-but-progressing test). The parent workflow has been red for ≥8 days for multiple unrelated reasons, so shard-level persistence alone is weak evidence for any single cause.
  • How to verify: re-run the shard with the step wrapped in per-test timeouts + rocm-smi/py-spy dumps on stall; correlate the ~72-min mark with the model under test; check runner-agent/system logs on the MI325 host for a GPU reset or OOM around 05:46 UTC.
• [LOW] Step-level 90-min timeout.
  • Supporting: shard 0 (~90 min) sits right at timeout-minutes: 90.
  • Disconfirming: shard 1 ran only ~72 min — it did not reach the step timeout, so this does not explain this job.
• [LOW] Same weight-shape assertion seen in shard 3.
  • Supporting: same job family / suite.
  • Disconfirming: shard 3 is a different LPT partition and it produced a log; shard 1 produced none. A pytest assertion would have uploaded a log + failure artifact. Different failure mode.

In-flight Fix Check

Searched open PRs with keywords diffusion timeout, amd hang, multimodal_gen amd:

• In-flight fix (mitigation, not exact): 🟡 #31068 "[AMD] ci: detect and recover hung GPUs in MI35x pre-flight" and #30431 "[AMD] ci: self-heal driver-level VRAM leaks + graceful GPU cleanup" target the hung-GPU/runner class but are MI35x/general and do not identify the failing diffusion test.
• In-flight fix (exact test-level): ❌ none found — no open PR addresses a specific multimodal-gen-test-1-gpu-amd shard-1 diffusion test.

Suggested Triage Steps

• Reproduce the missing-log problem first: it is the primary blocker. Re-run shard 1 with per-test/docker exec timeouts shorter than the runner-loss point (e.g. wrap each case) plus stall-time rocm-smi + py-spy dumps, so the failing model/test is captured before the runner dies.
• Once a log exists, pin the exact diffusion test (T2V/T2I/I2V/LoRA model) that hangs at the ~72-min mark and file it at test-file+function granularity.
• Chase reviewers on #31068 / #30431 — GPU-hang recovery would at least make these jobs fail loudly with a log instead of silently losing the runner.
• Separately, triage the shard-3 weight-shape AssertionError in test_update_weights_from_disk.py (linear.py:410) — it is a distinct, log-backed failure in the same suite and warrants its own investigation.

Status

• Persistence: multimodal-gen-test-1-gpu-amd (…,1) has failed on every completed scheduled run for ≥5 days (8/8 in the query window); all failing runs share the unretrievable-log signature. Test-file-level first-observed date is unknown (no logs).
• Scope: Within this run, the same no-log abnormal-termination signature affects the long-running 1-GPU shards 0 & 1 and 2-GPU shard 0; shard 2 passed and shard 3 failed with a different, log-backed assertion. Parent workflow pr-test-amd.yml has been red on every scheduled run for ≥8 days (multiple unrelated causes).
• In-flight fix: 🟡 mitigation-only PRs #31068 / #30431; ❌ no exact test-level fix.

stage-b-test-1-gpu-small-amd-mi35x (linux-mi35x-gpu-1) — failed step(s): Run test

Commit Info

• sglang: b98a577fb
• aiter: 9127c94a1 (v0.1.17.dev110+g9127c94a1; source: dev image preserved, matches Dockerfile default, AITER_COMMIT_OVERRIDE empty)

Failed Tests

Test File	Test Function	Error	Log
test/registered/quant/test_quark_mxfp4.py	TestFP8ToMXFP4MOETP1.test_gsm8k	AssertionError: 0.912 not greater than 0.92	Log (step 6)

The other 4 registered files in this job passed (test_dsa_skip_logits_clean.py, test_verify_splitkv.py, test_mamba_state_scatter_triton.py, test_deepseek_mla_dispatch.py). Within test_quark_mxfp4.py: 1 error, 1 pass (TestFP8ToMXFP4DenseTP1.test_gsm8k @ 0.888 vs threshold 0.868), 3 skipped.

Failure Summary

The GSM8K accuracy gate for online FP8→MXFP4 requantization of Qwen/Qwen3-30B-A3B-Instruct-2507-FP8 (TP=1) failed. TestFP8ToMXFP4MOETP1.test_gsm8k asserts accuracy > 0.92 (test_quark_mxfp4.py:251 → :171). The eval produced accuracy 0.92 on the first attempt (not strictly greater → fail) and 0.912 on the single retry, so the retry wrapper exhausted and the job exited 255. No crash, no numerical NaN — this is a threshold miss.

Regression Status

Flaky test (marginal accuracy threshold). The MOE GSM8K accuracy oscillates directly across the strictly-greater-than 0.92 gate from run to run (observed range ~0.906–0.928, retry-aware final verdict below).

Recent history of test/registered/quant/test_quark_mxfp4.py (TestFP8ToMXFP4MOETP1.test_gsm8k) in job stage-b-test-1-gpu-small-amd-mi35x (most recent comparable completed scheduled runs: branch=main, event=schedule):

Date (UTC)	Run	Test File Status	MOE accuracy (attempt/retry)	Result
Jul 23 12:33	30007380837	❌ Failed	0.920 / 0.912	fail
Jul 23 00:27	29969145884	❌ Failed	0.912 / 0.914	fail
Jul 22 12:32	29920027108	✅ Passed	0.928	pass
Jul 22 00:26	29880220845	❌ Failed	0.912 / 0.920	fail
Jul 21 12:32	29830494225	✅ Passed	0.922	pass
Jul 21 00:27	29790347494	✅ Passed	(file passed)	pass

3 of the last 6 comparable runs failed on this exact test. The test file is recent (first appears ~Jul 21 in the queryable window), and has been intermittently red since introduction — this is not a clean regression from a datable green baseline, but intrinsic eval variance sitting on the threshold. The Dense FP8 sibling (TestFP8ToMXFP4DenseTP1.test_gsm8k, threshold 0.868) is stable (~0.888).

Failure Cluster

GSM8K accuracy threshold miss on online FP8→MXFP4 MoE requant (Qwen3-30B-A3B)

Facts (verified observations)

• test_quark_mxfp4.py:251 sets accuracy_threshold=0.92; :171 asserts self.assertGreater(metrics["accuracy"], accuracy_threshold) — strictly-greater, so exactly 0.92 fails (FACT).
• Current run: attempt=0.920, retry=0.912; invalid=0.0 both times (no eval infra breakage) — accuracy is genuinely below gate (FACT).
• Retry wrapper: retry() exceed maximum number of retries after 1 retry (sglang/srt/utils/common.py:3192) (FACT).
• Across 6 comparable scheduled runs the MOE accuracy ranged 0.906–0.928, straddling 0.92; the same job passed at 0.922/0.928 and failed at 0.912/0.920 (FACT, from historical job logs).
• Only the MoE variant is affected; the Dense FP8 variant (threshold 0.868, ~0.888) is comfortably above its gate (FACT).
• invalid=0.0, latency/throughput normal — server healthy, no crash/OOM (FACT).

Hypothesised Causes (with confidence)

• [HIGH] The 0.92 gate is too tight for the observed run-to-run GSM8K variance of this online-requantized MoE model.
  • Supporting: retry-aware final accuracies across 6 runs cluster at 0.906–0.928 with mean ≈0.918, i.e. the gate sits inside the noise band; passes and fails alternate with no code change to the model path. assertGreater also makes an exact-0.92 landing (seen this run) a guaranteed fail.
  • Disconfirming: none found — no sglang commit in the window is known to touch the MXFP4 requant numerics, and the Dense sibling is stable, arguing against a broad accuracy regression.
  • How to verify: run TestFP8ToMXFP4MOETP1.test_gsm8k N×10 on an MI35x (gfx950) container; confirm the accuracy distribution and whether the mean is < 0.92 (true accuracy regression) or straddling it (threshold-too-tight).
• [LOW] A genuine accuracy regression in the FP8→MXFP4 MoE requant path or an aiter MoE kernel change lowered mean accuracy below 0.92.
  • Supporting: several passes are only marginally above (0.922); a small real drop could tip the balance.
  • Disconfirming: aiter version is unchanged (dev image preserved, no override) across passing and failing runs; the Dense path is unaffected; accuracy still reaches 0.928 in some runs — inconsistent with a systematic degradation.
  • How to verify: bisect MoE/MXFP4-quant commits between a green run (e4eea7ce2f, Jul 21 12:32) and the failing SHA b98a577fb, re-running the test; compare mean accuracy before/after.

In-flight Fix Check

• In-flight fix: ❌ none found — no open PR adjusts this threshold or the MXFP4 MoE accuracy path. Searched is:open quark_mxfp4, is:open quark mxfp4 threshold, is:open mxfp4 accuracy. The feature that introduced this test is #29328 (Online MXFP4 quantization 4/N, still open); no threshold-tuning follow-up exists.

Suggested Triage Steps

• Collect the empirical accuracy distribution: run TestFP8ToMXFP4MOETP1.test_gsm8k ~10× on an MI35x/gfx950 runner. If mean ≥ 0.92 with variance crossing the gate → loosen the threshold (e.g. 0.90) and/or switch assertGreater→assertGreaterEqual, mirroring the Dense variant's looser gate.
• If the empirical mean is < 0.92, treat as a real accuracy regression: bisect MoE/MXFP4 requant commits in e4eea7ce2f..b98a577fb and re-run the test per candidate.
• Consider raising num_questions (currently 500) to reduce sampling variance if the model accuracy is genuinely near 0.92, so the gate is statistically meaningful.

Status

• Persistence: Intermittent since at least Jul 22 00:26 UTC — 3 of the last 6 comparable scheduled runs failed on this test; passes and fails alternate.
• Scope: 1 test function (TestFP8ToMXFP4MOETP1.test_gsm8k) in 1 job (stage-b-test-1-gpu-small-amd-mi35x) on pr-test-amd.yml. MI35x-only (requires gfx95x FP4 hardware); the Dense MXFP4 gate in the same file is unaffected.
• Blocked work: Flakes the AMD MI35x stage-b small-GPU signal ~50% of scheduled runs; blocks a clean green on pr-test-amd.yml.
• In-flight fix: ❌ none found — needs a threshold review or new investigation.

stage-b-test-large-8-gpu-mi35x-disaggregation-amd (linux-mi35x-gpu-8.fabric) — failed step(s): Run test_disaggregation

The report is complete. That background task was the test-file results check I already read and incorporated. No further action needed — the analysis is finished.

Summary of findings: test_disaggregation_pp.py::test_gsm8k failed with KeyError: 'answer' in the MI35x 8-GPU PP disaggregation job. It's a flaky/intermittent failure (7/13 completed scheduled runs failed with the identical signature, interleaved with passes), most likely a nondeterministic dropped request in the PP prefill→decode KV-transfer path rather than a clean regression. No direct in-flight fix exists.

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-07-23 18:35 UTC)