[CI Monitor] Daily Report - 2026-06-23

[amd-bot]

Daily Cross-Workflow Summary — 2026-06-23

Snapshot: 2026-06-23 21:01 UTC · Only completed runs/jobs counted in trends · Auto-updated every 30 min

TL;DR

🔴 RED · ~16 active clusters · 3 🆕 today (R204, R205, R206 — all aiter-caused) · ~12 carrying over · 3 review-ready fixes (#27757, #27141, #28889) + 1 merged (#28941)
👉 Today's ask: The Jun-22 AMD AITER Scout 27983814629 is now COMPLETE and — unlike yesterday's in-flight "zero aiter-caused" snapshot — its MI35x jobs reveal 3 A/B-confirmed aiter-caused accuracy regressions under candidate aiter a0320631: 🆕 R204 (GLM-5.1-FP8 DSA collapse), 🆕 R205 (DeepSeek-V3.2-MTP gsm8k), 🆕 R206 (gpt-oss-20b). Triage these before the next Thu scout — re-run each MI35x job with default aiter (AITER_COMMIT_OVERRIDE= empty) to confirm, then flag the aiter bump. Also chase the 3 review-ready fixes. ⚠️ Both nightly Jun-23 runs are IN-FLIGHT (only the early 2-GPU accuracy jobs done) — counts partial. Good news: both release-docker workflows are ✅ green today, and pr-test-amd-rocm720 produced a clean ✅ SUCCESS run (28027779773) — the concurrency churn is still present but a full run completed.

Workflow status

Workflow	Latest completed run	✅	❌	Trend (completed real failures)	Δ vs yesterday
nightly-test-amd	Jun-22 27975987522; Jun-23 28046906309 (IN-FLIGHT)	0	7 (+2 partial)	5·11·9·7·(IN-FLIGHT)	pending (Jun-23 in-flight)
nightly-test-amd-rocm720	Jun-22 27975850577; Jun-23 28046795462 (IN-FLIGHT)	0	11 (+2 partial)	12·11·(IN-FLIGHT)	pending (Jun-23 in-flight)
release-docker-amd-nightly	Jun-23 28028570862	✅	0	0·0·0	0
release-docker-amd-rocm720-nightly	Jun-23 28027960865	✅	0	0·0·0	0
amd-aiter-scout	Jun-22 27983814629 (now COMPLETE)	0	~13 real (3 aiter-caused · ~8 pre-existing · rest infra/cancel)	—	scout completed; +3 aiter-caused surfaced
pr-test-amd	Jun-23 28009346928 (schedule) + PR runs	0	~4/run	~4–6	~same
pr-test-amd-rocm720	Jun-23 ✅ 28027779773; 28046873327 concurrency-cancelled	✅	0 (clean run)	~10·~10·0	much better (clean run landed)

Notes: (1) Both nightlies' Jun-23 runs (28046906309, 28046795462) are queued/IN-FLIGHT at snapshot — only the early 2-GPU accuracy jobs have reported; the 8-GPU jobs (Qwen3.5/Kimi/MXFP4) have not run yet, so their carry-over clusters are NOT yet confirmed for today. (2) The scout run is the Mon/Thu cron from Jun-22, now completed (conclusion cancelled overall, but real test failures ran first). No new scout today (next ≈ Thu Jun-25). (3) pr-test-amd-rocm720 still loses overlapping scheduled runs to cancel-in-progress, but run 28027779773 completed green — the in-flight CI fix #28987 (draft) targets this churn.

Failure clusters (active, deduplicated across all workflows)

R204 · 🆕 GLM-5.1-FP8 DSA accuracy collapse under candidate aiter — MI35x (scout, aiter-caused)

• Status: NEW today (surfaced now scout completed). Near-zero accuracy, no crash, on candidate aiter a0320631.
• Top hypothesis: [MEDIUM] aiter override regresses the GLM-5.1 FP8 sparse-MLA/DSA compute path; disconfirming: sister baseline-job log expired (404) — pass inferred from job success, not directly read.
• In-flight fix: ❌ none (#28922 would move nightly off GLM-5.1→GLM-5.2; #28975 adds an opt-in gfx950 GLM5 kernel — neither a fix).

Workflow	Job (shard)	Test File	Test Function	Origin	Error	Log
amd-aiter-scout	nightly-8-gpu-mi35x-glm51-rocm720	test/registered/amd/accuracy/mi35x/test_glm51_eval_mi35x.py	test_glm51_accuracy	aiter-caused	near-zero accuracy (no crash)	Log

R205 · 🆕 DeepSeek-V3.2-MTP gsm8k TP-divergence/accuracy — MI35x (scout, aiter-caused)

• Status: NEW today. Sister (default aiter) passed at 0.965; scout (candidate aiter) hangs/diverges during gsm8k decode (DSA + EAGLE MTP).
• Top hypothesis: [MEDIUM] aiter override changes a collective/MTP code path → TP-rank scheduling divergence → deadlock; disconfirming: hang vs accuracy drop are different symptoms (R205 vs R204) — may be two aiter regressions.
• In-flight fix: ❌ none (aiter-override-only; sglang fix not expected).

Workflow	Job (shard)	Test File	Test Function	Origin	Error	Log
amd-aiter-scout	nightly-accuracy-8-gpu-mi35x-deepseek-v32-mtp-rocm720	test_deepseek_v32_mtp_eval_mi35x.py	TestDeepseekV32TPMTP.test_a_gsm8k	aiter-caused	collective deadlock during gsm8k decode	Log

R206 · 🆕 gpt-oss-20b GSM8K below threshold (AITER MXFP4 fused-MoE, gfx950) — MI35x (scout, aiter-caused)

• Status: NEW today. Sister (default aiter) passed at 0.520; scout below threshold.
• Top hypothesis: [MEDIUM] candidate aiter MXFP4 fused-MoE kernel regresses gpt-oss-20b accuracy on gfx950; disconfirming: gpt-oss thresholds are tight — margin not stated, could be borderline.
• In-flight fix: ❌ none.

Workflow	Job (shard)	Test File	Test Function	Origin	Error	Log
amd-aiter-scout	nightly-accuracy-8-gpu-mi35x-rocm720	test/registered/amd/accuracy/mi35x/test_gpt_oss_eval_mi35x.py	test_gpt_oss_accuracy	aiter-caused	gsm8k below threshold	Log

R195 · Mamba extra_buffer needs CUDA/MUSA/NPU (FLA) rejected on ROCm (Qwen3.5) — 4 nightly jobs

• Status: persistent since ~Jun-18/19 on all four Qwen3.5 nightly jobs (Jun-22 completed; Jun-23 8-GPU jobs not yet run). Top hypothesis: [HIGH] #28151 made auto→extra_buffer, removing the ROCm fallback. In-flight fix: ✅ #27141 (open, ready, blocked) — unblock/land.

Workflow	Job (shard)	Test File	Test Function	Error	Log
nightly-test-amd	nightly-8-gpu-qwen35	test_qwen35_eval_amd.py	setUpClass	extra_buffer needs CUDA/MUSA/NPU (FLA)	Log
nightly-test-amd	nightly-8-gpu-mi35x-qwen35	test_qwen35_eval_mi35x.py	test_lm_eval	same assert	Log
nightly-test-amd-rocm720	nightly-8-gpu-qwen35-rocm720	test_qwen35_eval_amd.py	setUpClass	same assert	Log
nightly-test-amd-rocm720	nightly-8-gpu-mi35x-qwen35-rocm720	test_qwen35_eval_mi35x.py	test_lm_eval	same assert	Log

R2 · Mistral/Mixtral GSM8K below threshold (chat-eval) — both nightlies + scout (pre-existing)

• Status: never-passed in window (≥Jun-13); Mistral/Mixtral score ~0.17–0.37 vs 0.46–0.69. Scout A/B confirms pre-existing (sglang) (Mistral-7B fails ~0.36 with default aiter too) — aiter exonerated. Still failing Jun-23 (nightly-accuracy-2-gpu, IN-FLIGHT run). In-flight fix: ✅ #27757 (open, ready, blocked) — unblock.

Workflow	Job (shard)	Test File	Test Function	Error	Log
nightly-test-amd	nightly-accuracy-2-gpu (Jun-23)	test_gsm8k_eval_amd.py	test_gsm8k_all_models	Mistral/Mixtral < thr (+ Qwen2-57B server-start timeout)	Log
nightly-test-amd-rocm720	nightly-accuracy-2-gpu-rocm720	test_gsm8k_eval_amd.py	test_gsm8k_all_models	5 models below thr	Log
amd-aiter-scout	nightly-accuracy-2-gpu (pre-existing)	test_gsm8k_eval_amd.py	test_gsm8k_all_models	Mistral/Mixtral < thr	Log

R1 · VLM MMMU accuracy below threshold (5 models, ~random) — both nightlies + scout (pre-existing)

• Status: never-passed (≥Jun-13); same 5 models (Qwen2-VL, Qwen2.5-VL, Qwen3-VL-30B, MiniCPM-o, deepseek-vl2) ~0.26–0.29, bit-stable. Scout A/B confirms pre-existing (sglang). Still failing Jun-23. In-flight fix: ❌ none (stale WIP #26178 only touches perf tests).

Workflow	Job (shard)	Test File	Test Function	Error	Log
nightly-test-amd	nightly-accuracy-2-gpu-vlm (Jun-23)	test_vlms_mmmu_eval_amd.py	test_mmmu_vlm_models	4 models < thr (+deepseek-vl2 download hang)	Log
nightly-test-amd-rocm720	nightly-accuracy-2-gpu-vlm-rocm720 (Jun-23)	test_vlms_mmmu_eval_amd.py	test_mmmu_vlm_models	5 models < thr	Log
amd-aiter-scout	nightly-accuracy-2-gpu-vlm (pre-existing)	test_vlms_mmmu_eval_amd.py	test_mmmu_vlm_models	5 models < thr	Log

R196 · VLM DP-encoder CUDA-IPC invalid device pointer crash/hang — nightlies + scout (pre-existing)

• Status: ongoing since ~Jun-16; HIP invalid device pointer in cuda_ipc_transport_utils.py reconstruct path → exit -9; intermittent. Scout A/B confirms pre-existing (sglang). In-flight fix: ❌ none.

Workflow	Job (shard)	Test File	Test Function	Error	Log
nightly-test-amd	nightly-4-gpu	test_encoder_dp.py	test_vlm_mmmu_benchmark	mem fault → -9	Log
nightly-test-amd-rocm720	nightly-4-gpu-rocm720	test_encoder_dp.py	test_vlm_mmmu_benchmark	hipErrorInvalidDevicePointer	Log
amd-aiter-scout	nightly-4-gpu-rocm720 (pre-existing)	test_encoder_dp.py	test_vlm_mmmu_benchmark	invalid device pointer	Log

R192 · FLUX.2 modelopt-FP8 torch._scaled_mm HIPBLAS_STATUS_NOT_SUPPORTED — pr-test-amd multimodal-gen (now has fix)

• Status: recurring on every pr-test-amd multimodal-gen-2gpu run (flux2). In-flight fix: ✅ #28889 (open, ready, blocked) — "dequant→bf16 fallback in apply_fp8_linear", exact-symptom fix; unblock/land. ← upgraded from ❌ yesterday.

Workflow	Job (shard)	Test File	Test Function	Error	Log
pr-test-amd	multimodal-gen-test-2-gpu-amd (1)	diffusion (flux2)	N/A	HIPBLAS_STATUS_NOT_SUPPORTED	Log

R202 · Mooncake RDMA transfer-engine segfault / KV-transfer timeout (PD-disagg) — pr-test-amd MI35x

• Status: recurring on stage-b-test-large-8-gpu-mi35x-disaggregation-amd (RDMA ENOMEM / mooncake segfault / decode WaitingForInput 300s), cascades to GPU-occupied sibling tests. In-flight fix: ⚠️ #27012 (open, ready) — Revert "[AMD] Fix stage-b-test-large-8-gpu-mi35x-disaggregation-amd", exact job-name match; review before duplicating.

Workflow	Job (shard)	Test File	Test Function	Error	Log
pr-test-amd	stage-b-test-large-8-gpu-mi35x-disaggregation-amd	PD-disagg decode	N/A	mooncake RDMA segfault	Log

R199 · Meta-Llama-3.1-70B-FP8 HIP OOM during decode CUDA-graph capture — scout (unclear)

• Status: carried from Jun-22. Scout A/B: unclear — no FP8-70B sister baseline (sister ran only non-FP8 70B, which passed 0.910). One analysis labels it aiter-caused (LOW, likely flaky OOM); treat as unclear pending a default-aiter FP8-70B re-run. In-flight fix: ❌ none.

Workflow	Job (shard)	Test File	Test Function	Origin	Error	Log
amd-aiter-scout	nightly-accuracy-2-gpu-rocm720	test_gsm8k_eval_amd.py	test_gsm8k_all_models (70B-FP8)	unclear	HIP OOM at capture → -9	Log

R19 · Qwen3-235B-MXFP4 HIP stream-capture abort (hipErrorCapturedEvent) — MI35x, nightlies + scout

• Status: never-passed (Jun 10–22); --tp4 --ep2 mxfp4-specific custom-all-reduce capture abort. Scout A/B pre-existing (sglang). In-flight fix: ⚠️ #23581 (default SGLANG_USE_AITER_AR false) — candidate.

Workflow	Job (shard)	Test File	Test Function	Error	Log
nightly-test-amd	nightly-8-gpu-mi35x-qwen3-235b-mxfp4	test_qwen3_instruct_mxfp4.py	setUpClass	hipErrorCapturedEvent → -6	Log
amd-aiter-scout	nightly-8-gpu-mi35x-qwen3-235b-mxfp4-rocm720 (pre-existing)	test_qwen3_instruct_mxfp4.py	setUpClass	same	Log

R194 · Kimi-K2.6 MoE w2 narrow IndexError (TP8) — rocm720 nightly

• Status: narrow(start=shard_size*tp_rank) overruns a 64-wide w2 shard. In-flight fix: ⚠️ #28905 (open, DRAFT) — targets the w2 scale path; verify it also covers the w2 weight narrow.

Workflow	Job (shard)	Test File	Test Function	Error	Log
nightly-test-amd-rocm720	nightly-8-gpu-kimi-k26-rocm720	test_kimi_k26_eval_amd.py	setUpClass	IndexError narrow start out of range → -9	Log
nightly-test-amd	nightly-8-gpu-kimi-k26	test_kimi_k26_eval_amd.py	setUpClass	same	Log

Known/stable + resolved clusters (carrying over or fixed) · click to expand

ID	Cluster	Where (latest)	In-flight fix
R200	DeepSeek-V4 compress-state KV HIP OOM (rocm720 dsv4-flash)	rocm720 dsv4-flash	✅ #28941 MERGED (cee1caaf) — needs post-fix run to confirm
R197	anthropic SDK init — httpx 0.23.3 lacks socket_options (AMD image dep)	pr-test-amd stage-b-1gpu-small (13)	❌ none — bump AMD image httpx≥0.25
R193	torch._inductor AssertionError in diffusion torch.compile (rocm720)	quiet today (rocm720 ran green)	⚠️ #25969 (DRAFT)
R155	DeepSeek-V3.2 MI35x GSM8K catastrophic/below-thr (scout pre-existing)	rocm720 dsv32-mi35x	⚠️ #25559 (WIP)
R6	Qwen3-30B-A3B unquantized MoE accuracy ~0.38 (MI35x)	rocm720 accuracy-8gpu-mi35x	❌ none
R198	MiniMax-M2.7 GSM8K marginal 0.920 < 0.93 (borderline)	rocm720 minimax-m27	❌ none
R187	DeepSeek-R1-MXFP4 EAGLE/MTP accept-length low (MI35x)	pr-test-amd stage-c-mi35x	⚠️ #28378
R203	Qwen2-57B/72B-FP8 server-start / decode capture hang (TP2)	nightly-accuracy-2-gpu Jun-23	❌ none
R201	Qwen3-Coder-Next basic server hang (aiter JIT, flaky 1/3)	rocm720 accuracy-8gpu-mi35x	❌ none
R181	MORI build invalid-ELF-header → gtest discovery fail (infra)	pr-test-amd stage-a-1gpu-small	❌ none (infra)
—	Ideogram-4-fp8 gated-repo 403 at diffusion setup	pr-test-amd + scout (pre-existing)	✅ #28225 (draft)
—	RCCL allreduce SIGSEGV 8-GPU smoke test (test_rccl_multi_gpu.py)	pr-test-amd stage-c-8gpu (0)	❌ none

Infrastructure / orchestration noise (not test failures) · click to expand

• pr-test-amd-rocm720 concurrency churn: dozens of cancel-in-progress rows across runs (27975943807, 28046873327) — overlapping 30 17 daily + 0 */6 crons share github.ref. In-flight fix ⚠️ #28987 (draft). A clean run did complete today (28027779773 ✅).
• Scout gate-cancellation cascade: call-gate / pr-gate (82820330340) cancelled → extra-a-test jobs cancelled (0 steps). No test ran.
• HF weight-download stalls (xet): grok-1-W4A8KV8, GLM-5.1-MXFP4, Kimi-K2.6 MI35x, DeepSeek-R1 HiCache, DeepSeek-V3-KV-FP8, DeepSeek-V4-pro — across nightly + scout. These are infra/cache, not aiter (scout A/B explicitly exonerates aiter for the download hangs).
• grok2-rocm720 VRAM not cleared (zombie KFD context) — scout 82820327688; runner-ops (node reboot), upstream ROCm/aiter#2061.
• Diffusion 1-GPU infra: runner lost-communication, port-5555 contention, 45/90-min step timeouts on pr-test-amd multimodal-gen shards.
• jit-kernel-unit-test-amd (scout 82820364884): C++ template compile error — already fixed by merged #27947; stale on the scout's older sglang SHA.

Workflow drill-down (per-workflow view)

amd-aiter-scout · Jun-22 [27983814629](https://github.com/sgl-project/sglang/actions/runs/27983814629) (now COMPLETE) · ~13 real · override [`a0320631`](https://github.com/ROCm/aiter/commit/a0320631f5567e8723916a5d3b24ae7d6c5a32c2)

Job (shard)	Test File	Test Function	Cluster	Origin	Error
nightly-8-gpu-mi35x-glm51-rocm720	test_glm51_eval_mi35x.py	test_glm51_accuracy	R204🆕	aiter-caused	accuracy collapse
nightly-accuracy-8-gpu-mi35x-deepseek-v32-mtp-rocm720	test_deepseek_v32_mtp_eval_mi35x.py	test_a_gsm8k	R205🆕	aiter-caused	collective deadlock
nightly-accuracy-8-gpu-mi35x-rocm720	test_gpt_oss_eval_mi35x.py	test_gpt_oss_accuracy	R206🆕	aiter-caused	gsm8k < thr
nightly-accuracy-2-gpu	test_gsm8k_eval_amd.py	test_gsm8k_all_models	R2 / R199	pre-existing / unclear	Mistral < thr + 70B-FP8 OOM
nightly-accuracy-2-gpu-rocm720	test_gsm8k_eval_amd.py	test_gsm8k_all_models	R2 / R199	pre-existing / unclear	Mistral < thr + 70B-FP8 OOM
nightly-accuracy-2-gpu-vlm	test_vlms_mmmu_eval_amd.py	test_mmmu_vlm_models	R1	pre-existing	5 < thr
nightly-accuracy-2-gpu-vlm-rocm720	test_vlms_mmmu_eval_amd.py	test_mmmu_vlm_models	R1	pre-existing	5 < thr
nightly-4-gpu-rocm720	test_encoder_dp.py	test_vlm_mmmu_benchmark	R196	pre-existing	invalid device pointer
nightly-8-gpu-mi35x-qwen3-235b-mxfp4-rocm720	test_qwen3_instruct_mxfp4.py	setUpClass	R19	pre-existing	hipErrorCapturedEvent
nightly-accuracy-8-gpu-mi35x-deepseek-v32-rocm720	test_deepseek_v32_eval_mi35x.py	test_deepseek_v32_accuracy	R155	pre-existing	catastrophic ~0.01
2× minimax-m27 (amd/rocm720)	test_minimax_m27_eval_amd.py	accuracy	R198	pre-existing	0.920 < 0.93
6+ download/gate/grok2 jobs	various	setUpClass/N/A	infra	unclear/infra	HF xet stall / cancel / VRAM

Scout verdict: yesterday's in-flight snapshot said "zero aiter-caused"; the completed run shows 3 A/B-confirmed aiter-caused MI35x accuracy regressions (R204/R205/R206) under candidate aiter a0320631. All other real failures are pre-existing (sglang) (R1/R2/R19/R155/R196/R198) or unclear/infra (R199 + downloads).

nightly-test-amd · Jun-22 [27975987522](https://github.com/sgl-project/sglang/actions/runs/27975987522) (7) · Jun-23 [28046906309](https://github.com/sgl-project/sglang/actions/runs/28046906309) IN-FLIGHT (2 so far)

Job (shard)	Test File	Test Function	Cluster	Error
nightly-accuracy-2-gpu-vlm (Jun-23)	test_vlms_mmmu_eval_amd.py	test_mmmu_vlm_models	R1	4 < thr + deepseek-vl2 download
nightly-accuracy-2-gpu (Jun-23)	test_gsm8k_eval_amd.py	test_gsm8k_all_models	R2 / R203	Mistral < thr + Qwen2-57B server-start
nightly-4-gpu	test_encoder_dp.py	test_vlm_mmmu_benchmark	R196	IPC mem fault → -9
nightly-accuracy-2-gpu-vlm	test_vlms_mmmu_eval_amd.py	test_mmmu_vlm_models	R1	5 < thr
nightly-accuracy-2-gpu	test_gsm8k_eval_amd.py	test_gsm8k_all_models	R2	Mistral/Mixtral+gemma
nightly-8-gpu-qwen35 + mi35x-qwen35	test_qwen35_eval_*.py	setUpClass/test_lm_eval	R195	extra_buffer assert
nightly-8-gpu-mi35x-qwen3-235b-mxfp4	test_qwen3_instruct_mxfp4.py	setUpClass	R19	HIP capture abort -6
nightly-8-gpu-kimi-k26	test_kimi_k26_eval_amd.py	setUpClass	R194	narrow IndexError
grok1-int4 / mi35x-kimi-k26 / mi35x-dsr1-hicache	download	setUpClass	infra	HF xet stall

nightly-test-amd-rocm720 · Jun-22 [27975850577](https://github.com/sgl-project/sglang/actions/runs/27975850577) (11) · Jun-23 [28046795462](https://github.com/sgl-project/sglang/actions/runs/28046795462) IN-FLIGHT (2 so far)

Job (shard)	Test File	Test Function	Cluster	Error
nightly-accuracy-2-gpu-vlm-rocm720 (Jun-23)	test_vlms_mmmu_eval_amd.py	test_mmmu_vlm_models	R1	near-chance
nightly-accuracy-2-gpu-rocm720 (Jun-23)	test_gsm8k_eval_amd.py	test_gsm8k_all_models	R2	Mistral + watchdog storms
nightly-4-gpu-rocm720	test_encoder_dp.py	test_vlm_mmmu_benchmark	R196	invalid device pointer
nightly-8-gpu-qwen35 + mi35x-qwen35	test_qwen35_eval_*.py	setUpClass/test_lm_eval	R195	extra_buffer assert
nightly-8-gpu-kimi-k26-rocm720	test_kimi_k26_eval_amd.py	setUpClass	R194	narrow IndexError
accuracy-8-gpu-mi35x-rocm720	test_qwen3_moe_eval_mi35x.py / ..._coder_next...	accuracy	R6 / R201	0.380 / basic hang
minimax-m27-rocm720	test_minimax_m27_eval_amd.py	accuracy	R198	0.920 < 0.93
dsv32-mi35x-rocm720	test_deepseek_v32_eval_mi35x.py	accuracy	R155	below thr / timeout
glm5-mxfp4 / glm51 / mi35x-kimi-k26 / dsr1-hicache / dsv4-pro	download	setUpClass	infra	HF xet stall

pr-test-amd · Jun-23 [28009346928](https://github.com/sgl-project/sglang/actions/runs/28009346928) (schedule) + [28048998470](https://github.com/sgl-project/sglang/actions/runs/28048998470) IN-FLIGHT

Job (shard)	Test File	Test Function	Cluster	Error
multimodal-gen-2gpu (1)	diffusion (flux2)	N/A	R192	HIPBLAS_STATUS_NOT_SUPPORTED
stage-b-mi35x-disaggregation	PD-disagg decode	N/A	R202	mooncake RDMA segfault
stage-c-8gpu (0)	test_rccl_multi_gpu.py	N/A	RCCL	SIGSEGV smoke test
stage-a-1gpu-small	mori build	N/A	R181	invalid ELF header
stage-b-1gpu-small (13) / extra-a	test_anthropic_server.py	test_in_messages_system_role	R197	httpx socket_options
multimodal-gen-1gpu / 2gpu	diffusion	N/A	infra	runner lost / port-5555 / timeout

pr-test-amd-rocm720 · Jun-23 ✅ [28027779773](https://github.com/sgl-project/sglang/actions/runs/28027779773) SUCCESS · [28046873327](https://github.com/sgl-project/sglang/actions/runs/28046873327) concurrency-cancelled

A clean scheduled run completed green today. The remaining "failures" in the daily issue are cancel-in-progress concurrency supersessions (overlapping crons share github.ref) — orchestration noise, not test failures. In-flight CI fix ⚠️ #28987 (draft) targets this. The substantive rocm720 cluster when runs do complete is R200 (DSV4 OOM), now ✅ fixed by merged #28941.

How this report is generated

• Only status == "completed" runs/jobs counted in trends. Both nightly Jun-23 runs are IN-FLIGHT (only 2-GPU accuracy jobs reported) — their rows are partial; 8-GPU carry-over clusters (R195/R194/R19) are NOT yet confirmed for Jun-23. Scout 27983814629 is now COMPLETE. Both release-docker workflows ✅ green.
• 🆕 NEW today: R204 (GLM-5.1-FP8 DSA collapse), R205 (DeepSeek-V3.2-MTP gsm8k), R206 (gpt-oss-20b) — all aiter-caused, surfaced because the scout completed. R200 resolved (#28941 merged); R192 gained a fix (#28889).
• Scout Origins are A/B-derived: R204/R205/R206 aiter-caused; R1/R2/R19/R155/R196/R198 pre-existing (sglang); R199 + all download hangs unclear/infra. PR states verified live: #27757/#27141/#28889 open+ready+blocked; #28905/#28987 draft; #28941 merged; #27012 open+ready.
• Confidence: FACT/HIGH/MEDIUM/LOW/SPECULATION. Bot does NOT assign Priority — engineers decide from cluster size + persistence + fix availability.

---

Generated by amd-bot · last updated 2026-06-23 21:01 UTC

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-06-23 21:01 UTC)

---

CI Monitor — 2026-06-23

Repo: sgl-project/sglang

Monitored Workflows:

• nightly-test-amd.yml
• nightly-test-amd-rocm720.yml
• release-docker-amd-nightly.yml
• release-docker-amd-rocm720-nightly.yml
• amd-aiter-scout.yml
• pr-test-amd.yml
• pr-test-amd-rocm720.yml

Per-workflow failure reports are appended as comments below; the Daily Cross-Workflow Summary is rendered above this section.

[amd-bot]

nightly-test-amd.yml — 12 failure(s)

Run started (UTC): 2026-06-22 18:46 UTC
Last scanned (UTC): 2026-06-23 20:59 UTC
Commits: sglang ed26a10

Summary

Counts: 14 test-level failures across 12 jobs · 8 clusters · 11 carrying over (≥3 days) · 3 new today (2 scheduled runs: 27975987522 Jun 22, 28046906309 Jun 23)

#	Cluster	Job	Test File	Test Function	Status	Hypothesis (confidence)
1	HF weight-download stall/timeout at startup	nightly-8-gpu-mi35x-deepseek-r1-hicache	test/registered/amd/test_deepseek_r1_hicache_mi35x.py	N/A (whole-file TIMEOUT 3600s)	recurring since Jun 20	[HIGH] cold-cache single-threaded HF download under global filelock
2	HF weight-download stall/timeout at startup	nightly-8-gpu-mi35x-kimi-k26	test/registered/amd/accuracy/mi35x/test_kimi_k26_eval_mi35x.py	test_kimi_k26_gsm8k_accuracy	recurring (Jun 20 & 22; Jun 21 cache-hit crash)	[HIGH] xet_get download timeout (7200s), cache miss
3	HF weight-download stall/timeout at startup	nightly-8-gpu-mi35x-grok1-int4	test/registered/amd/accuracy/mi35x/test_grok1_int4_eval_mi35x.py	test_grok1_int4_accuracy	🆕 new today (passed Jun 20/21)	[MEDIUM] xet_get stall, no progress past Load weight begin
4	HF weight-download stall/timeout at startup	nightly-accuracy-2-gpu-vlm	test/registered/amd/accuracy/mi30x/test_vlms_mmmu_eval_amd.py	test_mmmu_vlm_models (deepseek-vl2-small)	🆕 new today	[FACT] deepseek-vl2 xet_get stall (py-spy)
5	GSM8K accuracy below threshold (Mistral/Mixtral) MI30x	nightly-accuracy-2-gpu	test/registered/amd/accuracy/mi30x/test_gsm8k_eval_amd.py	test_gsm8k_all_models	recurring since ≤Jun 14	[MEDIUM] chat-eval prompt/answer-extraction → #27757
6	GSM8K accuracy below threshold (Mistral/Mixtral) MI30x	nightly-accuracy-2-gpu	test/registered/amd/accuracy/mi30x/test_gsm8k_eval_amd.py	test_gsm8k_all_models	recurring since ≤Jun 14	[MEDIUM] chat-eval prompt/answer-extraction → #27757
7	VLM MMMU accuracy below threshold MI30x	nightly-accuracy-2-gpu-vlm	test/registered/amd/accuracy/mi30x/test_vlms_mmmu_eval_amd.py	test_mmmu_vlm_models	recurring/never-passed (Jun 14–22)	[MEDIUM] thresholds miscalibrated vs near-random scores
8	VLM MMMU accuracy below threshold MI30x	nightly-accuracy-2-gpu-vlm	test/registered/amd/accuracy/mi30x/test_vlms_mmmu_eval_amd.py	test_mmmu_vlm_models	recurring/never-passed	[LOW] thresholds vs genuine VLM regression
9	ROCm mamba extra_buffer assert at startup (Qwen3.5)	nightly-8-gpu-mi35x-qwen35	test/registered/amd/accuracy/mi35x/test_qwen35_eval_mi35x.py	TestQwen35EvalMI35x.test_lm_eval	never-passed (Jun 20–22)	[HIGH] extra_buffer gated CUDA/MUSA/NPU → #27141
10	ROCm mamba extra_buffer assert at startup (Qwen3.5)	nightly-8-gpu-qwen35	test/registered/amd/accuracy/mi30x/test_qwen35_eval_amd.py	TestQwen35EvalAMD.setUpClass	recurring (Jun 20–22)	[HIGH] extra_buffer gated CUDA/MUSA/NPU → #27141
11	HIP capturing-stream abort during decode CUDA-graph capture	nightly-8-gpu-mi35x-qwen3-235b-mxfp4	test/registered/amd/test_qwen3_instruct_mxfp4.py	TestQwen3Instruct2507MXFP4.setUpClass	never-passed (Jun 15–22)	[MEDIUM] RCCL watchdog queries HIP event in capturing stream
12	Kimi-K2.6 MoE w2 weight-load IndexError	nightly-8-gpu-kimi-k26	test/registered/amd/accuracy/mi30x/test_kimi_k26_eval_amd.py	TestKimiK26EvalAMD.setUpClass	recurring 7d (since Jun 17, last green Jun 16)	[HIGH] f06e2d3d1f (#27690) flipped load_full_w2 → #28905
13	GPU memory access fault (write to read-only page) in VLM forward	nightly-4-gpu	test/registered/vlm/test_encoder_dp.py	test_vlm_mmmu_benchmark	recurring since Jun 20, last green Jun 15	[LOW] regression in 33719cfb..ff1fc1fb or base-image drift
14	Qwen2-57B server failed to start (launch timeout)	nightly-accuracy-2-gpu	test/registered/amd/accuracy/mi30x/test_gsm8k_eval_amd.py	test_gsm8k_all_models (Qwen2-57B-A14B)	🆕 new today (passed Jun 21/22)	[LOW] prior Mixtral ~50min hang left GPUs degraded

Cluster: HF weight-download stall/timeout at startup

• Affected: deepseek-r1-hicache (test_deepseek_r1_hicache_mi35x.py), kimi-k26-mi35x (test_kimi_k26_eval_mi35x.py::test_kimi_k26_gsm8k_accuracy), grok1-int4 (test_grok1_int4_eval_mi35x.py::test_grok1_int4_accuracy), vlm-Jun23 (test_vlms_mmmu_eval_amd.py::test_mmmu_vlm_models deepseek-vl2-small).
• Shared facts: all stuck in download_weights_from_hf (weight_utils.py:554/593) → xet_get/filelock; cache-miss correlated (kimi loads in 1326s on cache-hit days); aiter constant; pure network/storage stall, not sglang code.
• Top hypothesis: [HIGH] runner HF-cache eviction + slow/hung Xet backend on large MoE checkpoints. Disconfirming: same models load fine on cache-hit days, so not a deterministic code path.
• In-flight fix: ❌ none across all four.
• Suggested triage: pre-warm/persist /sgl-data/hf-cache for these models; test HF_HUB_DISABLE_XET=1; reconcile per-file timeout vs est_time (deepseek-r1 est_time=5400 > timeout=3600).

Cluster: GSM8K accuracy below threshold (Mistral/Mixtral) MI30x

• Affected: nightly-accuracy-2-gpu Jun22 + Jun23, both test_gsm8k_eval_amd.py::test_gsm8k_all_models.
• Shared facts: Mistral-7B (~0.36), Mixtral-8x7B (~0.29), FP8 variants deterministically below threshold every run; 12 other models pass. gemma-2-27b-it borderline (0.855–0.861).
• Top hypothesis: [MEDIUM] chat-eval prompt/answer-extraction mishandles Mistral/Mixtral formatting. Disconfirming: gemma near-miss is a threshold-margin issue, separate cause.
• In-flight fix: ✅ #27757 (open since 2026-06-10) — chase reviewers.
• Suggested triage: validate #27757 recovers scores on MI30x; review gemma 0.86 threshold separately.

Cluster: VLM MMMU accuracy below threshold MI30x

• Affected: vlm Jun22 + Jun23, test_vlms_mmmu_eval_amd.py::test_mmmu_vlm_models.
• Shared facts: Qwen2-VL/Qwen2.5-VL/Qwen3-VL-30B/MiniCPM-o bit-stable ~0.26–0.29 (near 4-choice random ~0.25); 6 other VLMs pass; 100-sample eval is noisy.
• Top hypothesis: [MEDIUM] thresholds miscalibrated for AMD MI30x. Disconfirming: near-random scores could indicate genuine vision-path regression (image input dropped).
• In-flight fix: ❌ none.
• Suggested triage: A/B same models + 100-sample MMMU on NVIDIA; if NVIDIA ~0.28 → recalibrate thresholds, else bisect AMD multimodal path.

Cluster: ROCm mamba extra_buffer assert at startup (Qwen3.5)

• Affected: mi35x-qwen35 (test_qwen35_eval_mi35x.py::test_lm_eval), qwen35 (test_qwen35_eval_amd.py::setUpClass).
• Shared facts: identical AssertionError: extra_buffer needs CUDA/MUSA/NPU (FLA) at server_args.py:4366; auto mamba-radix-cache selects extra_buffer (FLA-gated to CUDA/MUSA/NPU) for hybrid Qwen3.5 on ROCm; 100% deterministic.
• Top hypothesis: [HIGH] auto→extra_buffer selection has no ROCm guard. Disconfirming: (none found).
• In-flight fix: ✅ #27141 (open since 2026-06-03) — chase reviewers.
• Suggested triage: verify #27141 relaxes the assert for ROCm; interim, run with --mamba-radix-cache-strategy no_buffer.

Cluster: HIP capturing-stream abort during decode CUDA-graph capture

• Affected: mi35x-qwen3-235b-mxfp4, test_qwen3_instruct_mxfp4.py::setUpClass.
• Shared facts: ProcessGroupNCCL.cpp:2055 watchdog raises hipErrorCapturedEvent/hipErrorStreamCaptureUnsupported during Capture decode CUDA graph (bs=512); aiter JIT-builds pa_ragged inside capture; SIGABRT exit -6; identical across Jun 15–22.
• Top hypothesis: [MEDIUM] RCCL watchdog queries a HIP event in a stream under capture. Disconfirming: abort site ≠ trigger; underlying op not pinned.
• In-flight fix: ❌ none ( #23581 / #27650 speculative only).
• Suggested triage: re-run with --disable-decode-cuda-graph; test SGLANG_USE_AITER_AR=0; pre-warm pa_ragged.

Cluster: Kimi-K2.6 MoE w2 weight-load IndexError

• Affected: nightly-8-gpu-kimi-k26, test_kimi_k26_eval_amd.py::setUpClass.
• Shared facts: IndexError: start out of range … got {192,256,320,448} (=64×k) at fused_moe_triton/layer.py:566 _load_w2.narrow; loader narrows already-per-rank (dim-64) w2 at full-TP offset. First seen Jun 17 (f5b041622b), last green Jun 16 (33f205d8c5).
• Top hypothesis: [HIGH] f06e2d3d1f (#27690) flipped load_full_w2 = (actorder != "static") → allocates full-TP-width w2 for per-rank Kimi checkpoint. Disconfirming: crash is on the w2 weight narrow vs the changed w2 scale alloc; 0ae4740bd1 (#27328) also in window.
• In-flight fix: ✅ #28905 (open since 2026-06-22) — directly targets this; only tested MI35x, MI30x unverified.
• Suggested triage: apply #28905, re-run on 8×MI30x; else bisect the two #27690/#27328 commits.

Cluster: GPU memory access fault (write to read-only page) in VLM forward

• Affected: nightly-4-gpu, test_encoder_dp.py::test_vlm_mmmu_benchmark.
• Shared facts: Memory access fault by GPU node-2 … Write access to a read-only page on first inference (Qwen3-VL-32B & Qwen2.5-VL-72B, --mm-enable-dp-encoder --tp 4) → exit -9; last green Jun 15, fails Jun 20–22; aiter constant.
• Top hypothesis: [LOW] regression in sglang delta 33719cfb..ff1fc1fb OR base-image drift; no commit touches VLM/DP-encoder paths. Disconfirming: none of the 20 commits is a plausible culprit on inspection.
• In-flight fix: ❌ none.
• Suggested triage: image A/B (rerun current SHA in -20260615 image) first; then git bisect the window; capture coredump.

Cluster: Qwen2-57B server failed to start (launch timeout)

• Affected: nightly-accuracy-2-gpu Jun23, test_gsm8k_eval_amd.py::test_gsm8k_all_models (Qwen2-57B-A14B, TP=2).
• Shared facts: TimeoutError: Server failed to start after ~50min of Mixtral soft-watchdog timeouts; passed Jun 21/22 (0.79–0.81); converted the run from accuracy-FAIL to ERROR.
• Top hypothesis: [LOW] prior Mixtral hang left GPUs/processes degraded. Disconfirming: Jun 22 also had Mixtral timeouts yet Qwen started fine; could be transient infra.
• In-flight fix: ❌ none — needs a re-run to classify flake vs regression.
• Suggested triage: re-run to confirm flake; if persistent bisect 43b1fb95..ed26a109; isolate per-model server lifecycle.

Suggested triage order

Rank	Triage step	Targets	Why this first
1	Verify & merge in-flight #28905 on 8×MI30x (MI30x unverified)	row 12 (Kimi w2 IndexError)	Existing fix, highest-confidence root cause (7d persistent regression)
2	Verify & merge in-flight #27141	rows 9–10 (Qwen3.5 mamba)	Existing fix resolves 2 jobs, [HIGH] deterministic assert
3	Pre-warm/persist HF cache + try HF_HUB_DISABLE_XET=1; reconcile timeouts	rows 1–4 (download stalls)	Largest cluster (4 rows), cheap infra fix, no code change
4	Verify & merge in-flight #27757	rows 5–6 (gsm8k Mistral/Mixtral)	Existing fix, 2 jobs
5	A/B MMMU scores AMD vs NVIDIA to decide threshold vs regression	rows 7–8 (VLM MMMU)	Cheap discriminator before any code work
6	Re-run + bisect new-today single failures	rows 2,3,4,14 (grok1, deepseek-vl2, Qwen2-57B)	Confirm flake vs fresh regression cheaply
7	Image A/B then bisect 33719cfb..ff1fc1fb; re-run with --disable-decode-cuda-graph	rows 11, 13 (HIP capture, VLM mem fault)	[LOW]-confidence, need evidence-gathering before fix direction

---

Job	Failed Steps	Started
nightly-accuracy-2-gpu-vlm	Nightly Accuracy Test (2-GPU VLM MMMU)	2026-06-22T19:41
nightly-8-gpu-mi35x-deepseek-r1-hicache	Accuracy Test MI35x (8-GPU DeepSeek-R1 HiCache)	2026-06-22T22:31
nightly-accuracy-2-gpu	Nightly Test (2-GPU)	2026-06-22T19:20
nightly-8-gpu-mi35x-grok1-int4	Accuracy Test MI35x (8-GPU Grok1-INT4)	2026-06-22T19:15
nightly-8-gpu-mi35x-qwen35	Accuracy Test MI35x (8-GPU Qwen 3.5)	2026-06-22T21:19
nightly-8-gpu-qwen35	Accuracy Test (8-GPU Qwen 3.5)	2026-06-22T21:43
nightly-4-gpu	Nightly Test (4-GPU)	2026-06-22T18:46
nightly-8-gpu-mi35x-qwen3-235b-mxfp4	Accuracy Test + Performance Test MI35x (8-GPU Qwen3-235B-MXFP4)	2026-06-22T20:19
nightly-8-gpu-mi35x-kimi-k26	Accuracy Test MI35x (8-GPU Kimi-K2.6)	2026-06-22T23:31
nightly-8-gpu-kimi-k26	Accuracy Test (8-GPU Kimi-K2.6)	2026-06-23T01:08
nightly-accuracy-2-gpu-vlm	Nightly Accuracy Test (2-GPU VLM MMMU)	2026-06-23T18:39
nightly-accuracy-2-gpu	Nightly Test (2-GPU)	2026-06-23T19:01

Per-Job Analysis

nightly-accuracy-2-gpu-vlm — failed step(s): Nightly Accuracy Test (2-GPU VLM MMMU)

Commit Info

• sglang: 43b1fb95 ("[CI] Optimize nixl dependency installation script", #28950)
• aiter: 7d604afe5 (v0.1.14rc1.dev240+g7d604afe5) — source: dev image ([CI-AITER-CHECK] Dev/patched version detected → skipping rebuild; Dockerfile default matches the same SHA)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi30x/test_vlms_mmmu_eval_amd.py	test_mmmu_vlm_models	AssertionError: The following models failed (5 of 11 VLMs below MMMU threshold)	Log

The single test parametrizes 11 VLMs; 6 passed, 5 failed:

Model	TP	Score	Threshold	Status
Qwen/Qwen2-VL-7B-Instruct	1	0.280	0.30	❌
Qwen/Qwen2.5-VL-7B-Instruct	1	0.280	0.33	❌
Qwen/Qwen3-VL-30B-A3B-Instruct	2	0.260	0.29	❌
openbmb/MiniCPM-o-2_6	1	0.280	0.32	❌
deepseek-ai/deepseek-vl2-small	1	0.280	0.31	❌
InternVL2_5-2B / MiniCPM-v-2_6 / Janus-Pro-7B / Kimi-VL / MiMo-VL / DeepSeek-OCR-2	—	0.29–0.35	—	✅

Failure Summary

The AMD MI30x VLM MMMU accuracy test failed because 5 of 11 vision-language models scored below their configured MMMU accuracy thresholds (test_vlms_mmmu_eval_amd.py:380). The achieved scores cluster around 0.26–0.28 — close to MMMU 4-choice random chance (~0.25) — while the thresholds for these models sit at 0.29–0.33. The HEAD commit (43b1fb95, a nixl CI install script change) does not touch any VLM or multimodal code path.

Regression Status

Known recurring failure (the four core models are Never-passed in the queryable window).

Recent history of test_vlms_mmmu_eval_amd.py (test_mmmu_vlm_models) in job nightly-accuracy-2-gpu-vlm (branch main, event=schedule, completed runs only):

Date	Run	Test File Status	Failing models (score)
Jun 22	27975987522	❌ Failed	Qwen2-VL (0.280), Qwen2.5-VL (0.280), Qwen3-VL (0.260), MiniCPM-o (0.280), deepseek-vl2 (0.280)
Jun 21	27912890405	❌ Failed	same 5 (Qwen3-VL 0.2601, deepseek-vl2 0.2802)
Jun 20	27879266774	❌ Failed	Qwen2.5-VL, Qwen3-VL, MiniCPM-o, deepseek-vl2 (Qwen2-VL passed at exactly 0.300)
Jun 19–16	(4 runs)	— Cancelled	not comparable (excluded)
Jun 15	27568949589	❌ Failed	same 5 (Qwen3-VL 0.2601, deepseek-vl2 0.2802)
Jun 14	27507337214	❌ Failed	same 5 (Qwen3-VL 0.2601, deepseek-vl2 0.2802)

First observed failure for these models: at/before Jun 14 (oldest queryable completed run; Jun 16–19 runs were cancelled and the only earlier comparable data is older). No green run for Qwen2.5-VL / Qwen3-VL / MiniCPM-o / deepseek-vl2 exists in the queryable window → treat as Never-passed. Qwen2-VL is borderline/flaky (passed once at exactly 0.300 on Jun 20).

Failure Cluster

VLM MMMU accuracy below threshold on AMD MI30x (scores pinned near random chance)

Facts (verified observations)

• AssertionError raised at test_vlms_mmmu_eval_amd.py:380 via _check_results; job exit code 255 (log).
• The same 4 models fail in every completed scheduled run from Jun 14 → Jun 22 (Jun 16–19 cancelled, excluded).
• Scores are not just below threshold but bit-stable across runs: Qwen3-VL-30B = 0.2601 on Jun 14/15/21/22; deepseek-vl2-small = 0.2802 on Jun 14/15/20/21/22; Qwen2.5-VL = 0.2802 on Jun 15/20/21/22. Identical per-category breakdowns repeat across runs.
• Configured thresholds (test_vlms_mmmu_eval_amd.py:50–89): Qwen2-VL 0.30, Qwen2.5-VL 0.33, Qwen3-VL 0.29, MiniCPM-o 0.32, deepseek-vl2 0.31.
• aiter is unchanged (dev-image default, no override) across these runs — [CI-AITER-CHECK] markers report the same 7d604afe5 each run.
• HEAD commit 43b1fb95 modifies only the nixl CI install script; no VLM/multimodal code in the diff.

Hypothesised Causes (with confidence)

• [MEDIUM] Thresholds in test_vlms_mmmu_eval_amd.py are miscalibrated (too high) for actual MI30x VLM accuracy — i.e. a test-config issue, not a runtime regression.
  • Supporting: scores are stable to 4 decimal places across many independent runs and never approach the thresholds; the comment at line 43 calls them "conservative thresholds" yet 5 models sit consistently under them; 6 other models pass comfortably, so the harness/eval itself works.
  • Disconfirming: the failing scores (~0.26–0.28) hover at MMMU 4-choice random chance, which could instead indicate the models are not actually consuming image input (a real accuracy bug), making the thresholds correct and the models broken.
  • How to verify: run the same models + MMMU 100-sample eval on an NVIDIA reference and compare scores; if NVIDIA also lands ~0.28, the threshold is wrong; if NVIDIA scores ~0.5+, AMD multimodal is regressed.
• [LOW] Genuine AMD MI30x VLM accuracy regression in image preprocessing / vision path (models guessing at random chance).
  • Supporting: scores pinned near 0.25–0.28 across diverse architectures (Qwen, MiniCPM, deepseek-vl2) is the signature of vision features being dropped; several models share an exact 0.2802.
  • Disconfirming: 6 other VLMs (InternVL, Kimi-VL, MiMo-VL, DeepSeek-OCR-2, Janus-Pro, MiniCPM-v) score 0.29–0.35 and pass, so the multimodal path is not globally broken; failure is model-specific.
  • How to verify: inspect raw model generations for one failing model (e.g. Qwen2.5-VL) to confirm whether answers reference image content or are degenerate.

No hypothesis reaches HIGH; the bit-stable repeated scores point at a config/calibration issue more than a fresh regression. This is not introduced by the HEAD commit.

In-flight Fix Check

• In-flight fix: ❌ none found targeting the AMD MMMU VLM accuracy thresholds. Searched open PRs for mmmu+vlm, Qwen2.5-VL+accuracy — results are unrelated feature/bugfix PRs (e.g. #28828 GPU JPEG-to-RGB grayscale-crash fix, #26398 Qwen2.5-VL fused rotary), none addressing this nightly accuracy failure.

Suggested Triage Steps

• Run the 5 failing models through the identical MMMU 100-sample eval on an NVIDIA nightly runner; compare scores to decide between "thresholds too high" vs "AMD VLM accuracy regression".
• For one failing model (e.g. Qwen2.5-VL-7B), dump raw generations on MI30x and confirm whether responses reflect image content — distinguishes config issue from broken vision path.
• If NVIDIA scores match (~0.28), recalibrate the thresholds in test/registered/amd/accuracy/mi30x/test_vlms_mmmu_eval_amd.py:50–89 to AMD-realistic values (and consider widening the 100-sample eval, which is noisy).
• If NVIDIA scores are materially higher, bisect the AMD multimodal/vision path; the regression predates Jun 14 (oldest queryable completed run), so widen the lookback beyond the Jun 16–19 cancelled-run gap to find the last green scheduled run.

Status (factual)

• Persistence: The 4 core models (Qwen2.5-VL, Qwen3-VL, MiniCPM-o, deepseek-vl2) have failed in every completed scheduled run from Jun 14 through Jun 22 (Jun 16–19 cancelled); no green run in the queryable window. Qwen2-VL is borderline (passed once, Jun 20).
• Scope: 1 job (nightly-accuracy-2-gpu-vlm) in nightly-test-amd.yml; not observed as a separate cluster elsewhere. Blocks AMD VLM accuracy CI signal.
• In-flight fix: ❌ none found — needs a new PR (threshold recalibration) or investigation into AMD VLM accuracy.

nightly-8-gpu-mi35x-deepseek-r1-hicache — failed step(s): Accuracy Test MI35x (8-GPU DeepSeek-R1 HiCache)

Commit Info

• sglang: 43b1fb95
• aiter: 7d604afe5 (v0.1.14rc1.dev240+g7d604afe5; source: dev/patched image preserved — [CI-AITER-CHECK] Dev/patched version detected … → skipping rebuild)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/test_deepseek_r1_hicache_mi35x.py	N/A (whole-file timeout; server never reached ready)	TIMEOUT after 3600s — server stuck loading weights	Log

Failure Summary

The single test file test/registered/amd/test_deepseek_r1_hicache_mi35x.py (DeepSeek-R1-0528, TP8, on MI35x) was killed by the harness TIMEOUT after 3600 seconds (Test Summary: 0/1 passed, Process completed with exit code 255). The SGLang server never became ready: all 8 TP scheduler processes spent the entire ~55 min (22:45 → 23:35 UTC) blocked inside HF weight download, and the TokenizerManager kept hitting _wait_for_scheduler_ready watchdog timeouts. No test assertion ran.

Regression Status

Known recurring failure — test_deepseek_r1_hicache_mi35x.py has not passed in the queryable window (every completed scheduled run since Jun 20 timed out at 3600s; Jun 16–19 runs were cancelled, not comparable). Note the failure symptom changed today.

Recent history of test_deepseek_r1_hicache_mi35x.py in job nightly-8-gpu-mi35x-deepseek-r1-hicache (branch main, event schedule, completed runs only):

Date	Run	Job	Test File Status	Failure mode
Jun 22	run	job	❌ Failed	TIMEOUT 3600s — stuck in HF weight download the entire time; no GPU activity, no retry
Jun 21	run	job	❌ Failed	TIMEOUT 3600s — got past download, then Memory access fault on all 8 GPU nodes; retried, timed out
Jun 20	run	job	❌ Failed	TIMEOUT 3600s — Memory access fault on all 8 GPU nodes; retried, timed out
Jun 19	run	—	⏭️ cancelled	run cancelled — not comparable
Jun 15	run	job	❌ Failed	(not inspected in detail)

No green run for this test file found in the queryable window (Jun 15 → Jun 22 completed scheduled runs).

Failure Cluster

"DeepSeek-R1-0528 hicache test timeout at 3600s — server never ready" (symptom-level grouping). Today's sub-symptom: stuck in single-threaded HF weight download under the global cache lock.

Facts (verified observations)

• ✗ TIMEOUT: …/test_deepseek_r1_hicache_mi35x.py after 3600 seconds → Test Summary: 0/1 passed → Process completed with exit code 255 (log lines 9784–9801).
• All 8 TP scheduler MainThreads idle in the same stack for the full run (first dump 22:45:50, last dump 23:35:54), top frames:
  select (libc) → _poll_until_acquired (filelock/_api.py:552) → acquire (_api.py:506) → download_weights_from_hf (model_loader/weight_utils.py:554) → ci_download_with_validation_and_retry (model_loader/ci_weight_validation.py:1955) → … → load_model (model_runner.py:1423) → initialize (model_runner.py:693).
• weight_utils.py:554 is with get_lock(model_name_or_path, cache_dir): — one global lock wrapping validation+download. ci_weight_validation.py:1955 forces max_workers=1 (single-threaded download, intentional to avoid NFS .incomplete races). The lock holder (TP6, pid 3634) could not be py-spy'd (Pyspy failed … Failed to get stack traces) — consistent with it being busy in the native single-threaded download; the other 7 ranks block on its lock.
• TokenizerManager MainThread idle in _wait_for_scheduler_ready (engine.py:1355) → repeated TokenizerManager watchdog timeout and Scheduler watchdog timeout (watchdog_timeout=300, soft=True).
• Today's log: Memory access fault count = 0; no retry attempt — i.e. download never completed, so the GPU/forward path was never reached (unlike Jun 20/21).
• Config mismatch: the test declares est_time=5400 (test_deepseek_r1_hicache_mi35x.py:37) but the harness runs with --timeout-per-file 3600 (log line 4180). The test's own estimate (90 min) already exceeds the enforced 60 min limit.
• Runner network flakiness earlier in the job: [local-pull] Error response from daemon: Get "https://10.245.143.50:5000/v2/": context deadline exceeded (line 1281) before falling back to pulling rocm/sgl-dev:v0.5.13.post1-rocm700-mi35x-20260622.
• Harness warning: Unknown suite nightly-amd-8-gpu-mi35x-deepseek-r1-hicache for backend AMD, nightly=True (line 4192) — suite still resolved and ran.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [HIGH] Infrastructure/timeout: cold-cache, single-threaded HF download of the ~671B DeepSeek-R1-0528 weights did not finish within the 3600s file timeout.
  • Supporting: all 8 ranks stuck on the global download lock for the entire run with zero progress past load_model; max_workers=1 forced download; test est_time=5400 > timeout 3600; no GPU fault and no eval ever started.
  • Disconfirming: cannot see download byte-progress in the log (py-spy can't dump the lock-holder), so a true hang (vs merely-slow) on the HF/NFS side cannot be fully excluded.
  • How to verify: re-run with weights pre-warmed in the runner cache, or raise --timeout-per-file to ≥ est_time (5400s) and watch whether load completes; check HF/NFS throughput on the MI35x runner.
• [LOW] GPU-side regression (the Jun 20/21 "Memory access fault during/after load on all 8 nodes").
  • Supporting: prior two completed runs crashed with GPU memory faults once load completed — a real failure mode for this config.
  • Disconfirming: today never reached the GPU path at all (0 faults), so it cannot explain today's timeout; it is a separate downstream symptom of the same job.
  • How to verify: only investigable on a run where weights load successfully; bisect would require a green load first.
• [SPECULATION] aiter dev build v0.1.14rc1.dev240+g7d604afe5.
  • Supporting: AMD-only path.
  • Disconfirming: failure is entirely in HF weight download (pure Python/filelock/network) before any kernel runs — aiter is not on the stuck stack.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for deepseek-r1-hicache, timeout-per-file, hicache+mi35x, ci_weight_validation; no PR addresses this job's download timeout. (Closest unrelated: #28430 HiCache TP-deadlock, #28455 DeepSeek-V4 fp8 KV — neither touches weight download or this test's timeout.)

Suggested Triage Steps

• Pre-warm the DeepSeek-R1-0528 weight cache on the MI35x runner (or use a shared/NFS snapshot) so the test doesn't pay a cold single-threaded download under the 3600s budget.
• Reconcile the timeout budget: either raise --timeout-per-file to ≥ est_time (5400s) for this suite, or lower est_time if a warm cache is guaranteed — currently est_time=5400 > timeout=3600.
• Separately track the Jun 20/21 GPU Memory access fault symptom: once weights load in time, re-run and capture a coredump/py-spy at fault time to triage the GPU-side crash (distinct from today's download timeout).
• Confirm runner network/registry health — the early 10.245.143.50:5000 registry timeout suggests intermittent network pressure on this runner that could also slow HF downloads.

Status (factual)

• Persistence: test_deepseek_r1_hicache_mi35x.py failing every completed scheduled run since at least Jun 20 (3 consecutive completed runs; also failed Jun 15). No green run in the queryable window.
• Scope: single job nightly-8-gpu-mi35x-deepseek-r1-hicache in nightly-test-amd.yml; single test file. Symptom varies run-to-run (today: download timeout; Jun 20/21: GPU memory fault after load).
• Blocked work: DeepSeek-R1-0528 hicache nightly accuracy/perf signal on MI35x — no usable data produced.
• In-flight fix: ❌ none found in open PRs.

nightly-accuracy-2-gpu — failed step(s): Nightly Test (2-GPU)

Commit Info

• sglang: 43b1fb95
• aiter: 7d604afe5 (v0.1.14rc1.dev240+g7d604afe5, source: dev image preserved — [CI-AITER-CHECK] Dev/patched version detected ... → skipping rebuild)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi30x/test_gsm8k_eval_amd.py	test_gsm8k_all_models	AssertionError: The following models failed: (5 models below GSM8K threshold)	Log

The 5 sub-models below threshold:

Model	TP	Score	Threshold
mistralai/Mistral-7B-Instruct-v0.3	1	0.364	0.47
google/gemma-2-27b-it	1	0.858	0.86 (borderline, -0.002)
mistralai/Mixtral-8x7B-Instruct-v0.1	2	0.301	0.69
neuralmagic/Mistral-7B-Instruct-v0.3-FP8	1	0.325	0.46
neuralmagic/Mixtral-8x7B-Instruct-v0.1-FP8	2	0.189	0.69

Failure Summary

test_gsm8k_all_models evaluates 17 nightly instruction models on GSM8K; 12 passed and 5 failed (Passed: 12 / Failed: 5). The Mistral-7B and Mixtral-8x7B families (both bf16 and FP8 variants) score far below their thresholds (0.19–0.36 vs 0.46–0.69), while gemma-2-27b-it misses by a hair (0.858 vs 0.86). The assertion is raised in check_model_scores (test_gsm8k_eval_amd.py:201), called from the test at line 332.

Regression Status

Known recurring failure / Never-passed in queryable window — the same Mistral/Mixtral models fail with near-identical scores on every recent completed schedule run.

Recent history of test_gsm8k_eval_amd.py (test_gsm8k_all_models) in job nightly-accuracy-2-gpu (branch main, event schedule, completed runs only):

Date	Run	Job	Test File Status	Failed Functions / Models	Notes
Jun 22	27975987522	nightly-accuracy-2-gpu	❌ Failed	Mistral-7B 0.364, Mixtral 0.301, both FP8, gemma-2-27b 0.858	this run
Jun 21	27912890405	nightly-accuracy-2-gpu	❌ Failed	Mistral-7B 0.363, Mixtral 0.294, both FP8, gemma-2-27b 0.859	same 5 models
Jun 20	27879266774	nightly-accuracy-2-gpu	❌ Failed	Mistral-7B ~0.37, gemma 0.855 (log truncated mid-run)	same pattern
Jun 15	27568949589	nightly-accuracy-2-gpu	❌ Failed	Mistral-7B ~0.36 (log truncated mid-run)	same pattern
Jun 14	27507337214	nightly-accuracy-2-gpu	❌ Failed	Mixtral ~0.30 + a RemoteProtocolError (truncated model download) on top	same pattern + infra noise

No passing run for this test file found in the queryable window (Jun 14–22; earlier runs Jun 16–19 were cancelled). The bf16/FP8 Mistral & Mixtral failures are deterministic (score variance < 0.02 run-to-run, score:std ≈ 0.46–0.48), which rules out flakiness.

Failure Cluster

GSM8K accuracy below threshold for Mistral/Mixtral chat models (AMD MI30x nightly).

Facts (verified observations, no interpretation)

• Verbatim assertion: AssertionError: The following models failed: listing Mistral-7B (0.3638/0.47), gemma-2-27b-it (0.8584/0.86), Mixtral-8x7B (0.3014/0.69), Mistral-7B-FP8 (0.3250/0.46), Mixtral-8x7B-FP8 (0.1895/0.69). Final stats Passed: 12 / Failed: 5. (log line 37871–37888)
• The same 5 models fail with the same scores on the Jun 21 run (Mistral 0.363, Mixtral 0.294, Mistral-FP8 0.330, Mixtral-FP8 0.194, gemma 0.859) — confirmed from [METRIC] gsm8k_score=... lines in 82592972349.
• score:std for the failing models is ~0.46–0.48 (binary right/wrong with ~20–36% correct), consistent across runs.
• aiter is unchanged across these runs (dev image v0.1.14rc1.dev240+g7d604afe5 preserved; no override, AITER_COMMIT_OVERRIDE empty).
• 12 other models (Llama-3.1-8B/70B, DeepSeek-Coder-V2-Lite, Qwen2/2.5/3, gemma-passing variants) pass their thresholds in the same run.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [MEDIUM] GSM8K chat-eval prompt / answer-extraction does not handle Mistral/Mixtral instruction-tuned output formatting, producing low parsed scores rather than a genuine model-quality collapse.
  • Supporting: only the Mistral/Mixtral families (both bf16 and FP8) are affected while structurally similar models pass; scores are stable and deterministic; in-flight PR #27757 ("Fix Mistral GSM8K chat eval") states the root cause is prompt/extraction (prefers explicit Answer: <number> extraction, plumbs stop sequences) and notes Mistral-7B still scored ~0.417 even after moving off FA3.
  • Disconfirming: gemma-2-27b-it is a near-miss (0.858 vs 0.86), which looks more like a threshold-margin issue than an extraction failure — it may not belong to the same cause as the Mistral/Mixtral models.
  • How to verify: run the GSM8K eval against mistralai/Mistral-7B-Instruct-v0.3 on MI30x with PR #27757's extraction patch and compare parsed score; inspect raw completions for unparsed answers.
• [LOW] Genuine accuracy regression in the Mistral/Mixtral forward path on AMD.
  • Supporting: scores are very low (0.19–0.36).
  • Disconfirming: no green baseline exists in the window to regress from; scores are flat across ≥5 runs (a regression would show a step change); FP8 and bf16 variants fail identically, which is more consistent with an eval-harness issue than two independent kernel regressions.
  • How to verify: bisect is not actionable without a known-good run; instead diff raw model outputs vs reference answers.

This is a nightly-test-amd.yml (non-scout) run, so no Baseline A/B / Failure Origin section applies.

In-flight Fix Check

• In-flight fix: ✅ #27757 ("Fix Mistral GSM8K chat eval", open since 2026-06-10, last updated 2026-06-22) — directly targets the Mistral/Mixtral GSM8K chat-eval prompt + answer extraction + stop sequences. Chase reviewers; do NOT open a duplicate. (API query: search/issues?q=repo:sgl-project/sglang+is:pr+is:open+gsm8k+accuracy.)
• Note: PR #27757 likely does not address the gemma-2-27b-it near-miss (0.858 vs 0.86), which may need a separate threshold review.

Suggested Triage Steps

• Verify #27757 recovers the Mistral/Mixtral scores by running test_gsm8k_eval_amd.py on AMD MI30x with the PR branch; if so, prioritize its review/merge.
• Separately, inspect the gemma-2-27b-it margin (0.858 vs 0.86 = -0.002): decide whether the 0.86 threshold is too tight for run-to-run noise or whether it should stay and gemma be tracked independently.
• If #27757 does not fully recover scores, dump raw GSM8K completions for mistralai/Mistral-7B-Instruct-v0.3 to distinguish unparsed-answer (harness) from wrong-answer (model) cases.

Status (factual, no priority assignment)

• Persistence: Failing on every completed schedule run in the queryable window (at least Jun 14 → Jun 22); no green run found for this test file in that window.
• Scope: nightly-accuracy-2-gpu job in nightly-test-amd.yml; 5 of 17 nightly models (Mistral-7B, Mixtral-8x7B, their FP8 variants, gemma-2-27b-it near-miss).
• Blocked work: Blocks a green nightly accuracy signal on AMD MI30x for the Mistral/Mixtral model families.
• In-flight fix: ✅ #27757 (open since 2026-06-10) — awaiting merge.

nightly-8-gpu-mi35x-grok1-int4 — failed step(s): Accuracy Test MI35x (8-GPU Grok1-INT4)

Commit Info

• sglang: 43b1fb95
• aiter: N/A (no [CI-AITER-CHECK] markers in this job log; nightly job runs against the prebuilt rocm/sgl-dev:v0.5.13.post1-rocm700-mi35x-20260622 image)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi35x/test_grok1_int4_eval_mi35x.py	test_grok1_int4_accuracy	TimeoutError: Server failed to start within the timeout period	Log

Failure Summary

The single test test_grok1_int4_accuracy (Grok-1 INT4 GSM8K eval on MI35x, TP=8) failed because the server never became ready within its 3600s startup budget. Weight loading began at 19:24:12 but never completed; py-spy dumps show all 8 TP scheduler processes blocked downloading the amd/grok-1-W4A8KV8 weights from HuggingFace. The test was killed after Ran 1 test in 3609.457s.

Regression Status

New failure today (this test passed in the 3 most recent completed scheduled runs).

Recent history of test_grok1_int4_eval_mi35x.py (test_grok1_int4_accuracy) in job nightly-8-gpu-mi35x-grok1-int4 (branch main, event schedule, completed runs only):

Date	Run	Job	Test File Status	Failed Function	Error
Jun 22	27975987522	nightly-8-gpu-mi35x-grok1-int4	❌ Failed	test_grok1_int4_accuracy	TimeoutError: Server failed to start
Jun 21	27912890405	nightly-8-gpu-mi35x-grok1-int4	✅ Passed	—	—
Jun 20	27879266774	nightly-8-gpu-mi35x-grok1-int4	✅ Passed	—	—
Jun 15	27568949589	nightly-8-gpu-mi35x-grok1-int4	✅ Passed	—	—

(Jun 16–19 scheduled runs were cancelled, so excluded. First observed failure: Jun 22; last known pass: Jun 21.)

Failure Cluster

HuggingFace weight-download stall during server startup (Xet xet_get hang → watchdog/startup timeout)

Facts (verified observations, no interpretation)

• Server launch command: sglang serve --model-path amd/grok-1-W4A8KV8 --tp 8 --quantization fp8 --attention-backend aiter --mem-fraction-static 0.85 ... (log line 4239–4241).
• Load weight begin logged at 19:24:12 for all 8 TP ranks (log lines 4446–4453); no "Load weight end" ever appears.
• First watchdog timeout at 19:28:54 (TokenizerManager) and 19:29:02 (all Scheduler ranks) — watchdog_timeout=300, self.soft=True (log lines 4511, 4561+).
• py-spy dumps (both the early 19:29 dumps and the final 20:19 dump) show the active thread stuck in xet_get (huggingface_hub/file_download.py:566) inside _download_to_tmp_and_move → snapshot_download → ci_download_with_validation_and_retry (model_loader/ci_weight_validation.py:1955) → download_weights_from_hf (weight_utils.py:593) → _prepare_presharded_weights (grok.py:960). xet_get appears 11 times across the dumps — the download was hung for the full ~55 min.
• Final result: TimeoutError: Server failed to start within the timeout period (test_utils.py:1028), Ran 1 test in 3609.457s, FAILED (errors=1), job exit code 255.
• Earlier transient infra note (non-fatal): Local docker registry pull failed; falling back to public registry at 19:15:44 (log lines 1281–1282) — image still pulled successfully.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [MEDIUM] HuggingFace Xet-backend download stall / network flake while fetching the amd/grok-1-W4A8KV8 presharded weights — the run hung in xet_get for the entire hour and never progressed past "Load weight begin".
  • Supporting: every py-spy stack across two timepoints points to the same xet_get download call; no model/forward code was ever reached; the symptom is a stalled network read, not a crash or numeric error.
  • Disconfirming: cannot fully rule out a slow/contended runner disk or a node-local caching miss vs. an upstream HF outage from the log alone.
  • How to verify: re-run the job; check whether the weights were already cached on the runner; inspect HF Hub / Xet availability for amd/grok-1-W4A8KV8 around 19:24–20:19 UTC on 2026-06-22.
• [LOW] sglang code regression in the weight-download path (ci_weight_validation.py / weight_utils.py / grok.py) merged since the Jun 21 pass.
  • Supporting: the test passed on Jun 21 (7f67965b) and failed on Jun 22 (43b1fb95).
  • Disconfirming: the hang is inside third-party huggingface_hub xet_get, not in sglang code; a download stall is the classic signature of a network/infra flake, not a code change. No git log 7f67965b..43b1fb95 commit is implicated by the stack trace.
  • How to verify: git log --oneline 7f67965b4d..43b1fb95f6 -- python/sglang/srt/model_loader/ python/sglang/srt/models/grok.py to see if any download-path code changed in the window.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for grok1 int4 (2 unrelated PRs: #27209, #20894) and xet download (0 results). No PR targets this download-stall symptom.

Suggested Triage Steps

• Re-run nightly-8-gpu-mi35x-grok1-int4 — if it passes, this confirms a transient HF/Xet download stall (no code action needed).
• If it recurs, check whether amd/grok-1-W4A8KV8 weights are pre-cached on the MI35x runner and consider pre-staging them or setting HF_HUB_DISABLE_XET=1 to fall back to the standard HTTP download path.
• Only if reruns consistently fail: git log --oneline 7f67965b4d..43b1fb95f6 -- python/sglang/srt/model_loader/ python/sglang/srt/models/grok.py to check for any download-path change in the Jun 21→22 window.

Status (factual, no priority assignment)

• Persistence: 1 failure (first seen 2026-06-22 ~20:23 UTC); passed in the immediately preceding completed run on Jun 21.
• Scope: this one job/test only (nightly-8-gpu-mi35x-grok1-int4); symptom is server-startup weight download, not specific to attention/MoE/quant code.
• Blocked work: blocks Grok-1 INT4 accuracy signal on MI35x for the Jun 22 nightly.
• In-flight fix: ❌ none found in open PRs.

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-06-23 20:59 UTC)

[amd-bot]

nightly-test-amd.yml — Per-Job Analysis (continued 2/3)

nightly-8-gpu-mi35x-qwen35 — failed step(s): Accuracy Test MI35x (8-GPU Qwen 3.5)

Commit Info

• sglang: 43b1fb95
• aiter: 7d604afe5 (v0.1.14rc1.dev240+g7d604afe5; source: dev/patched image, rebuild skipped — AITER_COMMIT_OVERRIDE empty)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi35x/test_qwen35_eval_mi35x.py	TestQwen35EvalMI35x.test_lm_eval	Exception: Server process exited with code 1 → AssertionError: extra_buffer needs CUDA/MUSA/NPU (FLA).	Log

Failure Summary

The single accuracy test in this job (test_lm_eval, model Qwen/Qwen3.5-397B-A17B, --tp 8 --attention-backend aiter) failed because the sglang server process exited with code 1 on startup, both on the first attempt and the one retry. The server crashed during ServerArgs.__post_init__ while validating the mamba radix-cache configuration. No GSM8K/lm-eval scoring was ever reached.

Regression Status

Never-passed (no green run found in queryable window).

Recent history of test_qwen35_eval_mi35x.py (TestQwen35EvalMI35x.test_lm_eval) in job nightly-8-gpu-mi35x-qwen35 (branch main, event schedule, completed runs only):

Date	Run	Job	Test File Status	Failed Function	Error
Jun 22	27975987522	nightly-8-gpu-mi35x-qwen35	❌ Failed	test_lm_eval	AssertionError: extra_buffer needs CUDA/MUSA/NPU (FLA).
Jun 21	27912890405	nightly-8-gpu-mi35x-qwen35	❌ Failed	test_lm_eval	same assertion
Jun 20	27879266774	nightly-8-gpu-mi35x-qwen35	❌ Failed	test_lm_eval	same assertion

Runs of Jun 16–19 were cancelled (not comparable). Lookback window: last 8 scheduled runs (Jun 15–22). No passing run exists for this test file in that window — it is a recently added "Day 0" Qwen3.5 test that has never been green on AMD MI35x.

Failure Cluster

Server launch crash — AssertionError: extra_buffer needs CUDA/MUSA/NPU (FLA) for hybrid mamba model on ROCm.

Facts (verified observations)

• Verbatim crash (step 6 log):

  File ".../sglang/srt/server_args.py", line 4366, in _validate_mamba_extra_buffer
      assert (
  AssertionError: extra_buffer needs CUDA/MUSA/NPU (FLA).
  

  Call chain: __post_init__ → _handle_model_specific_adjustments (server_args.py:4208) → _handle_mamba_radix_cache (:4395) → _validate_mamba_extra_buffer (:4366).
• The assertion at python/sglang/srt/server_args.py:4366-4368 is assert is_cuda() or is_musa() or is_npu(), "extra_buffer needs CUDA/MUSA/NPU (FLA).". On AMD ROCm none of these are true (is_hip() is the live backend), so the assert fails deterministically.
• Server command: sglang serve --model-path Qwen/Qwen3.5-397B-A17B --tp 8 --attention-backend aiter --trust-remote-code --model-loader-extra-config '{"enable_multithread_load": true}' --watchdog-timeout 1200 --device cuda (--device cuda is the ROCm convention; the runtime backend is still HIP).
• Qwen3.5 is a hybrid mamba model ("MoE, Hybrid Attention with Gated Delta Networks" per the test docstring), so the mamba radix-cache path is exercised.
• _handle_mamba_radix_cache (:4378) with default args (radix cache on, mamba_radix_cache_strategy="auto", overlap schedule on) auto-selects extra_buffer then calls _validate_mamba_extra_buffer, which trips the CUDA/MUSA/NPU assert.
• Reproducibility: identical assertion + identical returned exit code 1 confirmed in all 3 completed runs (Jun 20/21/22). 1 test attempted, 0 passed; both retries failed within ~48s.
• This is NOT an amd-aiter-scout.yml run (workflow nightly-test-amd.yml, no override), so no Baseline A/B / Failure Origin section applies; aiter is not implicated.

Hypothesised Causes (with confidence)

• [HIGH] The extra_buffer mamba radix-cache strategy is gated to CUDA/MUSA/NPU only and is unsupported on ROCm, but the auto-selection logic picks it for hybrid Qwen3.5 on AMD, crashing server startup.
  • Supporting: the failing assert at server_args.py:4366-4368 literally excludes ROCm; the auto path at :4383-4395 selects extra_buffer whenever overlap/paging is wanted and the arch supports the extra buffer, with no ROCm guard; failure is 100% deterministic across 3 runs.
  • Disconfirming: (none found) — the crash is in pure arg validation, independent of aiter/GPU/runtime, and reproduces identically every run.
  • How to verify: launch Qwen/Qwen3.5-397B-A17B on MI35x with --mamba-radix-cache-strategy no_buffer (and --disable-overlap-schedule) or --disable-radix-cache; expect the server to start, confirming the extra_buffer path is the sole blocker.

In-flight Fix Check

• In-flight fix: ✅ #27141 "[ROCm] Support radix-cache with mamba-extra-buffer for Qwen3.5" (open since 2026-06-03, last updated 2026-06-22, not draft) — directly targets enabling this exact path on ROCm. Chase reviewers / verify it covers the _validate_mamba_extra_buffer assert; do NOT open a duplicate.
  • Search run: is:pr is:open mamba radix rocm and is:pr is:open mamba extra_buffer on sgl-project/sglang.

Suggested Triage Steps

• Verify #27141 merges cleanly against 43b1fb95 and that it relaxes/branches the is_cuda() or is_musa() or is_npu() assert at server_args.py:4366-4368 for ROCm; re-run job nightly-8-gpu-mi35x-qwen35.
• As an interim signal, the new test could pass --mamba-radix-cache-strategy no_buffer (or --disable-radix-cache) in other_args so the suite exercises Qwen3.5 accuracy on AMD until the extra_buffer ROCm path lands. Decision for maintainers.
• If #27141 is stalled, treat this as a Never-passed AMD port gap and consider gating the test off until the ROCm extra_buffer path is supported.

Status

• Persistence: Failing every completed run since the test appeared — Jun 20, 21, 22 (3/3 completed scheduled runs); earlier runs were cancelled, so no green baseline exists.
• Scope: Job nightly-8-gpu-mi35x-qwen35 (and the sibling non-mi35x nightly-8-gpu-qwen35, which also failed in the Jun 20/21 runs — likely the same assertion). Single workflow nightly-test-amd.yml.
• Blocked work: Blocks Qwen3.5-397B-A17B accuracy CI signal on AMD MI35x.
• In-flight fix: ✅ #27141 (open since 2026-06-03) — ROCm support for mamba-extra-buffer radix cache.

nightly-8-gpu-qwen35 — failed step(s): Accuracy Test (8-GPU Qwen 3.5)

Commit Info

• sglang: 43b1fb95
• aiter: 7d604afe5fa7efba63c0dce323b95d9daf2db112 (source: dev image preserved — Dev/patched version detected: v0.1.14rc1.dev240+g7d604afe5 → skipping rebuild)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi30x/test_qwen35_eval_amd.py	TestQwen35EvalAMD.setUpClass	AssertionError: extra_buffer needs CUDA/MUSA/NPU (FLA). (server exits code 1)	Log

Failure Summary

The single registered test test_qwen35_eval_amd.py failed in setUpClass because the SGLang server (Qwen/Qwen3.5-397B-A17B, --tp 8 --attention-backend aiter --device cuda) never came up — it exited with code 1 during argument validation. The crash is an AssertionError: extra_buffer needs CUDA/MUSA/NPU (FLA). raised in ServerArgs.__post_init__, so zero tests ran (Ran 0 tests).

Regression Status

Known recurring failure (not new today).

Recent history of test_qwen35_eval_amd.py (TestQwen35EvalAMD.setUpClass) in job nightly-8-gpu-qwen35 (branch main, event schedule, completed runs only):

Date	Run	Job	Test File Status	Failed Function	Error
Jun 22	27975987522	nightly-8-gpu-qwen35	❌ Failed	setUpClass	extra_buffer needs CUDA/MUSA/NPU (FLA)
Jun 21	27912890405	nightly-8-gpu-qwen35	❌ Failed	setUpClass	extra_buffer needs CUDA/MUSA/NPU (FLA)
Jun 20	27879266774	nightly-8-gpu-qwen35	❌ Failed	setUpClass	extra_buffer needs CUDA/MUSA/NPU (FLA)

Failing on every completed scheduled run in the queryable window (Jun 20–22) with the identical assertion. Jun 17–19 runs were cancelled (not comparable). No green run found in the queryable window; lookback covered the last 6 scheduled runs.

Failure Cluster

ROCm mamba extra_buffer assertion at server startup (Qwen3.5)

Facts (verified observations, no interpretation)

• Server launch command (verbatim): sglang serve --model-path Qwen/Qwen3.5-397B-A17B --tp 8 --attention-backend aiter --trust-remote-code --model-loader-extra-config '{"enable_multithread_load": true}' --watchdog-timeout 1200 --device cuda --host 127.0.0.1 --port 11000
• Crash stack trace (verbatim, log):
  • server_args.py:2523 __post_init__ → _handle_model_specific_adjustments()
  • server_args.py:4208 _handle_model_specific_adjustments → _handle_mamba_radix_cache(model_arch)
  • server_args.py:4395 _handle_mamba_radix_cache → _validate_mamba_extra_buffer(model_arch)
  • server_args.py:4366 _validate_mamba_extra_buffer → assert (...) "extra_buffer needs CUDA/MUSA/NPU (FLA)."
• Source confirms _validate_mamba_extra_buffer (python/sglang/srt/server_args.py:4361-4368) asserts is_cuda() or is_musa() or is_npu(). On ROCm none of these are true, so the assert fails.
• Source confirms _handle_mamba_radix_cache (server_args.py:4378+) with default mamba_radix_cache_strategy == "auto" selects extra_buffer when overlap schedule or paging is wanted AND the arch supports the extra-buffer cache — then immediately validates it, hitting the ROCm assertion.
• Platform confirmed ROCm: INFO ... Automatically detected platform rocm. (--device cuda is the standard alias used on AMD nightly jobs).
• aiter version unchanged and irrelevant to this code path: [CI-AITER-CHECK] AITER version inside CI image: v0.1.14rc1.dev240+g7d604afe5.
• Reproducibility: identical assertion + same test file in 3/3 completed scheduled runs (Jun 20, 21, 22).

Hypothesised Causes (with confidence, with disconfirming evidence)

• [HIGH] The auto mamba-radix-cache strategy resolves to extra_buffer for Qwen3.5, but extra_buffer (FLA path) is gated to CUDA/MUSA/NPU only, so the server aborts at startup on ROCm before any model load.
  • Supporting: stack trace lands exactly on _validate_mamba_extra_buffer assert is_cuda() or is_musa() or is_npu() (server_args.py:4366); the source path from _handle_mamba_radix_cache to this assert is direct; the assertion message matches verbatim; reproduces on every run with the same arch (Qwen3.5 hybrid mamba) on ROCm.
  • Disconfirming: (none found) — the failure is a deterministic config-validation assert, not a flake; no green ROCm run of this test exists in the window to suggest a recent break.
  • How to verify: launch Qwen/Qwen3.5-397B-A17B on AMD MI325 with the same flags plus --mamba-radix-cache-strategy no_buffer (and --disable-overlap-schedule); if the server then starts, the auto→extra_buffer selection on ROCm is confirmed as the trigger.

In-flight Fix Check

• In-flight fix: ✅ #27141 [ROCm] Support radix-cache with mamba-extra-buffer for Qwen3.5 (open since 2026-06-03, last updated 2026-06-22) — directly targets enabling the extra_buffer radix cache path on ROCm for Qwen3.5. Chase reviewers; do NOT open a duplicate.
• Related (not the same fix): #27220 (TTFT regression for hybrid mamba radix cache), #26127 (tune mamba defaults for Qwen3.5).
• API used: search/issues?q=repo:sgl-project/sglang+is:pr+is:open+mamba+extra_buffer.

Suggested Triage Steps

• Verify #27141 covers this exact assertion (extra_buffer needs CUDA/MUSA/NPU (FLA)) and merges cleanly; if so, ping reviewers — it should unblock this nightly job.
• As an interim signal check, re-run test_qwen35_eval_amd.py on AMD MI325 with --mamba-radix-cache-strategy no_buffer --disable-overlap-schedule to confirm the server starts and the test exercises accuracy (validates the auto-selection hypothesis without the fix).
• If #27141 stalls, consider making the auto strategy in _handle_mamba_radix_cache fall back to no_buffer on non-CUDA/MUSA/NPU platforms rather than selecting extra_buffer and asserting — but defer to the PR author / maintainer.

Status (factual, no priority assignment)

• Persistence: Failing on every completed scheduled run since at least 2026-06-20 (3 consecutive completed runs; Jun 17–19 were cancelled, not comparable).
• Scope: nightly-8-gpu-qwen35 job in nightly-test-amd.yml. The sibling nightly-8-gpu-mi35x-qwen35 job also failed in both prior runs (likely same assertion — not separately confirmed here), so impact spans both Qwen3.5 AMD accuracy shards.
• Blocked work: Blocks Qwen3.5-397B-A17B accuracy signal on AMD MI30x (and likely MI35x).
• In-flight fix: ✅ #27141 (open since 2026-06-03) — ROCm mamba-extra-buffer radix-cache support; chase reviewers.

nightly-4-gpu — failed step(s): Nightly Test (4-GPU)

Commit Info

• sglang: 43b1fb95
• aiter: 7d604afe5 (v0.1.14rc1.dev240+g7d604afe5, source: dev image baked into rocm/sgl-dev:...-mi30x-20260622, preserved — no override)

Failed Tests

Test File	Test Function	Error	Log
test/registered/vlm/test_encoder_dp.py	test_vlm_mmmu_benchmark	AssertionError: Server process exited with code -9 (preceded by Memory access fault by GPU node-2 … Write access to a read-only page)	Log

Failure Summary

The single test test_vlm_mmmu_benchmark in test/registered/vlm/test_encoder_dp.py failed. The SGLang server launches and reaches "Application startup complete" for Qwen/Qwen3-VL-32B-Instruct (TP=4, --mm-enable-dp-encoder), then on the first inference a Memory access fault by GPU node-2 … Write access to a read-only page is raised; the server stops serving (health checks return 503) and is killed with exit code -9. The same crash repeats for Qwen/Qwen2.5-VL-72B-Instruct after retry, so the job ends FAILED (errors=1) after 1528s.

Regression Status

Known recurring failure (failing for 3 consecutive completed scheduled runs; last green 2026-06-15).

Recent history of test/registered/vlm/test_encoder_dp.py (test_vlm_mmmu_benchmark) in job nightly-4-gpu (branch=main, event=schedule, completed runs only):

Date	Run	Job	Test File Status	Failed Function	Error
Jun 22	27975987522	nightly-4-gpu	❌ Failed	test_vlm_mmmu_benchmark	mem access fault → exit -9
Jun 21	27912890405	nightly-4-gpu	❌ Failed	test_vlm_mmmu_benchmark	mem access fault → exit -9
Jun 20	27879266774	nightly-4-gpu	❌ Failed	test_vlm_mmmu_benchmark	mem access fault → exit -9
Jun 19–16	(4 runs)	nightly-4-gpu	⏭️ run cancelled	—	not comparable
Jun 15	27568949589	nightly-4-gpu	✅ Passed	—	1/1 passed, 360s

First observed failure: 2026-06-20. Last known pass: 2026-06-15. Regression window (head SHAs): 33719cfb..ff1fc1fb (20 commits; Jun 16–19 nightlies were cancelled so the window cannot be narrowed further from CI history).

Failure Cluster

GPU memory access fault (write to read-only page) during VLM forward pass on AMD MI30x

Facts (verified observations, no interpretation)

• Server reaches Application startup complete and answers GET /model_info 200 OK at 19:00:19, then Memory access fault by GPU node-2 … Reason: Write access to a read-only page at 19:00:20 (log:4512). Subsequent /health_generate returns 503; process later killed with code -9.
• Both Qwen/Qwen3-VL-32B-Instruct and Qwen/Qwen2.5-VL-72B-Instruct fail identically; launch flags include --mm-enable-dp-encoder --tp=4 (log:6103).
• aiter version is identical across the last passing (Jun 15) and all failing runs (Jun 20/21/22): v0.1.14rc1.dev240+g7d604afe5 (verified via [CI-AITER-CHECK] markers in each job log).
• CI base image differs only by build date tag (...-mi30x-20260615 for the pass vs ...-20260620/-20260622 for the fails); ROCm major tag rocm700 unchanged.
• No commit in the regression window modifies test_encoder_dp.py, mmmu_vlm_kit.py, or the Qwen VL model / multimodal forward paths (verified via git log <range> -- <paths>).

Hypothesised Causes (with confidence, with disconfirming evidence)

• [LOW] A regression landed in the sglang delta 33719cfb..ff1fc1fb (20 commits) that corrupts GPU memory in the VLM forward/DP-encoder path.
  • Supporting: failure starts exactly at the Jun 15→Jun 20 boundary; consistent across 3 runs; aiter held constant, so the change is on the sglang or base-image side.
  • Disconfirming: none of the 20 commits touches Qwen VL model code, multimodal, or the DP-encoder path; the AMD forward-pass commit in the window 47cad39f only modifies deepseek_v4 files (not reachable from Qwen VLM). No single commit is a plausible culprit on inspection.
  • How to verify: git bisect across the window running test_encoder_dp.py on an MI30x 4-GPU runner.
• [LOW] A base-image dependency change (torch / transformers / rocm libs rebuilt in the daily sgl-dev image between -20260615 and -20260620) introduced a faulty VLM kernel path.
  • Supporting: image build-date tag is the only other variable that changed alongside the sglang delta; the fault is a low-level GPU page-protection violation typical of a kernel/library bug, not Python logic.
  • Disconfirming: ROCm major tag (rocm700) and aiter commit are unchanged; cannot confirm sub-package deltas from the log alone.
  • How to verify: re-run today's sglang SHA inside the -20260615 image; if it passes, the regression is in the image, not sglang code.
• [LOW] aiter kernel involvement (the [aiter] … using torch solution:0 lines print immediately before the fault).
  • Supporting: aiter log lines are the last output before the fault.
  • Disconfirming: aiter commit is byte-identical in the Jun 15 passing run and all failing runs — strongly exonerates aiter; log-line proximity is not causation.
  • How to verify: not actionable; aiter is constant across pass/fail.

No hypothesis above LOW confidence; needs git bisect between 33719cfb..ff1fc1fb, and an image A/B (rerun current SHA in the -20260615 image) to separate the sglang delta from base-image drift.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for memory access fault VLM (0 results) and encoder_dp (0 results).

Suggested Triage Steps

• Run the image A/B first (cheapest discriminator): rerun sglang 43b1fb95 inside rocm/sgl-dev:v0.5.13.post1-rocm700-mi30x-20260615. Pass ⇒ regression is in the base image (torch/transformers/rocm), not sglang Python.
• If it still faults, git bisect 33719cfb..ff1fc1fb running python3 test/registered/vlm/test_encoder_dp.py on an MI30x 4-GPU runner.
• Capture a real backtrace by setting CUDA_ENABLE_USER_TRIGGERED_COREDUMP=1 (the log notes coredumps were disabled) to localize the offending kernel.
• If reproducible only with --mm-enable-dp-encoder, retry without that flag to isolate the DP-encoder path.

Status (factual)

• Persistence: Failing for 3 consecutive completed scheduled runs (since 2026-06-20; last green 2026-06-15). Jun 16–19 nightlies were cancelled.
• Scope: nightly-4-gpu job of nightly-test-amd.yml; affects both Qwen3-VL-32B and Qwen2.5-VL-72B VLM MMMU benchmarks on MI30x.
• Blocked work: Blocks AMD 4-GPU VLM (encoder-DP) nightly accuracy signal.
• In-flight fix: ❌ none found in open PRs.

nightly-8-gpu-mi35x-qwen3-235b-mxfp4 — failed step(s): Accuracy Test + Performance Test MI35x (8-GPU Qwen3-235B-MXFP4)

Commit Info

• sglang: 43b1fb95
• aiter: 7d604afe5 (source: dev image preserved — [CI-AITER-CHECK] Dev/patched version detected: v0.1.14rc1.dev240+g7d604afe5 → skipping rebuild, matches Dockerfile gfx950 default)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/test_qwen3_instruct_mxfp4.py	TestQwen3Instruct2507MXFP4.setUpClass	Exception: Server process exited with code 1 → RuntimeError: Rank 0 scheduler died during initialization (exit code: -6)	Log

This is a server-launch (setUpClass) failure, so Ran 0 tests — no per-test assertions executed.

Failure Summary

The amd/Qwen3-235B-A22B-Instruct-2507-mxfp4 server (--tp 4 --ep 2 --attention-backend aiter) failed to start. After weights loaded and the KV cache / memory pool were allocated, all four ranks entered decode CUDA graph capture and immediately aborted with a HIP stream-capture violation, killing rank 0 with SIGABRT (exit -6). The test never ran any assertions.

Regression Status

Never-passed (no green run found in queryable window).

Recent history of test/registered/amd/test_qwen3_instruct_mxfp4.py (setUpClass) in job nightly-8-gpu-mi35x-qwen3-235b-mxfp4 (branch main, event schedule, completed runs only):

Date	Run	Job	Test File Status	Failed Function	Error
Jun 22	27975987522	job	❌ Failed	setUpClass	HIP capturing-stream error (exit -6)
Jun 21	27912890405	job	❌ Failed	setUpClass	same
Jun 20	27879266774	job	❌ Failed	setUpClass	same
Jun 19–16	27841462093 …	—	⏭️ Cancelled (not comparable)	—	—
Jun 15	27568949589	job	❌ Failed	setUpClass	same (HIP capturing-stream)

First observed failure: at least Jun 15 (earliest completed run in window). No passing run found; lookback covered Jun 11–22 scheduled runs (Jun 16–19 were cancelled, not comparable).

Failure Cluster

HIP "operation not permitted in a capturing stream" abort during decode CUDA graph capture (Qwen3-235B-mxfp4, AMD MI35x, aiter backend).

Facts (verified observations, no interpretation)

• Weights loaded successfully but slowly: Load weight end. elapsed=596.51 s … 666.88 s (4 ranks) — exceeds the soft watchdog (watchdog_timeout=300, soft=True), which fired at 20:32:50 but did NOT kill the process (py-spy at that moment showed download_weights_from_hf → _poll_until_acquired filelock during the still-in-progress load).
• KV Cache is allocated … #tokens: 4153365 and Memory pool end. avail mem≈58.9 GB completed on all ranks at 20:39:12.
• At 20:39:13 all ranks logged Capture target decode CUDA graph begin. backend=full … bs=[1…512]; capture started at bs=512.
• During capture, aiter JIT-compiled the paged-attention kernel: [aiter] start build … pa_ragged_…, waiting for baton release at /root/.aiter/build/pa_ragged_…/lock.
• At 20:39:33 (≈20 s into capture) every rank aborted: ProcessGroupNCCL.cpp:2055 … Process group watchdog thread terminated with exception: HIP error: operation not permitted on an event last recorded in a capturing stream (hipErrorCapturedEvent) and operation not permitted when stream is capturing (hipErrorStreamCaptureUnsupported) → terminate called after throwing an instance of 'c10::DistBackendError' → Fatal Python error: Aborted.
• Final: RuntimeError: Rank 0 scheduler died during initialization (exit code: -6) → Server process exited with code 1 → FAILED (errors=1), Ran 0 tests in 736.454s.
• Identical sequence (Capture cuda graph begin → capturing stream HIP error → exit -6) verified in jobs 82592972485, 82503885123, and 81500156314.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [MEDIUM] The ROCm RCCL/ProcessGroupNCCL watchdog thread queries a HIP event last recorded in a stream that is concurrently under HIP graph capture; ROCm forbids this (hipErrorCapturedEvent / hipErrorStreamCaptureUnsupported), aborting all ranks. This is the directly-logged failure point.
  • Supporting: error is raised by ProcessGroupNCCL.cpp:2055 (the watchdog thread, not the capture thread) at the exact moment Capture target decode CUDA graph begin is running; reproduces identically across 4 runs spanning Jun 15–22 on the same MI35x/tp4-ep2/aiter config.
  • Disconfirming: the symptom is the abort site, not necessarily the trigger — the underlying trigger (what issues a collective/event op inside the capturing stream) is not pinned. AMD-only; NVIDIA not exercised by this job.
  • How to verify: re-run with --disable-cuda-graph (or --disable-decode-cuda-graph) in other_args; if the server launches, the capture path is confirmed as the trigger.
• [LOW] aiter lazy JIT of pa_ragged paged-attention kernel happening inside the first graph capture (waiting for baton release at …pa_ragged…/lock) perturbs the capturing stream (file-lock / cross-stream sync) and provokes the illegal event op.
  • Supporting: the aiter pa_ragged build log lines are interleaved with capture and immediately precede the abort.
  • Disconfirming: JIT build normally completes; cannot confirm it touches the capturing stream from the log alone.
  • How to verify: pre-warm/AOT-compile the aiter pa_ragged kernel before capture (or warm a 1-step forward pass outside capture), then re-run.
• [LOW] Slow weight load (>300 s) tripping the soft watchdog is not the root cause — it is a separate symptom; the soft watchdog does not kill and weights finished loading before the crash.
  • Disconfirming: crash is ~7 min after the soft-watchdog log line and at a different code site (graph capture).

In-flight Fix Check

• In-flight fix: ❌ none found directly targeting this test/symptom. Candidate AMD HIP-graph-capture PRs exist but none cites Qwen3-235B-mxfp4 or this capturing-stream abort: #23581 ("[AMD] Default SGLANG_USE_AITER_AR to false to avoid HIP graph capture invalidation") and #27650 ("[AMD] Fix PCG capture fail with aiter fused all-reduce+RMSNorm"). Both are speculative matches only. (Searches run: is:pr is:open capturing stream cuda graph, cuda graph capture nccl rocm, qwen3 235b amd nightly.)

Suggested Triage Steps

• Re-run the job with --disable-decode-cuda-graph added to other_args in test/registered/amd/test_qwen3_instruct_mxfp4.py:30 to confirm decode-graph capture is the trigger (cheapest disconfirmation).
• If confirmed, isolate the watchdog/collective interaction: try disable_custom_all_reduce and/or running with SGLANG_USE_AITER_AR=0 (cf. #23581) and re-run capture on MI35x.
• Pre-warm the aiter pa_ragged kernel (one forward pass before capture) to test the [LOW] JIT-during-capture hypothesis.
• Separately, raise this job's watchdog_timeout (currently 300 s) above the observed ~670 s weight-load time so the slow-load soft-watchdog noise stops masking the real capture crash.
• This is AMD-only (no NVIDIA sister job in this workflow); if capture cannot be made graph-safe quickly, consider gating decode CUDA graph off for this model on MI35x until fixed.

Status (factual, no priority assignment)

• Persistence: Failing every completed scheduled run in the window (Jun 15, 20, 21, 22); never observed passing. Jun 16–19 runs were cancelled (not comparable).
• Scope: 1 job (nightly-8-gpu-mi35x-qwen3-235b-mxfp4) in nightly-test-amd.yml; blocks all CI signal for Qwen3-235B-A22B-mxfp4 on AMD MI35x.
• In-flight fix: ❌ none found directly targeting this symptom; #23581 / #27650 are speculative candidates only.

nightly-8-gpu-mi35x-kimi-k26 — failed step(s): Accuracy Test MI35x (8-GPU Kimi-K2.6)

Commit Info

• sglang: 43b1fb95
• aiter: 7d604afe (v0.1.14rc1.dev240+g7d604afe5, source: dev image preserved — matches Dockerfile default)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi35x/test_kimi_k26_eval_mi35x.py	TestKimiK26EvalMI35x.test_kimi_k26_gsm8k_accuracy	TIMEOUT after 7200s — schedulers stuck in download_weights_from_hf / xet_get (HF weight download never completed)	Log

Failure Summary

The accuracy test for moonshotai/Kimi-K2.6 (tp_size=8, MI35x/gfx950) never finished loading model weights. After Load weight begin at 23:40:45, all 8 TP scheduler ranks remained blocked inside download_weights_from_hf — one rank actively downloading via the HF xet_get backend, the others waiting on the shared filelock — for ~2 hours. The test harness killed it at the hard 7200s per-file timeout (exit code 255); the GSM8K accuracy assertion was never reached.

Regression Status

Known recurring failure / Never-passed in queryable window (last 3 completed scheduled runs all failed for this test file).

Recent history of test_kimi_k26_eval_mi35x.py (test_kimi_k26_gsm8k_accuracy) in job nightly-8-gpu-mi35x-kimi-k26 (branch main, event schedule, completed runs only):

Date	Run	Job	Test File Status	Failed Function	Error
Jun 22	27975987522	mi35x-kimi-k26	❌ Failed	test_kimi_k26_gsm8k_accuracy	TIMEOUT 7200s — xet_get download hang
Jun 21	27912890405	mi35x-kimi-k26	❌ Failed	test_kimi_k26_gsm8k_accuracy	Server process exited with code -9 + IndexError: start out of range (different mode; weights were cached, elapsed 1326s)
Jun 20	27879266774	mi35x-kimi-k26	❌ Failed	test_kimi_k26_gsm8k_accuracy	TIMEOUT 7200s — xet_get download hang (same as today)

No passing run found in the queryable window (Jun 20–22). Today's symptom (download timeout) matches Jun 20; Jun 21 was a distinct server-crash failure that occurred after weights were already cached.

Failure Cluster

Kimi-K2.6 HF weight download timeout (xet backend) during model load on MI35x

Facts (verified observations, no interpretation)

• server_args=ServerArgs(model_path='moonshotai/Kimi-K2.6', ... tp_size=8 ...), watchdog_timeout=1200.0, soft_watchdog_timeout=300 (log line 4238).
• Load weight begin. avail mem=277.x GB on all 8 ranks at 23:40:45 (log ~4523).
• py-spy dumps from 23:45:33 onward show every scheduler rank blocked in download_weights_from_hf (model_loader/weight_utils.py:554) → filelock _poll_until_acquired (waiting ranks), and the downloading rank in ci_download_with_validation_and_retry (model_loader/ci_weight_validation.py:1955) → download_weights_from_hf (weight_utils.py:593) → xet_get (huggingface_hub/file_download.py:566).
• TokenizerManager/Scheduler watchdog timeout (watchdog_timeout=300, soft=True) repeats every ~5 min for the full 2-hour window (no progress).
• Only tiktoken.model/config were cache-resident (/sgl-data/hf-cache/.../snapshots/7eb5002f...); the large safetensors shards were a cache miss requiring fresh HF download.
• ✗ TIMEOUT: ...test_kimi_k26_eval_mi35x.py after 7200 seconds; elapsed=7200, estimated_time=5400.0; exit code 255 (log lines 15811, 15824, 15828).
• AITER_COMMIT_OVERRIDE empty; this is nightly-test-amd.yml (not a scout run).

Hypothesised Causes (with confidence, with disconfirming evidence)

• [HIGH] The immediate failure is an infrastructure / network issue: the HuggingFace download of the Kimi-K2.6 checkpoint (~1T-param MoE, hundreds of safetensors shards) via the xet backend did not complete within the 2-hour per-file budget.
  • Supporting: all 8 ranks are unambiguously stuck in download_weights_from_hf/xet_get/filelock across every py-spy snapshot; timeout hit exactly at elapsed=7200; the model was a cache miss this run; identical signature on Jun 20.
  • Disconfirming: on Jun 21 the same checkpoint loaded in ~1326s (cache hit) — so download is not always the bottleneck; cache state varies run-to-run. The download path itself is not an sglang code regression.
  • How to verify: check /sgl-data/hf-cache persistence/eviction on the MI35x runner; pre-stage/warm the Kimi-K2.6 weights before the nightly; measure raw HF/xet throughput from the runner.
• [LOW] A latent server-side defect (the code -9 SIGKILL + IndexError: start out of range seen Jun 21) may be the real blocker once weights are cached, with the download timeout merely masking it on cache-miss days.
  • Supporting: Jun 21 reached model init and crashed with -9 and an indexer range error.
  • Disconfirming: today's run never got past weight download, so there is no evidence the crash recurred today.
  • How to verify: warm the cache, then re-run to see whether the Jun 21 -9/IndexError reproduces independently of the download.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for xet download (none) and kimi k26 (only unrelated PRs: #27976 ReasoningParser, #28159 cookbook docs, #27209 Dockerfile gfx950).

Suggested Triage Steps

• Inspect the MI35x runner's /sgl-data/hf-cache persistence and eviction policy — the failure correlates with cache-miss runs (Jun 20 & 22 missed, Jun 21 hit). Pre-staging/warming the Kimi-K2.6 weights before the nightly would likely eliminate the download-timeout mode.
• Measure raw HF/xet download throughput from the runner; if xet is the bottleneck, try HF_HUB_DISABLE_XET=1 / hf_transfer, or raise --timeout-per-file for this one very large model.
• Once the cache is warm, re-run to determine whether the Jun 21 server crash (exit code -9 + IndexError: start out of range) reproduces — that is a separate, code-level issue worth its own investigation on kimi_k25.py / DeepSeek-V2 weight-load + indexer paths.

Status (factual, no priority assignment)

• Persistence: This test file has failed in all 3 completed scheduled runs Jun 20–22; today's download-timeout mode matches Jun 20. No green run in the queryable window.
• Scope: Job nightly-8-gpu-mi35x-kimi-k26 in nightly-test-amd.yml. The sister job nightly-8-gpu-kimi-k26 (non-mi35x) also failed in the Jun 20/21 runs (not analyzed here).
• Blocked work: Kimi-K2.6 accuracy signal on MI35x (gfx950) is unavailable.
• In-flight fix: ❌ none found.

nightly-8-gpu-kimi-k26 — failed step(s): Accuracy Test (8-GPU Kimi-K2.6)

Commit Info

• sglang: 43b1fb95
• aiter: 7d604afe (source: Dockerfile default gfx942; CI image carried dev build v0.1.14rc1.dev240+g7d604afe5, rebuild skipped — Dev/patched version detected)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi30x/test_kimi_k26_eval_amd.py	TestKimiK26EvalAMD.setUpClass	IndexError: start out of range (expected to be in range of [-64, 64], but got 320) → server exits -9	Log

Failure Summary

The Kimi-K2.6 server failed to launch on 8×MI30x: every TP rank crashed during weight loading. The accuracy test never ran (Ran 0 tests) because setUpClass could not bring up the server — popen_launch_server reported Server process exited with code -9. The underlying crash is an IndexError while narrowing the MoE w2 (down_proj) weight in fused_moe_triton/layer.py:566.

Regression Status

Known recurring failure — failing on every completed nightly-8-gpu-kimi-k26 run since 2026-06-17. Last green: 2026-06-16.

Recent history of test_kimi_k26_eval_amd.py (TestKimiK26EvalAMD.setUpClass) in job nightly-8-gpu-kimi-k26 (branch main, event schedule, completed runs only):

Date	Run	Test File Status	Failed Function	Error
Jun 23	27975987522	❌ Failed	setUpClass	IndexError: ... but got 320
Jun 21	27912890405	❌ Failed	setUpClass	IndexError: ... but got 320
Jun 20	27879266774	❌ Failed	setUpClass	IndexError: ... but got 192
Jun 19	27841462093	❌ Failed	setUpClass	IndexError: ... but got 448
Jun 18	27780272530	❌ Failed	setUpClass	RuntimeError: Rank 1 scheduler died (exit -11) (different symptom)
Jun 17	27710225107	❌ Failed	setUpClass	IndexError: ... but got 192/256
Jun 16	27640378823	✅ Passed	—	—

(First observed IndexError: 2026-06-17 (f5b041622b). Last green: 2026-06-16 (33f205d8c5). The varying "got N" value — 192/256/320/448, all = 64×k — reflects which expert/rank wins the multithreaded-load race; same root cause. Jun 18 shows a different crash symptom (-11).)

Failure Cluster

Kimi-K2.6 MoE w2 weight-load IndexError (narrow out of range during server warmup)

Facts (verified observations)

• Stack trace (verbatim, log L43852–43864):
  kimi_k25.py:801 load_weights → deepseek_v2.py:2778 → deepseek_weight_loader.py:373 future.result() → fused_moe_triton/layer.py:566 _load_w2 → loaded_weight.narrow(shard_dim, shard_size*tp_rank, shard_size) → IndexError: start out of range (expected to be in range of [-64, 64], but got 320).
• shard_size*tp_rank ∈ {192,256,320,448} = 64×{3,4,5,7}; the loaded_weight shard dim is 64. The loader narrows a w2 tensor that is already per-rank (dim 64) as if it were full-TP-width.
• Server launch command (FACT, log L43895): sglang serve --model-path moonshotai/Kimi-K2.6 --tp 8 --decode-attention-backend triton --prefill-attention-backend aiter.
• Regression window 33f205d8c5..f5b041622b (Jun 16→17) = 44 commits; two of them touch the exact file PR #28905 fixes (compressed_tensors_wNa16_moe.py): f06e2d3d1f (#27690) and 0ae4740bd1 (#27328).
• f06e2d3d1f changed the w2-scale gating from load_full_w2 = self.actorder to load_full_w2 = (self.actorder != "static") (compressed_tensors_wNa16_moe.py:130).

Hypothesised Causes

• [HIGH] Commit f06e2d3d1f (#27690, "Support asymmetric compressed-tensors MoE") flipped load_full_w2 to (self.actorder != "static"). For Kimi-K2.6 actorder is missing/None, so None != "static" ⇒ True ⇒ the scheme now allocates full-TP-width w2 scales for a checkpoint whose w2 is already per-rank, and the MoE loader then narrows the per-rank (dim-64) w2 tensor at a full-width offset → IndexError.
  • Supporting: commit lands in the regression window (Jun 16→17), exactly matching first IndexError appearance Jun 17; it modifies the precise file & line the in-flight fix reverts; in-flight fix #28905 explicitly states "Treat missing/false/static actorder as non-actorder" and "Avoid … full TP-width w2_weight_scale for checkpoints whose w2 scales are already per-rank, such as Kimi-K2.6."
  • Disconfirming: the crash line is in fused_moe_triton/layer.py:566 (w2 weight narrow), whereas the changed/ fixed code is in compressed_tensors_wNa16_moe.py (w2 scale param allocation) — the param-shape → loader-narrow linkage is plausible but not traced line-by-line here. Also 0ae4740bd1 (#27328) touches the same file in the same window and is not fully excluded.
  • How to verify: on a branch, cherry-pick #28905 (or revert the load_full_w2 line of f06e2d3d1f) and re-run test_kimi_k26_eval_amd.py on 8×MI30x.

In-flight Fix Check

• In-flight fix: ✅ #28905 "Fix Kimi K2.6 compressed-tensors MoE w2 scale loading" (open since 2026-06-22) — modifies compressed_tensors_wNa16_moe.py: treats missing/false/static actorder as non-actorder and avoids full-TP-width w2 scale allocation for per-rank checkpoints (Kimi-K2.6). Directly targets this symptom. Chase reviewers; do NOT open a duplicate. (Search: is:pr is:open kimi k2.)

Suggested Triage Steps

• Verify #28905 resolves this: apply it and re-run nightly-8-gpu-kimi-k26 (test_kimi_k26_eval_amd.py) on 8×MI30x — its own CI ran MI35x only (run 27938672153), so MI30x is still unverified.
• If #28905 does not fully fix it, bisect the two window commits f06e2d3d1f and 0ae4740bd1 over 33f205d8c5..f5b041622b by reverting each on a branch.
• Confirm the param-shape linkage by logging layer.w2_weight_scale.shape and loaded_weight.shape in _load_w2 for Kimi-K2.6 to confirm the per-rank vs full-width mismatch.

Status

• Persistence: IndexError failing every completed run since 2026-06-17 (7 days); last green 2026-06-16.
• Scope: nightly-8-gpu-kimi-k26 (MI30x). The sister nightly-8-gpu-mi35x-kimi-k26 job also shows failure in the same runs — likely the same cluster, worth confirming separately.
• Blocked work: Kimi-K2.6 accuracy signal on AMD nightly (test cannot even start the server).
• In-flight fix: ✅ #28905 (open since 2026-06-22, awaiting merge/review).

nightly-accuracy-2-gpu-vlm — failed step(s): Nightly Accuracy Test (2-GPU VLM MMMU)

Commit Info

• sglang: ed26a109
• aiter: 7d604afe (v0.1.14rc1.dev240+g7d604afe5, source: dev image preserved — matches Dockerfile default, no override)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi30x/test_vlms_mmmu_eval_amd.py	test_mmmu_vlm_models (model=deepseek-ai/deepseek-vl2-small)	TimeoutError: Server failed to start within the timeout period (HF weight download hung)	Log
test/registered/amd/accuracy/mi30x/test_vlms_mmmu_eval_amd.py	test_mmmu_vlm_models (model=Qwen/Qwen2-VL-7B-Instruct)	MMMU score 0.280 < threshold 0.30	Log
test/registered/amd/accuracy/mi30x/test_vlms_mmmu_eval_amd.py	test_mmmu_vlm_models (model=Qwen/Qwen2.5-VL-7B-Instruct)	MMMU score 0.280 < threshold 0.33	Log
test/registered/amd/accuracy/mi30x/test_vlms_mmmu_eval_amd.py	test_mmmu_vlm_models (model=Qwen/Qwen3-VL-30B-A3B-Instruct)	MMMU score 0.260 < threshold 0.29	Log
test/registered/amd/accuracy/mi30x/test_vlms_mmmu_eval_amd.py	test_mmmu_vlm_models (model=openbmb/MiniCPM-o-2_6)	MMMU score 0.290 < threshold 0.32	Log

Passed in this run: OpenGVLab/InternVL2_5-2B (0.290 / 0.29), openbmb/MiniCPM-v-2_6 (0.310 / 0.25).

Failure Summary

The job runs a single parameterized test (test_mmmu_vlm_models) over a list of VLMs on the MMMU benchmark. Two distinct problems occurred: (1) four models produced MMMU scores below their configured thresholds (Qwen2-VL, Qwen2.5-VL, Qwen3-VL-30B-A3B, MiniCPM-o-2_6), and (2) the deepseek-ai/deepseek-vl2-small server never started — its HuggingFace weight download hung (stuck in xet_get), tripping ~10 consecutive 300s soft watchdog timeouts before popen_launch_server raised TimeoutError. The deepseek TimeoutError is what unittest reports as errors=1; the four accuracy shortfalls are recorded as ❌ FAILED per-model.

Regression Status

Accuracy threshold failures — Known recurring failure. deepseek-vl2-small server-start timeout — New today (infrastructure/download hang).

Recent history of test_vlms_mmmu_eval_amd.py (test_mmmu_vlm_models) in job nightly-accuracy-2-gpu-vlm (same branch main, event=schedule, completed runs only):

Date	Run	Job	Test File Status	Failing models / mode	Notes
Jun 23	28046906309	nightly-accuracy-2-gpu-vlm	❌ Failed	Qwen2-VL 0.280, Qwen2.5-VL 0.280, Qwen3-VL-30B 0.260, MiniCPM-o 0.290; deepseek-vl2 server timeout	deepseek download hang NEW today
Jun 22	27975987522	nightly-accuracy-2-gpu-vlm	❌ Failed	Qwen2-VL 0.280, Qwen2.5-VL 0.280, Qwen3-VL-30B 0.260, MiniCPM-o 0.280	deepseek-vl2 started fine; test ran to completion (2976s)
Jun 21	27912890405	nightly-accuracy-2-gpu-vlm	❌ Failed	(same job failing)	—
Jun 20	27879266774	nightly-accuracy-2-gpu-vlm	❌ Failed	(same job failing)	—

Lookback window: last 10 scheduled runs. Runs Jun 16–19 were cancelled (excluded). No green run for this job found in the queryable window ≥ Jun 14.

Failure Cluster

Two clusters: "VLM MMMU accuracy below threshold (AMD MI30x)" (4 models) and "VLM server startup timeout — HF weight download hang (deepseek-vl2-small)".

Facts (verified observations)

• The MMMU scores are deterministic across days: Qwen2-VL 0.280 (Jun 22 & 23), Qwen2.5-VL 0.280 (both), Qwen3-VL-30B 0.260 (both). MiniCPM-o 0.280 (Jun 22) vs 0.290 (Jun 23). These are not random flakes.
• Thresholds are defined in test/registered/amd/accuracy/mi30x/test_vlms_mmmu_eval_amd.py:45 as "conservative thresholds on 100 MMMU samples". MMMU is multiple-choice; 100 samples is a small, noisy eval set, and the failing scores (~0.26–0.29) are close to the thresholds.
• deepseek-vl2-small hang: py-spy dump at 20:16:45 shows the scheduler stuck in load_weights (deepseek_vl2.py:246) → download_weights_from_hf (weight_utils.py:593) → snapshot_download → xet_get (huggingface_hub/file_download.py:566) — i.e. a network/storage stall in the HF Xet downloader, not an sglang code crash.
• On Jun 22 the same deepseek-vl2-small model launched successfully (local HF snapshot present, download completed); today the snapshot had no files matching ['*.safetensors'] and the live download stalled. This points to an environment/cache/network condition, not a code change.
• aiter unchanged (dev image preserved; no override). NVIDIA not affected (AMD-only test file).

Hypothesised Causes (with confidence)

• [FACT] deepseek-vl2-small server-start TimeoutError is caused by the HF weight download stalling in xet_get (py-spy dump is direct evidence).
  • Supporting: 10× consecutive 300s watchdog timeouts (19:26→20:16 UTC) while the only active thread is ThreadPoolExecutor in xet_get; Jun 22 ran the same model fine when the snapshot was cached.
  • Disconfirming: none — the stall is in the downloader, not sglang code.
  • How to verify: re-run the job (transient network), or pre-warm /sgl-data/hf-cache for deepseek-ai/deepseek-vl2-small, or set HF_HUB_DISABLE_XET=1 to bypass the Xet path.
• [LOW] The four accuracy shortfalls reflect thresholds set above what these models achieve on the 100-sample AMD MMMU harness (i.e. mis-tuned thresholds), rather than a serving regression.
  • Supporting: scores are stable/deterministic across ≥2 days and sit just under the thresholds; comment in the test calls them "conservative thresholds on 100 MMMU samples".
  • Disconfirming: scores ~0.26–0.29 are only marginally above 4-choice random (~0.25), which could equally indicate a genuine multimodal-serving correctness problem on AMD producing near-random answers — cannot distinguish without a known-good baseline score for these models on MI30x.
  • How to verify: compare against the last run where this job was green (none in window — needs older history or a manual baseline on MI30x); run one model with full MMMU set to reduce noise; A/B the same model on NVIDIA.
• [LOW] A pre-existing sglang VLM serving regression depresses scores.
  • Supporting: no green run found in the queryable window (Jun 14+); failures predate today.
  • Disconfirming: scores are stable, not degrading; no crash/stack-trace in the eval path for these four models.
  • How to verify: bisect against the last known green run of this job (older than the 10-run window).

In-flight Fix Check

• Accuracy thresholds: In-flight fix: ❌ none found — searched open PRs for mmmu vlm (no threshold/accuracy fix; results are unrelated VLM features).
• deepseek-vl2 download hang: In-flight fix: ❌ none found — searched open PRs for deepseek-vl2 (results are model features/cleanups, not a download-hang fix).

Suggested Triage Steps

• For deepseek-vl2-small: re-run the job once to confirm the download hang is transient; if it recurs, pre-warm the MI30x HF cache for deepseek-ai/deepseek-vl2-small or try HF_HUB_DISABLE_XET=1 on the runner. Consider a shorter per-model download/start budget so one hung download doesn't consume ~50 min and abort the whole suite.
• For the accuracy cluster: establish whether ~0.28 is the expected MI30x score for these models (run one model — e.g. Qwen2.5-VL-7B — on the full MMMU set and compare NVIDIA vs AMD). If expected, adjust thresholds in test_vlms_mmmu_eval_amd.py:45; if not, bisect VLM serving against the last green run of this job (older than the current 10-run window).
• Decouple per-model results so the suite reports all model verdicts even when one model's server fails to start (today the deepseek TimeoutError aborts collection of any models queued after it).

Status

• Persistence: nightly-accuracy-2-gpu-vlm has failed every completed scheduled run Jun 20–23; no green run in the last 10 scheduled runs (Jun 16–19 were cancelled). The 4-model accuracy shortfall is the persistent component; the deepseek-vl2 download timeout is new today.
• Scope: one job (nightly-accuracy-2-gpu-vlm) in nightly-test-amd.yml, AMD MI30x only. NVIDIA unaffected (AMD-specific test file).
• In-flight fix: ❌ none found for either cluster.

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-06-23 20:59 UTC)

[amd-bot]

amd-aiter-scout.yml — 26 failure(s)

Run started (UTC): 2026-06-22 21:03 UTC
Last scanned (UTC): 2026-06-23 12:59 UTC
Commits: sglang bbc853d

Summary

Counts: 29 failures · 15 clusters · Origin: 4 aiter-caused · 16 pre-existing (sglang) · 9 unclear

⚠️ Parent scout run 27983814629 was still queued/in-flight at snapshot (≈23 success / 9 failure / 3 in-progress / 61 queued, of 180 jobs). These 26 analyzed jobs are completed-failed; remaining jobs may add failures. [IN-FLIGHT, partial data]

#	Cluster	Job	Test File	Test Function	Origin	Status	Hypothesis (confidence)
1	Server-launch timeout / HF weight-download hang	nightly-8-gpu-mi35x-deepseek-r1-hicache-rocm720	test/registered/amd/test_deepseek_r1_hicache_mi35x.py	whole-file	pre-existing (sglang)	recurring	[HIGH] --timeout-per-file 3600 < est_time 5400
2	Server-launch timeout / HF weight-download hang	nightly-8-gpu-mi35x-deepseek-r1-hicache	test/registered/amd/test_deepseek_r1_hicache_mi35x.py	setUpClass	pre-existing (sglang)	recurring	[HIGH] 3600s timeout < cold R1-0528 671B load
3	Server-launch timeout / HF weight-download hang	nightly-8-gpu-mi35x-kimi-k26-rocm720	test/registered/amd/accuracy/mi35x/test_kimi_k26_eval_mi35x.py	whole-file	pre-existing (sglang)	recurring	[HIGH] HF Kimi-K2.6 download stall, 7200s budget
4	Server-launch timeout / HF weight-download hang	nightly-8-gpu-mi35x-kimi-k26	test/registered/amd/accuracy/mi35x/test_kimi_k26_eval_mi35x.py	test_kimi_k26_gsm8k_accuracy	pre-existing (sglang)	recurring	[HIGH] HF weight fetch too slow → launch timeout
5	Server-launch timeout / HF weight-download hang	nightly-2-gpu-mi35x-glm51-mxfp4	test/registered/amd/accuracy/mi35x/test_glm51_mxfp4_tp2_gsm8k_mi35x.py	setUpClass	unclear (infra)	new (3 prior ✅)	[MEDIUM] snapshot lacked safetensors → live HF download hung
6	Server-launch timeout / HF weight-download hang	nightly-8-gpu-mi35x-glm5-mxfp4-rocm720	test/registered/amd/accuracy/mi35x/test_glm5_mxfp4_eval_mi35x.py	test_glm5_mxfp4_accuracy	unclear (infra)	sister ✅	[HIGH] xet download of amd/GLM-5-MXFP4 stalled
7	Server-launch timeout / HF weight-download hang	nightly-8-gpu-deepseek-v3-kv-fp8-rocm720	test/registered/amd/test_deepseek_v3_mtp_kv_fp8.py	N/A	unclear (infra)	sister ✅	[HIGH] DeepSeek-V3-0324 670B download > 3600s
8	Gate/job cancellation cascade (orchestration)	extra-a-test-1-gpu-small-amd-rocm720	N/A	N/A	unclear	cancelled, 0 steps	[LOW] concurrency/fast-fail teardown before runner pickup
9	Gate/job cancellation cascade (orchestration)	extra-a-test-1-gpu-large-amd-rocm720	N/A	N/A	unclear	cancelled, 0 steps	[LOW] cascade from cancelled pr-gate need
10	Gate/job cancellation cascade (orchestration)	call-pr-test-amd-extra-rocm720 / call-gate / pr-gate	N/A	N/A	unclear	cancelled, 0 steps	[LOW] concurrency-group supersession / manual cancel
11	MiniMax-M2.7 GSM8K accuracy < 0.93	nightly-8-gpu-minimax-m27	test/registered/amd/accuracy/mi30x/test_minimax_m27_eval_amd.py	test_minimax_m27_accuracy	unclear	0.925, sister 0.930 (on line)	[MEDIUM] borderline threshold vs sampling variance
12	MiniMax-M2.7 GSM8K accuracy < 0.93	nightly-8-gpu-minimax-m27-rocm720	test/registered/amd/accuracy/mi30x/test_minimax_m27_eval_amd.py	test_minimax_m27_accuracy	pre-existing (sglang)	declining 0.945→0.915	[MEDIUM] borderline gate; sister 0.920 also FAIL
13	VLM MMMU accuracy below threshold (MI30x)	nightly-accuracy-2-gpu-vlm	test/registered/amd/accuracy/mi30x/test_vlms_mmmu_eval_amd.py	test_mmmu_vlm_models	pre-existing (sglang)	persistent ≥Jun 14	[LOW] 5 VLMs near-random on AMD / threshold too high
14	VLM MMMU accuracy below threshold (MI30x)	nightly-accuracy-2-gpu-vlm-rocm720	test/registered/amd/accuracy/mi30x/test_vlms_mmmu_eval_amd.py	test_mmmu_vlm_models	pre-existing (sglang)	persistent ≥Jun 13	[MEDIUM] per-model thresholds > AMD-achievable
15	Mistral/Mixtral GSM8K accuracy below threshold	nightly-accuracy-2-gpu-rocm720	test/registered/amd/accuracy/mi30x/test_gsm8k_eval_amd.py	test_gsm8k_all_models	pre-existing (sglang)	recurring	[HIGH] pre-existing Mistral/Mixtral chat-eval bug
16	Mistral/Mixtral GSM8K accuracy below threshold	nightly-accuracy-2-gpu	test/registered/amd/accuracy/mi30x/test_gsm8k_eval_amd.py	test_gsm8k_all_models	pre-existing (sglang)	recurring	[HIGH] same Mistral/Mixtral scores in sister
17	HIP OOM during decode CUDA-graph capture (FP8-70B TP2)	nightly-accuracy-2-gpu-rocm720	test/registered/amd/accuracy/mi30x/test_gsm8k_eval_amd.py	test_gsm8k_all_models (70B-FP8)	unclear	no FP8-70B baseline	[LOW] mem-fraction headroom on FP8-70B TP2
18	HIP OOM during decode CUDA-graph capture (FP8-70B TP2)	nightly-accuracy-2-gpu	test/registered/amd/accuracy/mi30x/test_gsm8k_eval_amd.py	test_gsm8k_all_models (70B-FP8)	aiter-caused	baseline ✅ 0.910	[LOW] aiter override workspace footprint / flaky OOM
19	HIP error during decode CUDA-graph capture (Qwen3-235B custom all-reduce)	nightly-8-gpu-mi35x-qwen3-235b-mxfp4	test/registered/amd/test_qwen3_instruct_mxfp4.py	setUpClass	pre-existing (sglang)	recurring	[MEDIUM] aiter custom-AR IPC-meta gather during capture
20	HIP error during decode CUDA-graph capture (Qwen3-235B custom all-reduce)	nightly-8-gpu-mi35x-qwen3-235b-mxfp4-rocm720	test/registered/amd/test_qwen3_instruct_mxfp4.py	setUpClass	pre-existing (sglang)	recurring	[MEDIUM] AiterCustomAllreduce records event in capture → hipErrorCapturedEvent
21	GLM-5.1-FP8 DSA accuracy collapse under candidate aiter	nightly-8-gpu-mi35x-glm51-rocm720	test/registered/amd/accuracy/mi35x/test_glm51_eval_mi35x.py	test_glm51_accuracy	aiter-caused	sister ✅ (default aiter)	[MEDIUM] aiter 7d604afe..a40c487b regresses GLM-5.1 DSA on gfx950
22	DeepSeek-V3.2 DSA+EAGLE-MTP collective deadlock	nightly-accuracy-8-gpu-mi35x-deepseek-v32-mtp-rocm720	test/registered/amd/accuracy/mi35x/test_deepseek_v32_mtp_eval_mi35x.py	test_a_gsm8k	aiter-caused	sister ✅ 0.965	[MEDIUM] aiter override destabilizes DSA+MTP decode
23	gpt-oss-20b GSM8K accuracy below threshold (AITER MXFP4 MoE)	nightly-accuracy-8-gpu-mi35x-rocm720	test/registered/amd/accuracy/mi35x/test_gpt_oss_eval_mi35x.py	test_gpt_oss_accuracy	aiter-caused	0.465 vs thr 0.47	[LOW] aiter fmoe tuned-config-key change shifts MXFP4 numerics
24	DeepSeek-V3.2 MI35x GSM8K accuracy collapse	nightly-accuracy-8-gpu-mi35x-deepseek-v32-rocm720	test/registered/amd/accuracy/mi35x/test_deepseek_v32_eval_mi35x.py	test_deepseek_v32_accuracy	pre-existing (sglang)	bimodal ~0.01/~0.87	[LOW] non-deterministic DSA/NSA collapse (reproduces Jun 13-15 default aiter)
25	HIP "invalid device pointer" CUDA-IPC crash (MM DP-encoder)	nightly-4-gpu-rocm720	test/registered/vlm/test_encoder_dp.py	test_vlm_mmmu_benchmark	pre-existing (sglang)	flaky (retry-dependent)	[MEDIUM] --mm-enable-dp-encoder CUDA-IPC transport unreliable on ROCm
26	HF gated-repo 403 (ideogram-4-fp8)	multimodal-gen-test-2-gpu-amd	test/registered/.../test_server_2_gpu.py	test_diffusion_generation[ideogram4_fp8_tp2_t2i]	pre-existing (sglang)	recurring ≥Jun 21	[FACT] GatedRepoError 403, CI token not authorized
27	JIT activation.cuh ROCm compile failure + pgq timeout	jit-kernel-unit-test-amd	test/registered/jit/test_activation.py	test_activation_correctness	pre-existing (sglang)	fixed on main	[HIGH] stale base predates fix #27947
28	JIT activation.cuh ROCm compile failure + pgq timeout	jit-kernel-unit-test-amd	test/registered/jit/test_per_token_group_quant_8bit.py	test_per_token_group_quant_with_column_major	pre-existing (sglang)	fixed on main	[HIGH] not yet CUDA-only-gated in stale base (#27947)
29	GPU VRAM not cleared (zombie KFD context)	nightly-8-gpu-mi35x-grok2-rocm720	N/A (VRAM pre-check)	N/A	unclear (infra)	host-side	[HIGH] zombie KFD context holding ~8-9% VRAM on runner

Cluster: Server-launch timeout / HF weight-download hang

• Affected: rows 1-7 (R1-hicache ×2, Kimi-K2.6 ×2, GLM-5.1/5-MXFP4 ×2, DeepSeek-V3-KV-FP8). Largest cluster (7 jobs).
• Shared facts: server never reaches ready; all hang in huggingface_hub.snapshot_download/weight load, then per-file/test timeout (3600–7200s). Big-checkpoint models (671B R1, Kimi-K2.6, V3-0324).
• Top hypothesis: [HIGH] per-file timeout budget < cold-cache download+load time (R1: 3600s < 5400s est). Infra/network, not aiter or model code.
• In-flight fix: ❌ none found — these are timeout-budget/cache issues, needs config tuning or pre-warmed cache.
• Suggested triage: pre-warm HF cache on MI35x runners; raise --timeout-per-file for ≥600B models; verify download throughput on affected runners.

Cluster: Gate/job cancellation cascade (orchestration)

• Affected: rows 8-10 — only the call-pr-test-amd-extra-rocm720 leg; sibling rocm720 and non-rocm720 gates succeeded.
• Shared facts: pr-gate 82820330340 cancelled at scheduling (0 steps, completed_at before started_at); downstream extra-chain jobs cancelled in cascade; BlobNotFound logs.
• Top hypothesis: [LOW] concurrency-group supersession or fast-fail teardown; no code defect. Disconfirming: sibling gates in the same run succeeded.
• Suggested triage: re-run the call-pr-test-amd-extra-rocm720 leg once the parent run settles; check Actions timeline for concurrency cancellation near 21:03 UTC.

Cluster: aiter-caused failures (override a0320631 vs default 7d604afe)

• Affected: row 21 GLM-5.1-FP8 DSA collapse, row 22 DeepSeek-V3.2 DSA+EAGLE-MTP deadlock, row 23 gpt-oss-20b accuracy, row 18 FP8-70B OOM (LOW).
• Shared facts: each passes in the sister nightly with default aiter and fails only under the override. GLM-5.1/V3.2 are gfx950 DSA paths; gpt-oss is MXFP4 fused-MoE.
• Top hypotheses: [MEDIUM] GLM-5.1/V3.2 — aiter delta regresses DSA forward/decode on gfx950; [LOW] gpt-oss — aiter fmoe tuned-config-key commits (#3703, #3705), but 0.005 margin is within GSM8K noise.
• In-flight fix: ❌ none found.
• Suggested triage: re-run rows 21-23 ×3 with override a0320631 to separate noise from regression; if consistent, bisect aiter 7d604afe..a40c487b/..a0320631 over DSA + fmoe commits on MI35x.

Pre-existing sglang failures (not caused by aiter)

Rows 1-4, 12-16, 19-20, 24-28 reproduce in the sister nightly runs with the default aiter (AITER_COMMIT_OVERRIDE= empty) — not attributable to the override:

• VLM MMMU (rows 13-14): persistent since ≥Jun 13-14; 5 VLMs near-random on AMD MI30x — real accuracy issue or thresholds set too high.
• Mistral/Mixtral gsm8k (rows 15-16): same low scores (≈0.36/0.29) in default-aiter baseline — pre-existing chat-eval/accuracy bug.
• Qwen3-235B custom all-reduce graph capture (rows 19-20): hipErrorCapturedEvent during decode capture, default SGLANG_USE_AITER_AR=true.
• JIT compile (rows 27-28): scout base a7fd09207 predates merged fix #27947 (72d962be) — already green on current main.
• ideogram-4-fp8 403 (row 26): gated HF repo, CI token not authorized — access/infra.
• DeepSeek-V3.2 collapse (row 24): bimodal ~0.01/~0.87, reproduced Jun 13-15 without override.

Suggested triage order

Rank	Triage step	Targets	Why this first
1	Confirm jit job green on current main (fix #27947 merged); ignore in scout — it ran stale base	rows 27-28	Already fixed; zero further work
2	Re-run aiter-suspect tests ×3 under override a0320631; bisect aiter delta if consistent	rows 21-23 (+18)	Only genuinely aiter-attributable failures; isolates override regression
3	Raise per-file timeout / pre-warm HF cache for ≥600B models on MI35x runners	rows 1-7 (largest cluster)	Biggest blast radius; pure infra/budget fix
4	Re-trigger call-pr-test-amd-extra-rocm720 leg after parent run settles	rows 8-10	Orchestration cancellation, expected to clear on clean re-run
5	File AMD-side tickets for pre-existing accuracy gaps (VLM MMMU, Mistral/Mixtral, V3.2 collapse) + investigate Qwen3-235B custom-AR capture	rows 13-16, 19-20, 24	Pre-existing, independent of aiter; needs sglang/threshold review

---

Job	Failed Steps	Started
call-nightly-amd / nightly-2-gpu-mi35x-glm51-mxfp4	Accuracy Test MI35x (2-GPU GLM-5.1-MXFP4 GSM8K)	2026-06-22T21:03
call-nightly-amd / nightly-accuracy-2-gpu-vlm	Nightly Accuracy Test (2-GPU VLM MMMU)	2026-06-22T21:56
call-nightly-amd-rocm720 / nightly-4-gpu-rocm720	Nightly Test ROCm 7.2 (4-GPU)	2026-06-22T21:03
call-nightly-amd / nightly-8-gpu-minimax-m27	Accuracy Test (8-GPU MiniMax-M2.7)	2026-06-22T23:49
call-nightly-amd-rocm720 / nightly-accuracy-2-gpu-rocm720	Nightly Test ROCm 7.2 (2-GPU)	2026-06-22T21:38
call-nightly-amd-rocm720 / nightly-accuracy-2-gpu-vlm-rocm720	Nightly Accuracy Test ROCm 7.2 (2-GPU VLM MMMU)	2026-06-22T22:05
[`call-pr-test-amd-rocm720 / call-pr-test-amd-extra-rocm720 / extra-a-test-1-gpu-small-amd${{ ((inputs.rocm_version && format('-{0}', inputs.rocm_version))		'') }} (linux-${{ (inputs.runner_arch
[`call-pr-test-amd-rocm720 / call-pr-test-amd-extra-rocm720 / extra-a-test-1-gpu-large-amd${{ ((inputs.rocm_version && format('-{0}', inputs.rocm_version))		'') }} (linux-${{ (inputs.runner_arch
call-pr-test-amd-rocm720 / call-pr-test-amd-extra-rocm720 / call-gate / pr-gate	N/A	2026-06-22T21:03
call-pr-test-amd / multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 1)	Run diffusion server tests (2-GPU)	2026-06-22T21:51
call-pr-test-amd / jit-kernel-unit-test-amd (linux-mi325-1gpu-sglang)	Run JIT kernel unit tests	2026-06-22T23:27
call-nightly-amd / nightly-8-gpu-mi35x-qwen3-235b-mxfp4	Accuracy Test + Performance Test MI35x (8-GPU Qwen3-235B-MXFP4)	2026-06-23T00:33
call-nightly-amd / nightly-accuracy-2-gpu	Nightly Test (2-GPU)	2026-06-22T21:58
call-nightly-amd-rocm720 / nightly-accuracy-8-gpu-mi35x-deepseek-v32-rocm720	Accuracy Test MI35x ROCm 7.2 (8-GPU DeepSeek-V3.2)	2026-06-23T01:13
call-nightly-amd-rocm720 / nightly-8-gpu-minimax-m27-rocm720	Accuracy Test ROCm 7.2 (8-GPU MiniMax-M2.7)	2026-06-23T02:10
call-nightly-amd / nightly-8-gpu-mi35x-kimi-k26	Accuracy Test MI35x (8-GPU Kimi-K2.6)	2026-06-23T01:13
call-nightly-amd-rocm720 / nightly-8-gpu-mi35x-glm5-mxfp4-rocm720	Accuracy Test MI35x ROCm 7.2 (8-GPU GLM-5-MXFP4)	2026-06-23T02:52
call-nightly-amd-rocm720 / nightly-8-gpu-mi35x-glm51-rocm720	Accuracy Test MI35x ROCm 7.2 (8-GPU GLM-5.1 DSA)	2026-06-23T03:41
call-nightly-amd-rocm720 / nightly-8-gpu-mi35x-qwen3-235b-mxfp4-rocm720	Accuracy Test + Performance Test MI35x ROCm 7.2 (8-GPU Qwen3-235B-MXFP4)	2026-06-23T06:11
call-nightly-amd-rocm720 / nightly-8-gpu-mi35x-deepseek-r1-hicache-rocm720	Accuracy Test MI35x ROCm 7.2 (8-GPU DeepSeek-R1 HiCache)	2026-06-23T07:22
call-nightly-amd / nightly-8-gpu-mi35x-deepseek-r1-hicache	Accuracy Test MI35x (8-GPU DeepSeek-R1 HiCache)	2026-06-23T08:11
call-nightly-amd-rocm720 / nightly-8-gpu-mi35x-grok2-rocm720	Ensure VRAM is clear	2026-06-23T09:21
call-nightly-amd-rocm720 / nightly-8-gpu-deepseek-v3-kv-fp8-rocm720	DeepSeek-V3 KV FP8 Test ROCm 7.2 (8-GPU Basic + MTP)	2026-06-23T07:49
call-nightly-amd-rocm720 / nightly-accuracy-8-gpu-mi35x-deepseek-v32-mtp-rocm720	Accuracy Test MI35x ROCm 7.2 (8-GPU DeepSeek-V3.2 TP+MTP)	2026-06-23T09:30
call-nightly-amd-rocm720 / nightly-accuracy-8-gpu-mi35x-rocm720	Accuracy Test MI35x ROCm 7.2 (8-GPU GPT-OSS)	2026-06-23T10:31
call-nightly-amd-rocm720 / nightly-8-gpu-mi35x-kimi-k26-rocm720	Accuracy Test MI35x ROCm 7.2 (8-GPU Kimi-K2.6)	2026-06-23T10:39

Per-Job Analysis

call-nightly-amd / nightly-2-gpu-mi35x-glm51-mxfp4 — failed step(s): Accuracy Test MI35x (2-GPU GLM-5.1-MXFP4 GSM8K)

Commit Info

• sglang: bbc853df
• aiter: a0320631 (source: AITER_COMMIT_OVERRIDE, forced rebuild; Dockerfile default was 7d604afe)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi35x/test_glm51_mxfp4_tp2_gsm8k_mi35x.py	TestGLM51MXFP4TP2GSM8KMI35x.setUpClass	TimeoutError: Server failed to start within the timeout period	Log

Failure Summary

The server for amd/GLM-5.1-MXFP4 (sglang serve --tp 2) never finished starting, so setUpClass timed out after ~5420s and 0 tests ran. The scheduler hung inside HuggingFace weight loading: the local HF snapshot had no *.safetensors files, triggering a download that never completed. This is a model-weight download/cache problem on the MI35x runner, not a test-logic failure.

Regression Status

Infrastructure issue (HF weight download hang). The same job passes consistently in the sister nightly workflow.

Recent history of test_glm51_mxfp4_tp2_gsm8k_mi35x.py (setUpClass) in job nightly-2-gpu-mi35x-glm51-mxfp4, sister workflow nightly-test-amd.yml (branch main, event=schedule, completed runs only):

Date	Run	Job	Test File Status	Failed Function	Error
Jun 21	27912890405	nightly-2-gpu-mi35x-glm51-mxfp4	✅ Passed	—	—
Jun 20	27879266774	nightly-2-gpu-mi35x-glm51-mxfp4	✅ Passed	—	—
Jun 15	27568949589	nightly-2-gpu-mi35x-glm51-mxfp4	✅ Passed	—	—

(The current scout run 27983814629 still shows status=queued at the API level; this job has completed-failed. Other jobs in the scout may be in-flight.)

Failure Origin

unclear (infrastructure — not aiter-caused).

Test File	Test Function	Origin	Baseline Run	Baseline Status
test_glm51_mxfp4_tp2_gsm8k_mi35x.py	setUpClass	unclear (infra: HF download hang)	27912890405	✅ Passed (sister job, default aiter)

The sister baseline passed, which by the rote A/B table maps to "aiter-caused" — but that is the symptom-clustering trap. The actual hang is in HF weight download (weight_utils.py:556 filelock / download), a code path the aiter rebuild does not touch. The scout↔sister difference is the model-weight cache state on the runner (sister had weights cached and loaded in seconds; scout hit an empty snapshot and attempted a download that hung), not the aiter override. Hence unclear/infra, with zero aiter commits implicated.

Failure Cluster

HF model-weight download hang during server warmup (MI35x GLM-5.1-MXFP4).

Facts (verified observations)

• Server cmd: sglang serve --model-path amd/GLM-5.1-MXFP4 --tp 2 ... --device cuda (log line 7500).
• Local HF snapshot at .../models--amd--GLM-5.1-MXFP4/snapshots/74fb7daa... has no files matching ['*.safetensors', '*.bin', '*.pt']; will attempt download. (log line 4468).
• TP0 acquired the CI download lock, then entered download_weights_from_hf; TP1 py-spy dump shows it blocked at _poll_until_acquired (filelock/_api.py:552) → download_weights_from_hf (model_loader/weight_utils.py:556) (log lines 4474–4496). weight_utils.py:556 is with get_lock(model_name_or_path, cache_dir): (verified in source).
• Repeated Scheduler/DetokenizerManager watchdog timeout (timeout=300, soft=True) every 5 min from 21:17 to 22:37 (log lines 4522, 7480) — never progressed past weight load.
• Ran 0 tests in 5419.987s → TimeoutError: Server failed to start (log lines 7490–7497).
• aiter rebuilt from override a0320631 (log lines 4127–4131); aiter JIT modules (e.g. module_custom_all_reduce) built and imported successfully (log lines 4436–4442) — aiter was functional.
• Sister nightly job passed on Jun 21/20/15 with the same model and tp config.

Hypothesised Causes (with confidence)

• [MEDIUM] Model-weight download/cache failure on the MI35x runner: the local snapshot 74fb7daa lacked any safetensors, so weight loading fell back to a live HF download that hung (lock-wait, never completed in ~90 min).
  • Supporting: log explicitly states the snapshot has no weight files and "will attempt download"; py-spy stack is parked in download_weights_from_hf lock acquisition; sister runs that found cached weights loaded fine.
  • Disconfirming: cannot see from the log whether the hang is network-stalled vs. a genuine filelock deadlock between TP0/TP1; the underlying snapshot may have been mid-eviction/corrupted rather than simply absent.
  • How to verify: inspect the runner's /sgl-data/hf-cache/hub/models--amd--GLM-5.1-MXFP4 cache integrity; re-run the job with weights pre-staged; check HF connectivity from the runner.
• [LOW] aiter override a0320631 involvement.
  • Supporting: only the scout (override) run failed; sister (default aiter) passed.
  • Disconfirming: the hang is in HF weight download, a path aiter does not touch; aiter modules built/imported successfully before the hang; the scout/sister delta is more plausibly the weight-cache state, not aiter.
  • How to verify: re-run scout once weights are cached — expected to pass with the same aiter override, which would exonerate aiter.

In-flight Fix Check

• In-flight fix: ❌ none found — search repo:sgl-project/sglang is:pr is:open GLM-5.1-MXFP4 download returned 0 results. This is an infra/cache symptom, not a code bug, so a code PR is unlikely to be the right remedy.

Suggested Triage Steps

• Check the MI35x runner's HF cache for amd/GLM-5.1-MXFP4 (snapshot 74fb7daa): is it empty/partial/corrupted? Pre-stage the safetensors and re-run the job.
• Verify HF Hub connectivity / rate-limit state from the runner during the failure window (21:12–22:41 UTC).
• Re-run the scout job with the same aiter override a0320631 but with weights pre-cached; a pass confirms the failure is download/infra and exonerates aiter.
• If filelock contention between TP0/TP1 is suspected, inspect /sgl-data/hf-cache/hub/.sglang_locks/ for stale download_*.lock files.

Status

• Persistence: single observed failure in this scout run; the same job passed in the last 3 completed sister nightly runs (Jun 15/20/21).
• Scope: this job (nightly-2-gpu-mi35x-glm51-mxfp4) in scout run 27983814629; not reproduced in the sister baseline.
• Blocked work: GLM-5.1-MXFP4 TP2 GSM8K accuracy signal on MI35x for this scout run only.
• In-flight fix: ❌ none found — infra/cache investigation needed, not a code change.

call-nightly-amd / nightly-accuracy-2-gpu-vlm — failed step(s): Nightly Accuracy Test (2-GPU VLM MMMU)

Commit Info

• sglang: bbc853df
• aiter: a0320631 (source: AITER_COMMIT_OVERRIDE — scout forced rebuild; Dockerfile default is 7d604afe)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi30x/test_vlms_mmmu_eval_amd.py	test_mmmu_vlm_models (TestNightlyVLMMmmuEvalAMD)	AssertionError: The following models failed: — 5 VLM models below MMMU accuracy threshold	Log

The single test test_mmmu_vlm_models failed because 5 of 11 VLM models scored below their configured MMMU thresholds:

Model	Score	Threshold
Qwen/Qwen2-VL-7B-Instruct	0.2799	0.30
Qwen/Qwen2.5-VL-7B-Instruct	0.2901	0.33
Qwen/Qwen3-VL-30B-A3B-Instruct	0.2601	0.29
openbmb/MiniCPM-o-2_6	0.2802	0.32
deepseek-ai/deepseek-vl2-small	0.2802	0.31

Failure Summary

The accuracy job nightly-accuracy-2-gpu-vlm failed (exit code 255) when test_mmmu_vlm_models asserted that all configured VLM models meet their MMMU thresholds. 6 models passed, 5 failed. All failing scores cluster at ~0.26–0.29, just below their 0.29–0.33 thresholds. For a 4-choice MMMU benchmark, chance accuracy is ~0.25, so these scores are near-random.

Regression Status

Known recurring failure / effectively Never-passed in queryable window (lookback 2026-06-13 → 2026-06-22). The job ran under the scout override, but the same test fails identically in the regular nightly workflow on every queryable scheduled run.

Recent history of test_vlms_mmmu_eval_amd.py (test_mmmu_vlm_models) — sister workflow nightly-test-amd.yml, job nightly-accuracy-2-gpu-vlm (branch main, event schedule, completed runs only):

Date	Run	Job	Test File Status	Failed Function	Error
Jun 22 (scout)	27983814629	vlm	❌ Failed	test_mmmu_vlm_models	5 models below threshold
Jun 21	27912890405	vlm	❌ Failed	test_mmmu_vlm_models	same 5 models below threshold
Jun 20	27879266774	vlm	❌ Failed	test_mmmu_vlm_models	4 models below threshold
Jun 15	27568949589	vlm	❌ Failed	test_mmmu_vlm_models	below threshold
Jun 14	27507337214	vlm	❌ Failed	test_mmmu_vlm_models	same models below threshold

No passing run found for this test file in the lookback window (Jun 13–22). First observed failure predates the window.

Failure Origin

Test File	Test Function	Origin	Baseline Run	Baseline Status
test/registered/amd/accuracy/mi30x/test_vlms_mmmu_eval_amd.py	test_mmmu_vlm_models	pre-existing (sglang)	27912890405	❌ Same failure in sister job (default aiter, AITER_COMMIT_OVERRIDE= empty)

Baseline A/B check: The sister nightly run 27912890405 (Jun 21, most recent completed; the Jun 22 sister run 27975987522 is still queued/in-flight and excluded) ran the identical test with the Dockerfile-default aiter (AITER_COMMIT_OVERRIDE= empty, verified at log line 34) and produced the same failure on the same 5 models with near-identical scores (Qwen2-VL 0.2799, Qwen3-VL-30B 0.2601, deepseek-vl2-small 0.2802 — all identical; Qwen2.5-VL 0.2802 vs scout 0.2901; MiniCPM-o 0.2700 vs scout 0.2802 — both still below threshold). The failure reproduces without the aiter override → NOT aiter-caused.

Failure Cluster

VLM MMMU accuracy below threshold (near-random scores on AMD MI30x)

Facts (verified observations)

• Job nightly-accuracy-2-gpu-vlm failed step 6 with exit code 255; harness logged Exit code 255 indicates test failure (not network issue), not retrying.
• Ran 1 test in 3050.218s … FAILED (failures=1); final stats Passed: 6, Failed: 5.
• All 5 failing models score 0.26–0.29 on 100 MMMU samples (4-choice random baseline ≈ 0.25). Several passing models also pass only marginally (InternVL2_5-2B 0.290 vs 0.29; Janus-Pro-7B 0.290 vs 0.28).
• Scores are deterministic across days: Qwen2-VL=0.2799, Qwen3-VL-30B=0.2601, deepseek-vl2-small=0.2802 recur identically on Jun 14, 21, and 22.
• Sister baseline 82592972476 (default aiter) fails identically — AITER_COMMIT_OVERRIDE= empty confirmed.
• Test threshold config: AMD_VLM_MODELS in test_vlms_mmmu_eval_amd.py:45 (e.g. Qwen2.5-VL accuracy_threshold: 0.33, Qwen3-VL-30B 0.29).

Hypothesised Causes (with confidence)

• [FACT] The failure is not caused by the aiter override. The sister nightly baseline with default aiter reproduces the exact same test failure on the same 5 models.
  • Supporting: identical scores in scout and non-override baseline; AITER_COMMIT_OVERRIDE= empty in baseline log line 34.
  • Disconfirming: (none found).
• [LOW] The 5 models genuinely under-perform on MMMU on AMD MI30x (a real sglang-side VLM accuracy issue), with near-random scores (~0.26–0.29 vs 0.25 chance).
  • Supporting: scores are deterministic and clustered near random across 8+ days; affects multiple independent model families (Qwen, MiniCPM, deepseek-vl2).
  • Disconfirming: cannot confirm a specific regressing commit without a known green run — no passing run exists in the queryable window, so this may be a never-correctly-validated AMD port rather than a fresh regression.
• [LOW] Thresholds in AMD_VLM_MODELS are set higher than achievable on AMD (test-config issue rather than model regression).
  • Supporting: even passing models pass only marginally; scores barely move day-to-day, consistent with thresholds being mis-calibrated against actual AMD performance.
  • Disconfirming: scores near random (0.26) for a benchmark with 0.25 chance argues the models really are mis-performing, not merely that thresholds are slightly high.

No hypothesis above LOW for root cause: needs a known-green reference. Because no passing run exists in the queryable window, a git bisect has no green anchor; investigation should instead reproduce locally and compare AMD vs NVIDIA MMMU scores for these models.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for VLM MMMU accuracy and test_vlms_mmmu (https://api.github.com/search/issues?q=repo:sgl-project/sglang+is:pr+is:open+...); no PR targeting this VLM MMMU accuracy failure or these thresholds.

Suggested Triage Steps

• Reproduce locally on AMD MI30x: run test/registered/amd/accuracy/mi30x/test_vlms_mmmu_eval_amd.py for Qwen/Qwen2-VL-7B-Instruct and compare the 100-sample MMMU score against the same model/run on an NVIDIA box — confirm whether ~0.28 is AMD-specific (real accuracy bug) or also seen on NVIDIA (benchmark/threshold issue).
• If scores match NVIDIA, the thresholds in test_vlms_mmmu_eval_amd.py:45 (AMD_VLM_MODELS) are mis-calibrated — recalibrate per measured baselines.
• If AMD is materially lower than NVIDIA, this is a real VLM accuracy regression on AMD; bisect VLM forward-path / multimodal preprocessing commits, but note there is no green anchor in the queryable window — first establish any historical green run before bisecting.
• Do NOT investigate the aiter delta — the failure is reproducible without the override.

Status

• Persistence: Failing on every completed scheduled run of the sister workflow in the lookback window (at least Jun 14 → Jun 22, 2026); no green run found.
• Scope: test_mmmu_vlm_models in both amd-aiter-scout.yml (this job) and the regular nightly-test-amd.yml (nightly-accuracy-2-gpu-vlm). 5 VLM models affected.
• Blocked work: AMD nightly VLM MMMU accuracy signal is permanently red, masking any real VLM accuracy regressions on AMD.
• In-flight fix: ❌ none found in open PRs.

call-nightly-amd-rocm720 / nightly-4-gpu-rocm720 — failed step(s): Nightly Test ROCm 7.2 (4-GPU)

Commit Info

• sglang: bbc853df
• aiter: a0320631 (source: AITER_COMMIT_OVERRIDE, forced rebuild — confirmed via [CI-AITER-CHECK] markers). The override commit is "Tuned DSV4-Flash GEMM configs (#3814)".

Failed Tests

Test File	Test Function	Error	Log
test/registered/vlm/test_encoder_dp.py	TestVLMEncoderDP.test_vlm_mmmu_benchmark	AssertionError: 0.25222 not greater than or equal to 0.55 (Qwen3-VL-32B); server crashed with torch.AcceleratorError: HIP error: invalid device pointer	Log

Failure Summary

The VLM encoder-DP test (--mm-enable-dp-encoder --tp=4) crashed its scheduler workers with Failed to deserialize from CUDA IPC handle (HIP error: invalid device pointer) while reconstructing multimodal features across TP ranks. The server became unreachable (HTTP 503 / Connection error), so the mmmu evaluation returned ~random-chance accuracy 0.2522 (≈0.25 for 4-choice MMMU), failing the 0.55 threshold. Both randomly-selected models in this run — GLM-4.1V-9B-Thinking and Qwen3-VL-32B-Instruct — hit the identical 0.2522 crash signature.

Regression Status

Known recurring failure (intermittent at job level, masked by random model selection).

The test picks random.choice(MODELS) per run (test_encoder_dp.py:34) with one retry, so the job only goes red when an affected model (GLM-4.1V or Qwen3-VL-32B) is chosen for both the attempt and the retry. The underlying IPC crash is recurring.

Recent history of test_encoder_dp.py (test_vlm_mmmu_benchmark) in sister job nightly-4-gpu-rocm720 (nightly-test-amd-rocm720.yml, event=schedule, branch=main, completed runs only):

Date	Run	Model picked	Test File Status	Detail
Jun 22 (scout)	27983814629	GLM-4.1V + Qwen3-VL-32B	❌ Failed	Both 0.2522, IPC crash
Jun 21	27912828989	GLM-4.1V → InternVL2_5-8B	✅ Passed	GLM crashed (0.2522); retry hit InternVL (0.5311) → pass
Jun 20	27879239829	InternVL2_5-8B	✅ Passed	0.5333
Jun 15	27568866028	—	❌ Failed	timeout after 3600s (same server-instability family)

First clearly-observed IPC crash in queryable window: Jun 21 (GLM-4.1V, sister baseline); a related timeout on Jun 15. InternVL2_5-8B consistently passes — the crash is model-specific (GLM-4.1V, Qwen3-VL-32B).

Failure Origin

Test File	Test Function	Origin	Baseline Run	Baseline Status
test/registered/vlm/test_encoder_dp.py	test_vlm_mmmu_benchmark	pre-existing (sglang)	27912828989	❌ Same 0.2522 IPC crash on GLM-4.1V in sister job (default/dev aiter)

A/B check result: The sister nightly-test-amd-rocm720.yml run used the dev/Dockerfile aiter (no override) and GLM-4.1V crashed with the exact same 0.2522 signature there — the job only stayed green because the retry randomly landed on InternVL. The failure reproduces without the aiter override, so it is NOT aiter-caused. (The override commit is merely GEMM-config tuning, unrelated to multimodal CUDA-IPC transport.) Origin = pre-existing (sglang).

Failure Cluster

HIP "invalid device pointer" CUDA-IPC crash in multimodal DP-encoder feature transport (ROCm)

Facts (verified observations)

• Scheduler crash stack (verbatim, step 6 line ~4727):
  • scheduler.py:2092 handle_generate_request → scheduler.py:1908 _get_multimodal_inputs → schedule_batch.py:513 from_processor_output → schedule_batch.py:362 reconstruct → cuda_ipc_transport_utils.py:485 reconstruct_on_target_device → torch.UntypedStorage._new_shared_cuda(...) → torch.AcceleratorError: HIP error: invalid device pointer.
  • Followed by SIGQUIT received ... one child failed (TP1 and TP3 both crash).
• Symptom chain: scheduler crash → GET /health_generate ... 503 Service Unavailable → Health check failed. Server couldn't get a response from detokenizer → lmms-eval All 5 attempts failed. Last error: Connection error. → mmmu_acc = 0.2522.
• Reproducibility: identical 0.25222 on two distinct models (GLM-4.1V threshold 0.68; Qwen3-VL-32B threshold 0.55) in this run, and again on GLM-4.1V in the Jun 21 sister baseline with default aiter.
• aiter override a0320631 = "Tuned DSV4-Flash GEMM configs" — GEMM tuning only; does not touch multimodal IPC transport.
• cuda_ipc_transport_utils.py last modified by 5b3eeaf5 (#23377, 2026-06-21), which is an ancestor of the run SHA.

Hypothesised Causes (with confidence)

• [MEDIUM] The --mm-enable-dp-encoder CUDA-IPC feature transport (cuda_ipc_transport_utils.py:485, _new_shared_cuda) is unreliable on ROCm/HIP for GLM-4.1V and Qwen3-VL-32B, raising hipErrorInvalidDevicePointer when a TP rank reconstructs a shared-CUDA storage from an IPC handle.
  • Supporting: stack trace points directly at reconstruct_on_target_device → _new_shared_cuda; reproduces across two models and across two separate runs/aiter versions; InternVL2_5-8B (different mm feature shapes/path) does not crash.
  • Disconfirming: no clean A/B revert performed; cannot yet say whether it ever passed for GLM-4.1V/Qwen3-VL-32B on ROCm (may be never-passed for these models rather than a regression).
  • How to verify: run python3 test/registered/vlm/test_encoder_dp.py pinned to Qwen/Qwen3-VL-32B-Instruct on an MI30x ROCm 7.2 container; reproduce locally and bisect the encoder-DP IPC path.
• [LOW] Recent change 5b3eeaf5 (#23377, MM pool GPU alloc with base_gpu_id) to the crashing file affected handle/device-index redirection.
  • Supporting: most recent edit to cuda_ipc_transport_utils.py; touches device-index / handle rebuild logic adjacent to the failing _new_shared_cuda call.
  • Disconfirming: the crash persists after this commit (it is itself a fix attempt in this area), and the Jun 15 sister run already timed out before it merged — so the instability predates this commit. Unlikely to be the originating cause.
  • How to verify: not a priority; the IPC instability predates this commit.
• [LOW] aiter override a0320631.
  • Supporting: only AMD path affected.
  • Disconfirming: baseline with default aiter reproduces the identical crash; override is GEMM-only and unrelated to mm IPC. Effectively exonerated.

In-flight Fix Check

• In-flight fix: ❌ none found — Searched open PRs for CUDA IPC encoder and dp-encoder ipc. Matches (#24013, #16006, #21068) are VLM/DP-ViT feature/refactor/test PRs, none targeting the ROCm hipErrorInvalidDevicePointer in cuda_ipc_transport_utils.py. Needs a new investigation/PR.

Suggested Triage Steps

• Reproduce on an MI30x ROCm 7.2 container by pinning the test to Qwen/Qwen3-VL-32B-Instruct (or GLM-4.1V-9B-Thinking) instead of random.choice, with --mm-enable-dp-encoder --tp=4, and capture per-rank logs around cuda_ipc_transport_utils.py:485.
• Determine whether the encoder-DP IPC path ever passed for these two models on ROCm (check older nightly history); if never, treat as a ROCm IPC-support gap, not a fresh regression.
• If confirmed ROCm-specific, investigate a non-IPC fallback for multimodal feature transport on HIP, or guard _new_shared_cuda reconstruction with a HIP-validity check.
• Consider making the test deterministic per-model (or sharding the 4 models) so this crash stops being masked by random selection + retry.

Status

• Persistence: IPC crash observed Jun 21 (sister, GLM-4.1V) and Jun 22 (scout, GLM-4.1V + Qwen3-VL-32B); related timeout Jun 15. Recurring but intermittently green when InternVL2_5-8B is randomly picked.
• Scope: nightly-4-gpu-rocm720 job; affects multimodal DP-encoder path for GLM-4.1V-9B-Thinking and Qwen3-VL-32B-Instruct on ROCm. NVIDIA parity not assessed here.
• Origin: pre-existing (sglang) — reproduces with default aiter; not caused by the aiter override.
• In-flight fix: ❌ none found — needs new PR or investigation.

call-nightly-amd / nightly-8-gpu-minimax-m27 — failed step(s): Accuracy Test (8-GPU MiniMax-M2.7)

Commit Info

• sglang: bbc853df
• aiter: a0320631 (source: AITER_COMMIT_OVERRIDE — scout forced rebuild; Dockerfile default was 7d604afe / v0.1.14rc1.dev240+g7d604afe5)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi30x/test_minimax_m27_eval_amd.py	TestMiniMaxM27EvalAMD.test_minimax_m27_accuracy	AssertionError: Failed models: ['MiniMaxAI/MiniMax-M2.7 (TP8+EP8)'] — accuracy=0.925 threshold=0.93 FAIL	Log

Failure Summary

The single test in this job, test_minimax_m27_accuracy, ran the GSM8K completion benchmark (200 questions) against MiniMaxAI/MiniMax-M2.7 with TP8+EP8 and scored 0.925, just under the 0.93 threshold (test_minimax_m27_eval_amd.py:67). 0.925 = 185/200; the threshold requires 186/200 — the test failed by a single question. The server itself started and served all 200 requests successfully; this is an accuracy-margin failure, not a crash.

Regression Status

Flaky / borderline-threshold failure (first observed failure for this test file in scout history; surrounding runs pass within noise).

Recent history of test_minimax_m27_eval_amd.py (test_minimax_m27_accuracy) for the nightly-8-gpu-minimax-m27 job (same branch main, completed runs only):

Date	Run	Job	Test File Status	Accuracy	aiter
Jun 22 (this scout)	27983814629	call-nightly-amd / nightly-8-gpu-minimax-m27	❌ Failed	0.925	override a0320631
Jun 21 (sister nightly)	27912890405	nightly-8-gpu-minimax-m27	✅ Passed	0.930	default 7d604afe
Jun 18 (prev scout)	27788558636	call-nightly-amd / nightly-8-gpu-minimax-m27	✅ Passed	0.950	override a40c487b

Observed accuracy band across the last 3 completed runs: 0.925 / 0.930 / 0.950 (a ~5-question spread). The 0.93 threshold sits inside this noise band, and the passing baseline cleared it by exactly one question.

Failure Origin

unclear — degenerate A/B: the sister baseline passed, but only by a single question while sitting exactly on the threshold, so the comparison cannot separate aiter-causation from sampling variance.

Test File	Test Function	Origin	Baseline Run	Baseline Status
test_minimax_m27_eval_amd.py	test_minimax_m27_accuracy	unclear	27912890405	✅ Passed at 0.930 (sister job, default aiter — exactly on the 0.93 line)

Per the Baseline A/B table, "test passes in sister baseline" normally maps to aiter-caused. I am downgrading to unclear because the baseline is not "much greener": it passed by 0.000 margin (0.930 == threshold, 186/200), i.e. one question away from the same failure. With observed run-to-run variance of ~5 questions, a default-aiter run could plausibly land at 0.925 too. The A/B signal here is not strong enough to attribute causation to the aiter override.

Failure Cluster

MiniMax-M2.7 GSM8K accuracy below threshold (borderline, single-question margin)

Facts (verified observations)

• Log: accuracy=0.925 threshold=0.93 FAIL then AssertionError: Failed models: ['MiniMaxAI/MiniMax-M2.7 (TP8+EP8)'] at test_minimax_m27_eval_amd.py:241 (job log line 8218/8228).
• Threshold is accuracy_threshold=0.93 for the TP8+EP8 MiniMax-M2.7 config (test_minimax_m27_eval_amd.py:67); num_questions=200 (line 170). 0.925·200 = 185 correct; 0.93·200 = 186 → failed by 1 question.
• Server ran to completion: all 200 /generate requests returned 200 OK; decode used CUDA graph; no crash, OOM, or kernel fault in the log.
• aiter that actually ran: override a0320631, confirmed by [CI-AITER-CHECK] AITER_COMMIT_OVERRIDE=a0320631… → forcing rebuild (log line 4085).
• Sister nightly baseline (default aiter 7d604afe) scored exactly 0.930 PASS (baseline log line 7345).
• Prior scout (aiter override a40c487b) scored 0.950 PASS (prev-scout log line 8715).

Hypothesised Causes (with confidence)

• [MEDIUM] Borderline accuracy threshold flake — the 0.93 threshold lies inside the test's natural variance band (0.925–0.950 across 3 recent runs); a one-question drop is within sampling noise for a 200-question GSM8K eval with non-greedy generation.
  • Supporting: baseline passed by exactly 0.000 margin (186/200); the three most recent completed runs span 5 questions; failure delta is a single question.
  • Disconfirming: strictly, the override run failed where the default-aiter baseline passed, which is consistent with a small real aiter-induced numeric shift rather than pure noise.
  • How to verify: re-run test_minimax_m27_eval_amd.py 3–5× on AMD MI30x with the same override aiter a0320631 and observe the accuracy distribution vs the 0.93 line.
• [LOW] aiter override a0320631 introduces a small numerical regression vs the default 7d604afe.
  • Supporting: scout (override) failed at 0.925 while the immediately-adjacent default-aiter nightly passed at 0.930.
  • Disconfirming: the baseline passed by only one question, so the 0.005 gap is within run-to-run variance; the prior scout on a different aiter override (a40c487b) passed at 0.950, so override-vs-default is not a consistent predictor of accuracy. No code-level evidence linking any aiter diff to MiniMax accuracy.
  • How to verify: A/B the two aiter commits (7d604afe vs a0320631) on the same sglang SHA with multiple repeats; compare mean accuracy, not a single sample.

No hypothesis reaches HIGH; the dominant explanation is a borderline-threshold flake.

In-flight Fix Check

• In-flight fix: ❌ none found — GET /search/issues?q=repo:sgl-project/sglang+is:pr+is:open+minimax+m27+accuracy returned 0 results. No open PR adjusting this threshold or fixing MiniMax-M2.7 AMD accuracy.

Suggested Triage Steps

• Re-run the nightly-8-gpu-minimax-m27 scout job (or test_minimax_m27_eval_amd.py standalone) 3–5× on AMD MI30x to characterize the accuracy distribution around 0.93; if it straddles the line, treat as a flake.
• If repeated runs confirm a stable ~0.925 under override aiter a0320631 but ~0.930+ under default 7d604afe, escalate as a genuine aiter numeric regression and A/B-bisect the aiter delta 7d604afe..a0320631.
• Consider whether the 0.93 threshold has too little headroom for a 200-question GSM8K eval (one question = 0.005); a maintainer may want to either lower the threshold or raise GSM8K_NUM_QUESTIONS to reduce per-question quantization of the score.

Status

• Persistence: First failure of this test file in the queryable scout history; the two adjacent completed runs (Jun 21 sister nightly, Jun 18 scout) both passed. Not a sustained regression.
• Scope: 1 job, 1 test (nightly-8-gpu-minimax-m27), this scout run only.
• Blocked work: MiniMax-M2.7 TP8+EP8 accuracy signal on AMD MI30x for this scout cycle.
• In-flight fix: ❌ none found.

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-06-23 12:59 UTC)

[amd-bot]

amd-aiter-scout.yml — Per-Job Analysis (continued 2/5)

call-nightly-amd-rocm720 / nightly-accuracy-2-gpu-rocm720 — failed step(s): Nightly Test ROCm 7.2 (2-GPU)

Commit Info

• sglang: bbc853df
• aiter: a0320631 (source: override — AITER_COMMIT_OVERRIDE forced a rebuild over the Dockerfile default 7d604afe / image dev build v0.1.14rc1.dev240+g7d604afe5)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi30x/test_gsm8k_eval_amd.py	test_gsm8k_all_models (model=neuralmagic/Meta-Llama-3.1-70B-Instruct-FP8)	Exception: Server process exited with code -9 → HIP out of memory during decode CUDA graph capture	Log
test/registered/amd/accuracy/mi30x/test_gsm8k_eval_amd.py	test_gsm8k_all_models (model=neuralmagic/Mistral-7B-Instruct-v0.3-FP8)	gsm8k score 0.316 < threshold 0.46	Log
test/registered/amd/accuracy/mi30x/test_gsm8k_eval_amd.py	test_gsm8k_all_models (model=mistralai/Mistral-7B-Instruct-v0.3)	gsm8k score 0.356 < threshold 0.47	Log
test/registered/amd/accuracy/mi30x/test_gsm8k_eval_amd.py	test_gsm8k_all_models (model=mistralai/Mixtral-8x7B-Instruct-v0.1)	gsm8k score 0.302 < threshold 0.69	Log
test/registered/amd/accuracy/mi30x/test_gsm8k_eval_amd.py	test_gsm8k_all_models (model=neuralmagic/Meta-Llama-3.1-8B-Instruct-FP8)	gsm8k score 0.795 < threshold 0.80 (marginal, −0.005)	Log

The remaining 5 models in the suite passed: meta-llama/Llama-3.1-8B-Instruct (0.823), deepseek-ai/DeepSeek-Coder-V2-Lite-Instruct (0.818), google/gemma-2-27b-it (0.861), meta-llama/Llama-3.1-70B-Instruct (0.930, non-FP8 TP=2), Qwen/Qwen2-57B-A14B-Instruct (0.798).

Failure Summary

The single suite test test_gsm8k_all_models (a parameterized loop over 10 models) failed. The unittest framework recorded FAILED (errors=1) — the one hard error is the neuralmagic/Meta-Llama-3.1-70B-Instruct-FP8 server crashing with exit code −9 (HIP out of memory during decode CUDA-graph capture). Four additional models reported Status: ❌ FAILED for low gsm8k accuracy (the two Mistral/Mixtral families well below threshold; one FP8-8B marginally below). These are two distinct symptom families: an OOM server crash and an accuracy regression.

Regression Status

Known recurring failure (accuracy family) + unclear/insufficient-history (70B-FP8 OOM).

Recent history of test_gsm8k_eval_amd.py (test_gsm8k_all_models) in the sister job nightly-accuracy-2-gpu-rocm720 (workflow nightly-test-amd-rocm720.yml, branch main, event schedule, completed runs only):

Date	Run	Job	Test File Status	Failed Models	Note
Jun 21	27912828989	nightly-accuracy-2-gpu-rocm720	❌ Failed	mistralai/Mistral-7B-Instruct-v0.3 (0.363<0.47), google/gemma-2-27b-it (0.857<0.86)	non-override baseline
Jun 20	27879239829	(workflow failure)	❌ Failed	—	run-level failure
Jun 19	27841393725	—	⚠️ Cancelled	—	excluded
Jun 15	27568866028	(workflow failure)	❌ Failed	—	run-level failure

The sister nightly-test-amd-rocm720.yml schedule runs have been failure on every completed day (Jun 13, 14, 15, 20, 21; Jun 16–19 cancelled). The Mistral/Mixtral accuracy shortfall is long-standing on rocm720, NOT new today. No green run for the accuracy suite found in the queryable window.

Failure Origin

Test File	Failing Model	Origin	Baseline Run	Baseline Status
test_gsm8k_eval_amd.py	mistralai/Mistral-7B-Instruct-v0.3	pre-existing (sglang)	27912828989	❌ Same failure, ~same score (0.363 vs scout 0.356), default aiter
test_gsm8k_eval_amd.py	neuralmagic/Mistral-7B-Instruct-v0.3-FP8	pre-existing (sglang)	27912828989	❌ non-FP8 Mistral fails identically without override → accuracy issue independent of aiter
test_gsm8k_eval_amd.py	mistralai/Mixtral-8x7B-Instruct-v0.1	unclear	27912828989	Model not run in sister baseline; fits the same Mistral/Mixtral degradation pattern
test_gsm8k_eval_amd.py	neuralmagic/Meta-Llama-3.1-8B-Instruct-FP8	unclear	27912828989	FP8-8B not run in sister; marginal (−0.005)
test_gsm8k_eval_amd.py	neuralmagic/Meta-Llama-3.1-70B-Instruct-FP8	unclear	27912828989	FP8-70B not run in sister; non-FP8 70B PASSED (0.930) in both scout and baseline

A/B verdict: The aiter override does not explain the failures. The one model present in both override (scout) and non-override (sister) runs — mistralai/Mistral-7B-Instruct-v0.3 — fails with essentially the same score in both (0.356 vs 0.363), conclusively pre-existing in sglang. The 70B-FP8 OOM has no comparable baseline (sister ran only the non-FP8 70B, which passed), so it is unclear rather than aiter-attributable. Zero aiter commits are cited below.

Failure Cluster

• Cluster A: "Mistral/Mixtral gsm8k accuracy below threshold on AMD rocm720"
• Cluster B: "HIP out of memory during decode CUDA graph capture (FP8 70B, TP=2)"

Facts (verified observations)

• Exception: Server process exited with code -9 for neuralmagic/Meta-Llama-3.1-70B-Instruct-FP8; root server-side error: torch.OutOfMemoryError: HIP out of memory. Tried to allocate 20.00 MiB. GPU 1 has a total capacity of 255.98 GiB of which 0 bytes is free. Of the allocated memory 248.20 GiB is allocated by PyTorch — i.e. the GPU was already ~99% full when the failing 20 MiB allocation occurred.
• The OOM happened inside decode_cuda_graph_runner.py:481 → Capture cuda graph failed: HIP error: out of memory; the capture progress bar shows avail_mem falling to 0.00 GB (Capturing batches (bs=120 avail_mem=0.00 GB)) before the crash.
• The failing stack frame custom_all_reduce.py:588 out = torch.empty_like(inp) is in aiter, but it is merely the next allocation after the GPU was already exhausted — log-line proximity, not causation.
• Accuracy scores (scout): mistralai/Mistral-7B-Instruct-v0.3=0.356 (thr 0.47); neuralmagic/Mistral-7B-Instruct-v0.3-FP8=0.316 (thr 0.46); mistralai/Mixtral-8x7B-Instruct-v0.1=0.302 (thr 0.69); neuralmagic/Meta-Llama-3.1-8B-Instruct-FP8=0.795 (thr 0.80).
• Sister baseline (default aiter, 27912828989): mistralai/Mistral-7B-Instruct-v0.3=0.363 (thr 0.47, ❌), google/gemma-2-27b-it=0.857 (thr 0.86, ❌) — confirms Mistral failure is independent of the aiter override.
• [CI-AITER-CHECK] markers confirm override a0320631 rebuilt over Dockerfile default 7d604afe.

Hypothesised Causes (with confidence)

Cluster A — Mistral/Mixtral accuracy

• [HIGH] Pre-existing gsm8k chat-eval / model-default accuracy bug for Mistral & Mixtral on AMD, unrelated to aiter.
  • Supporting: same mistralai/Mistral-7B-Instruct-v0.3 fails at ~0.36 in both the scout (override) and the sister baseline (default aiter); the suite has been failing on every completed rocm720 schedule run; an open PR (#27757) explicitly attributes Mistral/Mixtral gsm8k failure to FA3 default selection + chat-prompt/answer-extraction issues.
  • Disconfirming: (none found) — the override-vs-default A/B is consistent.
  • How to verify: rerun the accuracy suite with #27757 applied on a branch; confirm Mistral/Mixtral recover above threshold.

Cluster B — FP8 70B OOM during CUDA graph capture

• [LOW] Memory pressure: mem-fraction-static / decode CUDA-graph capture leaves too little headroom for the FP8-70B TP=2 config on MI30x (gfx942), exhausting GPU 1.
  • Supporting: avail_mem reaches 0.00 GB mid-capture; OOM message shows 248/256 GiB already allocated; non-FP8 70B (different memory profile) passes.
  • Disconfirming: no comparable FP8-70B baseline exists, so a config/env or model-loading regression cannot be excluded; the failing alloc surfaces in aiter custom_all_reduce but that is post-exhaustion noise.
  • How to verify: relaunch neuralmagic/Meta-Llama-3.1-70B-Instruct-FP8 TP=2 on MI30x with --mem-fraction-static 0.7 and/or --cuda-graph-max-bs-decode 16 (the in-log suggested mitigations) and observe whether capture completes.
• No hypothesis above LOW confidence for Cluster B; the FP8-70B path lacks a queryable green/baseline run.

In-flight Fix Check

• Cluster A: In-flight fix ✅ #27757 "Fix Mistral GSM8K chat eval" (open since 2026-06-10) — keeps Mistral/Mixtral off FA3 default, fixes chat-eval prompt + Answer: extraction + stop sequences. Directly targets this accuracy cluster; chase reviewers, do NOT open a duplicate. (Search: repo:sgl-project/sglang is:pr is:open mistral accuracy.)
• Cluster B: In-flight fix ❌ none found matching FP8-70B CUDA-graph OOM on rocm720 (searched rocm cuda graph memory). Needs config tuning or a new issue.

Suggested Triage Steps

• Verify #27757 merges cleanly and re-run test_gsm8k_eval_amd.py on MI30x rocm720; expect Mistral/Mixtral (and likely the FP8 Mistral) to recover — this resolves Cluster A (4 of 5 failing models).
• For Cluster B, re-launch neuralmagic/Meta-Llama-3.1-70B-Instruct-FP8 TP=2 on an MI30x runner with --mem-fraction-static 0.7 (or --cuda-graph-max-bs-decode 16); if capture succeeds, the OOM is a memory-headroom config issue, not a code regression.
• Because the A/B check exonerates the aiter override, no aiter bisect is warranted for this job.

Status

• Persistence: Accuracy family failing on every completed rocm720 schedule run in the lookback window (Jun 13–21); not new today. FP8-70B OOM: first observed in this scout run (no comparable history).
• Scope: test_gsm8k_all_models in nightly-accuracy-2-gpu-rocm720, reproduced in the sister non-override nightly — affects the AMD rocm720 accuracy signal.
• Failure origin: pre-existing (sglang) for the Mistral/Mixtral accuracy failures; unclear (no FP8-70B baseline) for the OOM crash. Aiter override is exonerated.
• In-flight fix: Cluster A ✅ #27757; Cluster B ❌ none.

call-nightly-amd-rocm720 / nightly-accuracy-2-gpu-vlm-rocm720 — failed step(s): Nightly Accuracy Test ROCm 7.2 (2-GPU VLM MMMU)

Commit Info

• sglang: bbc853df
• aiter: a0320631 (source: AITER_COMMIT_OVERRIDE → forced rebuild; Dockerfile default was 7d604afe)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi30x/test_vlms_mmmu_eval_amd.py	test_mmmu_vlm_models	AssertionError: The following models failed (5 VLMs below MMMU threshold)	Log

Per-model MMMU scores vs threshold (all near the ~0.25 random-choice baseline):

Model	TP	Score	Threshold
Qwen/Qwen2-VL-7B-Instruct	1	0.2901	0.30
Qwen/Qwen2.5-VL-7B-Instruct	1	0.2901	0.33
Qwen/Qwen3-VL-30B-A3B-Instruct	2	0.2601	0.29
openbmb/MiniCPM-o-2_6	1	0.2802	0.32
deepseek-ai/deepseek-vl2-small	1	0.2802	0.31

(6 models passed: InternVL2_5-2B, MiniCPM-v-2_6, Janus-Pro-7B, Kimi-VL-A3B, MiMo-VL-7B-RL, DeepSeek-OCR-2.)

Failure Summary

The single test test_mmmu_vlm_models ran 11 VLMs through a 100-sample MMMU eval; 6 passed and 5 failed their accuracy_threshold gate, so _check_results raised an AssertionError (job exit 255). All five failing scores (0.26–0.29) sit just below their configured thresholds (0.29–0.33) and near the MMMU random-choice floor.

Regression Status

Known recurring failure (pre-existing on AMD rocm720) — Never-passed in queryable window.

Recent history of test_vlms_mmmu_eval_amd.py (test_mmmu_vlm_models) in the sister job nightly-accuracy-2-gpu-vlm-rocm720 (nightly-test-amd-rocm720.yml, event=schedule, branch main, completed runs only — used as the comparable baseline since the scout job runs the identical test):

Date	Run	Test File Status	Failed Functions	Note
Jun 21	27912828989	❌ Failed	test_mmmu_vlm_models	Same 5 models, identical scores
Jun 20	27879239829	❌ Failed	test_mmmu_vlm_models	4 models failed (Qwen2-VL passed at 0.30)
Jun 19–16	27841393725 … 27640284588	—	—	Runs cancelled (excluded)
Jun 15	27568866028	❌ Failed	test_mmmu_vlm_models	—
Jun 14	27507292665	❌ Failed	test_mmmu_vlm_models	—
Jun 13	27474496879	❌ Failed	test_mmmu_vlm_models	—

No green run for this test exists in the queryable window (Jun 13–22). First-observed failure ≥ Jun 13; last known passing date: none in window.

Failure Origin

Test File	Test Function	Origin	Baseline Run	Baseline Status
test/registered/amd/accuracy/mi30x/test_vlms_mmmu_eval_amd.py	test_mmmu_vlm_models	pre-existing (sglang)	27912828989	❌ Identical failure in sister job (empty AITER_COMMIT_OVERRIDE)

A/B check: The most recent completed sister scheduled run (Jun 21, no aiter override — AITER_COMMIT_OVERRIDE: empty, verified in log) failed on the same test function with the same 5 models and byte-identical scores (0.2901, 0.2901, 0.2601, 0.2802, 0.2802). The aiter override a0320631 is therefore exonerated; the failure reproduces without it.

Failure Cluster

"VLM MMMU accuracy below threshold on AMD MI30x (rocm720)"

Facts (verified observations)

• _check_results raised AssertionError at test_vlms_mmmu_eval_amd.py:380; job exit code 255.
• 5/11 models below threshold; failing scores 0.2601–0.2901 vs thresholds 0.29–0.33 (verbatim from log line 15163+).
• The aiter override was applied (AITER_COMMIT_OVERRIDE=a0320631…→forcing rebuild, log line ~3640), but the non-override sister baseline reproduces the identical failure.
• Failing scores cluster near the MMMU random-choice baseline (~0.25 for multiple choice), while the 6 passing models clear lower thresholds (0.25–0.28) — i.e. the gate is failing the models with the most aggressive thresholds.
• Thresholds are hard-coded per-model in AMD_VLM_MODELS (test_vlms_mmmu_eval_amd.py:44–123).

Hypothesised Causes (with confidence)

• [MEDIUM] Per-model accuracy_threshold values are set higher than the scores AMD MI30x/rocm720 actually achieves on the 100-sample MMMU eval, so the gate fails deterministically.
  • Supporting: every failing model misses by a small, stable margin (e.g. Qwen2-VL 0.2901 vs 0.30); scores are byte-identical across the Jun 21 baseline and this run; comment at line 43 calls them "conservative thresholds" but they exceed observed scores; the 6 passing models all have thresholds ≤ their score.
  • Disconfirming: the absolute scores (~0.26–0.29) are near the random-choice floor, which could instead indicate a genuine VLM-quality problem on this backend (broken image path / preprocessing) rather than merely strict thresholds — that would not be fixed by relaxing the gate.
  • How to verify: compare these MMMU scores against the NVIDIA nightly VLM eval for the same models; if NVIDIA scores are materially higher, the issue is AMD model quality, not threshold tuning.
• [LOW] A real accuracy/correctness regression in the AMD VLM forward path (vision encoder or attention) on rocm720 depresses scores to near-random.
  • Supporting: 100-sample scores hovering at the random baseline are suspicious for a serving-quality defect.
  • Disconfirming: 6 of 11 VLMs pass, including 7B-class models — a broad VLM-path breakage would likely depress all of them; scores are stable, not noisy.
  • How to verify: run one failing model (e.g. Qwen2.5-VL-7B) locally on MI30x with a larger MMMU sample and inspect sample-level outputs for image grounding.

In-flight Fix Check

• In-flight fix: ❌ none found matching this test — searched open PRs for mmmu+vlm and vlm+accuracy. The closest hit, #26178 "[AMD] WIP - relaxing thres for mi300" (open since 2026-05-23, last updated 2026-05-26), only modifies perf test files (test_bench_serving_1gpu_part1/2.py, test_vlm_perf_5090.py), not this accuracy test — so it does not cover this failure.

Suggested Triage Steps

• Cross-check the 5 failing models' MMMU scores against the NVIDIA nightly VLM eval to decide between "thresholds too strict" (relax the gate) vs "AMD VLM quality regression" (fix the forward path).
• If quality is the issue, bisect the AMD VLM image-preprocessing / vision-encoder path on rocm720 between the last known good window and now (no green run in Jun 13–22, so widen the search beyond the queryable schedule history).
• If thresholds are confirmed too aggressive, re-tune the 5 per-model accuracy_threshold values in test_vlms_mmmu_eval_amd.py:44–123 based on observed-score distributions, mirroring the perf-side effort in #26178.
• Do not pursue aiter-side investigation — the A/B baseline reproduces without the override.

Status

• Persistence: Failing in every completed sister scheduled run in the queryable window (Jun 13–21; Jun 16–19 cancelled). No green run found for this test.
• Scope: test_mmmu_vlm_models in nightly-accuracy-2-gpu-vlm-rocm720, reproduced in both the AITER Scout run and the regular nightly-test-amd-rocm720 baseline (override-independent).
• Blocked work: AMD MI30x rocm720 VLM accuracy CI signal — 5 of 11 tracked VLMs gated red.
• In-flight fix: ❌ none found matching this test (#26178 covers perf tests only).

call-pr-test-amd-rocm720 / call-pr-test-amd-extra-rocm720 / extra-a-test-1-gpu-small-amd${{ ((inputs.rocm_version && format('-{0}', inputs.rocm_version)) || '') }} (linux-${{ (inputs.runner_arch || 'mi325') }}-1gpu-sglang) — failed step(s): N/A

Commit Info

• sglang: bbc853df
• aiter: N/A — job log blob does not exist (BlobNotFound); no [CI-AITER-CHECK] markers available because the job never ran.

Failed Tests

Test File	Test Function	Error	Log
N/A	N/A	Job cancelled before execution — no test ran	Job

This is not a test failure. The job 82820336774 has conclusion: cancelled, zero steps executed, and its log blob returns BlobNotFound. The completed_at (21:03:09Z) precedes started_at (21:03:10Z), confirming it was torn down before any work began.

Failure Summary

The target job was cancelled before it ran any step — no dependency install, no server launch, no test. There is no failing test file or function to identify. The parent AMD AITER Scout run 27983814629 is still in-flight (status: queued): of 180 jobs, 23 succeeded, 9 failed, 3 in-progress, 61 still queued, 1 skipped, 3 cancelled.

Regression Status

Infrastructure issue (job cancelled, never executed) — not a test regression. No test-file history is applicable because no test ran in this job.

Failure Origin (REQUIRED for amd-aiter-scout.yml)

unclear — the job never executed, so there is no symptom to compare against a sister baseline. A Baseline A/B check is not meaningful for a cancelled job.

Test File	Test Function	Origin	Baseline Run	Baseline Status
N/A (job cancelled)	N/A	unclear	—	Job never ran; no comparison possible

Failure Cluster

Job cancelled before execution (no test run)

Facts (verified observations, no interpretation)

• [FACT] Job 82820336774 conclusion: cancelled, status: completed; started_at=2026-06-22T21:03:10Z, completed_at=2026-06-22T21:03:09Z (completed before started → torn down immediately).
• [FACT] The job's steps array is empty — no step ran.
• [FACT] Log endpoint returns BlobNotFound — no log was ever produced.
• [FACT] Parent run 27983814629 is still queued (in-flight): 23 success / 9 failure / 3 in-progress / 61 queued / 1 skipped / 3 cancelled, out of 180 jobs (snapshot 2026-06-23 00:28 UTC).

Hypothesised Causes (with confidence, with disconfirming evidence)

• [LOW] The scout run is still queueing its 180-job matrix; this shard was cancelled by orchestration (e.g. a fast-fail/concurrency cancellation or a superseding run) before its runner picked it up.
  • Supporting: zero steps, no log blob, completed≈started timestamp, parent run still queued with 61 jobs pending.
  • Disconfirming: cannot confirm the cancellation trigger without the run's cancellation event; 9 sibling jobs did reach failure, so the matrix was not globally cancelled.
  • How to verify: re-query this job after the parent run completes; check whether a newer scout run superseded this one, or whether bypass-fastfail was absent and an upstream gate failed.

In-flight Fix Check

Not applicable — no test/library failure to match. No fix lookup performed because there is no failing test signature.

Suggested Triage Steps

• Wait for parent run 27983814629 to finish (still queued); re-run this Job Failure Analysis against a completed run so a real signal exists.
• If this shard remains cancelled after the run completes, check the run's cancellation source (superseding scout run, concurrency group, or upstream fast-fail gate) — not a code issue.
• No commit-level or aiter investigation is warranted: nothing executed.

Status (factual, no priority assignment)

• Persistence: Single occurrence; job cancelled before execution at 2026-06-22T21:03 UTC. Not a recurring test failure.
• Scope: This one shard. Parent run still in-flight (61/180 jobs queued); 9 unrelated sibling jobs reached failure and would need separate analysis once logs exist.
• Blocked work: None attributable — no test signal lost beyond this shard not yet running.
• In-flight fix: N/A — no failing test to fix.

call-pr-test-amd-rocm720 / call-pr-test-amd-extra-rocm720 / extra-a-test-1-gpu-large-amd${{ ((inputs.rocm_version && format('-{0}', inputs.rocm_version)) || '') }} (linux-${{ (inputs.runner_arch || 'mi325') }}-1gpu-sglang) — failed step(s): N/A

Commit Info

• sglang: bbc853df
• aiter: N/A for this job — it never ran, so no [CI-AITER-CHECK] markers were emitted (job log blob does not exist). The scout's intended override (visible in the sibling pr-test-amd-extra-finish env) was AITER_COMMIT_OVERRIDE=a0320631, but it was never applied here.

Failed Tests

Test File	Test Function	Error	Log
N/A — no test ran	N/A	Job cancelled before a runner was assigned (0 steps executed)	Job

This is not a test failure. The job extra-a-test-1-gpu-large-amd … (linux-…-1gpu-sglang) (id 82820336506) was cancelled: runner: None, zero steps, started_at=21:03:10Z / completed_at=21:03:09Z (completion stamped before start = instant cancel, never executed), and the GitHub log blob returns BlobNotFound.

Failure Summary

The job was cancelled as part of a gate-cascade, not because any test failed. Within the call-pr-test-amd-extra-rocm720 reusable-workflow path, the upstream call-gate / pr-gate job (id 82820330340) was itself cancelled (also BlobNotFound, instant cancel at 21:03:09Z). Because both extra-a-test-1-gpu-large-amd (this job) and extra-a-test-1-gpu-small-amd needs that gate, they were cancelled before starting. The pr-test-amd-extra-finish aggregator (id 82820336106) then failed by design — its script does if result == "failure" || "cancelled": exit 1, and it saw call-gate: cancelled.

Regression Status

Infrastructure issue — not a code/test regression. No test file was exercised, so test-file-level history is N/A. The parent run 27983814629 was still in progress at snapshot (status=queued: 61 queued, 3 in_progress, 23 success, 9 failure, 3 cancelled, 1 skipped) — [IN-FLIGHT, partial data].

Failure Origin

unclear — the Baseline A/B check cannot be performed because no test ran. There is no scout test result to compare against a sister-workflow baseline; the cancellation is an orchestration event, attributable to neither aiter nor sglang code.

Test File	Test Function	Origin	Baseline Run	Baseline Status
N/A (job cancelled, 0 tests)	N/A	unclear	—	No comparable signal — job never executed

Failure Cluster

Gate-cascade cancellation (downstream matrix jobs cancelled when upstream pr-gate is cancelled)

Facts (verified observations)

• Job 82820336506: status=completed, conclusion=cancelled, runner=None, 0 steps, started=2026-06-22T21:03:10Z, completed=2026-06-22T21:03:09Z. Log API returns BlobNotFound (verbatim XML error).
• Upstream gate 82820330340 (call-pr-test-amd-extra-rocm720 / call-gate / pr-gate): conclusion=cancelled, instant (21:03:08→21:03:09), log also BlobNotFound.
• Sibling gates succeeded: call-pr-test-amd-rocm720 / call-gate / pr-gate 82820327730 and call-pr-test-amd / call-pr-test-amd-extra / call-gate / pr-gate 82820328195. Only the call-pr-test-amd-extra-rocm720 gate was cancelled.
• Finish job 82820336106 log (verbatim): json_needs shows call-gate: cancelled, extra-a-test-1-gpu-small-amd: cancelled, extra-a-test-1-gpu-large-amd: cancelled; script printed call-gate: cancelled → The above jobs failed. → ##[error]Process completed with exit code 1.
• The job name still contains unexpanded template literals (${{ ((inputs.rocm_version && …)) }}, (inputs.runner_arch || 'mi325')), confirming the AITER-Scout-via-workflow_call invocation of amd-aiter-scout.yml.

Hypothesised Causes (with confidence)

• [FACT] This job ran no tests; it was cancelled because its needs gate pr-gate was cancelled.
  • Supporting: runner=None, 0 steps, BlobNotFound; finish-job json_needs shows the gate cancelled.
  • Disconfirming: (none — directly observable)
  • How to verify: re-read the linked job/finish logs.
• [LOW] The pr-gate cancellation (and thus the cascade) was triggered by an external orchestration event — most likely concurrency-group supersession (a newer scout/PR run cancelled this path) or a manual/scheduler cancel — rather than a job error.
  • Supporting: the gate was cancelled in ~1 s with no log produced; sibling gates on other paths succeeded in the same run; parent run is still in-flight.
  • Disconfirming: no log evidence of the trigger (gate blob missing); cannot confirm a concurrency cancel vs. other cause from available data.
  • How to verify: inspect the run's cancellation/concurrency context via the Actions UI timeline for run 27983814629, and check whether a newer run on the same concurrency group started near 21:03 UTC.

In-flight Fix Check

Not applicable — there is no failing test or code defect to fix; this is a cancellation. No PR search performed (no test name / library / error keyword to query against). In-flight fix: ❌ N/A (orchestration cancellation, not a code bug).

Suggested Triage Steps

• Re-run the call-pr-test-amd-extra-rocm720 leg (or the whole scout run 27983814629) once the parent run settles — a cancelled gate is expected to clear on a clean re-trigger.
• Check the Actions UI timeline for run 27983814629 to confirm whether the pr-gate cancellation was a concurrency-group supersession (newer run cancelled this one) vs. a scheduler/manual cancel.
• If this gate-cancel cascade recurs across scout runs, review the concurrency group / cancel-in-progress config in amd-aiter-scout.yml and the call-pr-test-amd-extra-rocm720 reusable workflow's gate, since the cascade silently turns a cancelled gate into a finish-job failure.

Status (factual)

• Persistence: single observation in an in-flight run; no test-level history (job never executed).
• Scope: the call-pr-test-amd-extra-rocm720 path only — extra-a-test-1-gpu-large-amd (this job) + extra-a-test-1-gpu-small-amd 82820336774 cancelled, pr-test-amd-extra-finish 82820336106 failed. Sibling rocm720 and non-rocm720 extra gates were unaffected.
• Blocked work: the rocm720 "extra-a" 1-GPU test signal is missing for this scout run; no AMD test coverage was produced by this leg.
• In-flight fix: ❌ N/A — orchestration cancellation, not a code bug.

call-pr-test-amd-rocm720 / call-pr-test-amd-extra-rocm720 / call-gate / pr-gate — failed step(s): N/A

Commit Info

• sglang: bbc853df (scout ref amd/aiter-ci)
• aiter: a0320631 (source: override — AITER_COMMIT_OVERRIDE set in the extra-rocm720 chain)

Failed Tests

Test File	Test Function	Error	Log
N/A (no test ran)	N/A	Job call-gate / pr-gate cancelled before any step executed (0 steps; started 21:03:08Z, completed 21:03:09Z)	Log

This job is not a test job. It is the label/cool-down gate (pr-gate.yml) for the call-pr-test-amd-extra-rocm720 chain. No test file or function executed.

Failure Summary

The analyzed job is the call-gate / pr-gate gate of the ROCm 7.2 extra-test chain. It was cancelled at scheduling (zero steps, completed_at 21:03:09Z is before started_at 21:03:08Z — the hallmark of a job killed before it ever ran). The cancellation cascaded: the two extra test jobs (extra-a-test-1-gpu-large-amd, extra-a-test-1-gpu-small-amd) were also cancelled before running (their names still show unexpanded ${{ ... }} matrix templates), and pr-test-amd-extra-finish then exited 1 because all its needs resolved to cancelled.

Regression Status

Infrastructure issue (orchestration cancellation cascade) — not a code/test regression.

The sister non-extra gate in the same run, call-pr-test-amd-rocm720 / call-gate / pr-gate, succeeded with identical inputs, and the NVIDIA-side extra gate call-pr-test-amd / call-pr-test-amd-extra / call-gate / pr-gate also succeeded. Only the rocm720 extra chain's gate was cancelled. The overall scout run 27983814629 was still queued/in-progress at snapshot, with many nightly jobs still running — so this is a partial-run snapshot.

Failure Origin

unclear — infrastructure cancellation, not attributable to aiter or sglang code.

Test File	Test Function	Origin	Baseline	Baseline Status
N/A (gate cancelled)	N/A	unclear	sister gate call-pr-test-amd-rocm720 / call-gate / pr-gate	✅ Succeeded (same run, no test executed)

The Baseline A/B check does not meaningfully apply: there is no test failure to reproduce — the gate never ran a step. The aiter override is therefore irrelevant to this cancellation.

Failure Cluster

Gate job cancelled at scheduling → extra-chain finish cascade (orchestration cancellation)

Facts (verified observations)

• [FACT] Analyzed job call-gate / pr-gate (id 82820330340) conclusion = cancelled, 0 steps, started_at 2026-06-22T21:03:08Z, completed_at 2026-06-22T21:03:09Z. The log blob returns BlobNotFound (no log was ever produced).
• [FACT] The finish job log enumerates its needs: call-gate: cancelled, extra-a-test-1-gpu-small-amd: cancelled, extra-a-test-1-gpu-large-amd: cancelled; it printed The above jobs failed. then ##[error]Process completed with exit code 1 (finish job log lines 41-73).
• [FACT] In .github/workflows/pr-test-amd-extra.yml, the extra test jobs are guarded by if: ${{ !cancelled() && needs.call-gate.result == 'success' }} (lines 124, 156). A cancelled call-gate therefore forces the dependent test jobs to be cancelled, and pr-test-amd-extra-finish (needs call-gate) treats cancelled as failure → exit 1.
• [FACT] pr-gate.yml gate steps are all guarded if: github.event_name == 'pull_request'; on this schedule event they would all skip, so the gate is a no-op that should succeed (and its sister copy did succeed in the same run).
• [FACT] Scout concurrency group is amd-aiter-scout-${{ github.run_id }} (run-unique), so cross-run cancel-in-progress did NOT cancel this job.

Hypothesised Causes (with confidence)

• [LOW] Transient GitHub Actions scheduling/runner-allocation cancellation of the nested reusable-workflow gate job.
  • Supporting: job killed before any step with completed_at < started_at; identical sister gate in the same run succeeded with the same inputs; the gate's own steps are no-ops on a schedule event so it has no internal failure path.
  • Disconfirming: no log was retained (BlobNotFound), so the cancellation signal source cannot be confirmed from logs; could alternatively be a manual/automated cancellation of the extra sub-workflow.
  • How to verify: re-run the call-pr-test-amd-extra-rocm720 chain; check the GitHub Actions run timeline/annotations for who/what issued the cancel; confirm the gate succeeds on rerun.
• [SPECULATION] Nested-reusable-workflow concurrency interaction at the pr-test-amd-extra.yml level (its concurrency.cancel-in-progress: ${{ github.event_name != 'workflow_call' }} evaluates to false under workflow_call, so this is unlikely but not fully excludable).
  • Supporting: only the deepest-nested (extra-rocm720) chain was affected.
  • Disconfirming: the guard evaluates false for workflow_call, which is exactly how this chain is invoked.
  • How to verify: inspect run annotations for a concurrency-cancel event.

No hypothesis above LOW confidence; the missing gate log prevents a definitive cause.

In-flight Fix Check

• In-flight fix: ❌ none found — this is a transient orchestration cancellation, not a tracked code defect. Searched open PRs for gate/cancel keywords:
  • curl -s -H "Authorization: token $GH_PAT" "https://api.github.com/search/issues?q=repo:sgl-project/sglang+is:pr+is:open+pr-gate+cancelled" — no matching fix PR.

Suggested Triage Steps

• Re-run only the call-pr-test-amd-extra-rocm720 chain of scout run 27983814629 and confirm the call-gate / pr-gate job runs to success; a clean rerun confirms this was transient.
• Inspect the run's Actions timeline/annotations for the cancellation source (manual cancel vs. runner reclamation vs. concurrency) — the job log itself is unavailable (BlobNotFound).
• Do not treat this as an aiter or sglang regression: no test executed, and the sister non-extra rocm720 gate succeeded in the same run with the same aiter override.

Status

• Persistence: Single occurrence in this run; gate cancelled at 2026-06-22T21:03:08Z. Sister gate in same run passed.
• Scope: One gate job; cascaded to 2 cancelled extra test jobs + 1 failed finish job, all within the call-pr-test-amd-extra-rocm720 chain only. Other rocm720 and NVIDIA gates unaffected.
• Blocked work: The ROCm 7.2 extra test tier did not run in this scout run (no signal produced for those suites).
• In-flight fix: ❌ none found — transient infra cancellation; rerun to recover signal.

call-pr-test-amd / multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 1) — failed step(s): Run diffusion server tests (2-GPU)

Commit Info

• sglang: bbc853df
• aiter: a0320631 (source: AITER_COMMIT_OVERRIDE — scout forced rebuild; Dockerfile default was 7d604afe)

Failed Tests

Test File	Test Function	Error	Log
python/sglang/multimodal_gen/test/server/test_server_2_gpu.py	TestDiffusionServerTwoGpu::test_diffusion_generation[ideogram4_fp8_tp2_t2i]	huggingface_hub.errors.GatedRepoError: 403 Client Error → ValueError: Model or revision not found at ideogram-ai/ideogram-4-fp8	Log

Run result: 10 passed, 12 deselected, 1 error in 2821.39s.

Failure Summary

The 2-GPU multimodal-gen server test errored while loading the diffusion model ideogram-ai/ideogram-4-fp8. The Hugging Face Hub returned 403 Forbidden / GatedRepoError ("Access to model ideogram-ai/ideogram-4-fp8 is restricted and you are not in the authorized list"), so maybe_download_model (hf_diffusers_utils.py:853) raised and the server process exited before the test could run. All other 10 cases in the shard passed.

Regression Status

Known recurring failure (infrastructure / gated-repo access).

Recent history of test_server_2_gpu.py (test_diffusion_generation[ideogram4_fp8_tp2_t2i]), checked via the sister workflow pr-test-amd.yml (event=schedule, branch main, completed runs only — see Failure Origin below):

Date	Run	Job	Test File Status	Failed Function	Error
Jun 22 13:56	run	multimodal-gen-test-2-gpu-amd (…, 1)	❌ Error	[ideogram4_fp8_tp2_t2i]	GatedRepoError 403
Jun 22 08:09	run	multimodal-gen-test-2-gpu-amd (…, 1)	❌ Error	[ideogram4_fp8_tp2_t2i]	GatedRepoError 403
Jun 22 01:00	run	multimodal-gen-test-2-gpu-amd (…, 1)	❌ Error	[ideogram4_fp8_tp2_t2i]	GatedRepoError 403
Jun 21 18:35	run	multimodal-gen-test-2-gpu-amd (…, 1)	❌ Error	[ideogram4_fp8_tp2_t2i]	GatedRepoError 403

No green run for this test was found in the queryable window (Jun 21–22). Same symptom every run.

Failure Origin

Test File	Test Function	Origin	Baseline Run	Baseline Status
test_server_2_gpu.py	test_diffusion_generation[ideogram4_fp8_tp2_t2i]	pre-existing (sglang)	run 27958083310, job 82731802132	❌ Same GatedRepoError 403 failure in sister job (default aiter)

The sister pr-test-amd.yml scheduled run with the Dockerfile-default aiter reproduces the identical failure (same test, same gated-repo 403). The aiter override is exonerated — zero aiter commits cited.

Failure Cluster

HF gated-repo 403 on ideogram-ai/ideogram-4-fp8 during diffusion model load

Facts (verified observations)

• [FACT] Log: huggingface_hub.errors.GatedRepoError: 403 Client Error and Access to model ideogram-ai/ideogram-4-fp8 is restricted and you are not in the authorized list (log).
• [FACT] Failure raised in hf_diffusers_utils.py:853 (maybe_download_model) → diffusers_pipeline.py:401 (_load_diffusers_pipeline), i.e. at model download, before inference.
• [FACT] Only the ideogram4_fp8_tp2_t2i case errored; the other 10 cases in the shard passed.
• [FACT] Sister baseline (default aiter, run 27958083310) shows the same test erroring with the same 403, and the test errored in all 4 most-recent sister scheduled runs (Jun 21–22).

Hypothesised Causes

• [FACT] The HF repo ideogram-ai/ideogram-4-fp8 is gated and the CI token is not in its authorized list (403 Forbidden). This is an access/infrastructure condition, not a code defect.
  • Supporting: explicit 403 GatedRepoError; reproduces identically with and without the aiter override and across 4 consecutive sister runs.
  • Disconfirming: (none found) — no stack frame implicates sglang model/runtime logic beyond the download step.
  • How to verify: confirm the CI HF_TOKEN has been granted access at https://huggingface.co/ideogram-ai/ideogram-4-fp8, or check whether the model id/revision is correct.

In-flight Fix Check

• In-flight fix: ⚠️ candidate #28225 "Route Ideogram NF4 without gated metadata fetch" — targets the gated-metadata-fetch path for Ideogram; may or may not cover the fp8 variant. Verify scope before relying on it.
• Searched repo:sgl-project/sglang is:pr is:open ideogram (7 open PRs); none explicitly mentions ideogram-4-fp8 access for CI.

Suggested Triage Steps

• Grant the CI Hugging Face account access to ideogram-ai/ideogram-4-fp8 (the test is gated, not broken by code).
• If access cannot be granted, consider skipping/quarantining the ideogram4_fp8_tp2_t2i case on CI until authorized, or confirm #28225 removes the gated metadata fetch for the fp8 path and re-run.
• No aiter or sglang code bisect is warranted — the baseline reproduces the failure without the aiter override.

Status

• Persistence: Erroring in every completed sister run Jun 21 18:35 → Jun 22 13:56 UTC; no green run in the queryable window.
• Scope: multimodal-gen-test-2-gpu-amd (…, 1) shard in both amd-aiter-scout.yml and pr-test-amd.yml; single test case only.
• Blocked work: Blocks the ideogram4_fp8_tp2_t2i 2-GPU diffusion signal on AMD; other cases in the shard unaffected.
• In-flight fix: ⚠️ candidate #28225 (scope unconfirmed for fp8); root remediation is HF gated-repo access for the CI token.

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-06-23 12:59 UTC)

[amd-bot]

amd-aiter-scout.yml — Per-Job Analysis (continued 3/5)

call-pr-test-amd / jit-kernel-unit-test-amd (linux-mi325-1gpu-sglang) — failed step(s): Run JIT kernel unit tests

Commit Info

• sglang (under test): a7fd09207 — from installed version string sglang==0.0.0.dev1+ga7fd09207.d20260622 (committed 2026-06-16T03:28Z, "[AMD] Fix AITER Scout workflow permissions (#28360)"). Note: run head_sha is bbc853df (tip of the amd/aiter-ci scout branch), but the sglang source actually compiled in /sglang-checkout is a7fd09207.
• aiter: a0320631 (source: override — AITER_COMMIT_OVERRIDE, forced rebuild over Dockerfile default 7d604afe)

Failed Tests

Test File	Test Function	Error	Log
test/registered/jit/test_activation.py	test_activation_correctness[*], test_activation_out_param[*] (all params)	RuntimeError: ninja exited with status 1 — clang-HIP compile error in activation.cuh:213/217	Log
test/registered/jit/test_per_token_group_quant_8bit.py	test_per_token_group_quant_with_column_major[*] (1287 failed, 0 passed)	✗ TIMEOUT after 1200 seconds	Log

Failure Summary

The jit-kernel-unit-test-amd job fails on two JIT test files. test_activation.py fails because the JIT compile of activation.cuh aborts with a clang-HIP (gfx942) C++ template error: select_unary_kernel declared its return type as decltype(unary_kernel<kReLU2>) (a const-qualified auto), which clang-HIP refuses to initialize from a function-pointer lvalue / nullptr (activation.cuh:213, :217), compounded by forward<const auto &> used before definition (utils.cuh:301). test_per_token_group_quant_8bit.py ran 1287 parametrizations on AMD (all failed) and hit the 1200 s file timeout. Both are exactly the symptoms addressed by sglang #27947.

Regression Status

Known recurring failure on the scout base — already fixed on main. The sglang version compiled by this scout (a7fd09207, 2026-06-16 03:28Z) predates the fix 72d962be (#27947, merged 2026-06-16 06:44Z) by ~3 hours, so the scout base does not contain the fix.

Recent history of test_activation.py / test_per_token_group_quant_8bit.py in job jit-kernel-unit-test-amd (linux-mi325-1gpu-sglang) (sister workflow pr-test-amd.yml, event=schedule, branch main, completed runs only):

Date	Run	Job	Test File Status	Failed Function	Error
Jun 22	27958083310	jit-kernel-unit-test-amd	✅ Passed	—	—

On current main (head 4923bb93, with the fix) this job passes. The failure only reproduces on the scout's stale pinned base.

Failure Origin

Test File	Test Function	Origin	Baseline Run	Baseline Status
test/registered/jit/test_activation.py	test_activation_correctness / test_activation_out_param	pre-existing (sglang)	27958083310	✅ Passed (sister job, current main, default aiter)
test/registered/jit/test_per_token_group_quant_8bit.py	test_per_token_group_quant_with_column_major	pre-existing (sglang)	27958083310	✅ Passed (sister job, current main, default aiter)

The failure is a host-side C++ template compile error in sglang's own python/sglang/jit_kernel/csrc/elementwise/activation.cuh and an AMD-only test-coverage gap in per_token_group_quant_8bit. The aiter override is a kernel-binary swap and cannot affect sglang's JIT C++ compilation. Sister baseline (newer sglang, regular aiter) passes because it carries the fix; the differing variable is the sglang version, not aiter.

Failure Cluster

JIT activation.cuh ROCm (gfx942) C++ template compile failure + per_token_group_quant AMD coverage timeout

Facts (verified observations)

• [FACT] Compile error verbatim: activation.cuh:213:14: error: cannot initialize return object of type 'decltype(ActivationKernel<__half, false>::unary_kernel<ActivationKind::kReLU2>)' (aka 'const auto') with an lvalue of type 'void (*const)(const UnaryActivationParams)' and :217:12: ... with an rvalue of type 'std::nullptr_t'; also utils.cuh:301:14: error: function 'forward<const auto &>' with deduced return type cannot be used before it is defined. 2 warnings and 3 errors generated when compiling for gfx942. → ninja: build stopped.
• [FACT] Only compile errors in the whole log are in activation.cuh (73× line 213, 73× line 217). No compile errors attributable to aiter or to the pgq kernel.
• [FACT] test_per_token_group_quant_8bit.py → ✗ TIMEOUT after 1200 seconds, 1287 FAILED / 0 PASSED.
• [FACT] Installed sglang = 0.0.0.dev1+ga7fd09207.d20260622; a7fd09207 committed 2026-06-16T03:28Z.
• [FACT] Fix PR #27947 (72d962be) merged 2026-06-16T06:44Z; title: "[AMD] Fix jit-kernel-unit-test-amd: activation.cuh ROCm build + per_token CUDA-only (R165)" — names this exact job and both failing test files.
• [FACT] Current main source (activation.cuh:215) already returns the explicit unary_kernel_fn_t with a comment documenting the clang-HIP gfx942 const-pointer issue.
• [FACT] Sister job in pr-test-amd run 27958083310 (current main, default aiter) = success.
• [FACT] aiter used here is the override a0320631 over Dockerfile default 7d604afe.

Hypothesised Causes

• [HIGH] The scout's pinned sglang base a7fd09207 predates fix 72d962be (#27947) and therefore compiles the old select_unary_kernel whose decltype(unary_kernel<...>) const-auto return type is rejected by clang-HIP on gfx942.
  • Supporting: code-level error is in sglang's activation.cuh; the fix PR explicitly names this job and both failing test files; tested SHA is ~3 h older than the fix merge; current main (with fix) passes the same job.
  • Disconfirming: (none found) — all three evidence chains (code, fix-PR scope, temporal, A/B baseline) converge.
  • How to verify: rebuild test/registered/jit/test_activation.py on AMD MI325 gfx942 against a7fd09207 vs 72d962be; the former fails, the latter passes.
• [HIGH] test_per_token_group_quant_8bit fails/times out on AMD because, in the stale base, it is not yet gated CUDA-only (also addressed by #27947's "per_token CUDA-only").
  • Supporting: 1287/1287 fail on AMD, 0 pass; #27947 makes the test CUDA-only; passes on current main.
  • Disconfirming: per-test tracebacks were not emitted before the 1200 s timeout, so the exact per-test error string is not in the log (only the FAILED roster + timeout are observable).
  • How to verify: confirm test_per_token_group_quant_8bit.py is skipped/CUDA-gated at 72d962be and runs unconditionally at a7fd09207.
• aiter override a0320631: not implicated — the failure is a host C++ compile error in sglang source, reproduces independently of aiter, and the sister baseline (default aiter) passes once the sglang fix is present.

In-flight Fix Check

• Fix already merged: ✅ #27947 (merged 2026-06-16) — fixes both failing test files. Not in-flight; already on main.
• Searched open PRs: is:pr is:open jit-kernel-unit-test-amd / activation.cuh — no additional open fix needed for the code; the residual issue is the scout base being stale, not a missing code fix.

Suggested Triage Steps

• Update the AMD AITER Scout's pinned sglang base (amd/aiter-ci branch / scout config) from a7fd09207 to a base that includes #27947 (72d962be or later); this clears both failures.
• Re-run jit-kernel-unit-test-amd after rebasing the scout base; expect parity with the green sister job 82731802397.
• Do not investigate the aiter override or open a new code fix — the activation.cuh / per_token fix is already on main.

Status

• Persistence: Reproduces on every scout run whose sglang base predates 2026-06-16 06:44Z (#27947 merge). Already resolved on main (sister job green as of 2026-06-22).
• Scope: This job only (jit-kernel-unit-test-amd, linux-mi325-1gpu-sglang); two test files. No impact on current main CI.
• Blocked work: None on main — purely a stale-base artifact of the AITER Scout pipeline; produces noise in scout signal until the base is bumped.
• In-flight fix: ✅ Fix merged in #27947; action is to refresh the scout's sglang base, not to write new code.

call-nightly-amd / nightly-8-gpu-mi35x-qwen3-235b-mxfp4 — failed step(s): Accuracy Test + Performance Test MI35x (8-GPU Qwen3-235B-MXFP4)

Commit Info

• sglang: bbc853df (fix(schedule_batch): trim stop string when EOS matches in the same step #28802)
• aiter: a0320631 (Tuned DSV4-Flash GEMM configs #3814) — source: AITER_COMMIT_OVERRIDE (Dockerfile default was 7d604afe5, forced rebuild)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/test_qwen3_instruct_mxfp4.py	TestQwen3Instruct2507MXFP4.setUpClass	RuntimeError: Rank 0 scheduler died during initialization (exit code: -6) → server exited code 1	Log

Failure Summary

The nightly-8-gpu-mi35x-qwen3-235b-mxfp4 job failed because the SGLang server for amd/Qwen3-235B-A22B-Instruct-2507-mxfp4 (tp=4, ep=2, --attention-backend aiter) crashed during startup. The Rank 0 scheduler died (exit code -6, SIGABRT) while initializing the decode CUDA graph; the watchdog reported a HIP error: operation not permitted on an event last recorded in a capturing stream (hipErrorCapturedEvent). setUpClass therefore aborted and 0 tests ran.

Regression Status

Known recurring failure (no green run found in the queryable window).

Recent history of test/registered/amd/test_qwen3_instruct_mxfp4.py (TestQwen3Instruct2507MXFP4.setUpClass) in job nightly-8-gpu-mi35x-qwen3-235b-mxfp4 (sister workflow nightly-test-amd.yml, branch main, event schedule, completed runs only):

Date	Run	Test File Status	Failed Function	Error
Jun 21	27912890405	❌ Failed	setUpClass	hipErrorCapturedEvent during graph capture
Jun 20	27879266774	❌ Failed	setUpClass	scheduler died (exit -6)
Jun 17	27710225107	❌ Failed	setUpClass	scheduler died (exit -6), same hipErrorCapturedEvent
Jun 15	27568949589	❌ Failed	setUpClass	scheduler died (exit -6)

(Runs on Jun 16/18/19 were cancelled — job did not complete — and are excluded. No passing completed run for this test exists in the ~8-run lookback window since 2026-06-15.)

Failure Origin

Test File	Test Function	Origin	Baseline Run	Baseline Status
test/registered/amd/test_qwen3_instruct_mxfp4.py	TestQwen3Instruct2507MXFP4.setUpClass	pre-existing (sglang)	27912890405	❌ Same failure in sister job (default aiter 7d604afe5, no override)

The scout's aiter override (a0320631) is exonerated: the sister nightly job ran the Dockerfile-default aiter and crashed on the same test with the same hipErrorCapturedEvent symptom and identical stack frames.

Failure Cluster

HIP error during decode CUDA graph capture (aiter custom all-reduce IPC meta gather)

Facts (verified observations)

• Server launch command: sglang serve --model-path amd/Qwen3-235B-A22B-Instruct-2507-mxfp4 --tp 4 --ep 2 --trust-remote-code --attention-backend aiter --device cuda (log line 6190).
• Crash exit code: -6 (SIGABRT) — RuntimeError: Rank 0 scheduler died during initialization (log line 6172). Not OOM (would be -9).
• Watchdog exception: HIP error: operation not permitted on an event last recorded in a capturing stream (hipErrorCapturedEvent) and operation not permitted when stream is capturing (hipErrorStreamCaptureUnsupported), raised in ProcessGroupNCCL.cpp:2055.
• Crash stack (scout, faulting thread): aiter/dist/device_communicators/custom_all_reduce.py:336 _gather_ipc_meta → :303 flush_graph_buffers → :505 capture → parallel_state.py:537/1664 graph_capture → decode_cuda_graph_runner.py:768 capture / :479 __init__ → model_runner.py:2937 init_decode_cuda_graph → tp_worker.py:337 → scheduler.py:844/862/432.
• AITER markers: scout forced rebuild to override a0320631 (log lines 4152-4155); sister baseline used default 7d604afe5 with no override (AITER version inside CI image: v0.1.14rc1.dev240+g7d604afe5).
• Reproducibility: same test + symptom in 4/4 completed nightly runs (06-15, 06-17, 06-20, 06-21) AND in today's scout — across two distinct aiter versions.

Hypothesised Causes (with confidence)

• [MEDIUM] Pre-existing incompatibility between the aiter custom all-reduce IPC-meta gather (custom_all_reduce.py: flush_graph_buffers → _gather_ipc_meta) and the decode CUDA-graph capture path on MI35x/gfx950 for this tp=4/ep=2 mxfp4 config: a HIP event/IPC operation is issued while the stream is in capture mode, which HIP forbids (hipErrorCapturedEvent).
  • Supporting: the faulting frame is the IPC-meta gather invoked from inside parallel_state.graph_capture; the error is specifically about event/stream ops during capture; reproduces identically across two aiter versions, so it is the interaction of this code path with capture, not a specific aiter delta.
  • Disconfirming: same crash occurs with both default and override aiter — so this is not introduced by either aiter commit; the exact sglang commit that first exposed it is not isolated (test has no green completed run in the window to bound pass_sha..fail_sha).
  • How to verify: re-run the test with --disable-custom-all-reduce (or --disable-cuda-graph) on AMD MI35x; if it passes, the custom-all-reduce-in-capture interaction is confirmed.
• [LOW] A specific sglang commit before 2026-06-15 introduced the regression.
  • Supporting: failure is sglang-side (baseline reproduces without override).
  • Disconfirming: no green completed run exists in the queryable window to anchor a bisect range; the job was frequently cancelled on intervening days.
  • How to verify: extend nightly history lookback before 2026-06-15 to find the last green run, then bisect pass_sha..fail_sha.

Aiter commits are deliberately NOT listed: Origin is pre-existing (sglang).

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for custom_all_reduce graph capture and qwen3 mxfp4; matches returned (#28482, #28291, #28700) are unrelated MXFP4 feature work, not a fix for this graph-capture crash.

Suggested Triage Steps

• Re-run test_qwen3_instruct_mxfp4.py on AMD MI35x with --disable-custom-all-reduce to confirm the custom-all-reduce-during-capture hypothesis; if green, the IPC-meta gather in capture is the trigger.
• If confirmed, investigate guarding flush_graph_buffers/_gather_ipc_meta so IPC/event ops are not issued inside an active CUDA graph capture (aiter custom_all_reduce.py ↔ sglang parallel_state.graph_capture ordering).
• Extend the nightly lookback before 2026-06-15 to locate the last green completed run for this test, then bisect the pass_sha..fail_sha sglang window.
• Track this as a nightly-test-amd failure (not aiter-scout-specific) since the sister baseline reproduces it with default aiter.

Status

• Persistence: Failing on every completed nightly-test-amd run since at least 2026-06-15 (4 completed runs) plus today's scout 2026-06-23; no passing completed run in the queryable window.
• Scope: nightly-8-gpu-mi35x-qwen3-235b-mxfp4 in both amd-aiter-scout.yml (this run) and the sister nightly-test-amd.yml. Blocks Qwen3-235B MXFP4 8-GPU MI35x signal.
• In-flight fix: ❌ none found in open PRs — needs new PR or investigation.

call-nightly-amd / nightly-accuracy-2-gpu — failed step(s): Nightly Test (2-GPU)

Commit Info

• sglang: bbc853df
• aiter: a0320631 (source: override — AITER_COMMIT_OVERRIDE; Dockerfile default was 7d604afe, forcibly rebuilt)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi30x/test_gsm8k_eval_amd.py	test_gsm8k_all_models (model=mistralai/Mistral-7B-Instruct-v0.3)	gsm8k score 0.363 < threshold 0.47	Log
test/registered/amd/accuracy/mi30x/test_gsm8k_eval_amd.py	test_gsm8k_all_models (model=mistralai/Mixtral-8x7B-Instruct-v0.1)	gsm8k score 0.292 < threshold 0.69	Log
test/registered/amd/accuracy/mi30x/test_gsm8k_eval_amd.py	test_gsm8k_all_models (model=neuralmagic/Mistral-7B-Instruct-v0.3-FP8)	gsm8k score 0.322 < threshold 0.46	Log
test/registered/amd/accuracy/mi30x/test_gsm8k_eval_amd.py	test_gsm8k_all_models (model=neuralmagic/Meta-Llama-3.1-70B-Instruct-FP8)	torch.OutOfMemoryError: HIP out of memory → server launch failed (Capture cuda graph failed) → FAILED (errors=1)	Log
test/registered/debug_utils/test_dumper.py	(whole file)	TIMEOUT after 7200s (estimated 60s)	Log

Failure Summary

The nightly-accuracy-2-gpu job (step 6, "Nightly Test (2-GPU)") failed with exit code 255. Within test_gsm8k_all_models, three models fell below their GSM8K accuracy thresholds (Mistral-7B-v0.3 → 0.363, Mixtral-8x7B → 0.292, Mistral-7B-v0.3-FP8 → 0.322), and a fourth model (Meta-Llama-3.1-70B-Instruct-FP8) raised an error because its server failed to launch with a HIP OOM during decode CUDA-graph capture. A second file, test_dumper.py, then timed out after 7200s (normally ~400s).

Regression Status

Known recurring failure for the three accuracy sub-cases; unique-so-far for the 70B-FP8 OOM.

Recent history of test_gsm8k_eval_amd.py (test_gsm8k_all_models) — compared against the sister nightly-test-amd.yml nightly-accuracy-2-gpu job (same branch, event=schedule, completed runs only; the scout cron itself runs only Mon/Thu so sister-nightly is the queryable daily baseline):

Date	Run	Job	Mistral-7B / Mixtral / Mistral-FP8	70B-FP8
Jun 22 (scout)	27983814629	scout nightly-accuracy-2-gpu	❌ 0.363 / 0.292 / 0.322	❌ HIP OOM
Jun 21	27912890405	nightly nightly-accuracy-2-gpu	❌ 0.363 / 0.289 / 0.326	✅ 0.910
Jun 20	27879266774	nightly nightly-accuracy-2-gpu	❌ 0.367 (Mistral-7B); run cancelled before reaching Mixtral/FP8/70B	— (not reached)

The three Mistral/Mixtral accuracy failures reproduce identically in the default-aiter nightly baseline (≥2 days, scores match to ~±0.01). The 70B-FP8 OOM does not appear in the Jun-21 baseline (passed at 0.910) and the Jun-20 run was cancelled before reaching it, so it is so-far unique to this scout run.

Failure Origin

Test File	Test Function (model)	Origin	Baseline Run	Baseline Status
test_gsm8k_eval_amd.py	Mistral-7B-Instruct-v0.3	pre-existing (sglang)	27912890405	❌ Same failure, default aiter (~0.36)
test_gsm8k_eval_amd.py	Mixtral-8x7B-Instruct-v0.1	pre-existing (sglang)	27912890405	❌ Same failure, default aiter (~0.29)
test_gsm8k_eval_amd.py	Mistral-7B-Instruct-v0.3-FP8	pre-existing (sglang)	27912890405	❌ Same failure, default aiter (~0.33)
test_gsm8k_eval_amd.py	Meta-Llama-3.1-70B-Instruct-FP8	aiter-caused (LOW — likely flaky OOM)	27912890405	✅ Passed, default aiter (0.910)
test_dumper.py	(whole file)	unclear (likely downstream of the OOM crash)	27912890405	✅ Passed, default aiter (397s)

Failure Cluster

Two clusters: (1) GSM8K accuracy below threshold for Mistral/Mixtral models on AMD; (2) HIP OOM during decode CUDA-graph capture (70B-FP8).

Facts (verified observations)

• [FACT] Three accuracy sub-cases failed: Mistral-7B-v0.3 = 0.363 (thr 0.47), Mixtral-8x7B = 0.292 (thr 0.69), Mistral-7B-v0.3-FP8 = 0.322 (thr 0.46). Each was retried 3× with stable low scores (no recovery).
• [FACT] 70B-FP8 server launch raised torch.OutOfMemoryError: HIP out of memory. Tried to allocate 38.00 MiB. GPU 0 ... 0 bytes free ... 247.70 GiB allocated by PyTorch at decode_cuda_graph_runner.py:479 → capture() → logits_processor.py:859 _get_logits → tensor_model_parallel_all_gather (parallel_state.py:988 torch.empty). mem_fraction_static=0.76415.
• [FACT] aiter ran as override a0320631 ([CI-AITER-CHECK] AITER_COMMIT_OVERRIDE=... → forcing rebuild), vs Dockerfile default 7d604afe.
• [FACT] Sister nightly baseline 27912890405 (default aiter): Mistral/Mixtral/Mistral-FP8 fail with the same scores; 70B-FP8 passed at 0.910; test_dumper.py passed in 397s.
• [FACT] After the OOM Received sigquit from a child process, test_dumper.py started but timed out at 7200s vs its 397s baseline.

Hypothesised Causes

• [HIGH] The three Mistral/Mixtral accuracy failures are pre-existing in sglang, unrelated to aiter. Baseline A/B: same test file + same model sub-cases fail with the same scores in the default-aiter nightly run on Jun-21, and Mistral-7B fails on Jun-20 too.
  • Supporting: scout 0.363/0.292/0.322 vs baseline 0.363/0.289/0.326 — essentially identical; failure predates the aiter override.
  • Disconfirming: (none found) — baseline reproduces without the override.
  • How to verify: already verified via the sister baseline.
• [LOW] 70B-FP8 OOM is aiter-related OR a flaky memory-fragmentation event. Per A/B it passes in the default-aiter baseline (0.910) and differs only by the aiter override, so the override is a candidate (e.g. larger aiter kernel/workspace footprint pushing a mem_fraction_static=0.764 config over the edge during graph capture).
  • Supporting: only differing variable vs baseline is the aiter override; baseline passed cleanly.
  • Disconfirming: OOM during CUDA-graph capture at "0 bytes free" is a classic single-shot fragmentation symptom; only one occurrence, only one model, fresh per-model server launch — consistent with flakiness rather than a deterministic aiter regression. No code-level evidence linking a0320631 to memory growth was gathered.
  • How to verify: re-run the scout (or run 70B-FP8 with AITER_COMMIT_OVERRIDE=a0320631... vs default) on MI30x; if it OOMs deterministically with the override only, promote to MEDIUM. Otherwise lowering --mem-fraction-static would confirm a headroom issue.
• [LOW] test_dumper.py timeout is downstream of the 70B-FP8 crash, not an independent regression — the preceding sigquit/kill-process-tree may have left the 2-GPU device in a bad state; it passes in the baseline where no OOM occurred.
  • Disconfirming: not proven the GPU state carried over across the file boundary.
  • How to verify: re-run test_dumper.py standalone on MI30x.

In-flight Fix Check

• Accuracy cluster — In-flight fix: ✅ [#27757](https://github.com/sgl-project/sglang/pull/27757) "Fix Mistral GSM8K chat eval" (open since 2026-06-10) — directly targets Mistral/Mixtral GSM8K eval (keeps Mistral/Mixtral off FA3 auto-default, makes the chat-eval prompt explicit, plumbs stop sequences). Chase reviewers; do NOT open a duplicate. (Searched repo:sgl-project/sglang is:pr is:open Mistral gsm8k.)
• 70B-FP8 OOM — In-flight fix: ❌ none found (searched gsm8k amd accuracy, Mixtral accuracy amd) — needs reproduction first before any fix.

Suggested Triage Steps

• For the accuracy cluster: verify #27757 addresses all three (Mistral-7B-v0.3, Mixtral-8x7B, Mistral-7B-v0.3-FP8) and re-run test_gsm8k_eval_amd.py on MI30x; these are pre-existing sglang eval issues, not aiter.
• For the 70B-FP8 OOM: re-run the scout job to check determinism; if it recurs only with the aiter override, A/B AITER_COMMIT_OVERRIDE=a0320631... vs default on a single MI30x to confirm aiter-causation; otherwise treat as flaky and/or lower --mem-fraction-static.
• For test_dumper.py: re-run standalone after a clean VRAM reset to confirm the timeout was a downstream artifact of the OOM crash, not a real hang.

Status

• Persistence: Mistral/Mixtral/Mistral-FP8 GSM8K accuracy failures recurring ≥2 days (Jun-21 nightly + Jun-22 scout; Mistral-7B also Jun-20). 70B-FP8 OOM observed only in this scout run.
• Scope: nightly-accuracy-2-gpu (scout + sister nightly) on AMD MI30x. The accuracy failures are AMD-side and aiter-independent (reproduce with default aiter).
• Blocked work: blocks the AMD nightly GSM8K accuracy signal for Mistral/Mixtral.
• In-flight fix: accuracy → ✅ #27757 (open since 2026-06-10); 70B-FP8 OOM → ❌ none.

call-nightly-amd-rocm720 / nightly-accuracy-8-gpu-mi35x-deepseek-v32-rocm720 — failed step(s): Accuracy Test MI35x ROCm 7.2 (8-GPU DeepSeek-V3.2)

Commit Info

• sglang: bbc853df
• aiter: a0320631 (source: override via AITER_COMMIT_OVERRIDE; Dockerfile default was 7d604afe)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi35x/test_deepseek_v32_eval_mi35x.py	test_deepseek_v32_accuracy	AssertionError: Failed models: ['deepseek-ai/DeepSeek-V3.2 (basic)'] — accuracy=0.015 threshold=0.93	Log

Failure Summary

The single GSM8K accuracy test for DeepSeek-V3.2 (basic) on MI35x scored 0.015 against a 0.93 threshold. The server itself ran healthily — all 200 /generate requests returned 200 OK at ~880 tok/s and the eval completed all 200 samples — so this is a numerical/output-correctness failure (essentially garbage generations), not a crash. (An earlier py-spy dump on TP2 failed but is a watchdog diagnostic artifact, not the cause.)

Regression Status

Never-passed (no green run found) + bimodal/flaky. This test has not met its 0.93 threshold in the entire queryable window of the sister workflow (nightly-test-amd-rocm720.yml, schedule, main), with failure/cancelled conclusions going back to ~2026-05-26. The accuracy oscillates between a catastrophic regime (~0.01–0.025) and a near-pass regime (~0.87) across days, with no monotonic regression.

Recent history of test_deepseek_v32_accuracy in the sister job nightly-accuracy-8-gpu-mi35x-deepseek-v32-rocm720 (nightly-test-amd-rocm720.yml, default aiter, schedule, main, completed runs only):

Date	Run	Test File Status	Accuracy	Error
Jun 21	run	❌ Failed	0.870	below threshold (near-pass)
Jun 20	run	❌ Failed	0.865	below threshold (near-pass)
Jun 15	run	❌ Failed	0.010	catastrophic
Jun 14	run	❌ Failed	0.020	catastrophic
Jun 13	run	❌ Failed	0.025	catastrophic

(Scout run itself: 0.015, catastrophic — same regime as the Jun 13–15 default-aiter runs.)

Failure Origin

pre-existing (sglang) — NOT aiter-caused.

Test File	Test Function	Origin	Baseline Run	Baseline Status
test/registered/amd/accuracy/mi35x/test_deepseek_v32_eval_mi35x.py	test_deepseek_v32_accuracy	pre-existing (sglang)	run	❌ Same test fails in sister job (default aiter); catastrophic mode also reproduced Jun 13-15 without any aiter override

Baseline A/B verdict: The aiter override is exonerated. A naive comparison against only the most-recent baseline (Jun 21, 0.870) would suggest the override drove a catastrophic 0.870→0.015 collapse and look "aiter-caused" — but the full sister history shows the identical catastrophic ~0.01 accuracy with the Dockerfile-default aiter on Jun 13-15. The override is therefore not required to produce the 0.015 result.

Failure Cluster

DeepSeek-V3.2 MI35x GSM8K accuracy below threshold (bimodal: catastrophic ~0.01 vs near-pass ~0.87)

Facts (verified observations)

• Scout log: accuracy=0.015 threshold=0.93 ❌ FAIL → AssertionError: Failed models: ['deepseek-ai/DeepSeek-V3.2 (basic)'] (scout job).
• Server healthy during eval: 200/200 samples completed, /generate returned 200 OK, decode throughput ~880 tok/s; no crash/OOM/memory-fault in the log.
• aiter in scout = override a0320631 (forced rebuild, verified [CI-AITER-CHECK]); aiter in sister baseline = default 7d604afe (dev image, rebuild skipped).
• Sister (default-aiter) history shows the same catastrophic regime: 0.025 (Jun 13) / 0.020 (Jun 14) / 0.010 (Jun 15), and near-pass 0.865 (Jun 20) / 0.870 (Jun 21).
• Config bar: accuracy_threshold=0.93 for the basic variant (test_deepseek_v32_eval_mi35x.py:71).
• sglang delta baseline→scout is 49 commits, several touching DSA / piecewise CUDA graph (e.g. 6b2c730b "Fix DSA indexer in prefill piecewise CUDA graph" #28644); DeepSeek-V3.2 runs attention_backend='dsa'.

Hypothesised Causes (with confidence)

• [LOW] Non-deterministic DSA/NSA correctness issue on MI35x (gfx950) causing the model to intermittently collapse into garbage output (~0.01 accuracy).
  • Supporting: accuracy is bimodal across days, flipping between ~0.01 and ~0.87 with no monotonic trend; server runs cleanly each time, so it's an output-correctness (not stability) problem.
  • Disconfirming: no single sglang commit correlates with the flips; cannot rule out an environmental/seed/runner factor.
  • How to verify: re-run the sister job 3-5× on the same sglang SHA to measure the catastrophic-vs-near-pass hit rate; if bimodal under fixed code, it's nondeterminism, not a code regression.
• [LOW] aiter override a0320631 caused the catastrophic accuracy.
  • Supporting: scout (override) = 0.015 vs immediately-preceding baseline (default) = 0.870.
  • Disconfirming (strong): the same catastrophic ~0.01 accuracy occurred with default aiter on Jun 13-15 — the override is not necessary to reproduce 0.015. This hypothesis is effectively refuted.
  • How to verify: not actionable; baseline already reproduces without the override.
• [LOW] A DSA-path sglang commit in the 49-commit delta (e.g. 6b2c730b) shifts accuracy.
  • Supporting: DeepSeek-V3.2 uses the DSA backend; several delta commits touch DSA / piecewise CUDA graph.
  • Disconfirming: the catastrophic mode predates these commits (Jun 13-15 baselines), so they cannot be the originating cause.

No hypothesis above LOW confidence; the dominant signal is a long-standing Never-passed + bimodal test, not a fresh regression.

In-flight Fix Check

• In-flight fix: ⚠️ candidate #25559 "[AMD] WIP - Deepseek v3.2 : disable preshuffle for nsa path" — open WIP touching the AMD DeepSeek-V3.2 NSA path; potentially relevant to MI35x accuracy. Not confirmed to target this test. (Searched repo:sgl-project/sglang is:pr is:open deepseek v3.2 accuracy.)
• No open PR found that explicitly fixes this MI35x accuracy test.

Suggested Triage Steps

• Re-run the sister job nightly-accuracy-8-gpu-mi35x-deepseek-v32-rocm720 3-5× at a fixed sglang SHA to quantify the catastrophic-vs-near-pass rate and confirm nondeterminism vs a code regression.
• If deterministic-per-SHA, bisect the catastrophic↔near-pass flip across the schedule history around Jun 15→Jun 20 (when accuracy jumped 0.010→0.865) and Jun 21→Jun 22 in the sister (default-aiter) workflow to isolate the responsible sglang commit, focusing on DSA/NSA and piecewise-CUDA-graph changes.
• Track WIP #25559 (NSA preshuffle) to see if it stabilizes the catastrophic mode.
• Do NOT pursue aiter a0320631 as a cause — the default-aiter baseline reproduces the catastrophic accuracy.

Status

• Persistence: Never-passed in the queryable window — sister workflow failure/cancelled back to ~2026-05-26; catastrophic accuracy mode observed Jun 13-15 and again in this scout (Jun 22).
• Scope: test_deepseek_v32_accuracy (DeepSeek-V3.2 basic, MI35x) in both amd-aiter-scout.yml and the sister nightly-test-amd-rocm720.yml.
• Blocked work: DeepSeek-V3.2 accuracy CI signal on MI35x (ROCm 7.2) is unreliable (bimodal, never green).
• In-flight fix: ⚠️ candidate WIP #25559; no confirmed fix PR.

call-nightly-amd-rocm720 / nightly-8-gpu-minimax-m27-rocm720 — failed step(s): Accuracy Test ROCm 7.2 (8-GPU MiniMax-M2.7)

Commit Info

• sglang: bbc853df
• aiter: a0320631 (source: AITER_COMMIT_OVERRIDE — scout forced rebuild; Dockerfile default was 7d604afe)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi30x/test_minimax_m27_eval_amd.py	test_minimax_m27_accuracy	AssertionError: Failed models: ['MiniMaxAI/MiniMax-M2.7 (TP8+EP8)'] — GSM8K accuracy=0.915 threshold=0.93 FAIL	Log

Failure Summary

The 8-GPU MiniMax-M2.7 (TP8+EP8) GSM8K accuracy eval scored 0.915 against a 0.93 threshold and asserted failure. The benchmark completed all 200 questions and the server served every request (200 OK); the failure is purely the accuracy gate, not a crash. A soft TokenizerManager watchdog warning and "detokenizer 20s" health-check warnings appeared during the run but did not abort it — the eval ran to completion.

Regression Status

Known recurring failure (borderline accuracy drifting through the 0.93 gate). The same test on the sister workflow nightly-test-amd-rocm720.yml (default aiter) shows a steady downward drift across runs:

Recent history of test_minimax_m27_eval_amd.py (test_minimax_m27_accuracy), sister workflow nightly-test-amd-rocm720.yml, event=schedule, branch main, completed runs only:

Date	Run	Job	Test File Status	Accuracy	Error
Jun 22 (scout)	27983814629	nightly-8-gpu-minimax-m27-rocm720 (this)	❌ Failed	0.915	below 0.93
Jun 21	27912828989	nightly-8-gpu-minimax-m27-rocm720	❌ Failed	0.920	below 0.93
Jun 20	27879239829	nightly-8-gpu-minimax-m27-rocm720	✅ Passed	0.935	—
Jun 15	27568866028	nightly-8-gpu-minimax-m27-rocm720	✅ Passed	0.945	—

First observed FAIL: 2026-06-21 (sister). Last known PASS: 2026-06-20 (acc 0.935). Jun 16–19 sister runs were cancelled (test did not produce a result) and are excluded.

Failure Origin

Test File	Test Function	Origin	Baseline Run	Baseline Status
test/registered/amd/accuracy/mi30x/test_minimax_m27_eval_amd.py	test_minimax_m27_accuracy	pre-existing (sglang)	27912828989	❌ Same failure (default aiter, acc 0.920 < 0.93)

Baseline A/B result: aiter exonerated. The sister nightly-test-amd-rocm720.yml run on 2026-06-21 ran the identical test with the Dockerfile-default aiter (no override) and failed the same way (accuracy=0.920 threshold=0.93, same AssertionError). The scout's aiter override a0320631 is therefore not the cause. Per the A/B rule, no aiter commits are cited below.

Failure Cluster

MiniMax-M2.7 GSM8K accuracy below 0.93 threshold (borderline accuracy gate).

Facts (verified observations)

• [FACT] Scout: accuracy=0.915 threshold=0.93 FAIL (log line 7898); Ran 1 test in 830.776s, FAILED (failures=1), job exit code 255.
• [FACT] Sister baseline (default aiter): accuracy=0.920 threshold=0.93 FAIL, same AssertionError (run 27912828989, log line 6452).
• [FACT] Accuracy sequence on the sister (default aiter): 0.945 (Jun 15) → 0.935 (Jun 20) → 0.920 (Jun 21) → and scout 0.915 (Jun 22).
• [FACT] Eval is GSM8K, num_questions=200, num_shots=5 (test_minimax_m27_eval_amd.py:118-119); threshold 0.93 (:67). At n=200, each question = 0.5%, so 0.93 = 186/200 and the failing runs missed by 2–3 questions.
• [FACT] Server random_seed differs per run (scout 669906163, sister 767479822), so sampling/decode variance differs run-to-run.
• [FACT] Both runs emitted "detokenizer 20s" health-check warnings and a soft watchdog warning, but completed all 200 generations with 200 OK.

Hypothesised Causes

• [MEDIUM] Borderline accuracy gate vs run-to-run sampling variance. Accuracy sits 1.5–4.5% from threshold and differs by only 2–3 GSM8K questions; with only 200 questions and a per-run random seed, ±1–2% jitter is expected, putting 0.93 inside the noise band.
  • Supporting: small n (200), per-run seed differences, the 0.945→0.915 spread spans only ~6 questions.
  • Disconfirming: the sequence is monotonically decreasing (0.945→0.935→0.920→0.915), which pure noise would not usually produce — a small genuine numeric drift cannot be ruled out.
  • How to verify: re-run the eval 3–5× on the same head SHA with GSM8K_NUM_QUESTIONS raised (e.g. 1000) to shrink the confidence interval and see whether accuracy stabilizes above or below 0.93.
• [LOW] Small numeric regression in sglang between Jun 20 (pass) and Jun 21 (fail). Regression window is sister pass_sha ff1fc1fb..fail_sha 7f67965b (24 commits).
  • Supporting: pass→fail boundary aligns with that window; accuracy stepped from 0.935 to 0.920.
  • Disconfirming: none of the 24 commits in the window grep-match minimax / moe / fp8 / rope / attention / quant code paths; the drift also continued into the scout (0.915) on a later SHA, consistent with noise rather than one commit.
  • How to verify: bisect ff1fc1fb..7f67965b, re-running the eval at each step (noisy — average several runs per step).

In-flight Fix Check

• In-flight fix: ❌ none found — no open PR targets the MiniMax-M2.7 GSM8K accuracy threshold. Searched is:pr is:open minimax accuracy (36 unrelated hits) and is:pr is:open MiniMax-M2.7 (12 hits).
• Tangentially related (not an accuracy-gate fix): #27609 "Disable MiniMax-M2.7 custom all-reduce on gfx942" — this job runs on gfx942 (mi30x) with AiterCustomAllreduce, so it could shift numerics; worth watching but [LOW] relevance.

Suggested Triage Steps

• Re-run test_minimax_m27_eval_amd.py 3–5× on bbc853df (container rocm/sgl-dev:v0.5.13.post1-rocm720-mi30x-20260622, default aiter) to measure the accuracy distribution and decide whether this is noise or a real drift.
• If noise: consider raising GSM8K_NUM_QUESTIONS or lowering the 0.93 threshold to a value with margin against observed variance (maintainer decision).
• If the monotonic decline holds across reruns: bisect ff1fc1fb..7f67965b, averaging multiple eval runs per commit.
• Optionally A/B #27609 (disable custom all-reduce on gfx942) to see whether it moves accuracy.

Status

• Persistence: Failing since 2026-06-21 on the sister rocm720 nightly (default aiter); reproduced in this 2026-06-22 scout run. Last pass 2026-06-20 (acc 0.935).
• Scope: nightly-8-gpu-minimax-m27-rocm720 in both amd-aiter-scout.yml and nightly-test-amd-rocm720.yml. mi30x/gfx942, TP8+EP8.
• Blocked work: MiniMax-M2.7 8-GPU accuracy signal on AMD rocm720.
• In-flight fix: ❌ none found — needs rerun-for-variance assessment or threshold/bisect investigation.

call-nightly-amd / nightly-8-gpu-mi35x-kimi-k26 — failed step(s): Accuracy Test MI35x (8-GPU Kimi-K2.6)

Commit Info

• sglang: bbc853df
• aiter: a0320631 (source: AITER_COMMIT_OVERRIDE — forced rebuild; Dockerfile default was 7d604afe)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi35x/test_kimi_k26_eval_mi35x.py	TestKimiK26EvalMI35x.test_kimi_k26_gsm8k_accuracy	TimeoutError: Server failed to start within the timeout period (both attempts)	Log

Failure Summary

The single test test_kimi_k26_gsm8k_accuracy (model moonshotai/Kimi-K2.6, TP=8 on MI35x) failed because the sglang server never finished starting. py-spy dumps show all TP schedulers stuck inside HuggingFace snapshot_download → download_weights_from_hf (xet_get / hf_hub_download) loading the Kimi-K2.6 weight shards. The soft scheduler watchdog (watchdog_timeout=300) fired repeatedly throughout, and after popen_launch_server exhausted its timeout (the per-file budget was --timeout-per-file 7200), both the initial attempt and the one retry raised TimeoutError.

Regression Status

Known recurring failure — this test has failed in every recent completed nightly run, not just the scout.

Recent history of test_kimi_k26_eval_mi35x.py (test_kimi_k26_gsm8k_accuracy) in the sister job nightly-8-gpu-mi35x-kimi-k26 (nightly-test-amd.yml, event=schedule, branch=main, completed runs only):

Date	Run	Job	Test File Status	Failed Function	Error
Jun 22	27983814629 (scout)	nightly-8-gpu-mi35x-kimi-k26	❌ Failed	test_kimi_k26_gsm8k_accuracy	TimeoutError: Server failed to start
Jun 21	27912890405	nightly-8-gpu-mi35x-kimi-k26	❌ Failed	test_kimi_k26_gsm8k_accuracy	Server process exited with code -9 (during weight load)
Jun 20	27879266774	nightly-8-gpu-mi35x-kimi-k26	❌ Failed	test_kimi_k26_gsm8k_accuracy	server start failure

No passing run found for this test in the queryable window (Jun 20–22). Lookback limited to the last completed scheduled nightly runs; mark as effectively Never-passed in window.

Failure Origin

Test File	Test Function	Origin	Baseline Run	Baseline Status
test/registered/amd/accuracy/mi35x/test_kimi_k26_eval_mi35x.py	test_kimi_k26_gsm8k_accuracy	pre-existing (sglang)	27912890405	❌ Same test fails in sister job (default aiter) — server fails to start during Kimi-K2.6 weight load

A/B check: the sister nightly-test-amd.yml job ran with the Dockerfile-default aiter (no override) and the same test failed in the same stage (server never starts during download_weights_from_hf). The aiter override a0320631 is therefore exonerated — zero aiter commits are cited below.

Failure Cluster

Kimi-K2.6 server fails to start during HF weight download/load (server-launch timeout)

Facts (verified observations, no interpretation)

• Server args: model_path='moonshotai/Kimi-K2.6', tp_size=8, watchdog_timeout=1200.0, soft_watchdog_timeout=300, enable_multithread_load: true; test launched via run_suite.py … --timeout-per-file 7200.
• py-spy MainThread stack (scout, TP4): snapshot_download (_snapshot_download.py:467) → ci_download_with_validation_and_retry (ci_weight_validation.py:1955) → download_weights_from_hf (weight_utils.py:595) → stream_language_weights (kimi_k25.py:780). Worker threads stuck in xet_get (file_download.py:566) / _download_to_tmp_and_move.
• Tokenizer/config files were cache-hit (models--moonshotai--Kimi-K2.6/snapshots/7eb5002f…) but weight shards were being downloaded online (CI_OFFLINE: Cache validation failed … will use online mode).
• Scheduler watchdog timeout (self.watchdog_timeout=300, self.soft=True) fired on every TP rank repeatedly from ~02:02 through ~03:17.
• Both attempts failed: initial + 1 retry. Final: Test Summary: 0/1 passed, exit code 255.
• Sister baseline (default aiter, Jun 21): same test, same weight-load stage, terminated as Server process exited with code -9 after elapsed=1326. Same launch command (sglang serve --model-path moonshotai/Kimi-K2.6 --tp 8 …).

Hypothesised Causes (with confidence, with disconfirming evidence)

• [HIGH] Infrastructure / model-fetch issue: the Kimi-K2.6 weight shards are downloaded online during the test and the download is too slow / stalls, so the server never starts before the launch timeout.
  • Supporting: py-spy shows every rank parked in snapshot_download/xet_get; config was cache-hit but weights were online (Cache validation failed … online mode); reproduces identically in the sister baseline with a different aiter; failure is at startup, never reaching inference.
  • Disconfirming: the sister baseline died with exit -9 (SIGKILL — could indicate an OOM kill rather than a pure download stall), whereas the scout hit a clean TimeoutError. The terminal signal differs, so a host memory/OOM component during weight load cannot be fully ruled out.
  • How to verify: pre-stage moonshotai/Kimi-K2.6 weights into /sgl-data/hf-cache and re-run with HF_HUB_OFFLINE=1; if it then starts, the cause is the online download. Separately, watch host RSS during load to check for the -9 OOM path.
• [LOW] aiter override a0320631 involvement.
  • Supporting: none beyond it being a scout run.
  • Disconfirming: sister baseline with default aiter fails the same test at the same stage — the override is not required to reproduce.
  • How to verify: not actionable; already disconfirmed by the A/B baseline.

In-flight Fix Check

• In-flight fix: ❌ none found for the Kimi-K2.6 server-launch/weight-download failure. Searched open PRs for kimi-k26 (3 hits: #27976, #28159, #27209) and kimi download (6 hits) — none address server startup or weight fetching for this test. Note an in-progress branch bingxche/fix-kimi-k26-w2-scale exists but targets w2-scale, not the startup timeout.

Suggested Triage Steps

• Confirm whether the MI35x runner has moonshotai/Kimi-K2.6 weights pre-cached; if not, pre-stage them and re-run test_kimi_k26_eval_mi35x.py with HF_HUB_OFFLINE=1 to remove the online-download variable.
• If the host gets SIGKILLed (the sister -9 path), capture host memory during weight load to check for an OOM during the multithread load (enable_multithread_load: true).
• If download bandwidth is the bottleneck, consider raising the per-launch server timeout for this very large model or gating the test behind a cache-warm step.
• Because this fails identically on default aiter, do NOT pursue any aiter-side change for this failure.

Status (factual, no priority assignment)

• Persistence: Failing on every completed nightly nightly-8-gpu-mi35x-kimi-k26 run in the queryable window (Jun 20, 21) and on the Jun 22 scout — no green run found.
• Scope: nightly-8-gpu-mi35x-kimi-k26 across amd-aiter-scout.yml (this job) and nightly-test-amd.yml (sister). The sibling nightly-8-gpu-kimi-k26 job also failed in the same baseline runs.
• Blocked work: Kimi-K2.6 accuracy signal on MI35x is unavailable.
• In-flight fix: ❌ none found in open PRs — needs investigation (likely infra/cache, not code).

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-06-23 12:59 UTC)

[amd-bot]

pr-test-amd.yml — 28 failure(s)

Run started (UTC): 2026-06-22 01:00 UTC
Last scanned (UTC): 2026-06-23 20:58 UTC
Commits: sglang e0dc8b7

Summary

Counts: 28 failing jobs · 13 clusters · 2 clusters ≥3 days (flux2, ideogram4) · ~9 first seen Jun 22–23

#	Cluster	Job	Test File	Test Function	Status	Hypothesis (confidence)
1	flux2 FP8 HIPBLAS unsupported	multimodal-gen-test-2-gpu (1)	…/test_server_2_gpu.py	test_diffusion_generation[flux2_modelopt_fp8_tp2_t2i]	8 runs, Never-passed in window (≥3d)	[HIGH] hipBLAS rejects fp8 e4m3fn _scaled_mm in apply_fp8_linear (fp8_utils.py:1695)
2	ideogram4 gated HF 403	multimodal-gen-test-2-gpu (1)	…/test_server_2_gpu.py	test_diffusion_generation[ideogram4_fp8_tp2_t2i]	8 runs, recurring (access)	[FACT] CI HF token lacks access to gated ideogram-ai/ideogram-4-fp8
3	Mooncake PD KV-transfer fail (MI35x)	stage-b…mi35x-disaggregation	…/disaggregation/test_disaggregation_basic.py (+_pp.py)	TestDisaggregationAccuracy.test_gsm8k	5 runs since Jun 22 19:10	[MEDIUM] RDMA/mooncake ENOMEM·QP-teardown segfault·300s WaitingForInput timeout on .fabric runner
4	anthropic httpx socket_options	stage-b-test-1-gpu-small (13)	…/openai_server/basic/test_anthropic_server.py	test_in_messages_system_role	New since PR #26773 (Jun 21), Never-passed on AMD	[HIGH] anthropic 0.111.0 needs httpx≥0.25; image ships httpx 0.23.3 (pinned by lmms_eval==0.4.1)
5	1-GPU diffusion runner loss (no log)	multimodal-gen-test-1-gpu (1)	test_server_1_gpu.py (1-gpu part-1)	N/A (runner died, blob purged)	Flaky 3/7 runs	[MEDIUM] host OOM / GPU hang / pod eviction on MI325 runner
6	pr-test-amd-extra fast-fail cancel	extra-a-test-1-gpu-small	N/A (cancelled in mori build)	N/A	[IN-FLIGHT] parent run, cascade cancel	[HIGH] fast-fail cascade cancelled still-running extra jobs
7	Perf-suite 45-min job timeout	stage-b-test-1-gpu-large (1)	test_bench_serving_1gpu_part2.py	test_score_api_latency_throughput	New regression (1 run); p95 56.7ms > 50	[FACT] suite elapsed 2057s vs 1452s last pass; 13-commit window
8	Scheduler port 5555 strict-ports	multimodal-gen-test-2-gpu (0)	…/test_server_2_gpu.py	test_diffusion_generation[zimage_image_t2i_2_gpus] (+3)	New Jun 23	[HIGH] stale process holds TCP 5555 on runner → --strict-ports fast-fail
9	Kimi-K2 server-launch timeout	stage-c-test-large-8-gpu (2)	…/amd/test_kimi_k2_instruct.py	TestKimiK2Instruct0905.setUpClass	New Jun 23 (1/4)	[MEDIUM] cold HF cache; weight download exceeds SERVER_LAUNCH_TIMEOUT
10	RCCL allreduce SIGSEGV	stage-c-test-large-8-gpu (0)	scripts/ci/amd/test_rccl_multi_gpu.py	test_rccl_allreduce	New (1 run); gates+skips real tests	[MEDIUM] RCCL/driver IMA on 8gpu-sglang node
11	8-GPU runner abrupt termination	stage-c-test-large-8-gpu (1)	Unrecoverable (BlobNotFound)	Unrecoverable	New (1 run), no log	[LOW] runner/process loss; no log to attribute
12	mori build corrupt libabsl ELF	stage-a-test-1-gpu-small	N/A (dependency build)	N/A	New Jun 23 (1 run)	[MEDIUM] corrupt libabsl_base.so.20210324 (invalid ELF) on this runner
13	FLUX.1-dev generate 90-min hang	multimodal-gen-test-1-gpu (3)	…/cli/test_generate_t2i_perf.py	TestFlux_T2V::test_single_gpu	New Jun 23 (1 run)	[LOW] sglang generate hangs with zero output before model load

Cluster: flux2 FP8 HIPBLAS unsupported

• Affected: 8 multimodal-gen-test-2-gpu-amd shard-1 runs (jobs 82731802132, 82667263632, 82620731555, 82798487115, 82853563979, 82898649860, 82961543292, 83034452768).
• Shared facts: identical stack modelopt_quant.apply → apply_fp8_linear → torch._scaled_mm (fp8_utils.py:1695) on gfx942/MI325; Never-passed in lookback (≥2.5 days, ≥8 runs).
• Top hypothesis: [HIGH] hipBLAS lacks per-tensor fp8 e4m3fn GEMM support. Disconfirming: not bisected to an introducing commit — may be an unported path, not a fresh regression.
• In-flight fix: ✅ #28889 (open Jun 22, mergeable_state: blocked) — bf16 dequant fallback in apply_fp8_linear. Chase reviewers; do NOT duplicate.
• Suggested triage: unblock/merge #28889 and re-run shard 1.

Cluster: ideogram4 gated HF 403

• Affected: same 8 multimodal-2gpu shard-1 jobs as flux2.
• Shared facts: huggingface_hub.errors.GatedRepoError: 403 for ideogram-ai/ideogram-4-fp8 at server startup — access/infra, not a code defect.
• In-flight fix: ❌ none (needs HF token provisioning or scenario gating); #28225 is a draft NF4-routing change.
• Suggested triage: provision HF access for the gated repo or gate the scenario off in AMD CI.

Cluster: Mooncake PD KV-transfer fail (MI35x)

• Affected: 5 stage-b…mi35x-disaggregation-amd runs (82798487481, 82853564249, 82898650217, 82961543942, 83034453255). Same test_disaggregation_basic.py::test_gsm8k (200/200 KV-transfer timeouts) with _pp.py::setUpClass as a memory-imbalance cascade.
• Shared facts: only the mooncake PD path fails; test_mori_transfer_engine_e2e.py (mori engine) mostly passes. Symptom varies: ENOMEM (errno 12), 300s KVPoll.WaitingForInput, ibv_destroy_qp segfault — all on linux-mi35x-gpu-8.fabric.
• Top hypothesis: [MEDIUM] RDMA fabric / pinned-memory exhaustion + QP teardown instability. Disconfirming: memory-balance guard (model_runner.py:1279) predates the regression window — it trips as a downstream cascade, not a cause.
• In-flight fix: ⚠️ #27012 (revert PR, exact job-name match) and ⚠️ #26326 (mooncake-transfer-engine 0.3.11.post1 bump) — review before duplicating.
• Suggested triage: assess #27012/#26326; bisect basic-test window 4923bb93ae..adc203df; check .fabric runner RDMA/dmesg.

Cluster: anthropic httpx socket_options

• Affected: 3 stage-b-test-1-gpu-small (13) runs (82731803181, 82667264501, 82620731883).
• Shared facts: [FACT] TypeError: HTTPTransport.__init__() got an unexpected keyword argument 'socket_options' from anthropic/_base_client.py:943; install log shows anthropic 0.111.0 requires httpx>=0.25.0 but you have httpx 0.23.3; httpx not pinned in pyproject.toml; SGLang server itself healthy.
• Top hypothesis: [HIGH] image dependency mismatch; new SDK test from PR #26773 first exercises the path. Disconfirming: none.
• In-flight fix: ❌ none (open anthropic PRs #28906/#28955 address system-message handling, not the httpx pin).
• Suggested triage: pin/upgrade httpx≥0.25 in the AMD image or constrain lmms_eval; re-run shard 13.

Cluster: 1-GPU diffusion runner loss (no log)

• Affected: multimodal-gen-test-1-gpu part-1 (jobs 82667263584, 82620731588, 82798487110) — "self-hosted runner lost communication", step never finalized, ~53–55 min, blob purged; siblings on same commit pass.
• Top hypothesis: [MEDIUM] host resource starvation / GPU hang. Disconfirming: intermittent (recovered Jun 22 13:56) — points to infra flake, not code.
• In-flight fix: ❌ none. Suggested triage: inspect MI325 runner dmesg/pod-eviction during the failure windows; quarantine/retry partition-1.

Other single-job clusters (rows 6–13)

• pr-test-amd-extra cancel (jobs 82798476959, 82798476988): [HIGH] fast-fail cascade in [IN-FLIGHT] run 27977371862; not a real test failure. In-flight fix ❌ n/a. Triage: re-run, or bypass-fastfail label sparingly for full signal.
• Perf 45-min timeout (row 7): suite overran (2057s); test_score_api_latency_throughput p95 56.7ms > 50. ❌ no fix. Triage: bisect 13-commit window; check if threshold/regression.
• port 5555 strict-ports (row 8): ❌ no fix. Triage: hunt orphaned scheduler on 2gpu-sglang runner; reap stale processes.
• Kimi-K2 launch timeout (row 9) / mori ELF corruption (row 12) / 8-GPU no-log termination (row 11): infra-flavored, ❌ no fix; triage = runner health checks + re-run.
• RCCL SIGSEGV (row 10): ❌ no fix; triage: re-run smoke test, check RCCL/driver version in dev image.
• FLUX.1-dev 90-min hang (row 13): ❌ no fix; triage: reproduce sglang generate FLUX.1-dev on MI325, capture py-spy.

Suggested triage order

Rank	Triage step	Targets	Why this first
1	Unblock/merge #28889, re-run multimodal-2gpu shard 1	row 1 (8 jobs)	Largest code-defect cluster with a ready fix
2	Review #27012/#26326 + bisect disagg window	row 3 (5 jobs)	5 jobs, fresh regression, candidate PRs exist
3	Pin httpx≥0.25 in AMD image (or constrain lmms_eval)	row 4 (3 jobs)	Deterministic env fix, blocks anthropic CI signal
4	Provision/gate gated HF repo ideogram-ai/ideogram-4-fp8	row 2 (8 jobs)	Cheap access fix clears 8 recurring errors
5	Inspect MI325/MI35x/8-GPU runner host logs (dmesg, evictions)	rows 5, 11, 12, 10	One infra check covers all no-log/runner-loss flakes

---

Job	Failed Steps	Started
stage-b-test-1-gpu-small-amd (linux-mi325-1gpu-sglang, 13)	Run test	2026-06-22T15:24
multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 1)	Run diffusion server tests (2-GPU)	2026-06-22T14:03
multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 1)	N/A	2026-06-22T08:43
multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 1)	Run diffusion server tests (2-GPU)	2026-06-22T08:25
stage-b-test-1-gpu-small-amd (linux-mi325-1gpu-sglang, 13)	Run test	2026-06-22T10:20
multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 1)	Run diffusion server tests (2-GPU)	2026-06-22T01:00
stage-b-test-1-gpu-small-amd (linux-mi325-1gpu-sglang, 13)	Run test	2026-06-22T01:00
call-pr-test-amd-extra / extra-a-test-1-gpu-large-amd (linux-mi325-1gpu-sglang)	Install dependencies	2026-06-22T21:45
multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 1)	N/A	2026-06-22T01:00
call-pr-test-amd-extra / extra-a-test-1-gpu-small-amd (linux-mi325-1gpu-sglang)	Install dependencies	2026-06-22T21:45
multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 1)	N/A	2026-06-22T21:43
multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 1)	Run diffusion server tests (2-GPU)	2026-06-22T20:00
stage-b-test-1-gpu-large-amd (linux-mi325-1gpu-sglang, 1)	Run test	2026-06-22T21:56
stage-b-test-large-8-gpu-mi35x-disaggregation-amd (linux-mi35x-gpu-8.fabric)	Run test_disaggregation	2026-06-22T20:07
stage-c-test-large-8-gpu-amd (linux-mi325-8gpu-sglang, 2)	Run test	2026-06-23T00:12
multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 1)	Run diffusion server tests (2-GPU)	2026-06-23T00:59
stage-b-test-large-8-gpu-mi35x-disaggregation-amd (linux-mi35x-gpu-8.fabric)	Run test_disaggregation	2026-06-23T01:20
stage-b-test-large-8-gpu-mi35x-disaggregation-amd (linux-mi35x-gpu-8.fabric)	Run test_disaggregation	2026-06-23T10:06
multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 0)	Run diffusion server tests (2-GPU)	2026-06-23T09:40
multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 1)	Run diffusion server tests (2-GPU)	2026-06-23T09:41
multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 3)	Run diffusion server tests (1-GPU)	2026-06-23T09:55
stage-c-test-large-8-gpu-amd (linux-mi325-8gpu-sglang, 0)	Test RCCL multi-GPU communication	2026-06-23T12:18
multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 1)	Run diffusion server tests (2-GPU)	2026-06-23T14:27
stage-c-test-large-8-gpu-amd (linux-mi325-8gpu-sglang, 1)	N/A	2026-06-23T14:58
stage-b-test-large-8-gpu-mi35x-disaggregation-amd (linux-mi35x-gpu-8.fabric)	Run test_disaggregation	2026-06-23T16:16
stage-a-test-1-gpu-small-amd (linux-mi325-1gpu-sglang)	Install dependencies	2026-06-23T19:17
stage-b-test-large-8-gpu-mi35x-disaggregation-amd (linux-mi35x-gpu-8.fabric)	Run test_disaggregation	2026-06-23T19:32
multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 1)	Run diffusion server tests (2-GPU)	2026-06-23T19:35

Per-Job Analysis

stage-b-test-1-gpu-small-amd (linux-mi325-1gpu-sglang, 13) — failed step(s): Run test

Commit Info

• sglang: 4923bb93
• aiter: 7d604afe5 (v0.1.14rc1.dev240+g7d604afe5, source: dev image preserved; AITER_COMMIT_OVERRIDE empty)

Failed Tests

Test File	Test Function	Error	Log
test/registered/openai_server/basic/test_anthropic_server.py	test_in_messages_system_role	TypeError: HTTPTransport.__init__() got an unexpected keyword argument 'socket_options'	Log

All 6 other test files in this shard passed (test_radix_attention.py, test_external_models.py, test_causal_conv1d.py, etc.). Within test_anthropic_server.py, 6/7 tests passed; only test_in_messages_system_role errored.

Failure Summary

One test errored in this shard: test_in_messages_system_role in test/registered/openai_server/basic/test_anthropic_server.py. It instantiates anthropic.Anthropic(...) (line 235), which raises TypeError: HTTPTransport.__init__() got an unexpected keyword argument 'socket_options'. The test retried once and then failed with Exception: retry() exceed maximum number of retries.

Regression Status

Never-passed (no green run found). This test has failed in every completed scheduled run of pr-test-amd.yml since the test function was introduced.

Recent history of test_anthropic_server.py (test_in_messages_system_role) in job stage-b-test-1-gpu-small-amd (linux-mi325-1gpu-sglang, 13) (branch=main, event=schedule, completed runs only):

Date	Run	Job	Test File Status	Failed Function	Error
Jun 22	27958083310	…small-amd (…, 13)	❌ Failed	test_in_messages_system_role	TypeError: …socket_options
Jun 22	27938642367	…small-amd (…, 13)	❌ Failed	test_in_messages_system_role	TypeError: …socket_options
Jun 21	27897611658	…small-amd (…, 13)	❌ Failed	test_in_messages_system_role	TypeError: …socket_options

Test function was added b4dda8b3 ("fix(anthropic): handle mid-conversation system messages", PR #26773). Run 27897611658 (head b4dda8b3ce, Jun 21 07:43 UTC) is the first scheduled run containing the test, and it already failed. Lookback window searched: last 7 completed scheduled runs (Jun 21 01:00 → Jun 22 13:56 UTC); no passing run for this test on AMD.

Failure Cluster

anthropic SDK / httpx socket_options TypeError

Facts (verified observations, no interpretation)

• Direct log evidence (job 82731803181, log line 9591): TypeError: HTTPTransport.__init__() got an unexpected keyword argument 'socket_options', raised from anthropic/_base_client.py:943 → httpx HTTPTransport.__init__().
• Stack trace originates at test_anthropic_server.py:235 client = anthropic.Anthropic(...) → anthropic/_client.py:281 → anthropic/_base_client.py:1034,943.
• Dependency versions in this run (verified from pip output, log lines 2125, 2547-2558):
  • anthropic==0.111.0 installed.
  • httpx==0.28.1 initially present, then downgraded to httpx==0.23.3 by lmms_eval==0.4.1 (which pins httpx==0.23.3): "Uninstalling httpx-0.28.1 / Successfully uninstalled httpx-0.28.1" (lines 2547-2548), "Successfully installed … httpx-0.23.3 … lmms_eval-0.4.1 …" (line 2558).
• Reproducibility: identical socket_options TypeError observed in 3/3 inspected completed runs (jobs 82731803181, 82667264501, 82551960583) — every run since the test was added.
• This is a pure Python dependency conflict; the aiter version is unrelated to the failing code path.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [HIGH] Version incompatibility: lmms_eval==0.4.1 pins httpx==0.23.3, which pip installs by downgrading from 0.28.1; anthropic==0.111.0 passes socket_options to httpx.HTTPTransport, a kwarg that only exists in httpx ≥0.24. The two pins are mutually incompatible in the same environment.
  • Supporting: pip log shows the exact downgrade to httpx 0.23.3 (lines 2547-2558); socket_options is a documented httpx ≥0.24 feature; stack trace passes through anthropic/_base_client.py:943 → HTTPTransport(**transport_kwargs). Also flagged by pip itself: "ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts." (line 2550).
  • Disconfirming: (none found) — the test never passed on AMD, fully consistent with an environment-level conflict rather than a code regression.
  • How to verify: in the CI container, run python -c "import httpx,anthropic; print(httpx.__version__, anthropic.__version__)"; confirm httpx==0.23.3. Then pip install 'httpx>=0.24' (or pin a compatible lmms_eval/anthropic pair) and re-run python3 test/registered/openai_server/basic/test_anthropic_server.py.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for httpx socket_options, lmms_eval httpx, and anthropic httpx; the only matches were unrelated Qwen3.5 AMD PRs (full-text noise, not fixes for this conflict). API calls run: search/issues?q=repo:sgl-project/sglang+is:pr+is:open+httpx+socket_options (and +lmms_eval+httpx, +anthropic+httpx).

Suggested Triage Steps

• Confirm the httpx downgrade is the trigger: in the AMD CI image, check pip show httpx anthropic lmms_eval and verify httpx==0.23.3 alongside anthropic==0.111.0.
• Resolve the pin conflict in the AMD test dependency install: either upgrade/replace lmms_eval to a version not pinned to httpx==0.23.3, or install test_anthropic_server.py's deps in an environment without lmms_eval, or pin httpx>=0.24,<1 after lmms_eval is installed.
• If a quick fix isn't feasible, consider the test may need an AMD skip/quarantine until the dependency environment is fixed, since it has never passed on this shard.

Status (factual, no priority assignment)

• Persistence: Failing in every completed scheduled run of pr-test-amd.yml since the test was added (first seen 27897611658, Jun 21 07:43 UTC; 3/3 inspected runs red). Never observed passing on AMD.
• Scope: 1 test (test_in_messages_system_role) in shard stage-b-test-1-gpu-small-amd (…, 13). Other 6 tests in the same file and all 6 sibling test files in the shard pass.
• Blocked work: Anthropic-SDK end-to-end signal for the role:"system"-in-messages path on AMD CI.
• In-flight fix: ❌ none found — needs new PR or investigation.

multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 1) — failed step(s): Run diffusion server tests (2-GPU)

Commit Info

• sglang: 4923bb93ae176583651d840b3fc5c229dd7e65e7
• aiter: 7d604afe5fa7efba63c0dce323b95d9daf2db112 (v0.1.14rc1.dev240+g7d604afe5, source: dev image preserved — [CI-AITER-CHECK] Dev/patched version detected → skipping rebuild)

Failed Tests

Test File	Test Function	Error	Log
python/sglang/multimodal_gen/test/server/test_server_2_gpu.py	TestDiffusionServerTwoGpu::test_diffusion_generation[flux2_modelopt_fp8_tp2_t2i]	openai.InternalServerError: Internal Server Error ← server-side RuntimeError: CUDA error: HIPBLAS_STATUS_NOT_SUPPORTED in torch._scaled_mm	Log
python/sglang/multimodal_gen/test/server/test_server_2_gpu.py	TestDiffusionServerTwoGpu::test_diffusion_generation[ideogram4_fp8_tp2_t2i]	RuntimeError: Server exited early (code 1) ← huggingface_hub.errors.GatedRepoError: 403 Client Error for ideogram-ai/ideogram-4-fp8	Log

Result: 1 failed, 9 passed, 1 skipped, 12 deselected, 1 error.

Failure Summary

Two distinct problems in the 2-GPU diffusion server suite (shard 1/3). flux2_modelopt_fp8_tp2_t2i FAILED: the server returns HTTP 500 because every fp8 GEMM call in the FLUX.2 DiT raises HIPBLAS_STATUS_NOT_SUPPORTED — first at warmup, then on the first denoising step (to_q projection). ideogram4_fp8_tp2_t2i ERRORed at setup: the server never came up because the model repo ideogram-ai/ideogram-4-fp8 returns HTTP 403 (gated, no access token), so _wait_for_ready saw the process exit early.

Regression Status

Known recurring failure (both tests) — neither has a green run in the queryable window.

Recent history of flux2_modelopt_fp8_tp2_t2i in job multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 1) (branch main, event schedule, completed runs only):

Date	Run	Test Function Status	Error
Jun 22 13:56	27958083310	❌ Failed	HIPBLAS_STATUS_NOT_SUPPORTED
Jun 22 08:09	27938642367	❌ Failed	HIPBLAS_STATUS_NOT_SUPPORTED
Jun 22 01:00	27923265639	❌ Failed	HIPBLAS_STATUS_NOT_SUPPORTED
Jun 21 18:35	27913714740	❌ Failed	HIPBLAS_STATUS_NOT_SUPPORTED
Jun 21 12:45	27904738325	❌ Failed	HIPBLAS_STATUS_NOT_SUPPORTED
Jun 21 07:43	27897611658	❌ Failed	(same)
Jun 21 01:00	27889118263	❌ Failed	(same)
Jun 20 18:34	27880194201	❌ Failed	(same)

flux2_modelopt_fp8_tp2_t2i has failed in every completed run inspected (≥8, back to 2026-06-20 18:34 UTC) with the identical HIPBLAS error — no passing run found in the lookback window → Never-passed (in window). ideogram4_fp8_tp2_t2i shows the same 1 error count across all these runs (gated-repo setup error).

Failure Cluster

• flux2: "fp8 torch._scaled_mm HIPBLAS_STATUS_NOT_SUPPORTED on ROCm"
• ideogram4: "Gated HF repo 403 → server exits at startup" (separate cluster; infra/access, not a code defect)

Facts (verified observations, no interpretation)

• flux2 server-side stack trace (verbatim): fp8_utils.py:1695 → torch._scaled_mm → RuntimeError: CUDA error: HIPBLAS_STATUS_NOT_SUPPORTED when calling HIPBLAS_STATUS_NOT_SUPPORTED. Call chain: flux_2.py:1221 forward → flux_2.py:862 → flux_2.py:423 → _get_qkv_projections (flux_2.py:179) → attn.to_q → linear.py:427 → modelopt_quant.py:425 apply_fp8_linear → fp8_utils.py:1695.
• The fault fires first during synthetic warmup (Synthetic server warmup failed; continuing startup), so the server stays up but every real request 500s → client sees openai.InternalServerError.
• fp8_utils.py:1695 is the per-tensor-weights + per-tensor-activations fused torch._scaled_mm path (verified by reading the source) — this is the GEMM that hipBLAS rejects on the MI325/gfx942 runner.
• ideogram4: httpx ... 403 Forbidden for url https://huggingface.co/ideogram-ai/ideogram-4-fp8/resolve/.../.gitattributes → GatedRepoError → ValueError: Model or revision not found ... ensure you have access → Server exited early (code 1) at test_server_utils.py:472 _wait_for_ready.
• AITER_COMMIT_OVERRIDE is empty (regular schedule run, not an AITER Scout); aiter does not appear in either stack trace.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [HIGH] flux2: torch._scaled_mm with the modelopt per-tensor fp8 (e4m3fn) operands is unsupported by the hipBLAS backend on this gfx942/MI325 image, so the GEMM throws HIPBLAS_STATUS_NOT_SUPPORTED.
  • Supporting: the error is raised directly inside torch._scaled_mm at fp8_utils.py:1695; identical signature in ≥8 consecutive runs; only the fp8 modelopt scenario fails while the bf16 Wan/LTX/FLUX-dev scenarios in the same job pass; an open AMD PR (#28889) describes exactly this (Fix FP8 e4m3fn GEMM on ROCm ... HIPBLAS_STATUS_NOT_SUPPORTED) and adds a dequant→bf16 fallback in apply_fp8_linear.
  • Disconfirming: no green baseline exists in the window, so this is a never-passed AMD-port gap rather than a fresh regression from a specific commit; the precise unsupported operand combination (rowwise vs per-tensor, scale dtype) is not proven from the log alone.
  • How to verify: on an MI325 container, run flux2_modelopt_fp8_tp2_t2i with PR #28889 applied; confirm the dequant→bf16 fallback removes the HIPBLAS error.
• [FACT] ideogram4: the failure is a HuggingFace gated-repo 403, not a code defect — the CI environment lacks access to ideogram-ai/ideogram-4-fp8.
  • Supporting: verbatim GatedRepoError: 403 Client Error; server exits before serving any request.
  • Disconfirming: (none — this is directly observable).
  • How to verify: provide an HF token with access, or route ideogram without the gated metadata fetch (see #28225).

In-flight Fix Check

• flux2: In-flight fix: ✅ #28889 (open since 2026-06-22, non-draft) — "[AMD] Fix FP8 e4m3fn GEMM on ROCm: dequant->bf16 fallback in apply_fp8_linear (fixes flux2 modelopt-fp8 HIPBLAS_STATUS_NOT_SUPPORTED)". Directly targets this failure; chase reviewers, do NOT open a duplicate. (search: flux2+fp8+amd, HIPBLAS_STATUS_NOT_SUPPORTED.)
• ideogram4: In-flight fix: ✅ #28225 (open since 2026-06-15, draft) — "Route Ideogram NF4 without gated metadata fetch". Related to the gated-fetch path; still draft. (search: ideogram+gated.)

Suggested Triage Steps

• Verify #28889 resolves flux2: apply on a branch and re-run test_diffusion_generation[flux2_modelopt_fp8_tp2_t2i] on an MI325 2-GPU runner; confirm torch._scaled_mm no longer throws.
• For ideogram4, decide whether this scenario should run in AMD CI at all given the gated repo: either grant the CI HF token access to ideogram-ai/ideogram-4-fp8, advance draft PR #28225, or skip/deselect the scenario on runners without access so it stops surfacing as a hard ERROR.
• These are two independent clusters — track them separately; the ideogram 403 will persist even after the flux2 fp8 fix lands.

Status (factual, no priority assignment)

• Persistence: flux2 failing in every completed run inspected since at least 2026-06-20 18:34 UTC (≥8 runs, ~2 days); no green run in window. ideogram4 ERRORing across the same runs.
• Scope: job multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 1) in pr-test-amd.yml. flux2 is the only hard test failure; ideogram4 is a setup ERROR. 9 other scenarios in the shard pass.
• Blocked work: blocks the FLUX.2 modelopt-fp8 2-GPU diffusion signal on AMD; ideogram4 fp8 2-GPU coverage unavailable in CI.
• In-flight fix: flux2 → ✅ #28889 (open, non-draft, since 2026-06-22); ideogram4 → ✅ #28225 (open, draft, since 2026-06-15).

multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 1) — failed step(s): N/A

Commit Info

• sglang: db12bfcd ([JIT] Add kpool_topk_transform JIT kernel #28670)
• aiter: 7d604afe5 — v0.1.14rc1.dev240+g7d604afe5 (source: dev/patched image build, preserved; AITER_COMMIT_OVERRIDE empty)

Failed Tests

Test File	Test Function	Error	Log
N/A — multimodal_gen/test/run_suite.py --suite 1-gpu --partition-id 1	unknown (no test output captured)	Runner died during the test step — step "Run diffusion server tests (1-GPU)" produced no log and ended with conclusion None; job marked failure	Job

This is not a per-test assertion failure. The diffusion test step (step 9) never emitted any captured output; its log blob is absent (BlobNotFound) and the GitHub step conclusion is None, so no individual diffusion test can be pinned.

Failure Summary

The multimodal-gen-test-1-gpu-amd partition 1 job failed because the self-hosted MI325 runner stopped reporting during the diffusion test step. Setup steps 1–8 succeeded (GPU clean: 298 MB/274 GB VRAM used; 1.5 TiB RAM free; disk 16%), then step 9 ran for ~45 min and the job terminated as failure at 53 min total — well under the 90-min step timeout — with no test-step log, no failure artifacts, and no subsequent steps. The other three partitions of the same job on the same commit all passed.

Regression Status

Flaky / Infrastructure issue (partition-1-specific intermittent runner death). Not a code regression: siblings on the identical commit passed (part 0: 6 passed, 1 skipped; part 2: 8 passed; part 3: 1 passed).

Recent history of multimodal-gen-test-1-gpu-amd (…, 1) (same branch main, same event schedule, completed runs only):

Date (UTC)	Run	Job	Status	Failed Function	Error
Jun 22 13:56	27958083310	part 1	✅ Passed	—	—
Jun 22 08:09	27938642367	part 1	❌ Failed (THIS)	unknown (no log)	runner died in step 9
Jun 22 01:00	27923265639	part 1	❌ Failed	unknown (no log)	runner died in step 9
Jun 21 18:35	27913714740	part 1	❌ Failed	unknown (no log)	runner died in step 9
Jun 21 12:45	27904738325	part 1	✅ Passed	—	—
Jun 21 07:43	27897611658	part 1	✅ Passed	—	—

First observed failure for this partition: Jun 21 18:35 UTC. Last passing before: Jun 21 12:45 UTC. Recovered: Jun 22 13:56 UTC. Pattern: 3 consecutive failures bracketed by passes → intermittent, partition-1 only.

Failure Cluster

Self-hosted AMD runner died mid-step during diffusion test (no log, step conclusion None)

Facts (verified observations)

• Step 9 "Run diffusion server tests (1-GPU)" has step conclusion None and no log file in the run archive; direct job-log fetch returns BlobNotFound.
• Job ran 08:43:17 → 09:36:19 UTC (~53 min); step timeout is 90 min, so this was not a step timeout.
• Pre-test state healthy (step 8 log): VRAM 298 MB used of 274 GB; RAM 1.5 TiB free of 3.0 TiB; disk 16%. All 1-GPU diffusion models reported NOT CACHED, so the step had to download Wan2.1-T2V-1.3B, HunyuanVideo, Qwen-Image, FLUX.1-dev from HF.
• Sibling partitions on the same commit db12bfcd passed: part 0 6 passed, 1 skipped (26 min), part 2 8 passed (35 min), part 3 1 passed (2.6 min).
• The two prior part-1 failures (jobs 82620731588, 82595189869) share the identical signature: step 8 success, step 9 None, no log, ~53–55 min runtime, blob purged.
• aiter identical across this job and siblings (v0.1.14rc1.dev240+g7d604afe5), so aiter is not a differentiator.

Hypothesised Causes (with confidence)

• [MEDIUM] Self-hosted runner agent terminated mid-step (host OOM-killer, GPU/driver hang triggering a reset, or pod eviction) on the partition-1 runner instance.
  • Supporting: step 9 has conclusion None with zero captured log + steps 10/20 also None + job still marked failure — the canonical signature of a runner losing its connection mid-step rather than a test asserting/exiting. Reproduced identically across 3 consecutive part-1 jobs. Siblings on the same commit pass.
  • Disconfirming: pre-test host RAM was 1.5 TiB free and GPU was idle, so a host-RAM OOM at step start is unlikely (though the diffusion models loaded during the step could still spike memory/VRAM). No coredump or kernel-log evidence is available since the log was lost.
  • How to verify: check the runner host's dmesg / kubelet eviction events / amdgpu driver logs for the 08:51–09:36 UTC window on the part-1 MI325 node; correlate with rocm-smi / node memory at crash time.
• [LOW] Partition-1 test subset hits a specific large-model download/inference path that intermittently hangs or exhausts VRAM, killing the runner.
  • Supporting: failure is partition-1-specific; all models were uncached so the step did heavy downloads; the suite is split by --partition-id, so part 1 exercises a different model set than the passing partitions.
  • Disconfirming: part 1 passed before (Jun 21 12:45) and after (Jun 22 13:56) on the same partition split, so the subset is not inherently broken; intermittency points more to host/infra than to a deterministic code path.
  • How to verify: re-run part 1 in isolation with verbose logging and rocm-smi sampling; identify which model in the part-1 split is in flight at the ~45-min mark.

No hypothesis reaches HIGH confidence — the decisive evidence (the step-9 log) was lost with the runner.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs multimodal-gen+diffusion (110 hits, all feature work) and diffusion+flaky/runner (219 hits, none targeting partition-1 runner crashes). No PR addresses this infra flake. API: GET /search/issues?q=repo:sgl-project/sglang+is:pr+is:open+multimodal-gen+diffusion.

Suggested Triage Steps

• Inspect the part-1 MI325 runner node's system logs (dmesg, kubelet/pod-eviction events, amdgpu driver resets) for 2026-06-22 08:51–09:36 UTC to confirm whether the host or GPU died — this is the only way to convert the [MEDIUM] hypothesis to a root cause, since the CI log was lost.
• Re-run partition 1 alone (--partition-id 1 --total-partitions 4) with rocm-smi/free -h sampling during the step to capture the state the original run could not log; the job has since self-recovered (Jun 22 13:56 passed), so this may also confirm transience.
• If the crash reproduces, bisect the part-1 model split (-k per model) to see whether one model's download/inference triggers the runner death, vs. a host-level instability independent of the test.
• Consider adding step-level live-log flushing / a lightweight heartbeat to the diffusion test step so future runner deaths leave diagnosable output instead of an empty None step.

Status

• Persistence: 3 consecutive part-1 failures Jun 21 18:35 → Jun 22 08:09 UTC; recovered Jun 22 13:56 UTC. Intermittent.
• Scope: multimodal-gen-test-1-gpu-amd partition 1 only, in pr-test-amd.yml (schedule). Partitions 0/2/3 unaffected; no other workflow implicated.
• Blocked work: marks the 1-GPU diffusion CI signal red for the affected runs, but coverage is not lost (other partitions pass; the same partition passes on adjacent runs).
• In-flight fix: ❌ none found — needs an infra/runner investigation, not a code PR.

multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 1) — failed step(s): Run diffusion server tests (2-GPU)

Commit Info

• sglang: db12bfcd
• aiter: 7d604afe5 (v0.1.14rc1.dev240+g7d604afe5) — source: dev/patched image build preserved ([CI-AITER-CHECK] says "Dev/patched version detected → skipping rebuild")

Failed Tests

Test File	Test Function	Error	Log
sglang/multimodal_gen/test/server/test_server_2_gpu.py	TestDiffusionServerTwoGpu::test_diffusion_generation[flux2_modelopt_fp8_tp2_t2i]	openai.InternalServerError: Internal Server Error (server-side RuntimeError: CUDA error: HIPBLAS_STATUS_NOT_SUPPORTED in torch._scaled_mm)	Log
sglang/multimodal_gen/test/server/test_server_2_gpu.py	TestDiffusionServerTwoGpu::test_diffusion_generation[ideogram4_fp8_tp2_t2i]	huggingface_hub.errors.GatedRepoError: 403 Client Error (gated repo ideogram-ai/ideogram-4-fp8)	Log

Failure Summary

The 2-GPU diffusion server test shard failed on two scenarios. flux2_modelopt_fp8_tp2_t2i returned a 500 to the client because the server crashed during warmup: the ModelOpt FP8 linear in the Flux2 transformer's to_q projection called torch._scaled_mm, which raised HIPBLAS_STATUS_NOT_SUPPORTED on the MI325/ROCm backend. ideogram4_fp8_tp2_t2i errored before serving because the model weights live in a gated Hugging Face repo the CI runner is not authorized to download (403).

Regression Status

Known recurring failure (both tests). Shard 1 of this job has failed on every queryable completed scheduled run, with the same two test scenarios failing the same way. No passing run for these test functions found in the lookback window.

Recent history of test_server_2_gpu.py (flux2_modelopt_fp8_tp2_t2i & ideogram4_fp8_tp2_t2i) in job multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 1) (branch=main, event=schedule, completed runs only):

Date (UTC)	Run	Job	flux2 status	ideogram4 status
Jun 22 13:56	27958083310	job	❌ FAILED (HIPBLAS)	❌ ERROR (403 gated)
Jun 22 08:09	27938642367	job	❌ FAILED (HIPBLAS)	❌ ERROR (403 gated)
Jun 22 01:00	27923265639	job	❌ FAILED (HIPBLAS)	❌ ERROR (403 gated)
Jun 21 18:35	27913714740	job	❌ FAILED (HIPBLAS)	❌ ERROR (403 gated)
Jun 21 12:45	27904738325	job	❌ FAILED (HIPBLAS)	❌ ERROR (403 gated)

(Lookback: 5 completed scheduled runs, Jun 21 12:45 → Jun 22 13:56 UTC. Both functions Never-passed in this window. Run 27977371862 is queued and excluded.)

Failure Cluster

Two distinct clusters in one shard:

1. FP8 torch._scaled_mm unsupported on ROCm/hipBLAS (HIPBLAS_STATUS_NOT_SUPPORTED) — flux2.
2. Gated HF repo access denied (403) — ideogram4.

Facts (verified observations)

• flux2 server-side stack trace (verbatim): apply_fp8_linear → output = torch._scaled_mm(...) at python/sglang/srt/layers/quantization/fp8_utils.py:1695, reached from models/dits/flux_2.py:179 (attn.to_q) → layers/quantization/modelopt_quant.py:425. Terminal error: RuntimeError: CUDA error: HIPBLAS_STATUS_NOT_SUPPORTED.
• Crash occurs during server warmup ([denoising_step_0] Error during execution after 8841 ms), so the scheduler dies and the subsequent client images.generate gets HTTP 500 → openai.InternalServerError.
• ideogram4: Cannot access gated repo for url https://huggingface.co/ideogram-ai/ideogram-4-fp8/...; Access to model ideogram-ai/ideogram-4-fp8 is restricted and you are not in the authorized list → GatedRepoError: 403 → Rank 0 scheduler is dead → launcher EOFError.
• Both failures reproduce identically in all 5 completed runs above (FP8 HIPBLAS for flux2; 403 gated for ideogram4).
• aiter version unchanged across the window (dev image g7d604afe5); NVIDIA equivalents not affected by the HIPBLAS path (ROCm-specific symptom).

Hypothesised Causes (with confidence)

• [HIGH] flux2: ModelOpt FP8 (e4m3fn) GEMM via torch._scaled_mm is not supported by hipBLAS for this weight/scale layout on gfx942 (MI325).
  • Supporting: error is raised directly inside torch._scaled_mm at fp8_utils.py:1695; ROCm-only; the open fix PR explicitly targets this path with a dequant→bf16 fallback.
  • Disconfirming: (none found) — the stack trace points unambiguously at the scaled-mm call.
  • How to verify: apply #28889 on a branch and re-run test_server_2_gpu.py::...[flux2_modelopt_fp8_tp2_t2i] on MI325.
• [FACT] ideogram4: test model ideogram-ai/ideogram-4-fp8 is a gated/restricted HF repo and the CI HF token is not on the authorized list (verbatim 403 GatedRepoError). This is an access/config issue, not a code regression.
  • How to verify: confirm CI HF account has access to ideogram-ai/ideogram-4-fp8, or skip/gate the scenario when access is unavailable.

In-flight Fix Check

• flux2 (HIPBLAS FP8): ✅ #28889 "[AMD] Fix FP8 e4m3fn GEMM on ROCm: dequant->bf16 fallback in apply_fp8_linear (fixes flux2 modelopt-fp8 HIPBLAS_STATUS_NOT_SUPPORTED)" — open since 2026-06-22 06:14 UTC, not draft. Chase reviewers; do NOT open a duplicate. (search: is:pr is:open HIPBLAS_STATUS_NOT_SUPPORTED and flux2 fp8)
• ideogram4 (gated repo): ❌ none found — needs HF access grant or a CI skip-on-no-access guard, not a code PR.

Suggested Triage Steps

• Verify in-flight fix #28889 lands cleanly and re-run test_server_2_gpu.py::...[flux2_modelopt_fp8_tp2_t2i] on linux-mi325-2gpu-sglang to confirm it clears the HIPBLAS failure.
• For ideogram4: confirm whether the CI HF token is intended to have access to ideogram-ai/ideogram-4-fp8. If access cannot be granted, gate/skip the ideogram4_fp8_tp2_t2i scenario on AMD CI so it stops masking real shard-1 signal.

Status

• Persistence: flux2 + ideogram4 both failing on every completed scheduled run in the queryable window (since at least 2026-06-21 12:45 UTC); Never-passed for shard 1 in that window.
• Scope: multimodal-gen-test-2-gpu-amd shard 1 only (shards 0 and 2 pass); workflow pr-test-amd.yml. flux2 symptom is ROCm-specific.
• Blocked work: 2-GPU diffusion FP8 (Flux2 ModelOpt) and gated-model (ideogram4) coverage on AMD MI325.
• In-flight fix: flux2 ✅ #28889 (open since 2026-06-22); ideogram4 ❌ none (access/config issue).

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-06-23 20:58 UTC)

[amd-bot]

pr-test-amd.yml — Per-Job Analysis (continued 2/5)

stage-b-test-1-gpu-small-amd (linux-mi325-1gpu-sglang, 13) — failed step(s): Run test

Commit Info

• sglang: db12bfcdc8a686efc8ee0a2899bdb43be9cedcf8
• aiter: 7d604afe5fa7efba63c0dce323b95d9daf2db112 — v0.1.14rc1.dev240+g7d604afe5 (source: dev image preserved; [CI-AITER-CHECK] Dev/patched version detected → skipping rebuild, AITER_COMMIT_OVERRIDE empty)

Failed Tests

Test File	Test Function	Error	Log
test/registered/openai_server/basic/test_anthropic_server.py	TestAnthropicServer.test_in_messages_system_role	TypeError: HTTPTransport.__init__() got an unexpected keyword argument 'socket_options' (→ retry() exceed maximum number of retries)	Log

The other 6 tests in this file passed (Ran 7 tests … FAILED (errors=1)). All other 10 test files in the shard passed.

Failure Summary

The shard-13 job failed solely because test_in_messages_system_role in test_anthropic_server.py errored. It is the only test in the file that constructs an anthropic.Anthropic SDK client (line 235); the other six tests use raw HTTP via _make_request. Client construction raises TypeError: HTTPTransport.__init__() got an unexpected keyword argument 'socket_options', the retry wrapper exhausts its retries, and the test errors out.

Regression Status

New regression — started 2026-06-21 with the merge of PR #26773.

Recent history of test/registered/openai_server/basic/test_anthropic_server.py (test_in_messages_system_role) in job stage-b-test-1-gpu-small-amd (linux-mi325-1gpu-sglang, 13) (branch main, event schedule, completed runs only):

Date (UTC)	Run	Job	Test File Status	Failed Function	Error
Jun 22 08:09	27938642367	shard 13	❌ Failed	test_in_messages_system_role	socket_options TypeError
Jun 22 01:00	27923265639	shard 13	❌ Failed	test_in_messages_system_role	socket_options TypeError
Jun 21 07:43	27897611658	shard 13	❌ Failed	test_in_messages_system_role	socket_options TypeError
Jun 20 18:34	27880194201	shard 13	✅ Passed	—	—

First observed failure: 2026-06-21 07:43 UTC (the first scheduled run containing PR #26773, head b4dda8b3). Last passing: 2026-06-20 18:34 UTC (head d6d06cdc17, before the PR). Failing in every completed run since.

Failure Cluster

anthropic SDK / httpx version incompatibility (socket_options kwarg)

Facts (verified observations)

• Verbatim error: TypeError: HTTPTransport.__init__() got an unexpected keyword argument 'socket_options' at anthropic/_base_client.py:943 (default_transport = HTTPTransport(**transport_kwargs)), triggered from test_anthropic_server.py:235 (client = anthropic.Anthropic(...)).
• The image installs httpx==0.23.3, pinned transitively by lmms_eval==0.4.1 (Collecting httpx==0.23.3 (from lmms_eval==0.4.1)). httpx 0.23.3 predates the socket_options parameter on HTTPTransport, which the installed anthropic SDK passes unconditionally.
• A secondary AttributeError: 'SyncHttpxClientWrapper' object has no attribute '_state' is emitted from the SDK's __del__ — a side effect of the failed constructor, not an independent failure.
• PR #26773 (b4dda8b3, merged 2026-06-21 04:34 UTC) added 22 lines to test_anthropic_server.py, including test_in_messages_system_role — the first/only test exercising the anthropic Python SDK client. Verified via git show b4dda8b3 --stat.
• Same test + same error confirmed in the shard-13 logs of runs 27897611658 and 27923265639; shard 13 was ✅ green in the pre-PR run 27880194201.
• aiter unchanged (Dockerfile-default dev image, no override) — not an aiter-related failure.

Hypothesised Causes (with confidence)

• [HIGH] Environment incompatibility: the installed anthropic SDK requires an httpx new enough to accept socket_options on HTTPTransport, but the AMD CI image ships httpx==0.23.3 (pinned by lmms_eval==0.4.1). The newly-added SDK-based test in PR #26773 is the first test to exercise this path, so it surfaces the latent mismatch.
  • Supporting: socket_options is rejected by httpx 0.23.3's HTTPTransport.__init__; the log shows httpx==0.23.3 being collected; the failing line constructs the SDK client; failure begins exactly in the first run containing the PR and persists every run since; pre-PR run green.
  • Disconfirming: (none found) — the regression boundary and the dependency pin are both directly observable.
  • How to verify: in the AMD CI container, run python -c "import httpx, anthropic; print(httpx.__version__, anthropic.__version__)" and a minimal anthropic.Anthropic(base_url=..., auth_token=...) construction; confirm bumping httpx (or aligning the anthropic SDK pin) resolves it.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for anthropic+httpx and socket_options (GET /search/issues?q=repo:sgl-project/sglang+is:pr+is:open+anthropic+httpx). The two open anthropic PRs (#28906, #28955) concern protocol-level mid-conversation system-message handling, not the httpx version incompatibility. No PR addresses the dependency conflict.

Suggested Triage Steps

• Confirm the httpx vs anthropic version matrix inside the AMD CI image: pip show httpx anthropic / pip index versions httpx. The lmms_eval==0.4.1 pin (httpx==0.23.3) is the likely constraint to resolve.
• Option A: bump httpx to a version supporting socket_options (≥0.25.0) in the AMD image and re-run test_anthropic_server.py on linux-mi325-1gpu-sglang; verify lmms_eval still functions with the newer httpx.
• Option B: in test_in_messages_system_role, pass an explicit http_client=httpx.Client(...) (without socket_options) to anthropic.Anthropic(...) so the test does not rely on the SDK's default transport kwargs — decouples the test from the image's httpx pin.
• Verify whether NVIDIA CI runs this test and with which httpx; if it passes there, the divergence is image-specific to AMD and points squarely at the lmms_eval httpx pin.

Status

• Persistence: Failing since 2026-06-21 07:43 UTC, every completed scheduled run since (3 consecutive runs).
• Scope: 1 test function (test_in_messages_system_role) in 1 shard (stage-b-test-1-gpu-small-amd shard 13) of pr-test-amd.yml. Blocks Anthropic Messages-API SDK signal on AMD CI.
• In-flight fix: ❌ none found in open PRs — needs a new fix (image httpx bump or test decoupling).

multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 1) — failed step(s): Run diffusion server tests (2-GPU)

Commit Info

• sglang: 6779ca8d
• aiter: 7d604afe5fa7efba63c0dce323b95d9daf2db112 (source: dev image — [CI-AITER-CHECK] reports Dev/patched version detected: v0.1.14rc1.dev240+g7d604afe5 → skipping rebuild; AITER_COMMIT_OVERRIDE empty, so this is a regular pr-test-amd run, not a scout)

Failed Tests

Test File	Test Function	Error	Log
sglang/multimodal_gen/test/server/test_server_2_gpu.py	TestDiffusionServerTwoGpu::test_diffusion_generation[flux2_modelopt_fp8_tp2_t2i]	openai.InternalServerError (server-side RuntimeError: CUDA error: HIPBLAS_STATUS_NOT_SUPPORTED in torch._scaled_mm)	Log
sglang/multimodal_gen/test/server/test_server_2_gpu.py	TestDiffusionServerTwoGpu::test_diffusion_generation[ideogram4_fp8_tp2_t2i]	huggingface_hub.errors.GatedRepoError: 403 Client Error (gated repo ideogram-ai/ideogram-4-fp8)	Log

Failure Summary

Two of twelve scenarios in test_server_2_gpu.py failed (final tally: 1 failed, 9 passed, 1 skipped, 1 error). flux2_modelopt_fp8_tp2_t2i reaches the server but the diffusion worker crashes during the warmup denoising step with HIPBLAS_STATUS_NOT_SUPPORTED, surfacing to the test client as openai.InternalServerError. ideogram4_fp8_tp2_t2i errors at setup because the worker cannot download the gated HuggingFace repo ideogram-ai/ideogram-4-fp8 (403).

Regression Status

Known recurring failure (both tests) — no green run found for either test in the queryable window (4 most-recent completed scheduled runs, 2026-06-21 12:45 → 2026-06-22 08:09 UTC).

Recent history of test_server_2_gpu.py in job multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 1) (branch main, event schedule, completed runs only):

Date (UTC)	Run	flux2_modelopt_fp8_tp2_t2i	ideogram4_fp8_tp2_t2i
Jun 22 08:09	27938642367	❌ FAILED (HIPBLAS)	❌ ERROR (GatedRepo 403)
Jun 22 01:00	27923265639 ← this	❌ FAILED (HIPBLAS)	❌ ERROR (GatedRepo 403)
Jun 21 18:35	27913714740	❌ FAILED (HIPBLAS)	❌ ERROR (GatedRepo 403)
Jun 21 12:45	27904738325	❌ FAILED (HIPBLAS)	❌ ERROR (GatedRepo 403)

(The HIPBLAS error string appears 28× in each historical flux2 log — same crash signature throughout. First-observed failure date is at or before Jun 21 12:45; no last-known-passing date within the lookback window.)

Failure Cluster

• flux2: FP8 _scaled_mm HIPBLAS_STATUS_NOT_SUPPORTED on ROCm during denoising warmup
• ideogram4: GatedRepoError 403 fetching gated HF model

Facts (verified observations, no interpretation)

• flux2 server loads all modules successfully, then fails on the first warmup request: [denoising_step_0] Error during execution after 8396 ms: CUDA error: HIPBLAS_STATUS_NOT_SUPPORTED.
• flux2 stack trace (verbatim chain): flux_2.py:179 _get_qkv_projections → attn.to_q(...) → linear.py:427 forward → modelopt_quant.py:425 apply → apply_fp8_linear → fp8_utils.py:1695 output = torch._scaled_mm(...) → RuntimeError: CUDA error: HIPBLAS_STATUS_NOT_SUPPORTED.
• flux2 transformer is a ModelOpt FP8 checkpoint (quant_algo: FP8, quant_method: modelopt); aiter attention backend on ROCm (gfx942 / MI325).
• ideogram4 error (verbatim): GatedRepoError: 403 ... Cannot access gated repo for url https://huggingface.co/ideogram-ai/ideogram-4-fp8/resolve/.../.gitattributes. Access to model ideogram-ai/ideogram-4-fp8 is restricted and you are not in the authorized list. raised from hf_diffusers_utils.py:853 maybe_download_model.
• Reproducibility: both failures present in all 4 completed scheduled runs checked, same signatures.

Hypothesised Causes (with confidence, with disconfirming evidence)

flux2_modelopt_fp8_tp2_t2i

• [HIGH] ROCm/hipBLAS does not support the FP8 e4m3fn GEMM that torch._scaled_mm is invoked with in apply_fp8_linear for the ModelOpt-FP8 flux2 transformer.
  • Supporting: error originates directly inside torch._scaled_mm (fp8_utils.py:1695); the failing layer is an FP8 quantized linear (modelopt_quant.apply → apply_fp8_linear); error is HIPBLAS_STATUS_NOT_SUPPORTED (an explicit "operation not supported" from the BLAS backend), not a numerical/shape error; an open PR titled exactly "[AMD] Fix FP8 e4m3fn GEMM on ROCm: dequant->bf16 fallback in apply_fp8_linear (fixes flux2 modelopt-fp8 HIPBLAS_STATUS_NOT_SUPPORTED)" exists.
  • Disconfirming: (none found) — error is deterministic across 4 runs and the stack trace is unambiguous.
  • How to verify: apply #28889 on a branch and re-run test_server_2_gpu.py::...[flux2_modelopt_fp8_tp2_t2i] on MI325.

ideogram4_fp8_tp2_t2i

• [FACT] CI HF credentials are not on the authorized list for the gated repo ideogram-ai/ideogram-4-fp8; the 403 blocks model download at setup.
  • Supporting: HTTP 403 GatedRepoError verbatim in log; failure is at download/setup, before any compute.
  • Disconfirming: (none found) — this is an access-control fact, not a code bug.
  • How to verify: confirm the CI HF token has accepted the model's access terms at https://huggingface.co/ideogram-ai/ideogram-4-fp8, or mirror weights to an internal cache.

In-flight Fix Check

• flux2: ✅ #28889 — "[AMD] Fix FP8 e4m3fn GEMM on ROCm: dequant->bf16 fallback in apply_fp8_linear (fixes flux2 modelopt-fp8 HIPBLAS_STATUS_NOT_SUPPORTED)". Directly targets this exact symptom; chase reviewers, do NOT open a duplicate. (search: flux2 fp8 → 3 open PRs.)
• ideogram4: ❌ none directly fixes the 403. Related but not a fix: #28225 "Route Ideogram NF4 without gated metadata fetch" (NF4 path, not fp8) and #28727 "[Diffusion] Ideogram support bf16 dtype". Root cause is repo access, not code — likely needs an HF-access/cache change, not a PR.

Suggested Triage Steps

• flux2: verify #28889 applies cleanly on 6779ca8d, then re-run test_server_2_gpu.py::TestDiffusionServerTwoGpu::test_diffusion_generation[flux2_modelopt_fp8_tp2_t2i] on MI325 to confirm the HIPBLAS error clears.
• ideogram4: grant the CI HuggingFace token access to ideogram-ai/ideogram-4-fp8 (accept the gated terms) or pre-stage the weights in the runner's /sgl-data/hf-cache; alternatively skip/quarantine [ideogram4_fp8_tp2_t2i] on AMD CI until access is provisioned, since it is unrelated to any code change.

Status

• Persistence: both tests failing in every completed scheduled run of pr-test-amd in the window (≥ since 2026-06-21 12:45 UTC); no green run found for either.
• Scope: job multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 1); 1 FAILED + 1 ERROR per run, the only two reds in an otherwise 9-passed shard.
• In-flight fix: flux2 → ✅ #28889 (awaiting merge); ideogram4 → ❌ none (needs HF access provisioning, not a code PR).

stage-b-test-1-gpu-small-amd (linux-mi325-1gpu-sglang, 13) — failed step(s): Run test

Commit Info

• sglang: 6779ca8d
• aiter: 7d604afe5 — v0.1.14rc1.dev240+g7d604afe5 (source: dev/patched image, preserved — [CI-AITER-CHECK] line 4070; AITER_COMMIT_OVERRIDE empty)

Failed Tests

Test File	Test Function	Error	Log
test/registered/openai_server/basic/test_anthropic_server.py	TestAnthropicServer.test_in_messages_system_role	TypeError: HTTPTransport.__init__() got an unexpected keyword argument 'socket_options'	Log

The other 10 test files in this shard passed (Test Summary: 10/11 passed). Within test_anthropic_server.py, 6 of 7 tests passed — only test_in_messages_system_role errored (Ran 7 tests in 68.145s / FAILED (errors=1)).

Failure Summary

The job's only failure is test_in_messages_system_role in test/registered/openai_server/basic/test_anthropic_server.py:235. That test is the only one in the file that instantiates the real anthropic.Anthropic(...) SDK client; the constructor crashes before any request is sent. The remaining 6 tests in the file (which use raw HTTP requests) pass. The job exited 255 because this one test file returned exit code 1.

Regression Status

Never-passed (no green run found) — environment/dependency-driven, AMD-image-specific.

Recent history of test/registered/openai_server/basic/test_anthropic_server.py (test_in_messages_system_role) in job stage-b-test-1-gpu-small-amd (linux-mi325-1gpu-sglang, 13) (branch main, event schedule, completed runs only):

Date (UTC)	Run	Shard-13 Job	Test File Status	Failed Function	Error
Jun 22 01:00	27923265639	job	❌ Failed	test_in_messages_system_role	socket_options TypeError
Jun 21 18:35	27913714740	job	❌ Failed	test_in_messages_system_role	same
Jun 21 12:45	27904738325	job	❌ Failed	test_in_messages_system_role	same
Jun 21 07:43	27897611658	job	❌ Failed	test_in_messages_system_role	same

The test function was introduced by PR #26773 (b4dda8b3, merged Jun 20 2026 23:34 UTC). Run 27897611658 (head b4dda8b3) is the first scheduled run that contains the test, and it failed identically. No passing run exists in the queryable window (Jun 20–22). Lookback range: 4 most recent completed schedule runs + the introducing commit.

Failure Cluster

anthropic SDK client init crash — httpx too old for socket_options

Facts (verified observations)

• Verbatim error (log line 417 of the test block): TypeError: HTTPTransport.__init__() got an unexpected keyword argument 'socket_options', raised from anthropic/_base_client.py:943 → HTTPTransport(**transport_kwargs), reached via test_anthropic_server.py:235 client = anthropic.Anthropic(...). [FACT]
• Dependency conflict printed during install (job log lines 2550–2551): anthropic 0.111.0 requires httpx<1,>=0.25.0, but you have httpx 0.23.3 which is incompatible. and fastapi-cloud-cli 0.1.5 requires httpx>=0.27.0, but you have httpx 0.23.3 which is incompatible. [FACT]
• socket_options was added to httpx's HTTPTransport.__init__ in httpx 0.25.0; httpx 0.23.3 predates it, so the kwarg passed by anthropic 0.111.0 is rejected. [FACT]
• Secondary noise: AttributeError: 'SyncHttpxClientWrapper' object has no attribute '_state' during __del__ — a follow-on effect of the failed constructor, not an independent failure. [FACT]
• httpx is not pinned anywhere in python/pyproject.toml; sglang only declares anthropic>=0.20.0 (resolves to 0.111.0). The stale httpx 0.23.3 is baked into the AMD rocm/sgl-dev:v0.5.13.post1-rocm700-mi30x-20260621 image. [FACT]
• The crash is in the SDK client constructor — the SGLang server itself is healthy (the same shard's other anthropic tests issuing raw POST /v1/messages requests pass, returning expected 400s and 200s). [FACT]

Hypothesised Causes (with confidence)

• [HIGH] Dependency mismatch in the AMD CI image: anthropic 0.111.0 calls HTTPTransport(socket_options=...), but the image ships httpx 0.23.3 (needs ≥0.25.0).
  • Supporting: the pip resolver explicitly flags the incompatibility (lines 2550–2551); the traceback terminates exactly at the kwarg httpx 0.23.3 lacks; reproduced identically across 4/4 completed runs.
  • Disconfirming: (none found) — the evidence chain is self-contained in the log.
  • How to verify: in the rocm/sgl-dev container, run pip install 'httpx>=0.25.0,<1' then re-run python3 test/registered/openai_server/basic/test_anthropic_server.py::TestAnthropicServer::test_in_messages_system_role.
• [LOW] Test design assumes a host environment where httpx satisfies the anthropic SDK; the AMD image's old httpx makes the SDK path unreachable.
  • Supporting: only the single SDK-using test fails; raw-HTTP tests in the same file pass.
  • Disconfirming: this is the same root mismatch as above, not a distinct cause.
  • How to verify: same as above; alternatively check whether NVIDIA CI (newer httpx) passes this test.

This is not an aiter-caused failure (this is pr-test-amd.yml, not a scout run; aiter is the Dockerfile-default dev build). The aiter delta is irrelevant to this dependency conflict.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for httpx (8 results, none address the version pin) and anthropic (15 results; #28906/#28955 are about mid-conversation system-message handling, not the SDK/httpx dependency). API calls: search/issues?q=repo:sgl-project/sglang+is:pr+is:open+httpx and ...+anthropic.

Suggested Triage Steps

• Bump httpx in the AMD CI image: confirm httpx 0.23.3 origin in rocm/sgl-dev:v0.5.13.post1-rocm700-mi30x-20260621 and rebuild with httpx>=0.25.0,<1 to satisfy both anthropic 0.111.0 and fastapi-cloud-cli.
• Alternatively, add an explicit httpx>=0.25.0 floor to sglang's deps so the resolver upgrades the stale image package at install time.
• As a stopgap, gate/skip test_in_messages_system_role on AMD until the image httpx is upgraded (the SGLang server behavior it exercises is also covered by the raw-HTTP tests that pass).
• To confirm it is purely environmental, check the NVIDIA PR Test run of the same test on the same SHA — if green there, that isolates the cause to the AMD image's httpx version.

Status

• Persistence: Failing on every completed scheduled run since the test was introduced — first appearance Jun 21 07:43 UTC (head b4dda8b3, PR #26773); 4/4 completed runs red. Never passed on AMD.
• Scope: 1 test function (test_in_messages_system_role) in 1 file, in shard 13 of pr-test-amd.yml. Environment-driven (stale httpx in the AMD rocm image), not an sglang server-code regression.
• In-flight fix: ❌ none found — needs a new PR (image httpx upgrade or a dependency floor).

call-pr-test-amd-extra / extra-a-test-1-gpu-large-amd (linux-mi325-1gpu-sglang) — failed step(s): Install dependencies

Commit Info

• sglang: adc203df — fix(router): use full conversation for PD chat cache-aware routing (#26263) (#27430)
• aiter: N/A — the job was cancelled during Install dependencies before any [CI-AITER-CHECK] marker was emitted (AITER_COMMIT_OVERRIDE was empty; no override).

Failed Tests

Test File	Test Function	Error	Log
N/A — no test ran	N/A	Job cancelled during dependency install (building mori, step [61/141]), then ##[error]The operation was canceled.	Log

No test executed. The job's Install dependencies step (step 5) was cancelled while compiling the mori C++ dependency; the Run test step (step 6) was skipped.

Failure Summary

This is not a test failure. The job extra-a-test-1-gpu-large-amd was cancelled at 2026-06-22 21:53:10 UTC while still inside amd_ci_install_dependency.sh (mid-mori build at [61/141]), so it never reached any test. Its sister job extra-a-test-1-gpu-small-amd was cancelled at the exact same instant, and the aggregator pr-test-amd-extra-finish then exited 1 because both matrix jobs reported cancelled (verified in the finish-job log: extra-a-test-1-gpu-small-amd: cancelled → The above jobs failed.).

Regression Status

Infrastructure issue (cancellation cascade) — not a code/test regression.

• Job conclusion: cancelled (confirmed via API: step 5 Install dependencies = cancelled, step 6 Run test = skipped).
• Both extra-a matrix jobs (large + small) were cancelled simultaneously at 21:53:10Z, the signature of a workflow-level cancellation (concurrency cancel-in-progress from a newer scheduled run, or a fast-fail/cancel signal propagated to the call-pr-test-amd-extra matrix), not a per-test failure in this job.
• At snapshot time the parent run 27977371862 was still partially in progress (4 jobs queued, 2 in-progress) — [IN-FLIGHT, partial data] — so run-level pass/fail trend is not yet final.

Because no test ran, there is no test-file-level history to compare; a regression verdict cannot and should not be drawn from this job.

Failure Cluster

"Job cancelled during AMD dependency install (mori build) — fast-fail/concurrency cancellation"

Facts (verified observations, no interpretation)

• ##[error]The operation was canceled. at log line 3407, immediately after [61/141] Building CXX object src/io/.../engine.cpp.o (the mori dependency build).
• API job record: status=completed, conclusion=cancelled; steps — 4 Start CI container: success, 5 Install dependencies: cancelled, 6 Run test: skipped.
• Sister job extra-a-test-1-gpu-small-amd also cancelled at 21:53:10Z (same second).
• pr-test-amd-extra-finish log: call-gate: success, extra-a-test-1-gpu-small-amd: cancelled, The above jobs failed. → exit code 1. The finish job's "failure" is purely a consequence of the cancellations.
• No [CI-AITER-CHECK] markers present (install never reached the aiter check); AITER_COMMIT_OVERRIDE empty.
• Other genuinely-failed siblings in the same run (separate from this cancellation): multimodal-gen-test-1-gpu-amd (…,1), multimodal-gen-test-2-gpu-amd (…,1), stage-b-test-1-gpu-large-amd (…,1), stage-b-test-large-8-gpu-mi35x-disaggregation-amd. These are distinct jobs and outside the scope of this analysis.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [FACT] The job did not fail on its own merits — it was externally cancelled during dependency installation before any test ran.
  • Supporting: API step states (Install dependencies: cancelled, Run test: skipped); the ##[error]The operation was canceled. line mid-mori-build; no test output anywhere in the log.
  • Disconfirming: (none found)
• [LOW] The cancellation was triggered by workflow-level fast-fail or concurrency cancel-in-progress (e.g. a newer scheduled run superseding this one, or a sibling-stage cancel signal), rather than a problem in the mori build itself.
  • Supporting: both extra-a jobs cancelled at the identical second 21:53:10Z; run created 19:10Z but jobs only started 21:45Z (long queue, consistent with a superseding scheduled run); mori build was progressing normally (only deprecation warnings, no compile errors) up to the cancel point.
  • Disconfirming: the local docker-registry pull earlier fell back to the public registry (line 1261) — slow infra could in principle have pushed the job past a timeout; but a timeout would not cancel the sister job in the same instant, so a timeout is unlikely.
  • How to verify: inspect the workflow concurrency group for pr-test-amd-extra.yml and check whether a later schedule run of "PR Test (AMD)" started near 21:53:10Z; check the run's cancellation actor via the Actions UI.

In-flight Fix Check

Not applicable — there is no code/test defect to fix. A cancelled job is resolved by re-running, not by a code change.

• In-flight fix: ❌ none needed — this is a cancellation, not a reproducible test failure.

Suggested Triage Steps

• Re-run the job (extra-a-test-1-gpu-large-amd) once the parent run settles; a cancelled dependency-install step almost always clears on retry.
• Confirm the cancellation source: check the pr-test-amd-extra.yml concurrency: block and whether a newer scheduled "PR Test (AMD)" run cancelled this one in-progress. If concurrency cancel-in-progress is unintended for scheduled runs, that's the config to adjust.
• If extra-a jobs are repeatedly cancelled across recent scheduled runs (not the case in this single snapshot), investigate runner queue saturation on linux-mi325-1gpu-sglang — the 2h15m queue delay (19:10→21:45) is worth watching.
• Investigate the genuinely-failed siblings (multimodal-gen-test-*, stage-b-test-*) separately — they are unrelated to this cancellation.

Status (factual, no priority assignment)

• Persistence: single observed cancellation in this run; the job never executed its tests.
• Scope: both call-pr-test-amd-extra matrix jobs (extra-a large + small) cancelled simultaneously; the pr-test-amd-extra-finish aggregator failed as a downstream consequence. Parent run 27977371862 still [IN-FLIGHT] at snapshot.
• Blocked work: the extra-a (1-GPU large/small) test coverage for commit adc203df produced no signal — green/red unknown for these suites until re-run.
• In-flight fix: ❌ none needed — re-run required, not a code fix.

multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 1) — failed step(s): N/A

Commit Info

• sglang: 6779ca8d
• aiter: 7d604afe — v0.1.14rc1.dev240+g7d604afe5 (source: dev image preserved; Dev/patched version detected → skipping rebuild)

Failed Tests

Test File	Test Function	Error	Log
test/server/test_server_1_gpu.py (partition 1/4 of 1-gpu suite)	unrecoverable — see note	Step never finalized; runner lost the job during Run diffusion server tests (1-GPU)	Job

Note (test-level granularity unavailable): This was not a normal pytest assertion. Step 9 (Run diffusion server tests (1-GPU)) is stuck at status=in_progress while the job concluded failure, and the if: always() "Upload diffusion failure artifacts" step is pending (never ran). The step-9 log blob returns BlobNotFound and is absent from the run-logs archive. This means the runner/container died mid-step rather than the test exiting with a non-zero code — so no per-test log line, JUnit XML, or failure artifact exists to name the exact hung case. Partition 1 runs the LPT group-1 subset of ONE_GPU_CASES in test/server/test_server_1_gpu.py with -k "not flux_2" (T2V/T2I/I2V/LoRA across Wan2.1-T2V-1.3B, HunyuanVideo, Qwen-Image, FLUX.1).

Failure Summary

Shard 1 (partition 1 of 4) of the diffusion 1-gpu server-test suite failed. The test step ran ~55 min then the runner lost the job: step 9 is frozen in_progress, later always() steps never executed, and the log was never finalized. Sibling shards 0, 2, 3 of the same matrix all passed (36.1, 44.1, 11.8 min). This is the hang/runner-loss signature, not an assertion or OOM (those finalize the log and run the upload step).

Regression Status

Flaky / intermittent — 3 of the last 7 completed scheduled runs of this shard failed with the identical signature (step 9 never finalized, ~53–55 min runtime), and it recovered on the run immediately after this one.

Recent history of multimodal-gen-test-1-gpu-amd (…, 1) / partition 1 of the 1-gpu suite (workflow pr-test-amd.yml, branch main, event schedule, completed runs only):

Date (UTC)	Run	Shard-1 Status	Step 9	Duration	Signature
Jun 22 13:56	27958083310	✅ Passed	completed/success	44.5 min	—
Jun 22 08:09	27938642367	❌ Failed	in_progress (frozen)	53.0 min	runner-loss / hang
Jun 22 01:00	27923265639	❌ Failed (this run)	in_progress (frozen)	55.1 min	runner-loss / hang
Jun 21 18:35	27913714740	❌ Failed	in_progress (frozen)	53.0 min	runner-loss / hang
Jun 21 12:45	27904738325	✅ Passed	completed/success	47.4 min	—
Jun 21 07:43	27897611658	✅ Passed	completed/success	42.6 min	—
Jun 21 01:00	27889118263	✅ Passed	completed/success	45.0 min	—

First observed (this window): Jun 21 18:35. Last known passing before the streak: Jun 21 12:45. Recovered: Jun 22 13:56. Failure ratio ≈ 3/7 (43%).

Failure Cluster

Diffusion 1-GPU server test hang → runner loss during step 9 (no log finalization)

Facts (verified observations)

• Job 82620731588 concluded failure; step 9 Run diffusion server tests (1-GPU) is status=in_progress, step 10 Upload diffusion failure artifacts (if: always()) and the post-step are pending. (verified via Jobs API)
• The step-9 log blob returns <Error><Code>BlobNotFound></Code>; the run-logs archive contains only steps 1–8 for this shard and no combined log for shard 1. (verified)
• Step 9 has timeout-minutes: 90; the job died at ~55 min, so it did not hit the GitHub step timeout. (verified — workflow pr-test-amd.yml:712)
• Sibling shards 0/2/3 of the same matrix passed in the same run. (verified via Jobs API)
• Failing runs of this shard run ~10 min longer (53–55 min) than passing runs (42–47 min) before dying. (verified from history above)
• The suite code documents pre-existing timeout flakiness in this exact job: gpu_cases.py:862 disables test_update_weights_from_disk.py citing "24 timeout failures since 2026-04-09 across multimodal-gen-test-1-gpu". (verified)
• aiter is unchanged vs the Dockerfile default (dev image preserved, no override). (verified via [CI-AITER-CHECK] markers)

Hypothesised Causes (with confidence)

• [LOW] One of partition 1's diffusion server cases in test/server/test_server_1_gpu.py (a video/image E2E case among Wan2.1-T2V-1.3B / HunyuanVideo / Qwen-Image / FLUX.1) intermittently hangs, causing the runner to lose the job before the step can finalize.
  • Supporting: failing runs consistently run ~10 min longer than passing runs and never finalize step 9; the suite already has a documented history of timeout flakiness on this job (gpu_cases.py:862); only partition 1 is affected while sibling shards pass.
  • Disconfirming: the exact hung case is unidentifiable (no log/artifact); intermittency across different head SHAs and recovery without any fix commit means it is not tied to a code change in this commit window; could equally be a runner-pool hardware/GPU hang independent of the test.
  • How to verify: re-run shard 1 in isolation with verbose per-case timing and a py-spy dump / GPU watchdog on hang; correlate failures to specific runner host in the linux-mi325-1gpu-sglang pool.
• [LOW] Infrastructure: a specific MI325 runner in the pool hangs (GPU watchdog / driver) during long video generation, dropping the runner connection.
  • Supporting: identical signature across 3 runs (frozen step, no graceful exit) is more typical of runner/host loss than a Python exception; always() upload never ran.
  • Disconfirming: cannot confirm same physical host across the 3 failures from the API; the >10-min duration skew points partly at the workload, not pure infra.
  • How to verify: map the 3 failing jobs to runner hostnames; check dmesg/GPU reset logs on those hosts for the failure windows.

No hypothesis exceeds LOW confidence: the step log was never finalized and no failure artifact was produced, so the exact hung test cannot be pinned without a fresh instrumented repro.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for multimodal-gen timeout, diffusion test flaky, multimodal_gen hang; no open PR targets this shard's timeout/hang flakiness (results were unrelated keyword hits). The in-code mitigation at gpu_cases.py:862 (disabling test_update_weights_from_disk.py) is already merged but clearly does not cover the partition-1 case set.

Suggested Triage Steps

• Re-run shard 1 (partition 1) standalone several times to reproduce; add per-case start/end timestamps so the hang can be attributed to a specific case in test/server/test_server_1_gpu.py.
• On a reproduced hang, capture py-spy dump of the pytest/server process and rocm-smi + dmesg to distinguish a Python/model hang from a GPU/driver hang.
• Map the 3 failing jobs (27913714740, 27923265639, 27938642367) to runner hostnames to test the "bad host in pool" hypothesis.
• If a single case is implicated, consider a per-case watchdog/timeout in the suite so a hang produces a finalized log + artifact instead of a silent runner loss (improves future diagnosability).

Status

• Persistence: Intermittent — 3/7 most recent completed scheduled runs of this shard failed (Jun 21 18:35 → Jun 22 08:09 UTC); recovered Jun 22 13:56.
• Scope: This shard only (partition 1 of multimodal-gen-test-1-gpu-amd); sibling shards 0/2/3 passed in this run. Part of a broader, pre-existing timeout-flakiness pattern on multimodal-gen-test-1-gpu documented since 2026-04-09.
• Diagnosability gap: No test-level data — runner loss prevented log finalization and always() artifact upload, so the exact hung case is currently unknown.
• In-flight fix: ❌ none found in open PRs.

call-pr-test-amd-extra / extra-a-test-1-gpu-small-amd (linux-mi325-1gpu-sglang) — failed step(s): Install dependencies

Commit Info

• sglang: adc203df
• aiter: N/A — job was cancelled during the Install dependencies step before any [CI-AITER-CHECK] markers were emitted (0 found in log).

Failed Tests

Test File	Test Function	Error	Log
N/A (no test ran)	N/A	##[error]The operation was canceled. during dependency build (mori)	Log line 2842

No test file ran. The job was cancelled, not failed on a test. Step status: Start CI container ✅ → Install dependencies ❌ cancelled (21:51:40 → 21:53:08, mid-mori build) → Run test ⏭️ skipped.

Failure Summary

This is not a test failure. The job extra-a-test-1-gpu-small-amd was cancelled by GitHub Actions while still installing dependencies (building the mori library), ~7 minutes in. The Run test step never executed, so no SGLang test was exercised. Its sister job extra-a-test-1-gpu-large-amd in the same call-pr-test-amd-extra group was cancelled at the same instant (21:53:10Z).

Regression Status

Infrastructure issue (fast-fail cancellation) — NOT a code regression.

Recent history of job call-pr-test-amd-extra / extra-a-test-1-gpu-small-amd (branch main, event schedule, completed runs only):

Date	Run	Job Status	Note
Jun 22 19:10	27977371862	⏹️ Cancelled	this run — cancelled in Install dependencies, no test ran
Jun 22 13:56	27958083310	✅ Passed	—
Jun 22 08:09	27938642367	✅ Passed	—
Jun 22 01:00	27923265639	✅ Passed	—

The job passed in each of the 3 prior scheduled runs. No persistent or recurring failure for this job.

Failure Cluster

Fast-fail cancellation of call-pr-test-amd-extra jobs during dependency install (no test executed).

Facts (verified observations, no interpretation)

• FACT: Job conclusion = cancelled (verified via jobs API). Install dependencies step concluded cancelled at 2026-06-22T21:53:08Z; Run test step concluded skipped.
• FACT: Log line ##[error]The operation was canceled. appears while building mori ([18/141] Building CXX object ... offset_allocator.cpp.o) — the cancellation interrupted dependency compilation, not a test.
• FACT: The only non-warning log content is HIP/CXX compiler warnings (-Wpass-failed, -Wdeprecated-literal-operator) — these are benign build warnings, present in passing runs too, not errors.
• FACT: Two jobs in the same run failed before this cancellation: stage-b-test-large-8-gpu-mi35x-disaggregation-amd (completed 20:35:41Z, failure) and multimodal-gen-test-2-gpu-amd (…, 1) (completed 20:49:43Z, failure). The extra-a jobs were then cancelled at 21:53:10Z.
• FACT: Both call-pr-test-amd-extra / extra-a-* jobs were cancelled at the identical timestamp (21:53:10Z), and pr-test-amd-extra-finish failed at 21:53:17Z — consistent with a group-level cancellation, not independent failures.

Hypothesised Causes (with confidence)

• [HIGH] Fast-fail cascade: earlier job failures in run 27977371862 triggered cancellation of the still-running call-pr-test-amd-extra jobs.
  • Supporting: two jobs failed at 20:35 and 20:49, ~1 hour before the extra-a jobs were cancelled at 21:53; both extra-a jobs cancelled at the exact same timestamp; pr-test-amd-extra-finish failed immediately after. This is the classic fast-fail pattern documented in the PR-CI guidance.
  • Disconfirming: (none found) — the cancellation timestamps and prior failures align cleanly with a fast-fail cascade.
  • How to verify: inspect the call-gate/fast-fail logic for the call-pr-test-amd-extra reusable workflow; confirm it cancels pending jobs when an upstream PR-test job fails.
• [LOW] Runner/infra interruption (spot reclaim, timeout) independent of other failures.
  • Supporting: cancellation occurred mid dependency build.
  • Disconfirming: a generic runner kill would not cancel both extra-a jobs at the identical second nor coincide with the -finish job failure; the earlier same-run failures better explain it.
  • How to verify: check runner logs for the two extra-a runners around 21:53:08Z.

In-flight Fix Check

Not applicable — this is a transient fast-fail cancellation, not a defect requiring a code fix. No fix PR is warranted. (No keyword search performed because there is no failing test/library to key on.)

Suggested Triage Steps

• Treat this job result as no-signal, not a failure — the changed code was never tested here. Re-run the call-pr-test-amd-extra group (or wait for the next scheduled run) to obtain a real signal for the extra-a tests.
• Investigate the actual upstream failures that triggered the cascade instead: stage-b-test-large-8-gpu-mi35x-disaggregation-amd and multimodal-gen-test-2-gpu-amd (…, 1) — these are the real failing jobs in this run.
• If full extra-a signal is needed on a high-priority run despite unrelated early failures, consider the bypass-fastfail label (uses more CI resources; use sparingly).

Status (factual, no priority assignment)

• Persistence: One-off. This job passed in the 3 immediately prior scheduled runs; cancelled (not failed) only in this run.
• Scope: Both call-pr-test-amd-extra / extra-a-* jobs cancelled in run 27977371862; caused by ≥2 earlier job failures in the same run.
• Blocked work: extra-a 1-GPU small/large tests produced no signal for commit adc203df on this run.
• In-flight fix: N/A — transient cancellation, no code fix needed.

multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 1) — failed step(s): N/A

Commit Info

• sglang: adc203df
• aiter: N/A — the job log blob is unavailable (BlobNotFound); the runner died mid-step before uploading logs, so no [CI-AITER-CHECK] markers can be read.

Failed Tests

Test File	Test Function	Error	Log
N/A (infrastructure failure — not a test)	N/A	The self-hosted runner lost communication with the server	Job

No test-level granularity is recoverable: the runner lost communication during the Run diffusion server tests (1-GPU) step, so the step never finished and no log was uploaded (the log API returns BlobNotFound). This job runs sglang/multimodal_gen/test/run_suite.py --suite 1-gpu --partition-id 1 --total-partitions 4 -k "not flux_2" (pr-test-amd.yml:711-739).

Failure Summary

The job concluded failure because the self-hosted MI325 runner lost communication with GitHub ~54 min into the Run diffusion server tests (1-GPU) step (step timeout is 90 min, so this is not a step timeout). The failure check-run annotation is a GitHub infrastructure message, not a test assertion. No diffusion test result, stack trace, or log content is available.

Regression Status

Known recurring infrastructure failure (intermittent), not a code regression.

Recent history of multimodal-gen-test-1-gpu-amd (shard 1) in pr-test-amd.yml (branch main, event schedule, completed runs only):

Date (UTC)	Run	Job	Status	Failure type
Jun 22 21:43	27977371862	shard 1	❌ Failed	runner lost communication (~54 min)
Jun 22 13:56	27958083310	shard 1	✅ Passed	—
Jun 22 08:09	27938642367	shard 1	❌ Failed	runner lost communication (~53 min)
Jun 22 01:00	27923265639	shard 1	❌ Failed	runner lost communication (~55 min)
Jun 21 18:35	27913714740	shard 1	❌ Failed	runner lost communication (~53 min)
Jun 21 12:45	27904738325	shard 1	✅ Passed	—

First observed (in queryable window): Jun 21 18:35 UTC. Interspersed with passes on varying head SHAs → not SHA-deterministic.

Failure Cluster

Self-hosted MI325 runner lost communication during 1-GPU diffusion test (~54 min mark)

Facts (verified observations, no interpretation)

• Failure annotation (verbatim, [FACT]): "The self-hosted runner lost communication with the server. Verify the machine is running and has a healthy network connection. Anything in your workflow that terminates the runner process, starves it for CPU/Memory, or blocks its network access can cause this error."
• The job log API (/actions/jobs/82798487110/logs) returns BlobNotFound — no log was archived because the runner died mid-step.
• Step 9 Run diffusion server tests (1-GPU) is permanently in_progress in the job's step list; step timeout is 90 min, run lasted only 54 min → not a step timeout.
• All 4 recent failures of this shard crashed at a consistent 53-55 min elapsed, all during the same step, all with the identical "runner lost communication" annotation.
• The sister job multimodal-gen-test-2-gpu-amd (shard 1) (82798487115) failed differently — Process completed with exit code 1 (a real test/process failure), a separate cluster from this one.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [MEDIUM] Host resource starvation / OOM (or GPU hang) on the self-hosted MI325 runner causes the runner agent to be killed or starved, dropping its connection.
  • Supporting: GitHub's own message names "starves it for CPU/Memory" as a cause; all 4 failures crash at a remarkably consistent ~54 min into the same memory-heavy diffusion step; the workflow itself diagnoses memory/VRAM (free -h, rocm-smi) at pr-test-amd.yml:684-709, indicating a known resource-pressure context.
  • Disconfirming: no log was captured, so OOM/hang cannot be confirmed directly; the consistent timing could equally be a network/runner-host watchdog rather than OOM. The same shard passes on other runs, so it is not deterministic.
  • How to verify: re-run the shard and watch host dmesg/free -h/rocm-smi on the MI325 runner for OOM-killer or GPU-reset events around the 50-min mark; check runner-host system logs for the agent process exit.
• [LOW] Code regression in the diffusion 1-GPU suite at adc203df.
  • Supporting: the crash is inside the diffusion test step.
  • Disconfirming: failure is intermittent across varying head SHAs (passes at 27958083310 and 27904738325); a code bug would manifest as a test assertion / non-zero exit with an uploaded log, not "runner lost communication" with no log.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs (repo:sgl-project/sglang is:pr is:open multimodal diffusion runner, 16 hits); none address the runner-lost-communication / resource-starvation infra issue on the MI325 1-GPU diffusion shard.

Suggested Triage Steps

• Inspect the MI325 self-hosted runner host system logs (dmesg -T, OOM-killer, rocm-smi event log) for the failing runs around the 50-55 min mark to confirm/deny memory/GPU starvation.
• Re-run this shard (multimodal-gen-test-1-gpu-amd part 1) and monitor the in-step free -h / rocm-smi diagnostics already emitted by the workflow.
• If host memory pressure is confirmed, consider whether partition 1's model set (T2V/T2I/I2V/LoRA) peaks RAM/VRAM higher than the other shards — compare per-shard durations and memory.
• Treat as an infra/runner-health item, not a code revert; no commit bisect is warranted given the intermittent, log-less, SHA-independent pattern.

Status (factual, no priority assignment)

• Persistence: Intermittent — 3 of the last 6 completed scheduled runs of this shard failed identically (Jun 21 18:35, Jun 22 01:00, Jun 22 08:09, plus this run Jun 22 21:43), interspersed with passes.
• Scope: This specific job/shard (multimodal-gen-test-1-gpu-amd, part 1) on the linux-mi325-1gpu-sglang runner. The 2-GPU shard-1 failure in the same run is a different cluster (exit code 1, real process failure).
• Blocked work: Adds noise to AMD pr-test-amd scheduled CI signal for the 1-GPU diffusion suite; green/red is non-deterministic for this shard.
• In-flight fix: ❌ none found — needs runner-health investigation, not a code PR.

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-06-23 20:58 UTC)

[amd-bot]

pr-test-amd.yml — Per-Job Analysis (continued 3/5)

multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 1) — failed step(s): Run diffusion server tests (2-GPU)

Commit Info

• sglang: adc203df
• aiter: 7d604afe — v0.1.14rc1.dev240+g7d604afe5 (source: Dockerfile default; AITER_COMMIT_OVERRIDE empty)

Failed Tests

Test File	Test Function	Error	Log
python/sglang/multimodal_gen/test/server/test_server_2_gpu.py	TestDiffusionServerTwoGpu::test_diffusion_generation[flux2_modelopt_fp8_tp2_t2i]	openai.InternalServerError: Internal Server Error (client) ← server-side RuntimeError: CUDA error: HIPBLAS_STATUS_NOT_SUPPORTED in torch._scaled_mm	Log
python/sglang/multimodal_gen/test/server/test_server_2_gpu.py	TestDiffusionServerTwoGpu::test_diffusion_generation[ideogram4_fp8_tp2_t2i] (ERROR at setup)	huggingface_hub.errors.GatedRepoError: 403 for ideogram-ai/ideogram-4-fp8 → server exited code 1	Log

Overall shard result: 1 failed, 9 passed, 1 skipped, 1 error in 2554.86s.

Failure Summary

Two distinct, independent failures in the 2-GPU diffusion shard. (1) flux2_modelopt_fp8_tp2_t2i returned HTTP 500 to the client because the server's FP8 GEMM path raised HIPBLAS_STATUS_NOT_SUPPORTED from torch._scaled_mm during denoising (same fault hit at warmup and at the live request). (2) ideogram4_fp8_tp2_t2i errored at setup: the server could not download the gated HF repo ideogram-ai/ideogram-4-fp8 (403 Forbidden — CI token not in the authorized list).

Regression Status

Both: Known recurring failure (Never-passed in queryable window). Lookback: 11 completed schedule/main runs of pr-test-amd.yml from 2026-06-20 00:55 UTC through 2026-06-22 13:56 UTC — every one failed identically on this job for both tests. No green run found in the window.

Recent history of test_server_2_gpu.py (flux2_modelopt_fp8_tp2_t2i / ideogram4_fp8_tp2_t2i) in job multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 1) (same branch, same event, completed runs only):

Date	Run	Job	flux2 Status	ideogram4 Status	Cause (verified)
Jun 22 13:56	27958083310	job	❌ FAILED	❌ ERROR	HIPBLAS_NOT_SUPPORTED / GatedRepo 403
Jun 22 08:09	27938642367	job	❌ FAILED	❌ ERROR	HIPBLAS_NOT_SUPPORTED / GatedRepo 403
Jun 22 01:00	27923265639	job	❌ FAILED	❌ ERROR	HIPBLAS_NOT_SUPPORTED / GatedRepo 403

(The current run 27977371862 is overall queued, but this job 82798487115 completed with exit 1.)

Failure Cluster

• flux2: "ROCm FP8 torch._scaled_mm HIPBLAS_STATUS_NOT_SUPPORTED in diffusion DiT"
• ideogram4: "Gated HF repo 403 during model download at server startup"

Facts (verified observations)

• flux2 client error: openai.InternalServerError: Internal Server Error, raised at test_server_utils.py:1041 (generate_image).
• flux2 server stack (verbatim top frames): runtime/layers/linear.py:427 → runtime/layers/quantization/modelopt_quant.py:425 (apply_fp8_linear) → python/sglang/srt/layers/quantization/fp8_utils.py:1695 → torch._scaled_mm(...) → RuntimeError: CUDA error: HIPBLAS_STATUS_NOT_SUPPORTED. fp8_utils.py:1695 is the per-tensor weights + per-tensor activations branch; the checkpoint config logged dynamic: False, type: float, num_bits: 8 (static per-tensor FP8), consistent with that branch.
• flux2 fault first occurs at warmup ([denoising_step_0] Error ... HIPBLAS_STATUS_NOT_SUPPORTED) and again on the live request → both produce 500.
• ideogram4: httpx.HTTPStatusError: 403 Forbidden for https://huggingface.co/ideogram-ai/ideogram-4-fp8/... → GatedRepoError: Access to model ideogram-ai/ideogram-4-fp8 is restricted and you are not in the authorized list, raised at runtime/utils/hf_diffusers_utils.py:853 (maybe_download_model); worker died → RuntimeError: Server exited early (code 1) at test_server_utils.py:472.
• Both failures reproduce identically in the 3 prior completed runs inspected; flux2 HIPBLAS error confirmed verbatim in jobs 82731802132 and 82620731555.

Hypothesised Causes (with confidence)

• [FACT] ideogram4: the test model repo ideogram-ai/ideogram-4-fp8 is gated and the CI HF token lacks access → 403 at download. Reproducible every run; not a code regression. Disconfirming: none — it is a direct HTTP 403 in the log.
• [HIGH] flux2: ROCm/hipBLAS does not support the per-tensor FP8 torch._scaled_mm call at fp8_utils.py:1695 for this flux2 modelopt-fp8 checkpoint on MI325 (gfx942).
  • Supporting: stack trace lands exactly on torch._scaled_mm in the per-tensor FP8 branch; error is the hipBLAS-specific HIPBLAS_STATUS_NOT_SUPPORTED; only AMD CI is affected; an open AMD PR (#28889) targets this exact line with a dequant→bf16 fallback.
  • Disconfirming: not yet bisected to the commit that introduced flux2 modelopt-fp8 to this AMD suite — the test appears Never-passed in the queryable window, so this may be an unported path rather than a fresh regression.
  • How to verify: apply #28889 on a branch and re-run test_diffusion_generation[flux2_modelopt_fp8_tp2_t2i] on MI325.

In-flight Fix Check

• flux2 HIPBLAS: ✅ #28889 "[AMD] Fix FP8 e4m3fn GEMM on ROCm: dequant->bf16 fallback in apply_fp8_linear (fixes flux2 modelopt-fp8 HIPBLAS_STATUS_NOT_SUPPORTED)" (open, non-draft, since 2026-06-22) — directly targets this failure; chase review, do NOT open a duplicate.
• ideogram4 gated repo: ✅ #28225 "Route Ideogram NF4 without gated metadata fetch" (open, draft, since 2026-06-15) — candidate fix for the gated-metadata 403; may not fully cover the gated weight download. (Searched: flux2+modelopt, HIPBLAS_STATUS_NOT_SUPPORTED, ideogram.)

Suggested Triage Steps

• flux2: verify #28889 merges cleanly, apply on a branch, re-run flux2_modelopt_fp8_tp2_t2i on MI325 container; if green, push reviewers to merge.
• ideogram4: decide whether to (a) grant the CI HF token access to ideogram-ai/ideogram-4-fp8, or (b) land/finish draft #28225 to avoid the gated fetch, or (c) skip/quarantine ideogram4_fp8_tp2_t2i on AMD CI until access is provisioned.
• If neither fix is imminent, consider quarantining both scenarios on AMD CI so this shard stops masking other regressions.

Status (factual)

• Persistence: Both failing every completed run for ≥2.5 days (since at least 2026-06-20 00:55 UTC); no green run in the queryable window (Never-passed).
• Scope: This job (multimodal-gen-test-2-gpu-amd, partition 1) on pr-test-amd.yml; AMD-specific (flux2 HIPBLAS is ROCm-only; ideogram is access-only and would affect any runner without repo access).
• Blocked work: flux2 modelopt-fp8 and ideogram-4-fp8 diffusion coverage on AMD 2-GPU.
• In-flight fix: flux2 → ✅ #28889; ideogram4 → ✅ (draft) #28225.

stage-b-test-1-gpu-large-amd (linux-mi325-1gpu-sglang, 1) — failed step(s): Run test

Commit Info

• sglang: adc203df
• aiter: v0.1.14rc1.dev240+g7d604afe5 (7d604afe) — source: Dockerfile default; dev/patched image preserved (no AITER_COMMIT_OVERRIDE)

Failed Tests

Test File	Test Function	Error	Log
(job-level) Run test action	N/A	The action 'Run test' has timed out after 45 minutes — only 3/7 registered test files completed	Log
test/registered/perf/test_bench_serving_1gpu_part2.py	test_score_api_latency_throughput	AssertionError: 56.7309797508642 not less than 50 (p95_latency_ms), triggered a retry	Log

Failure Summary

The Run test step hit the 45-minute action timeout and was killed mid-way through the 4th of 7 registered test files (it was loading weights for test_qwen2_5_vl at termination). The dominant time sink was test/registered/perf/test_bench_serving_1gpu_part2.py, which alone consumed 2057s (vs estimated_time=900s). Inside that file, test_score_api_latency_throughput failed its p95-latency assertion (56.73ms vs the <50ms threshold) and triggered a full retry (a fresh server relaunch), adding further wall-clock time.

Regression Status

New regression (this run only) — [FACT] the same job passed in the previous 4 completed scheduled runs.

Recent history of job stage-b-test-1-gpu-large-amd (linux-mi325-1gpu-sglang, 1) (branch=main, event=schedule, completed runs only):

Date	Run	Job Status	Note
Jun 22 19:10	27977371862	❌ Failed	Timeout; part2 elapsed=2057s
Jun 22 13:56	27958083310	✅ Passed	part2 elapsed=1452s
Jun 22 08:09	27938642367	✅ Passed	—
Jun 22 01:00	27923265639	✅ Passed	—
Jun 21 18:35	27913714740	✅ Passed	—

Failure Cluster

Job-level 45-minute timeout driven by perf-suite overrun in test_bench_serving_1gpu_part2.py (+ p95 latency threshold breach).

Facts (verified observations)

• [FACT] Job conclusion: failure; failing step is step 6 Run test, killed by ##[error]The action 'Run test' has timed out after 45 minutes (log line 34468).
• [FACT] At timeout, the runner was at Begin (3/6) → test_piecewise_cuda_graph_support_1_gpu.py had completed and it was loading weights for a Qwen2.5-VL test; only files 0,1,2 fully completed (End (0/6), End (1/6), End (2/6) present).
• [FACT] test_bench_serving_1gpu_part2.py elapsed=2057s this run vs 1452s in the last passing run (82731802726) — both already exceeded estimated_time=900s.
• [FACT] test_score_api_latency_throughput failed assertLess(p95_latency_ms, 50) with 56.73ms, then retry() relaunched the server (Qwen/Qwen3-Reranker-0.6B) and re-ran the method. This assertion did NOT fire in the last passing run (no not less than in its log).
• [FACT] aiter is the dev image baked into the container (v0.1.14rc1.dev240+g7d604afe5), unchanged vs passing runs; AITER_COMMIT_OVERRIDE empty.
• [FACT] 13 sglang commits in window 4923bb93ae..adc203df.

Hypothesised Causes (with confidence)

• [LOW] Perf/runner-contention flake. The p95-latency assertion missed by a small margin (56.73 vs 50ms) and part2 was already at ~1452s (over its 900s estimate) in the last green run; a modest slowdown plus the retry's extra server relaunch pushed total over the 45-min wall.
  • Supporting: threshold breach is marginal; the previous run was already near the time budget; perf thresholds on shared MI325 runners are noisy; no code in the window obviously touches the bench-serving hot path.
  • Disconfirming: part2's jump of +605s is large for pure noise — could indicate a real slowdown.
  • How to verify: re-run the job; if green and part2 ≈1450s, confirm flake. If part2 stays >1800s, investigate a serving regression.
• [LOW] sglang commit in window slowed inference / startup. 13 commits merged in 4923bb93ae..adc203df; b28e9901 (ServerArgs migration to Annotated) touches arg parsing in every server launch path that part2 exercises repeatedly.
  • Supporting: temporal correlation; part2 launches many servers, so any per-launch overhead compounds.
  • Disconfirming: no code-level evidence it adds runtime cost; arg parsing is one-time per launch and cheap; the same commit was present and the job still passed in the immediate prior schedule? — No: it landed in this window, so it is a genuine candidate but unverified.
  • How to verify: bisect 4923bb93ae..adc203df, measuring part2 server-launch time.

No hypothesis above LOW confidence; this is most consistent with a perf/timeout flake on an already time-tight job.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for test_bench_serving_1gpu_part2 / score-latency / stage-b-test-1-gpu-large-amd timeout; no matching fix. (queries: search/issues?q=...+test_bench_serving_1gpu_part2..., ...+stage-b-test-1-gpu-large-amd...)

Suggested Triage Steps

• Re-run the job once; if green with part2 ≈1450s, classify as a perf/timeout flake (no code action).
• If part2 consistently exceeds ~1800s, bisect 4923bb93ae..adc203df measuring per-server-launch time, with b28e9901 (ServerArgs migration) as first suspect.
• Consider splitting test_bench_serving_1gpu_part2.py (already ~1450s of a 45-min, 7-file budget in the green case) or raising the action timeout — the job has little headroom and any single slow file tips it over.
• Investigate why test_score_api_latency_throughput p95 exceeds 50ms on MI325; the retry-on-failure doubles its cost and compounds the timeout.

Status

• Persistence: First failure in this job — failed only in run 27977371862 (Jun 22 19:10 UTC); previous 4 completed scheduled runs passed.
• Scope: Single job stage-b-test-1-gpu-large-amd (linux-mi325-1gpu-sglang, 1) in pr-test-amd.yml.
• Blocked work: Files 3–6 of this shard (Qwen2.5-VL eagle, piecewise CUDA graph follow-ons, gdn/lightning attention unittests, decode-set-metadata) were never reached — no signal for them this run.
• In-flight fix: ❌ none found.

stage-b-test-large-8-gpu-mi35x-disaggregation-amd (linux-mi35x-gpu-8.fabric) — failed step(s): Run test_disaggregation

Commit Info

• sglang: adc203df
• aiter: v0.1.14rc1.dev240+g7d604afe5 (source: dev/patched image preserved — [CI-AITER-CHECK] Dev/patched version detected → skipping rebuild; Dockerfile default was 7d604afe5). No AITER_COMMIT_OVERRIDE — this is a regular pr-test-amd.yml schedule run, not a scout run.

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/disaggregation/test_disaggregation_basic.py	TestDisaggregationAccuracy.test_gsm8k	KeyError: 'answer' → Exception: retry() exceed maximum number of retries (server returned HTTP 500 KVTransferError ... timed out after 300.0s in KVPoll.WaitingForInput)	Log
test/registered/amd/disaggregation/test_disaggregation_pp.py	TestDisaggregationDecodePPAccuracy.setUpClass	RuntimeError: Server process exited with code -9 (preceded by RuntimeError: The memory capacity is unbalanced ... pre_model_load_memory=279.6 + RDMA Failed to register memory: Cannot allocate memory [12])	Log
test/registered/amd/disaggregation/test_mori_transfer_engine_e2e.py	TestMoriTransferEngineE2E.setUpClass	RuntimeError: Server process exited with code -9 (preceded by torch.OutOfMemoryError: HIP out of memory)	Log

Failure Summary

The stage-b-test-large-8-gpu-mi35x-disaggregation-amd job ran 3 disaggregation test files (Test Summary: 0/3 passed) and all 3 failed in step 9 Run test_disaggregation. In test_disaggregation_basic.py, every /generate request returned HTTP 500 with KVTransferError ... timed out after 300.0s in KVPoll.WaitingForInput, so run_eval_few_shot_gsm8k got no answer field and raised KeyError: 'answer'. The two subsequent files (test_disaggregation_pp.py, test_mori_transfer_engine_e2e.py) failed at setUpClass because their servers were SIGKILLed (exit -9) under memory pressure.

Regression Status

New regression — this job passed in the previous 5 completed scheduled runs and failed for the first time in this run.

Recent history of test_disaggregation_basic.py (TestDisaggregationAccuracy.test_gsm8k) in job stage-b-test-large-8-gpu-mi35x-disaggregation-amd (branch main, event=schedule, completed runs only):

Date (UTC)	Run	Job	Test File Status	Failed Function	Error
Jun 22 19:10	27977371862	…mi35x-disaggregation-amd	❌ Failed	test_gsm8k	KeyError: 'answer' / KV transfer timeout
Jun 22 13:56	27958083310	…mi35x-disaggregation-amd	✅ Passed	—	—
Jun 22 08:09	27938642367	…mi35x-disaggregation-amd	✅ Passed	—	—
Jun 22 01:00	27923265639	…mi35x-disaggregation-amd	✅ Passed	—	—
Jun 21 18:35	27913714740	…mi35x-disaggregation-amd	✅ Passed	—	—
Jun 21 12:45	27904738325	…mi35x-disaggregation-amd	✅ Passed	—	—

First observed failure: 2026-06-22 19:10 UTC. Last known pass: run 27958083310 (head 4923bb93ae) at 2026-06-22 13:56 UTC. Regression window: 4923bb93ae..adc203df (13 commits).

Failure Cluster

RDMA memory registration failure (ENOMEM) in mooncake PD KV transfer on MI35x

Facts (verified observations)

• All 3 test files failed in step 9; Test Summary: 0/3 passed; the job step exited 255 and was not retried (Exit code 255 indicates test failure (not network issue), not retrying).
• test_disaggregation_basic.py: server returned HTTP 500 Decode transfer failed ... KVTransferError(bootstrap_room=...): Request ... timed out after 300.0s in KVPoll.WaitingForInput for many requests; client then raised KeyError: 'answer'. The basic test ran 1054 s (elapsed=1054).
• RDMA registration failures appear at server startup, BEFORE any KV transfer: at 20:17:16 (during the prefill/decode server launch of the FIRST test) the log shows E rdma_context.cpp:243] Failed to register memory 0x...: Cannot allocate memory [12] (errno 12 = ENOMEM). The same RDMA error recurs in the test_disaggregation_pp.py server (Failed to register memory ... Cannot allocate memory [12]).
• test_disaggregation_pp.py: RuntimeError: The memory capacity is unbalanced. Some GPUs may be occupied by other processes. pre_model_load_memory=279.6..., local_gpu_memory=972.8... then Received sigquit from a child process → Server process exited with code -9.
• test_mori_transfer_engine_e2e.py: torch.OutOfMemoryError: HIP out of memory. Tried to allocate 16.50 GiB. GPU 1 has a total capacity of 287.98 GiB of which 15.76 GiB is free → exit -9.
• Transfer backend in use is mooncake (disaggregation_transfer_backend='mooncake' in the logged server_args), with disaggregation_ib_device='rdma0..rdma7'.
• The 13-commit regression window contains no changes to the disaggregation / mooncake / RDMA KV-transfer code path:
  • adc203df (HEAD) touches only sgl-model-gateway/src/routers/http/pd_router.rs (the Rust gateway). These tests use launch_lb() (mini load balancer), not the Rust gateway — the changed code is not exercised.
  • 43b1fb95 edits only scripts/ci/cuda/ci_install_dependency.sh (CUDA install path; AMD jobs use amd_ci_install_dependency.sh).
  • b28e9901 is a mechanical ServerArgs→Annotated refactor; the disaggregation defaults (mooncake, ib_device None, bootstrap_port, etc.) are unchanged, and it ships test_server_args_migration.py to assert equivalence.
  • 99c18cce and 34e5e386 are docs-only.

Hypothesised Causes (with confidence)

• [MEDIUM] Host-level RDMA resource exhaustion on the linux-mi35x-gpu-8.fabric runner — mooncake's transfer engine cannot pin/register RDMA memory (Failed to register memory: Cannot allocate memory [12], errno 12 ENOMEM), so the prefill server never completes KV transfer and the decode side times out at 300 s in KVPoll.WaitingForInput.
  • Supporting: The RDMA Cannot allocate memory [12] error fires at server startup of the first test, before any traffic, and recurs across multiple tests; the user-visible failure is a transfer timeout (not a numerical/accuracy mismatch); errno 12 on ibv_reg_mr is a classic locked-memory (memlock ulimit) / pinned-host-memory exhaustion symptom; no code in the regression window touches the transfer path.
  • Disconfirming: This is a NEW failure after 5 consecutive passes, which would normally suggest a code change rather than a static infra limit — but no commit in the window touches disaggregation/RDMA/mooncake, weakening a code-regression explanation and favoring a transient/host resource issue (e.g. leaked pinned memory or another process on the shared 8-GPU node).
  • How to verify: re-run the job on the same runner; check ulimit -l (memlock) and nvidia-smi/rocm-smi for leftover processes inside the container at startup; inspect host pinned-memory usage. If it reproduces deterministically, bisect 4923bb93ae..adc203df.
• [LOW] Cascade-only failures for test_disaggregation_pp.py and test_mori_transfer_engine_e2e.py — these failed within 35 s / 45 s with exit -9 and "GPUs may be occupied by other processes" / HIP OOM, consistent with GPU/host memory not being released after the 1054 s basic test rather than an independent root cause.
  • Supporting: pre_model_load_memory=279.6 GB already occupied before model load in the pp test; both failed at setUpClass (server never came up), not in test logic.
  • Disconfirming: the pp server also shows its own RDMA Cannot allocate memory [12], so it may share the same root ENOMEM rather than being a pure leftover-memory cascade.
  • How to verify: run each test file in isolation on a clean node and confirm they pass when not preceded by the basic test.
• [LOW] Code regression in the 13-commit window.
  • Supporting: failure newly appeared at head adc203df.
  • Disconfirming: none of the 13 commits modify the disaggregation/mooncake/RDMA code actually exercised by these tests (see Facts).
  • How to verify: bisect the window only after the infra hypothesis is ruled out.

In-flight Fix Check

• In-flight fix: ❌ none found directly targeting RDMA Cannot allocate memory on MI35x disaggregation. Searches run: is:pr is:open disaggregation mi35x (15 hits, all NIXL/feature work) and is:pr is:open rdma register memory (18 hits, none matching this ENOMEM symptom).
• Note: open PR #27012 Revert "[AMD] Fix stage-b-test-large-8-gpu-mi35x-disaggregation-amd" targets this exact job name and indicates prior stability churn — worth checking with the AMD CI owners, though it is not a fix for this specific RDMA failure.

Suggested Triage Steps

• Re-run the job on linux-mi35x-gpu-8.fabric; if it passes, treat as a transient host RDMA/memory resource issue and capture ulimit -l, host pinned-memory, and rocm-smi at startup for the next occurrence.
• Inspect the runner for leftover processes / unreleased GPU memory before the job starts (the pre_model_load_memory=279.6 GB and "GPUs may be occupied by other processes" message suggest a dirty node).
• Run test_disaggregation_pp.py and test_mori_transfer_engine_e2e.py standalone to confirm they are cascade victims of the basic test rather than independent failures.
• Only if the failure reproduces deterministically across clean re-runs, bisect 4923bb93ae..adc203df — but the changed code in this window does not touch the exercised transfer path, so a code regression is unlikely.

Status

• Persistence: First failing run; passed the previous 5 consecutive scheduled runs (last pass 2026-06-22 13:56 UTC).
• Scope: 1 job (stage-b-test-large-8-gpu-mi35x-disaggregation-amd) in pr-test-amd.yml; 3 disaggregation test files (0/3 passed).
• Blocked work: Blocks the MI35x 8-GPU PD-disaggregation CI signal (mooncake transfer path).
• In-flight fix: ❌ none found targeting this RDMA ENOMEM symptom; related job-stability PR #27012 exists.

stage-c-test-large-8-gpu-amd (linux-mi325-8gpu-sglang, 2) — failed step(s): Run test

Commit Info

• sglang: adc203df
• aiter: 7d604afe5 (v0.1.14rc1.dev240+g7d604afe5, source: dev/patched image preserved — [CI-AITER-CHECK] reported "Dev/patched version detected → skipping rebuild")

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/test_kimi_k2_instruct.py	TestKimiK2Instruct0905.setUpClass	TimeoutError: Server failed to start within the timeout period	Log

Failure Summary

The single test in this partition, test/registered/amd/test_kimi_k2_instruct.py, failed in setUpClass: popen_launch_server raised TimeoutError: Server failed to start within the timeout period after the full 3600s budget (SERVER_LAUNCH_TIMEOUT = 3600, test elapsed 3626s). The server (TP=8, moonshotai/Kimi-K2-Instruct-0905) never reached ready state — Ran 0 tests in 3609.848s.

Regression Status

New regression in this run ([IN-FLIGHT] overall run — other jobs still queued; this job itself completed and failed). The same test file passed in the immediately-prior completed scheduled run.

Recent history of test_kimi_k2_instruct.py (TestKimiK2Instruct0905) in the stage-c-test-large-8-gpu-amd (linux-mi325-8gpu-sglang) matrix (branch main, event schedule, completed runs only; partition id is auto-assigned so the suite's overall result is used per Ground Rules):

Date	Run	Job	Test File Status	Failed Function	Error
Jun 23	27977371862	…8-gpu-amd (…, 2)	❌ Failed	setUpClass	TimeoutError: Server failed to start (elapsed 3626s)
Jun 22 13:56	27958083310	…8-gpu-amd (…, 2)	✅ Passed	—	1/1 passed, elapsed 1281s
Jun 22 08:09	27938642367	…8-gpu-amd (…, 2)	✅ Passed	—	suite green
Jun 22 01:00	27923265639	…8-gpu-amd (…, 2)	✅ Passed	—	suite green

First observed failure for this test file: 2026-06-23 (this run). Last known pass: 2026-06-22 13:56 UTC, where it completed in 1281s (vs 3626s here).

Failure Cluster

Server launch timeout during Kimi-K2 weight download (cold HF cache)

Facts (verified observations, no interpretation)

• TimeoutError: Server failed to start within the timeout period, raised at popen_launch_server (python/sglang/test/test_utils.py:1028), invoked from test_kimi_k2_instruct.py:45 setUpClass (verbatim traceback, log line 12897–12904).
• All 8 TP scheduler processes were stuck in weight download. py-spy dumps (e.g. TP5, log ~7960) show the stack: download_weights_from_hf (model_loader/weight_utils.py:554) → __enter__ (filelock/_api.py:581) → acquire (filelock/_api.py:506) → _poll_until_acquired (filelock/_api.py:552) → select (libc.so.6) — i.e. blocked acquiring the HF download file lock.
• At 01:14:16 (≈55 min in) the process was still inside hf_hub_download (huggingface_hub/file_download.py:1016) — the model download had not finished (log line 12880–12888).
• Soft watchdog fired repeatedly across TP ranks: Scheduler watchdog timeout (self.watchdog_timeout=300, self.soft=True); py-spy dump … Failed to get stack traces / failed to get os threadid recurred every 5 min (non-fatal).
• Launch config: --tp 8 --decode-attention-backend triton --prefill-attention-backend aiter, HF_HUB_OFFLINE=0 (download permitted), model moonshotai/Kimi-K2-Instruct-0905 (~1T-param checkpoint).
• Comparison: prior scheduled run 27958083310 ran the identical test on the same runner label and passed in 1281s (warm weights cache); this run consumed the full 3626s without the server ever starting.
• Runner instance differs: linux-mi325-8gpu-sglang-lmwqt-runner-jwxk2 ("Warning: could not parse GPU family from …", log line ~7360). Also an unrelated infra note: Local docker registry pull failed; falling back to public registry (log line 1288) — image pull succeeded via fallback.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [MEDIUM] Cold/incomplete HuggingFace weight cache on this runner instance → the full Kimi-K2-Instruct-0905 download did not finish within the 3600s SERVER_LAUNCH_TIMEOUT.
  • Supporting: every TP rank is blocked in download_weights_from_hf filelock acquire; at 55 min the process is still inside hf_hub_download; the prior run on a warm cache loaded the same model in 1281s — a 2.8× delta consistent with a from-scratch ~1TB download vs a cache hit.
  • Disconfirming: the exact bytes-downloaded / network throughput is not printed in the log, so "download too slow" vs "download stalled/deadlocked on the filelock" cannot be distinguished from the log alone. No "out of disk" or HTTP error was emitted.
  • How to verify: re-run the partition-2 job on the same runner; if it now passes quickly the weights are cached, confirming cold-cache. Alternatively inspect the runner's HF cache dir for a stale .lock / partial .incomplete file for this model.
• [LOW] Stale/abandoned HF download .lock left by an earlier process, causing the holder never to release and waiters to block indefinitely.
  • Supporting: stack is specifically in _poll_until_acquired (filelock), not in active byte transfer, across all ranks.
  • Disconfirming: one rank must hold the lock and be actively downloading for this to be benign-slow rather than deadlocked; the 01:14 dump shows an active hf_hub_download thread, which argues a real (slow) download is in progress, not a pure deadlock.
  • How to verify: check the runner's HF cache for a lingering lock file with no live owner; clear cache and re-run.
• [LOW] Code regression in this commit's weight-loading / server-startup path.
  • Supporting: failure is at server startup.
  • Disconfirming: strong — the identical test on the identical commit family passed in the previous 3 scheduled runs (last green 2026-06-22 13:56 UTC), and the failure is environmental (download/filelock), not a code exception. No traceback in sglang model code.
  • How to verify: not actionable until cache hypotheses are ruled out.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs kimi+timeout (13 results, none about CI server-launch timeout for this test) and server failed to start timeout (86 results, all unrelated PD/sampling/mooncake work). No PR addresses Kimi-K2 weight-download/server-launch timeout in AMD CI.

Suggested Triage Steps

• Re-run only the stage-c-test-large-8-gpu-amd (…, 2) partition; if it passes on the now-warm cache, this is a transient cold-cache/download-slowness flake on runner …-jwxk2, not a code issue.
• Inspect the runner's HF cache (~/.cache/huggingface/.../moonshotai--Kimi-K2-Instruct-0905) for a stale .lock or *.incomplete shard; clear and re-run to rule out a stuck filelock.
• If cold-cache downloads of this ~1TB model routinely approach the 3600s budget on fresh runners, consider (maintainer decision) pre-staging/persisting the Kimi-K2 weights on the 8-GPU runner pool or raising SERVER_LAUNCH_TIMEOUT for this specific test — investigate, do not assume.
• If a re-run reproduces the failure on a warm cache, escalate to a code-level startup investigation (would contradict the environmental hypothesis).

Status

• Persistence: First failure for this test file at 2026-06-23 (this run); last green 2026-06-22 13:56 UTC (passed in 1281s). 1/4 recent completed observations failed.
• Scope: Single job / single test (stage-c-test-large-8-gpu-amd (linux-mi325-8gpu-sglang, 2)); sibling stage-c 8-GPU partitions (0,1,3) and the mi35x matrix passed in prior runs. Overall run 27977371862 still has queued jobs [IN-FLIGHT].
• Blocked work: Kimi-K2-Instruct-0905 large-model accuracy/speed signal on AMD MI325 8-GPU.
• In-flight fix: ❌ none found — needs re-run to confirm transient, else new investigation.

multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 1) — failed step(s): Run diffusion server tests (2-GPU)

Commit Info

• sglang: ba9d5aed9 — "Fix nightly CI test for Kimi K2.5 INT4 + H200 (#28746)"
• aiter: 7d604afe5 (v0.1.14rc1.dev240+g7d604afe5) — source: dev image preserved (AITER_COMMIT_OVERRIDE empty; [CI-AITER-CHECK] Dev/patched version detected → skipping rebuild)

Failed Tests

Test File	Test Function	Error	Log
sglang/multimodal_gen/test/server/test_server_2_gpu.py	TestDiffusionServerTwoGpu::test_diffusion_generation[flux2_modelopt_fp8_tp2_t2i]	RuntimeError: CUDA error: HIPBLAS_STATUS_NOT_SUPPORTED in torch._scaled_mm	Log
sglang/multimodal_gen/test/server/test_server_2_gpu.py	TestDiffusionServerTwoGpu::test_diffusion_generation[ideogram4_fp8_tp2_t2i]	httpx.HTTPStatusError: 403 Forbidden / GatedRepoError (error at setup)	Log

Shard result: 1 failed, 9 passed, 1 skipped, 12 deselected, 1 error.

Failure Summary

Two parametrizations of test_diffusion_generation in test_server_2_gpu.py failed. flux2_modelopt_fp8_tp2_t2i (model black-forest-labs/FLUX.2-dev + lmsys/flux2-dev-modelopt-fp8-sglang-transformer, tp=2) failed in the denoising step: every torch._scaled_mm call raised HIPBLAS_STATUS_NOT_SUPPORTED, so server warmup and the actual generation request both produced no output (HTTP 500). ideogram4_fp8_tp2_t2i errored during setup because ideogram-ai/ideogram-4-fp8 is a gated HuggingFace repo returning 403 Forbidden; the model never loaded. The other 9 executed tests passed and 1 was skipped (qwen_image_t2i_2_gpus).

Regression Status

Known recurring failure (both tests) — no green run found for either test in the queryable window.

Recent history of test_server_2_gpu.py (test_diffusion_generation[flux2_modelopt_fp8_tp2_t2i] and [ideogram4_fp8_tp2_t2i]) in job multimodal-gen-test-2-gpu-amd (…, 1) (branch=main, event=schedule, completed runs only):

Date (UTC)	Run	flux2_modelopt_fp8	ideogram4_fp8
Jun 22 13:56	27958083310	❌ FAILED	❌ ERROR (setup)
Jun 22 08:09	27938642367	❌ FAILED	❌ ERROR (setup)
Jun 22 01:00	27923265639	❌ FAILED	❌ ERROR (setup)
Jun 21 18:35	27913714740	❌ FAILED	❌ ERROR (setup)
Jun 21 12:45	27904738325	❌ FAILED	❌ ERROR (setup)
Jun 21 07:43	27897611658	❌ FAILED	❌ ERROR (setup)
Jun 21 01:00	27889118263	❌ FAILED	❌ ERROR (setup)
Jun 20 12:42	27871498928	❌ FAILED	❌ ERROR (setup)
Jun 20 07:26	27864213328	❌ FAILED	❌ ERROR (setup)

Both tests failed identically in all 9 completed scheduled runs queried (Jun 20 07:26 → Jun 22 13:56). First-observed failure is at or before the start of this lookback window; no passing run exists in it. Current run 27994433780 and the next 27977371862 were still queued at snapshot — excluded from history above.

Failure Cluster

• flux2: HIPBLAS_STATUS_NOT_SUPPORTED in fp8 torch._scaled_mm (ROCm)
• ideogram4: Gated HF repo 403 at model setup

Facts (verified observations, no interpretation)

• flux2 stack trace (verbatim, run log): modelopt_quant.py:425 apply → fp8_utils.py:1695 apply_fp8_linear → output = torch._scaled_mm(...) → RuntimeError: CUDA error: HIPBLAS_STATUS_NOT_SUPPORTED when calling HIPBLAS_STATUS_NOT_SUPPORTED. Error repeats on warmup (Synthetic server warmup failed; continuing startup) and on the real request (POST /v1/images/generations → 500), which the test surfaces as Model generation returned no output.
• fp8_utils.py:1695 is the per-tensor-weights/per-tensor-activations _scaled_mm branch (confirmed by reading the file at this commit).
• ideogram4 error is at pytest setup, not in model code: Cannot access gated repo for url https://huggingface.co/ideogram-ai/ideogram-4-fp8/resolve/main/model_index.json → huggingface_hub.errors.GatedRepoError: 403 Client Error → ValueError: Model or revision not found at ideogram-ai/ideogram-4-fp8 … ensure you have access.
• aiter unchanged: no override, dev image preserved at v0.1.14rc1.dev240+g7d604afe5. Not a scout run (workflow pr-test-amd.yml), so no aiter A/B applies.
• Reproducibility: both failures in 9/9 completed runs over ~2.3 days, same shard, same symptoms.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [MEDIUM] flux2: torch._scaled_mm with e4m3fn fp8 operands is not supported by the hipBLAS path in this ROCm/MI325 image, so the modelopt-fp8 transformer GEMM raises HIPBLAS_STATUS_NOT_SUPPORTED.
  • Supporting: error is raised exactly at the _scaled_mm call in apply_fp8_linear; only the fp8 modelopt test fails while bf16/other diffusion tests on the same shard pass; an open AMD PR (#28889) titled "Fix FP8 e4m3fn GEMM on ROCm: dequant->bf16 fallback in apply_fp8_linear (fixes flux2 modelopt-fp8 HIPBLAS_STATUS_NOT_SUPPORTED)" describes this exact path.
  • Disconfirming: no green run exists in the queryable window, so a specific regressing sglang commit cannot be isolated from history alone — this may be a never-supported path on this hardware rather than a regression.
  • How to verify: apply #28889 on a branch and re-run test_diffusion_generation[flux2_modelopt_fp8_tp2_t2i] on MI325 2-GPU.
• [FACT] ideogram4: the test cannot run because the CI HF token lacks access to gated repo ideogram-ai/ideogram-4-fp8 (403). This is an access/infrastructure precondition, not an sglang code defect.
  • Supporting: GatedRepoError: 403 at setup before any model code runs; "you are not in the authorized list".
  • Disconfirming: (none — the 403 is directly observable in the log).
  • How to verify: grant the CI HF account access to ideogram-ai/ideogram-4-fp8, or route metadata without the gated fetch (see #28225), then re-run.

In-flight Fix Check

• flux2: ✅ #28889 (open, ready, since 2026-06-22) — "[AMD] Fix FP8 e4m3fn GEMM on ROCm: dequant->bf16 fallback in apply_fp8_linear (fixes flux2 modelopt-fp8 HIPBLAS_STATUS_NOT_SUPPORTED)". Directly targets this failure — chase reviewers, do NOT open a duplicate. (Searched flux2+modelopt+fp8 and HIPBLAS+scaled_mm.)
• ideogram4: ⚠️ #28225 (open, draft, since 2026-06-15) — "Route Ideogram NF4 without gated metadata fetch" partially relevant; also #28727 "[Diffusion] Ideogram support bf16 dtype". Neither directly fixes the gated-access 403; root cause is CI HF permissions. (Searched ideogram.)

Suggested Triage Steps

• flux2: verify #28889 merges cleanly, then re-run test_diffusion_generation[flux2_modelopt_fp8_tp2_t2i] on MI325 2-GPU to confirm the dequant→bf16 fallback resolves the HIPBLAS error.
• ideogram4: confirm whether the CI HuggingFace account should have access to ideogram-ai/ideogram-4-fp8; if access cannot be granted, either skip/gate this parametrization on AMD CI or land a non-gated metadata route (#28225).
• Because no green run exists in the window for either test, treat both as known/never-passing on this shard rather than bisecting a regression window.

Status

• Persistence: both failing in every completed scheduled run from Jun 20 07:26 UTC through Jun 22 13:56 UTC (9/9 runs); no passing run in the queryable window.
• Scope: multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 1) shard of pr-test-amd.yml; 2 of 22 parametrizations (1 FAILED + 1 ERROR), other 9 executed tests pass.
• Blocked work: AMD 2-GPU diffusion fp8 (FLUX.2 modelopt-fp8) coverage and the Ideogram-4-fp8 server test.
• In-flight fix: flux2 → ✅ #28889; ideogram4 → ❌ no direct fix (CI HF access issue).

stage-b-test-large-8-gpu-mi35x-disaggregation-amd (linux-mi35x-gpu-8.fabric) — failed step(s): Run test_disaggregation

Commit Info

• sglang: ba9d5aed
• aiter: N/A — no [CI-AITER-CHECK] markers in this log (this stage builds sgl-kernel and runs disaggregation tests; no aiter version-check step present).

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/disaggregation/test_disaggregation_basic.py	TestDisaggregationAccuracy.test_gsm8k	KeyError: 'answer' (after all 200 /generate requests returned HTTP 500 KVTransferError … timed out after 300.0s in KVPoll.WaitingForInput); retry() exceed maximum number of retries	Log
test/registered/amd/disaggregation/test_disaggregation_pp.py	TestDisaggregationDecodePPAccuracy.setUpClass	RuntimeError: Server process exited with code -9 (preceded by RDMA Failed to register memory … Cannot allocate memory [12] and RuntimeError: The memory capacity is unbalanced. Some GPUs may be occupied by other processes)	Log

(test_mori_transfer_engine_e2e.py passed — {"passed": true, "elapsed": 118}.)

Failure Summary

The 8-GPU MI35x disaggregation stage failed on two of three test files. In test_disaggregation_basic.py::test_gsm8k, every one of the 200 GSM8K requests returned HTTP 500 because the prefill→decode KV transfer timed out after 300 s in KVPoll.WaitingForInput, so no answer variable was ever populated, yielding KeyError: 'answer'. The subsequent test_disaggregation_pp.py aborted in setUpClass when the prefill server was SIGKILLed (exit code -9) after RDMA memory registration failed with Cannot allocate memory [12] and the model runner reported unbalanced GPU memory (other processes occupying the GPUs).

Regression Status

New regression today (for this job) — but symptoms are infrastructure-flavored, not a clear code regression.

Recent history of test_disaggregation_basic.py + test_disaggregation_pp.py in job stage-b-test-large-8-gpu-mi35x-disaggregation-amd (branch main, event schedule, completed runs only):

Date	Run	Job	Test File Status	Failed Function	Error
Jun 23 00:51	27994433780	job	❌ Failed	test_gsm8k, setUpClass	KV transfer timeout / server -9
Jun 22 13:56	27958083310	job	✅ Passed	—	—
Jun 22 08:09	27938642367	job	✅ Passed	—	—
Jun 22 01:00	27923265639	job	✅ Passed	—	—

First observed failure: 2026-06-23 00:51 UTC. Last known passing: 2026-06-22 13:56 UTC (run 27958083310). Regression window: 4923bb93ae..ba9d5aed98 (26 commits). The two scheduled runs after this one (27977371862, this one) were still queued at snapshot — excluded.

Failure Cluster

"Disaggregation KV-transfer timeout + RDMA/GPU memory exhaustion on MI35x fabric runner"

Facts (verified observations, no interpretation)

• test_gsm8k: progress bar shows 100%|██████████| 200/200 [10:00<00:00, 3.00s/it] — all 200 requests hit the 300 s server-side timeout window; every /generate logged 500 Internal Server Error.
• Verbatim error: KVTransferError(bootstrap_room=…): Request … timed out after 300.0s in KVPoll.WaitingForInput and Prefill transfer failed … Failed due to an unknown reason from another rank.
• test_gsm8k final failure: KeyError: 'answer' at python/sglang/test/few_shot_gsm8k.py:123 → retry() exceed maximum number of retries (python/sglang/srt/utils/common.py:2853). Basic test ran 1374 s (estimate 600 s).
• test_disaggregation_pp.py setUpClass: 4 RDMA errors rdma_context.cpp:243] Failed to register memory 0x… Cannot allocate memory [12] (errno 12 = ENOMEM), then RuntimeError: The memory capacity is unbalanced. Some GPUs may be occupied by other processes. pre_model_load_memory=45.5, local_gpu_memory=279.6 … — and one rank reported local_gpu_memory=612.5 (inconsistent free memory across ranks). Server then received sigquit from a child process and Server process exited with code -9.
• Runner: linux-mi35x-gpu-8.fabric. Job overall exit: Process completed with exit code 255.
• No commit in the regression window touches disaggregation or model-runner code: git log 4923bb93ae..ba9d5aed98 -- python/sglang/srt/disaggregation/ and -- python/sglang/srt/model_executor/model_runner.py both return empty.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [MEDIUM] Infrastructure: RDMA fabric / pinned-memory exhaustion on the linux-mi35x-gpu-8.fabric runner, with GPU contamination carried over from the timed-out basic test into the pp test.
  • Supporting: Cannot allocate memory [12] (ENOMEM) on RDMA register_memory; "GPUs may be occupied by other processes" with inconsistent per-rank free memory (279.6 vs 612.5 GB); the basic test overran its estimate by ~13 min before the pp test even started, so leftover/zombie GPU processes are plausible; both failures are memory/transport-layer, not numerical/logic.
  • Disconfirming: the job passed cleanly in the previous 3 scheduled runs on the same runner pool, so the fabric is not permanently broken — this could be a transient resource-contention event rather than a standing infra fault.
  • How to verify: re-run the job in isolation; check the runner for leftover sglang/python GPU processes and ulimit -l (locked-memory limit) / available hugepages before launch; inspect rocm-smi for residual memory before the pp test.
• [LOW] A commit in 4923bb93ae..ba9d5aed98 regressed startup memory/initialization causing the prefill server OOM-kill (-9).
  • Supporting: temporal correlation — first failing run; window includes the large ServerArgs migration #28919 (b28e9901) and FA4 page_size revert churn.
  • Disconfirming: no commit in the window modifies python/sglang/srt/disaggregation/ or model_runner.py (verified empty git log); the failing symptom is RDMA ENOMEM + cross-rank memory imbalance, which is environmental, not code-shaped; the basic-test KV-transfer timeout is independent of server startup memory.
  • How to verify: bisect the window only if a re-run reproduces on a clean runner; otherwise treat as infra.

In-flight Fix Check

• In-flight fix: ❌ none found for the KV-transfer-timeout / RDMA-ENOMEM symptom — repo:sgl-project/sglang is:pr is:open disaggregation KVTransfer timeout returned 0 results.
• Context (not a fix): #27012 Revert "[AMD] Fix stage-b-test-large-8-gpu-mi35x-disaggregation-amd" indicates this exact job has a history of instability. Several open PRs touch MI35x PD disaggregation CI (#28945, #28554, #28348) but none target this timeout.

Suggested Triage Steps

• Re-run the job on a freshly-reset linux-mi35x-gpu-8.fabric node; if it passes, classify as transient infra/contention (most likely given 3 prior green runs).
• Before launch, check the runner for leftover GPU processes (rocm-smi) and locked-memory limits (ulimit -l) — the Cannot allocate memory [12] on RDMA register_memory points at pinned-memory/hugepage exhaustion or residual allocations from the overrun basic test.
• Investigate test isolation: the pp test started while GPUs were still reported "occupied by other processes" — add a GPU-free/teardown wait between test_disaggregation_basic.py and test_disaggregation_pp.py so a timed-out basic test cannot poison the pp test.
• Only if a clean-runner re-run still fails: bisect 4923bb93ae..ba9d5aed98 — but note no disaggregation/model_runner code changed in this window, so a code root cause is unlikely.

Status

• Persistence: First failing completed run 2026-06-23 00:51 UTC; previous 3 scheduled runs (Jun 22) all passed this job. 2 newer scheduled runs still queued at snapshot.
• Scope: 1 job (stage-b-test-large-8-gpu-mi35x-disaggregation-amd) on runner linux-mi35x-gpu-8.fabric; 2 of 3 disaggregation test files affected.
• Blocked work: AMD MI35x PD-disaggregation CI signal (basic accuracy + PP decode accuracy).
• In-flight fix: ❌ none found for this symptom; job has prior instability history (see #27012).

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-06-23 20:58 UTC)

[amd-bot]

nightly-test-amd-rocm720.yml — 17 failure(s)

Run started (UTC): 2026-06-22 18:44 UTC
Last scanned (UTC): 2026-06-23 20:59 UTC
Commits: sglang ed26a10

Summary

Counts: 20 failures · 12 clusters · 14 carrying over (≥3 days) · 6 new today

#	Cluster	Job	Test File	Test Function	Status	Hypothesis (confidence)
1	HF weight download/load timeout	nightly-8-gpu-mi35x-deepseek-v32-rocm720	test_deepseek_v32_eval_mi35x.py	whole-file (eval never reached)	≥10 days, worsening	[HIGH] init (HF download + fp8 _load_w2 H2D copy) exceeds 3600s
2	HF weight download/load timeout	nightly-8-gpu-mi35x-kimi-k26-rocm720	test_kimi_k26_eval_mi35x.py	test_kimi_k26_gsm8k_accuracy	3 days (since Jun 20)	[HIGH] hf-xet download stall → 7200s timeout
3	HF weight download/load timeout	nightly-8-gpu-mi35x-deepseek-r1-hicache-rocm720	test_deepseek_r1_hicache_mi35x.py	N/A (stuck in load)	3 days, never-passed	[MEDIUM] xet download stall → 3600s timeout
4	HF weight download/load timeout	nightly-8-gpu-mi35x-glm5-mxfp4-rocm720	test_glm5_mxfp4_eval_mi35x.py	test_glm5_mxfp4_accuracy	new today (passed Jun 21)	[HIGH] xet/filelock download stall
5	HF weight download/load timeout	nightly-8-gpu-mi35x-glm51-rocm720	test_glm51_eval_mi35x.py	test_glm51_accuracy	new today (passed Jun 21)	[HIGH] xet/filelock download stall
6	HF weight download/load timeout	nightly-8-gpu-mi35x-deepseek-v4-pro-rocm720	test_deepseek_v4_pro_fp8.py	setUpClass	new today (passed Jun 21)	[HIGH] xet download stall (step-6 only; triton variant passed)
7	Mamba extra_buffer rejected on ROCm	nightly-8-gpu-qwen35-rocm720	test_qwen35_eval_amd.py	TestQwen35EvalAMD.setUpClass	3 days (since Jun 20)	[HIGH] commit 66a7fd5c0b (#28151)
8	Mamba extra_buffer rejected on ROCm	nightly-8-gpu-mi35x-qwen35-rocm720	test_qwen35_eval_mi35x.py	test_lm_eval	3 days (since Jun 20)	[MEDIUM] auto→extra_buffer unreachable on ROCm
9	VLM MMMU accuracy near chance	nightly-accuracy-2-gpu-vlm-rocm720	test_vlms_mmmu_eval_amd.py	test_mmmu_vlm_models	≥12 days, never-passed	[MEDIUM] systemic VLM degradation on ROCm 7.2
10	VLM MMMU accuracy near chance	nightly-accuracy-2-gpu-vlm-rocm720	test_vlms_mmmu_eval_amd.py	test_mmmu_vlm_models	≥9 days, never-passed	[MEDIUM] systemic VLM degradation on ROCm 7.2
11	Mistral/Mixtral gsm8k below threshold	nightly-accuracy-2-gpu-rocm720	test_gsm8k_eval_amd.py	test_gsm8k_all_models (Mistral-7B, Mixtral-8x7B)	recurring, never-passed	[MEDIUM] eval-harness/answer-extraction
12	Mistral/Mixtral gsm8k below threshold	nightly-accuracy-2-gpu-rocm720	test_gsm8k_eval_amd.py	check_model_scores (Mistral/Mixtral ±FP8, gemma-2-27b)	recurring, never-passed	[MEDIUM] eval-harness/answer-extraction
13	HIP invalid device pointer (DP-encoder IPC)	nightly-4-gpu-rocm720	test_encoder_dp.py	test_vlm_mmmu_benchmark	since Jun 14 (3/6 runs)	[HIGH] cuda_ipc_transport_utils.py:488 reconstruct fails on ROCm
14	HIP capturing-stream abort (aiter custom all-reduce)	nightly-8-gpu-mi35x-qwen3-235b-mxfp4-rocm720	test_qwen3_instruct_mxfp4.py	TestQwen3Instruct2507MXFP4.setUpClass	since ≥Jun 14, never-passed	[MEDIUM] custom_all_reduce.capture() event on capturing stream
15	MoE w2 narrow out-of-range (Kimi-K2.6)	nightly-8-gpu-kimi-k26-rocm720	test_kimi_k26_eval_amd.py	TestKimiK26EvalAMD.setUpClass	3 days (since Jun 20)	[LOW] per-rank w2 checkpoint vs use_presharded_weights=False
16	Qwen3-30B-A3B gsm8k far below threshold	nightly-accuracy-8-gpu-mi35x-rocm720	test_qwen3_moe_eval_mi35x.py	test_qwen3_moe_accuracy	since Jun 20, never-passed	[MEDIUM] ROCm MoE degraded or threshold too high
17	MiniMax-M2.7 gsm8k below threshold	nightly-8-gpu-minimax-m27-rocm720	test_minimax_m27_eval_amd.py	test_minimax_m27_accuracy	2 days, drifting (0.935→0.920→0.910)	[LOW] eval variance vs too-tight 0.93 threshold
18	GPU memory access fault → 0.000 (Qwen2-72B-FP8)	nightly-accuracy-2-gpu-rocm720	test_gsm8k_eval_amd.py	check_model_scores (Qwen2-72B-FP8)	new today (passed Jun 20)	[LOW] mem fault in ff1fc1fbdf..43b1fb95f or flake
19	Qwen3-Coder-Next fp8-kv-cache server hang	nightly-accuracy-8-gpu-mi35x-rocm720	test_qwen3_coder_next_eval_mi35x.py	test_qwen3_coder_next_accuracy	new today (passed Jun 20–21)	[LOW] flake vs regression in 7f67965b4d..43b1fb95f
20	Server-start timeout (GPU state)	nightly-accuracy-2-gpu-rocm720	test_gsm8k_eval_amd.py	test_gsm8k_all_models (Qwen2-57B)	new today (started Jun 20/22)	[LOW] GPU-state degradation after Mixtral watchdog storm

Cluster: HF weight download/load timeout (rows 1–6)

• Affected: deepseek-v32, mi35x-kimi-k26, deepseek-r1-hicache, glm5-mxfp4, glm51, deepseek-v4-pro-fp8.
• Shared facts: every job parks all 8 TP ranks in download_weights_from_hf→filelock._poll_until_acquired while the lock-holder stalls in huggingface_hub xet_get/_download_to_tmp_and_move; eval never runs. aiter unchanged (v0.1.14rc1.dev240+g7d604afe5) — not a variable. deepseek-v32 additionally has a slow fp8 _load_w2 H2D copy (fused_moe_triton/layer.py:571) past the download phase.
• Top hypothesis: [HIGH] transient/cold HF-Xet download stall on the mi35x runners; disconfirming: no byte-progress lines, so pure-stall vs slow-fetch can't be separated; sister/triton variants downloaded fine (rules out model/config).
• In-flight fix: ❌ none found (searched xet download, weight download timeout).
• Suggested triage: re-run the 3 "new today" jobs (glm5/glm51/v4-pro) to confirm flake; check whether mi35x HF cache volume is persisted across nightlies (Jun 19–20 cancelled window correlates with onset of the recurring ones); add a stalled-download (zero-progress) fast-fail/retry around snapshot_download.

Cluster: Mamba extra_buffer rejected on ROCm (rows 7–8)

• Affected: nightly-8-gpu-qwen35-rocm720, nightly-8-gpu-mi35x-qwen35-rocm720 (both Qwen3.5-397B-A17B).
• Shared facts: identical AssertionError: extra_buffer needs CUDA/MUSA/NPU (FLA). at server_args.py:4366-4368 via _handle_mamba_radix_cache; last green Jun 15, failing every run since Jun 20.
• Top hypothesis: [HIGH] commit 66a7fd5c0b (#28151, Jun 18) changed auto→extra_buffer and removed the ROCm guard; disconfirming: none (test passed before, fails after; guard+assertion both in this commit).
• In-flight fix: ✅ #27141 "[ROCm] Support radix-cache with mamba-extra-buffer for Qwen3.5" (open since 2026-06-03) — chase reviewers.
• Suggested triage: verify #27141 merges & resolves; interim, pass --mamba-radix-cache-strategy no_buffer for these AMD tests.

Cluster: VLM MMMU accuracy near chance (rows 9–10)

• Affected: nightly-accuracy-2-gpu-vlm-rocm720 on both Jun 23 (83026842498) and Jun 22 (82793291668).
• Shared facts: all 11 VLMs score in the 0.26–0.35 band (~random for 4-choice); identical deterministic scores (e.g. 0.2802) across multiple model families; same 4-model core never passes; aiter unchanged.
• Top hypothesis: [MEDIUM] systemic VLM accuracy degradation (or never-correct pipeline) on ROCm 7.2/MI30x; disconfirming: no green baseline in window — thresholds may simply be miscalibrated.
• In-flight fix: ⚠️ stale WIP #26178 (threshold relaxation only) — does not address a possible accuracy root cause.
• Suggested triage: A/B the same models/commit on NVIDIA nightly; inspect /tmp/mmmu_<model>.html for garbled vs coherent outputs before touching thresholds.

Cluster: Mistral/Mixtral gsm8k below threshold (rows 11–12)

• Affected: nightly-accuracy-2-gpu-rocm720 Jun 23 & Jun 22.
• Shared facts: Mistral-7B ~0.36 (thr 0.47), Mixtral-8x7B ~0.30 (thr 0.69) reproduce every nightly across 3 internal retries; gemma-2-27b borderline (0.860 vs 0.86).
• Top hypothesis: [MEDIUM] gsm8k chat-prompt/answer-extraction issue, not model correctness; disconfirming: no green baseline to pin a regressing commit.
• In-flight fix: ✅ #27757 "Fix Mistral GSM8K chat eval" (open since 2026-06-10) — chase reviewers.
• Suggested triage: rebase/merge #27757, re-run; review gemma-2-27b threshold.

Single-job clusters (rows 13–20)

• Row 13 — HIP invalid device pointer (DP-encoder IPC): [HIGH] torch.UntypedStorage._new_shared_cuda fails on ROCm in cuda_ipc_transport_utils.py:488; gated on --mm-enable-dp-encoder; masked when retry picks InternVL2_5-8B. ❌ no in-flight fix. Triage: pin Qwen3-VL-32B + A/B with/without --mm-enable-dp-encoder.
• Row 14 — HIP capturing-stream abort (aiter custom all-reduce): [MEDIUM] event queried on a capturing stream inside custom_all_reduce.capture()→flush_graph_buffers→_gather_ipc_meta. ⚠️ #27650 related but targets a different (fusion) config. Triage: re-run with --disable-custom-all-reduce.
• Row 15 — Kimi-K2.6 MoE w2 narrow out-of-range: [LOW] per-rank w2 checkpoint vs use_presharded_weights=False (fused_moe_triton/layer.py:566). ✅ #28905 (scale path — verify it covers the w2 weight narrow). Sister mi35x-kimi-k26 (row 2) is a different (download) cluster.
• Row 16 — Qwen3-30B-A3B gsm8k 0.380/0.75: [MEDIUM] ROCm MoE degraded or threshold too high; never-passed since the test was added Jun 20 by #28244. ❌ no in-flight fix.
• Row 17 — MiniMax-M2.7 gsm8k 0.910/0.93: [LOW] variance vs too-tight threshold (non-fixed seed), monotonic drift can't fully exclude small regression. ❌ no fix. Triage: re-run 3–5× with fixed seed.
• Row 18 — Qwen2-72B-FP8 mem fault → 0.000: [LOW] new today (passed Jun 20 at 0.937); GPU memory access fault → exit -6. ❌ no fix. Triage: re-run standalone to separate flake from regression.
• Row 19 — Qwen3-Coder-Next fp8-kv-cache hang: [LOW] basic (fp8_e4m3) variant hung (soft watchdog + detokenizer health-check); mtp variant passed. ❌ no fix. Triage: one confirming re-run.
• Row 20 — Qwen2-57B server-start timeout: [LOW] likely degraded GPU state after the ~50-min Mixtral watchdog storm in the same job. Triage: run Qwen2-57B first/in isolation.

Suggested triage order

Rank	Triage step	Targets	Why this first
1	Verify & merge #27141, re-run both qwen35 jobs	rows 7–8	HIGH-confidence known cause (66a7fd5c0b) + in-flight fix exists
2	Investigate mi35x HF-cache persistence; re-run the 3 new-today download stalls	rows 1–6	Largest cluster (6 jobs); cheap re-run separates infra-flake from persistent
3	Rebase/merge #27757, re-run	rows 11–12	In-flight fix targets exact Mistral/Mixtral path
4	A/B VLM MMMU on NVIDIA + inspect HTML reports	rows 9–10	Resolves a 12-day red signal; decides degradation vs threshold
5	Confirm #28905 covers w2-weight narrow; A/B --disable-custom-all-reduce (row 14) and --mm-enable-dp-encoder (row 13)	rows 13–15	Code-level HIP crashes; cheap flag A/B yields decisive evidence

---

Job	Failed Steps	Started
nightly-accuracy-2-gpu-vlm-rocm720	Nightly Accuracy Test ROCm 7.2 (2-GPU VLM MMMU)	2026-06-22T19:36
nightly-accuracy-2-gpu-rocm720	Nightly Test ROCm 7.2 (2-GPU)	2026-06-22T19:31
nightly-8-gpu-mi35x-deepseek-r1-hicache-rocm720	Accuracy Test MI35x ROCm 7.2 (8-GPU DeepSeek-R1 HiCache)	2026-06-22T21:22
nightly-accuracy-8-gpu-mi35x-rocm720	Accuracy Test MI35x ROCm 7.2 (8-GPU GPT-OSS)	2026-06-22T19:04
nightly-accuracy-8-gpu-mi35x-deepseek-v32-rocm720	Accuracy Test MI35x ROCm 7.2 (8-GPU DeepSeek-V3.2)	2026-06-22T22:07
nightly-4-gpu-rocm720	Nightly Test ROCm 7.2 (4-GPU)	2026-06-22T18:44
nightly-8-gpu-qwen35-rocm720	Accuracy Test ROCm 7.2 (8-GPU Qwen 3.5)	2026-06-22T20:59
nightly-8-gpu-minimax-m27-rocm720	Accuracy Test ROCm 7.2 (8-GPU MiniMax-M2.7)	2026-06-22T20:44
nightly-8-gpu-mi35x-glm5-mxfp4-rocm720	Accuracy Test MI35x ROCm 7.2 (8-GPU GLM-5-MXFP4)	2026-06-22T22:14
nightly-8-gpu-mi35x-glm51-rocm720	Accuracy Test MI35x ROCm 7.2 (8-GPU GLM-5.1 DSA)	2026-06-22T19:50
nightly-8-gpu-mi35x-qwen35-rocm720	Accuracy Test MI35x ROCm 7.2 (8-GPU Qwen 3.5)	2026-06-22T23:54
nightly-8-gpu-mi35x-kimi-k26-rocm720	Accuracy Test MI35x ROCm 7.2 (8-GPU Kimi-K2.6)	2026-06-22T19:41
nightly-8-gpu-mi35x-qwen3-235b-mxfp4-rocm720	Accuracy Test + Performance Test MI35x ROCm 7.2 (8-GPU Qwen3-235B-MXFP4)	2026-06-23T00:01
nightly-8-gpu-kimi-k26-rocm720	Accuracy Test ROCm 7.2 (8-GPU Kimi-K2.6)	2026-06-22T22:00
nightly-8-gpu-mi35x-deepseek-v4-pro-rocm720	Accuracy + Performance Test MI35x ROCm 7.2 (8-GPU DeepSeek-V4-Pro FP8 + FP4, unified_kv_triton)	2026-06-23T00:28
nightly-accuracy-2-gpu-vlm-rocm720	Nightly Accuracy Test ROCm 7.2 (2-GPU VLM MMMU)	2026-06-23T18:58
nightly-accuracy-2-gpu-rocm720	Nightly Test ROCm 7.2 (2-GPU)	2026-06-23T18:43

Per-Job Analysis

nightly-accuracy-2-gpu-vlm-rocm720 — failed step(s): Nightly Accuracy Test ROCm 7.2 (2-GPU VLM MMMU)

Commit Info

• sglang: 43b1fb95f
• aiter: 7d604afe5 (source: dev image — [CI-AITER-CHECK] reports Dev/patched version detected: v0.1.14rc1.dev240+g7d604afe5 → skipping rebuild; matches Dockerfile default)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi30x/test_vlms_mmmu_eval_amd.py	test_mmmu_vlm_models (TestNightlyVLMMmmuEvalAMD)	AssertionError: The following models failed: — 4 of 11 VLMs below MMMU accuracy threshold	Log

Per-model breakdown (MMMU, 100 samples):

Model	TP	Score	Threshold	Status
Qwen/Qwen2.5-VL-7B-Instruct	1	0.2802	0.33	❌
Qwen/Qwen3-VL-30B-A3B-Instruct	2	0.2601	0.29	❌
openbmb/MiniCPM-o-2_6	1	0.2802	0.32	❌
deepseek-ai/deepseek-vl2-small	1	0.2802	0.31	❌
(7 others)	—	0.29–0.35	—	✅

Failure Summary

The nightly-accuracy-2-gpu-vlm-rocm720 job ran test_mmmu_vlm_models, which evaluates 11 VLMs on the MMMU benchmark. 7 models passed and 4 (Qwen2.5-VL-7B, Qwen3-VL-30B-A3B, MiniCPM-o-2_6, deepseek-vl2-small) scored below their configured accuracy thresholds, raising an AssertionError and failing the job (exit 255). This is the only failing step in the job (build + install steps succeeded).

Regression Status

Known recurring failure — test_mmmu_vlm_models has failed in every recent completed scheduled run of this job. No green run found in the queryable window (lookback Jun 11–22, 2026; intervening Jun 16–19 runs were cancelled, not comparable).

Recent history of test_vlms_mmmu_eval_amd.py (test_mmmu_vlm_models) in job nightly-accuracy-2-gpu-vlm-rocm720 (branch main, event schedule, completed runs only):

Date	Run	Job	Test File Status	Failed Models
Jun 22	27975850577	job	❌ Failed	Qwen2.5-VL-7B, Qwen3-VL-30B, MiniCPM-o-2_6, deepseek-vl2-small
Jun 21	27912828989	job	❌ Failed	+ also Qwen2-VL-7B (5 total)
Jun 20	27879239829	job	❌ Failed	same 4 models
Jun 15	27568866028	job	❌ Failed (errors=1)	near-random scores across models
Jun 14	27507292665	job	❌ Failed	—

(Run-level status of 27975850577 is still queued for sibling jobs, but this job itself completed: failure at 20:32 UTC — its result is final.)

Failure Cluster

VLM MMMU accuracy below threshold on AMD ROCm 7.2 (scores near random chance)

Facts (verified observations)

• The 4 failing scores are deterministic and identical to 4 decimals across multiple runs: Qwen2.5-VL-7B, MiniCPM-o-2_6, and deepseek-vl2-small all report exactly 0.2802 in this run; Qwen3-VL-30B reports 0.2601 here and 0.2601 again on Jun 15 and Jun 21 (verified across job logs).
• MMMU is 4-option multiple choice ⇒ random-chance baseline ≈ 0.25. All 11 models — including the 7 "passing" ones — score in the 0.26–0.35 band, i.e. only marginally above random (verified from the SUMMARY table).
• Thresholds are hard-coded per model in the test file (test_vlms_mmmu_eval_amd.py:50-123): Qwen2.5-VL-7B 0.33, Qwen3-VL-30B 0.29, MiniCPM-o-2_6 0.32, deepseek-vl2-small 0.31.
• The failure is an AssertionError raised in _check_results (test_vlms_mmmu_eval_amd.py:380), called from test_mmmu_vlm_models (line 314).
• Build, install, and aiter version check all succeeded; aiter is the Dockerfile-default dev build (no override).

Hypothesised Causes (with confidence)

• [MEDIUM] Systemic VLM accuracy degradation on the AMD ROCm 7.2 path (vision/eval pipeline), not per-model regressions.
  • Supporting: every model scores near MMMU random chance; three structurally different models land on the exact same score (0.2802), which is unlikely if each model were independently mis-scoring — points to a shared upstream factor (image preprocessing, vision encoder output, or answer-extraction parsing) producing near-random answers.
  • Disconfirming: a few models do clear their (low) thresholds (Kimi-VL 0.35, MiMo-VL 0.31, MiniCPM-v-2_6 0.31), so the path is not uniformly broken; and the thresholds themselves are set well above the observed band, so this could equally be a thresholds-too-high problem rather than a true accuracy regression.
  • How to verify: run one failing model (e.g. Qwen2.5-VL-7B) on MMMU on an NVIDIA box with the same sglang SHA and compare score; inspect a few generated answers / the MMMU HTML report (/tmp/mmmu_*.html) for parsing or blank-output patterns.
• [LOW] Thresholds in test_vlms_mmmu_eval_amd.py are set higher than what these models reproducibly achieve on mi30x / ROCm 7.2 (i.e. a test-config/calibration issue).
  • Supporting: the open WIP PR #26178 "[AMD] relaxing thres for mi300" indicates maintainers already consider AMD thresholds miscalibrated; scores are stable run-to-run (not flaky), consistent with a fixed offset between calibrated and actual accuracy.
  • Disconfirming: scores near random chance suggest the models aren't merely a few points short of a reasonable target — lowering thresholds to ~0.26 would mask a possibly real quality problem.
  • How to verify: compare these AMD scores against the same models' MMMU scores on the NVIDIA nightly to establish the expected accuracy band.

I did not enumerate an aiter regression window: this job runs the Dockerfile-default aiter (no override), and the failure is a stable, recurring accuracy shortfall rather than a crash — aiter is not implicated by current evidence.

In-flight Fix Check

• In-flight fix: ⚠️ partial — #26178 "[AMD] WIP - relaxing thres for mi300" (open, created 2026-05-23, last updated 2026-05-26 — stale WIP). It directly targets relaxing AMD VLM thresholds but appears stalled; it would mask the failure only if the LOW-confidence "thresholds too high" hypothesis is the accepted explanation. Search performed: q=repo:sgl-project/sglang is:pr is:open vlm accuracy amd and ...MMMU VLM.
• No open PR found addressing a ROCm VLM accuracy degradation root cause.

Suggested Triage Steps

• Pull the MMMU result artifacts from the run (/tmp/mmmu_<model>.json / .html) for the 4 failing models and inspect per-category accuracy + sample generations to distinguish "near-random answers" (real degradation) from "answers correct but parsed wrong" (eval bug).
• A/B the same sglang SHA on NVIDIA nightly for Qwen2.5-VL-7B / Qwen3-VL-30B; if NVIDIA scores are materially higher, this is a ROCm-path accuracy issue (promote MEDIUM hypothesis); if NVIDIA scores are similarly low, the thresholds are miscalibrated (promote LOW hypothesis).
• If degradation is confirmed on ROCm, bisect the VLM forward / vision-encoder path on AMD; do not simply lower thresholds via #26178 until the near-random scores are explained.
• If thresholds are confirmed miscalibrated against a healthy NVIDIA baseline, revive #26178 with per-model values derived from the AMD baseline band.

Status

• Persistence: Failing in every completed scheduled run of this job in the queryable window (since at least Jun 14, 2026; no green run found Jun 11–22).
• Scope: 1 job (nightly-accuracy-2-gpu-vlm-rocm720), 4 of 11 VLM models below threshold; companion nightly-perf-2-gpu-vlm-rocm720 passes (perf, not accuracy).
• Blocked work: AMD ROCm 7.2 VLM accuracy signal is red; cannot distinguish a real VLM quality regression from threshold miscalibration until A/B baseline is run.
• In-flight fix: ⚠️ stale WIP #26178 (threshold relaxation only — does not address a possible accuracy root cause).

nightly-accuracy-2-gpu-rocm720 — failed step(s): Nightly Test ROCm 7.2 (2-GPU)

Commit Info

• sglang: 43b1fb95f6d9077c6750419546d73be8c710a754
• aiter: 7d604afe5fa7efba63c0dce323b95d9daf2db112 (v0.1.14rc1.dev240+g7d604afe5, source: dev image preserved — no override; AITER_COMMIT_OVERRIDE empty)

Failed Tests

The job runs one accuracy file; check_model_scores raises a single AssertionError listing every model below its gsm8k threshold.

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi30x/test_gsm8k_eval_amd.py	check_model_scores	AssertionError: The following models failed: (6 models below threshold)	Log

The 6 failing models (from the summary table + assertion at the end of the run):

Model	TP	FP8	Score	Threshold	Note
mistralai/Mistral-7B-Instruct-v0.3	1	no	0.362	0.47	low score
google/gemma-2-27b-it	1	no	0.860	0.86	borderline (0.8600 < 0.86)
mistralai/Mixtral-8x7B-Instruct-v0.1	2	no	0.301	0.69	low score
neuralmagic/Mistral-7B-Instruct-v0.3-FP8	1	yes	0.317	0.46	low score
neuralmagic/Mixtral-8x7B-Instruct-v0.1-FP8	2	yes	0.159	0.69	low score
neuralmagic/Qwen2-72B-Instruct-FP8	2	yes	0.000	0.86	server crash (GPU memory access fault)

(11 models passed, 6 failed. The job's 2nd file, test/registered/debug_utils/test_dumper.py, passed: 243 passed.)

Failure Summary

The nightly-accuracy-2-gpu-rocm720 gsm8k accuracy file failed because 6 of 17 models scored below threshold. Five of them — the Mistral/Mixtral family plus a borderline gemma-2-27b — are the same models that fail every recent nightly (a recurring low-score pattern). The sixth, neuralmagic/Qwen2-72B-Instruct-FP8, scored 0.000 because its server crashed with a GPU memory access fault during the run; this is new today.

Regression Status

Two distinct patterns, different statuses:

Cluster A — Mistral/Mixtral/gemma low gsm8k scores: Known recurring failure.

Recent history of test_gsm8k_eval_amd.py for the Mistral/Mixtral models (nightly-accuracy-2-gpu-rocm720, branch main, event=schedule, completed runs):

Date	Run	Test File Status	Mistral-7B / Mixtral-8x7B / FP8 variants
Jun 22	27975850577	❌ Failed	0.362 / 0.301 / 0.317 / 0.159 — all FAIL
Jun 21	27912828989	❌ Failed	0.363 / (job cut off at Mixtral load) — Mistral-7B + gemma already FAIL
Jun 20	27879239829	❌ Failed	0.371 / 0.296 / 0.308 / 0.171 — same 5 FAIL

Cluster B — Qwen2-72B-FP8 = 0.000: New regression today.

Date	Run	Qwen2-72B-Instruct-FP8
Jun 22	27975850577	❌ 0.000 (GPU memory access fault, server crashed exit -6)
Jun 21	27912828989	— (job cut off before reaching this model)
Jun 20	27879239829	✅ 0.937 PASSED

Last known good for Qwen2-72B-FP8: Jun 20 (ff1fc1fbdf). First observed crash: Jun 22 (43b1fb95f). Jun 21 did not reach this model, so the crash window cannot be narrowed below ff1fc1fbdf..43b1fb95f.

Failure Cluster

• Cluster A: "Mistral/Mixtral family low gsm8k accuracy on AMD" (symptom: scores ~0.3 vs ~0.5–0.69 thresholds; gemma-2-27b borderline).
• Cluster B: "GPU memory access fault → server crash → 0.000 score (Qwen2-72B-FP8)".

Facts (verified observations)

• Assertion verbatim (run summary): AssertionError: The following models failed: followed by the 6 model/score/threshold lines — Mistral-7B=0.3623/0.47, gemma-2-27b=0.8600/0.86, Mixtral-8x7B=0.3006/0.69, Mistral-7B-FP8=0.3166/0.46, Mixtral-8x7B-FP8=0.1591/0.69, Qwen2-72B-FP8=0.0000/0.86. Final stats: Passed: 11 / Failed: 6. Job exits 255.
• Qwen2-72B-FP8 crash evidence (verbatim): Memory access fault by GPU node-3 (Agent handle: ...) on address (nil). Reason: Unknown. → Fatal Python error: Aborted → Subprocess scheduler_1 (pid=49498) crashed with exit code -6. After the crash the eval emitted a continuous stream of Rate limit exception ... Connection error. (server dead) → score 0.000.
• Baseline Jun 20: Qwen2-72B-FP8 scored 0.936834... and PASSED, confirming the crash is new.
• Mistral/Mixtral low scores reproduce identically in Jun 20 and Jun 22 (same 5 models, near-identical scores) → not a fresh regression.
• aiter version unchanged (dev image v0.1.14rc1.dev240+g7d604afe5, no override) — not a variable here.
• This is nightly-test-amd-rocm720.yml (not a scout workflow), so no Baseline A/B / Failure Origin section applies.

Hypothesised Causes (with confidence)

Cluster A (Mistral/Mixtral/gemma):

• [MEDIUM] Eval-harness / answer-extraction issue, not a model-correctness regression.
  • Supporting: open PR #27757 "Fix Mistral GSM8K chat eval" states the root cause is the gsm8k chat prompt + answer extraction (not FA3/backend); the failure is stable across many nightlies at the same ~0.3 scores; gemma-2-27b sits exactly on its threshold (0.8600 vs 0.86), characteristic of a too-tight threshold rather than a break.
  • Disconfirming: no green run for these models exists in the queryable nightly window, so a "regression commit" cannot be pinned; the eval-harness hypothesis is plausible but not yet code-verified on this branch.
  • How to verify: merge/rebase #27757 and re-run test_gsm8k_eval_amd.py on AMD MI325; confirm Mistral/Mixtral recover.

Cluster B (Qwen2-72B-FP8 crash):

• [LOW] A commit in ff1fc1fbdf..43b1fb95f (67 commits) introduced/exposed a GPU memory access fault on FP8 dense 2-GPU inference.
  • Supporting: model passed at 0.937 on Jun 20 and crashed on Jun 22; window contains AMD/ROCm-touching commits (e.g. 2552b860a, d6d06cdc1, 6779ca8d7).
  • Disconfirming: the fault (address (nil)) has no stack trace tying it to any of those commits; the listed AMD commits are MoE-path changes whereas Qwen2-72B is dense; this is the 18th model in a ~2-hour single-GPU-pair sequence, so memory fragmentation / accumulated GPU state / infra instability is an equally plausible flake cause; only one occurrence observed (Jun 21 didn't reach this model).
  • How to verify: re-run the accuracy suite (or just Qwen2-72B-FP8 in isolation) on AMD MI325; if it passes alone, suspect cross-model state/fragmentation rather than a code regression. If it reproducibly crashes, bisect ff1fc1fbdf..43b1fb95f.

No hypothesis above MEDIUM for Cluster B; needs a rerun to separate flake from regression before bisecting.

In-flight Fix Check

• Cluster A: ✅ #27757 "Fix Mistral GSM8K chat eval" (open since 2026-06-10, last updated 2026-06-22) — directly targets Mistral/Mixtral gsm8k scoring. Chase reviewers; do NOT open a duplicate. (search: repo:...+is:pr+is:open+Mixtral+accuracy)
• Cluster B: ❌ none found for the Qwen2-72B-FP8 memory-access-fault — needs a rerun to confirm reproducibility, then a new issue/PR if it persists. (search: repo:...+is:pr+is:open+Qwen2-72B+memory)

Suggested Triage Steps

• Cluster A: rebase/merge #27757 and re-run test_gsm8k_eval_amd.py on AMD MI325; verify Mistral-7B/Mixtral-8x7B (+FP8) recover. Separately, review whether gemma-2-27b's 0.86 threshold should be loosened (it sits at 0.8584–0.8600 across runs).
• Cluster B: re-run neuralmagic/Qwen2-72B-Instruct-FP8 standalone on AMD MI325 to determine flake vs regression. If reproducible, bisect ff1fc1fbdf..43b1fb95f and capture a coredump/stack trace (the run's GPU coredump handler failed: execvp failed: No such file or directory).

Status

• Persistence: Cluster A (Mistral/Mixtral/gemma) failing every completed nightly at least Jun 20→Jun 22 at stable ~0.3 scores. Cluster B (Qwen2-72B-FP8 crash) first seen Jun 22; passed Jun 20.
• Scope: one job (nightly-accuracy-2-gpu-rocm720), one test file; 6 of 17 models. Blocks the AMD ROCm 7.2 nightly accuracy signal.
• In-flight fix: Cluster A → ✅ #27757; Cluster B → ❌ none.

nightly-8-gpu-mi35x-deepseek-r1-hicache-rocm720 — failed step(s): Accuracy Test MI35x ROCm 7.2 (8-GPU DeepSeek-R1 HiCache)

Commit Info

• sglang: 43b1fb95f6d9077c6750419546d73be8c710a754
• aiter: 7d604afe5fa7efba63c0dce323b95d9daf2db112 (v0.1.14rc1.dev240+g7d604afe5; source: dev/patched image build, preserved — no rebuild)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/test_deepseek_r1_hicache_mi35x.py	N/A (server never reached test body — stuck in model load)	✗ TIMEOUT after 3600 seconds → exit code 255	Log

Failure Summary

The single test file test_deepseek_r1_hicache_mi35x.py (DeepSeek-R1-0528, TP8, hierarchical cache) timed out after the 3600s per-file budget. The 8 TP scheduler processes never finished loading the model: rank TP3 (PID 3332) acquired the HF download lock at 21:32:30 and was actively downloading weights via xet_get, while all other ranks blocked on the same filelock (_poll_until_acquired). No "loading model weights" progress was ever logged; the run was killed at 22:31:34 still inside weight download.

Regression Status

Known recurring failure (test file has not passed in the queryable window).

Recent history of test/registered/amd/test_deepseek_r1_hicache_mi35x.py in job nightly-8-gpu-mi35x-deepseek-r1-hicache-rocm720 (branch main, event schedule, completed runs only):

Date	Run	Job	Test File Status	Failure Mode	Error
Jun 22	27975850577	job	❌ Failed	Weight-download stall	✗ TIMEOUT 3600s → exit 255
Jun 21	27912828989	job	❌ Failed	Download stall + scheduler_6 crash exit -6	timeout/crash
Jun 20	27879239829	job	❌ Failed	Download stall + scheduler_0 crash exit -6, then ✗ TIMEOUT 3600s	timeout/crash

(Jun 16–19 scheduled runs were cancelled, so not comparable. No green run for this test file found in the queryable window → effectively Never-passed here.)

Failure Cluster

DeepSeek-R1-0528 weight-download stall → 3600s file timeout during TP8 model load.

Facts (verified observations, no interpretation)

• server_args shows model_path='deepseek-ai/DeepSeek-R1-0528', tp_size=8, hierarchical cache enabled (log line 3683).
• TP3/PID 3332 [CI Download] ... ACQUIRED lock for deepseek-ai/DeepSeek-R1-0528 at 21:32:30 (log line 3876); subsequent py-spy dumps show it active in xet_get (huggingface_hub/file_download.py:566) (lines 4194, 9367 region).
• All other TP ranks idle in _poll_until_acquired (filelock/_api.py:552) under download_weights_from_hf (weight_utils.py:554) (e.g. lines 3942, 3993, 4048).
• Zero model-weight-loading progress messages in the entire log (grep for load progress = 0 hits).
• Repeated soft watchdog timeouts (watchdog_timeout=300, self.soft=True) from 21:37:02 through 22:27:14, every ~5 min — consistent with a stalled-but-not-crashed load.
• Killed at 22:31:34: ✗ TIMEOUT ... after 3600 seconds, Test Summary: 0/1 passed, exit code 255, "not retrying" (lines 9183–9190).
• Image pre-step: Local docker registry pull failed; falling back to public registry (line 1281) — infra/registry hiccup earlier in the job, recovered.
• Today: no scheduler crash. Jun 20 & Jun 21 runs additionally had Subprocess scheduler_* crashed with exit code -6 before/around the timeout.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [MEDIUM] Infrastructure: HuggingFace (Xet) download of the ~671B DeepSeek-R1-0528 checkpoint did not complete within the 3600s file budget on this runner.
  • Supporting: active thread stuck in xet_get; no load-progress logged; all peer ranks merely waiting on the download lock; the 3600s timeout fired mid-download, not mid-inference; earlier registry-pull fallback indicates network friction on this runner.
  • Disconfirming: cannot see download throughput numbers in the log, so a server-side/model-load hang (vs pure network slowness) cannot be fully excluded; the same test also failed the prior two days with an additional -6 crash, suggesting more than one failure mode for this test.
  • How to verify: re-run the job; check whether weights are cached on the runner (/sgl-data/hf-cache) and time the download in isolation; inspect ci_weight_validation.py retry behavior.
• [LOW] sglang/runtime regression in DeepSeek model load or hicache init for this config.
  • Supporting: test file has not passed in the queryable window.
  • Disconfirming: today's stack is entirely inside huggingface_hub/filelock download code — no sglang model-forward or hicache frame is implicated; the hang is pre-load, before any sglang-specific weight processing runs.
  • How to verify: with weights pre-cached, re-run and confirm the load completes; only then investigate hicache/runtime.
• [LOW] aiter involvement.
  • Supporting: none.
  • Disconfirming: aiter is the preserved dev build, unchanged; failure occurs during HF download before any kernel executes.
  • How to verify: not actionable.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for deepseek r1 hicache, weight download timeout, xet download; no matching fix. (curl .../search/issues?q=repo:sgl-project/sglang+is:pr+is:open+<kw> returned only unrelated PRs.)

Suggested Triage Steps

• Confirm whether DeepSeek-R1-0528 weights are pre-cached on the mi35x-rocm720 runner under /sgl-data/hf-cache; if the model is re-downloaded each run, pre-stage it or raise --timeout-per-file above 3600s for this 671B-model file.
• Time the HF/Xet download in isolation on this runner to distinguish network slowness from a server-side load hang.
• Cross-check the sibling NVIDIA / non-rocm720 DeepSeek-R1 hicache nightly to see whether the same download stalls there (would point to model-size/timeout config rather than ROCm720-specific infra).
• Separately investigate the exit code -6 scheduler crash seen Jun 20–21 (absent today) — it may be a distinct second failure mode for this test once weights do load.

Status (factual, no priority assignment)

• Persistence: This test file has failed in all 3 most recent completed scheduled runs (Jun 20, 21, 22); intervening Jun 16–19 runs were cancelled. No passing run in the queryable window.
• Scope: 1 job / 1 test file (nightly-8-gpu-mi35x-deepseek-r1-hicache-rocm720). Today's failure is a pure weight-download timeout; prior two days also showed a -6 scheduler crash.
• In-flight fix: ❌ none found in open PRs.

nightly-accuracy-8-gpu-mi35x-rocm720 — failed step(s): Accuracy Test MI35x ROCm 7.2 (8-GPU GPT-OSS)

Commit Info

• sglang: 43b1fb95f
• aiter: 7d604afe5 (source: dev image preserved — v0.1.14rc1.dev240+g7d604afe5, matched Dockerfile default, no rebuild)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi35x/test_qwen3_coder_next_eval_mi35x.py	test_qwen3_coder_next_accuracy	AssertionError: Failed models: ['Qwen/Qwen3-Coder-Next (basic)'] — basic (fp8_e4m3 kv-cache) variant produced no accuracy result; server hung (watchdog + detokenizer health-check timeouts)	Log
test/registered/amd/accuracy/mi35x/test_qwen3_moe_eval_mi35x.py	test_qwen3_moe_accuracy	AssertionError: Failed models: ['Qwen/Qwen3-30B-A3B'] — accuracy=0.380 threshold=0.75 FAIL	Log

The third file in the suite, test_gpt_oss_eval_mi35x.py, passed (gpt-oss-20b 0.520/0.47, gpt-oss-120b 0.865/0.79). Suite result: 1/3 passed; job exited 255.

Failure Summary

The MI35x 8-GPU nightly accuracy suite ran 3 files; 2 failed. test_qwen3_moe_accuracy for Qwen/Qwen3-30B-A3B scored 0.380 against a 0.75 GSM8K threshold. test_qwen3_coder_next_accuracy failed because its basic variant (the --kv-cache-dtype fp8_e4m3 configuration) hung — the server emitted repeated soft watchdog timeouts and detokenizer health-check failures and never returned an accuracy, so the test recorded it as an error. The mtp variant of the same model passed (0.975/0.9).

Regression Status

test_qwen3_moe_eval_mi35x.py (test_qwen3_moe_accuracy) — Never-passed since the test was added.
The file was introduced on 2026-06-20 by 111537366 (#28244, "[AMD] Fix garbled unquantized Qwen3-30B-A3B output on ROCm/aiter…"). It has failed in every completed scheduled run since.

Date	Run	Job	Test File Status	Accuracy	Error
Jun 22	27975850577	mi35x-rocm720	❌ Failed	0.380	below 0.75
Jun 21	27912828989	mi35x-rocm720	❌ Failed	0.405	below 0.75
Jun 20	27879239829	mi35x-rocm720	❌ Failed	0.375	below 0.75
Jun 15	27568866028	mi35x-rocm720	➖ Not in suite	—	(test did not exist yet)

(6/16–6/19 scheduled runs were cancelled → excluded. Lookback: 6/15–6/22. First observed failure 2026-06-20; no passing run exists in queryable history.)

test_qwen3_coder_next_eval_mi35x.py (test_qwen3_coder_next_accuracy) — New failure today (basic variant).

Date	Run	Job	Test File Status	basic / mtp	Error
Jun 22	27975850577	mi35x-rocm720	❌ Failed	basic=hang/ERROR · mtp=0.975 ✅	basic server hang
Jun 21	27912828989	mi35x-rocm720	✅ Passed	basic=0.965 · mtp=0.970	—
Jun 20	27879239829	mi35x-rocm720	✅ Passed	basic=0.965 · mtp=0.965	—
Jun 15	27568866028	mi35x-rocm720	✅ Passed	basic=0.965 · mtp=0.965	—

Last passing run 6/21 (head 7f67965b4dbb), first failing today (head 43b1fb95f). Regression window = 7f67965b4dbb..43b1fb95f (43 commits).

Failure Cluster

Two distinct clusters:

1. "Qwen3-30B-A3B GSM8K accuracy far below threshold (suspected garbled MoE output)"
2. "Qwen3-Coder-Next fp8-kv-cache server hang (soft watchdog + detokenizer health-check timeout)"

Facts (verified observations, no interpretation)

• accuracy=0.380 threshold=0.75 ❌ FAIL for Qwen/Qwen3-30B-A3B; benchmark completed 200/200 questions in 237s (it ran, then scored low) — verbatim from log L22384/L22389.
• Qwen3-30B-A3B accuracy has been tightly clustered ~0.375–0.405 across 3 completed runs (6/20, 6/21, 6/22), always below 0.75.
• The qwen3_moe test file was added 2026-06-20 by #28244 (111537366); test threshold hard-coded accuracy_threshold=0.75 (test file line 59).
• Coder-Next basic variant uses --kv-cache-dtype fp8_e4m3; mtp variant does not. Only basic failed; mtp scored 0.975 (L19591).
• During the coder_next window: 117 watchdog timeout log lines (soft, span 19:20:59→20:27:07) and 4 Health check failed. Server couldn't get a response from detokenizer lines (20:28:07→20:28:53).
• basic produced no accuracy=… threshold=… line, confirming it took the except Exception path (test file lines 248–257) rather than scoring low.
• gpt-oss-20b/120b passed; the failure is isolated to the two Qwen models.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [MEDIUM] Qwen3-30B-A3B unquantized MoE output is still degraded on ROCm/aiter (gfx950, rocm720), i.e. the fix in #28244 is incomplete or the 0.75 threshold is too high for this backend.

  • Supporting: the test was added by the very PR meant to fix garbled Qwen3-30B-A3B output, yet has never passed; accuracy ~0.38–0.40 (about half) is consistent with partially-garbled generations; symptom is model/backend-specific (gpt-oss passes).
  • Disconfirming: no green run of this test exists anywhere, so there is no baseline proving the model can reach 0.75 on this hardware — the threshold itself may simply be miscalibrated rather than the kernel being broken. Accuracy is stable (not noisy), which is unusual for a true correctness corruption.
  • How to verify: run Qwen/Qwen3-30B-A3B GSM8K manually on an MI35x/gfx950 + rocm720 + aiter container; inspect raw generations for garbling; compare aiter CK-vs-Triton fused-MoE path per #28244; if outputs are clean but score ~0.4, the threshold is wrong.

• [LOW] A commit in 7f67965b4dbb..43b1fb95f regressed the Coder-Next fp8-kv-cache path, causing the server/detokenizer hang.

  • Supporting: basic variant passed at 0.965 on 6/15, 6/20, 6/21 with the identical config and broke only today; window contains mamba/hicache-adjacent changes (e.g. 70883cb1b mamba hicache test rollback) and Qwen3-Coder-Next is a hybrid-mamba model.
  • Disconfirming: this is a single occurrence with a server-hang signature (soft watchdog + detokenizer non-response), which is the classic profile of a transient runner/GPU hang (flake), not a deterministic regression; the mtp variant on the same model/run passed cleanly; no commit in the window has been shown to touch the fp8-kv-cache decode/detokenizer path.
  • How to verify: re-run test_qwen3_coder_next_eval_mi35x.py once on MI35x; if basic passes, classify as flake; if it hangs again, bisect 7f67965b4dbb..43b1fb95f focusing on mamba/kv-cache/detokenizer commits.

No hypothesis above MEDIUM. The Coder-Next hang needs a confirming re-run before any bisect.

In-flight Fix Check

• Qwen3-30B-A3B accuracy: In-flight fix: ❌ none found — searches is:pr is:open Qwen3-30B-A3B+accuracy and Coder-Next+mi35x returned no PR targeting this accuracy gap (only loosely-matched unrelated PRs). Needs new investigation.
• Coder-Next fp8 hang: In-flight fix: ❌ none found — Qwen3-Coder-Next+fp8 returned only unrelated #22807 (SWA memory saver).

Suggested Triage Steps

• Re-run test_qwen3_coder_next_eval_mi35x.py once on an MI35x runner to disambiguate flake vs regression for the basic (fp8_e4m3) hang. If reproducible, bisect 7f67965b4dbb..43b1fb95f prioritizing mamba/kv-cache/detokenizer commits.
• For Qwen3-30B-A3B: manually run GSM8K on MI35x/gfx950 + rocm720 + aiter and inspect raw generations. If garbled, escalate to the #28244 author — the fix may not cover this code path on rocm720; if clean but ~0.4, the 0.75 threshold in the test (line 59) likely needs recalibration since the test has never passed.
• Confirm with maintainers whether Qwen3-30B-A3B was ever validated to clear 0.75 on this backend before #28244 enabled it in the nightly suite.

Status

• Persistence: test_qwen3_moe_accuracy — failing every completed run since 2026-06-20 (3/3), never passed. test_qwen3_coder_next_accuracy (basic) — first failure today (2026-06-22); passed the prior 3 completed runs.
• Scope: 1 job — nightly-accuracy-8-gpu-mi35x-rocm720 (nightly-test-amd-rocm720.yml), MI35x/gfx950. Not observed in the gpt-oss test of the same job.
• Blocked work: AMD MI35x rocm720 nightly accuracy signal for Qwen3-30B-A3B (persistently red) and Qwen3-Coder-Next fp8-kv-cache.
• In-flight fix: ❌ none found for either failure.

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-06-23 20:59 UTC)

[amd-bot]

nightly-test-amd-rocm720.yml — Per-Job Analysis (continued 2/3)

nightly-accuracy-8-gpu-mi35x-deepseek-v32-rocm720 — failed step(s): Accuracy Test MI35x ROCm 7.2 (8-GPU DeepSeek-V3.2)

Commit Info

• sglang: 43b1fb95f
• aiter: 7d604afe5 (v0.1.14rc1.dev240+g7d604afe5, source: dev image preserved — [CI-AITER-CHECK] Dev/patched version detected → skipping rebuild)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi35x/test_deepseek_v32_eval_mi35x.py	(whole-file harness run; eval never reached)	✗ TIMEOUT after 3600 seconds → exit code 255	Log

Failure Summary

The DeepSeek-V3.2 accuracy harness on MI35x (gfx950) / ROCm 7.2 timed out after 3600 s and was killed (exit 255, Test Summary: 0/1 passed). The server never finished initializing: Load weight begin printed at 22:17:37 but no Load weight end ever appeared. The eval (gsm8k/accuracy scoring) was never reached — the process was still loading model weights when the harness clock ran out.

Regression Status

Known recurring failure / Never-passed in queryable window.

Recent history of test_deepseek_v32_eval_mi35x.py in job nightly-accuracy-8-gpu-mi35x-deepseek-v32-rocm720 (branch=main, event=schedule, completed runs only):

Date	Run	Test File Status	Detail
Jun 22	27975850577	❌ Failed	TIMEOUT 3600s; weight load never completed
Jun 21	27912828989	❌ Failed	Load weight end. elapsed=1791.10 s; total > 3600s
Jun 20	27879239829	❌ Failed	watchdog timeouts; 0/1 passed
Jun 19–16	27841393725 …	⏭️ Cancelled	workflow cancelled — not comparable
Jun 15	27568866028	❌ Failed	Load weight end. elapsed=2425.94 s; 0/1 passed
Jun 14	27507292665	❌ Failed	0/1 passed

No passing run exists in the queryable window (lookback ~Jun 13–22; Jun 16–19 runs were cancelled). This test has been red for ≥10 days. The trend is worsening: weight load took 1791 s (Jun 15→2425 s), then never completed at all today.

Failure Cluster

DeepSeek-V3.2 fp8 MoE weight loading hang/timeout on MI35x ROCm 7.2 (_load_w2 HIP H2D memcpy)

Facts (verified observations)

• [FACT] Harness killed the run: ✗ TIMEOUT: …test_deepseek_v32_eval_mi35x.py after 3600 seconds → ##[error]Process completed with exit code 255 (log line 18124–18133). Retry skipped: "Exit code 255 indicates test failure (not network issue)".
• [FACT] Load weight begin. avail mem=276.88 GB at 22:17:37 (line 3844); no Load weight end anywhere in this log.
• [FACT] Server config: tp_size=8, quant=fp8/e4m3, mem_fraction_static=0.85, model_loader_extra_config='{"enable_multithread_load": true}', watchdog_timeout=1200, soft_watchdog_timeout=300.
• [FACT] Soft watchdog fired repeatedly from 22:22:10 to 23:12:25 (99 watchdog timeout lines) on TokenizerManager + all 8 schedulers — soft, so it dumped py-spy but did not kill.
• [FACT] Early py-spy (22:22:18, TP3 main thread): stuck in download_weights_from_hf (weight_utils.py:554) → _poll_until_acquired (filelock/_api.py:552) → acquire — i.e. blocked on the HF-cache download filelock.
• [FACT] Late py-spy (23:12:25, TP0 main thread): do_load_weights (deepseek_weight_loader.py:373) → as_completed — main thread waiting on worker threads.
• [FACT] Worker threads (ThreadPoolExecutor) stuck in _load_w2 (moe/fused_moe_triton/layer.py:571) → copy_ → at::native::copy_kernel_cuda → c10::hip::memcpy_and_sync (libtorch_hip.so) — a host→device HIP copy of MoE w2 weights.
• [FACT] Sister jobs on the SAME model + GPU passed in the same runs: nightly-perf-8-gpu-mi35x-deepseek-v32-basic-rocm720 ✅, …-mtp-rocm720 ✅, and nightly-accuracy-…-deepseek-v32-mtp-rocm720 ✅ (Jun 20/21). The model is loadable on this hardware.

Hypothesised Causes (with confidence + disconfirming evidence)

• [HIGH] Proximate cause: the test exceeds the 3600 s harness timeout because model initialization (HF weight download + fp8 MoE weight copy) is pathologically slow on this job.
  • Supporting: prior completed runs measured Load weight end. elapsed=1791 s / 2425 s — far beyond a healthy load; today it didn't finish in 3600 s. py-spy confirms the process is in weight download/load the entire window, never reaching eval.
  • Disconfirming: (none — this is directly observed).
  • How to verify: re-run with --watchdog-timeout unchanged but a larger harness timeout and capture Load weight end elapsed; or time download_weights_from_hf vs _load_w2 separately.
• [MEDIUM] Sub-cause: HF weight-download filelock contention / cold cache. The early stuck frame is download_weights_from_hf → filelock.acquire, meaning weights were still being downloaded (not just copied) and the rank was blocked on the shared download lock. On an 8-GPU node running several DeepSeek-V3.2 jobs concurrently, contention on the shared HF cache lock can consume a large fraction of the budget.
  • Supporting: filelock _poll_until_acquired in the 22:22 dump; monotonically worsening load times suggest an environmental (cache/disk/network) factor rather than a code change.
  • Disconfirming: sister jobs sharing the same node/cache loaded fine — so contention alone may not fully explain why only this variant hangs.
  • How to verify: check whether the HF cache for deepseek-ai/DeepSeek-V3.2 is pre-populated on the MI35x runner; inspect concurrent jobs holding the lock during this window.
• [LOW] Sub-cause: slow c10::hip::memcpy_and_sync H2D copy of fp8 MoE w2 weights under enable_multithread_load on ROCm 7.2.
  • Supporting: worker threads spend the whole late window inside the HIP memcpy; _load_w2 dominates the py-spy frames (89 occurrences).
  • Disconfirming: the basic/mtp jobs use the same loader path and copy and complete — so the copy primitive itself is not categorically broken; the difference is likely degree/contention, not a hard fault. No Memory access fault / HIP error appears (unlike the unrelated bingxche/debug-dsv32-mi35x-mtp-mem-access-fault debug branch).
  • How to verify: A/B with enable_multithread_load:false (or weight_loader_prefetch_checkpoints) and compare Load weight end elapsed.

No hypothesis for a specific culprit commit reaches MEDIUM — the failure is a long-standing slowness/timeout, not a sharp regression tied to one commit. A bisect on load-time across nightly-test-amd-rocm720.yml history would be needed to find when load time first crossed the timeout.

In-flight Fix Check

• In-flight fix: ❌ none found directly targeting MI35x DeepSeek-V3.2 slow weight loading. Searched is:pr is:open deepseek v32 mi35x and weight load slow.
• Possibly-relevant existing PRs (NOT confirmed fixes): #25498 "[CI] Enable weight loader prefetch in CI", #9305 "Support setting UNBALANCED_MODEL_LOADING_TIMEOUT_S through environment variables". Note an active debug branch bingxche/debug-dsv32-mi35x-mtp-mem-access-fault exists upstream, but that targets a mem-access-fault (different symptom), not this slow-load timeout.

Suggested Triage Steps

• Instrument the load phase: separate download_weights_from_hf time from _load_w2 copy time in a one-off run to confirm whether download-lock or H2D-copy dominates the 3600 s.
• Pre-warm / pin the deepseek-ai/DeepSeek-V3.2 HF cache on the MI35x runner so the download/filelock phase is skipped; re-run and re-measure Load weight end elapsed.
• A/B the loader: re-run with enable_multithread_load:false and with weight_loader_prefetch_checkpoints to see if either changes load time materially.
• If load is genuinely ~30–40 min on this hardware, raise the harness est_time/timeout for this specific accuracy test (currently 3600 s with est 5400 s) so eval can actually run — or split download from eval — pending a real load-speed fix.
• Bisect load-time across nightly-test-amd-rocm720.yml scheduled runs to find when this job's load first crossed 3600 s.

Status

• Persistence: Failing every completed scheduled run since at least Jun 14 (≥10 days); Jun 16–19 runs were cancelled (no green data). Worsening — load time 1791 s → 2425 s → did-not-finish.
• Scope: Only nightly-accuracy-8-gpu-mi35x-deepseek-v32-rocm720 (non-MTP accuracy variant). Sister jobs (-basic, -mtp perf, -mtp accuracy) pass — failure is isolated to this one job/test.
• Blocked work: DeepSeek-V3.2 non-MTP accuracy signal on MI35x ROCm 7.2.
• In-flight fix: ❌ none found directly; candidates #25498, #9305 unverified.

nightly-4-gpu-rocm720 — failed step(s): Nightly Test ROCm 7.2 (4-GPU)

Commit Info

• sglang: 43b1fb95f6
• aiter: 7d604afe5 (v0.1.14rc1.dev240+g7d604afe5, source: dev image preserved — [CI-AITER-CHECK] Dev/patched version detected → skipping rebuild, AITER_COMMIT_OVERRIDE empty)

Failed Tests

Test File	Test Function	Error	Log
test/registered/vlm/test_encoder_dp.py	test_vlm_mmmu_benchmark (model Qwen/Qwen3-VL-32B-Instruct)	AssertionError: 0.25222 not greater than or equal to 0.55 (server crashed mid-eval)	Log

Failure Summary

The 4-GPU VLM encoder-DP MMMU benchmark selected Qwen/Qwen3-VL-32B-Instruct and reported accuracy 0.2522, below the 0.55 threshold. The low score is not a genuine accuracy result — the scheduler crashed during the run with torch.AcceleratorError: HIP error: invalid device pointer in the DP-encoder multimodal CUDA/HIP IPC transport path (crash log line). After the crash (SIGQUIT received ... one child failed), every subsequent lmms-eval request failed with Connection error (10,800 occurrences), so the model effectively scored at ~random chance (0.2522 ≈ 25% on 4-choice MMMU).

Regression Status

Known recurring failure (deterministic crash, surfaces flakily at job level due to random model selection + retry).

Recent history of test_encoder_dp.py (test_vlm_mmmu_benchmark) in job nightly-4-gpu-rocm720 (branch main, event schedule, completed runs only):

Date	Run	Job conclusion	Model selected	Test File Status	Detail
Jun 22	27975850577	failure	Qwen3-VL-32B	❌ Failed	HIP IPC crash → acc 0.2522
Jun 21	27912828989	success	GLM-4.1V-9B	❌ Crashed (1st attempt)	HIP IPC crash → acc 0.2522; passed on retry (different model)
Jun 20	27879239829	success	InternVL2_5-8B	✅ Passed	acc 0.5333, no crash
Jun 15	27568866028	failure	—	❌ Failed	TIMEOUT after 3600s
Jun 14	27507292665	success	Qwen3-VL-32B	❌ Crashed (1st attempt)	HIP IPC crash → acc 0.2522; passed on retry (different model)
Jun 13	27474496879	success	InternVL2_5-8B	✅ Passed	acc 0.5311, no crash

First observed HIP-IPC crash for this test in the queryable window: Jun 14 2026. The crash reproduces every time a heavier VLM (Qwen3-VL-32B, GLM-4.1V-9B) is selected; it is masked when the random selector + retry lands on InternVL2_5-8B, which passes cleanly. Job-level pass/fail is therefore flaky, but the underlying crash is deterministic and pre-existing.

Failure Cluster

HIP "invalid device pointer" crash in DP-encoder multimodal CUDA/HIP IPC reconstruct

Facts (verified observations)

• Crash stack (verbatim from log):
  • scheduler.py:1945 _get_multimodal_inputs → schedule_batch.py:516 from_processor_output → schedule_batch.py:365 reconstruct → cuda_ipc_transport_utils.py:488 reconstruct_on_target_device → torch.UntypedStorage._new_shared_cuda(...) → torch.AcceleratorError: HIP error: invalid device pointer.
• Preceding line: [TP0] Failed to deserialize from CUDA IPC handle (HIP error: invalid device pointer), then SIGQUIT received ... one child failed.
• The crash occurs on the multimodal IPC handle reconstruct path, exercised because the test runs with --mm-enable-dp-encoder --tp=4 (feature ships vision features between encoder-DP ranks and the scheduler via CUDA/HIP IPC). Source confirms the failing call is torch.UntypedStorage._new_shared_cuda(*redirected_handle) in python/sglang/srt/utils/cuda_ipc_transport_utils.py:488.
• Degenerate score 0.25222 is byte-identical across Jun 14, Jun 21, Jun 22 — consistent with all requests failing post-crash rather than genuine model output.
• Reproducibility: HIP-IPC crash observed in 3 of the last 6 completed scheduled runs (every run where a non-InternVL model was selected on the first attempt).
• aiter version unchanged (dev image v0.1.14rc1.dev240+g7d604afe5) across the crashing and passing runs — not an aiter version delta.

Hypothesised Causes (with confidence)

• [HIGH] The DP-encoder multimodal feature transport reconstructs a CUDA IPC handle on a target HIP device via torch.UntypedStorage._new_shared_cuda, which fails on ROCm with hipErrorInvalidDevicePointer.
  • Supporting: the failing frame is exactly cuda_ipc_transport_utils.py:488 in the reconstruct path; crash is gated on --mm-enable-dp-encoder; identical stack on Jun 14/21/22; AMD-only path (HIP IPC).
  • Disconfirming: InternVL2_5-8B runs the same --mm-enable-dp-encoder path and passes (Jun 13, Jun 20), so the crash is not unconditional — it may be tied to feature-tensor size/layout, multi-rank timing, or pooled-handle lifetime for larger encoders rather than the IPC call being categorically broken on ROCm.
  • How to verify: run test_encoder_dp.py pinned to Qwen/Qwen3-VL-32B-Instruct on a 4-GPU MI325 ROCm 7.2 box with --mm-enable-dp-encoder --tp=4; if it crashes, A/B by removing --mm-enable-dp-encoder (expect no IPC reconstruct, no crash). Set AMD_SERIALIZE_KERNEL=3 for a precise async-HIP stack.
• [LOW] IPC handle lifetime / freed source storage (sender frees the device buffer before the scheduler reconstructs it, leaving a stale pointer).
  • Supporting: invalid device pointer is the classic symptom of a handle pointing at already-freed device memory; larger encoders take longer, widening the free-before-reconstruct race.
  • Disconfirming: no direct evidence of an early free in the log; needs code-level tracing of handle ownership in cuda_ipc_transport_utils.py.
  • How to verify: trace producer-side storage lifetime vs consumer reconstruct timing; test the pooled vs non-pooled handle path.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for dp encoder ipc and invalid device pointer; no matching fix. (API calls: search/issues?q=repo:sgl-project/sglang+is:pr+is:open+dp+encoder+ipc and ...+invalid+device+pointer. Nearest-but-unrelated: #16006 DP ViT refactor, #23377 MM pool GPU alloc — neither targets this HIP IPC crash.)

Suggested Triage Steps

• Reproduce deterministically: pin test_encoder_dp.py to Qwen/Qwen3-VL-32B-Instruct (or GLM-4.1V-9B-Thinking) on 4-GPU MI325 ROCm 7.2, since random model selection currently masks the crash on InternVL2_5-8B.
• A/B the feature flag: rerun the pinned model with vs without --mm-enable-dp-encoder to confirm the IPC reconstruct path is the trigger.
• Inspect python/sglang/srt/utils/cuda_ipc_transport_utils.py:488 (reconstruct_on_target_device) for HIP IPC handle/device-pointer validity and producer-side storage lifetime; capture an exact stack with AMD_SERIALIZE_KERNEL=3.
• If confirmed ROCm-specific and not quickly fixable, consider gating the random selector to skip crash-triggering models on AMD (or marking the DP-encoder IPC path unsupported on ROCm) until a fix lands — and separately investigate the Jun 15 3600s timeout, which may be a related hang in the same path.

Status

• Persistence: HIP-IPC crash recurring since at least Jun 14 2026; present in 3 of last 6 completed scheduled runs (every run where a non-InternVL model was picked first).
• Scope: nightly-4-gpu-rocm720 job, test_encoder_dp.py. Flaky at job level (passes when retry selects InternVL2_5-8B), so true failure rate is masked.
• Blocked work: VLM encoder-DP (--mm-enable-dp-encoder) signal on AMD ROCm 7.2 is unreliable; green runs do not verify the larger-VLM IPC path.
• In-flight fix: ❌ none found in open PRs.

nightly-8-gpu-qwen35-rocm720 — failed step(s): Accuracy Test ROCm 7.2 (8-GPU Qwen 3.5)

Commit Info

• sglang: 43b1fb95f
• aiter: N/A (no [CI-AITER-CHECK] markers in this nightly-test log; AITER_COMMIT_OVERRIDE empty)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi30x/test_qwen35_eval_amd.py	TestQwen35EvalAMD.setUpClass	AssertionError: extra_buffer needs CUDA/MUSA/NPU (FLA). (server exits code 1 → setUpClass error)	Log

Failure Summary

The 8-GPU Qwen3.5-397B-A17B accuracy test failed at setUpClass: the SGLang server process exited with code 1 before serving. The crash is an AssertionError: extra_buffer needs CUDA/MUSA/NPU (FLA). raised in server_args.py:4366 (_validate_mamba_extra_buffer) during ServerArgs.__post_init__. Qwen3.5-397B-A17B is a hybrid (Gated-Delta/mamba) MoE model, and the mamba radix-cache auto strategy selected extra_buffer, which is gated to CUDA/MUSA/NPU and rejects ROCm.

Regression Status

New regression (started 2026-06-20).

Recent history of test_qwen35_eval_amd.py (TestQwen35EvalAMD.setUpClass) in job nightly-8-gpu-qwen35-rocm720 (same branch main, event=schedule, completed runs only):

Date	Run	Job	Test File Status	Failed Function	Error
Jun 22	27975850577	nightly-8-gpu-qwen35-rocm720	❌ Failed	setUpClass	extra_buffer needs CUDA/MUSA/NPU (FLA)
Jun 21	27912828989	nightly-8-gpu-qwen35-rocm720	❌ Failed	setUpClass	same
Jun 20	27879239829	nightly-8-gpu-qwen35-rocm720	❌ Failed	setUpClass	same
Jun 16–19	—	—	⚪ Cancelled (no data)	—	—
Jun 15	27568866028	nightly-8-gpu-qwen35-rocm720	✅ Passed	—	—

First observed failure: 2026-06-20. Last known passing: 2026-06-15 (head 33719cfb31). Jun 16–19 nightlies were cancelled, so the first observable failure is Jun 20 (head ff1fc1fbdf). Regression window: 33719cfb31..ff1fc1fbdf (221 commits).

Failure Cluster

Mamba extra_buffer strategy rejected on ROCm (AssertionError at server arg validation)

Facts (verified observations, no interpretation)

• Verbatim crash (job log line 3781): AssertionError: extra_buffer needs CUDA/MUSA/NPU (FLA).
• Stack trace path (job log 3766–3781): __post_init__ → _handle_model_specific_adjustments → _handle_mamba_radix_cache (server_args.py:4395) → _validate_mamba_extra_buffer (server_args.py:4366).
• Server launch command: sglang serve --model-path Qwen/Qwen3.5-397B-A17B --tp 8 --attention-backend aiter --device cuda ... (job log 3789). Despite --device cuda, the runtime is ROCm, so is_cuda() returns False.
• Same test file + function failed with the identical assertion on Jun 20 (job 82503808402) and Jun 21 (job 82592807996); passed Jun 15.
• Current server_args.py:4366-4368 asserts is_cuda() or is_musa() or is_npu() for extra_buffer; auto selects extra_buffer at lines 4386–4389 when overlap/paging is wanted and the model supports it.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [HIGH] Commit 66a7fd5c0b ("refactor: mamba radix cache server args initialize", #28151, authored 2026-06-18) changed the mamba radix-cache auto default from no_buffer to extra_buffer for hybrid models and removed the prior explicit ROCm guard.
  • Supporting: The commit diff replaces the old auto path (which set mamba_scheduler_strategy = "no_buffer" with a TODO and an explicit # On ROCm, extra_buffer is unsupported. comment) with if (wants_overlap or wants_paging) and self._support_mamba_cache_extra_buffer(model_arch): self.mamba_radix_cache_strategy = "extra_buffer". It also introduces the exact assertion extra_buffer needs CUDA/MUSA/NPU (FLA). Overlap schedule is on by default and Qwen3.5 supports extra_buffer, so auto→extra_buffer→assert fails on ROCm. Authored 2026-06-18, inside the regression window; first failure Jun 20.
  • Disconfirming: (none found) — the test passed Jun 15 (before the commit) and has failed every completed run since; the removed ROCm guard and the new assertion are both in this single commit.
  • How to verify: On a branch, set mamba_radix_cache_strategy default to no_buffer on ROCm (or pass --mamba-radix-cache-strategy no_buffer) and re-run test_qwen35_eval_amd.py on AMD MI325; or git revert 66a7fd5c0b and re-run.

In-flight Fix Check

• In-flight fix: ✅ #27141 "[ROCm] Support radix-cache with mamba-extra-buffer for Qwen3.5" (open since 2026-06-03, last updated 2026-06-22, not draft) — directly targets enabling extra_buffer on ROCm for this exact model. Chase reviewers; do NOT open a duplicate.
• Search ran: is:pr is:open mamba+extra_buffer+ROCm (1 hit), extra_buffer+FLA (3 hits).

Suggested Triage Steps

• Confirm culprit: git revert 66a7fd5c0b (#28151) on a branch and re-run test_qwen35_eval_amd.py on AMD MI325 ROCm 7.2; expect the assertion to disappear.
• Verify #27141 merges cleanly and resolves the ROCm path; if so, prioritize its review — it is the intended forward fix.
• Interim unblock for nightly signal: pass --mamba-radix-cache-strategy no_buffer for the Qwen3.5 AMD test (or re-add a ROCm guard so auto falls back to no_buffer on ROCm) until #27141 lands.

Status (factual, no priority assignment)

• Persistence: Failing for 3 completed nightly runs (Jun 20, 21, 22); last green Jun 15. Jun 16–19 nightlies were cancelled.
• Scope: nightly-8-gpu-qwen35-rocm720 in nightly-test-amd-rocm720.yml. Sister job nightly-8-gpu-mi35x-qwen35-rocm720 shows the same failure pattern on Jun 20/21 (also ROCm), so the regression spans both ROCm runner variants for this model.
• Blocked work: Qwen3.5-397B-A17B GSM8K accuracy signal on AMD ROCm 7.2.
• In-flight fix: ✅ #27141 (open since 2026-06-03) — awaiting merge.

nightly-8-gpu-minimax-m27-rocm720 — failed step(s): Accuracy Test ROCm 7.2 (8-GPU MiniMax-M2.7)

Commit Info

• sglang: 43b1fb95f
• aiter: 7d604afe5 (source: dev image preserved — v0.1.14rc1.dev240+g7d604afe5, no override, no rebuild)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi30x/test_minimax_m27_eval_amd.py	test_minimax_m27_accuracy	AssertionError: Failed models: ['MiniMaxAI/MiniMax-M2.7 (TP8+EP8)'] — accuracy=0.910 threshold=0.93 FAIL	Log

Failure Summary

The GSM8K completion accuracy eval for MiniMaxAI/MiniMax-M2.7 (TP8+EP8) scored 0.910 against a hard threshold of 0.93, failing the single test in the job. The model loaded and served all 200 prompts successfully (200/200 generated, HTTP 200); the only failure is the accuracy assertion at test_minimax_m27_eval_amd.py:241. Exit code 255 → job failed.

Regression Status

Known recurring failure (marginal accuracy) — 2 consecutive completed runs below threshold, with a clear downward drift from the last pass.

Recent history of test_minimax_m27_eval_amd.py (test_minimax_m27_accuracy) in job nightly-8-gpu-minimax-m27-rocm720 (branch main, event schedule, completed runs only):

Date	Run	Test File Status	Accuracy	Error
Jun 22	27975850577	❌ Failed	0.910	below 0.93
Jun 21	27912828989	❌ Failed	0.920	below 0.93
Jun 20	27879239829	✅ Passed	0.935	margin +0.005

First observed failure: Jun 21 2026. Last known pass: Jun 20 2026 (at 0.935 — a razor-thin +0.005 margin). Jun 16-19 runs were cancelled (excluded). The eval is non-deterministic across runs — random_seed differs each run (Jun 20 896765843, Jun 21 767479822, Jun 22 480934612).

Failure Cluster

MiniMax-M2.7 GSM8K accuracy below threshold (TP8+EP8, AMD rocm720)

Facts (verified observations)

• accuracy=0.910 threshold=0.93 FAIL at log line 7217; assertion raised at test_minimax_m27_eval_amd.py:241.
• Threshold is accuracy_threshold=0.93 (hardcoded in the config, test_minimax_m27_eval_amd.py:67); dataclass default is 0.50.
• Accuracy sequence over the last 3 completed runs: 0.935 (PASS) → 0.920 (FAIL) → 0.910 (FAIL). On 200 GSM8K samples, 0.935→0.910 = 5 fewer correct answers (each sample = 0.5%).
• random_seed is not fixed — it differs every run, so run-to-run variance is expected.
• aiter version (v0.1.14rc1.dev240+g7d604afe5) is unchanged across all three runs (dev image preserved, [CI-AITER-CHECK] at log line 3552-3557).
• Validation failed for MiniMaxAI/MiniMax-M2.7: Missing shards ... Will attempt to download missing files appears in all three runs including the Jun 20 pass (log line 4271) — benign, recovers, not correlated with the failure.
• Health check failed ... no response from detokenizer appears late in both failing runs (log line 6226+) but generation completed (200/200) and accuracy was still computed — likely end-of-eval teardown noise, not the cause.
• Commit window from last pass to first fail (ff1fc1fbdf..7f67965b4d) is only 24 commits; none obviously touch the MiniMax/MoE/EP forward path (the one MoE-adjacent commit, 535180070 MTP acceptance, is speculative-decoding only and this config runs no speculative decoding).

Hypothesised Causes (with confidence)

• [LOW] GSM8K eval variance around a too-tight threshold. The last pass cleared 0.93 by only +0.005 with a different random seed; seeds vary every run, so a 0.910-0.935 spread is consistent with sampling noise rather than a code defect.
  • Supporting: razor-thin pass margin (0.935 vs 0.93); random_seed not fixed; model serves all 200 prompts cleanly.
  • Disconfirming: the trend is monotonically downward (0.935→0.920→0.910) over 3 runs, which is more than pure noise would typically produce — a small real regression cannot be ruled out.
  • How to verify: re-run the job 3-5× pinning a fixed seed; if accuracy oscillates ≥0.93 sometimes, it is variance/threshold; if it stays <0.93, it is a regression.
• [LOW] A sglang commit in ff1fc1fbdf..7f67965b4d introduced a small numerical regression in the MiniMax MoE/attention path.
  • Supporting: failure first appears in the Jun 21 run whose head is 7f67965b4d; window is small (24 commits).
  • Disconfirming: no commit in the window touches the MiniMax/MoE/EP forward path; aiter is unchanged; the eval seed is non-deterministic so the timing correlation is weak.
  • How to verify: bisect ff1fc1fbdf..7f67965b4d with a fixed seed on AMD MI30x TP8+EP8.

No hypothesis above LOW confidence; the non-deterministic seed plus a 0.005 historical pass margin make this most likely a too-tight threshold, but a small regression in the 24-commit window cannot be excluded without a fixed-seed bisect.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for minimax+accuracy (37 results, none about this eval/threshold) and minimax+threshold (3 results, all unrelated: router observability, M1 model support, gfx950 Dockerfile). No PR adjusts the MiniMax-M2.7 GSM8K threshold or addresses this regression.

Suggested Triage Steps

• Re-run test_minimax_m27_eval_amd.py 3-5× with a fixed random_seed on AMD MI30x (TP8+EP8) to separate eval variance from a real regression — this is the cheapest discriminator and should be done first.
• If accuracy is stable <0.93 with a fixed seed, bisect ff1fc1fbdf..7f67965b4d (24 commits) for the regression.
• If it is confirmed to be eval noise, consider whether the 0.93 threshold (last pass margin only +0.005) is too tight for a non-deterministic-seed eval, and/or increasing the GSM8K sample count above 200 to reduce variance — owner decision.

Status

• Persistence: Failing 2 consecutive completed runs (since Jun 21 2026); last pass Jun 20 at 0.935.
• Scope: 1 job, 1 test (nightly-8-gpu-minimax-m27-rocm720 in nightly-test-amd-rocm720.yml). Sister 4-GPU MiniMax-M2.5 job passed in the same runs.
• Trend: accuracy drifting down 0.935 → 0.920 → 0.910.
• In-flight fix: ❌ none found.

nightly-8-gpu-mi35x-glm5-mxfp4-rocm720 — failed step(s): Accuracy Test MI35x ROCm 7.2 (8-GPU GLM-5-MXFP4)

Commit Info

• sglang: 43b1fb95f
• aiter: 7d604afe5 (v0.1.14rc1.dev240+g7d604afe5, source: Dockerfile default — dev/patched image preserved, AITER_COMMIT_OVERRIDE empty)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi35x/test_glm5_mxfp4_eval_mi35x.py	test_glm5_mxfp4_accuracy (TestGLM5MXFP4EvalMI35x)	AssertionError: Failed models: ['amd/GLM-5-MXFP4 (mxfp4)'] (line 247)	Log

Failure Summary

The single GSM8K accuracy test for amd/GLM-5-MXFP4 (TP8, mi35x, ROCm 7.2) failed after running 5410s. The server never reached readiness: all 8 TP scheduler processes were stuck loading model weights — blocked on a HuggingFace download filelock while TP3 held the lock and stalled inside the HF Xet downloader. The benchmark therefore got no usable server and the harness recorded the model as failed.

Regression Status

New today / Infrastructure (model-download hang) — not a code regression.

Recent history of test_glm5_mxfp4_eval_mi35x.py (test_glm5_mxfp4_accuracy) in job nightly-8-gpu-mi35x-glm5-mxfp4-rocm720 (branch=main, event=schedule, completed runs only):

Date	Run	Job	Test File Status	Failed Function	Error
Jun 22	27975850577	nightly-8-gpu-mi35x-glm5-mxfp4-rocm720	❌ Failed	test_glm5_mxfp4_accuracy	weight-download hang → AssertionError
Jun 21	27912828989	nightly-8-gpu-mi35x-glm5-mxfp4-rocm720	✅ Passed	—	—
Jun 20	27879239829	nightly-8-gpu-mi35x-glm5-mxfp4-rocm720	✅ Passed	—	—
Jun 15	27568866028	nightly-8-gpu-mi35x-glm5-mxfp4-rocm720	✅ Passed	—	—

(Jun 16–19 scheduled runs were cancelled, so not comparable. First (and only) observed failure: Jun 22; last passing: Jun 21.)

Failure Cluster

HuggingFace weight-download hang during model load (Xet/filelock stall) → server never ready

Facts (verified observations, no interpretation)

• The test command launched sglang serve --model-path amd/GLM-5-MXFP4 ... --tp 8 --watchdog-timeout 1200 (log line ~11951).
• From 22:29 to 23:49 the soft watchdog fired every 300s on every TP rank. Each dump shows the scheduler MainThread blocked in:
  download_weights_from_hf (weight_utils.py:554) → __enter__ (filelock/_api.py:581) → acquire (filelock/_api.py:506) → _poll_until_acquired (filelock/_api.py:552) (e.g. TP4/TP5/TP1 at 22:29, TP6/TP7 at 23:49).
• The final dump of TP3 (pid 3332, the lock holder) at 23:49:12 shows it actively downloading, not deadlocked: download_weights_from_hf (weight_utils.py:593) → ci_download_with_validation_and_retry (ci_weight_validation.py:1955) → snapshot_download (_snapshot_download.py:467) → thread_map, with an active worker thread ThreadPoolExecutor-2_0 in xet_get (huggingface_hub/file_download.py:566) → _download_to_tmp_and_move (file_download.py:1861).
• [2026-06-22 22:44:05] _get_metadata_or_catch_error (huggingface_hub/file_download.py:1691) confirms HF metadata/download activity mid-run.
• py-spy dump --native --pid 3332 repeatedly failed (failed to get os threadid) because TP3 was busy in the native Xet download path; the non-native dump succeeded and revealed the active download.
• Test ran 5410.9s then FAILED (failures=1); harness exit code 255; "not retrying" (treated as test failure, not network).
• This is nightly-test-amd-rocm720.yml (not aiter-scout); AITER_COMMIT_OVERRIDE empty, so aiter is the Dockerfile default and not a variable here.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [HIGH] Model-weight download from HuggingFace stalled (HF Xet backend), so the server never finished loading within the test window and GSM8K had no server to hit.
  • Supporting: every watchdog dump for ~90 min shows the schedulers stuck in download_weights_from_hf; the lock-holding rank (TP3) is in an active xet_get download thread that never completes; no "server ready"/warmup/CUDA-graph-capture lines ever appear after launch.
  • Disconfirming: none found — the stack traces directly show the hang is in the download path, not in model forward/kernel code.
  • How to verify: re-run the job; check HF Hub / Xet availability and the runner's network egress to cdn-lfs/Xet at the failure time; inspect whether amd/GLM-5-MXFP4 shards were partially cached.
• [LOW] sglang code change in this nightly's commit window affecting the loader/download path.
  • Supporting: temporal — failure is new on Jun 22 (43b1fb95f) vs pass on Jun 21 (7f67965b4d).
  • Disconfirming: the hang is inside huggingface_hub (xet_get/snapshot_download), an external library, not in changed sglang code; the symptom is a network/IO stall, not a logic error. Strongly favors infrastructure over code.
  • How to verify: git log --oneline 7f67965b4d..43b1fb95f -- python/sglang/srt/model_loader/ to see if anything touched the download path; if empty, this hypothesis is effectively ruled out.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for xet+download+hang, weight+download+filelock, hf+download+timeout (GET /search/issues?q=repo:sgl-project/sglang+is:pr+is:open+<kw>); no matching PR. As infrastructure, a code fix may not be warranted.

Suggested Triage Steps

• Re-run nightly-8-gpu-mi35x-glm5-mxfp4-rocm720 — if it passes, this confirms a transient HF/Xet download stall and no code action is needed.
• If it recurs, check runner network egress to HuggingFace Xet/CDN at run time and whether amd/GLM-5-MXFP4 weights can be pre-cached on the mi35x runners to avoid live download during the test.
• Consider a download-level timeout/retry around snapshot_download/xet_get so a stalled shard fails fast instead of consuming the full 5400s budget (note ci_download_with_validation_and_retry is already in the path but did not break the Xet hang).
• Confirm code is not implicated: git log --oneline 7f67965b4d..43b1fb95f -- python/sglang/srt/model_loader/.

Status (factual, no priority assignment)

• Persistence: 1 completed run (Jun 22); the same job passed Jun 21, Jun 20, Jun 15. Not previously failing.
• Scope: this job only (nightly-8-gpu-mi35x-glm5-mxfp4-rocm720); single test file.
• Blocked work: GLM-5-MXFP4 nightly accuracy signal on mi35x / ROCm 7.2 for this run.
• In-flight fix: ❌ none found — likely transient infrastructure; re-run before opening any PR.

nightly-8-gpu-mi35x-glm51-rocm720 — failed step(s): Accuracy Test MI35x ROCm 7.2 (8-GPU GLM-5.1 DSA)

Commit Info

• sglang: 43b1fb95f
• aiter: 7d604afe5 (v0.1.14rc1.dev240+g7d604afe5, source: dev image preserved — [CI-AITER-CHECK] Dev/patched version detected → skipping rebuild, AITER_COMMIT_OVERRIDE empty)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi35x/test_glm51_eval_mi35x.py	test_glm51_accuracy	AssertionError: Failed models: ['zai-org/GLM-5.1-FP8 (dsa)']	Log

Failure Summary

The single accuracy test test_glm51_accuracy (model zai-org/GLM-5.1-FP8 with DSA tilelang backend, TP8) failed after running 5411s. The server never finished initializing: all 8 TP ranks logged Load weight begin at 20:00:42 but the weight load stalled inside the HuggingFace Hub download path for ~85 minutes until the test gave up. No evaluation/accuracy output was ever produced (accuracy= appears 0 times in the log).

Regression Status

New occurrence (new symptom today) / not yet recurring for this cluster.

Recent history of test/registered/amd/accuracy/mi35x/test_glm51_eval_mi35x.py (test_glm51_accuracy) in job nightly-8-gpu-mi35x-glm51-rocm720 (branch main, event schedule, completed runs only):

Date	Run	Job	Test File Status	Failed Function	Error
Jun 22	27975850577	glm51-rocm720	❌ Failed	test_glm51_accuracy	weight-download hang → Failed models
Jun 21	27912828989	glm51-rocm720	✅ Passed	—	—
Jun 20	27879239829	glm51-rocm720	✅ Passed	—	—
Jun 15	27568866028	glm51-rocm720	❌ Failed	test_glm51_accuracy	different cause: accuracy=0.005 threshold=0.93 (model loaded, eval ran)

(Jun 16–19 scheduled runs were cancelled and excluded. Last passing run: Jun 21. The Jun 15 failure is a different cluster — there the model loaded and produced a 0.005 accuracy score; today the model never loaded.)

Failure Cluster

GLM-5.1-FP8 weight download hang during server warmup (HF Hub xet_get / filelock stall — eval never runs)

Facts (verified observations, no interpretation)

• All 8 TP ranks: Load weight begin. avail mem≈277 GB at 20:00:42 (lines 3856-3863). No corresponding "load weight end" / server-ready line ever appears.
• Soft scheduler watchdog fired every ~5 min from 20:05:18 to 21:25:27 (Scheduler watchdog timeout (self.watchdog_timeout=300, self.soft=True)).
• py-spy dump of sglang::scheduler_TP3 (PID 3332) shows the scheduler MainThread blocked in the weight-download path:
  download_weights_from_hf (weight_utils.py:593) → ci_download_with_validation_and_retry (ci_weight_validation.py:1955) → snapshot_download (_snapshot_download.py:467) → thread_map/xet_get (file_download.py:566); sibling threads block in filelock _poll_until_acquired (_api.py:552) (lines 4080-4180).
• _poll_until_acquired appears 119 times and xet_get 17 times across the log — repeated stalls on the same download.
• xet_get stack frame (line 11905) confirms the HF Xet transfer backend is in use and is the stuck call.
• Test failed after Ran 1 test in 5411.290s; 0 accuracy= lines today (eval never executed). Exit code 255.
• Earlier at 19:51:27 a local docker-registry pull timed out (net/http: request canceled ... Client.Timeout, line 1283) and fell back to remote pull — unrelated to the weight-download hang but corroborates network flakiness on this runner.
• aiter unchanged from Jun 20/21 passing runs (same dev image lineage); not a variable here.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [HIGH] Proximate cause is a stalled HuggingFace Hub weight download (Xet backend / filelock contention), not model or kernel code.
  • Supporting: py-spy stacks pin the hang in snapshot_download→xet_get/filelock; no GPU/model forward frames; eval never ran; soft watchdog (informational) never escalated.
  • Disconfirming: (none found) — every watchdog dump points to the same download frame.
  • How to verify: re-run the job; if it passes with no code change, confirms transient infra. Check runner network/HF-Xet availability at 2026-06-22 20:00–21:25 UTC.
• [LOW] Underlying trigger is runner network / HF Hub Xet backend degradation (infrastructure).
  • Supporting: same test passed Jun 20 & Jun 21 on similar SHAs; a docker-registry pull also timed out earlier in this run (line 1283); no sglang code change touches the HF download path between Jun 21 and Jun 22 that would explain a hang.
  • Disconfirming: cannot fully rule out a CI weight-validation/caching change without enumerating the Jun 21→22 delta; the hang sits partly in ci_weight_validation.py (sglang code).
  • How to verify: git log --oneline 7f67965b4d..43b1fb95f -- python/sglang/srt/model_loader/ci_weight_validation.py python/sglang/srt/model_loader/weight_utils.py to check for any download-path change in the window.
• [SPECULATION] Pre-seeded weight cache missing on this runner, forcing a cold full download that exceeded the ~90 min budget.
  • Supporting: cold xet_get of a large FP8 GLM-5.1 checkpoint over a flaky link could plausibly exceed the window.
  • Disconfirming: no explicit cache-miss log line observed.
  • How to verify: inspect runner HF cache state / compare download start-to-stall duration against a passing run.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for xet download (https://api.github.com/search/issues?q=repo:sgl-project/sglang+is:pr+is:open+xet+download → 0 results). This is most likely transient infrastructure; no code fix is expected to be in flight.

Suggested Triage Steps

• Re-run job nightly-8-gpu-mi35x-glm51-rocm720 on run 27975850577; if it passes unchanged, close as transient HF-Xet/network stall.
• Check the MI35x runner's network and HuggingFace Hub Xet backend health for the 2026-06-22 20:00–21:25 UTC window (the docker-registry timeout at line 1283 suggests broader connectivity flakiness).
• Confirm no code regression: git log --oneline 7f67965b4d..43b1fb95f -- python/sglang/srt/model_loader/ to verify the download/validation path is unchanged since the Jun 21 passing run.
• If GLM-5.1-FP8 weights are not pre-seeded on this runner, consider pre-caching the checkpoint so the test does not depend on a cold cross-network download within the ~90 min budget.
• Keep this separate from the Jun 15 accuracy=0.005 failure — that is a distinct (numerical) cluster, not the download hang seen today.

Status (factual, no priority assignment)

• Persistence: First seen in this cluster today (2026-06-22). Same test passed Jun 20 and Jun 21; the only earlier failure (Jun 15) had a different root cause (accuracy mismatch, model loaded).
• Scope: 1 job (nightly-8-gpu-mi35x-glm51-rocm720) in workflow nightly-test-amd-rocm720.yml. Sister glm51 jobs in the same run are not in scope of this analysis.
• Blocked work: Blocks the GLM-5.1-FP8 (DSA) nightly accuracy signal on MI35x ROCm 7.2 for this run only.
• In-flight fix: ❌ none found — likely transient infra; re-run before opening a PR.

nightly-8-gpu-mi35x-qwen35-rocm720 — failed step(s): Accuracy Test MI35x ROCm 7.2 (8-GPU Qwen 3.5)

Commit Info

• sglang: 43b1fb95f
• aiter: N/A (no [CI-AITER-CHECK] markers; this nightly accuracy job runs from the prebuilt image rocm/sgl-dev:v0.5.13.post1-rocm720-mi35x-20260622, AITER_COMMIT_OVERRIDE empty)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi35x/test_qwen35_eval_mi35x.py	TestQwen35EvalMI35x.test_lm_eval	Exception: Server process exited with code 1 → server raised AssertionError: extra_buffer needs CUDA/MUSA/NPU (FLA).	Log

Failure Summary

The single test test_lm_eval could not start its server: sglang serve --model-path Qwen/Qwen3.5-397B-A17B --tp 8 --attention-backend aiter exits with code 1 during arg validation. The server dies in ServerArgs.__post_init__ → _handle_model_specific_adjustments → _handle_mamba_radix_cache → _validate_mamba_extra_buffer with AssertionError: extra_buffer needs CUDA/MUSA/NPU (FLA). (python/sglang/srt/server_args.py:4366-4368). After 1 retry the test errors out with retry() exceed maximum number of retries.

Regression Status

Known recurring failure (started between 2026-06-15 and 2026-06-20).

Recent history of test_qwen35_eval_mi35x.py (test_lm_eval) in job nightly-8-gpu-mi35x-qwen35-rocm720 (branch main, event schedule, completed runs only):

Date	Run	Job	Test File Status	Failed Function	Error
Jun 22	27975850577	qwen35-rocm720	❌ Failed	test_lm_eval	extra_buffer needs CUDA/MUSA/NPU (FLA)
Jun 21	27912828989	qwen35-rocm720	❌ Failed	test_lm_eval	same assertion (verified in log)
Jun 20	27879239829	qwen35-rocm720	❌ Failed	test_lm_eval	same assertion (verified in log)
Jun 16-19	—	—	⏭️ Cancelled	—	runs cancelled, not comparable
Jun 15	27568866028	qwen35-rocm720	✅ Passed	—	—

First observed failure: 2026-06-20. Last known pass: 2026-06-15 (33719cfb31). Regression window 33719cfb31..ff1fc1fbdf.

Failure Cluster

ServerArgs assertion: mamba extra_buffer radix cache unsupported on ROCm (FLA needs CUDA/MUSA/NPU)

Facts (verified observations)

• Verbatim server crash (tail of log): AssertionError: extra_buffer needs CUDA/MUSA/NPU (FLA). at server_args.py:4368, reached via _handle_mamba_radix_cache (:4395) → _validate_mamba_extra_buffer.
• The same assertion in _validate_mamba_extra_buffer appears in the Jun 20 and Jun 21 failing logs (2 × each, verified by grep) — identical failure, not a different test.
• Qwen3.5-397B-A17B is a hybrid (Gated Delta Network / mamba-style linear attention) model; its arch Qwen3_5MoeForConditionalGeneration is in _support_mamba_cache_extra_buffer (server_args.py:4337-4339), so with default radix cache + overlap schedule the auto strategy selects extra_buffer (server_args.py:4383-4389).
• On ROCm, is_cuda() or is_musa() or is_npu() is False, so _validate_mamba_extra_buffer (:4366-4368) asserts. [FACT]
• The assertion code and the test file both predate the local shallow-clone boundary (b225f48fc, 2026-06-20); none of the 20 commits in 33719cfb31..ff1fc1fbdf touch the mamba radix-cache path. The exact landing commit is not locally bisectable.

Hypothesised Causes (with confidence)

• [MEDIUM] Structural gap: the auto mamba-radix-cache strategy routes Qwen3.5 to extra_buffer (FLA), which _validate_mamba_extra_buffer only permits on CUDA/MUSA/NPU — so the path is unreachable on ROCm by design.
  • Supporting: code chain _handle_mamba_radix_cache→extra_buffer→assert is directly reachable from the failing launch command (radix cache on, overlap on, arch in support list); assertion message matches the crash exactly.
  • Disconfirming: this alone doesn't explain why Jun 15 passed — the assertion code was already present then, so a separate change in the Jun 15→20 window must have started routing this model/job into the extra_buffer branch (e.g. arch added to the support list, default strategy/overlap change, the test being enabled, or the online HF model config changing).
  • How to verify: bisect 33719cfb31..ff1fc1fbdf running this test on MI35x; check whether Qwen3_5MoeForConditionalGeneration was added to _support_mamba_cache_extra_buffer or the test was newly registered in that window.
• [LOW] The Jun 15 "pass" may have exercised a different code path (e.g. model arch not yet recognized as hybrid, or different default strategy) rather than passing the same validation.
  • Supporting: regression timing aligns with active mamba-radix work (many mamba/radix feature branches visible in the fetched ref list).
  • Disconfirming: cannot confirm locally — relevant commits are outside the shallow clone.

In-flight Fix Check

• In-flight fix: ✅ #27141 "[ROCm] Support radix-cache with mamba-extra-buffer for Qwen3.5" (open, non-draft, since 2026-06-03, last updated 2026-06-22) — directly targets this exact gap. Chase reviewers; do NOT open a duplicate. (Search: repo:sgl-project/sglang is:pr is:open mamba extra_buffer ROCm.)

Suggested Triage Steps

• Verify #27141 resolves this assertion and merges cleanly, then re-run nightly-8-gpu-mi35x-qwen35-rocm720.
• As an interim CI-green workaround, the test could pass a non-extra_buffer path on ROCm (e.g. --disable-radix-cache, or force mamba_radix_cache_strategy=no_buffer/--disable-overlap-schedule) — confirm acceptable for the accuracy run before applying.
• To pin the exact culprit, bisect 33719cfb31..ff1fc1fbdf on MI35x, watching for the commit that first routes Qwen3_5MoeForConditionalGeneration into the extra_buffer branch.

Status

• Persistence: Failing every completed scheduled run since 2026-06-20 (Jun 20, 21, 22); last green 2026-06-15. Jun 16-19 runs were cancelled.
• Scope: test_lm_eval in nightly-8-gpu-mi35x-qwen35-rocm720. The sister job nightly-8-gpu-qwen35-rocm720 is also red in the Jun 20/21 runs (likely the same assertion — worth confirming).
• Blocked work: Qwen3.5 GSM8K accuracy signal on AMD MI35x ROCm 7.2.
• In-flight fix: ✅ #27141 open since 2026-06-03 — awaiting merge.

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-06-23 20:59 UTC)

[amd-bot]

nightly-test-amd-rocm720.yml — Per-Job Analysis (continued 3/3)

nightly-8-gpu-mi35x-kimi-k26-rocm720 — failed step(s): Accuracy Test MI35x ROCm 7.2 (8-GPU Kimi-K2.6)

Commit Info

• sglang: 43b1fb95f
• aiter: 7d604afe5 (source: dev image preserved — [CI-AITER-CHECK] Dev/patched version detected: v0.1.14rc1.dev240+g7d604afe5 → skipping rebuild; AITER_COMMIT_OVERRIDE empty)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi35x/test_kimi_k26_eval_mi35x.py	TestKimiK26EvalMI35x.test_kimi_k26_gsm8k_accuracy	TimeoutError: Server failed to start within the timeout period → harness TIMEOUT after 7200s, exit 255	Log

Failure Summary

The test_kimi_k26_gsm8k_accuracy test never ran its accuracy assertions because the server failed to start within the budget. The Kimi-K2.6 weights were still downloading from HuggingFace via the hf-xet backend (xet_get → _download_to_tmp_and_move) for the entire ~2 hours; the test harness aborted the job at 7200s with exit code 255. The server was launched twice (19:50:39 and, after the first popen_launch_server timeout + 1 retry, again at 21:20:45) and both attempts stalled in the same weight download.

Regression Status

Known recurring failure (since 2026-06-20). Same xet_get download-stall signature in the two prior completed scheduled runs; the test last passed 2026-06-15.

Recent history of test_kimi_k26_eval_mi35x.py (test_kimi_k26_gsm8k_accuracy) in job nightly-8-gpu-mi35x-kimi-k26-rocm720 (branch=main, event=schedule, completed runs only):

Date	Run	Job	Test File Status	Failed Function	Error
Jun 22	27975850577	mi35x-kimi-k26	❌ Failed	test_kimi_k26_gsm8k_accuracy	hf-xet download stall → 7200s timeout
Jun 21	27912828989	mi35x-kimi-k26	❌ Failed	test_kimi_k26_gsm8k_accuracy	hf-xet download stall → server start timeout
Jun 20	27879239829	mi35x-kimi-k26	❌ Failed	test_kimi_k26_gsm8k_accuracy	hf-xet download stall → server start timeout
Jun 15	27568866028	mi35x-kimi-k26	✅ Passed	—	—

(Jun 16–19 scheduled runs were cancelled, so not comparable. First observed failure: 2026-06-20; last known pass: 2026-06-15.)

Failure Cluster

Kimi-K2.6 weight download stall via huggingface_hub hf-xet → server start timeout

Facts (verified observations)

• The failing step is step 6 ("Accuracy Test MI35x ROCm 7.2 (8-GPU Kimi-K2.6)"); harness printed ✗ TIMEOUT: .../test_kimi_k26_eval_mi35x.py after 7200 seconds then Process completed with exit code 255 (log lines 15140–15157).
• Every py-spy scheduler dump from 19:55 through 21:45 shows the active thread in xet_get (huggingface_hub/file_download.py:566) ← _download_to_tmp_and_move (file_download.py:1861) ← snapshot_download ← download_weights_from_hf (weight_utils.py:593) ← stream_language_weights (kimi_k25.py:780). One TP rank actively downloads; the other 7 ranks block in filelock _poll_until_acquired (weight_utils.py:554) waiting for it.
• First popen_launch_server attempt timed out at 21:20:34 (TimeoutError: Server failed to start within the timeout period, test_utils.py:1028), retry() fired (max 1 retry), second launch at 21:20:45 stalled identically.
• hf-xet version 1.5.1 in the image (log line 2099). No download progress percentage/throughput ever printed for the model weights.
• Same xet_get stall present in Jun 20 (82503808359) and Jun 21 (82592808002) logs; Jun 21 ended as ERROR ... Fail. Time elapsed: 2486.06s (server-start timeout after retries) rather than the full 7200s harness timeout, but the underlying download stall is identical.
• This is a nightly AMD workflow (nightly-test-amd-rocm720.yml), not AITER Scout — no Baseline A/B check applies.

Hypothesised Causes (with confidence)

• [HIGH] Environment/infrastructure: the Kimi-K2.6 checkpoint (very large MoE) is not present in the runner cache and the hf-xet download throughput is too low (or stalled) to complete within the job's time budget.
  • Supporting: every scheduler dump for ~2h sits in xet_get/_download_to_tmp_and_move; download never completes; identical signature across 3 consecutive failing runs; the same test passed Jun 15 (cached weights or faster fetch).
  • Disconfirming: cannot confirm from the log whether the download is fully stalled vs. merely slow (no byte-progress lines emitted); a server-side HF/xet outage or a re-uploaded moonshotai/Kimi-K2.6 revision forcing a full re-download cannot be ruled out from this log alone.
  • How to verify: on the MI35x runner, time a standalone huggingface-cli download moonshotai/Kimi-K2.6 and measure xet throughput; check whether the model cache volume is persisted across nightly runs; check HF for a recent revision bump on moonshotai/Kimi-K2.6.
• [LOW] sglang code regression in the Jun 15→Jun 20 window affecting the weight-download/loader path.
  • Supporting: the regression window aligns with a code-merge window.
  • Disconfirming: the hang is entirely inside huggingface_hub's xet_get (third-party download), not sglang loader logic; the symptom is a slow network fetch, not a code error/traceback in sglang. No green run exists after Jun 15 to narrow the window cheaply.
  • How to verify: only pursue if the infra hypothesis is disproven; then bisect 33719cfb31..ff1fc1fbdf for changes touching model_loader/weight_utils.py, kimi_k25.py, or download/cache config.

In-flight Fix Check

• In-flight fix: ❌ none found — searches is:pr is:open kimi download and is:pr is:open xet returned no matching fix (only unrelated PRs: #21023, #25071, #27681, #26260 "DO NOT MERGE"). API: search/issues?q=repo:sgl-project/sglang+is:pr+is:open+xet.

Suggested Triage Steps

• Measure standalone hf-xet download throughput for moonshotai/Kimi-K2.6 on the MI35x ROCm 7.2 runner; if it's far below what's needed to finish in <7200s, the fix is infra-side (pre-cache/persist the checkpoint on the runner, or pre-warm it before the test).
• Verify whether the runner's model cache volume is persisted across nightly runs — the Jun 15 pass vs. Jun 20+ failures suggests a cache was lost/evicted around Jun 19–20 (the cancelled-run window).
• Check HuggingFace for a recent revision/re-upload of moonshotai/Kimi-K2.6 that would force a fresh full download.
• If infra is ruled out, bisect 33719cfb31..ff1fc1fbdf for weight-loader/download changes, and consider whether the test's server-start timeout should account for a cold-cache download of a model this size.

Status

• Persistence: Failing every completed scheduled run since 2026-06-20 (3 consecutive runs); last pass 2026-06-15.
• Scope: nightly-8-gpu-mi35x-kimi-k26-rocm720 in nightly-test-amd-rocm720.yml. The sister job nightly-8-gpu-kimi-k26-rocm720 also failed on Jun 20/21 — likely the same Kimi-K2.6 download cluster (not separately confirmed here).
• Blocked work: Kimi-K2.6 GSM8K accuracy signal on AMD MI35x ROCm 7.2 nightly.
• In-flight fix: ❌ none found in open PRs — needs infra investigation (runner cache / xet throughput) or a new PR.

nightly-8-gpu-mi35x-qwen3-235b-mxfp4-rocm720 — failed step(s): Accuracy Test + Performance Test MI35x ROCm 7.2 (8-GPU Qwen3-235B-MXFP4)

Commit Info

• sglang: 43b1fb95f
• aiter: 7d604afe5 (v0.1.14rc1.dev240+g7d604afe5; source: dev image preserved — matches Dockerfile default for gfx950, no override)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/test_qwen3_instruct_mxfp4.py	TestQwen3Instruct2507MXFP4.setUpClass	RuntimeError: Rank 0 scheduler died during initialization (exit code: -6) — server abort during decode CUDA graph capture (HIP error: operation not permitted on an event last recorded in a capturing stream, hipErrorCapturedEvent)	Log

Failure Summary

The server for amd/Qwen3-235B-A22B-Instruct-2507-mxfp4 (TP4/EP2, --attention-backend aiter) aborted during startup, so setUpClass could not launch it and 0 tests ran (Ran 0 tests in 516.256s, FAILED (errors=1)). The abort occurs while capturing the decode CUDA graph: the NCCL ProcessGroup watchdog thread terminated with a HIP error because it queried a completion event that was last recorded on a stream currently in graph-capture mode. The capture path on the main thread is in aiter's custom all-reduce (custom_all_reduce.py:505 capture → flush_graph_buffers:305 → _gather_ipc_meta:333).

Regression Status

Known recurring failure (no green run found in queryable window).

Recent history of test/registered/amd/test_qwen3_instruct_mxfp4.py (TestQwen3Instruct2507MXFP4.setUpClass) in job nightly-8-gpu-mi35x-qwen3-235b-mxfp4-rocm720 (branch main, event schedule, completed runs only):

Date	Run	Job	Test File Status	Failed Function	Error
Jun 22	27975850577	job	❌ Failed	setUpClass	HIP capturing-stream event error → exit -6
Jun 21	27912828989	job	❌ Failed	setUpClass	same
Jun 20	27879239829	job	❌ Failed	setUpClass	same
Jun 15	27568866028	job	❌ Failed	setUpClass	same
Jun 14	27507292665	job	❌ Failed	setUpClass	same

No passing run for this test file was found in the queryable window (every completed schedule run of this job from Jun 14 → Jun 22 shows the identical capturing-stream abort; the parent workflow has also been red on every schedule run back to at least May 24). Treat as Never-passed in window; lookback range Jun 14–22 at job level, May 24–Jun 22 at workflow level.

Failure Cluster

HIP "operation not permitted on event in capturing stream" abort during decode CUDA graph capture (aiter custom all-reduce)

Facts (verified observations, no interpretation)

• Watchdog thread error (verbatim): Process group watchdog thread terminated with exception: HIP error: operation not permitted on an event last recorded in a capturing stream → hipErrorCapturedEvent; main thread then Fatal Python error: Aborted; scheduler reported exit code: -6.
• Main-thread capture stack (verbatim, top frames): aiter/dist/device_communicators/custom_all_reduce.py:333 _gather_ipc_meta ← :305 flush_graph_buffers ← :505 capture ← parallel_state.py:538 graph_capture ← parallel_state.py:1665 graph_capture ← decode_cuda_graph_runner.py:687 capture ← :353 __init__ ← model_runner.py:2583 init_decode_cuda_graph ← :913 init_cuda_graphs ← tp_worker.py:351 ← scheduler.py:855 init_all_cuda_graphs.
• Crash occurs at the start of decode-graph capture: log shows Capture target decode CUDA graph begin … bs=[1..512] then Capturing batches (bs=512 avail_mem=25.03 GB): 0% immediately before the abort. Available memory at capture begin ~25.9 GB (not an obvious OOM; exit code is -6/SIGABRT, not -9/SIGKILL).
• Config (verbatim from server_args): tp_size=4, ep_size=2, attention_backend='aiter', enable_aiter_allreduce_fusion=False, disable_custom_all_reduce=False, piecewise/enforce-piecewise-cuda-graph not set.
• Reproducibility: identical test file + function + symptom + stack in all 5 checked completed runs across two ranks (rank0/rank1 both show the watchdog termination).
• aiter version unchanged across these runs (dev image v0.1.14rc1.dev240+g7d604afe5, gfx950 Dockerfile default).

Hypothesised Causes (with confidence, with disconfirming evidence)

• [MEDIUM] A collective/event operation inside aiter's custom_all_reduce.capture() (flush_graph_buffers → _gather_ipc_meta) is executed while the stream is in HIP graph-capture mode; the ProcessGroupNCCL watchdog then calls WorkNCCL::isCompleted() → finishedGPUExecutionInternal() which queries an event recorded on the capturing stream, which HIP forbids (hipErrorCapturedEvent), aborting the process.
  • Supporting: the fatal stack terminates exactly in _gather_ipc_meta under the ca_comm.capture() context (parallel_state.py:538); watchdog frames finishedGPUExecutionInternal/isCompleted are the error site; symptom is capture-specific.
  • Disconfirming: stack-frame proximity is not proof of which op recorded the offending event; the watchdog error may be triggered by an unrelated in-flight NCCL work item whose event merely happened to land on the capturing stream.
  • How to verify: re-run the same command with --disable-custom-all-reduce (bypasses ca_comm.capture()); if capture succeeds, the aiter custom all-reduce capture path is confirmed. Alternatively set TORCH_NCCL_ENABLE_MONITORING=0/disable the PG watchdog to isolate whether it is the watchdog query vs. the capture op itself.
• [LOW] ROCm-7.2 / gfx950 (MI35x) aiter dev build incompatibility with custom-all-reduce graph capture.
  • Supporting: failure is AMD-ROCm-7.2-specific and uses an aiter dev image; symptom is HIP-only.
  • Disconfirming: aiter version is constant across all failing runs, so it does not explain an onset; no green baseline exists to contrast against.
  • How to verify: compare against the non-rocm720 nightly (nightly-test-amd.yml) run of the same model, if present, to see whether ROCm 7.2 specifically is implicated.

No hypothesis reaches HIGH; root cause requires the A/B reproduction above. Note this is a long-standing red test, not a fresh regression — there is no pass_sha..fail_sha window to bisect within the queryable history.

In-flight Fix Check

• In-flight fix: ⚠️ partial — #27650 "[AMD] Fix PCG capture fail with aiter fused all-reduce+RMSNorm" (open, updated 2026-06-22) addresses an aiter all-reduce + CUDA-graph-capture crash, but for a different configuration (--enable-aiter-allreduce-fusion + --enforce-piecewise-cuda-graph on Kimi-K2.5). This job sets neither flag and crashes in custom_all_reduce.capture() (not the fused all-reduce+RMSNorm path), so #27650 likely does not cover this case. Searched custom_all_reduce+capture, Qwen3-235B+mxfp4, capturing+stream+aiter.
• In-flight fix: ❌ none found directly matching the custom-all-reduce capture abort — needs new investigation.

Suggested Triage Steps

• Reproduce with --disable-custom-all-reduce added to the launch command (sglang serve --model-path amd/Qwen3-235B-A22B-Instruct-2507-mxfp4 --tp 4 --ep 2 --trust-remote-code --attention-backend aiter) on an MI35x/ROCm-7.2 runner; if capture succeeds, file/track the fix against aiter custom_all_reduce.py (flush_graph_buffers/_gather_ipc_meta must not run a collective or record/query a cross-stream event during graph capture).
• If the custom all-reduce path is confirmed, confirm with the aiter maintainers whether #27650 can be extended to make the plain (non-fusion) ca_comm.capture() capture-safe, or whether a separate aiter fix is needed.
• As an interim signal-restoring step, consider disabling custom all-reduce for this nightly job config until the aiter capture path is fixed (do not mask without a tracked fix).

Status

• Persistence: Failing on every completed schedule run of this job checked from Jun 14 → Jun 22 2026 (no green run in window); identical symptom each time.
• Scope: nightly-8-gpu-mi35x-qwen3-235b-mxfp4-rocm720 in nightly-test-amd-rocm720.yml; blocks all Qwen3-235B-MXFP4 accuracy/perf signal on MI35x ROCm 7.2. The capture abort pattern may affect other aiter + custom-all-reduce graph-capture configs on this hardware.
• In-flight fix: ⚠️ #27650 related but likely does not cover this config; no directly-matching open PR found.

nightly-8-gpu-kimi-k26-rocm720 — failed step(s): Accuracy Test ROCm 7.2 (8-GPU Kimi-K2.6)

Commit Info

• sglang: 43b1fb95f
• aiter: 7d604afe5 (v0.1.14rc1.dev240+g7d604afe5) — source: dev image preserved (CI image had a dev/patched aiter build; rebuild skipped per [CI-AITER-CHECK] markers, log line 3553)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi30x/test_kimi_k26_eval_amd.py	TestKimiK26EvalAMD.setUpClass	Exception: Server process exited with code -9 (server crashed during weight load with RuntimeError: start (64) + length (64) exceeds dimension size (64))	Log

Failure Summary

The Kimi-K2.6 server failed to start: during model weight loading, the MoE w2/down-proj weight loader raised RuntimeError: start (64) + length (64) exceeds dimension size (64) at fused_moe_triton/layer.py _load_w2 (loaded_weight.narrow(shard_dim, shard_size*tp_rank, shard_size)). All 8 TP schedulers hit watchdog timeouts and the parent process was SIGKILLed (exit -9), so setUpClass aborted before any eval ran (0 tests). This is a server-launch/weight-loading failure, not an accuracy regression.

Regression Status

Known recurring failure — failing since the Jun 20 nightly. Last green was Jun 15.

Recent history of test_kimi_k26_eval_amd.py (TestKimiK26EvalAMD.setUpClass) in job nightly-8-gpu-kimi-k26-rocm720 (branch main, event schedule, completed runs only):

Date	Run	Test File Status	Failed Function	Error
Jun 22	27975850577	❌ Failed	setUpClass	narrow … exceeds dimension size (64) → exit -9
Jun 21	27912828989	❌ Failed	setUpClass	same narrow error → exit -9
Jun 20	27879239829	❌ Failed	setUpClass	same narrow error → exit -9
Jun 16–19	(cancelled)	—	—	runs cancelled, not comparable
Jun 15	27568866028	✅ Passed	—	—

First observed failure: 2026-06-20. Last known pass: 2026-06-15. Regression window: 33719cfb..ff1fc1fb (20 commits). The sister nightly-8-gpu-mi35x-kimi-k26-rocm720 job fails identically in the same runs, so the failure is not runner-specific.

Failure Cluster

MoE w2/down-proj weight-load narrow out-of-range during Kimi-K2.6 server startup

Facts (verified observations)

• Stack trace (log lines 50575–50597): kimi_k25.py:801 load_weights → deepseek_v2.py:2778 → deepseek_weight_loader.py:373 → fused_moe_triton/layer.py:566 _load_w2 → loaded_weight.narrow(...) → RuntimeError: start (64) + length (64) exceeds dimension size (64).
• The narrowed dimension is size 64; the loader computes shard_size=64, tp_rank=1, so start=64, length=64 overruns a length-64 tensor. This is the classic signature of a checkpoint whose w2 is already per-rank/presharded while use_presharded_weights=False (the code at layer.py:565-570 only narrows when not self.use_presharded_weights).
• Crash sequence: exception raised on TP1 → Received sigquit from a child process (50599) → kill_process_tree → all TP schedulers watchdog-timeout (300 s) → server exits -9 → popen_launch_server raises in setUpClass.
• Same narrow error verbatim in Jun 20 and Jun 21 runs (logs for jobs 82503808418, 82592807982).
• The MoE narrow code path in fused_moe_triton/layer.py was not modified in the regression window (git log 33719cfb..ff1fc1fb -- …/layer.py returns only an unrelated NPU-warmup commit).

Hypothesised Causes (with confidence)

• [LOW] Kimi-K2.6 HF checkpoint format changed to per-rank w2 (compressed-tensors), which the loader doesn't recognize as presharded.
  • Supporting: open in-flight PR #28905 (created 2026-06-22) is titled "Fix Kimi K2.6 compressed-tensors MoE w2 scale loading" and explicitly states "Avoid allocating/loading full TP-width w2_weight_scale for checkpoints whose w2 scales are already per-rank, such as Kimi-K2.6" — the same per-rank-w2 symptom as this narrow overrun. The narrow code itself is unchanged in-window, pointing to an external (checkpoint) trigger consistent with a Jun 15→20 break.
  • Disconfirming: #28905 addresses the w2 scale path (compressed_tensors_wNa16_moe.py), whereas this stack trace is the w2 weight narrow in fused_moe_triton/layer.py; the two paths differ, so #28905 may not by itself fix this exact line. No direct confirmation the HF checkpoint changed on those dates.
  • How to verify: diff the moonshotai/Kimi-K2.6 HF revision/config.json+quantization_config used Jun 15 vs Jun 20; inspect a w2 tensor's on-disk shape vs intermediate_size/TP.
• [LOW] A sglang commit in the window changed presharded-weight detection / quantization config parsing for AMD MoE. Candidate: 111537366 (#28244, "[AMD] Fix garbled unquantized Qwen3-30B-A3B … pre-shuffled weights").
  • Supporting: only MoE/AMD-touching commit in the window; deals with aiter CK fused-MoE pre-shuffled weights and 128-alignment.
  • Disconfirming: it modifies unquant.py process_weights_after_loading/runner selection (runtime), not the load-time narrow path that crashes here; Kimi-K2.6 is quantized (compressed-tensors), so the Unquantized method likely isn't even on this path.
  • How to verify: git revert 111537366 on a branch, relaunch sglang serve --model-path moonshotai/Kimi-K2.6 --tp 8 on MI30x ROCm 7.2; if it still crashes, this commit is exonerated.
• No hypothesis reaches MEDIUM confidence; the narrow code is unchanged in-window, so a git bisect of 33719cfb..ff1fc1fb (or a checkpoint-revision check) is needed to attribute root cause.

In-flight Fix Check

• In-flight fix: ✅ #28905 (open since 2026-06-22) — "Fix Kimi K2.6 compressed-tensors MoE w2 scale loading"; targets the same per-rank-w2 Kimi-K2.6 symptom but on the scale path. Verify it also resolves the w2 weight narrow in fused_moe_triton/layer.py; if not, this exact crash needs an additional fix. Its own PR CI was still red at last update — chase, do not duplicate.
• Search run: GET /search/issues?q=repo:sgl-project/sglang+is:pr+is:open+kimi+weight+load and +Kimi+K2.6.

Suggested Triage Steps

• Confirm whether #28905 fixes the w2 weight narrow (line fused_moe_triton/layer.py:566) or only the scale path; if only scale, file/extend a fix for the weight load.
• Check if the moonshotai/Kimi-K2.6 HF checkpoint revision (config/quantization_config, w2 tensor shapes) changed between 2026-06-15 and 2026-06-20 — the unchanged narrow code + per-rank-w2 framing in #28905 points to an external checkpoint trigger.
• If checkpoint-driven, ensure the loader sets use_presharded_weights=True (or skips the shard_size*tp_rank narrow) for already-per-rank Kimi-K2.6 w2.
• Otherwise git bisect 33719cfb..ff1fc1fb, relaunching the Kimi-K2.6 TP8 server per commit on MI30x ROCm 7.2.

Status

• Persistence: Failing every completed scheduled run since 2026-06-20 (3 consecutive: Jun 20, 21, 22); last green 2026-06-15.
• Scope: Both nightly-8-gpu-kimi-k26-rocm720 and sister nightly-8-gpu-mi35x-kimi-k26-rocm720 jobs in nightly-test-amd-rocm720.yml; blocks all Kimi-K2.6 accuracy signal on AMD ROCm 7.2 (server cannot start).
• In-flight fix: ✅ #28905 open (scale path; verify it covers the weight-narrow crash).

nightly-8-gpu-mi35x-deepseek-v4-pro-rocm720 — failed step(s): Accuracy + Performance Test MI35x ROCm 7.2 (8-GPU DeepSeek-V4-Pro FP8 + FP4, unified_kv_triton)

Commit Info

• sglang: 43b1fb95
• aiter: 7d604afe5 (v0.1.14rc1.dev240+g7d604afe5, source: dev/patched image preserved — rebuild skipped)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/test_deepseek_v4_pro_fp8.py	TestDeepseekV4ProFp8.setUpClass	TimeoutError: Server failed to start within the timeout period	Log

Failure Summary

The job ran two backend variants. Step 6 (SGLANG_HACK_FLASHMLA_BACKEND=unified_kv_triton) reported 2/3 passed: test_deepseek_v4_pro_fp4.py and test_deepseek_v4_pro_fp4_cp.py passed, but test_deepseek_v4_pro_fp8.py failed in setUpClass with Ran 0 tests ... FAILED (errors=1) — the DeepSeek-V4-Pro-FP8 server never became healthy and popen_launch_server timed out. Step 7 (triton variant) ran the same three files and reported 3/3 passed, including the same fp8 model+config. The job's overall failure is driven solely by the step-6 fp8 server-start timeout.

Regression Status

New/isolated failure (job passed the previous two completed scheduled runs).

Recent history of test/registered/amd/test_deepseek_v4_pro_fp8.py in job nightly-8-gpu-mi35x-deepseek-v4-pro-rocm720 (branch main, event schedule, completed runs only):

Date	Run	Job	Test File Status	Failed Function	Error
Jun 23	27975850577	…deepseek-v4-pro-rocm720	❌ Failed (step 6 only)	setUpClass	TimeoutError: Server failed to start
Jun 21	27912828989	…deepseek-v4-pro-rocm720	✅ Passed	—	—
Jun 20	27879239829	…deepseek-v4-pro-rocm720	✅ Passed	—	—

(Jun 16–19 scheduled runs were cancelled, so excluded. First observed failure for this test file: Jun 23 2026; last known passing: Jun 21 2026.)

Failure Cluster

HF Xet weight-download hang during server startup (server failed to start within timeout).

Facts (verified observations, no interpretation)

• setUpClass traceback (verbatim, log line 67849–67858): ERROR: setUpClass (__main__.TestDeepseekV4ProFp8) → popen_launch_server (test_utils.py:1028) → raise TimeoutError(error_msg) → TimeoutError: Server failed to start within the timeout period. Preceded by Ran 0 tests in 5409.399s / FAILED (errors=1).
• The server command was sglang serve --model-path sgl-project/DeepSeek-V4-Pro-FP8 --tp 8 --attention-backend dsv4 ... --device cuda (log line 66918).
• All 8 TP scheduler processes were stuck acquiring a filelock during weight download. Repeated py-spy dumps from 02:40 through 03:45 show the identical stack on the schedulers:
  _poll_until_acquired (filelock/_api.py:552) → acquire (_api.py:506) → download_weights_from_hf (model_loader/weight_utils.py:554).
• The lock-holding worker was continuously stuck in the HF Xet download for ~65 minutes (every 5-min dump 02:40→03:30 shows the same frames): xet_get (huggingface_hub/file_download.py:566) → _download_to_tmp_and_move (file_download.py:1861) → _hf_hub_download_to_cache_dir → snapshot_download → ci_download_with_validation_and_retry (ci_weight_validation.py:1955) → download_weights_from_hf (weight_utils.py:593). The main process sat in launch_server (http_server.py:2485) → PyThread_acquire_lock_timed.
• Soft scheduler-watchdog timeouts (watchdog_timeout=300, soft=True) fired on every TP rank repeatedly from 03:30 onward; being soft, they only logged and did not kill the hung process.
• The same fp8 model + same dsv4 config passed in step 7 (triton variant, 3/3 passed, log line 155526), i.e. weights downloaded successfully on the next attempt.
• fp8 file elapsed in step 6 = 5415s; step exited with code 255 → Process completed with exit code 255 (log line 67860–67900).

Hypothesised Causes (with confidence, with disconfirming evidence)

• [HIGH] Transient HF Xet weight-download stall is the proximate cause of the server-start timeout.
  • Supporting: the lock-holding scheduler is parked in xet_get / _download_to_tmp_and_move for ~65 consecutive minutes with no progress; the other 7 ranks merely poll the shared filelock; the main process blocks in PyThread_acquire_lock_timed. No model-forward, CUDA-graph, or attention-backend frame ever appears — the hang is entirely in download, before any compute.
  • Disconfirming: (none found) — the step-7 pass on the identical fp8 model+config confirms the model/config themselves are sound, isolating the fault to the one stalled download.
  • How to verify: re-run the job; if green, confirms a flaky download. Optionally pre-warm/cache sgl-project/DeepSeek-V4-Pro-FP8 or disable Xet (HF_HUB_DISABLE_XET=1) for a comparison run.
• [LOW] Code regression in the unified_kv_triton MLA backend or sglang ahead of 43b1fb95.
  • Supporting: only the unified_kv_triton variant failed, and the job passed Jun 20–21.
  • Disconfirming: the failure occurs during HF weight download — strictly before any attention backend is exercised — so the backend env var is not on the failing code path; the variant difference is incidental (step 6 simply attempted the download first). Strong evidence against a code regression.
  • How to verify: not actionable; download stack trace already rules out the backend.
• [LOW] aiter dev build v0.1.14rc1.dev240+g7d604afe5.
  • Supporting: AMD-only job.
  • Disconfirming: aiter is not on the download code path; same aiter build also ran the passing step 7 and the fp4/fp4_cp tests.

In-flight Fix Check

• In-flight fix: ❌ none found — repo:sgl-project/sglang is:pr is:open xet download returned 0 results; weight download timeout returned only unrelated keyword matches. No open PR targets the HF Xet download hang.

Suggested Triage Steps

• Re-run the job; a green re-run confirms a flaky HF Xet download stall (no code change needed).
• If the stall recurs, try a comparison run with HF_HUB_DISABLE_XET=1 (or pre-cache sgl-project/DeepSeek-V4-Pro-FP8 weights on the runner) to A/B test whether Xet is the culprit.
• Consider whether popen_launch_server's startup timeout and/or the CI download retry (ci_download_with_validation_and_retry) should detect a stalled (zero-progress) Xet download and retry, so an 8-GPU job doesn't burn ~90 min on one hung download.
• No sglang-code bisect is warranted: the stack trace places the hang entirely in HF weight download, before any sglang/aiter compute path.

Status (factual, no priority assignment)

• Persistence: First and only failure of this test file in the queryable window — passed Jun 20 and Jun 21; failed Jun 23. Not persistent.
• Scope: One job, one workflow (nightly-test-amd-rocm720.yml), one of two backend variants; sister model variants (fp4, fp4_cp) and the triton fp8 run all passed in the same job.
• Blocked work: Blocks the nightly DeepSeek-V4-Pro FP8 unified_kv_triton signal on MI35x ROCm 7.2 for this run only.
• In-flight fix: ❌ none found — needs a re-run to confirm flakiness; new PR only if the download-hang recurs.

nightly-accuracy-2-gpu-vlm-rocm720 — failed step(s): Nightly Accuracy Test ROCm 7.2 (2-GPU VLM MMMU)

Commit Info

• sglang: ed26a109
• aiter: 7d604afe5 (v0.1.14rc1.dev240+g7d604afe5; source: dev/patched image preserved — [CI-AITER-CHECK] reported "Dev/patched version detected → skipping rebuild", matches Dockerfile default)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi30x/test_vlms_mmmu_eval_amd.py	test_mmmu_vlm_models (assert in _check_results, line 380)	AssertionError: The following models failed: (5 VLMs below MMMU threshold)	Log

The single test function evaluates 11 VLMs on a 100-sample MMMU subset; it asserts at the end if any model is below its per-model threshold. 5 models failed:

Model	Score	Threshold
Qwen/Qwen2-VL-7B-Instruct	0.2799	0.30
Qwen/Qwen2.5-VL-7B-Instruct	0.2802	0.33
Qwen/Qwen3-VL-30B-A3B-Instruct	0.2601	0.29
openbmb/MiniCPM-o-2_6	0.2802	0.32
deepseek-ai/deepseek-vl2-small	0.2802	0.31

Failure Summary

The nightly-accuracy-2-gpu-vlm-rocm720 job failed because test_mmmu_vlm_models raised an AssertionError: 5 of 11 VLMs scored below their MMMU accuracy thresholds. Notably, every model in the suite (including the 6 "passing" ones) scored in the 0.26–0.35 band — at or just above the ~0.25 random-guess baseline for a 4-choice benchmark — even though models like Qwen2.5-VL-7B normally score ~0.5+ on MMMU.

Regression Status

Known recurring failure (NOT a new regression on this commit). The accuracy-VLM job has failed in every completed scheduled run in the queryable window, and the same core set of models fails each time.

Recent history of test_vlms_mmmu_eval_amd.py (test_mmmu_vlm_models) in job nightly-accuracy-2-gpu-vlm-rocm720 (branch main, event schedule, completed runs only):

Date	Run	Test File Status	Failing models (score)
Jun 23	run	❌ Failed (5/11)	Qwen2-VL(0.28), Qwen2.5-VL(0.28), Qwen3-VL-30B(0.26), MiniCPM-o(0.28), deepseek-vl2(0.28)
Jun 22	run	❌ Failed (4/11)	Qwen2.5-VL(0.28), Qwen3-VL-30B(0.26), MiniCPM-o(0.28), deepseek-vl2(0.28)
Jun 21	run	❌ Failed (5/11)	Qwen2-VL(0.29), Qwen2.5-VL(0.29), Qwen3-VL-30B(0.26), MiniCPM-o(0.28), deepseek-vl2(0.28)
Jun 20	run	❌ Failed (4/11)	Qwen2.5-VL(0.28), Qwen3-VL-30B(0.27), MiniCPM-o(0.28), deepseek-vl2(0.28)
Jun 15	run	❌ Failed	—
Jun 14	run	❌ Failed	—

(Jun 16–19 scheduled runs were cancelled and excluded. No green run for this test file exists in the queryable window — Jun 12–23. The current run 28046795462 is overall still queued for sibling jobs, but this job completed with conclusion=failure.)

Per-model persistence:

• Never-passed in window (fail every completed run): Qwen2.5-VL-7B-Instruct, Qwen3-VL-30B-A3B-Instruct, MiniCPM-o-2_6, deepseek-vl2-small.
• Borderline-flaky (oscillates at threshold 0.30): Qwen2-VL-7B-Instruct — PASS 0.30 (Jun 22, Jun 20) / FAIL 0.28–0.29 (Jun 23, Jun 21).

Failure Cluster

VLM MMMU accuracy near chance level on ROCm 7.2 / MI30x (all models clustered ~0.25–0.35).

Facts (verified observations)

• AssertionError: The following models failed: raised at test_vlms_mmmu_eval_amd.py:380 (_check_results), called from test_mmmu_vlm_models at line 314 (verbatim from log).
• Final stats: Passed: 6 / Failed: 5, Ran 1 test in 2923.591s, job exit code 255.
• Every one of the 11 models scored 0.26–0.35; the 4-choice MMMU random baseline is ~0.25, and the harness grades only num=100 samples per model (test header: "conservative thresholds on 100 MMMU samples").
• The identical 4-model failure core (Qwen2.5-VL, Qwen3-VL-30B, MiniCPM-o, deepseek-vl2) recurs in every completed run Jun 20–23 with near-identical scores (≈0.26–0.29), i.e. stable, not random per-run noise.
• aiter version is unchanged (dev image v0.1.14rc1.dev240+g7d604afe5 preserved) — no aiter delta introduced by this run.

Hypothesised Causes (with confidence)

• [MEDIUM] Systemic VLM accuracy degradation (or never-correct VLM pipeline) on ROCm 7.2 / MI30x — all models, including the nominally "passing" ones, score near chance, far below their known MMMU capability (Qwen2.5-VL-7B ≈0.50 upstream vs 0.28 here).
  • Supporting: whole-suite clustering at chance; stable per-model scores across 4+ runs (not noise); affects multiple independent model families (Qwen, MiniCPM, DeepSeek).
  • Disconfirming: no green baseline for this harness exists in the queryable window, so I cannot prove scores were ever higher here — the suite may have been borderline since introduction; some models (Kimi-VL 0.35, MiniCPM-v 0.32) sit a bit above chance, suggesting partial functionality rather than total breakage.
  • How to verify: compare against the NVIDIA nightly VLM MMMU scores for the same models/commit; manually inspect a few /tmp/mmmu_<model>.html reports for whether model outputs are coherent answers vs garbage/empty; run one model (e.g. Qwen2.5-VL-7B) on full MMMU on an MI30x box.
• [LOW] Borderline test design — 100-sample subset with thresholds set only marginally above chance makes pass/fail inherently noisy at the margin.
  • Supporting: Qwen2-VL-7B flips PASS/FAIL run-to-run right at 0.30; small sample count.
  • Disconfirming: the 4 never-passing models miss by a wide, consistent margin (e.g. Qwen2.5-VL 0.28 vs 0.33), which sample noise alone does not explain.
  • How to verify: not actionable until the systemic hypothesis is checked.

This is not a regression introduced by ed26a109 — the failure predates it across the whole window.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for VLM MMMU AMD and mmmu threshold; matches returned (#18186 "[AMD] Fix AMD CI", #26606 Gemma vision perf) are not targeted at this VLM accuracy failure.

Suggested Triage Steps

• Cross-check the same 11 models' MMMU scores on a working backend (NVIDIA nightly, same commit) to confirm whether ~0.28 is AMD/ROCm720-specific degradation or a suite-wide low baseline.
• Inspect the generated /tmp/mmmu_<model>.html / .json reports (e.g. for Qwen2.5-VL-7B) to see whether model responses are coherent answers or malformed — this distinguishes a broken VLM image/prompt path from a grading/threshold issue.
• If outputs are coherent but uniformly low, the thresholds and 100-sample subset likely need revisiting (test-config issue); if outputs are garbage, bisect VLM/multimodal forward-path changes on ROCm720 between an older suspected-good period and now.
• For the borderline Qwen2-VL-7B (0.30 == 0.30), consider whether the margin warrants a small threshold buffer to stop it oscillating — separate from the 4 wide-margin failures.

Status

• Persistence: Failing every completed scheduled run since at least Jun 12, 2026 (12+ days); 4 models never pass in the queryable window.
• Scope: 1 job (nightly-accuracy-2-gpu-vlm-rocm720) in nightly-test-amd-rocm720.yml; the sibling nightly-perf-2-gpu-vlm-rocm720 job passes, so this is accuracy-specific, not a server/launch failure.
• Blocked work: AMD ROCm 7.2 VLM accuracy signal is permanently red, masking any genuine new VLM accuracy regression.
• In-flight fix: ❌ none found in open PRs.

nightly-accuracy-2-gpu-rocm720 — failed step(s): Nightly Test ROCm 7.2 (2-GPU)

Commit Info

• sglang: ed26a109
• aiter: 7d604afe5fa7efba63c0dce323b95d9daf2db112 (source: dev image — [CI-AITER-CHECK] reports v0.1.14rc1.dev240+g7d604afe5, "Dev/patched version detected → skipping rebuild"; matches Dockerfile default)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi30x/test_gsm8k_eval_amd.py	test_gsm8k_all_models (model=Qwen/Qwen2-57B-A14B-Instruct)	TimeoutError: Server failed to start within the timeout period	Log
test/registered/amd/accuracy/mi30x/test_gsm8k_eval_amd.py	test_gsm8k_all_models (model=mistralai/Mistral-7B-Instruct-v0.3)	gsm8k score 0.361 < threshold 0.47 (❌ FAILED, 3 attempts)	Log
test/registered/amd/accuracy/mi30x/test_gsm8k_eval_amd.py	test_gsm8k_all_models (model=mistralai/Mixtral-8x7B-Instruct-v0.1)	gsm8k score 0.315 < threshold 0.69 (❌ FAILED, 3 attempts; preceded by ~10 watchdog-timeout cycles)	Log

Failure Summary

The 2-GPU nightly gsm8k accuracy test test_gsm8k_all_models failed. The unittest framework reported FAILED (errors=1): the hard error was a TimeoutError launching the Qwen/Qwen2-57B-A14B-Instruct server (TP=2). In the same run, Mistral-7B-Instruct-v0.3 (0.361 vs 0.47) and Mixtral-8x7B-Instruct-v0.1 (0.315 vs 0.69) scored well below their accuracy thresholds, and the Mixtral server logged a continuous ~50-minute storm of soft watchdog timeouts (Scheduler/TokenizerManager watchdog timeout, self.watchdog_timeout=300) plus earlier Health check failed messages.

Regression Status

Known recurring failure (accuracy thresholds) + new sub-failure today (Qwen server-start timeout).

• The accuracy threshold failures for Mistral-7B-Instruct-v0.3 and Mixtral-8x7B-Instruct-v0.1 recur in every completed nightly run examined (scores are stable and far below threshold: Mistral ~0.36, Mixtral ~0.30).
• The Qwen/Qwen2-57B-A14B-Instruct server-start timeout is new today — on Jun 20 and Jun 22 that server started and was evaluated successfully (the run failed instead via the accuracy AssertionError).

Recent history of test_gsm8k_eval_amd.py (test_gsm8k_all_models) in job nightly-accuracy-2-gpu-rocm720 (branch main, event schedule, completed runs only):

Date	Run	Test File Status	Failing Detail
Jun 23	28046795462	❌ Failed	Qwen2-57B server-start TimeoutError (errors=1); Mistral 0.361, Mixtral 0.315 below threshold
Jun 22	27975850577	❌ Failed	accuracy AssertionError (failures=1); Mistral 0.362, Mixtral 0.301, gemma-2-27b 0.860, FP8 models 0.159–0.317
Jun 21	27912828989	❌ Failed	Mistral 0.363, gemma-2-27b 0.857 below threshold (log truncated)
Jun 20	27879239829	❌ Failed	accuracy AssertionError (failures=1); Mixtral 0.296, FP8 models 0.170–0.308 below threshold

(Jun 19/18/17/16 runs were cancelled → excluded. No green run for this test file in the queryable window Jun 12–23; treat the accuracy failures as Never-passed in that window.)

Failure Cluster

AMD ROCm 2-GPU gsm8k accuracy below threshold + server instability (watchdog-timeout storms / server-start timeout) on nightly eval.

Facts (verified observations)

• Per-model results today (verbatim from log): Mistral-7B-Instruct-v0.3 Score 0.361 (threshold 0.47) → ❌ FAILED; Mixtral-8x7B-Instruct-v0.1 Score 0.315 (threshold 0.69) → ❌ FAILED. Passing models in the same run: Llama-3.1-8B 0.818, DeepSeek-Coder-V2-Lite 0.811, gemma-2-27b-it 0.861, Llama-3.1-70B 0.932.
• Hard error (verbatim): TimeoutError: Server failed to start within the timeout period at test_gsm8k_eval_amd.py:258 → :139 → test_utils.py:1028 (popen_launch_server), on Qwen/Qwen2-57B-A14B-Instruct (TP=2).
• During the Mixtral run, TokenizerManager/Scheduler watchdog timeout (self.watchdog_timeout=300, self.soft=True) repeated roughly every 5 min from 19:30 to 20:21 UTC; Health check failed. Server couldn't get a response from detokenizer for last 20 seconds appeared at 18:56 and 19:09.
• Test ran 5510 s vs estimated 3600 s; step 6 "Nightly Test ROCm 7.2 (2-GPU)" concluded failure; runner then logged First attempt failed with exit code 255.
• aiter is unchanged vs the Dockerfile default (dev image preserved, 7d604afe5...), so the aiter version is not a variable here.
• Each model is retried up to 3× internally; Mistral (0.371/0.361/0.361) and Mixtral (0.301/0.303/0.315) never reached threshold across attempts — consistent, not a single-shot flake.

Hypothesised Causes (with confidence)

• [MEDIUM] The Mistral/Mixtral accuracy shortfalls are a persistent, reproducible problem on AMD ROCm 7.2 2-GPU (not flakiness).
  • Supporting: identical low scores (~0.36 / ~0.30) across 4 consecutive nightly runs and across 3 internal retries each; other models in the same job pass cleanly.
  • Disconfirming: no green baseline exists in the Jun 12–23 window, so I cannot point to a specific regressing commit — the thresholds may also be mis-calibrated for the ROCm chat path (see in-flight PR below). This is symptom-level; root cause is not yet code-verified.
  • How to verify: run test_gsm8k_eval_amd.py for just Mistral/Mixtral on an MI30x ROCm 7.2 container and compare against an NVIDIA nightly score for the same models; if NVIDIA also low, it's a threshold/eval-harness issue, not an AMD kernel regression.
• [LOW] The Qwen2-57B server-start TimeoutError and the Mixtral watchdog-timeout storm reflect runner/GPU-state instability rather than a clean accuracy regression.
  • Supporting: Qwen2-57B started fine on Jun 20 & 22 with the same job; today it timed out only after the GPU had been through ~50 min of Mixtral watchdog timeouts — suggesting a degraded/leaked GPU state by the time Qwen launched.
  • Disconfirming: I did not confirm an OOM/HIP fault in the Qwen launch logs; the timeout could also be a slow weight download/load. No memory-access fault or HIP error was found in the log.
  • How to verify: inspect the Qwen2-57B server stdout in the job (look for OOM / load progress); re-run the job in isolation to see if Qwen starts when the GPU is fresh.

In-flight Fix Check

• In-flight fix: ✅ #27757 "Fix Mistral GSM8K chat eval" (open since 2026-06-10, updated 2026-06-22) — directly targets the Mistral gsm8k chat eval; chase reviewers rather than opening a duplicate for the Mistral failure.
• In-flight fix: ❌ none found for the Mixtral threshold shortfall or the Qwen2-57B server-start timeout (searched open PRs for gsm8k accuracy, nightly amd threshold, Qwen2-57B).

Suggested Triage Steps

• Verify #27757 addresses the Mistral gsm8k chat-eval path and re-run nightly-accuracy-2-gpu-rocm720 to confirm it lifts the Mistral score above 0.47.
• For Mixtral, A/B the same model+threshold on an NVIDIA nightly to decide whether this is an AMD-specific accuracy regression vs a threshold mis-calibration; if AMD-specific, bisect MoE/aiter-related changes touching the Mixtral forward path.
• For the Qwen2-57B server-start timeout, re-run the job in isolation (or first in the matrix) to check whether it starts on a fresh GPU; if it only fails after the Mixtral watchdog storm, the actionable fix is upstream Mixtral stability, not Qwen.
• Investigate the Mixtral watchdog-timeout storm (watchdog_timeout=300) as the likely trigger for the cascading slowdown/timeout — capture a py-spy/cuda dump during a Mixtral run.

Status

• Persistence: accuracy threshold failures present in every completed nightly run Jun 20–23 (no green run found Jun 12–23). Qwen2-57B server-start timeout first observed today (Jun 23).
• Scope: single job nightly-accuracy-2-gpu-rocm720 (workflow nightly-test-amd-rocm720.yml); blocks the AMD ROCm 7.2 2-GPU nightly accuracy signal.
• In-flight fix: ✅ #27757 for Mistral; ❌ none for Mixtral / Qwen2-57B.

---

⏳ 33 job(s) still running (run) — will update when complete

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-06-23 20:59 UTC)

[amd-bot]

pr-test-amd.yml — Cross-Run Pattern Summary (8 runs in lookback window)

Last scanned (UTC): 2026-06-23 21:00 UTC

Agent assessment

Persistent clusters

1. FLUX.2 modelopt-fp8 HIPBLAS_STATUS_NOT_SUPPORTED + ideogram4 gated-repo 403 — multimodal-gen-test-2-gpu-amd (…, 1)

• Jobs: latest shard-1 — same shard fails in all 8/8 runs in window (Never-passed; predates window, ≥4 days).
• test_server_2_gpu.py::test_diffusion_generation[flux2_modelopt_fp8_tp2_t2i] (FAILED, torch._scaled_mm → fp8_utils.py:1695) and [ideogram4_fp8_tp2_t2i] (ERROR at setup, HF 403).
• [HIGH] flux2 = hipBLAS fp8 e4m3fn GEMM unsupported on gfx942; disconfirming: never-passed in window, so an unported path, not a fresh regression. ideogram4 [FACT] = CI HF token lacks access (not code).
• In-flight fix: flux2 ✅ #28889 (open since 2026-06-22, non-draft); ideogram4 ❌ none (access/infra; draft #28225 partial).

2. MI35x mooncake PD KV-transfer timeout / RDMA segfault — stage-b-test-large-8-gpu-mi35x-disaggregation-amd

• Jobs: latest — failing 5 consecutive runs since first failure 2026-06-22 20:34 UTC (last green run 27958083310, Jun 22 14:15).
• test_disaggregation_basic.py::test_gsm8k (KVTransfer 300s WaitingForInput timeout; latest run a SIGSEGV in Mooncake ibv_destroy_qp) + test_disaggregation_pp.py::setUpClass (cascade, GPU mem not released, exit -9). test_mori_transfer_engine_e2e.py intermittent (2/4).
• [MEDIUM] infra/Mooncake RDMA fabric instability on linux-mi35x-gpu-8.fabric; disconfirming: clean pass→fail boundary at sha adc203df is unusually deterministic for pure infra — but no commit in window 4923bb93ae..adc203df touches mooncake/RDMA/transfer code.
• In-flight fix: ❌ none confirmed; related #26326 (mooncake bump), #27012 (job revert, exact-name match), #28440.

3. anthropic SDK / httpx socket_options TypeError — stage-b-test-1-gpu-small-amd (…, 13)

• Jobs: run 27958083310 — failed first 3 window runs (27923265639, 27938642367, 27958083310); not seen since Jun 22 14:03 → appears self-resolved (likely image httpx bump).
• test_anthropic_server.py::test_in_messages_system_role; [HIGH] lmms_eval==0.4.1 pins httpx==0.23.3, anthropic 0.111.0 needs ≥0.25. In-flight fix: ❌ none found (resolved environmentally).

Regression candidates (latest run 28048998470 only)

• stage-a-test-1-gpu-small-amd → job: mori build fails — libabsl_base.so.20210324: invalid ELF header during gtest discovery (N/A test; build step). Fresh regression vs flake: passed last 3 runs with the identical mori commit bf99bdf1 12 h earlier → [MEDIUM] corrupt system lib on this runner (not code; sha e0dc8b71 compiles no code here). Triage: re-run on clean runner; readelf -h the lib; check the public-registry fallback image pull. Not a commit-bisect candidate.

Flakes / intermittent

• multimodal-gen-test-1-gpu-amd runner-loss / hang — shard 1 failed 3/8 (27923265639, 27938642367, 27977371862, all ~53-55 min runner-lost-comm, no log); shard 3 hung once (28009346928, 90-min FLUX.1-dev timeout). Investigate runner host (dmesg/OOM/GPU reset); add heartbeat logging. No in-flight fix.
• stage-c-test-large-8-gpu-amd — three different single-run infra symptoms: Kimi-K2 launch timeout cold-cache (27977371862 …,2), RCCL allreduce SIGSEGV pre-flight (28009346928 …,0), runner abrupt-termination/no-log (28028218911 …,1). Each 1/run; retry; none exercise sglang regressions (RCCL test imports no sglang).
• multimodal-gen-test-2-gpu-amd (…, 0) port 5555 — 1/8 (28009346928), --strict-ports orphan-port conflict. Retry; reap stale schedulers between jobs.
• stage-b-test-1-gpu-large-amd — 1/8 (27977371862), 45-min job timeout from test_bench_serving_1gpu_part2.py perf overrun (p95 56.7ms > 50). Retry; consider splitting the file.
• call-pr-test-amd-extra / extra-a-* — 1/8 (27977371862), fast-fail cancellation cascade from that run's earlier failures (no test ran). No-signal, not a failure.

Suggested next actions

• Chase reviewers on #28889 to clear cluster 1's flux2 leg (largest persistent cluster, 8/8 runs); separately decide HF-access vs skip for ideogram4.
• For cluster 2, re-run stage-b-…-mi35x-disaggregation-amd at last-green sha 4923bb93 and inspect RDMA NIC health (transport retry counter exceeded, QP Invalid argument [22]) — separates Mooncake/image drift from a code trigger before any bisect of 4923bb93ae..adc203df.
• Re-run latest stage-a libabsl regression candidate on a clean runner; if widespread across jobs pulling the fallback image, suspect a corrupt published layer.
• Treat the multimodal-gen-1-gpu and stage-c flakes as a runner-health workstream (host dmesg/OOM/GPU-reset around the failure windows); add per-step heartbeat/live-log flushing so future runner-loss failures leave diagnosable output.
• Confirm cluster 3 (anthropic httpx) is truly resolved — verify current AMD image ships httpx>=0.25; if so, no action.

---

Runs analysed

Run	Started (UTC)	Failed jobs
27923265639	2026-06-22 01:00	3
27938642367	2026-06-22 08:25	3
27958083310	2026-06-22 14:03	2
27977371862	2026-06-22 20:00	7
27994433780	2026-06-23 00:59	2
28009346928	2026-06-23 09:40	5
28028218911	2026-06-23 14:27	3
28048998470	2026-06-23 19:17	3

Persistent failures (in every run)

• multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 1) — in 8/8 runs (3265639, 8642367, 8083310, 7371862, 4433780, 9346928, 8218911, 8998470)

Latest-only failures (potential regressions)

• stage-a-test-1-gpu-small-amd (linux-mi325-1gpu-sglang) — in 1/8 runs (8998470)

Flaky / intermittent failures

• call-pr-test-amd-extra / extra-a-test-1-gpu-large-amd (linux-mi325-1gpu-sglang) — in 1/8 runs (7371862)
• call-pr-test-amd-extra / extra-a-test-1-gpu-small-amd (linux-mi325-1gpu-sglang) — in 1/8 runs (7371862)
• multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 1) — in 3/8 runs (3265639, 8642367, 7371862)
• multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 3) — in 1/8 runs (9346928)
• multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 0) — in 1/8 runs (9346928)
• stage-b-test-1-gpu-large-amd (linux-mi325-1gpu-sglang, 1) — in 1/8 runs (7371862)
• stage-b-test-1-gpu-small-amd (linux-mi325-1gpu-sglang, 13) — in 3/8 runs (3265639, 8642367, 8083310)
• stage-b-test-large-8-gpu-mi35x-disaggregation-amd (linux-mi35x-gpu-8.fabric) — in 5/8 runs (7371862, 4433780, 9346928, 8218911, 8998470)
• stage-c-test-large-8-gpu-amd (linux-mi325-8gpu-sglang, 0) — in 1/8 runs (9346928)
• stage-c-test-large-8-gpu-amd (linux-mi325-8gpu-sglang, 1) — in 1/8 runs (8218911)
• stage-c-test-large-8-gpu-amd (linux-mi325-8gpu-sglang, 2) — in 1/8 runs (7371862)

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-06-23 21:00 UTC)

[amd-bot]

pr-test-amd-rocm720.yml — 94 failure(s)

Run started (UTC): 2026-06-22 00:42 UTC
Last scanned (UTC): 2026-06-23 19:23 UTC
Commits: sglang ed26a10

Summary

Counts: 94 failures · 3 clusters · 90 are CI concurrency/scheduling cancellations (no test signal) · 4 substantive test-level failures · 1 new today

#	Cluster	Job	Test File	Test Function	Status	Hypothesis (confidence)
1	Concurrency/scheduling cancellation	dsv4-pro-fp4-amd-rocm720 (+89 more)	N/A	N/A	90 jobs; recurring across Jun 22–23	[FACT] run 28046873327 superseded by newer scheduled run 28048657152 (cancel-in-progress)
2	HIP OOM in DSV4 compress-state KV-score buffer	dsv4-flash-fp4-fp8-amd-rocm720	test/registered/amd/test_deepseek_v4_flash_fp4.py	TestDeepseekV4Fp4.setUpClass	3 runs persistent (Jun 22)	[HIGH] over-aggressive pool sizing; fix #28941 merged post-run
3	HIP OOM in DSV4 compress-state KV-score buffer	dsv4-flash-fp4-fp8-amd-rocm720	test/registered/amd/test_deepseek_v4_flash_fp8.py	TestDeepseekV4Fp8.setUpClass	3 runs persistent (Jun 22)	[HIGH] same pool-sizing; fix #28941 merged post-run
4	HIP-runtime hang in DSV4-Pro MoE weight load	dsv4-pro-fp4-amd-rocm720	test/registered/amd/test_deepseek_v4_pro_fp4.py	N/A (server never ready)	new today (Jun 23); passed 6.5h earlier	[MEDIUM] ROCm-7.2 HIP mutex hang during weight memcpy

Cluster: Concurrency/scheduling cancellation

• Affected: ~90 of 94 jobs — the entire run 28046873327 (40 jobs killed at 18:40:51Z) plus earlier superseded/runner-starved scheduled runs. No test verdict was reached on any of these.
• Shared facts: Workflow concurrency group keys on github.ref with cancel-in-progress: true for schedule events (pr-test-amd-rocm720.yml:117-123). Two overlapping crons (30 17 * * * and 0 */6 * * *) plus scheduler delays cause back-to-back scheduled runs to cancel each other. Cancelled-job log blobs return BlobNotFound.
• Top hypothesis: [FACT] newer scheduled run created at 18:40:50Z; all 40 jobs cancelled at 18:40:51–52Z. Disconfirming: (none — timing is exact and config directly produces this).
• In-flight fix: ❌ none — this is a workflow concurrency/cron config matter, not a code bug.
• Suggested triage: Treat as non-failures (no test regression). Consider whether the two overlapping crons should be de-conflicted so scheduled runs stop cancelling each other; re-read results from the surviving newer run instead.

Cluster: HIP OOM in DSV4 compress-state KV-score buffer

• Affected: dsv4-flash-fp4-fp8-amd-rocm720 across runs 27957777753, 27938067935, 27922836455 — both test_deepseek_v4_flash_fp4.py and _fp8.py fail in setUpClass.
• Shared facts: Identical stack — torch.OutOfMemoryError: HIP out of memory allocating ~5.76 GiB at deepseek_v4_compress_state.py:149 _alloc_kv_score_buffer, with ~275 GiB already held; server killed exit -9. aiter unchanged (dev image preserved). MI35x/gfx950 ROCm 7.2.
• Top hypothesis: [HIGH] c4_kv_pool/c128_kv_pool allocated unconditionally before the score buffer exhaust the 288 GiB. Disconfirming: fix edits deepseek_v4_memory_pool.py (upstream) not the throwing line; needs a post-fix green re-run to confirm.
• In-flight fix: ✅ (already merged) #28941 (cee1caaf, merged 2026-06-22 14:35 UTC). These runs predate it — do NOT open a duplicate.
• Suggested triage: Confirm the next scheduled run on a post-cee1caaf SHA shows the job green; if so, closed by #28941.

Cluster: HIP-runtime hang in DSV4-Pro MoE weight load

• Affected: dsv4-pro-fp4-amd-rocm720 in run 28008999561 — test_deepseek_v4_pro_fp4.py, server never reached test phase.
• Shared facts: py-spy shows schedulers stuck in load_weights (deepseek_v4.py:2390) → _load_w13/_load_w2 (fused_moe_triton/layer.py) → copy_ → pthread_mutex_lock in libamdhip64.so.7.2.70200; Scheduler watchdog timeout repeats until job cancelled (~33 min). Commit window ba9d5aed98..c4376aaa88 is 14 commits, none touching the weight-load path.
• Top hypothesis: [MEDIUM] intermittent ROCm-7.2 HIP-runtime hang during MoE weight memcpy. Disconfirming: only one hung run observed; same job passed 6.5h earlier — flakiness frequency not yet established.
• In-flight fix: ❌ none found.
• Suggested triage: Re-run on the same MI35x runner; if it passes, classify as infra flake. If it reproduces, capture rocm-smi/dmesg/py-spy --native during the stall.

Suggested triage order

Rank	Triage step	Targets	Why this first
1	Confirm dsv4-flash job passes on a post-cee1caaf SHA	rows 2–3	Fix already merged (#28941); just needs verification, highest ROI
2	De-conflict the two overlapping scheduled crons / concurrency group	row 1 (90 jobs)	Largest blast radius; removes the noise masking real signal
3	Re-run dsv4-pro-fp4 on same runner; capture rocm-smi/py-spy if it reproduces	row 4	New today, no in-flight fix; cheap rerun establishes flake vs regression

---

Job	Failed Steps	Started
dsv4-pro-fp4-amd-rocm720	N/A	2026-06-22T21:52
call-pr-test-amd-extra-rocm720 / extra-a-test-1-gpu-large-amd-rocm720 (linux-mi325-1gpu-sglang)	N/A	2026-06-22T21:53
dsv4-flash-fp4-fp8-amd-rocm720	N/A	2026-06-22T21:52
dsv4-pro-fp4-amd-rocm720	N/A	2026-06-22T18:45
call-pr-test-amd-extra-rocm720 / extra-a-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang)	N/A	2026-06-22T21:53
call-pr-test-amd-extra-rocm720 / extra-a-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang)	N/A	2026-06-22T18:45
call-pr-test-amd-extra-rocm720 / extra-a-test-1-gpu-large-amd-rocm720 (linux-mi325-1gpu-sglang)	N/A	2026-06-22T18:45
dsv4-flash-fp4-fp8-amd-rocm720	N/A	2026-06-22T18:45
stage-a-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang)	N/A	2026-06-22T18:45
stage-b-test-1-gpu-small-amd-nondeterministic-rocm720 (linux-mi325-1gpu-sglang)	N/A	2026-06-22T18:45
stage-c-test-large-8-gpu-amd-mi35x-rocm720 (linux-mi35x-gpu-8, 1)	N/A	2026-06-22T18:45
stage-b-test-large-8-gpu-mi35x-disaggregation-amd-rocm720 (linux-mi35x-gpu-8.fabric)	N/A	2026-06-22T18:45
stage-c-test-large-8-gpu-amd-mi35x-rocm720 (linux-mi35x-gpu-8, 0)	N/A	2026-06-22T18:45
stage-b-test-2-gpu-large-amd-rocm720 (linux-mi325-2gpu-sglang, 0)	N/A	2026-06-22T18:45
stage-c-test-large-8-gpu-amd-rocm720 (linux-mi325-8gpu-sglang, 3)	N/A	2026-06-22T18:45
stage-b-test-1-gpu-large-amd-rocm720 (linux-mi325-1gpu-sglang, 0)	N/A	2026-06-22T18:45
stage-c-test-large-8-gpu-amd-rocm720 (linux-mi325-8gpu-sglang, 2)	N/A	2026-06-22T18:45
stage-c-test-large-8-gpu-amd-rocm720 (linux-mi325-8gpu-sglang, 0)	N/A	2026-06-22T18:45
stage-b-test-1-gpu-large-amd-rocm720 (linux-mi325-1gpu-sglang, 1)	N/A	2026-06-22T18:45
multimodal-gen-test-1-gpu-amd-rocm720 (linux-mi325-1gpu-sglang, 0)	N/A	2026-06-22T18:45
multimodal-gen-test-2-gpu-amd-rocm720 (linux-mi325-2gpu-sglang, 1)	N/A	2026-06-22T18:45
multimodal-gen-test-2-gpu-amd-rocm720 (linux-mi325-2gpu-sglang, 2)	N/A	2026-06-22T18:45
stage-b-test-2-gpu-large-amd-rocm720 (linux-mi325-2gpu-sglang, 1)	N/A	2026-06-22T18:45
multimodal-gen-test-2-gpu-amd-rocm720 (linux-mi325-2gpu-sglang, 0)	N/A	2026-06-22T18:45
stage-c-test-large-8-gpu-amd-rocm720 (linux-mi325-8gpu-sglang, 1)	N/A	2026-06-22T18:45
stage-b-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang, 1)	N/A	2026-06-22T18:45
multimodal-gen-test-1-gpu-amd-rocm720 (linux-mi325-1gpu-sglang, 1)	N/A	2026-06-22T18:45
stage-b-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang, 0)	N/A	2026-06-22T18:45
multimodal-gen-test-1-gpu-amd-rocm720 (linux-mi325-1gpu-sglang, 2)	N/A	2026-06-22T18:45
stage-b-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang, 2)	N/A	2026-06-22T18:45
multimodal-gen-test-1-gpu-amd-rocm720 (linux-mi325-1gpu-sglang, 3)	N/A	2026-06-22T18:45
stage-b-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang, 8)	N/A	2026-06-22T18:45
stage-b-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang, 7)	N/A	2026-06-22T18:45
stage-b-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang, 12)	N/A	2026-06-22T18:45
stage-b-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang, 4)	N/A	2026-06-22T18:45
stage-b-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang, 13)	N/A	2026-06-22T18:45
stage-b-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang, 11)	N/A	2026-06-22T18:45
stage-b-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang, 9)	N/A	2026-06-22T18:45
stage-b-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang, 6)	N/A	2026-06-22T18:45
stage-b-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang, 5)	N/A	2026-06-22T18:45
stage-b-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang, 10)	N/A	2026-06-22T18:45
stage-c-test-4-gpu-amd-rocm720	N/A	2026-06-22T19:05
dsv4-flash-fp4-fp8-amd-rocm720	Accuracy Test MI35x ROCm 7.2 (8-GPU DeepSeek-V4-Flash FP4 + FP8)	2026-06-22T13:52
stage-b-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang, 3)	N/A	2026-06-22T18:45
call-pr-test-amd-extra-rocm720 / extra-a-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang)	N/A	2026-06-22T13:52
call-pr-test-amd-extra-rocm720 / extra-a-test-1-gpu-large-amd-rocm720 (linux-mi325-1gpu-sglang)	N/A	2026-06-22T13:52
dsv4-flash-fp4-fp8-amd-rocm720	Accuracy Test MI35x ROCm 7.2 (8-GPU DeepSeek-V4-Flash FP4 + FP8)	2026-06-22T07:58
call-pr-test-amd-extra-rocm720 / extra-a-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang)	N/A	2026-06-22T07:58
call-pr-test-amd-extra-rocm720 / extra-a-test-1-gpu-large-amd-rocm720 (linux-mi325-1gpu-sglang)	N/A	2026-06-22T07:58
dsv4-flash-fp4-fp8-amd-rocm720	Accuracy Test MI35x ROCm 7.2 (8-GPU DeepSeek-V4-Flash FP4 + FP8)	2026-06-22T05:02
call-pr-test-amd-extra-rocm720 / extra-a-test-1-gpu-large-amd-rocm720 (linux-mi325-1gpu-sglang)	Run test	2026-06-22T00:42
call-pr-test-amd-extra-rocm720 / extra-a-test-1-gpu-large-amd-rocm720 (linux-mi325-1gpu-sglang)	N/A	2026-06-23T00:39
call-pr-test-amd-extra-rocm720 / extra-a-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang)	N/A	2026-06-23T00:39
dsv4-pro-fp4-amd-rocm720	Accuracy Test MI35x ROCm 7.2 (8-GPU DeepSeek-V4-Pro FP4)	2026-06-23T12:22
dsv4-flash-fp4-fp8-amd-rocm720	N/A	2026-06-23T18:10
call-pr-test-amd-extra-rocm720 / extra-a-test-1-gpu-large-amd-rocm720 (linux-mi325-1gpu-sglang)	N/A	2026-06-23T18:10
dsv4-pro-fp4-amd-rocm720	N/A	2026-06-23T18:10
call-pr-test-amd-extra-rocm720 / extra-a-test-2-gpu-large-amd-rocm720 (linux-mi325-2gpu-sglang)	Start CI container	2026-06-23T18:39
stage-b-test-1-gpu-small-amd-nondeterministic-rocm720 (linux-mi325-1gpu-sglang)	N/A	2026-06-23T18:11
call-pr-test-amd-extra-rocm720 / extra-a-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang)	N/A	2026-06-23T18:10
stage-b-test-2-gpu-large-amd-rocm720 (linux-mi325-2gpu-sglang, 1)	N/A	2026-06-23T18:11
multimodal-gen-test-2-gpu-amd-rocm720 (linux-mi325-2gpu-sglang, 0)	N/A	2026-06-23T18:11
stage-b-test-1-gpu-large-amd-rocm720 (linux-mi325-1gpu-sglang, 0)	N/A	2026-06-23T18:11
stage-b-test-1-gpu-large-amd-rocm720 (linux-mi325-1gpu-sglang, 1)	N/A	2026-06-23T18:11
stage-b-test-large-8-gpu-mi35x-disaggregation-amd-rocm720 (linux-mi35x-gpu-8.fabric)	N/A	2026-06-23T18:11
stage-a-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang)	N/A	2026-06-23T18:11
stage-c-test-large-8-gpu-amd-mi35x-rocm720 (linux-mi35x-gpu-8, 1)	N/A	2026-06-23T18:11
stage-c-test-large-8-gpu-amd-rocm720 (linux-mi325-8gpu-sglang, 2)	N/A	2026-06-23T18:11
multimodal-gen-test-1-gpu-amd-rocm720 (linux-mi325-1gpu-sglang, 2)	N/A	2026-06-23T18:11
multimodal-gen-test-1-gpu-amd-rocm720 (linux-mi325-1gpu-sglang, 0)	N/A	2026-06-23T18:11
stage-c-test-large-8-gpu-amd-rocm720 (linux-mi325-8gpu-sglang, 0)	N/A	2026-06-23T18:11
stage-c-test-large-8-gpu-amd-rocm720 (linux-mi325-8gpu-sglang, 3)	N/A	2026-06-23T18:11
multimodal-gen-test-2-gpu-amd-rocm720 (linux-mi325-2gpu-sglang, 2)	N/A	2026-06-23T18:11
stage-c-test-large-8-gpu-amd-mi35x-rocm720 (linux-mi35x-gpu-8, 0)	N/A	2026-06-23T18:11
stage-c-test-large-8-gpu-amd-rocm720 (linux-mi325-8gpu-sglang, 1)	N/A	2026-06-23T18:11
stage-b-test-2-gpu-large-amd-rocm720 (linux-mi325-2gpu-sglang, 0)	N/A	2026-06-23T18:11
multimodal-gen-test-2-gpu-amd-rocm720 (linux-mi325-2gpu-sglang, 1)	N/A	2026-06-23T18:11
stage-b-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang, 12)	N/A	2026-06-23T18:11
stage-b-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang, 1)	N/A	2026-06-23T18:11
multimodal-gen-test-1-gpu-amd-rocm720 (linux-mi325-1gpu-sglang, 3)	N/A	2026-06-23T18:11
stage-b-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang, 3)	N/A	2026-06-23T18:11
stage-b-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang, 6)	N/A	2026-06-23T18:11
stage-b-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang, 5)	N/A	2026-06-23T18:11
multimodal-gen-test-1-gpu-amd-rocm720 (linux-mi325-1gpu-sglang, 1)	N/A	2026-06-23T18:11
stage-b-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang, 2)	N/A	2026-06-23T18:11
stage-b-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang, 8)	N/A	2026-06-23T18:11
stage-b-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang, 4)	N/A	2026-06-23T18:11
stage-b-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang, 9)	N/A	2026-06-23T18:11
stage-b-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang, 11)	N/A	2026-06-23T18:11
stage-b-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang, 7)	N/A	2026-06-23T18:11
stage-b-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang, 13)	N/A	2026-06-23T18:11
stage-c-test-4-gpu-amd-rocm720	N/A	2026-06-23T18:40
stage-b-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang, 10)	N/A	2026-06-23T18:11
stage-b-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang, 0)	N/A	2026-06-23T18:11

Per-Job Analysis

dsv4-pro-fp4-amd-rocm720 — failed step(s): N/A

Commit Info

• sglang: adc203df
• aiter: N/A — the cancelled job's log blob no longer exists (BlobNotFound), so [CI-AITER-CHECK] markers are unreadable. The sibling finish job shows AITER_COMMIT_OVERRIDE= (empty), i.e. Dockerfile default would have been used.

Failed Tests

Test File	Test Function	Error	Log
N/A — no test failure	N/A	Job cancelled (conclusion: cancelled), not a test failure	Job

The job ran test/registered/amd/test_deepseek_v4_pro_fp4.py (GSM8K accuracy, SGLANG_DSV4_ACCURACY_ONLY=1) but was cancelled mid-run before any pass/fail verdict. Job logs are no longer retrievable.

Failure Summary

dsv4-pro-fp4-amd-rocm720 did not fail a test — it was cancelled ~2h10m after starting (started 21:52:55, cancelled 00:03:13), far below its timeout-minutes: 480. All four executing test jobs in run 27977093213 (dsv4-pro-fp4, dsv4-flash-fp4-fp8, and the two extra workflow_call jobs) were cancelled at the identical instant 00:03:13. The two jobs reporting failure are the *-finish gate jobs, which exit 1 by design when an upstream job is cancelled.

Regression Status

Infrastructure / workflow-config issue — not a code regression.

Recent scheduled runs of pr-test-amd-rocm720.yml (branch=main, event=schedule, completed runs only):

Date (UTC)	Run	Conclusion	Note
Jun 22 19:05	27977093213	❌ cancelled	this run
Jun 22 18:45	27975943807	❌ cancelled	cancelled at 21:52:55 = exactly when this run's jobs started
Jun 22 13:52	27957777753	❌ cancelled
Jun 22 07:58	27938067935	❌ cancelled
Jun 22 00:42	27922836455	❌ cancelled
Jun 21 18:31	27913598623	❌ cancelled

Every scheduled run for 3+ days concluded cancelled. The only success in the window was a manual workflow_dispatch run 27917172408 (Jun 21 20:54) — which gets a unique concurrency group and is therefore immune to the cancellation. No green scheduled run exists in the queryable window.

Failure Cluster

Scheduled dsv4 ROCm 7.2 run cancelled before completion (no test verdict)

Facts (verified observations)

• conclusion: cancelled for dsv4-pro-fp4-amd-rocm720 (verified via Jobs API). [FACT]
• All 4 executing jobs cancelled simultaneously at 2026-06-23T00:03:13Z; runtime ≈ 2h10m vs timeout-minutes: 480 → not a timeout. [FACT]
• The 17 non-dsv4 stage jobs are skipped → this run executed the 0 */6 * * * (dsv4-only) cron path, which skips call-gate and cascades skips. [FACT] (finish-job log: every stage job prints : skipped)
• pr-test-amd-rocm720-finish and pr-test-amd-extra-finish are failure solely because their loop hits result == "cancelled" and runs exit 1 (verified verbatim in finish-job log). [FACT]
• Concurrency: group: pr-test-amd-rocm720-…|| github.ref → for schedule events this resolves to pr-test-amd-rocm720-refs/heads/main, shared by all scheduled runs; cancel-in-progress: true for schedule. (.github/workflows/pr-test-amd-rocm720.yml:117-123) [FACT]
• dsv4 job timeouts: dsv4-pro-fp4 = 480 min, dsv4-flash-fp4-fp8 = 300 min (:1221, :1171); cron cadence is every 6h (:9). [FACT]
• The cancelled job's log blob is gone (BlobNotFound) — no test-level data recoverable. [FACT]

Hypothesised Causes (with confidence, with disconfirming evidence)

• [MEDIUM] Shared-concurrency self-cancellation between consecutive scheduled runs. All scheduled runs share group pr-test-amd-rocm720-refs/heads/main with cancel-in-progress: true. The previous run 27975943807 was cancelled at 21:52:55 — the exact instant this run's jobs started — which is the textbook signature of a newer same-group run evicting the older one. With dsv4 jobs needing up to 8h but a fresh schedule run arriving every ~6h (plus large queue delays), in-flight runs are routinely killed before finishing.
  • Supporting: previous→this run cancel/start timestamps line up to the second; every scheduled run for 3+ days is cancelled; the only success used a unique-group manual dispatch.
  • Disconfirming: this run (created 19:05) is the newest scheduled run in the queryable list — no newer scheduled run is visible to have cancelled it at 00:03:13. So either the 00:00 0 */6 cron run that should have evicted it is not surfaced by the API, or the 00:03:13 cancel came from a different trigger. This weakens "a newer schedule run cancelled this one" specifically, though the multi-day all-cancelled pattern still implicates the concurrency design.
  • How to verify: re-run via workflow_dispatch (unique group) and confirm dsv4-pro completes; inspect org audit log / actions/runs/27977093213 timeline for the cancel actor; check whether the 0 */6 00:00 cron created a run that the API list is omitting.
• [LOW] Runner-pool contention / external cancel on the linux-mi35x-gpu-8 self-hosted pool at 00:03:13.
  • Supporting: a burst of pull_request runs was created around 00:00–00:20 (e.g. 27992547075 at 00:03:12), competing for the same self-hosted AMD runners.
  • Disconfirming: PR runs use a different concurrency group (refs/pull/N/merge), so they cannot cancel a schedule run via concurrency; the timestamp overlap may be coincidental.
  • How to verify: inspect runner queue/availability metrics for linux-mi35x-gpu-8 at 00:03 UTC.

In-flight Fix Check

• In-flight fix: ❌ none found — search repo:sgl-project/sglang is:pr is:open pr-test-amd-rocm720 cancel returned 0 results. The 9 open dsv4/rocm720 PRs (e.g. #28455, #28237) are DeepSeek-V4 correctness fixes, unrelated to the run cancellation.

Suggested Triage Steps

• Confirm dsv4-pro can actually pass: trigger a manual workflow_dispatch of pr-test-amd-rocm720.yml (unique concurrency group, immune to cancel) and let it run to completion — the last such run 27917172408 succeeded.
• Inspect the cancellation actor: pull the run timeline for 27977093213 (org audit log or gh run view) to confirm whether the cancel was concurrency-driven, timeout, or manual.
• Review the scheduling/concurrency design: a job needing up to 480 min cannot reliably finish under a 6h cron that shares cancel-in-progress: true across all scheduled runs. Consider giving scheduled dsv4 runs a per-run unique group (like dispatch/run_all_tests already get), or reducing the cron frequency for the long dsv4 path.
• Determine why the 0 */6 00:00 UTC cron does not appear as a separate scheduled run while the cancel landed at 00:03:13 — clarifies which hypothesis holds.

Status (factual)

• Persistence: every scheduled run of pr-test-amd-rocm720.yml has concluded cancelled for 3+ days (Jun 20 → Jun 23); no green scheduled run in the queryable window.
• Scope: affects all jobs in each scheduled run — both dsv4 accuracy jobs (dsv4-pro-fp4, dsv4-flash-fp4-fp8) and the extra workflow_call jobs, plus cascading failure on the two *-finish gate jobs.
• Blocked work: scheduled DeepSeek-V4 FP4 accuracy signal on AMD MI35x ROCm 7.2 is not being produced — no scheduled run completes.
• In-flight fix: ❌ none found — needs new investigation/PR.

call-pr-test-amd-extra-rocm720 / extra-a-test-1-gpu-large-amd-rocm720 (linux-mi325-1gpu-sglang) — failed step(s): N/A

Commit Info

• sglang: adc203df
• aiter: N/A — the job produced no log (BlobNotFound for job 82829301913 logs; zero steps recorded), so no [CI-AITER-CHECK] markers are available.

Failed Tests

Test File	Test Function	Error	Log
—	—	No test ran — job conclusion was cancelled, not failure	job

No test file or function failed. The job extra-a-test-1-gpu-large-amd-rocm720 was cancelled mid-flight before producing any test results (no steps executed, log blob absent).

Failure Summary

This is not a test failure — it is a run cancellation. The job started at 21:53:25Z and was cancelled at 00:03:13Z (~2h10m later) with no recorded steps. All four in-flight jobs in run 27977093213 (dsv4-flash-fp4-fp8, dsv4-pro-fp4, extra-a-test-1-gpu-large, extra-a-test-1-gpu-small) were cancelled at the identical instant (00:03:13Z), indicating a run-level cancellation, not a per-job error or timeout.

Regression Status

Infrastructure / scheduling artifact (not a code regression).

Recent scheduled runs of pr-test-amd-rocm720.yml (branch=main, event=schedule, completed runs only) — the workflow-run conclusion, since this job emitted no per-test signal:

Date (UTC)	Run	Conclusion	Note
Jun 22 19:05	27977093213	❌ cancelled	this run — 4 in-flight jobs cancelled at 00:03:13Z
Jun 22 18:45	27975943807	❌ cancelled	cancelled 21:52:55Z — exactly when this run's jobs started
Jun 22 13:52	27957777753	❌ cancelled
Jun 22 07:58	27938067935	❌ cancelled
Jun 22 00:42	27922836455	❌ cancelled
Jun 21 18:31	27913598623	❌ cancelled

Every scheduled run in the queryable window (Jun 20–23) is cancelled. The only non-cancelled recent runs are two workflow_dispatch runs (27917172408, 27915608366, both ✅ success) — which get unique concurrency groups and so are immune to the cancellation chain.

Failure Cluster

"Scheduled run cancelled by concurrency group (no test signal)"

Facts (verified observations)

• Job conclusion = cancelled; steps: []; log endpoint returns BlobNotFound (verified via API).
• All 4 in-flight jobs in the run ended at 00:03:13Z exactly; the run was updated to cancelled at 00:03:20Z.
• Concurrency config (pr-test-amd-rocm720.yml:117-123): for schedule events group = pr-test-amd-rocm720-refs/heads/main (constant) with cancel-in-progress: true. Thus every scheduled run shares one group and a newer scheduled run cancels the older in-progress one.
• The cancellation chain is visible in the data: run 18:45 was cancelled at 21:52:55Z, which is the exact second this run's jobs started — i.e. this run cancelled its predecessor on entry.
• This run only executed dsv4-* + extra-a-* jobs (17 other jobs skipped) — consistent with the 0 */6 cron path described in the workflow ("target dsv4 accuracy jobs only").
• The target job's test step has timeout-minutes: 60 (pr-test-amd-extra.yml:176); the job ran ~130 min, so a 60-min step timeout (which would mark failure) was not the cause.

Hypothesised Causes (with confidence)

• [MEDIUM] Run-level cancellation by the next scheduled trigger via the shared cancel-in-progress concurrency group.
  • Supporting: this is the exact, repeating mechanism observed across the whole window (each run cancelled precisely when the next run's jobs begin; e.g. the 18:45 run cancelled at 21:52:55Z = this run's job start). The 0 */6 cron is due at 00:00 UTC, matching the 00:03:13Z cancellation.
  • Disconfirming: the GitHub API shows no scheduled run created after 23:30 on Jun 22 (total_count = 0); this run is still the newest scheduled run. If a 00:00 cron run had cancelled ours it should appear as a newer run — it does not (GitHub may have dropped the 00:00 scheduled trigger under load).
  • How to verify: check the run's cancellation actor/event in the Actions UI (the timeline shows whether cancellation came from a concurrency replacement vs. runner loss).
• [LOW] Self-hosted AMD (MI325) runner pool dropped connection at 00:03, cancelling all jobs on those runners simultaneously.
  • Supporting: all 4 jobs cancelled at the identical second; no newer workflow run exists to explain a concurrency cancel.
  • Disconfirming: runner-loss often surfaces as failure with a "runner lost communication" line, not a clean cancelled; no log exists to confirm.
  • How to verify: inspect AMD MI325 runner availability/health logs around 00:03Z Jun 23.

In-flight Fix Check

• In-flight fix: ❌ none found. Search: curl ".../search/issues?q=repo:sgl-project/sglang+is:pr+is:open+pr-test-amd-rocm720+concurrency" — no open PR addresses the scheduled-run cancellation pattern. (Note: this is a CI orchestration symptom, not a product bug; a fix may not warrant a PR.)

Suggested Triage Steps

• Open the run timeline in the Actions UI and read the cancellation reason on the cancelled jobs to disambiguate concurrency-cancel vs. runner-loss.
• If concurrency is the cause: consider whether overlapping scheduled runs on main should serialize-and-cancel at all — e.g. give scheduled runs a per-run-id concurrency group (like dispatch runs already get) or reduce cron overlap between 30 17 * * * and 0 */6 * * *, so a long dsv4/extra run isn't routinely killed by the next cron.
• If runner-loss is the cause: check MI325 self-hosted runner health around 00:03Z Jun 23.
• No code-level action on commit adc203df is warranted — the job never executed test code.

Status

• Persistence: Every schedule-triggered run of pr-test-amd-rocm720.yml in the queryable window (Jun 20–23) ended cancelled; workflow_dispatch runs (unique concurrency group) succeed.
• Scope: Run-level — affected all 4 in-flight jobs of run 27977093213 simultaneously; the 17 other jobs were skipped (cron path), and the two *-finish aggregator jobs reported failure purely because their upstream jobs were cancelled.
• Blocked work: Scheduled AMD ROCm 7.2 CI produces no usable pass/fail signal for extra-a-test-1-gpu-large (and siblings) while this cancellation pattern persists.
• In-flight fix: ❌ none found.

dsv4-flash-fp4-fp8-amd-rocm720 — failed step(s): N/A

Commit Info

• sglang: adc203df
• aiter: N/A — the analyzed job (82829214658) was cancelled before acquiring a runner, so no container started and no [CI-AITER-CHECK] markers exist. Sibling runs that did execute used the Dockerfile-default aiter (no AITER_COMMIT_OVERRIDE; this is a schedule run).

Failed Tests

Test File	Test Function	Error	Log
N/A (job cancelled)	N/A	Job cancelled while queued — 0 steps executed, no runner assigned	Job

No test ran in this job. The "failure" surfaced via the gate job pr-test-amd-rocm720-finish, which exit 1s whenever any dependency is failure/cancelled.

Failure Summary

The dsv4-flash-fp4-fp8-amd-rocm720 job was cancelled, not failed. It sat queued from 21:52:55Z waiting for a scarce linux-mi35x-gpu-8 runner, never acquired one (empty runner_name, 0 steps), and was cancelled at 00:03:13Z. All four still-queued GPU jobs in the run (both dsv4-* on mi35x-gpu-8; both extra-a-* on mi325-1gpu) were cancelled at the identical timestamp 2026-06-23T00:03:13Z, the signature of a run-level cancellation of the queued tail. The -finish gate then exited 1 on the cancelled deps.

Regression Status

Infrastructure issue (cancellation), not a test regression — for this run. However, the underlying dsv4-flash accuracy job has a separate, recurring OOM whenever it actually executes (see Facts).

Recent history of job dsv4-flash-fp4-fp8-amd-rocm720 (branch main, event schedule, completed runs only):

Date (UTC)	Run	Outcome	Ran?	Failure
Jun 22 21:52→Jun 23 00:03	27977093213	cancelled	❌ no runner	none (cancelled while queued)
Jun 22 18:45→19:05	27975943807	cancelled	❌ no runner	concurrency cancel-in-progress (superseded by run 27977093213, created 19:05:26Z)
Jun 22 13:52	27957777753	failure	✅ 8 steps	HIP OOM in _alloc_kv_score_buffer
Jun 22 07:58	27938067935	failure	✅ 8 steps	HIP OOM (same)
Jun 22 05:02	27922836455	failure	✅ 8 steps	HIP OOM (same)
Jun 21 19:59	27913598623	failure	✅ 8 steps	HIP OOM (same)

Failure Cluster

Two distinct clusters: (1) "Queued GPU job cancelled — no mi35x-gpu-8 runner available" (this run's actual outcome); (2) "HIP out of memory in DeepSeek-V4 compress-state KV+score buffer alloc" (the recurring failure when the job runs).

Facts (verified observations)

• This run: job cancelled, runner_name='', steps=0, queued 21:52:55Z → cancelled 00:03:13Z. Log blob returns BlobNotFound (no log because no execution). All 4 queued GPU jobs cancelled at the same instant 00:03:13Z; the two -finish jobs ran 2-6 s later and failed on the cancelled deps (verified in finish log: dsv4-flash-fp4-fp8-amd-rocm720: cancelled → exit code 1).
• Concurrency: workflow group is pr-test-amd-rocm720-${{ github.ref }} with cancel-in-progress: true for schedule events (.github/workflows/pr-test-amd-rocm720.yml:117-123). This demonstrably cancelled the previous run 27975943807 at 19:05:27Z — the exact creation time of this run. No newer schedule/main run exists after this one in the API, so this run's 00:03:13Z cancellation was not caused by a successor run.
• Recurring OOM (when executed): identical stack in all 4 executed runs — torch.OutOfMemoryError: HIP out of memory at python/sglang/srt/mem_cache/deepseek_v4_compress_state.py:149 (_alloc_kv_score_buffer, torch.empty((self._size, self.last_dim), …)), reached via deepseek_v4_memory_pool.py:623 → :849 → :803 → deepseek_v4_compress_state.py:128. GPU reports ~275 GiB already allocated of 287.98 GiB; tries to allocate ~5.4-5.8 GiB. Server child exits -9 → setUpClass of both test/registered/amd/test_deepseek_v4_flash_fp4.py and test_deepseek_v4_flash_fp8.py fails with Server process exited with code -9 (run 27957777753).

Hypothesised Causes

• [FACT] This job did not fail a test — it was cancelled while queued for a linux-mi35x-gpu-8 runner. The reported failure is the -finish gate cascading on the cancelled dependency.
  • Supporting: conclusion=cancelled, 0 steps, empty runner_name; 4 GPU jobs cancelled simultaneously at 00:03:13Z; finish-job log shows it tripping on the cancelled dep.
  • Disconfirming: (none) — the job never executed any sglang code.
  • How to verify: re-run the stage via workflow_dispatch target_stage=dsv4-flash-fp4-fp8-amd-rocm720 when an mi35x-gpu-8 runner is free.
• [MEDIUM] The recurring OOM seen in executed runs (Jun 21-22) is likely already addressed by merged fix #28941 (cee1caaf, "[AMD] Fix nightly-8-gpu-mi35x-deepseek-v4-flash-rocm720 OOM issue").
  • Supporting: fix merged 2026-06-22 14:35 UTC; all OOM runs are before that time (latest OOM run 4923bb93 at 13:52Z, fix is not its ancestor); fix is confirmed an ancestor of this run's head adc203df.
  • Disconfirming: the current SHA carrying the fix never ran (cancelled), so there is no post-fix green signal yet for this job; the fix's effectiveness on mi35x-gpu-8 is unconfirmed.
  • How to verify: dispatch dsv4-flash-fp4-fp8-amd-rocm720 at HEAD and confirm the GSM8K accuracy test completes without HIP OOM.

In-flight Fix Check

• OOM: ✅ already merged — #28941 (cee1caaf), in HEAD. Searched is:pr is:open deepseek-v4 OOM → 11 open PRs, none targeting this OOM. No new PR needed; just needs a post-fix run to confirm.
• Cancellation/runner-starvation: ❌ no in-flight PR found; this is a CI capacity/scheduling matter, not a code fix.

Suggested Triage Steps

• Re-dispatch the stage at HEAD when an mi35x-gpu-8 runner is free: workflow_dispatch on pr-test-amd-rocm720.yml with target_stage=dsv4-flash-fp4-fp8-amd-rocm720, to confirm fix #28941 clears the OOM.
• Investigate runner starvation: the mi35x-gpu-8 pool had no free runner for the full 21:52→00:03 window. Check whether the daily 30 17 cron + 0 */6 cron + the chained extra suite are oversubscribing the scarce 8-GPU MI35x pool. Earlier same-day runs (05:02, 07:58, 13:52) did get runners within seconds, so contention is intermittent.
• Consider whether the back-to-back schedule cadence plus cancel-in-progress is repeatedly killing queued long-pole GPU jobs before they can acquire a runner (the prior run 27975943807 was cancelled this way). No code change recommended without maintainer input — this is a CI-policy decision.

Status

• Persistence: This job's cancellation occurred in the last 2 scheduled runs (since ~18:45Z Jun 22); the separate OOM recurred in every executed run Jun 21 19:59Z → Jun 22 13:52Z, then a fix merged 14:35Z Jun 22.
• Scope: Run-level cancellation hit 4 queued GPU jobs (dsv4-flash, dsv4-pro, 2× extra-a-*) in this single run; OOM scoped to dsv4-flash accuracy job (both fp4 + fp8 test files) on mi35x-gpu-8.
• Blocked work: No DeepSeek-V4-Flash accuracy signal produced by this run. Note: this is a schedule run with --continue-on-error, so it does not gate merges.
• In-flight fix: OOM → ✅ merged #28941 (in HEAD, awaiting a post-fix run to confirm); cancellation → ❌ none (CI capacity issue).

dsv4-pro-fp4-amd-rocm720 — failed step(s): N/A

Commit Info

• sglang: 43b1fb95 "[CI] Optimize nixl dependency installation script (#28950)"
• aiter: N/A — the job never executed, so no [CI-AITER-CHECK] markers were emitted (log blob does not exist).

Failed Tests

Test File	Test Function	Error	Log
N/A	N/A	Job cancelled before execution (0 steps ran)	Job

No test ran. The job log endpoint returns BlobNotFound because the job was cancelled while still queued, before any step (including checkout / dependency install) executed.

Failure Summary

The job dsv4-pro-fp4-amd-rocm720 did not fail a test — it was cancelled (conclusion: cancelled, 0 steps executed) while waiting for an linux-mi35x-gpu-8 runner. The entire parent run 27975943807 was cancelled at 19:05:27 UTC, the same instant the next scheduled run 27977093213 was created (19:05:26 UTC) — i.e. the older scheduled run was superseded by concurrency cancellation. 39 of 48 jobs in the run were cancelled; the only 2 failure jobs are the -finish aggregators (call-pr-test-amd-extra-rocm720 / …finish, pr-test-amd-rocm720-finish), which fail by design when upstream jobs are cancelled.

Regression Status

Infrastructure issue (not a test regression).

Recent scheduled runs of pr-test-amd-rocm720.yml (same branch main, event=schedule, completed only):

Date	Run	Conclusion	Note
Jun 22 19:05	27977093213	cancelled	superseded the run below / or by a later one
Jun 22 18:45	27975943807	cancelled	this run — superseded at 19:05:26
Jun 22 13:52	27957777753	cancelled	—
Jun 22 07:58	27938067935	cancelled	—
Jun 22 00:42	27922836455	cancelled	—

Every recent scheduled run of this workflow ends in cancelled — this is the steady-state pattern for this scheduled pipeline (runs are routinely superseded), not a new break introduced by 43b1fb95.

Failure Cluster

Job cancelled before execution (concurrency supersede / never-ran)

Facts (verified observations, no interpretation)

• Job dsv4-pro-fp4-amd-rocm720 (id 82793634364): status=completed, conclusion=cancelled, steps count = 0, runner label linux-mi35x-gpu-8. Started 18:45:41 UTC, completed 19:05:27 UTC.
• Job log API returns BlobNotFound — no log was ever produced (consistent with 0 steps executed).
• Parent run 27975943807: event=schedule, head_sha=43b1fb95, conclusion=cancelled. Job conclusions: 7 success · 39 cancelled · 2 failure.
• The 2 failure jobs are both *-finish aggregator jobs (cascade from cancelled upstream), not test jobs.
• Next scheduled run 27977093213 was created at 19:05:26 UTC — 1 s before this run's cancel timestamp (19:05:27 UTC), confirming a concurrency-group supersede.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [HIGH] The run (and therefore this queued job) was cancelled by concurrency-group supersede: a newer schedule-triggered run started and the workflow's concurrency policy cancelled the in-flight older run.
  • Supporting: next scheduled run created 19:05:26 UTC vs this run cancelled 19:05:27 UTC (1 s apart); job had 0 steps so it was still queued; all recent scheduled runs share the cancelled conclusion.
  • Disconfirming: (none found) — no step ever ran, so an in-job test/runtime cause is ruled out.
  • How to verify: inspect concurrency: block in pr-test-amd-rocm720.yml and confirm cancel-in-progress: true; cross-check the cancel time of older runs against creation time of the next scheduled run.
• [LOW] Runner starvation: the linux-mi35x-gpu-8 runner pool may have been busy, leaving the job queued long enough to be caught by the supersede.
  • Supporting: job sat ~20 min with 0 steps before cancellation.
  • Disconfirming: cancellation timing aligns exactly with the next run's creation, so supersede is the proximate cause regardless of queue time.
  • How to verify: check linux-mi35x-gpu-8 runner availability/queue depth in the 18:45–19:05 window.

In-flight Fix Check

Not applicable — no test or code failure to fix. This is an orchestration/cancellation outcome, not a defect.

• In-flight fix: ❌ none needed — no test failure occurred.

Suggested Triage Steps

• Treat this job result as non-actionable: no test executed, no signal for 43b1fb95.
• If a real signal is needed for dsv4-pro-fp4-amd-rocm720, find the most recent scheduled run where this job actually executed (non-zero steps) and analyze that, or re-dispatch the workflow when no newer scheduled run will supersede it.
• If these mass cancellations are unwanted, review the concurrency configuration in pr-test-amd-rocm720.yml (consider whether cancel-in-progress should apply to schedule events, or whether the cron cadence overlaps run duration).

Status (factual, no priority assignment)

• Persistence: Every recent schedule run of pr-test-amd-rocm720.yml ends cancelled (≥ Jun 21); this run was superseded ~20 min after creation.
• Scope: Run-wide — 39/48 jobs cancelled in this run; the target job never executed.
• Blocked work: No real CI signal produced for the FP4 DSv4-Pro AMD ROCm 7.2 path on commit 43b1fb95; not a code defect.
• In-flight fix: none needed — no test failure occurred.

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-06-23 19:23 UTC)

[amd-bot]

pr-test-amd-rocm720.yml — Per-Job Analysis (continued 2/13)

call-pr-test-amd-extra-rocm720 / extra-a-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang) — failed step(s): N/A

Commit Info

• sglang: adc203df
• aiter: N/A — the job was cancelled before producing any log ([CI-AITER-CHECK] markers never emitted; log blob returns BlobNotFound).

Failed Tests

Test File	Test Function	Error	Log
N/A — no test ran to completion	N/A	Job cancelled mid-run (not a test failure)	Job (log unavailable: BlobNotFound)

Failure Summary

This job did not fail a test — its conclusion is cancelled, not failure. The job started at 21:53:25Z and was forcibly cancelled at 00:03:13Z (mid-run, ~2h10m in) together with all three sibling GPU jobs in the same run, at the exact same second. No job log was ever uploaded (the GitHub log API returns BlobNotFound), so no test function ever reported a pass/fail. The two *-finish gate jobs then reported failure only because their needs were cancelled.

Regression Status

Infrastructure issue (CI orchestration / scheduled-run cancellation) — not a code regression.

Recent scheduled runs of pr-test-amd-rocm720.yml (same branch main, event=schedule, completed runs only):

Date	Run	Event	Run Conclusion
Jun 22 19:05	27977093213	schedule	❌ cancelled (this run)
Jun 22 18:45	27975943807	schedule	❌ cancelled
Jun 22 13:52	27957777753	schedule	❌ cancelled
Jun 22 07:58	27938067935	schedule	❌ cancelled
Jun 22 00:42	27922836455	schedule	❌ cancelled
Jun 21 18:31	27913598623	schedule	❌ cancelled

Every scheduled run is cancelled (systemic, spans many days). By contrast, the two recent manual workflow_dispatch runs (27917172408, 27915608366) both concluded success — consistent with the cancellation being driven by the schedule/concurrency config, which workflow_dispatch is exempt from.

Failure Cluster

Scheduled run self-cancelled by concurrency cancel-in-progress (CI orchestration, not a test failure).

Facts (verified observations)

• Job conclusion is cancelled, not failure; steps: [] (no step ever recorded). Verified via jobs API for job 82829301918.
• Job log API returns <Error><Code>BlobNotFound</Code>...</Error> — no log was uploaded.
• All four GPU jobs in the run cancelled at the identical timestamp 2026-06-23T00:03:13Z: dsv4-flash-fp4-fp8-amd-rocm720 (82829214658), dsv4-pro-fp4-amd-rocm720 (82829214659), extra-a-test-1-gpu-large-amd-rocm720 (82829301913), and this job (82829301918). A simultaneous cancel of independent jobs is the signature of a run-level cancellation, not a per-job timeout (per-job timeout would yield failure at staggered times).
• The pr-test-amd-rocm720-finish gate log shows dsv4-flash-fp4-fp8-amd-rocm720: cancelled followed by ##[error]Process completed with exit code 1 — the finish job's failure is a downstream consequence of the cancellation, not an independent failure.
• Workflow concurrency config (.github/workflows/pr-test-amd-rocm720.yml:117-123): for schedule events the group resolves to pr-test-amd-rocm720-refs/heads/main with cancel-in-progress: true.
• The workflow has two overlapping schedules (.github/workflows/pr-test-amd-rocm720.yml:7-8): 30 17 * * * (daily) and 0 */6 * * * (every 6h at 00/06/12/18 UTC). Both land in the same concurrency group on main.
• This run (created 19:05, the delayed daily 17:30 cron) was cancelled at 00:03:13Z — immediately after the 00:00 UTC 6h cron window.

Hypothesised Causes (with confidence)

• [HIGH] The scheduled run was cancelled by a newer scheduled run entering the shared concurrency group pr-test-amd-rocm720-refs/heads/main (cancel-in-progress: true). The two cron schedules (30 17 * * * + 0 */6 * * *) repeatedly preempt each other, so each long-running scheduled run is killed by the next cron trigger before its GPU jobs can finish.
  • Supporting: every schedule run is cancelled while every workflow_dispatch run is success (workflow_dispatch gets a unique group + cancel-in-progress:false per the config at line 122-123); all 4 jobs cancel at the same instant; cancel time (00:03) aligns with the 0 */6 * * * 00:00 window; concurrency config directly produces this behavior.
  • Disconfirming: I could not directly observe the canceller run record in the API at snapshot time (no pr-test-amd-rocm720.yml run created after 23:00 UTC was visible yet — GitHub frequently delays scheduled run creation, e.g. the prior 00:00 cron appeared at 00:42). The exact canceller could therefore not be pinned to a specific run ID, though the mechanism and pattern are unambiguous.
  • How to verify: re-query .../workflows/pr-test-amd-rocm720.yml/runs?event=schedule&branch=main after the next cron and confirm a newer run with created_at just before 00:03:13Z; or inspect the run's cancellation actor in the Actions UI timeline.
• [LOW] Per-job step timeout (timeout-minutes: 45, .github/workflows/pr-test-amd-extra.yml:145).
  • Supporting: the job ran ~2h10m wall time, far exceeding 45 min.
  • Disconfirming: a step timeout produces failure with a timeout message, not cancelled; and it could not synchronize 4 different jobs to the same second. Most of the 2h10m was almost certainly runner-queue/setup wait, not the 45-min test step.

This is not a scout workflow, so no aiter Failure Origin / A/B baseline check applies. Commit adc203df is not implicated — the cancellation is orchestration-driven and independent of the head commit.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for pr-test-amd-rocm720 concurrency (1 unrelated hit, #24651, a Qwen3.5 FP8 kernel PR) and rocm720 cron cancel (0 hits). No open PR addresses the scheduled-run cancellation.

Suggested Triage Steps

• Confirm the canceller: after the next 0 */6 * * * window, check pr-test-amd-rocm720.yml schedule runs for the run whose start preempts 27977093213; confirm via the Actions UI "cancelled by" actor.
• If confirmed, consider giving the two cron schedules distinct concurrency groups (e.g. include github.event.schedule or github.run_id in the group: key at .github/workflows/pr-test-amd-rocm720.yml:122) so the daily 30 17 run and the 6h 0 */6 runs no longer preempt each other on main.
• Alternatively, set cancel-in-progress: false for schedule events (keeping it on for PR pushes), so a long extra-tier daily run can finish.
• These are investigation/design options for the maintainer — not directives.

Status (factual)

• Persistence: Every scheduled run of pr-test-amd-rocm720.yml on main is cancelled across at least the last ~3 days (Jun 20–22, all schedule runs cancelled); manual dispatch runs succeed.
• Scope: Run-level — all 4 GPU test jobs in this run (this extra-a-small job plus extra-a-large, dsv4-flash-fp4-fp8, dsv4-pro-fp4) are cancelled together each run; the two *-finish gate jobs consequently report failure.
• Blocked work: Scheduled ROCm 7.2 AMD extra-tier + dsv4 accuracy coverage never completes, so this workflow produces no usable scheduled CI signal.
• In-flight fix: ❌ none found in open PRs — needs a new PR or investigation.

call-pr-test-amd-extra-rocm720 / extra-a-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang) — failed step(s): N/A

Commit Info

• sglang: 43b1fb95
• aiter: N/A — the job log blob is expired/unavailable (BlobNotFound) and the job never reached the dependency-install step, so no [CI-AITER-CHECK] markers exist.

Failed Tests

Test File	Test Function	Error	Log
N/A	N/A	Job cancelled by Actions concurrency (no step executed, no test ran)	Job

This is not a test failure. The job extra-a-test-1-gpu-small-amd-rocm720 was cancelled (job conclusion: cancelled, zero steps recorded). The only failure-status jobs in the run are the two aggregator gates (pr-test-amd-rocm720-finish and call-pr-test-amd-extra-rocm720 / pr-test-amd-extra-finish), which fail by design (exit 1) whenever any dependency result is cancelled — see finish log line 150: multimodal-gen-test-1-gpu-amd-rocm720: cancelled → The above jobs failed.

Failure Summary

The entire run 27975943807 (PR Test ROCm 7.2 (AMD), event=schedule, conclusion cancelled) was superseded by a newer scheduled run. Of 48 jobs: 7 succeeded, 2 finish-gates failed, and 39 (including the target) were cancelled. No test code in the target job was ever exercised.

Regression Status

Infrastructure issue (concurrency cancellation) — not a code regression.

Recent history of target job extra-a-test-1-gpu-small-amd-rocm720 (same branch main, event schedule, completed runs only):

Date	Run	Job Conclusion	Test File Status	Note
Jun 22 19:05	27977093213	❌ cancelled	—	superseded run too
Jun 22 18:45	27975943807	❌ cancelled	—	this run
Jun 22 13:52	27957777753	❌ cancelled	—
Jun 22 07:58	27938067935	❌ cancelled	—

All 8 most-recent scheduled runs of pr-test-amd-rocm720.yml on main carry conclusion cancelled — overlapping scheduled runs on the same main ref repeatedly cancel each other. No green scheduled baseline exists in the queryable window for this job.

Facts (verified observations)

• [FACT] Target job conclusion: cancelled, started 18:45:42Z, completed 19:05:27Z, zero steps recorded (GitHub API job object).
• [FACT] Target job was cancelled at 19:05:27Z; the next scheduled run 27977093213 was created at 19:05:26Z — same second.
• [FACT] Concurrency config (.github/workflows/pr-test-amd-rocm720.yml:122-123): group pr-test-amd-rocm720-${{ ... || github.ref }} (= …-refs/heads/main for schedule) with cancel-in-progress: true for schedule events. A newer scheduled run on main therefore cancels the in-flight one.
• [FACT] Run-level conclusion is cancelled; only the two *-finish gates show failure, and their logs show they exit 1 purely because a dependency result was cancelled.
• [FACT] Job log blob returns BlobNotFound (expired); no test output retrievable.

Hypothesised Causes

• [FACT] Concurrency cancellation by a superseding scheduled run.
  • Supporting: cancellation timestamp (19:05:27Z) coincides exactly with creation of run 27977093213 (19:05:26Z); workflow concurrency group keys on github.ref with cancel-in-progress: true for schedule; zero steps ran in the cancelled job.
  • Disconfirming: (none found) — evidence is direct and self-consistent.
  • How to verify: confirm in run 27975943807 UI that cancelled jobs were terminated, not failed; both share concurrency group pr-test-amd-rocm720-refs/heads/main.

There is no code-level failure to attribute. The two schedule crons (30 17 * * * and 0 */6 * * *) plus GitHub schedule-delivery jitter cause scheduled runs to bunch up on main ~20 min apart and cancel each other.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for concurrency / cancel-in-progress keywords:

  curl -s -H "Authorization: token $GH_PAT" "https://api.github.com/search/issues?q=repo:sgl-project/sglang+is:pr+is:open+concurrency+cancel-in-progress"
  

  No open PR targets this scheduled-run self-cancellation pattern.

Suggested Triage Steps

• No code investigation warranted — this is an orchestration artifact, not a test/regression failure. The run is safe to ignore for test-signal purposes.
• If recurring cancelled scheduled runs are undesirable noise, consider scoping the schedule concurrency group per-run (e.g. include github.run_id for schedule like the workflow_dispatch/run_all_tests branches already do) or reconcile the two overlapping crons so scheduled runs on main don't supersede each other mid-flight.
• To get real test signal for this job, re-run the workflow via workflow_dispatch (which uses a unique full-<run_id> concurrency group and won't be cancelled by the schedule crons).

Status

• Persistence: All 8 most-recent schedule runs of pr-test-amd-rocm720.yml on main ended cancelled; the target job specifically was cancelled in each of the last 4 checked runs.
• Scope: Run-wide — 39/48 jobs cancelled in 27975943807; the 2 reported failures are finish-gate cascades, not independent failures.
• Blocked work: Scheduled ROCm 7.2 AMD test signal is effectively lost on overlapping schedule runs (no completed test results), but no PR is blocked by this.
• In-flight fix: ❌ none found.

call-pr-test-amd-extra-rocm720 / extra-a-test-1-gpu-large-amd-rocm720 (linux-mi325-1gpu-sglang) — failed step(s): N/A

Commit Info

• sglang: 43b1fb95 (#28950 "[CI] Optimize nixl dependency installation script")
• aiter: N/A (job produced no log — no [CI-AITER-CHECK] markers available to read)

Failed Tests

Test File	Test Function	Error	Log
N/A	N/A	Job cancelled mid-run before any test step executed	Job

No test ran. The GitHub API returns no steps for this job and the log blob is absent (BlobNotFound), confirming the job was terminated before its test steps produced output.

Failure Summary

The job extra-a-test-1-gpu-large-amd-rocm720 did not fail a test — it was cancelled (conclusion: cancelled) ~20 minutes after starting, with zero recorded steps. The entire parent run 27975943807 was cancelled: 39 of 48 jobs flipped to cancelled at the identical timestamp 2026-06-22T19:05:27Z. This is a CI concurrency/cancellation event, not a code defect.

Regression Status

Infrastructure issue (run cancelled by concurrency) — not a test regression.

Recent schedule, branch=main runs of pr-test-amd-rocm720.yml (completed runs only):

Date	Run	Conclusion	Note
Jun 22 19:05	27977093213	cancelled	Created 19:05:26Z — superseded this run
Jun 22 18:45	27975943807	cancelled	this run
Jun 22 13:52	27957777753	cancelled
Jun 22 07:58	27938067935	cancelled
Jun 22 00:42	27922836455	cancelled

Every recent scheduled run of this workflow ends in cancelled — consistent with a concurrency group that cancels the older in-progress run whenever a newer scheduled run starts.

Failure Cluster

Run cancelled by CI concurrency (superseding scheduled run) — no test executed.

Facts (verified observations)

• [FACT] Job conclusion: cancelled, status: completed; started 2026-06-22T18:45:42Z, completed 2026-06-22T19:05:27Z. The job has an empty steps array in the API response.
• [FACT] The job log endpoint returns BlobNotFound (no log was ever persisted) — the job never reached a logging test step.
• [FACT] 39 of 48 jobs in run 27975943807 share the exact cancellation timestamp 2026-06-22T19:05:27Z — a bulk cancellation, not per-job test failures.
• [FACT] The next scheduled run 27977093213 was created at 2026-06-22T19:05:26Z, one second before this run's jobs were cancelled.
• [FACT] The only two failure jobs in the run are the aggregator gates call-pr-test-amd-extra-rocm720 / pr-test-amd-extra-finish and pr-test-amd-rocm720-finish, which fail by design when upstream jobs are cancelled.
• [FACT] 7 jobs (gates, check-changes, kernel unit tests, mi35x small) completed success before/independent of the cancellation.

Hypothesised Causes

• [HIGH] This run was cancelled because a newer scheduled run of the same workflow superseded it via the GitHub Actions concurrency group.
  • Supporting: next schedule run 27977093213 created 19:05:26Z; all in-progress jobs of this run cancelled 19:05:27Z (1s later); every recent scheduled run of this workflow shows the same cancelled outcome.
  • Disconfirming: I could not read the workflow file's concurrency: block in this run's context to confirm cancel-in-progress: true directly (inferred from timing + the uniform cancellation pattern).
  • How to verify: inspect concurrency in .github/workflows/pr-test-amd-rocm720.yml; confirm the group key collides between consecutive scheduled runs and cancel-in-progress: true.

In-flight Fix Check

Not applicable — there is no test/code failure to fix. No PR search performed (a concurrency cancellation is expected CI behavior, not a defect).

Suggested Triage Steps

• No action needed on application code — 43b1fb95 is not implicated; no test ran.
• If full scheduled signal is desired despite overlap, review the concurrency config in .github/workflows/pr-test-amd-rocm720.yml: the schedule cadence appears to overlap (an 18:45 run and a 19:05 run within 20 min), so each new run cancels the previous. Consider widening the cron interval or excluding scheduled runs from cancel-in-progress.
• To get a real pass/fail signal for extra-a-test-1-gpu-large-amd-rocm720, re-run it from a non-superseded run or trigger a manual workflow_dispatch.

Status

• Persistence: All recent scheduled runs of this workflow end cancelled; this is recurring concurrency behavior, not a persisting test failure.
• Scope: Run-wide — 39 jobs cancelled together; this target job is one of them. No test-level failure observed.
• Blocked work: Scheduled pr-test-amd-rocm720 runs do not complete a full test matrix because each is cancelled by the next; this masks real signal for jobs that hadn't finished.
• In-flight fix: N/A (no code defect).

dsv4-flash-fp4-fp8-amd-rocm720 — failed step(s): N/A

Commit Info

• sglang (this run 27975943807): 43b1fb95
• aiter (this run): N/A — the job never started (no runner, 0 steps), so no [CI-AITER-CHECK] markers were emitted.
• Representative failing run 27957777753 (sha 4923bb93): aiter v0.1.14rc1.dev240+g7d604afe5 (source: dev image preserved; Dockerfile expects 7d604afe).

Failed Tests

This run's job was cancelled before it ran — no test executed. The table below is from the most recent run where the job actually acquired a runner (27957777753, job 82730668049), shown for context.

Test File	Test Function	Error	Log
test/registered/amd/test_deepseek_v4_flash_fp4.py	TestDeepseekV4Fp4.setUpClass	torch.OutOfMemoryError: HIP out of memory (alloc 5.76 GiB, ~2.4 GiB free) → server exit -9	Log
test/registered/amd/test_deepseek_v4_flash_fp8.py	TestDeepseekV4Fp8.setUpClass	torch.OutOfMemoryError: HIP out of memory (alloc 5.37 GiB, ~3.97 GiB free) → server exit -9	Log

Failure Summary

The asked-about job dsv4-flash-fp4-fp8-amd-rocm720 in run 27975943807 did not fail on a test — it was cancelled while queued. It never acquired a linux-mi35x-gpu-8 runner (empty runner_name, 0 steps) and was cancelled at 19:05:27Z after ~20 min in queue, together with 38 other still-queued jobs in the same run. Separately, when this job does get a runner it is a recurring real failure: both DeepSeek-V4-Flash FP4 and FP8 accuracy servers crash during setUpClass with a HIP OOM while allocating the KV+score buffer.

Regression Status

Two distinct things, both ongoing:

1. This run = infrastructure cancellation (concurrency supersede). Every recent schedule run of pr-test-amd-rocm720.yml concludes cancelled.
2. The dsv4-flash test = recurring OOM failure whenever the job actually runs.

Recent history of dsv4-flash-fp4-fp8-amd-rocm720 (main, event=schedule, completed runs only):

Date (UTC)	Run	Conclusion	Ran?	Symptom
Jun 22 21:52	27977093213	cancelled	❌ no runner	concurrency cancel
Jun 22 18:45	27975943807 (this)	cancelled	❌ no runner	concurrency cancel
Jun 22 13:52	27957777753	failure	✅ 8 steps	HIP OOM (verified)
Jun 22 07:58	27938067935	failure	✅ 8 steps	OOM (conclusion only)
Jun 22 05:02	27922836455	failure	✅ 8 steps	OOM (conclusion only)
Jun 21 18:31	27913598623	failure	✅ 8 steps	OOM (conclusion only)

No passing run of this job found in the queryable schedule window (Jun 21–22). When it runs, it fails; when it doesn't get a runner, it is cancelled.

Failure Cluster

Two clusters: (a) Queued job cancelled by concurrency supersede (this run); (b) HIP out-of-memory in DeepSeek-V4 KV+score buffer allocation during server warmup (the substantive recurring failure).

Facts (verified observations)

• This job: runner_name empty, steps: 0, queued 18:45:41Z, cancelled 19:05:27Z. Required label linux-mi35x-gpu-8.
• 39 of 48 jobs in this run were cancelled at the same instant 19:05:27Z; all 39 had 0 steps (none ever got a runner). Only 7 jobs (those that acquired runners) succeeded; the 2 *-finish aggregators failed because dependents were cancelled.
• The next schedule run 27977093213 was created 19:05:26Z — one second before the mass cancellation.
• Workflow concurrency config (.github/workflows/pr-test-amd-rocm720.yml:122-123): for schedule, group = pr-test-amd-rocm720-refs/heads/main (shared across all schedule runs) and cancel-in-progress: true. A newer schedule run therefore cancels the in-progress one.
• OOM (verified in run 27957777753 log): torch.OutOfMemoryError: HIP out of memory raised at python/sglang/srt/mem_cache/deepseek_v4_compress_state.py:149 in _alloc_kv_score_buffer (the HIP branch torch.empty((self._size, self.last_dim), ...)), on all 8 TP ranks. GPU capacity 287.98 GiB, ~275 GiB already allocated by PyTorch, only ~2–4 GiB free, tried to allocate 5.37–5.76 GiB → server exit code -9 → setUpClass error → exit 255.

Hypothesised Causes (with confidence)

• [FACT] This run's cancellation = concurrency supersede. The superseding schedule run was created 1 s before the cancel; the workflow uses a shared cancel-in-progress: true group for schedule events. The job's ~20 min queue wait for a scarce linux-mi35x-gpu-8 runner is what left it still-queued (and thus killable) when the supersede landed.
  • Disconfirming: (none found) — timing, the shared concurrency group, and the 0-step/no-runner state are all consistent.
  • How to verify: compare created_at of 27977093213 vs cancel time (already done).
• [MEDIUM] The recurring OOM is a memory-budget/sizing issue for the 8-GPU DeepSeek-V4-Flash accuracy config on MI35x (gfx950), not an obvious code regression: the KV+score buffer alloc tips an already ~95%-full HBM over the edge during warmup. deepseek_v4_compress_state.py was last touched only by an unrelated NPU commit (b225f48f), so the regression (if any) is more likely upstream of the alloc (model/KV-pool sizing, --mem-fraction-static, or a recently changed default), not in this file.
  • Supporting: identical stack on all 8 ranks; failure is during setUpClass server launch, before any test runs; consistent across all runner-acquired runs Jun 21–22.
  • Disconfirming: no green baseline found in the window, so this may be a never-passed config on MI35x rather than a fresh regression — cannot distinguish without older history.
  • How to verify: bisect against the last known green dsv4-flash run (search further back than Jun 21); or rerun the two test files locally on linux-mi35x-gpu-8 with a lower --mem-fraction-static to confirm it's purely a memory-budget tip-over.

In-flight Fix Check

• Concurrency cancellation: not a bug to "fix" — it is the configured behavior; the actionable item is runner availability (see Status).
• OOM: In-flight fix: ❌ none found — searched open PRs for deepseek v4 memory; matches (#28033, #26915 BF16 KV cache; #27686 ROCm fp8 KV) are feature work, none targets the MI35x dsv4-flash OOM directly.

Suggested Triage Steps

• Treat this specific run's status as cancelled/not-tested, not a code failure — the dsv4-flash signal for sha 43b1fb95 is simply missing.
• For the chronic cancellation: investigate why linux-mi35x-gpu-8 runners are unavailable for ~20 min so schedule runs finish before the next one supersedes them; or consider giving schedule runs a per-run concurrency group (or cancel-in-progress: false) so overlapping crons don't kill each other.
• For the OOM: find the last green dsv4-flash-fp4-fp8 run (look before Jun 21), then git log --oneline <pass_sha>..4923bb93 filtered for mem, kv, compress, mem_fraction, deepseek_v4 to spot a sizing change; if none, treat as a config tip-over and reduce static memory budget for these two tests.
• Verify the OOM reproduces by rerunning test/registered/amd/test_deepseek_v4_flash_fp8.py on an 8-GPU MI35x container.

Status

• Persistence: dsv4-flash-fp4-fp8 has produced no green schedule run in the Jun 21–22 window; it is failure on every run that acquired a runner and cancelled on every run that didn't.
• Scope: This run — 39/48 jobs cancelled (queue starvation + concurrency supersede). The OOM affects both dsv4-flash test files (fp4 + fp8) on linux-mi35x-gpu-8.
• Blocked work: No accuracy signal for DeepSeek-V4-Flash FP4/FP8 on AMD MI35x ROCm 7.2 — both when cancelled (no data) and when run (OOM crash).
• In-flight fix: ❌ none found for the OOM; cancellation is config/runner-capacity driven.

stage-a-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang) — failed step(s): N/A

Commit Info

• sglang: 43b1fb95 — [CI] Optimize nixl dependency installation script (#28950)
• aiter: N/A — job was cancelled with no recorded steps; the log blob returns BlobNotFound, so no [CI-AITER-CHECK] markers are available.

Failed Tests

Test File	Test Function	Error	Log
—	—	No test failure — job was cancelled, not failed	Job

No tests ran. The job conclusion is cancelled (not failure), it recorded zero steps, and its log blob does not exist.

Failure Summary

This is not a test failure. The job stage-a-test-1-gpu-small-amd-rocm720 was cancelled at 2026-06-22 19:05:27 UTC, ~20 min after starting, along with 38 of the run's other jobs — all cancelled in the same second. The two *-finish gate jobs show failure only because their upstream jobs were cancelled. This is a workflow-level concurrency cancellation, not a code or test problem.

Regression Status

Infrastructure issue (concurrency cancellation) — not a regression.

Recent schedule-triggered runs of pr-test-amd-rocm720.yml on main (run-level conclusion, completed only):

Date (UTC)	Run	Conclusion	Note
2026-06-22 19:05	27977093213	cancelled	superseded the run below
2026-06-22 18:45	27975943807	cancelled	this run
2026-06-22 13:52	27957777753	cancelled
2026-06-22 07:58	27938067935	cancelled
2026-06-22 00:42	27922836455	cancelled

Every recent scheduled run carries conclusion cancelled — this is the steady-state behavior of this workflow's concurrency group, not a new failure.

Failure Cluster

Run-wide concurrency cancellation (newer scheduled run superseded this one)

Facts (verified observations, no interpretation)

• Job conclusion is cancelled, not failure (verified via /actions/jobs/82793671938). Job recorded an empty steps array.
• Log fetch returns <Error><Code>BlobNotFound> — no log content available.
• Run 27975943807: 48 jobs → 7 success, 39 cancelled, 2 failure. The 2 failure jobs are gate jobs (pr-test-amd-rocm720-finish, call-pr-test-amd-extra-rocm720 / pr-test-amd-extra-finish) that fail when upstream is cancelled.
• All 39 cancelled jobs stopped at 2026-06-22T19:05:27–28Z — the same instant.
• The next scheduled run 27977093213 (event=schedule, head_branch=main) was created at 2026-06-22T19:05:26Z — one second before the cancellation.
• Workflow concurrency block (.github/workflows/pr-test-amd-rocm720.yml:117-123): group: pr-test-amd-rocm720-${{ ... || github.ref }} → refs/heads/main for schedule events, with cancel-in-progress: true for schedule events.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [HIGH] The run was cancelled by GitHub Actions concurrency cancel-in-progress when a newer scheduled run entered the same concurrency group.
  • Supporting: superseding scheduled run 27977093213 was created at 19:05:26Z; this run's jobs were all cancelled at 19:05:27Z; both are event=schedule on main and resolve to the same group pr-test-amd-rocm720-refs/heads/main; cancel-in-progress is true for schedule events (.github/workflows/pr-test-amd-rocm720.yml:123).
  • Disconfirming: (none found) — the one-second alignment and shared concurrency group are conclusive.
  • How to verify: already verified by the timestamp/group match above.

In-flight Fix Check

Not applicable — no code defect to fix. This is expected concurrency behavior, not a bug.

• In-flight fix: ❌ none needed

Suggested Triage Steps

• No action required for code or tests. The job did not fail; it was cancelled by a newer scheduled run in the same concurrency group.
• If the upstream goal is to obtain a clean signal for commit 43b1fb95, inspect the superseding run 27977093213 (same head_sha) for actual test results.
• If these back-to-back schedule cancellations are undesirable, consider whether the 30 17 daily cron and the 0 */6 6h cron overlapping with long-running prior runs should share the refs/heads/main concurrency group — but this is a workflow-tuning decision, not a regression to fix.

Status (factual, no priority assignment)

• Persistence: Every recent schedule run of this workflow ends cancelled; this is steady-state, not a new break.
• Scope: Run-wide — 39 of 48 jobs in run 27975943807 cancelled simultaneously; not specific to this job.
• Blocked work: No test signal produced by this run for 43b1fb95; the superseding run carries the same SHA.
• In-flight fix: ❌ none needed — concurrency cancellation, not a defect.

stage-b-test-1-gpu-small-amd-nondeterministic-rocm720 (linux-mi325-1gpu-sglang) — failed step(s): N/A

Commit Info

• sglang: 43b1fb95
• aiter: N/A (no job log was produced — the log blob returns BlobNotFound because no step ever executed, so no [CI-AITER-CHECK] markers exist)

Failed Tests

Test File	Test Function	Error	Log
N/A	N/A	No test ran — job cancelled before any step started (0 steps recorded)	Job

Failure Summary

The job did not fail a test. Its conclusion is cancelled (not failure), it recorded 0 steps, and its log blob does not exist. It started 2026-06-22T18:45:52Z and was cancelled 2026-06-22T19:05:27Z (~20 min), spent entirely queued for a linux-mi325-1gpu-sglang runner without executing a single step. The parent run PR Test ROCm 7.2 (AMD) (27975943807) was itself cancelled (7 success, 39 cancelled, 2 failure — the 2 failures are *-finish aggregator jobs that fail when their needed jobs are cancelled).

Regression Status

Infrastructure issue (concurrency cancellation) — not a code regression and not a flaky test.

Recent schedule runs of pr-test-amd-rocm720.yml on main (completed runs only):

Date (UTC)	Run	Created	Conclusion
Jun 22 19:05	27977093213	19:05:26Z	cancelled
Jun 22 18:45	27975943807 (this run)	18:45:31Z	cancelled
Jun 22 13:52	27957777753	13:52:09Z	cancelled
Jun 22 07:58	27938067935	07:58:03Z	cancelled
Jun 22 00:42	27922836455	00:42:29Z	cancelled
Jun 21 18:31	27913598623	18:31:03Z	cancelled
Jun 21 18:01	27912869737	18:01:47Z	cancelled
Jun 21 12:38	27904585132	12:38:43Z	cancelled

Every recent schedule run on main is cancelled — none completed in the queryable window.

Failure Cluster

CI run cancelled by concurrency group (no test executed)

Facts (verified observations, no interpretation)

• [FACT] Job 82793671954 conclusion = cancelled, status = completed, steps array is empty (0 steps) — verified via GET /actions/jobs/82793671954.
• [FACT] The job's log endpoint returns BlobNotFound — no log was ever written, consistent with 0 steps executed.
• [FACT] Job timing: started 2026-06-22T18:45:52Z, completed (cancelled) 2026-06-22T19:05:27Z (~19.5 min), all of it pre-step (queued for the linux-mi325-1gpu-sglang runner).
• [FACT] Parent run 27975943807 (event=schedule, branch main) conclusion = cancelled; of 48 jobs: 7 success, 39 cancelled, 2 failure (both *-finish aggregators).
• [FACT] The next schedule run 27977093213 (also event=schedule, head_branch=main) was created 2026-06-22T19:05:26Z — within 1 second of this job's cancellation timestamp (19:05:27Z).
• [FACT] pr-test-amd-rocm720.yml:122-123 sets the concurrency group to github.ref (= refs/heads/main) for schedule events and cancel-in-progress: true (since run_all_tests, workflow_dispatch, and workflow_call are all false for a schedule trigger).

Hypothesised Causes (with confidence, with disconfirming evidence)

• [HIGH] This run was cancelled by the concurrency group when the next schedule run on main started.
  • Supporting: concurrency group resolves to refs/heads/main with cancel-in-progress: true for schedule events (pr-test-amd-rocm720.yml:122-123); the superseding run 27977093213 was created 19:05:26Z, exactly when this run's job was cancelled (19:05:27Z); both runs share branch main and event schedule, hence the same group.
  • Disconfirming: (none found) — the timing alignment and shared concurrency group are dispositive.
• [MEDIUM] The job never got a runner in time: it sat ~20 min queued (0 steps) before the cancellation arrived, suggesting linux-mi325-1gpu-sglang runner capacity/contention prevents schedule runs from finishing before the next schedule fires.
  • Supporting: 0 steps recorded over ~19.5 min wall-clock; all 8 recent schedule runs are cancelled, none completed; the daily 30 17 and 6-hourly 0 */6 crons (plus apparent delayed firings at 18:45 and 19:05) overlap on the same main group.
  • Disconfirming: cannot confirm runner saturation without runner-queue telemetry, which isn't in the job metadata; some of the 7 success jobs in this run did execute, so capacity is not zero.

In-flight Fix Check

• Searched open PRs:
  • curl -s -H "Authorization: token $GH_PAT" "https://api.github.com/search/issues?q=repo:sgl-project/sglang+is:pr+is:open+pr-test-amd-rocm720+concurrency"
• In-flight fix: ❌ none found for the concurrency-cancellation pattern. Note: this is an orchestration/runner-capacity matter, likely better addressed by CI maintainers than a code PR.

Suggested Triage Steps

• Confirm this is purely cancellation (not a masked failure): re-run the workflow via workflow_dispatch (which gets a unique concurrency group per pr-test-amd-rocm720.yml:122 and won't be cancelled) and observe whether the stage-b-test-1-gpu-small-amd-nondeterministic-rocm720 job completes and passes.
• If schedule runs must complete, consider whether the overlapping 30 17 * * * and 0 */6 * * * crons plus delayed firings are causing back-to-back schedule starts on the same refs/heads/main group; CI maintainers could review whether schedule runs should share a cancel-in-progress group or be queued instead.
• Investigate linux-mi325-1gpu-sglang runner queue depth around 18:45–19:05 UTC — the ~20 min pre-step queue time suggests runner contention is the reason the run couldn't finish before the next schedule superseded it.

Status (factual, no priority assignment)

• Persistence: All 8 most recent schedule runs of pr-test-amd-rocm720.yml on main (Jun 21 12:38 → Jun 22 19:05 UTC) concluded cancelled; none completed in the queryable window.
• Scope: This is a run-level concurrency cancellation affecting the whole run (39/48 jobs cancelled), not specific to this one job or test file. No test was executed.
• Blocked work: Scheduled ROCm 7.2 AMD CI signal on main is not being produced — runs are cancelled before completing.
• In-flight fix: ❌ none found in open PRs — likely needs CI maintainer attention (concurrency/cron/runner-capacity), not a source code fix.

stage-c-test-large-8-gpu-amd-mi35x-rocm720 (linux-mi35x-gpu-8, 1) — failed step(s): N/A

Commit Info

• sglang: 43b1fb95
• aiter: N/A — the job was cancelled while queued and never reached the dependency-install step, so no aiter version was resolved (no [CI-AITER-CHECK] markers exist; the job log blob returns BlobNotFound). The sister extra-finish job confirms AITER_COMMIT_OVERRIDE was empty (Dockerfile default would apply had the job run).

Failed Tests

Test File	Test Function	Error	Log
N/A	N/A	Job cancelled before any test ran (0 steps executed)	Job

Failure Summary

The job stage-c-test-large-8-gpu-amd-mi35x-rocm720 (linux-mi35x-gpu-8, 1) did not fail a test — it was cancelled at 2026-06-22T19:05:27Z while still queued waiting for an 8-GPU MI35x runner (it recorded zero steps). The cancellation was triggered by GitHub Actions concurrency: the next scheduled run 27977093213 was created at 19:05:26Z and, sharing the same cancel-in-progress concurrency group, cancelled this in-progress run. No sglang or aiter code was exercised.

Regression Status

Infrastructure issue (concurrency cancellation) — not a test regression.

Recent scheduled runs of pr-test-amd-rocm720.yml (branch main, event=schedule, completed only):

Date (UTC)	Run	Conclusion	Note
Jun 22 19:05	27977093213	cancelled	superseded the run below
Jun 22 18:45	27975943807	cancelled	this run — target job cancelled
Jun 22 13:52	27957777753	cancelled
Jun 22 07:58	27938067935	cancelled
Jun 22 00:42	27922836455	cancelled
Jun 21 18:31	27913598623	cancelled

Every recent scheduled run resolves to cancelled, consistent with each new schedule superseding the prior in-progress run before the long-queued 8-GPU jobs can finish.

Failure Cluster

Concurrency cancellation of queued scheduled run (no test executed)

Facts (verified observations, no interpretation)

• Job 82793671971 conclusion = cancelled; steps: [] (empty) — never executed. Started 18:45:52Z, cancelled 19:05:27Z.
• Job log blob returns BlobNotFound (no log was ever produced — consistent with zero steps).
• All 39 cancellations in the run share the exact timestamp 2026-06-22T19:05:27Z.
• Next scheduled run 27977093213 created_at = 2026-06-22T19:05:26Z — 1s before the cancellations.
• Concurrency block (.github/workflows/pr-test-amd-rocm720.yml:122-123): group pr-test-amd-rocm720-refs/heads/main for schedule events with cancel-in-progress: true.
• Sibling stage-b-test-1-gpu-small-amd-mi35x-rocm720 82793671973 succeeded at 19:01:12Z — 1-GPU MI35x runners were available; the 8-GPU job was still queued.
• The two failure jobs (extra-finish 82797494542, pr-test-amd-rocm720-finish 82829194412) are gate jobs whose logic is if result==cancelled → exit 1; their log shows extra-a-test-1-gpu-small-amd: cancelled / The above jobs failed.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [FACT] The job was cancelled by the workflow's cancel-in-progress concurrency policy when the next scheduled run started, while it was still queued for an 8-GPU MI35x runner.
  • Supporting: next scheduled run created 19:05:26Z, all cancellations at 19:05:27Z; concurrency group + cancel-in-progress: true for schedule events; target job has zero steps.
  • Disconfirming: (none found) — the timing and config are unambiguous.
  • How to verify: compare created_at of 27977093213 vs the cancellation timestamps (already done).
• [MEDIUM] Contributing factor: 8-GPU MI35x runner starvation. The job queued ~20 min and never got a runner before the next schedule arrived; runs keep superseding each other so large-GPU scheduled jobs rarely complete.
  • Supporting: all 8 recent scheduled runs are cancelled; the 1-GPU MI35x sibling completed but the 8-GPU job did not; ~20 min queue with no steps.
  • Disconfirming: cannot confirm pure runner starvation vs. scheduling cadence overlap from this data alone; some 8-GPU MI325 jobs in the same run were also still queued when cancelled.
  • How to verify: inspect runner-pool availability/queue metrics for linux-mi35x-gpu-8 over the schedule window; check whether any recent scheduled run ever completed its 8-GPU stage-c jobs.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs:
  • curl -s -H "Authorization: token $GH_PAT" "https://api.github.com/search/issues?q=repo:sgl-project/sglang+is:pr+is:open+concurrency+rocm720" → no matching PR addressing scheduled-run concurrency cancellation.

Suggested Triage Steps

• Confirm this is expected concurrency behavior: two scheduled crons (30 17 * * * and 0 */6 * * *) plus GitHub schedule delays can overlap, and cancel-in-progress: true then kills the older run — by design for PR-style triggers but harsh for long-queued 8-GPU scheduled jobs.
• If full scheduled coverage of stage-c-test-large-8-gpu-amd-mi35x is desired, consider scoping the concurrency group to include the run id for schedule events (so scheduled runs don't cancel each other), or investigate 8-GPU MI35x runner capacity so these jobs dequeue before the next schedule fires.
• No code/test investigation is warranted for this specific job: no sglang/aiter code ran.

Status (factual, no priority assignment)

• Persistence: All 8 most recent scheduled runs (Jun 21–22) resolve to cancelled; this specific 8-GPU MI35x job never reached execution in this run.
• Scope: Run-wide — 39 jobs cancelled in 27975943807; 2 gate "finish" jobs reported failure as a consequence. Pattern repeats across consecutive scheduled runs.
• Blocked work: Scheduled 8-GPU MI35x stage-c CI signal is effectively not being collected on main (runs cancel before these jobs run).
• In-flight fix: ❌ none found in open PRs.

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-06-23 19:23 UTC)