[CI Monitor] Daily Report - 2026-06-24

[amd-bot]

Daily Cross-Workflow Summary — 2026-06-24

Snapshot: 2026-06-24 23:10 UTC · Only completed runs/jobs counted in trends · Auto-updated every 30 min

TL;DR

🔴 RED · ~13 active clusters · 5 🆕 today (R207–R211) · ~8 carrying over · 3 review-ready fixes (#29158, #27757, #28889)
👉 Today's ask: Good news first — the AMD AITER Scout is clean today (no failures; next cron ≈ Thu Jun-25) and release-docker-amd-nightly is ✅ green. The single highest-value merge is 🆕 #29158 (open, non-draft) — a direct ROCm fallback that fixes the R195 mamba extra_buffer assertion now red on 4 Qwen3.5 nightly jobs across both nightlies; land it plus the two other review-ready fixes #27757 (R2 Mistral) and #28889 (R192 FLUX.2 FP8). Then triage the 5 NEW clusters (R207–R211, none have a fix). ⚠️ pr-test-amd-rocm720 produced ZERO real signal again — all 88 "failures" are the recurring cron self-cancellation (draft fix #28987). Both nightlies' Jun-24 runs are partially IN-FLIGHT.

Workflow status

Workflow	Latest run	✅	❌	Trend (completed real failures)	Δ vs yesterday
nightly-test-amd	Jun-23 28046906309 (10); Jun-24 28119497313 (IN-FLIGHT)	0	10 (+4 partial)	11·9·7·10·(IN-FLIGHT)	+3 (worse, full run completed)
nightly-test-amd-rocm720	Jun-23 28046795462 (9); Jun-24 28119401197 (7)	0	9 → 7	12·11·9·7	-2 (slightly better)
release-docker-amd-nightly	Jun-24 (latest)	✅	0	0·0·0	0
release-docker-amd-rocm720-nightly	Jun-24 28099472424	⚠️	2 (infra)	0·0·2	+2 (1 leg cancelled)
amd-aiter-scout	— (no run/failures today)	—	0	—	scout clean; next ≈ Thu Jun-25
pr-test-amd	rolling (Jun-23→24)	0	~6–8/run	~6	~same
pr-test-amd-rocm720	28119463964 + others	0	0 real (82 concurrency-cancel · 6 infra)	0·0·0 real	no signal (cron churn)

Notes: (1) nightly Jun-24 runs (28119497313, 28119401197) reported only their early 2-GPU + some 8-GPU jobs at snapshot — late shards are partial. (2) release-docker-amd-rocm720-nightly: gfx942 publish leg cancelled mid-build (runner reclamation, [MEDIUM]) → push_local_registry failed on missing artifact; gfx950 leg succeeded. One-off infra, re-run to recover. (3) pr-test-amd-rocm720's two crons (30 17 + 0 */6) still share one cancel-in-progress group → 82 self-cancellations; draft fix #28987.

Failure clusters (active, deduplicated across all workflows)

R195 · Mamba extra_buffer needs CUDA/MUSA/NPU (FLA) rejected on ROCm (Qwen3.5) — 4 nightly jobs, both nightlies

• Status: persistent since ~Jun-19 on all four Qwen3.5 nightly jobs. Top hypothesis: [HIGH] #28151 (66a7fd5c) set mamba_radix_cache_strategy="auto" and removed the ROCm fallback. In-flight fix: ✅ #29158 (open, non-draft, NEW) — direct ROCm no_buffer fallback; land this. (Also #27141 open, enables the feature.)

Workflow	Job (shard)	Test File	Test Function	Error	Log
nightly-test-amd	nightly-8-gpu-mi35x-qwen35	test_qwen35_eval_mi35x.py	test_lm_eval	extra_buffer needs CUDA/MUSA/NPU (FLA)	Log
nightly-test-amd	nightly-8-gpu-qwen35	test_qwen35_eval_amd.py	setUpClass	same	Log
nightly-test-amd-rocm720	nightly-8-gpu-mi35x-qwen35-rocm720	test_qwen35_eval_mi35x.py	test_lm_eval	same	Log
nightly-test-amd-rocm720	nightly-8-gpu-qwen35-rocm720	test_qwen35_eval_amd.py	setUpClass	same	Log

R2 · Mistral/Mixtral GSM8K below threshold (chat-eval) — both nightlies

• Status: never-passed in window (≥Jun-13); Mistral-7B ~0.36–0.37, Mixtral ~0.29–0.32 vs thresholds 0.47/0.69. Top hypothesis: [HIGH] chat-eval answer-extraction + threshold calibration, not a kernel regression. In-flight fix: ✅ #27757 (open, non-draft) — land it.

Workflow	Job (shard)	Test File	Test Function	Error	Log
nightly-test-amd	nightly-accuracy-2-gpu (IN-FLIGHT)	test_gsm8k_eval_amd.py	test_gsm8k_all_models	Mistral 0.366 < 0.47	Log
nightly-test-amd-rocm720	nightly-accuracy-2-gpu-rocm720	test_gsm8k_eval_amd.py	test_gsm8k_all_models	Mistral 0.369 < 0.47	Log

R1 · VLM MMMU accuracy below threshold (5 models, ~random) — both nightlies

• Status: never-passed (≥Jun-13); same 5 VLMs (Qwen2-VL, Qwen2.5-VL, Qwen3-VL-30B, MiniCPM-o, deepseek-vl2) ~0.26–0.29, bit-stable. Top hypothesis: [MEDIUM] systemic VLM degradation / thresholds calibrated above achievable on MI30x. In-flight fix: ❌ none (#26178 stale, perf-only).

Workflow	Job (shard)	Test File	Test Function	Error	Log
nightly-test-amd	nightly-accuracy-2-gpu-vlm	test_vlms_mmmu_eval_amd.py	test_mmmu_vlm_models	5 models < thr	Log
nightly-test-amd-rocm720	nightly-accuracy-2-gpu-vlm-rocm720	test_vlms_mmmu_eval_amd.py	test_mmmu_vlm_models	5 models < thr	Log

R192 · FLUX.2 modelopt-FP8 torch._scaled_mm HIPBLAS_STATUS_NOT_SUPPORTED — pr-test-amd (8 jobs, has fix)

• Status: never-passed on AMD; every multimodal-gen-test-2-gpu-amd shard-1 run (flux2). Top hypothesis: [HIGH] FP8 e4m3fn GEMM unsupported by hipBLAS on gfx942. In-flight fix: ✅ #28889 (open, non-draft) — dequant→bf16 fallback in apply_fp8_linear; land it.

Workflow	Job (shard)	Test File	Test Function	Error	Log
pr-test-amd	multimodal-gen-test-2-gpu-amd (1)	test_server_2_gpu.py	test_diffusion_generation[flux2_modelopt_fp8_tp2_t2i]	HIPBLAS_STATUS_NOT_SUPPORTED	Log

R202 · PD-disaggregation KV-transfer 300s timeout / RDMA mem-registration fault — pr-test-amd MI35x (8 jobs)

• Status: persistent ≥6 days on stage-b-test-large-8-gpu-mi35x-disaggregation-amd (KVPoll.WaitingForInput 300s → KeyError: 'answer'; PP setUpClass exit -9, RDMA Cannot allocate memory [12]). Top hypothesis: [MEDIUM] faulty/resource-exhausted RDMA NIC rdma5 on the fabric runner; [LOW] sglang regression. In-flight fix: ⚠️ #29127 (DRAFT, DEBUG-only) — isolates the faulty NIC, not a code fix.

Workflow	Job (shard)	Test File	Test Function	Error	Log
pr-test-amd	stage-b-test-large-8-gpu-mi35x-disaggregation-amd	test_disaggregation_basic.py	TestDisaggregationAccuracy.test_gsm8k	KVTransfer 300s → KeyError	Log
pr-test-amd	stage-b-test-large-8-gpu-mi35x-disaggregation-amd	test_disaggregation_pp.py	TestDisaggregationDecodePPAccuracy.setUpClass	exit -9, RDMA ENOMEM [12]	Log

R19 · Qwen3-235B-MXFP4 HIP hipErrorCapturedEvent abort during decode CUDA-graph capture — MI35x, both nightlies

• Status: never-passed (≥May-25); NCCL watchdog hipEventQuery races with graph capture under aiter custom all-reduce. Top hypothesis: [HIGH] ProcessGroupNCCL watchdog polls events on a capturing stream → illegal on ROCm. In-flight fix: ❌ none direct (⚠️ #27650, #23581 candidates).

Workflow	Job (shard)	Test File	Test Function	Error	Log
nightly-test-amd	nightly-8-gpu-mi35x-qwen3-235b-mxfp4	test_qwen3_instruct_mxfp4.py	setUpClass	hipErrorCapturedEvent → -6	Log
nightly-test-amd-rocm720	nightly-8-gpu-mi35x-qwen3-235b-mxfp4-rocm720	test_qwen3_instruct_mxfp4.py	setUpClass	same	Log

R155 · DeepSeek-V3.2 (basic) MI35x GSM8K below threshold — both nightlies

• Status: never-passed on rocm720 (0.85–0.87 vs 0.93); nightly-test-amd borderline 0.925 (new today, 1-question short). Top hypothesis: [MEDIUM] tilelang DSA backend numerics on gfx950 (MTP variant passes). In-flight fix: ❌ none (related #25559, #29050).

Workflow	Job (shard)	Test File	Test Function	Error	Log
nightly-test-amd	nightly-accuracy-8-gpu-mi35x-deepseek-v32	test_deepseek_v32_eval_mi35x.py	test_deepseek_v32_accuracy	0.925 < 0.93	Log
nightly-test-amd-rocm720	nightly-accuracy-8-gpu-mi35x-deepseek-v32-rocm720	test_deepseek_v32_eval_mi35x.py	test_deepseek_v32_accuracy	0.850 < 0.93	Log

R211 · 🆕 DeepSeek-R1 HiCache MI35x — GPU memory access fault / GSM8K timeout — both nightlies

• Status: failing since ~Jun-20, surfaced today (8-GPU jobs completed). Two symptoms: nightly-test-amd + rocm720 Jun-23 = Memory access fault by GPU → -6/255; rocm720 Jun-24 = per-file 3600s timeout (est_time 5400s > 3600s budget). Top hypothesis: [LOW] illegal access in hicache fp8-KV / aiter block-scale GEMM (gfx950); plus [HIGH] structural timeout misconfig for the eval. In-flight fix: ❌ none.

Workflow	Job (shard)	Test File	Test Function	Error	Log
nightly-test-amd	nightly-8-gpu-mi35x-deepseek-r1-hicache	test_deepseek_r1_hicache_mi35x.py	test_gsm8k	mem access fault → 3600s timeout	Log
nightly-test-amd-rocm720	nightly-8-gpu-mi35x-deepseek-r1-hicache-rocm720	test_deepseek_r1_hicache_mi35x.py	TestDeepSeekR1HiCacheMI35x.test_gsm8k	TIMEOUT 3600s (63% done)	Log

R209 · 🆕 VLM MMMU benchmark 1800s timeout (Qwen2.5-VL-3B) — pr-test-amd nondeterministic

• Status: NEW today; model swapped to Qwen2.5-VL-3B → untuned vision GEMMs exceed 1800s budget. Top hypothesis: [HIGH] model-swap #29095 (MERGED) larger GEMMs (torch fallback) over budget. In-flight fix: ❌ none (raise timeout or tune GEMMs).

Workflow	Job (shard)	Test File	Test Function	Error	Log
pr-test-amd	stage-b-test-1-gpu-small-amd-nondeterministic	test_vlm_models.py	TestVLMModels.test_vlm_mmmu_benchmark (Qwen2.5-VL-3B)	TIMEOUT 1800s	Log

R207 · 🆕 GLM-5.1-FP8 weight-load stall (MI35x) — nightly-test-amd

• Status: NEW regression today (passed prior 3 runs; Jun-22 acc 0.955). Server never ready; TP7 weight load hung past 5411s budget (distributed hang at post-load monitored_barrier). Top hypothesis: [MEDIUM] checkpoint I/O / load slowdown (infra-leaning). In-flight fix: ❌ none.

Workflow	Job (shard)	Test File	Test Function	Error	Log
nightly-test-amd	nightly-8-gpu-mi35x-glm51	test_glm51_eval_mi35x.py	test_glm51_accuracy	weight-load stall, server never ready	Log

R208 · 🆕 VLM perf server-side KeyError (premature HTTP response) — pr-test-amd-rocm720 perf

• Status: NEW regression Jun-24 (passed prior 4 runs); Qwen3-VL-30B benchmark aborted by server KeyError in _wait_one_response → client ChunkedEncodingError. Top hypothesis: [LOW] zmq IPC refactor #29012 (34dd9c28, MERGED) altered output delivery → duplicate-output/deleted-state race. Disconfirming: only one occurrence; could be flake. In-flight fix: ❌ none.

Workflow	Job (shard)	Test File	Test Function	Error	Log
nightly-test-amd-rocm720	nightly-perf-2-gpu-vlm-rocm720	test_vlms_perf_amd.py	test_bench_one_batch	server KeyError → premature response	Log

R210 · 🆕 Qwen3.5 triton-DCP GSM8K eval timeout — nightly-test-amd

• Status: NEW (file test_qwen3p5_triton_dcp.py added by #25090, first scheduled run); full 1314-sample GSM8K (est_time 4800s) doesn't fit 3600s timeout, stalled at 46% on repeated Connection error. Top hypothesis: [MEDIUM] timeout-budget misconfig. In-flight fix: ❌ none.

Workflow	Job (shard)	Test File	Test Function	Error	Log
nightly-test-amd	nightly-8-gpu-mi35x-qwen35	test_qwen3p5_triton_dcp.py	TestQwen35TritonDCPGsm8k.test_a_gsm8k	TIMEOUT 3600s	Log

Known/stable + resolved clusters (carrying over) · click to expand

ID	Cluster	Where (latest)	In-flight fix
R6	Qwen3-30B-A3B unquantized MoE accuracy 0.305 (MI35x)	rocm720 accuracy-8gpu-mi35x	❌ none
R203	Qwen2-57B-FP8 server-start TimeoutError (TP2)	nightly-test-amd accuracy-2-gpu (Jun-23 only)	❌ none
R181	mori build invalid-ELF (libabsl .so)	pr-test-amd stage-a-1gpu-small + rocm720	❌ none (infra)
—	ideogram-4-fp8 gated-repo 403 at diffusion setup	pr-test-amd multimodal-gen-2gpu	⚠️ #28225 (draft)
—	RCCL allreduce SIGSEGV 8-GPU smoke (test_rccl_multi_gpu.py)	pr-test-amd stage-c-8gpu (0)	❌ none
R200	DeepSeek-V4 compress-state KV HIP OOM	quiet today	✅ #28941 MERGED
R196	VLM DP-encoder CUDA-IPC invalid device pointer	quiet today (8-GPU shards partial)	❌ none
R194	Kimi-K2.6 MoE w2 narrow IndexError	did NOT recur — today Kimi is download timeout (infra)	⚠️ #28905 (draft)

Infrastructure / orchestration noise (not test failures) · click to expand

• pr-test-amd-rocm720 cron self-cancellation: 82 of 88 jobs cancelled via cancel-in-progress — two crons (30 17 daily + 0 */6) share one concurrency group on main. Two mass events: run 28046873327 (40 jobs) and 28119463964 (~39 jobs). Draft fix ⚠️ #28987. This workflow produced 0 real test signal today.
• HF weight-download hangs (xet_get): Kimi-K2.6 (MI35x both nightlies), DeepSeek-V4-Pro-FP8 (rocm720, alternates pass/cancel on timing), deepseek-vl2-small, grok1-int4. Infra/cache, not code.
• Self-hosted runner lost-communication / abrupt termination (BlobNotFound): pr-test-amd stage-c-8gpu + multimodal-gen 1-GPU shards (jobs lost log).
• release-docker-amd-rocm720-nightly: gfx942 publish leg cancelled mid-build (runner reclamation [MEDIUM]) → push_local_registry artifact-missing cascade. Re-run 28099472424 to recover.
• Diffusion 1-GPU flakes: FLUX.1-dev 90-min hang, zimage warmup hang → port-5555 leak cascade, priority-scheduling decode-capture hang — assorted one-offs.
• Grok1-FP8 decode-capture hang (nightly-accuracy-8-gpu, new today, 1 occurrence, infra flake).

Workflow drill-down (per-workflow view)

nightly-test-amd · Jun-23 [28046906309](https://github.com/sgl-project/sglang/actions/runs/28046906309) (10) · Jun-24 [28119497313](https://github.com/sgl-project/sglang/actions/runs/28119497313) IN-FLIGHT

Job (shard)	Test File	Test Function	Cluster	Error
nightly-accuracy-2-gpu	test_gsm8k_eval_amd.py	test_gsm8k_all_models	R2 / R203	Mistral < thr (+ Qwen2-57B server-start, Jun-23)
nightly-accuracy-2-gpu-vlm	test_vlms_mmmu_eval_amd.py	test_mmmu_vlm_models	R1	5 VLMs < thr
nightly-8-gpu-mi35x-qwen35	test_qwen35_eval_mi35x.py / test_qwen3p5_triton_dcp.py	test_lm_eval / test_a_gsm8k	R195 / R210🆕	extra_buffer assert + DCP timeout
nightly-8-gpu-qwen35	test_qwen35_eval_amd.py	setUpClass	R195	extra_buffer assert
nightly-8-gpu-mi35x-qwen3-235b-mxfp4	test_qwen3_instruct_mxfp4.py	setUpClass	R19	hipErrorCapturedEvent -6
nightly-accuracy-8-gpu-mi35x-deepseek-v32	test_deepseek_v32_eval_mi35x.py	test_deepseek_v32_accuracy	R155	0.925 < 0.93
nightly-8-gpu-mi35x-deepseek-r1-hicache	test_deepseek_r1_hicache_mi35x.py	test_gsm8k	R211🆕	mem fault → timeout
nightly-8-gpu-mi35x-glm51	test_glm51_eval_mi35x.py	test_glm51_accuracy	R207🆕	weight-load stall
nightly-8-gpu-mi35x-kimi-k26	test_kimi_k26_eval_mi35x.py	test_kimi_k26_gsm8k_accuracy	infra	HF xet download 7200s timeout
nightly-accuracy-8-gpu	test_grok1_fp8_eval_amd.py	(server never ready)	infra	Grok1-FP8 capture hang (new)

nightly-test-amd-rocm720 · Jun-23 [28046795462](https://github.com/sgl-project/sglang/actions/runs/28046795462) (9) · Jun-24 [28119401197](https://github.com/sgl-project/sglang/actions/runs/28119401197) (7)

Job (shard)	Test File	Test Function	Cluster	Error
nightly-accuracy-2-gpu-rocm720	test_gsm8k_eval_amd.py	test_gsm8k_all_models	R2	Mistral/Mixtral < thr
nightly-accuracy-2-gpu-vlm-rocm720	test_vlms_mmmu_eval_amd.py	test_mmmu_vlm_models	R1	5 VLMs < thr
nightly-perf-2-gpu-vlm-rocm720	test_vlms_perf_amd.py	test_bench_one_batch	R208🆕	server KeyError
nightly-8-gpu-mi35x-qwen35-rocm720 + qwen35-rocm720	test_qwen35_eval_*.py	test_lm_eval/setUpClass	R195	extra_buffer assert
nightly-8-gpu-mi35x-qwen3-235b-mxfp4-rocm720	test_qwen3_instruct_mxfp4.py	setUpClass	R19	hipErrorCapturedEvent -6
nightly-accuracy-8-gpu-mi35x-deepseek-v32-rocm720	test_deepseek_v32_eval_mi35x.py	test_deepseek_v32_accuracy	R155	0.850 < 0.93
nightly-accuracy-8-gpu-mi35x-rocm720	test_qwen3_moe_eval_mi35x.py	test_qwen3_moe_accuracy	R6	0.305 < 0.75
nightly-8-gpu-mi35x-deepseek-r1-hicache-rocm720	test_deepseek_r1_hicache_mi35x.py	test_gsm8k	R211🆕	mem fault / 3600s timeout
nightly-8-gpu-mi35x-kimi-k26-rocm720 / dsv4-pro	download	setUpClass	infra	HF xet stall

pr-test-amd · rolling Jun-23→24 · 7 clusters (28 jobs)

Job (shard)	Test File	Test Function	Cluster	Error
multimodal-gen-2gpu (1) ×8	test_server_2_gpu.py	flux2_modelopt_fp8_tp2_t2i	R192	HIPBLAS_STATUS_NOT_SUPPORTED
stage-b-mi35x-disaggregation ×8	test_disaggregation_*.py	test_gsm8k / setUpClass	R202	KV-transfer 300s / RDMA ENOMEM
stage-b-1gpu-small-nondeterministic ×2	test_vlm_models.py	test_vlm_mmmu_benchmark	R209🆕	1800s timeout (Qwen2.5-VL-3B)
multimodal-gen-2gpu (1) ×4	test_server_2_gpu.py	ideogram4_fp8_tp2_t2i	ideogram-403	gated repo 403
stage-a-1gpu-small ×2	mori build	N/A	R181	invalid ELF header
runner-lost / RCCL / flux1-hang / port-5555 / priority-sched-hang	various	N/A	infra	runner loss / SIGSEGV / hangs

pr-test-amd-rocm720 · [28119463964](https://github.com/sgl-project/sglang/actions/runs/28119463964) + others · 0 real failures

All 88 "failures" are non-substantive: 82 concurrency cancel-in-progress cancellations (two crons share one group on main) + 6 infra (4 DeepSeek-V4-Pro slow weight-load/timeout cancellations, 1 mori libabsl_time_zone.so invalid-ELF build fail, runner starvation). Zero failing pytest test file+function; nothing to bisect. Draft CI fix ⚠️ #28987 targets the cron collision — the top item to restore ROCm 7.2 scheduled signal.

How this report is generated

• Only status == "completed" runs/jobs counted in trends. Both nightly Jun-24 runs are partly IN-FLIGHT (late shards pending); their carry-over 8-GPU clusters are confirmed from the Jun-23 completed runs. amd-aiter-scout had no run/failures today (next cron ≈ Thu Jun-25). release-docker-amd-nightly ✅ green.
• 🆕 NEW today (R207–R211): R207 GLM-5.1 weight-load stall, R208 VLM perf KeyError (zmq race), R209 VLM MMMU 1800s timeout (Qwen2.5-VL-3B), R210 Qwen3.5 triton-DCP timeout, R211 DeepSeek-R1 HiCache mem-fault/timeout (failing since ~Jun-20, surfaced now). R200 resolved (#28941 merged); R194 did not recur (Kimi is download-hang today, not the w2 narrow IndexError).
• PR states verified live: #29158 / #27757 / #28889 / #27141 open + non-draft; #28987 / #29127 draft; #29012 / #29095 merged.
• Confidence: FACT/HIGH/MEDIUM/LOW/SPECULATION. Bot does NOT assign Priority — engineers decide from cluster size + persistence + fix availability.

---

Generated by amd-bot · last updated 2026-06-24 23:10 UTC

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-06-24 23:10 UTC)

---

CI Monitor — 2026-06-24

Repo: sgl-project/sglang

Monitored Workflows:

• nightly-test-amd.yml
• nightly-test-amd-rocm720.yml
• release-docker-amd-nightly.yml
• release-docker-amd-rocm720-nightly.yml
• amd-aiter-scout.yml
• pr-test-amd.yml
• pr-test-amd-rocm720.yml

Per-workflow failure reports are appended as comments below; the Daily Cross-Workflow Summary is rendered above this section.

[amd-bot]

nightly-test-amd-rocm720.yml — 16 failure(s)

Run started (UTC): 2026-06-23 18:43 UTC
Last scanned (UTC): 2026-06-24 23:07 UTC
Commits: sglang d6aacd2

Summary

Counts: 16 failures · 10 clusters · 9 clusters carrying over (≥3 days) · 1 new today

#	Cluster	Job	Test File	Test Function	Status	Hypothesis (confidence)
1	HF weight download stall/timeout	nightly-8-gpu-mi35x-kimi-k26-rocm720	test/registered/amd/accuracy/mi35x/test_kimi_k26_eval_mi35x.py	test_kimi_k26_gsm8k_accuracy	persistent ≥4d (last green Jun 15)	[MEDIUM] HF xet_get/filelock download exceeds budget on mi35x runner (infra)
2	HF weight download stall/timeout	nightly-8-gpu-mi35x-kimi-k26-rocm720	test/registered/amd/accuracy/mi35x/test_kimi_k26_eval_mi35x.py	test_kimi_k26_gsm8k_accuracy	persistent since Jun 20	[MEDIUM] mi35x download throughput (sister non-mi35x job passes)
3	HF weight download stall/timeout	nightly-8-gpu-mi35x-deepseek-v4-pro-rocm720	test/registered/amd/test_deepseek_v4_pro_fp8.py	setUpClass	flaky (Jun 22+24 fail, Jun 20-21 green)	[HIGH] cache-wipe → xet_get re-download stall → 6h cancel
4	HF weight download stall/timeout	nightly-accuracy-2-gpu-rocm720	test/registered/amd/accuracy/mi30x/test_gsm8k_eval_amd.py	test_gsm8k_all_models	NEW Jun 23 (Qwen2-57B startup)	[MEDIUM] Qwen2-57B HF download hang (also persistent Mistral/Mixtral, see C3)
5	DeepSeek-V3.2 GSM8K acc < threshold (tilelang DSA)	nightly-accuracy-8-gpu-mi35x-deepseek-v32-rocm720	test/registered/amd/accuracy/mi35x/test_deepseek_v32_eval_mi35x.py	test_deepseek_v32_accuracy	never-passed ≥12d (Jun 12+)	[MEDIUM] tilelang DSA backend numerics on gfx950 (MTP variant passes)
6	DeepSeek-V3.2 GSM8K acc < threshold (tilelang DSA)	nightly-accuracy-8-gpu-mi35x-deepseek-v32-rocm720	test/registered/amd/accuracy/mi35x/test_deepseek_v32_eval_mi35x.py	test_deepseek_v32_accuracy	never-passed (0.850/0.93)	[MEDIUM] tilelang DSA path accuracy degradation
7	VLM MMMU accuracy near-random floor	nightly-accuracy-2-gpu-vlm-rocm720	test/registered/amd/accuracy/mi30x/test_vlms_mmmu_eval_amd.py	test_mmmu_vlm_models	never-passed ≥3d (5 VLMs ~0.28)	[MEDIUM] systemic VLM degradation on ROCm 7.2 (whole suite ~random)
8	VLM MMMU accuracy near-random floor	nightly-accuracy-2-gpu-vlm-rocm720	test/registered/amd/accuracy/mi30x/test_vlms_mmmu_eval_amd.py	test_mmmu_vlm_models	never-passed Jun 20-24 (5/11 fail)	[MEDIUM] shared multimodal path near-random across all VLMs
9	Mamba extra_buffer radix-cache assert on ROCm	nightly-8-gpu-mi35x-qwen35-rocm720	test/registered/amd/accuracy/mi35x/test_qwen35_eval_mi35x.py	test_lm_eval	regression since Jun 19 (5d)	[HIGH] 66a7fd5c (#28151) removed ROCm no_buffer fallback
10	Mamba extra_buffer radix-cache assert on ROCm	nightly-8-gpu-qwen35-rocm720	test/registered/amd/accuracy/mi30x/test_qwen35_eval_amd.py	TestQwen35EvalAMD.setUpClass	never-passed (same assert)	[HIGH] extra_buffer gated to CUDA/MUSA/NPU; fires on ROCm
11	HIP error during decode CUDA graph capture	nightly-8-gpu-mi35x-qwen3-235b-mxfp4-rocm720	test/registered/amd/test_qwen3_instruct_mxfp4.py	TestQwen3Instruct2507MXFP4.setUpClass	never-passed ~30d (May 25+)	[HIGH] NCCL watchdog polls events mid-capture → hipErrorCapturedEvent → exit -6
12	GPU mem access fault — DeepSeek-R1 fp8 prefill GEMM	nightly-8-gpu-mi35x-deepseek-r1-hicache-rocm720	test/registered/amd/test_deepseek_r1_hicache_mi35x.py	(benchmark phase)	never-passed ≥12d (Jun 12+)	[LOW] aiter fp8 block-scale GEMM mem fault — no candidate ≥ MEDIUM
13	Per-file timeout — DeepSeek-R1 HiCache GSM8K eval	nightly-8-gpu-mi35x-deepseek-r1-hicache-rocm720	test/registered/amd/test_deepseek_r1_hicache_mi35x.py	test_gsm8k	never-passed since Jun 20	[HIGH] --timeout-per-file 3600 < test est_time=5400
14	Mistral/Mixtral GSM8K chat-eval < threshold	nightly-accuracy-2-gpu-rocm720	test/registered/amd/accuracy/mi30x/test_gsm8k_eval_amd.py	test_gsm8k_all_models	persistent ≥5d (~0.37)	[HIGH] chat-eval extraction + threshold miscalibration (#27757)
15	Qwen3-30B-A3B GSM8K acc < threshold	nightly-accuracy-8-gpu-mi35x-rocm720	test/registered/amd/accuracy/mi35x/test_qwen3_moe_eval_mi35x.py	test_qwen3_moe_accuracy	4/5 fail (1 marginal pass Jun 23)	[LOW] unstable triton+aiter MoE numerics — no candidate ≥ MEDIUM
16	Server-side KeyError in batched-request handler	nightly-perf-2-gpu-vlm-rocm720	test/registered/amd/perf/mi30x/test_vlms_perf_amd.py	test_bench_one_batch	NEW Jun 24 (1/5, green prior 4)	[LOW] zmq IPC refactor 34dd9c28 vs deleted-state race

Cluster: HF weight download stall/timeout (filelock/xet)

• Affected: Kimi-K2.6 mi35x (rows 1-2), DeepSeek-V4-Pro FP8 (row 3), Qwen2-57B in 2-gpu accuracy (row 4).
• Shared facts: all park in download_weights_from_hf → filelock _poll_until_acquired / xet_get; no Load weight end; soft watchdog loops to the per-file/job timeout. Non-mi35x Kimi sister job passed Jun 23, and the V4-Pro repo loaded earlier in the same run — points to runner/network, not the artifact.
• Top hypothesis: [MEDIUM] HF download throughput/cache on mi35x runners (xet); [HIGH] for V4-Pro the cache-validation wipe forces a fatal re-download. Disconfirming: no byte-rate line proving advance vs hard stall.
• In-flight fix: ❌ none (nearest #23400 HF resilience, insufficient).
• Suggested triage: confirm mi35x persists/pre-seeds HF cache; time standalone huggingface-cli download; for V4-Pro avoid full-cache deletion in ci_weight_validation.py.

Cluster: DeepSeek-V3.2 GSM8K accuracy below threshold (tilelang DSA)

• Affected: rows 5-6 (same job, two runs). Score band 0.850-0.870 vs 0.93; MTP variant passes.
• Top hypothesis: [MEDIUM] tilelang DSA prefill/decode numerics on gfx950. Disconfirming: MTP differs by more than backend; no non-tilelang A/B.
• In-flight fix: ❌ none (#29050, #27705 DSA-path, not confirmed).
• Suggested triage: re-run basic config with non-tilelang DSA backend; recover authoring-PR reference score.

Cluster: VLM MMMU accuracy near-random floor

• Affected: rows 7-8 (same job, two runs). All 11 VLMs cluster 0.26-0.34 (random ≈0.25), stable day-over-day.
• Top hypothesis: [MEDIUM] systemic multimodal degradation on MI30x/ROCm 7.2. Disconfirming: 6 models still "pass"; no green baseline to prove regression vs always-tight thresholds.
• In-flight fix: ❌ none (#26178 WIP, mi300-scoped, doesn't touch this file).
• Suggested triage: inspect raw generations for one failing model; cross-check same model on NVIDIA nightly.

Cluster: Mamba extra_buffer radix-cache assert on ROCm

• Affected: Qwen3.5 mi35x (row 9) + 8-gpu (row 10). AssertionError: extra_buffer needs CUDA/MUSA/NPU (FLA) at server_args.py:4386.
• Top hypothesis: [HIGH] 66a7fd5c (#28151) removed the ROCm no_buffer fallback; auto now resolves to extra_buffer on ROCm. Boundary aligns exactly (Jun 18 pass → Jun 19 fail). Disconfirming: (none found).
• In-flight fix: ✅ #29158 (fall back to no_buffer on ROCm, open Jun 24) — chase reviewers; alt #27141.
• Suggested triage: verify #29158 clears the assert on both jobs.

Cluster: HIP error during decode CUDA graph capture (NCCL watchdog)

• Affected: row 11. hipErrorCapturedEvent from ProcessGroupNCCL watchdog while main thread in graph_capture; exit -6. 8/8 runs May 25-Jun 22.
• Top hypothesis: [HIGH] watchdog hipEventQuery during HIP capture is illegal → abort. Disconfirming: exact timing condition not pinned. Not from current commit (predates ~1 month).
• In-flight fix: ❌ none.
• Suggested triage: re-run with TORCH_NCCL_ENABLE_MONITORING=0 / --disable-cuda-graph / --disable-custom-all-reduce one at a time.

Cluster: GPU memory access fault — DeepSeek-R1 fp8 prefill GEMM

• Affected: row 12 (Jun 23 run). 8× Memory access fault in aiter_w8a8_block_fp8_linear → gemm_a8w8_blockscale on large-M prefill; exit -6 → timeout.
• Top hypothesis: [LOW] aiter fp8 block-scale GEMM fault (log-proximity ≠ causation; aiter unchanged across window). No candidate ≥ MEDIUM.
• In-flight fix: ❌ none; debug branch bingxche/debug-dsv32-mi35x-mtp-mem-access-fault exists (not a PR).
• Suggested triage: capture GPU coredump (HSA_ENABLE_COREDUMP=1); CK-vs-triton GEMM A/B; hicache on/off A/B.

Cluster: Per-file timeout — DeepSeek-R1 HiCache GSM8K eval

• Affected: row 13 (Jun 24 run, same job name as row 12 but distinct symptom: clean run killed at 3600s @63%).
• Top hypothesis: [HIGH] --timeout-per-file 3600 < test est_time=5400; siblings use 7200. Disconfirming: est_time is an estimate; no green baseline proves 7200 suffices.
• In-flight fix: ❌ none.
• Suggested triage: re-run with --timeout-per-file 7200; profile ~40min startup + eval latency.

Cluster: Mistral/Mixtral GSM8K chat-eval below threshold

• Affected: row 14 (Mistral 0.369/0.47); also the persistent component of row 4's job. Stable ~0.37 across 5 runs, aiter unchanged.
• Top hypothesis: [HIGH] chat-completion eval extraction + 5-shot-derived threshold mismatch. Disconfirming: PR author notes part of gap may be genuine (off-FA3 alone didn't recover).
• In-flight fix: ✅ #27757 (open Jun 10) — chase reviewers.
• Suggested triage: verify #27757 lifts Mistral/Mixtral over threshold; NVIDIA A/B for residual gap.

Cluster: Qwen3-30B-A3B GSM8K accuracy below threshold

• Affected: row 15. 0.305/0.75; swings 0.305→0.765 run-to-run (one marginal pass Jun 23).
• Top hypothesis: [LOW] unstable unquantized triton+aiter MoE numerics on gfx950. No candidate ≥ MEDIUM.
• In-flight fix: ❌ none.
• Suggested triage: re-run ~5× for variance; SGLANG_USE_AITER=0 vs =1 A/B; inspect for INVALID-parsed completions.

Cluster: Server-side KeyError in batched-request handler

• Affected: row 16. KeyError in _wait_one_response (tokenizer_manager.py:1435) → premature HTTP response; 144 "state was deleted" lines; only Qwen3-VL-30B (tp=2) tripped it.
• Top hypothesis: [LOW] zmq IPC refactor 34dd9c28 (#29012) altered output timing, or pre-existing race. No candidate ≥ MEDIUM. Disconfirming: failing code path unchanged; perf job green 4/4 prior.
• In-flight fix: ❌ none.
• Suggested triage: re-run same SHA 2-3× (regression vs flake); bisect ed26a109ee..d6aacd2801 starting at 34dd9c28.

Suggested triage order

Rank	Triage step	Targets	Why this first
1	Verify #29158 merges & re-run both qwen35 jobs	C4 (rows 9-10)	HIGH-confidence root cause + in-flight fix ready; clears a 5-day regression
2	Chase #27757; re-run Mistral/Mixtral subtests	C3 (row 14) + row 4 persistent part	HIGH-confidence, fix already open
3	Fix mi35x HF cache seeding / xet download (and V4-Pro cache-wipe)	C5 (rows 1-4)	Largest cluster (4 jobs); infra fix unblocks several jobs at once
4	Re-run DeepSeek-R1 HiCache with --timeout-per-file 7200	C8 (row 13)	HIGH-confidence config fix, single-line workflow change
5	Toggle NCCL watchdog/cuda-graph on Qwen3-235B mxfp4	C6 (row 11)	HIGH-confidence interaction; longest-running (~30d)
6	Backend A/B (tilelang DSA, VLM path, Qwen3 MoE) + coredump for R1 mem fault	C2, C1, C10, C7 (rows 5-8,12,15)	Accuracy/mem-fault clusters need GPU repro to move past MEDIUM/LOW
7	Same-SHA re-run + bisect from 34dd9c28	C9 (row 16)	New today; classify regression vs flake cheaply

---

Job	Failed Steps	Started
nightly-accuracy-2-gpu-vlm-rocm720	Nightly Accuracy Test ROCm 7.2 (2-GPU VLM MMMU)	2026-06-23T18:58
nightly-accuracy-8-gpu-mi35x-deepseek-v32-rocm720	Accuracy Test MI35x ROCm 7.2 (8-GPU DeepSeek-V3.2)	2026-06-23T23:25
nightly-accuracy-2-gpu-rocm720	Nightly Test ROCm 7.2 (2-GPU)	2026-06-23T18:43
nightly-8-gpu-mi35x-qwen3-235b-mxfp4-rocm720	Accuracy Test + Performance Test MI35x ROCm 7.2 (8-GPU Qwen3-235B-MXFP4)	2026-06-24T00:25
nightly-8-gpu-mi35x-qwen35-rocm720	Accuracy Test MI35x ROCm 7.2 (8-GPU Qwen 3.5)	2026-06-24T00:40
nightly-8-gpu-mi35x-deepseek-r1-hicache-rocm720	Accuracy Test MI35x ROCm 7.2 (8-GPU DeepSeek-R1 HiCache)	2026-06-24T00:14
nightly-8-gpu-mi35x-kimi-k26-rocm720	Accuracy Test MI35x ROCm 7.2 (8-GPU Kimi-K2.6)	2026-06-24T01:10
nightly-8-gpu-mi35x-deepseek-v4-pro-rocm720	Accuracy + Performance Test MI35x ROCm 7.2 (8-GPU DeepSeek-V4-Pro FP8 + FP4, triton), Accuracy + Performance Test MI35x ROCm 7.2 (8-GPU DeepSeek-V4-Pro FP8 + FP4, unified_kv_triton)	2026-06-24T01:45
nightly-8-gpu-qwen35-rocm720	Accuracy Test ROCm 7.2 (8-GPU Qwen 3.5)	2026-06-24T10:04
nightly-accuracy-2-gpu-rocm720	N/A	2026-06-24T19:58
nightly-perf-2-gpu-vlm-rocm720	Performance Test ROCm 7.2 (2-GPU VLM Models)	2026-06-24T20:28
nightly-accuracy-2-gpu-vlm-rocm720	Nightly Accuracy Test ROCm 7.2 (2-GPU VLM MMMU)	2026-06-24T20:22
nightly-accuracy-8-gpu-mi35x-deepseek-v32-rocm720	Accuracy Test MI35x ROCm 7.2 (8-GPU DeepSeek-V3.2)	2026-06-24T21:10
nightly-8-gpu-mi35x-kimi-k26-rocm720	Accuracy Test MI35x ROCm 7.2 (8-GPU Kimi-K2.6)	2026-06-24T20:20
nightly-8-gpu-mi35x-deepseek-r1-hicache-rocm720	Accuracy Test MI35x ROCm 7.2 (8-GPU DeepSeek-R1 HiCache)	2026-06-24T21:00
nightly-accuracy-8-gpu-mi35x-rocm720	Accuracy Test MI35x ROCm 7.2 (8-GPU GPT-OSS)	2026-06-24T22:31

Per-Job Analysis

nightly-accuracy-2-gpu-vlm-rocm720 — failed step(s): Nightly Accuracy Test ROCm 7.2 (2-GPU VLM MMMU)

Commit Info

• sglang: ed26a109e
• aiter: 7d604afe5 (source: dev image — v0.1.14rc1.dev240+g7d604afe5 detected in CI image, rebuild skipped; matches Dockerfile default)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi30x/test_vlms_mmmu_eval_amd.py	test_mmmu_vlm_models	AssertionError: The following models failed: (5 VLMs below MMMU threshold)	Log

Per-model breakdown (single subTest-driven assertion; one test function, 5 sub-failures):

Model	TP	Score	Threshold	Verdict
Qwen/Qwen2-VL-7B-Instruct	1	0.2799	0.30	❌
Qwen/Qwen2.5-VL-7B-Instruct	1	0.2802	0.33	❌
Qwen/Qwen3-VL-30B-A3B-Instruct	2	0.2601	0.29	❌
openbmb/MiniCPM-o-2_6	1	0.2802	0.32	❌
deepseek-ai/deepseek-vl2-small	1	0.2802	0.31	❌

Failure Summary

The single test test_mmmu_vlm_models failed because 5 of the configured VLMs scored below their MMMU accuracy thresholds on 100 samples (log line 15207). All 5 failing scores cluster in a tight 0.26–0.28 band. Notably, the models that passed (InternVL2_5-2B 0.29, MiniCPM-v-2_6 0.32, Janus-Pro 0.29, Kimi-VL 0.35, MiMo-VL 0.31, DeepSeek-OCR-2 0.32) also scored near the same 0.28–0.35 floor — they passed only because their thresholds are lower (0.25–0.29).

Regression Status

Known recurring failure (not new today, not flaky).

Recent history of test_vlms_mmmu_eval_amd.py (test_mmmu_vlm_models) in job nightly-accuracy-2-gpu-vlm-rocm720 (branch=main, event=schedule, completed runs only):

Date	Run	Test File Status	Failing Models (score)
Jun 23	28046795462	❌ Failed	Qwen2-VL 0.28, Qwen2.5-VL 0.28, Qwen3-VL 0.26, MiniCPM-o 0.28, deepseek-vl2 0.28
Jun 22	27975850577	❌ Failed	Qwen2.5-VL 0.28, Qwen3-VL 0.26, MiniCPM-o 0.28, deepseek-vl2 0.28
Jun 21	27912828989	❌ Failed	Qwen2-VL 0.29, Qwen2.5-VL 0.29, Qwen3-VL 0.26, MiniCPM-o 0.28, deepseek-vl2 0.28

Scores are nearly identical run-to-run (variance ≤ 0.01), so this is deterministic, not flaky. No green run for this test was found in the queryable schedule window (Jun 21–23 all failed; Jun 16–19 runs were cancelled; Jun 12–15 also failure). First passing run unknown — lookback limited to ~12 most-recent schedule runs, none green for this test.

Failure Cluster

VLM MMMU accuracy at near-random floor (~0.28) on ROCm 7.2

Facts (verified observations)

• All 5 failing models score within 0.26–0.28; MMMU is 4-choice multiple-choice so random chance ≈ 0.25. (log 15208–15212)
• Qwen/Qwen2.5-VL-7B-Instruct scored 0.2802 against a 0.33 threshold — this model's reference MMMU is ~0.50, so 0.28 is a ~0.22 absolute deficit, far beyond eval noise on 100 samples.
• The 6 passing models also scored only 0.29–0.35 — i.e. the entire VLM suite is at the near-random floor, not just the 5 that tripped their thresholds.
• Identical failure set + identical scores reproduce across 3 consecutive completed scheduled runs (Jun 21/22/23).
• aiter version is unchanged across all 3 runs (Dockerfile default 7d604afe5, dev image preserved) — confirmed via [CI-AITER-CHECK] markers (log 19:07:12).
• Benign noise in log: Ignore import error when loading ...mimo_v2: No module named 'torchcodec' — unrelated to the accuracy failure (MiMo-VL still ran and passed).

Hypothesised Causes (with confidence)

• [MEDIUM] Systemic VLM inference degradation on the ROCm 7.2 / MI30x backend (vision encoder, multimodal preprocessing, or attention path producing near-random outputs across all VLMs).
  • Supporting: every VLM in the suite — passing and failing — sits at the ~0.28–0.35 random-chance floor; a model expected at ~0.50 (Qwen2.5-VL) lands at 0.28; the pattern is uniform across architectures (Qwen, MiniCPM, DeepSeek), pointing to a shared backend code path rather than per-model regressions.
  • Disconfirming: scores are not exactly 0.25 and some models (Kimi-VL 0.35, MiniCPM-v 0.32) sit slightly above the floor, which is not what a fully-broken pipeline (all responses garbage) would produce — partial functionality cannot be excluded. No green baseline was found, so it cannot be proven the suite ever scored higher on this backend.
  • How to verify: launch one model (e.g. Qwen2.5-VL-7B) on an MI30x ROCm 7.2 container and manually inspect raw generations on a few MMMU images — are answers coherent or random? Compare the same model+commit on a CUDA runner.
• [LOW] Thresholds set too high for the AMD/ROCm 7.2 backend (test-config issue rather than a model regression).
  • Supporting: thresholds (0.30–0.33) were presumably calibrated on different hardware; an open PR #26178 ("[AMD] WIP - relaxing thres for mi300") suggests maintainers already consider AMD VLM thresholds miscalibrated.
  • Disconfirming: 0.28 for Qwen2.5-VL is implausibly low for a correctly running model regardless of hardware — this looks like broken inference, not a slightly-optimistic threshold. Relaxing thresholds would mask a real correctness bug.
  • How to verify: confirm the manual-generation check above first; only adjust thresholds if outputs are coherent and the deficit is genuine eval variance.
• [LOW] sglang commit in the regression window altered the VLM/multimodal path. The window cannot be narrowed because no green run exists in the queryable history; the failure predates Jun 21.
  • How to verify: extend the schedule-run lookback (GitHub API ?page=2) to find the last green nightly-accuracy-2-gpu-vlm-rocm720 run, then git log --oneline <pass_sha>..<fail_sha> and grep for multimodal/vision/attention changes.

No hypothesis reaches HIGH confidence; the dominant signal (whole-suite near-random floor) supports the systemic-degradation hypothesis but lacks a confirmed green baseline or a manual generation inspection.

In-flight Fix Check

• In-flight fix: ⚠️ partial — #26178 "[AMD] WIP - relaxing thres for mi300" (open, WIP) targets AMD VLM threshold relaxation but is scoped to mi300 and is a threshold change, not a correctness fix. It would suppress the symptom, not address the near-random scores. (searched: is:pr is:open vlm mmmu accuracy, vlm threshold.)
• In-flight fix: ❌ none found for an actual ROCm 7.2 VLM correctness fix.

Suggested Triage Steps

• Manually launch Qwen/Qwen2.5-VL-7B-Instruct on an MI30x ROCm 7.2 container at commit ed26a109e and inspect raw MMMU generations — coherent vs. random distinguishes "broken inference" from "tight threshold".
• Cross-check the same model+commit on a CUDA nightly VLM run; if CUDA scores ~0.5 and ROCm 7.2 scores ~0.28, the bug is AMD/ROCm-specific multimodal inference.
• Page the GitHub Actions API beyond the last 12 schedule runs to locate the last green nightly-accuracy-2-gpu-vlm-rocm720, then bisect git log <pass_sha>..<fail_sha> for vision/multimodal/attention commits.
• Do not simply relax thresholds (e.g. via #26178) until the manual generation check confirms outputs are coherent — otherwise a real correctness regression would be masked.

Status

• Persistence: Failing every completed scheduled run Jun 21–23 (3 consecutive); deterministic, scores stable within ±0.01. No green run in the queryable window.
• Scope: 1 job (nightly-accuracy-2-gpu-vlm-rocm720); 5 models hard-failing, with the entire VLM suite (incl. passing models) sitting at the near-random floor.
• Blocked work: AMD ROCm 7.2 VLM accuracy signal — currently provides no protection, since the whole suite is at chance level.
• In-flight fix: ⚠️ only a WIP threshold-relaxation PR #26178 (mi300-scoped, symptom-only); no correctness fix found.

nightly-accuracy-8-gpu-mi35x-deepseek-v32-rocm720 — failed step(s): Accuracy Test MI35x ROCm 7.2 (8-GPU DeepSeek-V3.2)

Commit Info

• sglang: ed26a109e
• aiter: 7d604afe5 (v0.1.14rc1.dev240+g7d604afe5, source: dev image preserved — matches Dockerfile default commit, no override; AITER_COMMIT_OVERRIDE empty)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi35x/test_deepseek_v32_eval_mi35x.py	test_deepseek_v32_accuracy	AssertionError: Failed models: ['deepseek-ai/DeepSeek-V3.2 (basic)'] — GSM8K accuracy=0.870 threshold=0.93 ❌ FAIL	Log

Failure Summary

The single test test_deepseek_v32_accuracy failed because DeepSeek-V3.2 (basic, TP=8) scored 0.870 on the GSM8K completion benchmark (200 questions), below the configured pass threshold of 0.93 (test file line 71). The server started and served all 200 requests successfully (no crash); the test asserted on the accuracy gap at line 245.

Regression Status

Never-passed (no green run found in queryable window). The nightly-accuracy-8-gpu-mi35x-deepseek-v32-rocm720 job — running this single test file — failed in every completed scheduled run from Jun 12 → Jun 23. The failures alternate between two distinct symptoms on the same test file: an accuracy-threshold miss (~0.865–0.870 vs 0.93) and a server watchdog-timeout hang (TokenizerManager + Scheduler watchdog timeout, exit 255 at the 3600s file timeout). The current job is the accuracy-miss variant.

Recent history of test_deepseek_v32_eval_mi35x.py (test_deepseek_v32_accuracy) in job nightly-accuracy-8-gpu-mi35x-deepseek-v32-rocm720 (branch main, event schedule, completed runs only):

Date	Run	Job	Test File Status	Symptom	Detail
Jun 23	28046795462	job	❌ Failed	accuracy miss	acc=0.870 < 0.93
Jun 22	27975850577	job	❌ Failed	watchdog hang	timeout, exit 255 @3600s
Jun 21	27912828989	job	❌ Failed	accuracy miss	acc=0.870 < 0.93
Jun 20	27879239829	job	❌ Failed	accuracy miss	acc=0.865 < 0.93
Jun 16	27568866028	job	❌ Failed	watchdog hang	timeout
Jun 12	27434236636	job	❌ Failed	watchdog hang	timeout

(Jun 17–19 runs were cancelled — excluded. The current overall run 28046795462 reports queued because other matrix jobs are still pending, but this job itself completed with failure. No passing run for this test file exists in the entire queryable window.)

Failure Cluster

GSM8K accuracy below threshold — DeepSeek-V3.2 (basic) on MI35x/ROCm 7.2 (sibling cluster: DeepSeek-V3.2 server watchdog-timeout hang on the same test).

Facts (verified observations, no interpretation)

• Verbatim from log: accuracy=0.870 threshold=0.93 ❌ FAIL then AssertionError: Failed models: ['deepseek-ai/DeepSeek-V3.2 (basic)'] (raw log line 17024–17031). Test ran 200/200 GSM8K questions; all /generate requests returned 200 OK.
• Configured threshold is accuracy_threshold=0.93 for the basic variant (test file line 71).
• Server config: attention_backend='dsa', dsa_prefill_backend='tilelang', dsa_decode_backend='tilelang', dsa_topk_backend='sgl-kernel', tp_size=8, kv_cache_dtype='bfloat16', mem_fraction_static=0.85.
• aiter version inside CI image: v0.1.14rc1.dev240+g7d604afe5 (dev build of Dockerfile-default commit 7d604afe5); no override applied ([CI-AITER-CHECK] markers, log lines 3628–3634).
• Reproducibility: same test file fails in 6/6 completed scheduled runs Jun 12–23; the accuracy-miss symptom specifically in 3 of those (Jun 20: 0.865, Jun 21: 0.870, Jun 23: 0.870). The accuracy is stable and consistently ~6 points below threshold.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [LOW] The 0.93 threshold may be too tight for the current DSA/tilelang backend numerics on MI35x, i.e. a real but small accuracy gap rather than a fresh code regression.
  • Supporting: accuracy is tightly clustered (0.865–0.870) across three independent runs — not noisy, suggesting a stable systematic gap rather than a flake; the server runs to completion with no errors.
  • Disconfirming: no green baseline exists in the queryable window, so it is unknown whether this test ever passed at 0.93 — the threshold could be correct and a real regression predates the window. Cannot distinguish "threshold too high" from "long-standing accuracy regression" without an earlier passing run.
  • How to verify: find the commit that introduced this test/threshold (git log for the file) and the first nightly run after it; check whether it ever passed at 0.93.
• [LOW] A DSA/tilelang attention or topk kernel numerical issue (the dsa/tilelang/sgl-kernel path used here) produces slightly degraded outputs on gfx950.
  • Supporting: the failure is specific to the DSA-backend DeepSeek-V3.2 config on MI35x; the sibling watchdog-hang symptom on the same job also points to instability in this serving path.
  • Disconfirming: no code-level evidence tying any specific commit to the numerical gap; the accuracy is consistent, not erratic, which is unusual for a kernel correctness bug. Related open work #29050 (dense-MHA prefill for DSA on gfx950) suggests the DSA prefill path on MI350/355 is still being actively tuned.
  • How to verify: re-run with an alternate attention backend (if supported for V3.2) or compare GSM8K accuracy on MI300/MI325 vs MI35x for the same commit to isolate gfx950-specific numerics.

No hypothesis reaches MEDIUM confidence — needs a passing baseline (or git bisect of the test-introduction commit) to separate a too-tight threshold from a genuine accuracy regression.

In-flight Fix Check

• In-flight fix: ❌ none found directly targeting the accuracy regression. Candidate-adjacent open PRs (not confirmed fixes): #25559 "[AMD] WIP — Deepseek v3.2: disable preshuffle for nsa path" (WIP, touches the NSA/DSA path) and #29050 "[AMD][Perf] Enable dense-MHA prefill on gfx950 for DSA models (DeepSeek-V3.2)" (perf-focused). Search: repo:sgl-project/sglang is:pr is:open DeepSeek-V3.2 accuracy (39 hits, none scoped to this MI35x accuracy threshold).

Suggested Triage Steps

• Identify the test/threshold-introduction commit: git log --oneline -- test/registered/amd/accuracy/mi35x/test_deepseek_v32_eval_mi35x.py, then check the first scheduled nightly after it merged — establish whether this test EVER passed at 0.93 (distinguishes "too-tight threshold" from "real regression").
• If never passed: treat as an unvalidated AMD port; either correct the threshold to a justified value or keep the job red until the DSA/tilelang accuracy gap is closed.
• Isolate gfx950 numerics: run the same GSM8K eval for DeepSeek-V3.2 on MI300/MI325 at the same sglang commit; a pass there implicates the gfx950 DSA/tilelang kernel path.
• Track #25559 and #29050 — if either lands, re-run this job to check whether accuracy recovers.
• Note the sibling watchdog-hang symptom (Jun 12/16/22) on the same job — investigate server stability on this DSA config separately; the two symptoms may share a root cause in the DSA/tilelang serving path.

Status

• Persistence: Test file red in every completed scheduled run Jun 12 → Jun 23 (≥12 days); no green run in the queryable window. Accuracy-miss symptom in 3 runs (stable 0.865–0.870), watchdog-hang symptom in 3 runs.
• Scope: 1 job (nightly-accuracy-8-gpu-mi35x-deepseek-v32-rocm720) in nightly-test-amd-rocm720.yml. Sibling -mtp- accuracy job passes, so this is specific to the DeepSeek-V3.2 basic/DSA config on MI35x.
• Blocked work: DeepSeek-V3.2 accuracy signal on AMD MI35x / ROCm 7.2 nightly.
• In-flight fix: ❌ none directly targeting this; candidate-adjacent PRs #25559, #29050.

nightly-accuracy-2-gpu-rocm720 — failed step(s): Nightly Test ROCm 7.2 (2-GPU)

Commit Info

• sglang: ed26a109e
• aiter: v0.1.14rc1.dev240+g7d604afe5 (source: dev/patched image preserved — [CI-AITER-CHECK] reported Dev/patched version detected ... → skipping rebuild; Dockerfile default was 7d604afe5)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi30x/test_gsm8k_eval_amd.py	TestNightlyGsm8KEval.test_gsm8k_all_models (model=Qwen/Qwen2-57B-A14B-Instruct)	TimeoutError: Server failed to start within the timeout period	Log

The single unittest result was FAILED (errors=1) — the TimeoutError on Qwen2-57B-A14B-Instruct (model 7 of the suite) aborted the test before it reached the per-model accuracy assertion (check_model_scores). Before that abort, two models had already scored below threshold (reported only as in-loop ❌ FAILED status, not yet asserted):

Model	Score	Threshold	In-loop status
mistralai/Mistral-7B-Instruct-v0.3	0.361	0.47	❌ below threshold
mistralai/Mixtral-8x7B-Instruct-v0.1	0.315	0.69	❌ below threshold
Qwen/Qwen2-57B-A14B-Instruct	—	0.76	server startup TimeoutError

(Passed before the abort: Llama-3.1-8B 0.818, DeepSeek-Coder-V2-Lite 0.811, gemma-2-27b 0.861, Llama-3.1-70B 0.932.)

Failure Summary

test_gsm8k_all_models in job nightly-accuracy-2-gpu-rocm720 errored out when the Qwen2-57B-A14B-Instruct server failed to start within the launch timeout. The pyspy watchdog dump shows the scheduler stuck inside HuggingFace weight download (download_weights_from_hf → filelock _poll_until_acquired, with a sibling thread in http_get). Independently, the Mistral-7B-v0.3 and Mixtral-8x7B accuracy scores were far below threshold, as they have been in every recent run.

Regression Status

Two distinct issues, different histories.

(A) Qwen2-57B-A14B server-startup TimeoutError — NEW today. Qwen2-57B started fine and PASSED in the two preceding completed runs.

(B) Mistral-7B-v0.3 / Mixtral-8x7B accuracy below threshold — pre-existing, persistent (≥4 days).

Recent history of test_gsm8k_eval_amd.py (test_gsm8k_all_models) in job nightly-accuracy-2-gpu-rocm720 (branch main, event schedule, completed runs only):

Date	Run	Test File Status	Qwen2-57B	Mistral-7B-v0.3	Mixtral-8x7B
Jun 23	28046795462	❌ Failed (errors=1)	❌ startup TimeoutError	0.361 (<0.47)	0.315 (<0.69)
Jun 22	27975850577	❌ Failed (failures=1)	✅ 0.789	0.362 (<0.47)	0.301 (<0.69)
Jun 21	27912828989	❌ Failed	(run truncated at Mixtral)	0.363 (<0.47)	(not reached in extract)
Jun 20	27879239829	❌ Failed (failures=1)	✅ 0.783	0.371 (<0.47)	0.296 (<0.69)

test_gsm8k_all_models is Never-passed in the queryable window (Jun 20–23 all failure; Jun 14–19 runs were cancelled/not comparable). Lookback range: last 12 scheduled runs (Jun 12–23).

Failure Cluster

Two clusters in this job:

• Cluster 1 — "Qwen2-57B server startup hang on HF weight download" (NEW today; the proximate cause of today's errors=1).
• Cluster 2 — "Mistral/Mixtral GSM8K accuracy below threshold on AMD" (persistent symptom-level grouping, ≥4 days).

Facts (verified observations)

• TimeoutError: Server failed to start within the timeout period, raised at test_gsm8k_eval_amd.py:258 → :139 → python/sglang/test/test_utils.py:1028 (verbatim from log).
• pyspy watchdog dump (PID 31986, sglang::scheduler_TP0): MainThread blocked in download_weights_from_hf (model_loader/weight_utils.py:554) → filelock _poll_until_acquired (filelock/_api.py:552); a worker thread blocked in http_get (huggingface_hub/file_download.py:433). Scheduler watchdog timeout (self.watchdog_timeout=300, self.soft=True) fired on both TP0 and TP1. The stuck loader is qwen2_moe.py:1107 (Qwen2-57B-A14B is a qwen2_moe architecture).
• Qwen2-57B-A14B PASSED in the two prior completed runs (0.789 on Jun 22, 0.783 on Jun 20) — startup was not a problem then.
• Mistral-7B-v0.3 scores: 0.361 / 0.362 / 0.363 / 0.371 across Jun 23/22/21/20 — stable and consistently below the 0.47 threshold.
• Mixtral-8x7B scores: 0.315 / 0.301 / 0.296 across Jun 23/22/20 — stable and far below the 0.69 threshold.
• aiter is unchanged across these runs (dev image v0.1.14rc1.dev240+g7d604afe5 preserved).

Hypothesised Causes (with confidence, with disconfirming evidence)

• [MEDIUM] Cluster 1: Qwen2-57B startup TimeoutError is caused by a slow/stalled HuggingFace weight download (network/cache infrastructure), not an sglang code regression.

  • Supporting: pyspy dump shows the scheduler blocked in download_weights_from_hf → filelock acquire + http_get, i.e. waiting on weight bytes/lock, not in a kernel or model-init code path; the same model started fine in the two prior runs with no code change to the loader implicated.
  • Disconfirming: a single pyspy snapshot proves where it was stuck at that instant, not why the download was slow; cannot fully exclude a filelock-contention or stale-cache issue introduced in the image. No second confirming data point (only one timeout occurrence so far).
  • How to verify: re-run the job; if Qwen2-57B starts and scores ~0.78 again, this is a transient HF-download/infra flake. If it recurs, inspect runner HF cache state and network to huggingface.co during the launch window.

• [LOW] Cluster 2: Mistral-7B-v0.3 / Mixtral-8x7B low GSM8K accuracy is a pre-existing eval/chat-template or attention-backend issue on AMD, independent of today's run.

  • Supporting: scores are stable across ≥4 days and well below threshold; open PR #27757 identifies a GSM8K chat-eval prompt/extraction + FA backend interaction for exactly these Mistral/Mixtral models.
  • Disconfirming: this cluster did not cause today's specific errors=1 signature (the test aborted earlier on the Qwen timeout, before the accuracy assertion); root cause not yet code-confirmed on AMD.
  • How to verify: track PR #27757; after it merges, confirm Mistral/Mixtral scores cross threshold on this job.

In-flight Fix Check

• Cluster 2 (Mistral/Mixtral accuracy): ✅ #27757 "Fix Mistral GSM8K chat eval" (open since 2026-06-10) — makes the GSM8K chat prompt explicit, prefers Answer: <number> extraction, plumbs stop sequences, and keeps Mistral/Mixtral off FA3 auto-default. Directly targets this cluster — chase reviewers, do NOT open a duplicate. (Search: is:pr is:open Mixtral accuracy.)
• Cluster 1 (Qwen2-57B startup timeout): ❌ none found (is:pr is:open Qwen2-57B returned no results) — likely infra/transient; no PR needed unless it recurs.

Suggested Triage Steps

• Cluster 1: Re-run nightly-accuracy-2-gpu-rocm720 and check whether Qwen2-57B-A14B starts within the timeout. If it recovers, treat as a transient HF-download flake. If it recurs, inspect the runner's HF cache / huggingface.co connectivity and consider pre-staging Qwen2-57B weights on the MI30x runners.
• Cluster 2: Verify #27757 addresses Mistral-7B-v0.3 and Mixtral-8x7B; if it merges, re-run this job and confirm both cross threshold. Until then this test cannot pass regardless of the Qwen timeout.

Status

• Persistence: Cluster 2 (Mistral/Mixtral accuracy) — failing every completed run for ≥4 days (since at least Jun 20). Cluster 1 (Qwen2-57B startup timeout) — first observed today (Jun 23); not seen in the two prior completed runs.
• Scope: One job (nightly-accuracy-2-gpu-rocm720), one test function (test_gsm8k_all_models). The Qwen timeout aborts the suite early, masking the FP8-model accuracy results that fail in full prior runs.
• Blocked work: AMD ROCm 7.2 nightly GSM8K accuracy signal is red and has not produced a green run in the queryable window.
• In-flight fix: Cluster 2 → ✅ #27757 (open since 2026-06-10). Cluster 1 → ❌ none found.

nightly-8-gpu-mi35x-qwen3-235b-mxfp4-rocm720 — failed step(s): Accuracy Test + Performance Test MI35x ROCm 7.2 (8-GPU Qwen3-235B-MXFP4)

Commit Info

• sglang: ed26a109 — [Fix] Return streaming logprobs when reasoning/tool parser is active (#28601)
• aiter: 7d604afe5 — v0.1.14rc1.dev240+g7d604afe5 (source: dev image preserved; [CI-AITER-CHECK] reports Dev/patched version detected → skipping rebuild, AITER_COMMIT_OVERRIDE empty, Dockerfile default is the same SHA)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/test_qwen3_instruct_mxfp4.py	TestQwen3Instruct2507MXFP4.setUpClass	RuntimeError: Rank 0 scheduler died during initialization (exit code: -6) — server abort during decode CUDA graph capture	Log

Failure Summary

The server for amd/Qwen3-235B-A22B-Instruct-2507-mxfp4 (--tp 4 --ep 2 --attention-backend aiter) launched, loaded weights, allocated KV cache, and crashed during decode CUDA graph capture. The torch NCCL ProcessGroupNCCL watchdog thread threw HIP error: operation not permitted on an event last recorded in a capturing stream (hipErrorCapturedEvent), which raised c10::DistBackendError, called terminate, and aborted rank 0 with SIGABRT (exit -6). setUpClass then failed because the server never became ready (0 tests ran).

Regression Status

Never-passed (no green run found in queryable window). This exact failure has occurred in every completed scheduled run of this job from May 25 through Jun 22 — same test, same hipErrorCapturedEvent during graph capture, same exit -6.

Recent history of test/registered/amd/test_qwen3_instruct_mxfp4.py (TestQwen3Instruct2507MXFP4.setUpClass) in job nightly-8-gpu-mi35x-qwen3-235b-mxfp4-rocm720 (branch main, event schedule, completed runs only):

Date	Run	Job	Test File Status	Failed Function	Error
Jun 23	28046795462	job	❌ Failed	setUpClass	hipErrorCapturedEvent → exit -6
Jun 22	27975850577	job	❌ Failed	setUpClass	hipErrorCapturedEvent → exit -6
Jun 21	27912828989	job	❌ Failed	setUpClass	hipErrorCapturedEvent → exit -6
Jun 20	27879239829	job	❌ Failed	setUpClass	hipErrorCapturedEvent → exit -6
Jun 15	27568866028	job	❌ Failed	setUpClass	hipErrorCapturedEvent → exit -6
Jun 10	27296828333	job	❌ Failed	setUpClass	hipErrorCapturedEvent → exit -6
Jun 02	26840660185	job	❌ Failed	setUpClass	hipErrorCapturedEvent → exit -6
May 25	26413553204	job	❌ Failed	setUpClass	hipErrorCapturedEvent → exit -6

(Runs Jun 16–19 were workflow-level cancelled and excluded. Lookback window searched: May 25 – Jun 23; no passing run for this test file found.)

Failure Cluster

HIP error during decode CUDA graph capture (NCCL watchdog event query in capturing stream)

Facts (verified observations, no interpretation)

• Server command (verbatim from log): sglang serve --model-path amd/Qwen3-235B-A22B-Instruct-2507-mxfp4 --tp 4 --ep 2 --trust-remote-code --attention-backend aiter --device cuda.
• Init progressed normally: weights loaded (Load weight end ... quant=quark), KV Cache is allocated ... #tokens: 4138168, Memory pool end. avail mem=58.75 GB, then Capture target decode CUDA graph begin. backend=full ... avail mem=25.95 GB.
• During capture the watchdog threw (verbatim): [PG ID 5 PG GUID 23 Rank 1] Process group watchdog thread terminated with exception: HIP error: operation not permitted on an event last recorded in a capturing stream and operation not permitted when stream is capturing (hipErrorCapturedEvent / hipErrorStreamCaptureUnsupported).
• Watchdog stack (verbatim): c10d::ProcessGroupNCCL::WorkNCCL::finishedGPUExecutionInternal() → WorkNCCL::isCompleted() → Watchdog::runLoop() → Watchdog::run(). Then terminate called after throwing an instance of 'c10::DistBackendError' and Fatal Python error: Aborted.
• Main-thread location at abort: custom_all_reduce.py:505 capture (aiter) ← parallel_state.py:538 graph_capture ← decode_cuda_graph_runner.py:685 capture ← base_cuda_graph_runner.py:55 freeze_gc (gc.collect() in the __exit__ of the capture context).
• Final result: RuntimeError: Rank 0 scheduler died during initialization (exit code: -6); Ran 0 tests in 606.165s; Test Summary: 0/1 passed; job exit 255.
• A non-fatal Scheduler watchdog timeout (... self.soft=True) fired earlier (~00:39:57) during the ~446–537 s weight load (exceeds the 300 s soft watchdog); this is a separate soft warning, not the crash.
• Reproducibility: identical symptom (hipErrorCapturedEvent, exit -6, same test) in 8/8 sampled completed runs spanning May 25 – Jun 22.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [HIGH] The crash is the torch NCCL ProcessGroupNCCL watchdog thread polling work-completion events (hipEventQuery via WorkNCCL::isCompleted) while a HIP stream is mid graph-capture, which is illegal on ROCm → hipErrorCapturedEvent → terminate → SIGABRT.
  • Supporting: watchdog stack frames and the operation not permitted ... capturing stream message are directly in the log; the main thread is simultaneously inside graph_capture/decode_cuda_graph_runner.py:685 capture.
  • Disconfirming: the precise condition that lets the watchdog poll during capture (timing/config-specific) is not pinned from the log alone; other AMD nightly jobs that also capture CUDA graphs do not all crash this way, so a config interaction specific to this TP4/EP2 mxfp4 run is likely.
  • How to verify: re-run with the NCCL watchdog disabled/monitoring off (e.g. TORCH_NCCL_ENABLE_MONITORING=0) or with --disable-cuda-graph; if the crash disappears, the watchdog/capture interaction is confirmed.
• [LOW] Environment/ROCm-7.2 + aiter custom all-reduce interaction during capture (custom_all_reduce.py:505 is the aiter capture call site on the main thread).
  • Supporting: the main thread aborts inside the aiter custom all-reduce capture; AMD-only path.
  • Disconfirming: the actual thrown exception originates in the torch NCCL watchdog thread, not in the aiter call; the aiter frame is merely where the main thread happened to be when the process aborted.
  • How to verify: re-run with --disable-custom-all-reduce and observe whether the watchdog still aborts during capture.
• Not a regression from the current commit. ed26a109 only touches streaming logprobs; the failure predates it by ~1 month, so no bisect of a recent window is warranted.

In-flight Fix Check

• In-flight fix: ❌ none found — hipErrorCapturedEvent search returned 0 open PRs; the 3 "watchdog + capturing" matches (#23317, #17356, #13023) are unrelated (FlashInfer symm-mem autotune, VLM Eagle, allocator RFC). The one Qwen3-235B mxfp4 PR (#27209) is a gfx950 Dockerfile change, not a fix for this crash. (API calls: search/issues q=...hipErrorCapturedEvent, ...watchdog+capturing+stream, ...Qwen3-235B+mxfp4.)

Suggested Triage Steps

• Reproduce on an MI35x (gfx950) ROCm-7.2 runner with the exact command, then toggle one variable at a time: (a) TORCH_NCCL_ENABLE_MONITORING=0 / disable the NCCL watchdog, (b) --disable-cuda-graph, (c) --disable-custom-all-reduce. The first toggle that makes the run pass identifies the responsible interaction.
• If the watchdog is confirmed, investigate suppressing/parking the ProcessGroupNCCL watchdog event polling during the graph_capture window in python/sglang/srt/distributed/parallel_state.py:538 / decode_cuda_graph_runner.py:685.
• Since this test has never passed on this AMD config in the queryable window, consider quarantining test_qwen3_instruct_mxfp4.py on the rocm720/MI35x nightly suite until the capture/watchdog interaction is fixed, to restore signal for the rest of the workflow.

Status (factual, no priority assignment)

• Persistence: Failing in every completed scheduled run of this job from at least 2026-05-25 through 2026-06-23 (8/8 sampled runs); never observed passing in the queryable window.
• Scope: nightly-8-gpu-mi35x-qwen3-235b-mxfp4-rocm720 job in nightly-test-amd-rocm720.yml; AMD MI35x (gfx950) ROCm-7.2 only.
• Blocked work: No accuracy/serving signal for Qwen3-235B-A22B-Instruct-2507-mxfp4 on MI35x ROCm-7.2 nightly.
• In-flight fix: ❌ none found — needs new investigation/PR.

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-06-24 23:07 UTC)

[amd-bot]

nightly-test-amd.yml — 14 failure(s)

Run started (UTC): 2026-06-23 18:39 UTC
Last scanned (UTC): 2026-06-24 23:06 UTC
Commits: sglang d6aacd2

Summary

Counts: 17 failures · 11 clusters · 11 carrying over (≥3 days) · 6 new today

#	Cluster	Job	Test File	Test Function	Status	Hypothesis (confidence)
1	mamba extra_buffer rejects ROCm	nightly-8-gpu-mi35x-qwen35	test/registered/amd/accuracy/mi35x/test_qwen35_eval_mi35x.py	test_lm_eval	Never-passed (Jun 20–24)	[HIGH] extra_buffer gated to CUDA/MUSA/NPU; auto-selected on ROCm
2	mamba extra_buffer rejects ROCm	nightly-8-gpu-qwen35	test/registered/amd/accuracy/mi30x/test_qwen35_eval_amd.py	TestQwen35EvalAMD.setUpClass	Never-passed (Jun 20–24)	[FACT] same _validate_mamba_extra_buffer assert
3	mamba extra_buffer rejects ROCm	nightly-8-gpu-mi35x-qwen35 [IN-FLIGHT]	test/registered/amd/accuracy/mi35x/test_qwen35_eval_mi35x.py	test_lm_eval	Recurring (3 runs)	[HIGH] same assert
4	HIP captured-event abort in CUDA-graph capture	nightly-8-gpu-mi35x-qwen3-235b-mxfp4	test/registered/amd/test_qwen3_instruct_mxfp4.py	TestQwen3Instruct2507MXFP4.setUpClass	Never-passed ≥11d	[MEDIUM] NCCL watchdog event-query on capturing stream / aiter custom_all_reduce
5	HIP captured-event abort in CUDA-graph capture	nightly-8-gpu-mi35x-qwen3-235b-mxfp4 [IN-FLIGHT]	test/registered/amd/test_qwen3_instruct_mxfp4.py	TestQwen3Instruct2507MXFP4.setUpClass	Never-passed ≥11d	[MEDIUM] same
6	VLM MMMU accuracy below threshold	nightly-accuracy-2-gpu-vlm	test/registered/amd/accuracy/mi30x/test_vlms_mmmu_eval_amd.py	test_mmmu_vlm_models	Persistent ≥5d	[no candidate ≥ MEDIUM] (LOW threshold calibration)
7	VLM MMMU accuracy below threshold	nightly-accuracy-2-gpu-vlm [IN-FLIGHT]	test/registered/amd/accuracy/mi30x/test_vlms_mmmu_eval_amd.py	test_mmmu_vlm_models	Persistent ≥5d	[no candidate ≥ MEDIUM]
8	GSM8K Mistral/Mixtral below threshold	nightly-accuracy-2-gpu	test/registered/amd/accuracy/mi30x/test_gsm8k_eval_amd.py	test_gsm8k_all_models (Mistral/Mixtral)	Persistent ≥5d	[MEDIUM] chat-eval extraction under-scores Mistral
9	GSM8K Mistral/Mixtral below threshold	nightly-accuracy-2-gpu [IN-FLIGHT]	test/registered/amd/accuracy/mi30x/test_gsm8k_eval_amd.py	test_gsm8k_all_models (Mistral-7B)	Persistent ≥5d	[MEDIUM] same
10	HF Hub weight-download hang → timeout	nightly-8-gpu-mi35x-kimi-k26	test/registered/amd/accuracy/mi35x/test_kimi_k26_eval_mi35x.py	test_kimi_k26_gsm8k_accuracy	Recurring (Jun 21/23/24)	[HIGH] Xet download stall on MI35x runner
11	HF Hub weight-download hang → timeout	nightly-accuracy-2-gpu-vlm	test/registered/amd/accuracy/mi30x/test_vlms_mmmu_eval_amd.py	test_mmmu_vlm_models (deepseek-vl2-small)	New today	[HIGH] HF Hub snapshot_download hang
12	GPU mem fault — R1 hicache prefill	nightly-8-gpu-mi35x-deepseek-r1-hicache	test/registered/amd/test_deepseek_r1_hicache_mi35x.py	test_gsm8k	Never-passed (Jun 20+)	[no candidate ≥ MEDIUM] (LOW hicache fp8 path)
13	DeepSeek-V3.2 GSM8K borderline	nightly-accuracy-8-gpu-mi35x-deepseek-v32	test/registered/amd/accuracy/mi35x/test_deepseek_v32_eval_mi35x.py	test_deepseek_v32_accuracy	New Jun 24 (0.925<0.93)	[MEDIUM] benchmark noise on 1-question margin
14	Grok1-FP8 decode graph-capture hang	nightly-accuracy-8-gpu	test/registered/amd/accuracy/mi30x/test_grok1_fp8_eval_amd.py	(eval never reached)	New Jun 24	[no candidate ≥ MEDIUM] (LOW infra flake)
15	Qwen3.5 triton-DCP GSM8K timeout	nightly-8-gpu-mi35x-qwen35 [IN-FLIGHT]	test/registered/amd/test_qwen3p5_triton_dcp.py	TestQwen35TritonDCPGsm8k.test_a_gsm8k	New Jun 24 (new test)	[MEDIUM] full GSM8K (1314) > 3600s timeout
16	GLM-5.1-FP8 weight-load stall (barrier hang)	nightly-8-gpu-mi35x-glm51	test/registered/amd/accuracy/mi35x/test_glm51_eval_mi35x.py	test_glm51_accuracy	New Jun 24	[MEDIUM] checkpoint I/O slowdown (infra-leaning)
17	Qwen2-57B server-start hang	nightly-accuracy-2-gpu	test/registered/amd/accuracy/mi30x/test_gsm8k_eval_amd.py	test_gsm8k_all_models (Qwen2-57B)	New Jun 23	[no candidate ≥ MEDIUM] (LOW teardown/flake)

Cluster: mamba extra_buffer rejects ROCm (server-launch assert)

• Affected: rows 1–3 — Qwen3.5-397B on MI35x + MI30x.
• Shared facts: _validate_mamba_extra_buffer asserts is_cuda() or is_musa() or is_npu() at server_args.py:4472-4474; auto strategy picks extra_buffer for hybrid-mamba Qwen3.5 then aborts on ROCm. Deterministic, every completed run.
• In-flight fix: ✅ #29158 (ROCm no_buffer fallback, open Jun 24) — directly targets this; related #27141.
• Suggested triage: verify #29158 clears the assert, re-run both qwen35 jobs; interim, pass --mamba-radix-cache-strategy no_buffer.

Cluster: HIP captured-event abort in decode CUDA-graph capture

• Affected: rows 4–5 — Qwen3-235B MXFP4, all 4 ranks, ≥11 days never-passed.
• Shared facts: hipErrorCapturedEvent thrown from ProcessGroupNCCL watchdog while main thread is in decode_cuda_graph_runner.capture → aiter custom_all_reduce.flush_graph_buffers. Plain custom-all-reduce path (enable_aiter_allreduce_fusion=False).
• In-flight fix: ⚠️ #27650 adjacent (aiter all-reduce capture-safety) but targets the fusion+PCG path — unconfirmed for this path. No direct fix.
• Suggested triage: A/B --disable-custom-all-reduce then --disable-cuda-graph on MI35x to localize watchdog vs aiter; consider suspending WorkNCCL polling during graph_capture on ROCm.

Cluster: VLM MMMU accuracy below threshold (MI30x)

• Affected: rows 6–7 — Qwen2-VL/Qwen2.5-VL/Qwen3-VL-30B/MiniCPM-o/deepseek-vl2; scores byte-identical across 5 days (deterministic, not flaky).
• Top hypothesis: [LOW] thresholds set above achievable MI30x scores; disconfirming: no green baseline in window, so a pre-Jun-20 regression cannot be excluded.
• In-flight fix: ❌ none. Suggested triage: git log -p thresholds on full clone to find any green baseline; compare vs NVIDIA MMMU scores.

Cluster: GSM8K Mistral/Mixtral accuracy below threshold

• Affected: rows 8–9 — Mistral-7B-v0.3 (~0.36 vs 0.47), Mixtral-8x7B (~0.29 vs 0.69); stable every night.
• Top hypothesis: [MEDIUM] GSM8K chat-eval extraction under-scores instruction-tuned Mistral (per #27757); disconfirming: #27757's cited 0.417 still < 0.47, so an AMD-specific gap may remain.
• In-flight fix: ✅ #27757 (Fix Mistral GSM8K chat eval, open Jun 10). Suggested triage: cherry-pick #27757, re-run on MI30x, check score clears 0.47.

Cluster: HF Hub weight-download hang → server timeout

• Affected: row 10 (Kimi-K2.6, xet_get stall, recurring Jun 21/23/24) + row 11 (deepseek-vl2-small, snapshot_download hang, new today).
• Top hypothesis: [HIGH] HF Xet/Hub download stalls/too slow on the runner; cache not pre-populated; disconfirming: no explicit Xet network error captured; Jun 22 Kimi reached load (download can succeed).
• In-flight fix: ❌ none. Suggested triage: confirm /sgl-data/hf-cache persists across runs; A/B HF_HUB_DISABLE_XET=1; pre-stage snapshots.

Cluster: GPU memory access fault — DeepSeek-R1 hicache prefill

• Affected: row 12 — TP8 fp8-KV + hierarchical cache; all 8 GPUs fault on first big prefill chunk → 3600s timeout. Never-passed; non-hicache R1 siblings pass.
• Top hypothesis: [LOW] illegal device memory access in hicache fp8-KV path; disconfirming: Reason: Unknown, no faulting kernel; aiter fused_moe log line is proximity not proof.
• In-flight fix: ❌ none. Suggested triage: repro with AMD_LOG_LEVEL=4; A/B toggle --hicache-io-backend kernel→direct, fp8→bf16, hicache off.

Cluster: DeepSeek-V3.2 GSM8K borderline below threshold

• Affected: row 13 — 0.925 vs 0.93 (185 vs 186/200); accuracy trend 0.940→0.935→0.935→0.925; server ran clean.
• Top hypothesis: [MEDIUM] benchmark variance on a 1-question margin (200-q ~±0.018 1σ); disconfirming: monotonic decline hints at small real drift.
• In-flight fix: ❌ none. Suggested triage: re-run 2–3× at same SHA; if reproducible, bisect 43b1fb95..ed26a109 (attention/MoE commits).

Cluster: Grok1-FP8 decode CUDA-graph capture hang

• Affected: row 14 — silent stall at Capturing batches (bs=256), no stack/error; first failure after 3 passes. GPT-OSS step in same job passed.
• Top hypothesis: [LOW] transient runner hang during capture; disconfirming: single occurrence, can't confirm transience.
• In-flight fix: ❌ none. Suggested triage: re-run the Grok1-FP8 step; if it hangs again, capture py-spy/cuda-coredump at the freeze.

Cluster: Qwen3.5 triton-DCP GSM8K timeout

• Affected: row 15 — new test (added by #25090 Jun 24); reached 46% (608/1314) then Connection error retries → 3600s timeout (est_time=4800).
• Top hypothesis: [MEDIUM] full 1314-sample GSM8K exceeds the 3600s harness timeout; disconfirming: connection errors could mask a server stall.
• In-flight fix: ❌ none. Suggested triage: re-run isolated with timeout ≥5400s; inspect server log at 46% to separate slow-vs-stalled.

Cluster: GLM-5.1-FP8 weight-load stall (barrier hang)

• Affected: row 16 — TP7 never finished MoE weight load (copy_ in fused_moe_triton/layer.py), others blocked at monitored_barrier; 5411s budget exhausted. Passed prior 3 runs.
• Top hypothesis: [MEDIUM] checkpoint I/O slowdown (load-time variance trending up 2472s→4384s+); disconfirming: single observation, code cause not excluded.
• In-flight fix: ❌ none. Suggested triage: re-run (cheap discriminator); if reproducible, profile per-shard read throughput / local-vs-network cache.

Cluster: Qwen2-57B server-start hang

• Affected: row 17 — TimeoutError launching Qwen2-57B-A14B (TP2) after ~50 min of watchdog soft-timeouts; new Jun 23 (passed prior 2 runs). Truncated run to 7/18 models.
• Top hypothesis: [LOW] unclean teardown of prior hung Mixtral server blocked Qwen2 startup; disconfirming: log buffering makes the owning server ambiguous; no OOM line.
• In-flight fix: ❌ none. Suggested triage: re-run once; if recurs, launch Qwen2-57B standalone on MI325 2-GPU to test startup in isolation.

Suggested triage order

Rank	Triage step	Targets	Why this first
1	Verify & merge in-flight #29158, re-run qwen35 jobs	rows 1–3 (largest cluster)	Direct fix exists for the 3-job ROCm mamba assert
2	Cherry-pick & validate #27757 on MI30x	rows 8–9	In-flight fix for persistent GSM8K Mistral cluster
3	Confirm HF cache persistence + A/B HF_HUB_DISABLE_XET=1	rows 10–11	One infra change resolves two download-hang failures
4	Re-run GLM51, Grok1-FP8, DeepSeek-V3.2 jobs at same SHA	rows 13, 14, 16	Cheap flake-vs-regression discriminator for new Jun 24 failures
5	A/B --disable-custom-all-reduce/--disable-cuda-graph on MI35x	rows 4–5	Localize the ≥11-day never-passed HIP capture abort; check #27650 coverage

---

Job	Failed Steps	Started
nightly-8-gpu-mi35x-qwen3-235b-mxfp4	Accuracy Test + Performance Test MI35x (8-GPU Qwen3-235B-MXFP4)	2026-06-23T23:53
nightly-accuracy-2-gpu	Nightly Test (2-GPU)	2026-06-23T19:01
nightly-accuracy-2-gpu-vlm	Nightly Accuracy Test (2-GPU VLM MMMU)	2026-06-23T18:39
nightly-accuracy-8-gpu-mi35x-deepseek-v32	Accuracy Test MI35x (8-GPU DeepSeek-V3.2)	2026-06-24T00:50
nightly-8-gpu-mi35x-qwen35	Accuracy Test MI35x (8-GPU Qwen 3.5)	2026-06-24T02:45
nightly-8-gpu-qwen35	Accuracy Test (8-GPU Qwen 3.5)	2026-06-24T02:57
nightly-8-gpu-mi35x-deepseek-r1-hicache	Accuracy Test MI35x (8-GPU DeepSeek-R1 HiCache)	2026-06-24T03:19
nightly-8-gpu-mi35x-glm51	Accuracy Test MI35x (8-GPU GLM-5.1 DSA)	2026-06-24T02:54
nightly-8-gpu-mi35x-kimi-k26	Accuracy Test MI35x (8-GPU Kimi-K2.6)	2026-06-24T02:39
nightly-accuracy-8-gpu	N/A	2026-06-24T05:50
nightly-accuracy-2-gpu	N/A	2026-06-24T20:48
nightly-8-gpu-mi35x-qwen3-235b-mxfp4	Accuracy Test + Performance Test MI35x (8-GPU Qwen3-235B-MXFP4)	2026-06-24T21:57
nightly-8-gpu-mi35x-qwen35	Accuracy Test MI35x (8-GPU Qwen 3.5)	2026-06-24T21:01
nightly-accuracy-2-gpu-vlm	Nightly Accuracy Test (2-GPU VLM MMMU)	2026-06-24T21:56

Per-Job Analysis

nightly-8-gpu-mi35x-qwen3-235b-mxfp4 — failed step(s): Accuracy Test + Performance Test MI35x (8-GPU Qwen3-235B-MXFP4)

Commit Info

• sglang: ed26a109
• aiter: 7d604afe5 v0.1.14rc1.dev240+g7d604afe5 (source: Dockerfile default / dev image preserved — [CI-AITER-CHECK] Dev/patched version detected ... → skipping rebuild, AITER_COMMIT_OVERRIDE empty)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/test_qwen3_instruct_mxfp4.py	TestQwen3Instruct2507MXFP4.setUpClass	RuntimeError: Rank 0 scheduler died during initialization (exit code: -6) → Server process exited with code 1	Log

Failure Summary

The server failed to launch for amd/Qwen3-235B-A22B-Instruct-2507-mxfp4 (--tp 4 --ep 2 --attention-backend aiter). The rank-0 scheduler aborted (SIGABRT, exit code -6) during decode CUDA graph capture, so setUpClass never reached any test — Ran 0 tests. No test functions executed; this is a server-init crash, not an assertion failure.

Regression Status

Known recurring failure / Never-passed in queryable window. This job (nightly-8-gpu-mi35x-qwen3-235b-mxfp4) has failed in every completed scheduled run with the identical test, symptom, and exit code, as far back as I queried (2026-06-12). No green run found.

Recent history of test/registered/amd/test_qwen3_instruct_mxfp4.py (TestQwen3Instruct2507MXFP4.setUpClass) in job nightly-8-gpu-mi35x-qwen3-235b-mxfp4 (branch main, event schedule, completed runs only):

Date	Run	Test File Status	Failed Function	Error
Jun 23	28046906309	❌ Failed	setUpClass	scheduler died exit -6 (HIP capture error)
Jun 22	27975987522	❌ Failed	setUpClass	same
Jun 21	27912890405	❌ Failed	setUpClass	same
Jun 20	27879266774	❌ Failed	setUpClass	same
Jun 15	27568949589	❌ Failed	setUpClass	same
Jun 14	27507337214	❌ Failed	setUpClass	same
Jun 12	27434333913	❌ Failed	setUpClass	same

(Jun 16–19 runs were cancelled — excluded. Lookback range searched: 2026-06-12 → 2026-06-23.)

Failure Cluster

HIP stream-capture violation in aiter custom all-reduce during decode CUDA graph capture (→ SIGABRT -6)

Facts (verified observations, no interpretation)

• Crash sequence (verbatim from log, run 28046906309):
  • Capturing batches (bs=512 avail_mem=25.33 GB): 0% then immediately:
  • [rank0]:[E624 00:11:06 ProcessGroupNCCL.cpp:2055] ... Process group watchdog thread terminated with exception: HIP error: operation not permitted on an event last recorded in a capturing stream (hipErrorCapturedEvent)
  • also HIP error: operation not permitted when stream is capturing (hipErrorStreamCaptureUnsupported)
  • terminate called after throwing an instance of 'c10::DistBackendError' → terminate called recursively → Fatal Python error: Aborted
• Python stack at abort (all 4 ranks identical):
  • aiter/dist/device_communicators/custom_all_reduce.py:338 in _gather_ipc_meta
  • aiter/.../custom_all_reduce.py:305 in flush_graph_buffers
  • aiter/.../custom_all_reduce.py:505 in capture
  • sglang/srt/distributed/parallel_state.py:538 / :1682 in graph_capture
  • sglang/srt/model_executor/runner/decode_cuda_graph_runner.py:687 in capture ← :353 __init__
  • model_runner.py:2583 init_decode_cuda_graph ← :913 init_cuda_graphs ← tp_worker.py:351 ← scheduler.py:856 init_all_cuda_graphs
• Preceding warning (may be unrelated): aiter/tuned_gemm.py:420: bgemm_internal_cublaslt error: HIPBLAS_STATUS_INTERNAL_ERROR ... Will attempt to recover by calling cublas instead.
• Reproducibility: identical fault on all 4 TP ranks, every completed scheduled run 2026-06-12 → 2026-06-23.
• Runner/arch: GPU_ARCH=mi35x / gfx950; aiter dev image v0.1.14rc1.dev240+g7d604afe5 unchanged across the failing window.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [MEDIUM] aiter custom_all_reduce.flush_graph_buffers → _gather_ipc_meta performs IPC-handle/event operations on a stream that is mid-capture, which HIP forbids (hipErrorCapturedEvent / hipErrorStreamCaptureUnsupported), aborting the scheduler.
  • Supporting: the Python abort stack tops out exactly at custom_all_reduce.py:338 _gather_ipc_meta invoked from flush_graph_buffers (line 305) inside capture (line 505), reached from sglang's decode_cuda_graph_runner.capture; the HIP error message names a capturing-stream violation. Consistent across all 4 ranks and 7 runs.
  • Disconfirming: the error is reported by the ProcessGroupNCCL watchdog thread, so the abort may be a secondary effect of an earlier async HIP error (the HIPBLAS_STATUS_INTERNAL_ERROR GEMM fallback prints immediately before the capture); the true first fault could be the bgemm path rather than _gather_ipc_meta. Stack proximity ≠ causation.
  • How to verify: launch the failing command with AMD_SERIALIZE_KERNEL=3 to force synchronous error attribution; if the fault still resolves to _gather_ipc_meta during capture, the hypothesis holds.
• [LOW] Pre-existing AMD port gap for this model config (mxfp4 Qwen3-235B, tp4/ep2) on gfx950 — capture path never worked on this runner.
  • Supporting: never-passed in the entire queryable window; no sglang/aiter version change correlates with an onset (it was already red on the oldest run checked).
  • Disconfirming: cannot see a green baseline to bound a regression window, so "never worked" vs "broke before 2026-06-12" is unresolved.
  • How to verify: extend the lookback past 2026-06-12 to find any green run; if none exists, treat as an un-ported config rather than a regression.

In-flight Fix Check

• In-flight fix: 🟡 candidate #27650 "[AMD] Fix PCG capture fail with aiter fused all-reduce+RMSNorm" (open since 2026-06-09, updated 2026-06-23) — makes aiter's fused all-reduce capture-safe during piecewise CUDA graph capture on gfx950/MXFP4. Caveat: that PR targets --enable-aiter-allreduce-fusion + --enforce-piecewise-cuda-graph (KimiK2.5), whereas this job's launch command sets neither flag, so it may not cover this exact path. Worth checking whether it also fixes custom_all_reduce.flush_graph_buffers.
  • Search run: is:pr is:open "capturing stream" and is:pr is:open mxfp4 qwen3 on sgl-project/sglang.
• No open PR found that explicitly names _gather_ipc_meta / flush_graph_buffers capture-safety for the non-fusion path.

Suggested Triage Steps

• Re-run the failing command with AMD_SERIALIZE_KERNEL=3 (and optionally TORCH_USE_HIP_DSA) in container rocm/sgl-dev:v0.5.13.post1-rocm700-mi35x-20260623 on an MI35x/gfx950 box to confirm whether _gather_ipc_meta is the first fault or a downstream watchdog symptom of the HIPBLAS_STATUS_INTERNAL_ERROR GEMM.
• Verify whether #27650 (or a sibling aiter patch) makes custom_all_reduce.flush_graph_buffers/_gather_ipc_meta capture-safe; if so, ping reviewers and re-run this job against that branch rather than opening a duplicate.
• As an A/B isolation step, re-run with the custom all-reduce path disabled (e.g. --disable-custom-all-reduce) to confirm the crash is gated on the aiter custom all-reduce capture path.
• Extend history before 2026-06-12 to determine whether this config ever passed; if never, classify as an un-ported AMD config and consider disabling/quarantining the test on gfx950 until the capture-safety fix lands.

Status (factual, no priority assignment)

• Persistence: Failing on every completed scheduled run since at least 2026-06-12 (7/7 completed runs red with the identical fault); no green run in the queryable window.
• Scope: 1 job (nightly-8-gpu-mi35x-qwen3-235b-mxfp4) in nightly-test-amd.yml; all 4 TP ranks abort identically. Blocks the Qwen3-235B-A22B MXFP4 nightly accuracy/serving signal on MI35x.
• In-flight fix: 🟡 candidate #27650 (open, may not cover the non-fusion custom_all_reduce path) — verify coverage before relying on it.

nightly-accuracy-2-gpu — failed step(s): Nightly Test (2-GPU)

Commit Info

• sglang: ed26a109
• aiter: 7d604afe5 (source: dev image — [CI-AITER-CHECK] reports v0.1.14rc1.dev240+g7d604afe5, "Dev/patched version detected → skipping rebuild")

Failed Tests

All failures are in the single unittest method test_gsm8k_all_models (TestNightlyGsm8KEval), which loops over models. Reported per-model:

Test File	Test Function (model)	Error	Log
test/registered/amd/accuracy/mi30x/test_gsm8k_eval_amd.py	test_gsm8k_all_models (Qwen/Qwen2-57B-A14B-Instruct, TP=2)	TimeoutError: Server failed to start within the timeout period (raised the unittest error; preceded by ~50 min of scheduler/tokenizer watchdog soft-timeouts)	Log
test/registered/amd/accuracy/mi30x/test_gsm8k_eval_amd.py	test_gsm8k_all_models (mistralai/Mixtral-8x7B-Instruct-v0.1, TP=2)	gsm8k score 0.289 < threshold 0.69 + server hang	Log
test/registered/amd/accuracy/mi30x/test_gsm8k_eval_amd.py	test_gsm8k_all_models (mistralai/Mistral-7B-Instruct-v0.3, TP=1)	gsm8k score 0.365 < threshold 0.47	Log

The job exited via ✗ FAILED: …test_gsm8k_eval_amd.py returned exit code 1 → First attempt failed with exit code 255.

Failure Summary

The accuracy suite ran 7 of its ~18 models before the unittest raised TimeoutError while launching the Qwen/Qwen2-57B-A14B-Instruct server (TP=2), aborting the test (Ran 1 test in 5460.838s, FAILED (errors=1)). The abort was preceded by ~50 minutes of repeated TokenizerManager/Scheduler watchdog soft-timeouts (watchdog_timeout=300, soft=True) on TP0/TP1 spanning 19:44–20:34 UTC, indicating a live server hang during/after the Mixtral-8x7B-Instruct-v0.1 stage. Two models were additionally below their gsm8k thresholds (Mistral-7B-Instruct-v0.3 0.365<0.47; Mixtral-8x7B 0.289<0.69), but these are long-standing sub-threshold failures, not today's regression.

Regression Status

• Qwen/Qwen2-57B-A14B-Instruct server-start hang/timeout: New regression (first observed today, 2026-06-23). It passed in all recent runs.
• test_gsm8k_eval_amd.py (file-level): Known recurring failure — the file has been red every scheduled run Jun 20–23. The chronic culprits are sub-threshold Mistral-7B-Instruct-v0.3 and Mixtral-8x7B-Instruct-v0.1 (and borderline gemma-2-27b-it).

History of test_gsm8k_eval_amd.py in nightly-accuracy-2-gpu (branch main, event=schedule, completed runs only). Per-model status for the two TP=2 models relevant today:

Date	Run	File Status	Qwen2-57B-A14B	Mixtral-8x7B	Mistral-7B-v0.3
Jun 23	28046906309	❌ Failed	❌ Server start TimeoutError (new)	❌ 0.289 + hang	❌ 0.365
Jun 22	27975987522	❌ Failed	✅ 0.795	❌ 0.301	❌ 0.364
Jun 21	27912890405	❌ Failed	✅ 0.812	❌ 0.294	❌ 0.363
Jun 20	27879266774	❌ Failed	(not reached — file failed)	❌ (run truncated)	❌ 0.370

Note: today's run only reached 7 models (vs 18 on Jun 22) because the hang/abort consumed the per-file budget; FP8 and Qwen3 models did not run today and are not comparable.

Last known passing of Qwen2-57B-A14B-Instruct: Jun 22 (43b1fb95f6). First failing: today (ed26a109).

Failure Cluster

Server hang → watchdog soft-timeout cascade → next-model server-start TimeoutError during nightly accuracy eval

Facts (verified observations)

• Traceback: test_gsm8k_eval_amd.py:258 → popen_launch_server_wrapper (:139) → test_utils.py:1028 popen_launch_server → raise TimeoutError("Server failed to start within the timeout period") (log line ~22052).
• The unittest collapsed to one method: Ran 1 test in 5460.838s / FAILED (errors=1).
• TP0/TP1 Scheduler watchdog timeout and TokenizerManager watchdog timeout (watchdog_timeout=300, self.soft=True) repeat every ~5 min from 19:44:05 to 20:34:16 UTC (~10 cycles). A scheduler emitting watchdog timeouts means a server was running and stuck, not merely failing to start.
• Earlier Health check failed. Server couldn't get a response from detokenizer for last 20 seconds appeared at 19:10 (Llama-3.1-8B) and 19:23 (DeepSeek) — but those models PASSED, so transient detokenizer health-check warnings are not by themselves fatal.
• Per-model results today: Llama-3.1-8B 0.818✅, Mistral-7B-v0.3 0.365❌, DeepSeek-Coder-V2-Lite 0.814✅, gemma-2-27b-it 0.861✅, Llama-3.1-70B 0.927✅, Mixtral-8x7B 0.289❌, Qwen2-57B ERROR.
• AITER unchanged from a dev image (7d604afe5); not rebuilt.

Hypothesised Causes (with confidence + disconfirming evidence)

• [LOW] Unclean teardown of the hung Mixtral-8x7B server (TP=2) left GPU/process state that prevented the subsequent Qwen2-57B server from starting within the launch timeout.
  • Supporting: TP0/TP1 scheduler watchdog soft-timeouts run continuously for ~50 min and end right before the Qwen2-57B TimeoutError; the symptom is a hang, not a crash/stack-trace in model code.
  • Disconfirming: log buffering makes the exact server (Mixtral vs Qwen2) owning the watchdog window ambiguous; no explicit "GPU in use"/OOM line was captured.
  • How to verify: re-run test_gsm8k_eval_amd.py on MI325 2-GPU; if Qwen2-57B starts cleanly when run in isolation, the issue is cross-model teardown/flake, not a Qwen2-specific regression.
• [LOW] A commit in the regression window 43b1fb95f6..ed26a109 (44 commits) regressed TP=2 server bring-up or model forward stability. Closest keyword match: 6cd8d286 "Vectorize _create_custom_4d_mask in CustomQwen2Decoder" (#27527).
  • Supporting: it touches the Qwen2 decoder path and lands in-window.
  • Disconfirming: the failure is a server-start timeout, not an inference/accuracy fault — a forward-pass mask change would not normally block server startup; and the watchdog hang began during the Mixtral stage (a different model), so a Qwen2-only change cannot explain the preceding hang.
  • How to verify: bisect 43b1fb95f6..ed26a109; specifically A/B 6cd8d286 by launching Qwen2-57B-A14B-Instruct TP=2 with/without the revert.
• No hypothesis reaches MEDIUM. A single hang-then-startup-timeout with no captured stack trace in model code is consistent with a flake/infra/teardown issue as much as a code regression; needs one re-run to disambiguate before bisecting.

In-flight Fix Check

• In-flight fix: ❌ none found — is:pr is:open Qwen2-57B nightly returned 0 results. Open AMD-nightly-accuracy PRs (e.g. #26730 "test(amd): run full GSM8K for accuracy checks") concern test config, not this server-start hang.

Suggested Triage Steps

• Re-run nightly-accuracy-2-gpu (or just test_gsm8k_eval_amd.py) once. If Qwen2-57B-A14B-Instruct passes, classify today as a flake/teardown hang and watch the next scheduled run.
• If it recurs: launch Qwen2-57B-A14B-Instruct standalone (sglang serve --model-path Qwen/Qwen2-57B-A14B-Instruct --tp 2 --trust-remote-code) on MI325 2-GPU to confirm whether startup itself is broken vs. only-after-Mixtral.
• If standalone startup hangs, bisect 43b1fb95f6..ed26a109 (44 commits); A/B 6cd8d286 first given its Qwen2-decoder touch (low prior).
• Separately (pre-existing, not today's regression): the chronic sub-threshold Mistral-7B-Instruct-v0.3 (~0.36 vs 0.47) and Mixtral-8x7B (~0.29 vs 0.69) need their own investigation or threshold review — they have failed every run in the window.

Status (factual)

• Persistence: test_gsm8k_eval_amd.py red every scheduled run Jun 20–23. The Qwen2-57B server-start hang/timeout is new today (Jun 23); prior 2 completed runs had it passing.
• Scope: single job nightly-accuracy-2-gpu in nightly-test-amd.yml. The hang also truncated today's run to 7/18 models, so FP8/Qwen3 coverage was lost today.
• In-flight fix: ❌ none found in open PRs.

nightly-accuracy-2-gpu-vlm — failed step(s): Nightly Accuracy Test (2-GPU VLM MMMU)

Commit Info

• sglang: ed26a109ee
• aiter: 7d604afe5fa7efba63c0dce323b95d9daf2db112 (source: Dockerfile default; dev image v0.1.14rc1.dev240+g7d604afe5 preserved, AITER_COMMIT_OVERRIDE empty)

Failed Tests

All failures are subTests of the single test function test_mmmu_vlm_models in test/registered/amd/accuracy/mi30x/test_vlms_mmmu_eval_amd.py. Job: nightly-accuracy-2-gpu-vlm.

Test File	Test Function (subTest model)	Error	Log
test/registered/amd/accuracy/mi30x/test_vlms_mmmu_eval_amd.py	test_mmmu_vlm_models (deepseek-ai/deepseek-vl2-small)	TimeoutError: Server failed to start within the timeout period — caused the unittest ERROR	Log
test/registered/amd/accuracy/mi30x/test_vlms_mmmu_eval_amd.py	test_mmmu_vlm_models (Qwen/Qwen2-VL-7B-Instruct)	MMMU score 0.280 < threshold 0.30	Log
test/registered/amd/accuracy/mi30x/test_vlms_mmmu_eval_amd.py	test_mmmu_vlm_models (Qwen/Qwen2.5-VL-7B-Instruct)	MMMU score 0.280 < threshold 0.33	Log
test/registered/amd/accuracy/mi30x/test_vlms_mmmu_eval_amd.py	test_mmmu_vlm_models (Qwen/Qwen3-VL-30B-A3B-Instruct)	MMMU score 0.260 < threshold 0.29	Log
test/registered/amd/accuracy/mi30x/test_vlms_mmmu_eval_amd.py	test_mmmu_vlm_models (openbmb/MiniCPM-o-2_6)	MMMU score 0.290 < threshold 0.32	Log

Models that passed today: OpenGVLab/InternVL2_5-2B (0.290/0.29), openbmb/MiniCPM-v-2_6 (0.310/0.25). Models never reached today (Janus-Pro-7B, Kimi-VL-A3B-Instruct, MiMo-VL-7B-RL, DeepSeek-OCR-2) because the deepseek-vl2-small timeout aborted the loop.

Failure Summary

The job failed with FAILED (errors=1). The error is a TimeoutError raised by popen_launch_server (python/sglang/test/test_utils.py:1028) when the deepseek-ai/deepseek-vl2-small server did not finish starting within the launch timeout. A py-spy dump of the scheduler shows it stuck downloading model weights from HuggingFace Hub. Separately, four VLMs (Qwen2-VL, Qwen2.5-VL, Qwen3-VL-30B, MiniCPM-o) scored just under their MMMU thresholds — a pattern unchanged from prior runs.

Regression Status

Mixed: persistent accuracy failures + one new infrastructure failure today.

• Accuracy threshold failures (Qwen2-VL / Qwen2.5-VL / Qwen3-VL-30B / MiniCPM-o): Known recurring failure, with near-identical scores every run.
• deepseek-vl2-small server timeout: New today — in the prior two completed runs its server started in ~120 s and it failed only on the accuracy threshold (score 0.280). Today it never started.

Recent history of test_mmmu_vlm_models in job nightly-accuracy-2-gpu-vlm (same branch main, event=schedule, completed runs only):

Date	Run	Test File Status	Failing subTests	Note
Jun 23	28046906309	❌ Failed (errors=1)	Qwen2-VL, Qwen2.5-VL, Qwen3-VL-30B, MiniCPM-o (accuracy) + deepseek-vl2-small (server timeout)	timeout aborted run early
Jun 22	27975987522	❌ Failed (failures=1)	Qwen2-VL (0.28), Qwen2.5-VL (0.28), Qwen3-VL-30B (0.26), MiniCPM-o (0.28), deepseek-vl2-small (0.28, accuracy)	server started in 121 s
Jun 21	27912890405	❌ Failed (failures=1)	same 4 accuracy + deepseek-vl2-small (accuracy)	server started fine
Jun 20	27879266774	❌ Failed	(accuracy cluster)	—

No green run for this test was found in the queryable window (Jun 12–23; Jun 16–19 runs were cancelled). Lookback range: last 12 scheduled runs.

Failure Cluster

Two distinct clusters:

1. AMD VLM MMMU accuracy below threshold (persistent) — multiple VLMs scoring ~0.26–0.29 vs thresholds 0.29–0.33.
2. VLM server startup timeout — HF Hub weight download hang (new today) — deepseek-vl2-small.

Facts (verified observations)

• Verbatim error (log tail): TimeoutError: Server failed to start within the timeout period, raised at popen_launch_server → test/registered/amd/accuracy/mi30x/test_vlms_mmmu_eval_amd.py:216 → python/sglang/test/test_utils.py:1028.
• py-spy dump of Process 24594: sglang::scheduler shows the MainThread blocked in snapshot_download (huggingface_hub/_snapshot_download.py:467) and an active worker thread in xet_get (huggingface_hub/file_download.py:566), reached via load_weights (deepseek_vl2.py:246) → download_weights_from_hf (model_loader/weight_utils.py:593). The scheduler/TokenizerManager hit soft watchdog timeouts (watchdog_timeout=300, soft=True) while the download was in progress.
• This run’s popen_launch_server exception is raised outside the per-model try block (test_vlms_mmmu_eval_amd.py:216), so it propagates out of the model loop, skips _check_results, and yields errors=1 (vs failures=1 in prior runs). It also prevented the remaining 4 models from running.
• Accuracy scores are essentially identical across runs: Qwen2-VL 0.280, Qwen2.5-VL 0.280, Qwen3-VL-30B 0.260, MiniCPM-o 0.280–0.290 — all just below their per-model thresholds. Random-chance baseline for MMMU multiple-choice is ~0.25.
• aiter version unchanged (Dockerfile-default dev build, no override).

Hypothesised Causes (with confidence)

• [HIGH] The deepseek-vl2-small failure is a HuggingFace Hub weight-download hang (xet backend), not an sglang code regression.
  • Supporting: py-spy stack shows the scheduler blocked in huggingface_hub snapshot_download/xet_get during initial weight load; the same model started in ~120 s on Jun 21 and Jun 22 with the same code paths; only the startup phase (download), not inference, is affected.
  • Disconfirming: no second run today to confirm it's transient; cannot fully exclude a model-loader change in the Jun 22→23 window without a re-run.
  • How to verify: re-run the job (or just this model) on an MI30x runner; if the server starts within timeout, confirms transient HF download. Optionally pre-warm/pre-cache the HF snapshot.
• [LOW] The persistent accuracy failures reflect thresholds calibrated above the harness's stable score floor (100-sample MMMU, max_tokens=30) rather than a fresh model regression.
  • Supporting: scores are stable run-over-run and several other VLMs pass; failing models sit 0.01–0.05 below threshold; eval harness clearly works (passes for InternVL2_5-2B, MiniCPM-v, Janus-Pro, Kimi-VL, MiMo-VL).
  • Disconfirming: scores near random (~0.28) for capable 7B+ VLMs could also indicate a genuine multimodal-accuracy problem on AMD; no green baseline exists in the window to prove these ever passed.
  • How to verify: bisect for when these models last passed (if ever) by extending history before Jun 12; compare the same models' MMMU scores on NVIDIA nightly to separate AMD-specific degradation from threshold miscalibration.

No hypothesis for the accuracy cluster reaches MEDIUM; root cause needs history beyond the queryable window and an AMD-vs-NVIDIA score comparison.

In-flight Fix Check

• VLM MMMU accuracy: In-flight fix: ❌ none found — searched open PRs for VLM+MMMU (15 results, none address AMD MMMU accuracy thresholds) and mmmu+accuracy+amd (5 results, none relevant).
• deepseek-vl2-small startup/download: In-flight fix: ❌ none found — searched open PRs for deepseek-vl2 (6 results: #24474, #24622, #24624, #21301 — feature/refactor work, none address HF download timeout).

Suggested Triage Steps

• Re-run job 28046906309 (or just deepseek-ai/deepseek-vl2-small) on an MI30x runner to confirm the server timeout is a transient HF Hub/xet download hang; if it recurs, investigate runner network/HF cache or pre-stage the snapshot.
• Consider making per-model server-launch failures non-fatal to the loop (move popen_launch_server inside the per-model try in test_vlms_mmmu_eval_amd.py:216) so one model's infra hang doesn't skip the remaining 4 models — a test-harness robustness improvement, maintainer's call.
• For the accuracy cluster: pull MMMU history before Jun 12 to find any green baseline for Qwen2-VL/Qwen2.5-VL/Qwen3-VL-30B/MiniCPM-o; compare against NVIDIA nightly MMMU scores to decide whether thresholds are miscalibrated or there is a real AMD multimodal regression.

Status

• Persistence: Accuracy cluster failing every completed scheduled run for ≥4 days (since at least Jun 20); deepseek-vl2-small server timeout first seen today (Jun 23).
• Scope: Single job nightly-accuracy-2-gpu-vlm in nightly-test-amd.yml; 5 of 7 reached models failing, plus 4 models not exercised due to the early abort.
• Blocked work: Reliable AMD VLM MMMU accuracy signal; the new timeout further reduces coverage by cutting the run short.
• In-flight fix: ❌ none found for either cluster.

nightly-accuracy-8-gpu-mi35x-deepseek-v32 — failed step(s): Accuracy Test MI35x (8-GPU DeepSeek-V3.2)

Commit Info

• sglang: ed26a109
• aiter: 7d604afe5 (v0.1.14rc1.dev240+g7d604afe5, source: dev image preserved — [CI-AITER-CHECK] reported "Dev/patched version detected → skipping rebuild")

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi35x/test_deepseek_v32_eval_mi35x.py	test_deepseek_v32_accuracy (TestDeepSeekV32EvalMI35x)	AssertionError: Failed models: ['deepseek-ai/DeepSeek-V3.2 (basic)'] — GSM8K accuracy=0.925 threshold=0.93	Log

Failure Summary

The MI35x DeepSeek-V3.2 (basic, TP=8) GSM8K accuracy eval scored 0.925 against a hard threshold of 0.93, tripping the AssertionError at test_deepseek_v32_eval_mi35x.py:245. With num_questions=200, 0.925 = 185/200 correct vs the required 186/200 — a miss of exactly one question. The server ran cleanly to completion (all 200 /generate calls returned 200 OK); this is purely a benchmark-score shortfall, not a crash.

Regression Status

New regression today (first observed failure for this test file). The job passed in each of the last three completed scheduled runs, but with a small and declining margin above the 0.93 threshold.

Recent history of test_deepseek_v32_eval_mi35x.py (test_deepseek_v32_accuracy) in job nightly-accuracy-8-gpu-mi35x-deepseek-v32 (branch main, event schedule, completed runs only):

Date	Run	Job	Test File Status	Accuracy	Error
Jun 24	28046906309	job	❌ Failed	0.925	below 0.93 by 1 question
Jun 22	27975987522	job	✅ Passed	0.935	—
Jun 21	27912890405	job	✅ Passed	0.935	—
Jun 20	27879266774	job	✅ Passed	0.940	—

First observed failure for this test file: 2026-06-24. Last known passing: 2026-06-22 (acc 0.935). Lookback window: last 4 completed scheduled runs.

Failure Cluster

DeepSeek-V3.2 MI35x GSM8K accuracy just below threshold (borderline eval)

Facts (verified observations, no interpretation)

• Verbatim log: accuracy=0.925 threshold=0.93 ❌ FAIL (job log, line 23825), then AssertionError: Failed models: ['deepseek-ai/DeepSeek-V3.2 (basic)'] (line 23837), Ran 1 test in 2582.043s, exit code 255.
• Threshold is a hard accuracy_threshold=0.93 for the basic variant (test_deepseek_v32_eval_mi35x.py:71); num_questions=200 (line 174), so each question is worth 0.005 — the pass/fail boundary is 186 vs 185 correct.
• Last 4 completed runs (incl. today): accuracy sequence 0.940 → 0.935 → 0.935 → 0.925. The margin over threshold has shrunk each run (+0.010, +0.005, +0.005, −0.005).
• The server completed the full 200-request GSM8K sweep with no errors (all /generate returned 200 OK; decode/prefill batches normal). No crash, no GPU fault, no kernel error in the log.
• aiter version is the dev image build, unchanged from the Dockerfile dev tag ([CI-AITER-CHECK] markers, lines 4183-4189).
• Commit window since last pass 43b1fb95..ed26a109: 44 commits.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [MEDIUM] Benchmark statistical noise on a tight threshold — the eval naturally fluctuates within ±1–2 questions and today landed one question under the 0.93 line.
  • Supporting: 4-run accuracy spread (0.925–0.940) is a ±0.0075 band around the threshold; GSM8K with only 200 questions has ~±0.018 (1σ) sampling noise, so a 1-question swing is well within expected variance. The run produced no errors and completed normally.
  • Disconfirming: the trend is monotonically decreasing (0.940→0.935→0.935→0.925) rather than random scatter, which could hint at a small real drift rather than pure noise.
  • How to verify: re-run the job at the same head SHA 2–3×; if it oscillates around 0.925–0.940 the threshold is simply too tight for 200 questions.
• [LOW] A sglang change in the 44-commit window subtly shifted DeepSeek-V3.2 numerics (attention/MoE/quant path). Candidate commits touching relevant paths: d70726a3 (revert of FA4 page_size auto-force) and 6c212a5d (the original FA4 page_size change), 8dc27f63 (MoE quant-arg dedup).
  • Supporting: these touch attention-backend / MoE quant config that DeepSeek-V3.2 exercises; they land in the regression window.
  • Disconfirming: none of them is DeepSeek-V3.2- or MI35x-specific; the FA4 change was reverted in the same window; a 0.5% accuracy move is far smaller than any of these would be expected to cause if they functionally broke the model. No stack-trace or numerical-mismatch evidence ties any commit to the score drop.
  • How to verify: bisect 43b1fb95..ed26a109 re-running the MI35x basic eval, but only after confirming the result is repeatable (rule out noise first).

No hypothesis reaches HIGH confidence; the most likely explanation is benchmark variance on a threshold that leaves only a 1-question margin.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for DeepSeek-V3.2 accuracy (39 results, none addressing the MI35x nightly accuracy threshold) and deepseek_v32 mi35x (0 results). No PR adjusting the 0.93 threshold or fixing a V3.2 accuracy regression is open. API: search/issues?q=repo:sgl-project/sglang+is:pr+is:open+DeepSeek-V3.2+accuracy.

Suggested Triage Steps

• Re-run nightly-accuracy-8-gpu-mi35x-deepseek-v32 at the same SHA ed26a109 2–3× to establish whether 0.925 is reproducible or noise. If it oscillates above/below 0.93, the threshold is too tight for 200 questions — consider raising GSM8K_NUM_QUESTIONS to reduce variance, or adding a small tolerance margin.
• If the score is consistently ≤0.925 on re-run, bisect 43b1fb95..ed26a109, starting with the attention/MoE commits (d70726a3, 6c212a5d, 8dc27f63), re-running the MI35x basic eval per step.
• Cross-check the sister NVIDIA DeepSeek-V3.2 accuracy job for the same window to see if the drift is AMD-specific or model-wide.

Status (factual, no priority assignment)

• Persistence: First failing run 2026-06-24; passed the prior 3 completed scheduled runs (Jun 20–22).
• Scope: 1 job (nightly-accuracy-8-gpu-mi35x-deepseek-v32) in 1 workflow (nightly-test-amd.yml). The -mtp sister accuracy job and the perf jobs passed in recent runs.
• Blocked work: Blocks the MI35x DeepSeek-V3.2 basic accuracy signal only; no crash or infra impact.
• In-flight fix: ❌ none found in open PRs.

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-06-24 23:06 UTC)

[amd-bot]

pr-test-amd.yml — 28 failure(s)

Run started (UTC): 2026-06-23 00:59 UTC
Last scanned (UTC): 2026-06-24 23:07 UTC
Commits: sglang d5c566e

Summary

Counts: 28 failing jobs · 13 distinct test failures · 11 clusters · 3 persistent (≥2 days, A/B/C) · 4 new today (D, K, runner-infra F/G)

#	Cluster	Job (latest)	Test File	Test Function	Status	Hypothesis (confidence)
1	PD KV-transfer timeout (MI35x)	stage-b…mi35x-disagg	test/registered/amd/disaggregation/test_disaggregation_basic.py	TestDisaggregationAccuracy.test_gsm8k	regression since 06-22, 8 jobs	[LOW] RDMA NIC fault; in-flight #29127
2	PD KV-transfer timeout (MI35x)	stage-b…mi35x-disagg	test/registered/amd/disaggregation/test_disaggregation_pp.py	TestDisaggregationDecodePPAccuracy.setUpClass	regression since 06-22, 8 jobs	[LOW] RDMA mem-reg ENOMEM [12] / mem-unbalanced
3	PD KV-transfer timeout (MI35x)	stage-b…mi35x-disagg	test/registered/amd/disaggregation/test_mori_transfer_engine_e2e.py	setUpClass	new 06-23, 1 job (HIP OOM cascade)	[MEDIUM] GPU-cleanup cascade; #27093 partial
4	ROCm FP8 _scaled_mm HIPBLAS unsupported	multimodal-gen-2gpu (1)	multimodal_gen/test/server/test_server_2_gpu.py	test_diffusion_generation[flux2_modelopt_fp8_tp2_t2i]	never-passed, 8 jobs	[HIGH] fp8_utils.py:1695 torch._scaled_mm; fix #28889
5	Gated HF repo 403 (ideogram-4-fp8)	multimodal-gen-2gpu (1)	multimodal_gen/test/server/test_server_2_gpu.py	test_diffusion_generation[ideogram4_fp8_tp2_t2i]	persistent, 4 jobs	[FACT] GatedRepoError 403 — model-access/data, not code
6	VLM MMMU benchmark timeout (Qwen2.5-VL-3B)	stage-b-1gpu-nondeterm	test/registered/models/test_vlm_models.py	TestVLMModels.test_vlm_mmmu_benchmark	new 06-24, 2 jobs	[HIGH] model swap 0fc815aa (#29095) over 1800s budget
7	mori build — invalid ELF header (libabsl)	multimodal-gen-2gpu (2)	N/A (build)	N/A	intermittent, 2 jobs	[LOW] corrupted system lib in image
8	Self-hosted runner lost communication	multimodal-gen-1gpu (1)	N/A (infra)	N/A	transient, 2 jobs	[no candidate ≥ MEDIUM] infra
9	1-GPU diffusion server abrupt termination	multimodal-gen-1gpu (1)	multimodal_gen/test/server/test_server_1_gpu.py	unidentifiable (BlobNotFound)	new, 2 jobs	[LOW] process crash/OOM during warmup
10	Diffusion 2-GPU warmup hang → port 5555 cascade	multimodal-gen-2gpu (0)	multimodal_gen/test/server/test_server_2_gpu.py	test_diffusion_generation[zimage/flux_2/ltx…]	flaky 1 run	[LOW] warmup hang leaks scheduler port
11	RCCL allreduce SIGSEGV (8-GPU)	stage-c-8gpu (0)	scripts/ci/amd/test_rccl_multi_gpu.py	test_rccl_allreduce	flaky 1/5	[no candidate ≥ MEDIUM] transient
12	FLUX.1-dev T2I hang → 90-min timeout	multimodal-gen-1gpu (3)	multimodal_gen/test/cli/test_generate_t2i_perf.py	TestFlux_T2V.test_single_gpu	flaky (passed before & after)	[no candidate ≥ MEDIUM] — code-regression disconfirmed
13	Decode CUDA-graph-capture hang	stage-b-1gpu-small (11)	test/registered/scheduler/test_priority_scheduling.py	TestPriorityScheduling.setUpClass	new 06-24, 1 job	[MEDIUM] MI325 host GPU stall in hipDeviceSynchronize

Cluster: PD KV-transfer timeout (MI35x) — rows 1-3

• Affected: 8 stage-b-test-large-8-gpu-mi35x-disaggregation-amd jobs across runs 27994433780, 28028218911, 28009346928, 28048998470, 28067315750, 28081940667, 28099674087, 28121318394.
• Shared facts: 200/200 /generate → HTTP 500; KVTransferError … timed out after 300.0s in KVPoll.WaitingForInput; pp setUpClass SIGKILL (-9) after rdma_context.cpp:243 Failed to register memory … Cannot allocate memory [12] + "memory capacity unbalanced". aiter unchanged across pass→fail. Regression window 4923bb93..adc203df (last green 06-22 14:15).
• Top hypothesis: [LOW] faulty RDMA NIC rdma5 / runner RLIMIT_MEMLOCK; sharp pass→fail at one window argues against pure flake. Disconfirming: no code-level evidence any disaggregation default flipped in the window.
• In-flight fix: ✅ #29127 (isolate faulty rdma5); ⚠️ #27093 (mori PD) possibly relevant to row 3.
• Suggested triage: confirm rdma5 health on linux-mi35x-gpu-8.fabric; check ulimit -l; run test_mori_transfer_engine_e2e.py in isolation to confirm the OOM is a cleanup cascade.

Cluster: ROCm FP8 _scaled_mm HIPBLAS unsupported — row 4

• Affected: 8 multimodal-gen-test-2-gpu-amd (…, 1) jobs, every completed schedule run Jun 21→24 (never green).
• Shared facts: server-side RuntimeError: CUDA error: HIPBLAS_STATUS_NOT_SUPPORTED at fp8_utils.py:1695 torch._scaled_mm via modelopt_quant.py:425; AMD-only (gfx942); other 10 scenarios in shard pass.
• Top hypothesis: [HIGH] hipBLASLt lacks support for the ModelOpt FP8 (e4m3fn) scale layout. Disconfirming: not a regression — missing-ROCm-support gap.
• In-flight fix: ✅ #28889 (dequant→bf16 fallback in apply_fp8_linear, open since 06-22) — touches the exact file; chase reviewers.

Cluster: Gated HF repo 403 (ideogram-4-fp8) — row 5

• Affected: 4 shard-1 jobs (co-located with row 4). huggingface_hub.errors.GatedRepoError: 403 on ideogram-ai/ideogram-4-fp8 at server startup → setup ERROR. Not a code defect — CI lacks access/license to the gated repo.
• In-flight fix: ❌ none — needs HF token/license grant or deselecting the scenario.

Other clusters (rows 6-13, brief)

• Row 6 VLM MMMU timeout: [HIGH] model swap to Qwen/Qwen2.5-VL-3B-Instruct (#29095) drives large untuned AMD GEMMs over the 1800s budget. In-flight ❌ none. Triage: time the Qwen MMMU eval on a quiet MI325; raise budget or tune.
• Row 7 mori build (invalid ELF): libabsl_*.so invalid ELF header corrupts the mori C++ build; passed in adjacent runs → corrupted/partially-written system lib in image. ❌ none. Triage: re-run; verify image digest.
• Rows 8-9 runner infra: "runner lost communication" (rows 1/18) and 1-GPU diffusion server abrupt termination with BlobNotFound logs (rows 16/26) — no clean test failure recorded; treat as infra/process-crash until reproduced. Triage: re-run on different runners; check GPU/dmesg health.
• Rows 10-12 flakes: diffusion warmup port-5555 cascade (1 run), RCCL allreduce SIGSEGV (1/5), FLUX.1-dev 90-min hang (passed before & after) — all single-run flakes; re-run to confirm.
• Row 13 decode graph-capture hang: [MEDIUM] MI325 host GPU stall in first decode CUDA-graph capture (passed prior 4 runs in ~150s; preceding heartbeat health-check warnings). ❌ none. Triage: re-run on a clean MI325; if reproduced, bisect 0460f277b7..5e6d7c1615.

Suggested triage order

Rank	Triage step	Targets	Why this first
1	Land/verify #28889, re-run shard-1	row 4 (8 jobs)	Largest cluster with a ready, file-exact fix
2	Confirm/merge #29127; check rdma5 + ulimit -l	rows 1-3 (8 jobs)	Second-largest; in-flight NIC fix exists
3	Grant HF access for ideogram-4-fp8 or deselect	row 5 (4 jobs)	Cheap, non-code; clears 4 setup errors
4	Time Qwen2.5-VL-3B MMMU vs 1800s budget (#29095)	row 6 (2 jobs)	New today; likely needs budget/model revert
5	Re-run infra/flake jobs on clean runners	rows 7-13	Cheap A/B to separate flakes from real regressions

---

Job	Failed Steps	Started
stage-c-test-large-8-gpu-amd (linux-mi325-8gpu-sglang, 1)	N/A	2026-06-23T14:58
multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 1)	Run diffusion server tests (2-GPU)	2026-06-23T14:27
multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 3)	Run diffusion server tests (1-GPU)	2026-06-23T09:55
stage-b-test-large-8-gpu-mi35x-disaggregation-amd (linux-mi35x-gpu-8.fabric)	Run test_disaggregation	2026-06-23T16:16
multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 0)	Run diffusion server tests (2-GPU)	2026-06-23T09:40
multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 1)	Run diffusion server tests (2-GPU)	2026-06-23T09:41
stage-c-test-large-8-gpu-amd (linux-mi325-8gpu-sglang, 0)	Test RCCL multi-GPU communication	2026-06-23T12:18
stage-b-test-large-8-gpu-mi35x-disaggregation-amd (linux-mi35x-gpu-8.fabric)	Run test_disaggregation	2026-06-23T10:06
multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 1)	Run diffusion server tests (2-GPU)	2026-06-23T00:59
stage-a-test-1-gpu-small-amd (linux-mi325-1gpu-sglang)	Install dependencies	2026-06-23T19:17
stage-b-test-large-8-gpu-mi35x-disaggregation-amd (linux-mi35x-gpu-8.fabric)	Run test_disaggregation	2026-06-23T01:20
multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 1)	Run diffusion server tests (2-GPU)	2026-06-23T19:35
stage-b-test-large-8-gpu-mi35x-disaggregation-amd (linux-mi35x-gpu-8.fabric)	Run test_disaggregation	2026-06-23T19:32
multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 1)	Run diffusion server tests (2-GPU)	2026-06-24T01:42
stage-b-test-large-8-gpu-mi35x-disaggregation-amd (linux-mi35x-gpu-8.fabric)	Run test_disaggregation	2026-06-24T02:24
multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 1)	N/A	2026-06-24T02:07
stage-b-test-1-gpu-small-amd (linux-mi325-1gpu-sglang, 11)	Run test	2026-06-24T02:42
multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 1)	N/A	2026-06-24T08:25
stage-b-test-large-8-gpu-mi35x-disaggregation-amd (linux-mi35x-gpu-8.fabric)	Run test_disaggregation	2026-06-24T08:53
multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 1)	Run diffusion server tests (2-GPU)	2026-06-24T10:53
stage-b-test-1-gpu-small-amd-nondeterministic (linux-mi325-1gpu-sglang)	Run test	2026-06-24T17:14
stage-b-test-large-8-gpu-mi35x-disaggregation-amd (linux-mi35x-gpu-8.fabric)	Run test_disaggregation	2026-06-24T18:56
multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 1)	Run diffusion server tests (2-GPU)	2026-06-24T19:57
multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 2)	Install dependencies	2026-06-24T21:20
stage-b-test-1-gpu-small-amd-nondeterministic (linux-mi325-1gpu-sglang)	Run test	2026-06-24T20:31
multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 1)	N/A	2026-06-24T20:24
stage-b-test-large-8-gpu-mi35x-disaggregation-amd (linux-mi35x-gpu-8.fabric)	Run test_disaggregation	2026-06-24T21:06
multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 1)	Run diffusion server tests (2-GPU)	2026-06-24T21:56

Per-Job Analysis

stage-c-test-large-8-gpu-amd (linux-mi325-8gpu-sglang, 1) — failed step(s): N/A

Commit Info

• sglang: 0460f277
• aiter: N/A (job log blob was never uploaded — BlobNotFound — so no [CI-AITER-CHECK] markers are available)

Failed Tests

Test File	Test Function	Error	Log
N/A (runner crash)	N/A	The self-hosted runner lost communication with the server	Job (log unavailable — BlobNotFound)

This is not a test assertion failure. The runner (AMD MI325 8-GPU, linux-mi325-8gpu-sglang) died during the Run test step before any test result could be recorded or the log uploaded.

Failure Summary

The stage-c-test-large-8-gpu-amd (part 1/4) job ran for ~25 min (14:58:47 → 15:23:48 UTC) and then the self-hosted runner lost communication with GitHub mid-test. Step 7 Run test is frozen at in_progress, step 14 Post Checkout code never started, and the job's log blob does not exist on the server. The job's GitHub check annotation states the runner lost communication. This partition normally runs large-model suites (Kimi K2, DeepSeek V3/V3.2, MoRI EP); the exact file being executed at crash time cannot be determined because no log was uploaded.

Regression Status

Infrastructure issue (isolated, non-reproducing so far) — not a code regression.

Recent history of stage-c-test-large-8-gpu-amd (part 1) (same branch main, event=schedule, completed runs only):

Date (UTC)	Run	Job	Run-test Step	Status
Jun 23 13:03	28028218911	part 1	in_progress (frozen)	❌ Runner lost comms
Jun 23 07:20	28009346928	part 1	completed/success	✅ Passed
Jun 23 00:51	27994433780	part 1	completed/success	✅ Passed
Jun 22 19:10	27977371862	part 1	completed/success	✅ Passed
Jun 22 13:56	27958083310	part 1	completed/success	✅ Passed
Jun 22 08:09	27938642367	part 1	completed/success	✅ Passed

Partition 1 passed in all 5 prior scheduled runs. This is the only run where it failed, and the failure mode is a runner disconnect, not a test failure.

Failure Cluster

Self-hosted AMD runner lost communication (mid-test crash, no log uploaded)

Facts (verified observations)

• [FACT] Job check annotation (verbatim): "The self-hosted runner lost communication with the server. Verify the machine is running and has a healthy network connection. Anything in your workflow that terminates the runner process, starves it for CPU/Memory, or blocks its network access can cause this error." (GET /check-runs/82961544956/annotations)
• [FACT] Step 7 Run test status = in_progress / conclusion = None; step 14 Post Checkout code = pending (job never ran post-steps). (GET /actions/jobs/82961544956)
• [FACT] Log download returns BlobNotFound — the runner never uploaded the log archive, consistent with an abrupt death.
• [FACT] Job duration ~25 min (14:58:47 → 15:23:48 UTC) before disconnect.
• [FACT] Same partition (part 1) was ✅ green in the previous 5 scheduled runs on main.
• [FACT] Other failing jobs in this run (multimodal-gen-test-2-gpu-amd, stage-b-…-disaggregation-amd) had their test steps complete with a normal failure conclusion — a different, unrelated signature.

Hypothesised Causes (with confidence)

• [HIGH] Self-hosted runner infrastructure failure — the runner lost contact with GitHub during the test step.
  • Supporting: GitHub's own annotation explicitly reports lost communication; frozen in_progress step + missing post-steps + missing log blob are the canonical signature of this; partition passed in all 5 prior runs so the suite content is healthy.
  • Disconfirming: (none found) — no evidence of a code-level test failure since no test result was ever emitted.
  • How to verify: re-run the job; if it passes on retry, infra is confirmed. Check the MI325 runner host's dmesg/system logs around 2026-06-23 15:23 UTC for OOM-kill, GPU hang, or network drop.
• [LOW] CPU/memory starvation (host OOM) during a large-model test (MoRI EP / Kimi K2 / DeepSeek V3.2 are memory-heavy) starved the runner agent and killed the connection.
  • Supporting: this partition runs the heaviest 8-GPU model suites; the annotation lists CPU/Memory starvation as a possible trigger.
  • Disconfirming: cannot confirm which file ran — no log; the suite has been stable across prior runs, so a deterministic OOM is unlikely.
  • How to verify: inspect host memory/GPU telemetry for the runner at crash time; if a specific large test correlates, watch its peak host-RAM usage.

In-flight Fix Check

• In-flight fix: ❌ none found — this is a transient self-hosted-runner infrastructure event, not a code defect; there is no code PR to chase. Searched scope: open sglang PRs (no relevant test/library keyword applies since no test failed).

Suggested Triage Steps

• Re-run the failed job (part 1); a green retry confirms a transient infra blip.
• Check the linux-mi325-8gpu-sglang runner host logs (dmesg / agent logs / GPU telemetry) around 2026-06-23 15:23 UTC for OOM-kill, GPU reset/hang, or network loss.
• If retries also disconnect on this host, drain and inspect that specific runner — it may be unhealthy.
• No source-code change is indicated; do not bisect commits for this failure.

Status

• Persistence: Single occurrence on 2026-06-23 13:03 UTC scheduled run; partition 1 was green in the previous 5 scheduled runs.
• Scope: 1 job (stage-c-test-large-8-gpu-amd, part 1) on one MI325 8-GPU runner. The other two failures in this run are unrelated normal test failures.
• Blocked work: Blocks the large-model 8-GPU AMD signal for this scheduled run only until re-run.
• In-flight fix: ❌ none found — infrastructure event, no code PR applicable.

multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 1) — failed step(s): Run diffusion server tests (2-GPU)

Commit Info

• sglang: 0460f277
• aiter: 7d604afe5 — v0.1.14rc1.dev240+g7d604afe5 (source: dev image preserved; AITER_COMMIT_OVERRIDE empty, not a scout run)

Failed Tests

Test File	Test Function	Error	Log
sglang/multimodal_gen/test/server/test_server_2_gpu.py	TestDiffusionServerTwoGpu::test_diffusion_generation[flux2_modelopt_fp8_tp2_t2i]	openai.InternalServerError ← server RuntimeError: CUDA error: HIPBLAS_STATUS_NOT_SUPPORTED in torch._scaled_mm	Log
sglang/multimodal_gen/test/server/test_server_2_gpu.py	TestDiffusionServerTwoGpu::test_diffusion_generation[ideogram4_fp8_tp2_t2i]	huggingface_hub.errors.GatedRepoError: 403 Forbidden on ideogram-ai/ideogram-4-fp8 (setup error)	Log

Failure Summary

Shard 1 of multimodal-gen-test-2-gpu-amd ran 12 diffusion server cases; 10 passed/skipped, 2 failed. flux2_modelopt_fp8_tp2_t2i crashes the server during the denoising step's FP8 linear projection with HIPBLAS_STATUS_NOT_SUPPORTED, surfacing to the client as openai.InternalServerError. ideogram4_fp8_tp2_t2i errors at setup because the model repo ideogram-ai/ideogram-4-fp8 returns HTTP 403 (gated/no access) during snapshot download.

Regression Status

Known recurring failure — both tests fail in every completed scheduled run in the queryable window. Not new today.

Recent history of test_server_2_gpu.py (flux2_modelopt_fp8_tp2_t2i + ideogram4_fp8_tp2_t2i) in job multimodal-gen-test-2-gpu-amd (…, 1) (same branch=main, event=schedule, completed runs only):

Date	Run	Job	Test File Status	Failed Function	Error
Jun 23 13:03	28028218911	shard 1	❌ Failed	both	HIPBLAS / 403
Jun 23 07:20	28009346928	shard 1	❌ Failed	both	HIPBLAS / 403
Jun 23 00:51	27994433780	shard 1	❌ Failed	both	HIPBLAS / 403
Jun 22 19:10	27977371862	shard 1	❌ Failed	both	HIPBLAS / 403
Jun 22 13:56	27958083310	shard 1	❌ Failed	both	HIPBLAS / 403

No passing run for these two test cases found in the last ~5 completed scheduled runs (lookback Jun 22 13:56 → Jun 23 13:03 UTC). Shards 0 and 2 of the same job pass; only shard 1 (which holds these two FP8 cases) is red. First-observed failure predates the queryable window — likely Never-passed on AMD for both cases.

Failure Cluster

Two distinct clusters in this job:

1. FP8 modelopt GEMM unsupported on ROCm (HIPBLAS_STATUS_NOT_SUPPORTED in torch._scaled_mm) — flux2 case.
2. Gated HF repo 403 on ideogram-ai/ideogram-4-fp8 — ideogram case (infrastructure/access, not code).

Facts (verified observations)

• flux2 server stack trace (verbatim path): multimodal_gen/runtime/layers/quantization/modelopt_quant.py:425 apply → srt/layers/quantization/fp8_utils.py:1695 apply_fp8_linear → output = torch._scaled_mm(...) → RuntimeError: CUDA error: HIPBLAS_STATUS_NOT_SUPPORTED. The crash first appears during synthetic server warmup (server warmup req (768x768, 2/5 steps) processing failed; "Synthetic server warmup failed; continuing startup"), then again on the actual generation request → client receives openai.InternalServerError.
• flux2 model: black-forest-labs/FLUX.2-dev + transformer lmsys/flux2-dev-modelopt-fp8-sglang-transformer, tp_size=2, attention_backend aiter. Model downloaded & loaded successfully (transformer 15.6 GiB); failure is at compute, not load.
• ideogram error (verbatim): httpx.HTTPStatusError: Client error '403 Forbidden' for url 'https://huggingface.co/ideogram-ai/ideogram-4-fp8/resolve/.../.gitattributes' → huggingface_hub.errors.GatedRepoError: 403 → ValueError: Model or revision not found at ideogram-ai/ideogram-4-fp8. … ensure you have access. Raised at setup (test_server_common.py:191 diffusion_server), so the test never runs generation.
• Reproducibility: identical failure signature for both cases in all 5 completed scheduled runs examined (Jun 22–23). aiter version unchanged across those runs.
• Both test case IDs are defined in python/sglang/multimodal_gen/test/server/gpu_cases.py:568 (flux2) and :580 (ideogram).

Hypothesised Causes (with confidence)

• [HIGH] flux2: torch._scaled_mm (FP8 e4m3fn GEMM) is not supported by the hipBLAS path in this ROCm/MI325 image for the modelopt-fp8 transformer.
  • Supporting: stack trace lands exactly on torch._scaled_mm in fp8_utils.py:1695; error is HIPBLAS_STATUS_NOT_SUPPORTED (a backend-capability error, not a numerical one); only the two FP8 cases on shard 1 fail while non-FP8 cases (flux, wan, ltx) pass; an open PR explicitly titles this exact symptom. NVIDIA path is unaffected.
  • Disconfirming: (none found) — failure is deterministic across 5 runs and isolated to the FP8 code path.
  • How to verify: apply PR #28889 (dequant→bf16 fallback) on a branch and re-run test_server_2_gpu.py::...[flux2_modelopt_fp8_tp2_t2i] on MI325.
• [FACT] ideogram: the failure is an HTTP 403 gated-repo access denial, not an sglang code defect.
  • Supporting: GatedRepoError: 403 raised during HF snapshot download at test setup; no model code executes.
  • Disconfirming: (none) — this is an access/infra condition, observable directly in the log.
  • How to verify: confirm the CI HF token has accepted the license / has access to ideogram-ai/ideogram-4-fp8, or route the case without a gated metadata fetch.

In-flight Fix Check

• flux2 HIPBLAS: In-flight fix: ✅ #28889 "[AMD] Fix FP8 e4m3fn GEMM on ROCm: dequant->bf16 fallback in apply_fp8_linear (fixes flux2 modelopt-fp8 HIPBLAS_STATUS_NOT_SUPPORTED)" (open since 2026-06-22) — directly targets this symptom; chase reviewers, do NOT open a duplicate. (Searched is:pr is:open HIPBLAS_STATUS_NOT_SUPPORTED and flux2 fp8.)
• ideogram 403: In-flight fix: ⚠️ #28225 "Route Ideogram NF4 without gated metadata fetch" (open since 2026-06-15, draft) — related to the gated-fetch path but not yet ready and not FP8-specific. Underlying cause is likely a CI HF-access/token issue; needs maintainer confirmation. (Searched is:pr is:open ideogram.)

Suggested Triage Steps

• flux2: Verify #28889 applies cleanly on 0460f277 and re-run shard 1 in the AMD MI325 2-GPU container; if green, ping reviewers to merge.
• ideogram: Check whether the CI Hugging Face token has accepted the ideogram-ai/ideogram-4-fp8 license / has repo access (the 403 is identical across all runs). If access can't be granted, consider gating/skipping the case on AMD CI until #28225 (or equivalent gated-fetch routing) lands.
• These two cases are isolated to shard 1; shards 0 and 2 of the same job are green, so no broader 2-GPU diffusion regression is implicated.

Status

• Persistence: Both cases failing in every completed scheduled run examined (Jun 22 13:56 → Jun 23 13:03 UTC); no green run for them in the lookback window.
• Scope: 1 job (multimodal-gen-test-2-gpu-amd), shard 1 only, in pr-test-amd.yml. Two independent root causes (FP8 GEMM unsupported; gated-repo 403).
• Blocked work: AMD CI signal for flux2 modelopt-FP8 and ideogram-4-fp8 2-GPU diffusion serving.
• In-flight fix: flux2 → ✅ #28889 (open since 2026-06-22); ideogram → ⚠️ #28225 (draft, since 2026-06-15) + likely a CI HF-access fix needed.

multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 3) — failed step(s): Run diffusion server tests (1-GPU)

Commit Info

• sglang: 743ce88bc5
• aiter: N/A — AITER_COMMIT_OVERRIDE empty; aiter module_aiter_core was JIT-built from the image default. No [CI-AITER-CHECK] markers in this diffusion job log.

Failed Tests

Test File	Test Function	Error	Log
python/sglang/multimodal_gen/test/cli/test_generate_t2i_perf.py	TestFlux_T2V::test_single_gpu	Job timeout — sglang generate (FLUX.1-dev) hung 88 min, no output	Log

Failure Summary

The shard collected and ran exactly one test, test_generate_t2i_perf.py::TestFlux_T2V::test_single_gpu (FLUX.1-dev single-GPU text-to-image generation). The test launched the sglang generate subprocess at 10:16:16 UTC and then produced zero further output until the GitHub Actions step Run diffusion server tests (1-GPU) hit its 90-minute hard timeout at 11:44:22 UTC. No pixel output, no stage logs, no error trace were emitted.

Regression Status

Flaky test / isolated hang — the same job partition passed in the immediately preceding and following scheduled runs.

Recent history of test_generate_t2i_perf.py (TestFlux_T2V::test_single_gpu) in job multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 3) (branch=main, event=schedule, completed runs only):

Date	Run	Job	Test File Status	Failed Function	Error
Jun 23 13:03	28028218911	partition 3	✅ Passed (165.94s)	—	—
Jun 23 07:20	28009346928	partition 3	❌ Hung/timeout	test_single_gpu	90-min action timeout
Jun 23 00:51	27994433780	partition 3	✅ Passed	—	—
Jun 22 19:10	27977371862	partition 3	✅ Passed	—	—
Jun 22 13:56	27958083310	partition 3	✅ Passed	—	—

First observed failure: 2026-06-23 07:20 UTC (this run). Last known passing: 2026-06-23 00:51 UTC (and again 13:03 UTC). Single failure surrounded by green ⇒ not a persistent regression.

Failure Cluster

"FLUX.1-dev T2I generation hang → 90-min job timeout (no output)"

Facts (verified observations, no interpretation)

• The shard ran 1 item: test_generate_t2i_perf.py::TestFlux_T2V::test_single_gpu (log: "collected 1 item" / "Running 1 items in this shard").
• Last log line before the hang (10:16:16): Running command: sglang generate --prompt A curious raccoon ... --model-path=black-forest-labs/FLUX.1-dev .... Next line (11:44:22): ##[error]The action 'Run diffusion server tests (1-GPU)' has timed out after 90 minutes.
• No intermediate output at all between those two timestamps — not even the Setting distributed timeout to 3600 seconds line that appears within ~17s of the same command in the passing run.
• In the passing sister run 28028218911, the identical command produced Setting distributed timeout, TextEncodingStage finished in 32.49s, DenoisingStage, DecodingStage, PASSED, and 1 passed ... in 165.94s — full run under 3 minutes.
• A 403 gated repo error for ideogram-ai/ideogram-4-fp8 appears during registry config-read at 10:15:50, before test collection. It is unrelated noise: that model is not the test that ran, and the same error is benign in passing runs.
• AITER_COMMIT_OVERRIDE is empty (regular pr-test-amd run, not a scout).

Hypothesised Causes (with confidence, with disconfirming evidence)

• [MEDIUM] Infrastructure/startup hang — the sglang generate subprocess stalled during model load / server init (before any pipeline stage), most likely on the MI325 runner (GPU/RCCL init, HF model fetch, or aiter JIT/MIOpen cache contention).
  • Supporting: zero output for 88 min including the absence of the very first startup log line; the same code+command completes in ~165s 6h later on the next scheduled run; failure is a clean hang, not an exception.
  • Disconfirming: no GPU/RCCL/HF error was logged (a true infra fault often prints something); cannot point to a specific subsystem from the log alone.
  • How to verify: re-run the job; if it passes, confirms transient. For a repeat, attach py-spy dump / faulthandler to the hung sglang generate PID on the runner to capture where it blocks.
• [LOW] Code regression in commit 743ce88bc5 ("Fix flaky optimistic prefill retry test").
  • Supporting: it is the head commit of the failing run.
  • Disconfirming: strong — that commit touches a prefill-retry test, not the diffusion/FLUX path; and the next scheduled run 28028218911 (which includes this commit) passed the same test. A code regression would persist.
  • How to verify: not actionable — already disconfirmed by the subsequent green run.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs repo:sgl-project/sglang is:pr is:open diffusion timeout (17 results, none target a FLUX/diffusion startup hang). No fix is warranted unless the hang recurs.

Suggested Triage Steps

• Re-run the failed job; given green runs on both sides, expect it to pass (confirms transient flake).
• If the hang recurs on partition 3, capture a py-spy dump/faulthandler stack of the hung sglang generate process to localize the stall (GPU/RCCL init vs HF download vs aiter JIT/MIOpen cache).
• Consider adding a per-test subprocess timeout (shorter than the 90-min step budget) to test_generate_t2i_perf.py so a single hung generation fails fast with a stack dump instead of consuming the whole job budget with no diagnostics.

Status (factual, no priority assignment)

• Persistence: Single occurrence on 2026-06-23 07:20 UTC; the 4 preceding and 1 following scheduled runs of this job all passed (FLUX test ~165s).
• Scope: 1 job, 1 test (multimodal-gen-test-1-gpu-amd partition 3 only) in pr-test-amd.yml.
• Blocked work: None ongoing — next scheduled run already green.
• In-flight fix: ❌ none found — no action needed unless the hang recurs.

stage-b-test-large-8-gpu-mi35x-disaggregation-amd (linux-mi35x-gpu-8.fabric) — failed step(s): Run test_disaggregation

Commit Info

• sglang: 0460f277b770f8b5a404da2e50e7a9573ef8db40
• aiter: 7d604afe5fa7efba63c0dce323b95d9daf2db112 (v0.1.14rc1.dev240+g7d604afe5, source: Dockerfile default, no override)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/disaggregation/test_disaggregation_basic.py	TestDisaggregationAccuracy.test_gsm8k	KeyError: 'answer' → Exception: retry() exceed maximum number of retries (downstream of mooncake KVTransferError decode-transfer timeout 300s)	Log
test/registered/amd/disaggregation/test_disaggregation_pp.py	setUpClass (TestDisaggregationDecodePPAccuracy)	RuntimeError: Server process exited with code -9 (prefill server SIGKILL; scheduler raised "memory capacity is unbalanced … GPUs may be occupied by other processes")	Log

test_mori_transfer_engine_e2e.py passed (2/2). test_first_token_finish (the other test in basic) passed.

Failure Summary

The 8-GPU MI35x disaggregation job failed on two of three test files. In test_disaggregation_basic.py, the mooncake prefill→decode KV transfer timed out (KVTransferError … timed out after 300.0s in KVPoll.WaitingForInput) for many requests; gsm8k therefore got empty generations, raising KeyError: 'answer', and the retry wrapper exhausted (basic-test runtime ballooned from ~254s when healthy to 1372s). In test_disaggregation_pp.py, the prefill server (Qwen3-8B, tp2/pp2, mooncake) was killed with exit code -9 during init, with the scheduler reporting GPU memory already occupied (pre_model_load_memory=45.6 GB) before model load.

Regression Status

New regression — both test files regressed together.

Recent history of test_disaggregation_basic.py and test_disaggregation_pp.py in job stage-b-test-large-8-gpu-mi35x-disaggregation-amd (branch main, event schedule, completed runs only):

Date	Run	Job	basic.py	pp.py	mori_e2e.py
06-23 13:03	28028218911	job	❌ Failed	❌ Failed	✅ Passed
06-23 07:20	28009346928	job	❌ Failed	❌ Failed	✅ Passed
06-23 00:51	27994433780	job	❌ Failed	❌ Failed	✅ Passed
06-22 19:10	27977371862	job	❌ Failed	❌ Failed	❌ Failed
06-22 13:56	27958083310	job	✅ Passed	✅ Passed	✅ Passed

First observed failure: 2026-06-22 19:10 UTC. Last known passing: 2026-06-22 13:56 UTC. Failing every completed run since (4 consecutive). Regression window: 4923bb93ae..adc203dfee.

Failure Cluster

"Mooncake PD KV-transfer 300s timeout + decode/prefill server death on MI35x 8-GPU"

Facts (verified observations)

• [FACT] Test Summary 1/3 passed; test_disaggregation_basic.py and test_disaggregation_pp.py both returned exit code 1; test_mori_transfer_engine_e2e.py passed.
• [FACT] basic.py error (verbatim): KVTransferError(bootstrap_room=…): Request … timed out after 300.0s in KVPoll.WaitingForInput repeated for many requests, then test_gsm8k raised KeyError: 'answer' at few_shot_gsm8k.py:123 → retry() exceed maximum number of retries.
• [FACT] basic.py uses --disaggregation-transfer-backend mooncake for both prefill and decode.
• [FACT] pp.py error (verbatim): RuntimeError: Server process exited with code -9; underlying scheduler exception RuntimeError: The memory capacity is unbalanced. Some GPUs may be occupied by other processes. pre_model_load_memory=45.611… at model_runner.py:1280.
• [FACT] The last passing run (82731802599) and the first failing run (82798487481) used the same docker image rocm/sgl-dev:v0.5.13.post1-rocm700-mi35x-20260622. aiter unchanged (Dockerfile default 7d604afe5, no override).
• [FACT] basic.py runtime: 254s in the passing run vs 1372–1374s in failing runs (driven by 300s mooncake timeouts × retries).
• [FACT] 13 sglang commits in the regression window; only b28e9901 (ServerArgs Annotated migration, server_args.py ±4625 lines), b43bd682 (B300 flashinfer, +10), and 04d952ea (AMD deepseek-v4 env cleanup, +12) touch server_args.py. None touch python/sglang/srt/disaggregation/.

Hypothesised Causes (with confidence)

• [LOW] Server-args refactor b28e9901 (#28919) altered some disaggregation/memory default during the mechanical ServerArgs→Annotated migration.
  • Supporting: lands in the regression window; touches server_args.py broadly; both failing tests launch servers with many disaggregation flags, and the pp server dies at init memory accounting.
  • Disconfirming: it is a mechanical refactor with no diff hunk reaching the mooncake transfer path; the basic-test symptom is an RDMA/mooncake transfer timeout (KVPoll.WaitingForInput), not an arg-parsing error; the small server_args commits (b43bd682, 04d952ea) are B300/deepseek-v4 specific and won't affect Llama/Qwen.
  • How to verify: git checkout 4923bb93ae (last green) vs adc203dfee (first red) on the MI35x 8-GPU node and run test_disaggregation_basic.py; if green→red reproduces, bisect the 13-commit window.
• [LOW] pp.py failure is a cascade of the basic.py failure: the timed-out/crashed mooncake servers from basic.py left GPU memory occupied (pre_model_load_memory=45.6 GB), so the pp prefill server hit the memory-balance guard and was SIGKILLed.
  • Supporting: pp.py runs after basic.py in the same job on the same 8 GPUs; the error names "GPUs may be occupied by other processes"; pp.py passed (204s) only in the run where basic.py also passed.
  • Disconfirming: not confirmed that inter-file teardown leaks memory — the runner is expected to kill processes between files; pp uses a different backend invocation, so it could be an independent mooncake/RDMA failure.
  • How to verify: run pp.py in isolation (skip basic.py) on a clean node; if it passes, the cascade is confirmed.
• [LOW] MI35x RDMA fabric / mooncake infrastructure issue on the linux-mi35x-gpu-8.fabric runner.
  • Supporting: failure is RDMA KV-transfer timeout; AMD-specific 8-GPU fabric.
  • Disconfirming: persistent across 4 runs and across two daily images (20260622→20260623), which argues against a one-off transient flake.

No hypothesis above LOW confidence; needs git bisect between 4923bb93ae..adc203dfee on MI35x 8-GPU hardware.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for disaggregation+mooncake; matches (#29028 "Default transfer queue size to 1", #27730 "docker(rocm) bump Mooncake", #28515 D2P KV transfer) are perf/feature work, none targets this 300s-timeout regression. (API: search/issues?q=repo:sgl-project/sglang+is:pr+is:open+disaggregation+mooncake)

Suggested Triage Steps

• Bisect 4923bb93ae..adc203dfee (13 commits) on the MI35x 8-GPU fabric node, running test_disaggregation_basic.py with the mooncake backend — the same docker image (20260622) spans the pass→fail boundary, so the trigger is in this sglang range (or in node/fabric state).
• Prime suspect to A/B first: revert b28e9901 (#28919) on a branch and re-run; if green, diff its server_args.py defaults for any disaggregation/mem_fraction change.
• Run test_disaggregation_pp.py in isolation on a clean node to confirm/deny the cascade hypothesis (basic.py leftover GPU memory → pp.py SIGKILL).
• If bisect finds no implicated commit, treat as MI35x RDMA/mooncake fabric instability and check runner GPU memory/zombie-process cleanup between test files.

Status

• Persistence: Failing for ~24h (since 2026-06-22 19:10 UTC), every completed run since (4 consecutive).
• Scope: 2 test files (test_disaggregation_basic.py, test_disaggregation_pp.py) in one job (stage-b-test-large-8-gpu-mi35x-disaggregation-amd) on the MI35x 8-GPU fabric runner; blocks AMD PD-disaggregation CI signal.
• In-flight fix: ❌ none found in open PRs.

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-06-24 23:07 UTC)

[amd-bot]

pr-test-amd.yml — Per-Job Analysis (continued 2/5)

multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 0) — failed step(s): Run diffusion server tests (2-GPU)

Commit Info

• sglang: 743ce88b
• aiter: 7d604afe5 (v0.1.14rc1.dev240+g7d604afe5, source: dev image preserved — [CI-AITER-CHECK] Dev/patched version detected → skipping rebuild; AITER_COMMIT_OVERRIDE empty)

Failed Tests

Test File	Test Function	Error	Log
test/server/test_server_2_gpu.py	test_diffusion_generation[zimage_image_t2i_2_gpus]	TimeoutError: Server not ready within 1200.0s	Log
test/server/test_server_2_gpu.py	test_diffusion_generation[zimage_image_t2i_2_gpus_non_square]	RuntimeError: Scheduler port 5555 is unavailable and --strict-ports is enabled	Log
test/server/test_server_2_gpu.py	test_diffusion_generation[flux_2_image_t2i_2_gpus]	RuntimeError: Scheduler port 5555 is unavailable and --strict-ports is enabled	Log
test/server/test_server_2_gpu.py	test_diffusion_generation[ltx_2.3_one_stage_ti2v]	RuntimeError: Scheduler port 5555 is unavailable and --strict-ports is enabled	Log

(Full path: python/sglang/multimodal_gen/test/server/test_server_2_gpu.py::TestDiffusionServerTwoGpu)

Failure Summary

This 2-GPU diffusion (multimodal_gen) job ran 12 cases; the first 8 passed. The 9th, zimage_image_t2i_2_gpus (Z-Image-Turbo, --ulysses-degree 2), started its server — the scheduler began listening on port 5555 at 10:24:42 — but the server never became ready and was killed by a TimeoutError: Server not ready within 1200.0s at ~10:43:39. The hung/leaked server process kept holding scheduler port 5555, so the next 3 tests (zimage_image_t2i_2_gpus_non_square, flux_2_image_t2i_2_gpus, ltx_2.3_one_stage_ti2v) all failed at setup with RuntimeError: Scheduler port 5555 is unavailable and --strict-ports is enabled. The retry attempt (2/7) re-ran the same 4 items and all 4 errored on port 5555 again, confirming the leaked process was never reclaimed between attempts.

Regression Status

Flaky / transient (isolated to this run; shard 0 passed in the immediately preceding and following scheduled runs).

Recent history of test_server_2_gpu.py (test_diffusion_generation[zimage_image_t2i_2_gpus]) in job multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 0) (branch=main, event=schedule, completed runs only):

Date	Run	Job (shard 0)	Test File Status	Note
Jun 23 13:03	28028218911	shard 0	✅ Passed	head 0460f277b7 (after)
Jun 23 07:20	28009346928	shard 0	❌ Failed	this run, head 743ce88b
Jun 23 00:51	27994433780	shard 0	✅ Passed	head ba9d5aed98 (before)
Jun 22 19:10	27977371862	shard 0	✅ Passed	—
Jun 22 13:56	27958083310	shard 0	✅ Passed	—

Shard 0 passed in the runs both before (ba9d5aed98) and after (0460f277b7) this one, with nearly identical code, so this is not a code regression. (Note: shard 1 of this job fails persistently across all these runs — a separate, unrelated failure not in scope here.)

Failure Cluster

Diffusion server warmup hang → leaked scheduler port 5555 → cascade --strict-ports setup errors

Facts (verified observations)

• zimage_image_t2i_2_gpus server reached Rank 0 scheduler listening on tcp://*:5555 at 10:24:42 (log line 7699), then produced no further output until ERROR at 10:43:39 (line 7700) — ~19 min silence during server warmup (warmup_mode=server, warmup_steps=1).
• Setup failure for this case: TimeoutError: Server not ready within 1200.0s at test_server_utils.py:492 (lines 8087-8089).
• The 3 subsequent cases fail at test_server_utils.py:472 → RuntimeError: Server exited early (code 1), whose underlying server error is RuntimeError: Scheduler port 5555 is unavailable and --strict-ports is enabled raised in server_args.py:748 _require_port (lines 7738, 7774, 7814, 8713, 8756, 8803).
• Each subsequent server logs kill_process_tree called: ... include_parent=False and exits before binding — i.e. the new process refused to start because 5555 was still held by the leaked zimage server.
• WARNING ... ROCm GPU memory not fully cleared after 60s (used: 28.68 GB) precedes each cascade case (line 8308 etc.), consistent with the leaked zimage server still resident on the GPUs.
• During zimage load, the RunAI distributed streamer raised EOFError: Ran out of input in all_gather_object while loading the customized text_encoder (line 7669), but it fell back to the native loader, which loaded successfully (line 7676); the server continued past component load, so this fallback is not the hang point.
• Both pytest attempts (1/7 initial, 2/7 retry) failed on the same 4 items with the same port-5555 errors (lines 8872-8886).

Hypothesised Causes (with confidence)

• [MEDIUM] The primary failure is a one-off server warmup hang in zimage_image_t2i_2_gpus (Z-Image-Turbo, ulysses-degree=2 sequence parallel). The server hung after the scheduler started, between 10:24:42 and the 1200s deadline, never completing warmup.
  • Supporting: scheduler bound and started listening, then 19 min of total silence with no error until the test harness timeout fired; the case uses 2-GPU sequence/ulysses parallelism where a collective-op stall during the warmup forward pass would present exactly as a silent hang.
  • Disconfirming: shard 0 (which runs this exact case) passed in the runs immediately before and after with essentially identical code, so this is not reproducible — strongly suggests a transient/environmental stall (e.g. RCCL/NCCL collective hang, GPU not fully cleared from a prior case) rather than a deterministic code bug.
  • How to verify: re-run multimodal-gen-test-2-gpu-amd shard 0; if it reproduces, capture py-spy dump / RCCL debug on the hung Z-Image worker during warmup.
• [LOW] Secondary (test-infra) issue: when the server-ready wait times out, the harness does not reliably kill the spawned server, leaking scheduler port 5555 and turning one hang into 4 failures.
  • Supporting: the 3 cascade failures and the retry-attempt failures are all pure consequences of the leaked port, not independent defects; _wait_for_ready raising TimeoutError (test_server_utils.py:492) is not followed by a successful teardown of the prior process.
  • Disconfirming: cannot confirm the teardown code path from logs alone; the leak may instead be a wedged GPU process that resists kill_process_tree.
  • How to verify: read test/server/test_server_utils.py teardown around _wait_for_ready/start; check whether a timeout path force-kills process and waits for port release before the next case.

In-flight Fix Check

• In-flight fix: ❌ none found directly targeting "diffusion server warmup hang / port-5555 leak / strict-ports cascade". Searched open PRs for strict-ports+scheduler+port (1 unrelated hit: #24227), zimage+diffusion, and diffusion+server+port.
• Tangentially related (not a fix for this): #28503 fix(multimodal_gen): disable proxy env in warmup health probe touches diffusion warmup probing — worth a glance but does not address a hung warmup forward pass or port leak.

Suggested Triage Steps

• Re-run job multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 0) on run 28009346928; given shard 0 passed before and after, expect it to recover — confirms flaky classification.
• If zimage_image_t2i_2_gpus warmup hang reproduces, attach py-spy dump to the hung Z-Image worker and enable NCCL_DEBUG=INFO/RCCL tracing to locate the stalled collective during the ulysses-degree=2 warmup forward pass.
• Harden the test harness so a TimeoutError in _wait_for_ready force-kills the spawned server and waits for scheduler port 5555 to be released before the next case — this would convert the single hang into 1 failure instead of 4 and prevent the retry from inheriting the leak (test/server/test_server_utils.py).
• Track shard 1's persistent failure separately — it is unrelated to this zimage hang and fails in every recent scheduled run.

Status

• Persistence: Isolated to run 28009346928 (Jun 23 07:20 UTC). Shard 0 passed in the preceding (00:51) and following (13:03) scheduled runs.
• Scope: 4 test cases in one job/shard (multimodal-gen-test-2-gpu-amd shard 0); 1 root hang + 3 port-leak cascades. AMD MI325 2-GPU only.
• Blocked work: Transient flake on the 2-GPU diffusion CI signal; no evidence of an ongoing block.
• In-flight fix: ❌ none found in open PRs.

multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 1) — failed step(s): Run diffusion server tests (2-GPU)

Commit Info

• sglang: 743ce88b
• aiter: 7d604afe5 (v0.1.14rc1.dev240+g7d604afe5, source: dev image preserved — [CI-AITER-CHECK] Dev/patched version detected → skipping rebuild)

Failed Tests

Test File	Test Function	Error	Log
python/sglang/multimodal_gen/test/server/test_server_2_gpu.py	TestDiffusionServerTwoGpu::test_diffusion_generation[flux2_modelopt_fp8_tp2_t2i]	RuntimeError: CUDA error: HIPBLAS_STATUS_NOT_SUPPORTED from torch._scaled_mm	Log
python/sglang/multimodal_gen/test/server/test_server_2_gpu.py	TestDiffusionServerTwoGpu::test_diffusion_generation[ideogram4_fp8_tp2_t2i] (setup error)	GatedRepoError / ValueError: 403 Client Error downloading ideogram-ai/ideogram-4-fp8	Log

Failure Summary

Shard 1 of multimodal-gen-test-2-gpu-amd ended 1 failed, 9 passed, 1 skipped, 1 error. The flux2 test failed because the server returned HTTP 500 — the FP8 torch._scaled_mm call in apply_fp8_linear raised HIPBLAS_STATUS_NOT_SUPPORTED on the MI325 (gfx942). The ideogram4 test errored at setUpClass/server startup because the gated HuggingFace repo ideogram-ai/ideogram-4-fp8 returned 403 Forbidden, so the server exited early (code 1).

Regression Status

Never-passed (no green run found in queryable window) for both tests on this shard.

Recent history of shard (linux-mi325-2gpu-sglang, 1) — test_server_2_gpu.py, both flux2_modelopt_fp8_tp2_t2i & ideogram4_fp8_tp2_t2i (branch main, event schedule, completed runs only):

Date	Run	Job	flux2 / ideogram4 Status	Error
Jun 23 13:03	28028218911	shard 1	❌ FAILED / ❌ ERROR	same
Jun 23 07:20 (this run)	28009346928	shard 1	❌ FAILED / ❌ ERROR	same
Jun 23 00:51	27994433780	shard 1	❌ FAILED / ❌ ERROR	same
Jun 22 19:10	27977371862	shard 1	❌ FAILED / ❌ ERROR	same
Jun 22 13:56	27958083310	shard 1	❌ FAILED / ❌ ERROR	same
Jun 22 08:09	27938642367	shard 1	❌ FAILED / ❌ ERROR	same
Jun 21 01:00	27889118263	shard 1	❌ FAILED / ❌ ERROR	same

Both tests failed identically in every completed schedule run back to the oldest queryable run (Jun 21 01:00 UTC). Lookback window: Jun 21 01:00 → Jun 23 13:03 UTC (11 runs); no passing run for either test found.

Failure Cluster

• flux2: FP8 _scaled_mm HIPBLAS_STATUS_NOT_SUPPORTED on ROCm/gfx942
• ideogram4: Gated HuggingFace repo 403 during server startup

Facts (verified observations)

• flux2 stack trace (verbatim, log line 5180-5192): flux_2.py:179 _get_qkv_projections → linear.py:427 forward → modelopt_quant.py:425 apply → fp8_utils.py:1695 apply_fp8_linear → output = torch._scaled_mm(...) → RuntimeError: CUDA error: HIPBLAS_STATUS_NOT_SUPPORTED when calling HIPBLAS_STATUS_NOT_SUPPORTED. Server returned 500 Internal Server Error on POST /v1/images/generations; test then saw openai.InternalServerError.
• ideogram4 (log line 5401, 5429): huggingface_hub.errors.GatedRepoError: 403 Client Error and ValueError: Model or revision not found at ideogram-ai/ideogram-4-fp8 ... ensure you have access to the repository while fetching model_index.json / .gitattributes. Server exited early (code 1); pytest reported ERROR at setup of ...[ideogram4_fp8_tp2_t2i].
• Both failures reproduce in 11/11 completed runs in the window; the other two shards (0, 2) of this job pass.
• aiter unchanged across runs (dev image v0.1.14rc1.dev240+g7d604afe5 preserved; no override).

Hypothesised Causes (with confidence)

• [HIGH] flux2: ROCm/hipBLASLt does not support the FP8 e4m3fn torch._scaled_mm path used by apply_fp8_linear (fp8_utils.py:1695) on gfx942.
  • Supporting: error is HIPBLAS_STATUS_NOT_SUPPORTED raised directly from the _scaled_mm call in the stack trace; AMD-only (NVIDIA unaffected); open fix PR #28889 explicitly targets "FP8 e4m3fn GEMM on ROCm" with a dequant→bf16 fallback in apply_fp8_linear.
  • Disconfirming: no green run exists in the window, so this is a standing platform-support gap rather than a code regression — cannot bisect to a single commit.
  • How to verify: apply #28889 on a branch and re-run test_diffusion_generation[flux2_modelopt_fp8_tp2_t2i] on MI325.
• [HIGH] ideogram4: the CI HF token lacks access to the gated repo ideogram-ai/ideogram-4-fp8, and the model path fetches gated metadata at startup.
  • Supporting: GatedRepoError: 403 on model_index.json/.gitattributes; open PR #28225 "Route Ideogram NF4 without gated metadata fetch" addresses exactly this metadata fetch.
  • Disconfirming: a 403 is also consistent with a pure credential/infra problem on the runner (not a code issue) — PR #28225 only helps if it removes the gated fetch entirely.
  • How to verify: confirm the runner's HF_TOKEN has access to ideogram-ai/ideogram-4-fp8; if not, this is infra, not code.

In-flight Fix Check

• flux2: In-flight fix: ✅ #28889 (open since 2026-06-22, not draft) — "[AMD] Fix FP8 e4m3fn GEMM on ROCm: dequant->bf16 fallback in apply_fp8_linear (fixes flux2 modelopt-fp8 HIPBLAS_STATUS_NOT_SUPPORTED)". Chase reviewers; do NOT open a duplicate. (search: q=HIPBLAS_STATUS_NOT_SUPPORTED)
• ideogram4: In-flight fix: ✅ #28225 (open since 2026-06-15, draft) — "Route Ideogram NF4 without gated metadata fetch". Relevant but still draft; also verify CI HF credentials. (search: q=ideogram+gated)

Suggested Triage Steps

• flux2: review/merge #28889, then re-run test_diffusion_generation[flux2_modelopt_fp8_tp2_t2i] on MI325 (gfx942) shard 1 to confirm the dequant→bf16 fallback resolves HIPBLAS_STATUS_NOT_SUPPORTED.
• ideogram4: first confirm whether the runner's HF credentials can access gated ideogram-ai/ideogram-4-fp8; if access is not expected in CI, finish #28225 (move out of draft) or deselect the ideogram4_fp8_tp2_t2i case on AMD until access/routing is fixed.

Status

• Persistence: both tests failing in every completed schedule run in the queryable window (since at least Jun 21 01:00 UTC, 11/11 runs); no green run found — likely standing failures, not a fresh regression.
• Scope: shard 1 of multimodal-gen-test-2-gpu-amd only; shards 0 and 2 pass. AMD/ROCm-specific (flux2 is a ROCm FP8 support gap).
• Blocked work: flux2 modelopt-FP8 and ideogram4-FP8 diffusion e2e signal on AMD 2-GPU.
• In-flight fix: flux2 ✅ #28889 (open, awaiting merge); ideogram4 ✅ #28225 (draft) + verify CI HF access.

stage-c-test-large-8-gpu-amd (linux-mi325-8gpu-sglang, 0) — failed step(s): Test RCCL multi-GPU communication

Commit Info

• sglang: 743ce88b
• aiter: 7d604afe5 (v0.1.14rc1.dev240+g7d604afe5, source: dev image preserved — Dev/patched version detected → skipping rebuild; AITER_COMMIT_OVERRIDE empty, so this is a regular pr-test-amd run, not a scout)

Failed Tests

Test File	Test Function	Error	Log
scripts/ci/amd/test_rccl_multi_gpu.py	test_rccl_allreduce (N/A — standalone torchrun script, not pytest)	ChildFailedError: Signal 11 (SIGSEGV), exitcode -11 on ranks 0-5	Log

Failure Summary

The job's pre-test "Test RCCL multi-GPU communication" step (step 6) crashed. All 8 ranks reported Initialized successfully and the device/memory probes succeeded, but during/after the dist.all_reduce call ranks 0-5 died with SIGSEGV (exitcode -11). The job never reached the actual run_suite.py test suite. This is an 8-GPU RCCL/NCCL smoke test, not an sglang unit test.

Regression Status

Flaky / infrastructure issue (isolated one-off).

Recent history of the RCCL multi-GPU communication step in job stage-c-test-large-8-gpu-amd (linux-mi325-8gpu-sglang, 0) (branch=main, event=schedule, completed runs only):

Date	Run	RCCL Step Status	Job Concl	Error
Jun 23 13:03	28028218911	✅ Passed	success	—
Jun 23 07:20	28009346928	❌ SIGSEGV	failure	Signal 11 on ranks 0-5
Jun 23 00:51	27994433780	✅ Passed	success	—
Jun 22 19:10	27977371862	✅ Passed	success	—
Jun 22 13:56	27958083310	✅ Passed	success	—

The RCCL step passed in both the immediately preceding (00:51) and the immediately following (13:03) scheduled runs — 1/5 recent runs failed. First (and only) observed RCCL-step failure: 2026-06-23 07:20 UTC.

Failure Cluster

RCCL/NCCL allreduce SIGSEGV during 8-GPU smoke test

Facts (verified observations)

• Step 6 command: docker exec ci_sglang bash -c "cd /sglang-checkout && NCCL_DEBUG=INFO RCCL_DEBUG=INFO torchrun --nproc_per_node=8 scripts/ci/amd/test_rccl_multi_gpu.py" (log line 4116).
• All 8 ranks printed Initialized successfully; ranks 0-6 printed device name AMD Instinct MI300X / 206.14 GB (lines 4129-4149). No rank printed the Before allreduce / After allreduce lines that the script emits — the crash occurred at or just before dist.all_reduce.
• failed (exitcode: -11) local_rank: 0 (pid: 2993) … ranks 1-5 also exitcode -11 … Signal 11 (SIGSEGV) (lines 4152, 4171-4214). Ranks 6/7 received SIGTERM (clean teardown, lines 4150-4151).
• Process completed with exit code 1 (line 4216).
• scripts/ci/amd/test_rccl_multi_gpu.py is unchanged in this window (last modified at 8e890391, well before this run). It imports only torch / torch.distributed — no sglang code is exercised.
• The same step passed on the runs immediately before and after with no relevant change.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [MEDIUM] Transient RCCL/runner/GPU-driver fault on the 8-GPU MI325 runner (infrastructure flake).
  • Supporting: SIGSEGV inside the RCCL collective with all ranks healthy through init; the test script and its (torch/RCCL-only) code path are unchanged; the step passed in the adjacent runs at 00:51 and 13:03 the same day; failure is non-deterministic (1/5 runs).
  • Disconfirming: only a single occurrence captured here, so a low-rate deterministic regression cannot be fully excluded from this evidence alone.
  • How to verify: re-run the job on the same runner; if green, confirms flake. If it recurs, capture an rocgdb/core dump of rank 0 and check RCCL/ROCm versions on that specific runner host (4e2854a8fd8c).
• [LOW] sglang code regression at 743ce88b.
  • Supporting: temporal — this is the SHA that failed.
  • Disconfirming: the failing step runs a standalone script that imports zero sglang modules, and a later run on a newer SHA (0460f277) passed the same step. No code-level path connects the diff to RCCL allreduce.
  • How to verify: not actionable — the step does not import sglang.

In-flight Fix Check

• In-flight fix: ❌ none found — searched repo:sgl-project/sglang is:pr is:open RCCL multi-gpu (2 hits: #28605 shadow-runner test, #17628 torch-compile enablement — neither addresses an RCCL SIGSEGV). No dedicated fix needed if this stays a one-off flake.

Suggested Triage Steps

• Re-run job stage-c-test-large-8-gpu-amd (linux-mi325-8gpu-sglang, 0) for run 28009346928; a green re-run confirms a transient infra flake and the failure can be dismissed.
• If it recurs, inspect runner host 4e2854a8fd8c for RCCL/ROCm version drift or a sick GPU, and capture a core dump (rocgdb) of rank 0 at the dist.all_reduce call.
• Consider quarantining/retrying the RCCL smoke step (e.g. one automatic retry) since a flake here blocks the entire 8-GPU large-test job before any real test runs.

Status

• Persistence: Single isolated failure at 2026-06-23 07:20 UTC; RCCL step passed in the runs immediately before (00:51) and after (13:03) the same day (1/5 recent scheduled runs failed).
• Scope: This one job/step (8-GPU large test, partition 0). Failure occurs in the pre-suite RCCL smoke test, so it blocks the entire run_suite.py stage-c run for this partition.
• Blocked work: Gates the 8-GPU large test suite on AMD MI325 for this run only.
• In-flight fix: ❌ none found — likely a flake; no PR warranted unless it recurs.

stage-b-test-large-8-gpu-mi35x-disaggregation-amd (linux-mi35x-gpu-8.fabric) — failed step(s): Run test_disaggregation

Commit Info

• sglang: 743ce88bc5
• aiter: 7d604afe5 (source: dev image — [CI-AITER-CHECK] reports v0.1.14rc1.dev240+g7d604afe5, "Dev/patched version detected → skipping rebuild")

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/disaggregation/test_disaggregation_basic.py	TestDisaggregationAccuracy.test_gsm8k	KeyError: 'answer' → retry() exceed maximum number of retries (underlying: KVTransferError ... timed out after 300.0s in KVPoll.WaitingForInput)	Log
test/registered/amd/disaggregation/test_disaggregation_pp.py	TestDisaggregationDecodePPAccuracy.setUpClass	RuntimeError: Server process exited with code -9 (underlying: The memory capacity is unbalanced ... pre_model_load_memory=45.59 GB)	Log

(test_mori_transfer_engine_e2e.py passed in this run.)

Failure Summary

The 8-GPU MI35x PD-disaggregation job failed on two test files. In test_disaggregation_basic.py::test_gsm8k, every decode request timed out with KVTransferError ... timed out after 300.0s in KVPoll.WaitingForInput (mooncake/RDMA transfer never completed), so no answer variable was produced → KeyError: 'answer'; the test ran 1373s vs ~254s when passing. Then test_disaggregation_pp.py::setUpClass failed fast (29s) when the prefill server was SIGKILLed (-9) after the memory-balance check found pre_model_load_memory=45.59 GB already resident on the GPUs (unbalanced), which is consistent with leftover/leaked server processes from the preceding hung basic test.

Regression Status

New regression (started 2026-06-22 19:10 UTC).

Recent history of test_disaggregation_basic.py (test_gsm8k) and test_disaggregation_pp.py (setUpClass) in job stage-b-test-large-8-gpu-mi35x-disaggregation-amd (branch main, event schedule, completed runs only):

Date (UTC)	Run	Job	basic.py	pp.py
Jun 23 13:03	28028218911	job	❌ Failed (1388s)	❌ Failed
Jun 23 07:20	28009346928	job	❌ Failed (1373s)	❌ Failed
Jun 23 00:51	27994433780	job	❌ Failed	❌ Failed
Jun 22 19:10	27977371862	job	❌ Failed (1054s)	❌ Failed
Jun 22 13:56	27958083310	job	✅ Passed (254s)	✅ Passed (204s)
Jun 22 08:09	27938642367	job	✅ Passed	✅ Passed

First observed failure: 2026-06-22 19:10 UTC. Last known passing: 2026-06-22 13:56 UTC. Failing every completed run since (4 consecutive) — persistent, not a one-off flake.

Failure Cluster

PD-disaggregation mooncake KV transfer timeout on AMD MI35x (with downstream GPU-memory-imbalance SIGKILL on the PP server).

Facts (verified observations)

• Basic test: repeated RuntimeError: ... Decode transfer failed ... KVTransferError(bootstrap_room=...): Request ... timed out after 300.0s in KVPoll.WaitingForInput for many requests; final KeyError: 'answer' at few_shot_gsm8k.py:123 → retry() exceed maximum number of retries (verbatim, log lines ~21009–21317).
• PP test: prefill launched with --tp-size 2 --pp-size 2 --disaggregation-transfer-backend mooncake --attention-backend aiter; schedulers raised The memory capacity is unbalanced ... pre_model_load_memory=45.59375, local_gpu_memory=279.64.../612.58... at model_runner.py:1280, then Received sigquit from a child process and Server process exited with code -9 (log lines ~21651–21740).
• Timing: basic-test wall time jumped from ~254s (passing) to 1054–1388s (failing) — consistent with 300s KV-transfer timeouts.
• Scope: AMD MI35x 8-GPU .fabric runner; RDMA/mooncake transfer path (rdma0..rdma7).
• Reproducibility: identical pattern in 4 consecutive completed scheduled runs; in the first failing run (Jun 22 19:10) test_mori_transfer_engine_e2e.py also failed, but it passed in this run.
• Regression window 4923bb93ae..adc203dfee contains 13 commits (compare).

Hypothesised Causes (with confidence)

• [LOW] A commit in the window changed behaviour of the PD/mooncake transfer or memory accounting path.
  • Supporting: failure begins exactly between 4923bb93ae (pass) and adc203dfee (first fail); persistent thereafter.
  • Disconfirming: NONE of the 13 window commits touch the mooncake transfer engine, the KVPoll/bootstrap path, or the memory-balance check at model_runner.py:1280. The router PD commit adc203dfee only edits the Rust gateway (sgl-model-gateway/src/routers/http/pd_router.rs), which these fixtures (python disaggregation_fixture.py mini-LB) do not use. The ServerArgs Annotated refactor b28e9901 preserves disaggregation defaults (mooncake, bootstrap port 8998) — verified in the diff.
  • How to verify: bisect 4923bb93ae..adc203dfee running test_disaggregation_basic.py on an MI35x 8-GPU .fabric runner.
• [LOW] pp.py::setUpClass -9 is a cascade from the hung basic test leaking prefill/decode server processes (leftover GPU memory), not an independent failure.
  • Supporting: pre_model_load_memory=45.59 GB already resident before model load; pp fails in <35s only after basic has hung; both pass together when basic passes.
  • Disconfirming: pre_model_load_memory=45.59 is suspiciously constant across runs — could instead be a genuine per-run runner contention issue rather than a leak.
  • How to verify: check whether stale sglang.launch_server processes survive after the basic test on the runner; run pp test in isolation on a clean runner.
• [LOW] Infrastructure: RDMA/runner GPU contention on the MI35x .fabric host (not a code regression).
  • Supporting: AMD/RDMA-specific; mooncake transfer timeouts and "GPUs may be occupied by other processes" message.
  • Disconfirming: 4 consecutive runs fail identically starting at a sharp commit boundary, which is more consistent with a code change than random contention.

No hypothesis above LOW confidence; needs git bisect between 4923bb93ae..adc203dfee on MI35x hardware.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for disaggregation memory; nearest is #28652 ("disaggregation: Fix FakeKVSender queue accumulation"), which targets a different KV-sender mechanism and is not an obvious match for this mooncake WaitingForInput timeout. No open PR addresses this specific MI35x PD-disaggregation regression.

Suggested Triage Steps

• Bisect the 13 commits in 4923bb93ae..adc203dfee, running test_disaggregation_basic.py on an MI35x 8-GPU .fabric runner — focus first on commits that could alter PD transfer/arg defaults (b28e9901, 99c18cce, 04d952ea).
• On a failing run, capture whether sglang.launch_server processes survive after the basic test (to confirm/deny the leftover-memory cascade hypothesis for the pp -9).
• Confirm whether the mooncake KVPoll.WaitingForInput timeout is reproducible against the prior aiter dev image to rule out the dev-build aiter (7d604afe5) as a confounder, since this job ran a non-Dockerfile dev aiter.
• If bisect finds no code culprit, treat as runner/RDMA infra and check MI35x .fabric host health and GPU process cleanup between tests.

Status

• Persistence: Failing every completed scheduled run since 2026-06-22 19:10 UTC (4 consecutive runs through Jun 23 13:03 UTC).
• Scope: 1 job (stage-b-test-large-8-gpu-mi35x-disaggregation-amd, linux-mi35x-gpu-8.fabric) in pr-test-amd.yml; 2 test files (test_disaggregation_basic.py, test_disaggregation_pp.py).
• Blocked work: AMD MI35x PD-disaggregation CI signal (gsm8k accuracy + PP decode).
• In-flight fix: ❌ none found in open PRs.

multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 1) — failed step(s): Run diffusion server tests (2-GPU)

Commit Info

• sglang: ba9d5aed98ef304338da1f7c48bd71456095b728
• aiter: 7d604afe5fa7efba63c0dce323b95d9daf2db112 — v0.1.14rc1.dev240+g7d604afe5 (source: dev/patched image preserved; matches Dockerfile default AITER_COMMIT, rebuild skipped)

Failed Tests

Test File	Test Function	Error	Log
python/sglang/multimodal_gen/test/server/test_server_2_gpu.py	TestDiffusionServerTwoGpu::test_diffusion_generation[flux2_modelopt_fp8_tp2_t2i]	openai.InternalServerError: Internal Server Error (server-side RuntimeError: CUDA error: HIPBLAS_STATUS_NOT_SUPPORTED in torch._scaled_mm)	Log
python/sglang/multimodal_gen/test/server/test_server_2_gpu.py	TestDiffusionServerTwoGpu::test_diffusion_generation[ideogram4_fp8_tp2_t2i]	ERROR at setup → RuntimeError: Server exited early (code 1) ← huggingface_hub.errors.GatedRepoError: 403 on ideogram-ai/ideogram-4-fp8	Log

Failure Summary

Two of 12 scenarios in the 2-GPU diffusion server shard failed; the other 9 passed and 1 (qwen_image_t2i_2_gpus) was skipped. flux2_modelopt_fp8_tp2_t2i got a 500 from the server because the FLUX.2 modelopt-FP8 forward pass crashes during the denoising step. ideogram4_fp8_tp2_t2i never got past server setup because the model repo is gated on Hugging Face (403).

Regression Status

Known recurring failure (both tests). Not new in this run.

Recent history of test_server_2_gpu.py (test_diffusion_generation[flux2_modelopt_fp8_tp2_t2i] / [ideogram4_fp8_tp2_t2i]) in job multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 1) (branch=main, event=schedule, completed runs only):

Date	Run	Job	flux2 Status	ideogram4 Status
Jun 23 13:03	28028218911	job	❌ FAILED (HIPBLAS)	❌ ERROR (403)
Jun 23 07:20	28009346928	job	❌ FAILED (HIPBLAS)	❌ ERROR (403)
Jun 23 00:51	27994433780	job	❌ FAILED (HIPBLAS)	❌ ERROR (403)
Jun 22 19:10	27977371862	job	❌ FAILED (HIPBLAS)	❌ ERROR (403)
Jun 22 13:56	27958083310	job	❌ FAILED (HIPBLAS)	❌ ERROR (403)
Jun 21 01:00	27889118263	job	❌ FAILED (HIPBLAS)	❌ ERROR (403)

Both tests fail in every completed schedule run checked. No passing run found in the queryable window (lookback Jun 21 01:00 → Jun 23 13:03 UTC) → effectively Never-passed within this window for both scenarios.

Failure Cluster

• flux2: HIPBLAS_STATUS_NOT_SUPPORTED in FP8 torch._scaled_mm (modelopt-fp8 GEMM on ROCm)
• ideogram4: Gated HF repo 403 during server startup (model access)

Facts (verified observations)

• Server-side crash for flux2 (verbatim): RuntimeError: CUDA error: HIPBLAS_STATUS_NOT_SUPPORTED when calling \HIPBLAS_STATUS_NOT_SUPPORTED`, raised at python/sglang/srt/layers/quantization/fp8_utils.py:1695inapply_fp8_linear→output = torch._scaled_mm(...)(verified: line 1695 isoutput = torch._scaled_mm(`).
• Call chain (from log stack trace): flux_2.py:179 _get_qkv_projections → attn.to_q → linear.py:427 → modelopt_quant.py:425 apply → apply_fp8_linear → fp8_utils.py:1695 torch._scaled_mm. Verified modelopt_quant.py:425 is the apply_fp8_linear( call.
• The crash fires during synthetic server warmup AND the first real generation request (POST /v1/images/generations → 500 Internal Server Error), which surfaces to the test as openai.InternalServerError.
• ideogram4 setup error (verbatim): huggingface_hub.errors.GatedRepoError: 403 Client Error and httpx.HTTPStatusError: Client error '403 Forbidden' for url 'https://huggingface.co/ideogram-ai/ideogram-4-fp8/...' → RuntimeError: Server exited early (code 1) at test_server_utils.py:472 _wait_for_ready.
• 28 occurrences of HIPBLAS_STATUS_NOT_SUPPORTED per run, consistently across all 6 inspected runs.
• AITER version identical (dev/patched v0.1.14rc1.dev240+g7d604afe5) and AITER_COMMIT_OVERRIDE empty — this is a regular pr-test-amd.yml schedule run, not a scout run.

Hypothesised Causes (with confidence)

• [HIGH] flux2: the modelopt-FP8 (e4m3fn) GEMM path via torch._scaled_mm is unsupported by hipBLAS for this weight/activation/scale layout on the MI325 (gfx942) runner.
  • Supporting: stack trace ends exactly at torch._scaled_mm with a hipBLAS "not supported" status; the matching in-flight PR (#28889) is titled "Fix FP8 e4m3fn GEMM on ROCm: dequant->bf16 fallback in apply_fp8_linear (fixes flux2 modelopt-fp8 HIPBLAS_STATUS_NOT_SUPPORTED)", independently confirming the diagnosis.
  • Disconfirming: (none found) — other FP8 scenarios in the same shard pass, but those do not necessarily exercise the same e4m3fn _scaled_mm layout.
  • How to verify: apply #28889 on a branch and re-run test_server_2_gpu.py::...[flux2_modelopt_fp8_tp2_t2i] on MI325.
• [FACT] ideogram4: ideogram-ai/ideogram-4-fp8 is a gated/inaccessible HF repo for the CI token (403). This is an environment/access issue, not an sglang code defect.
  • Disconfirming: (none) — the 403 is returned by Hugging Face directly.
  • How to verify: confirm CI HF token has access (or that the repo is public); if intentionally gated, the test should be skipped/guarded on AMD CI.

In-flight Fix Check

• flux2: In-flight fix: ✅ [#28889](https://github.com/sgl-project/sglang/pull/28889) "[AMD] Fix FP8 e4m3fn GEMM on ROCm: dequant->bf16 fallback in apply_fp8_linear" (open since 2026-06-22, not draft) — directly targets this HIPBLAS failure; chase reviewers, do NOT open a duplicate. (search: is:pr is:open HIPBLAS_STATUS_NOT_SUPPORTED)
• ideogram4: In-flight fix: ❌ none found (search: is:pr is:open flux2 scaled_mm returned 0; no open PR addressing the ideogram gated-repo access).

Suggested Triage Steps

• Verify #28889 applies cleanly, then re-run test_server_2_gpu.py::TestDiffusionServerTwoGpu::test_diffusion_generation[flux2_modelopt_fp8_tp2_t2i] on an MI325 2-GPU runner to confirm it clears the HIPBLAS failure.
• For ideogram4: confirm whether the CI Hugging Face token is expected to have access to ideogram-ai/ideogram-4-fp8. If the model is intentionally gated, guard/skip the scenario on AMD CI; if not, fix token/repo access.
• Until both land, this shard will remain red; consider tracking flux2 and ideogram4 as two separate known failures (different root causes).

Status

• Persistence: both tests failing in every completed schedule run inspected, Jun 21 01:00 → Jun 23 13:03 UTC (no green run in window).
• Scope: multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 1) shard of pr-test-amd.yml; 2 of 12 scenarios (flux2 FAILED, ideogram4 ERROR). Other 9 scenarios pass, 1 skipped.
• Blocked work: FLUX.2 modelopt-FP8 and Ideogram-4-FP8 2-GPU diffusion server coverage on AMD.
• In-flight fix: flux2 → ✅ #28889 (open since 2026-06-22); ideogram4 → ❌ none found.

stage-a-test-1-gpu-small-amd (linux-mi325-1gpu-sglang) — failed step(s): Install dependencies

Commit Info

• sglang: e0dc8b71
• aiter: N/A — failure occurred in Install dependencies (step 5) during the mori source build, before any [CI-AITER-CHECK] marker was emitted; no aiter version info in the log.

Failed Tests

Test File	Test Function	Error	Log
N/A (build failure)	N/A	mori build aborted: cmake --build . -j 256 returned exit status 1	Log

No sglang test ran. Step 5 Install dependencies failed and step 6 Run test was skipped.

Failure Summary

The Install dependencies step (scripts/ci/amd/amd_ci_install_dependency.sh) builds the AMD mori library from source. During that build, two gtest discovery post-build steps failed because the freshly-linked executables could not be launched: test_ssd_copy_pipeline and test_router_dedup both aborted with error while loading shared libraries: /lib/x86_64-linux-gnu/libabsl_base.so.20210324: invalid ELF header (exit 127). This stopped ninja, failed cmake --build, and aborted mori/setup.py, which exited the job with code 1.

Regression Status

Infrastructure issue (not a code regression).

Recent history of stage-a-test-1-gpu-small-amd (linux-mi325-1gpu-sglang) (Install dependencies / mori build) on main, event=schedule, completed runs only:

Date	Run	Job	Step Status	Detail
Jun 23 18:46	28048998470	stage-a-test-1-gpu-small-amd	❌ Failed	libabsl_base.so invalid ELF header during mori build
Jun 23 13:03	28028218911	stage-a-test-1-gpu-small-amd	✅ Passed	—
Jun 23 07:20	28009346928	stage-a-test-1-gpu-small-amd	✅ Passed	—
Jun 23 00:51	27994433780	stage-a-test-1-gpu-small-amd	✅ Passed	—

This job passed on the 3 immediately preceding scheduled runs with the same install script — first observed failure is this run (Jun 23 18:46 UTC). The corruption is environment-local, not introduced by a commit.

Failure Cluster

Corrupted system library (libabsl_base.so invalid ELF header) breaking mori dependency build

Facts (verified observations, no interpretation)

• Failing step: step 5 Install dependencies = failure; step 6 Run test = skipped (verified via job steps API).
• Verbatim error (log lines 4018, 4036): /sgl-workspace/mori/build/src/umbp/tests/test_ssd_copy_pipeline: error while loading shared libraries: /lib/x86_64-linux-gnu/libabsl_base.so.20210324: invalid ELF header and same for test_router_dedup. Both gtest discovery runs reported Result: 127.
• Build terminator (log line 4100): ninja: build stopped: subcommand failed. → subprocess.CalledProcessError: Command '['cmake', '--build', '.', '-j', '256']' returned non-zero exit status 1. (log line 4133).
• The shared library exists at the expected path but its header is invalid → the file is truncated/corrupt on the runner, not missing.
• Same job + same mori build passed in the previous 3 scheduled runs (verified: jobs 82961543205, 82898649729, 82853563929 all success).
• An unrelated transient occurred earlier (log line 1270-1271): local docker registry pull timed out (10.245.143.50:5000 context deadline exceeded), but it fell back to the public registry and succeeded — not the cause of this failure.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [HIGH] Corrupted/truncated libabsl_base.so.20210324 on the runner's container filesystem — the abseil shared lib has an invalid ELF header, so any executable linking it fails to load (exit 127).
  • Supporting: explicit loader error invalid ELF header on a specific system .so; two independent executables fail identically on the same library; the library is present (path resolved) but unreadable as ELF.
  • Disconfirming: none found — the error is a FACT in the log; only the underlying why the file is corrupt (bad docker layer, partial mount, disk issue) is unverified.
  • How to verify: on the runner/container, run ls -l /lib/x86_64-linux-gnu/libabsl_base.so.20210324 and readelf -h / file on it; compare size against a healthy node; re-pull the image to rule out a corrupt cached docker layer.
• [LOW] Bad/partial docker image layer for rocm/sgl-dev:v0.5.13.post1-rocm700-mi30x-20260623 on this runner producing the corrupt file.
  • Supporting: a local-registry pull timed out and fell back to public pull on this run (log line 1270), hinting at image-distribution flakiness on this node.
  • Disconfirming: the fallback pull reported success and the build progressed to 124/141 targets, so the image was mostly intact.
  • How to verify: docker rmi the image on this runner and re-pull from the public registry, then rerun the job.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs repo:sgl-project/sglang is:pr is:open mori install (5 results, none address abseil/libabsl/ELF corruption). This is a runner-environment problem, not a code defect; a PR is unlikely to be the right remedy.

Suggested Triage Steps

• Re-run the job (stage-a-test-1-gpu-small-amd) — if it passes, confirms a transient corrupt-file/infra flake on this runner.
• Inspect the runner: file/readelf -h /lib/x86_64-linux-gnu/libabsl_base.so.20210324; if header is invalid, force a fresh image pull (docker rmi + re-pull) or drain/recycle the node.
• If the corruption recurs across runners, check the rocm/sgl-dev:...-20260623 image build for a bad abseil layer rather than treating it as a per-node flake.

Status (factual, no priority assignment)

• Persistence: First and only failure this run (Jun 23 18:46 UTC); prior 3 scheduled runs of this job passed.
• Scope: 1 job, 1 workflow (pr-test-amd.yml, stage-a-test-1-gpu-small-amd); failure in dependency install, so the Run test step was skipped — no sglang test coverage produced by this job in this run.
• Blocked work: this run's stage-a 1-GPU-small AMD test signal is missing (tests skipped, not failed).
• In-flight fix: ❌ none found — infrastructure issue; recycle runner / re-pull image rather than open a PR.

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-06-24 23:07 UTC)

[amd-bot]

pr-test-amd.yml — Per-Job Analysis (continued 3/5)

stage-b-test-large-8-gpu-mi35x-disaggregation-amd (linux-mi35x-gpu-8.fabric) — failed step(s): Run test_disaggregation

Commit Info

• sglang: ba9d5aed
• aiter: 7d604afe (v0.1.14rc1.dev240+g7d604afe5, source: Dockerfile default — AITER_COMMIT_OVERRIDE empty)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/disaggregation/test_disaggregation_basic.py	TestDisaggregationAccuracy.test_gsm8k	Exception: retry() exceed maximum number of retries (underlying KeyError: 'answer' because every PD generation returned HTTP 500: KVTransferError ... timed out after 300.0s in KVPoll.WaitingForInput)	Log
test/registered/amd/disaggregation/test_disaggregation_pp.py	TestDisaggregationDecodePPAccuracy.setUpClass	RuntimeError: Server process exited with code -9 (underlying: RDMA Failed to register memory: Cannot allocate memory [12] + The memory capacity is unbalanced. Some GPUs may be occupied by other processes. pre_model_load_memory=45.5GB)	Log

test_mori_transfer_engine_e2e.py passed (1/3 overall: 1 passed, 2 failed). Job exited 255.

Failure Summary

Two of three PD-disaggregation test files in this MI35x 8-GPU job failed. In test_disaggregation_basic.py::test_gsm8k, every prefill→decode KV transfer over the MoRI/RDMA backend timed out (KVPoll.WaitingForInput, 300 s) so the eval never produced an answer field; the test ran 1358 s (vs 254 s when healthy) before exhausting retries. The subsequent test_disaggregation_pp.py then crashed in setUpClass because its decode/prefill servers could not register RDMA memory (ENOMEM) and reported the GPUs already holding ~45.5 GB from prior processes, killing the server with SIGKILL (-9).

Regression Status

New regression — both test files began failing in run 27977371862 (2026-06-22 19:10 UTC) and have failed in every completed run since.

Recent history of test_disaggregation_basic.py (test_gsm8k) and test_disaggregation_pp.py (setUpClass) in job stage-b-test-large-8-gpu-mi35x-disaggregation-amd (branch=main, event=schedule, completed runs only):

Date	Run	basic.py	pp.py	Failed Function
Jun 23 13:03	28028218911	❌ Failed	❌ Failed	test_gsm8k / setUpClass
Jun 23 07:20	28009346928	❌ Failed	❌ Failed	test_gsm8k / setUpClass
Jun 23 00:51 (this run)	27994433780	❌ Failed	❌ Failed	test_gsm8k / setUpClass
Jun 22 19:10	27977371862	❌ Failed	❌ Failed	first observed failure (also test_mori_e2e failed → 0/3)
Jun 22 13:56	27958083310	✅ Passed	✅ Passed	— (3/3 passed)

First observed failure: 2026-06-22 19:10 UTC. Last known pass: 2026-06-22 13:56 UTC (sha 4923bb93).

Failure Cluster

PD-disaggregation MoRI/RDMA KV-transfer timeout on MI35x (prefill→decode transfer stuck in KVPoll.WaitingForInput; PP server then SIGKILLed during RDMA memory registration).

Facts (verified observations)

• test_gsm8k every request returns HTTP 500 with "Decode transfer failed ... KVTransferError(bootstrap_room=...): Request ... timed out after 300.0s in KVPoll.WaitingForInput"; prefill side logs the mirror "Prefill transfer failed ... Failed due to an unknown reason from another rank" (job log lines ~9329–9650).
• Healthy run 27958083310: test_disaggregation_basic.py elapsed 254 s, passed; failing runs elapsed 1358–1388 s. (verified from TIMINGS JSON in each log)
• test_disaggregation_pp.py decode server: E rdma_context.cpp:243 Failed to register memory ...: Cannot allocate memory [12] followed by RuntimeError: The memory capacity is unbalanced ... pre_model_load_memory=45.509GB on all PP/TP ranks, then Server process exited with code -9 (log lines ~21540–21660).
• Regression window 4923bb93ae..adc203dfee = 13 commits (verified via git log). Both backends use --disaggregation-transfer-backend mori, --attention-backend aiter.
• aiter is the Dockerfile default (no override); CI image rocm/sgl-dev:v0.5.13.post1-rocm700-mi35x-20260622.

Hypothesised Causes (with confidence)

• [LOW] Commit b28e9901 "Migrate all ServerArgs fields to Annotated style (#28919)" — largest change in the window (server_args.py, 4625 lines), touches the disaggregation_* field block.
  • Supporting: lands in the regression window; refactors the exact PD-disaggregation arg definitions consumed by these tests.
  • Disconfirming: the disaggregation defaults are byte-for-byte preserved (disaggregation_transfer_backend="mooncake", bootstrap_port=8998, ib_device=None, etc., verified by reading b28e9901:server_args.py:2213-2260), and the tests pass all values explicitly on the CLI — so a default change cannot explain it.
  • How to verify: git revert b28e9901 on a branch, rerun test_disaggregation_basic.py on MI35x 8-GPU.
• [LOW] CI docker image / bundled MoRI+RDMA libraries changed (image …-20260622).
  • Supporting: symptom is purely at the MoRI/RDMA transport layer (KV transfer timeout + RDMA ENOMEM on memory registration), not in Python eval logic; the image is rebuilt daily and is outside the git commit window.
  • Disconfirming: the last-passing run (Jun 22 14:08) and first-failing run (Jun 22 19:10) are close in time; could not confirm a different image tag between them.
  • How to verify: compare the image digest/aiter+mori versions used by run 27958083310 vs 27977371862; re-run last-passing sglang sha 4923bb93ae on today's image.
• [LOW] The pp.py failure is a cascade from the hung basic.py, not independent: the "GPUs occupied by other processes (45.5 GB)" + RDMA Cannot allocate memory indicate residual GPU/pinned memory left by the preceding 1358 s-hung basic test rather than a fresh PP-specific defect.
  • Supporting: pp fails in setUpClass after only 30 s with leftover-memory errors; in the healthy run pp passed in 204 s.
  • Disconfirming: in first-fail run 27977371862 the mori-e2e test also failed (0/3), suggesting node-state contamination that does not perfectly track basic.py alone.
  • How to verify: run test_disaggregation_pp.py in isolation (no preceding basic.py) on a clean MI35x node.

No hypothesis above LOW confidence; needs git bisect across 4923bb93ae..adc203dfee plus an image-version A/B to separate sglang-code vs image causes.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for disaggregation+mori, KVPoll+WaitingForInput, disaggregation+timeout, mori+memory; matches (e.g. #28652, #27093) address unrelated FakeKVSender/DSV4 paths, not this MI35x MoRI KV-transfer regression.

Suggested Triage Steps

• Bisect the 13-commit window 4923bb93ae..adc203dfee, rerunning test_disaggregation_basic.py::test_gsm8k on MI35x 8-GPU each step.
• Independently confirm whether the CI image changed: compare aiter/mori versions and image digest between last-pass run 27958083310 and first-fail run 27977371862; if the image differs, re-run sglang sha 4923bb93ae on today's image to isolate image-vs-code.
• Run test_disaggregation_pp.py standalone on a freshly cleaned node to confirm whether its -9/RDMA-ENOMEM failure is a cascade from the hung basic test rather than an independent defect.
• If bisect implicates b28e9901, revert on a branch and re-run the disaggregation suite to confirm.

Status

• Persistence: Failing for ~4 completed scheduled runs since 2026-06-22 19:10 UTC (~18 h); every completed run since the last green at 2026-06-22 13:56 UTC.
• Scope: 2 test files (test_disaggregation_basic.py, test_disaggregation_pp.py) in job stage-b-test-large-8-gpu-mi35x-disaggregation-amd of pr-test-amd.yml. Blocks PD-disaggregation (MoRI/RDMA) CI signal on AMD MI35x.
• In-flight fix: ❌ none found in open PRs — needs new investigation/PR.

multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 1) — failed step(s): Run diffusion server tests (2-GPU)

Commit Info

• sglang: e0dc8b71
• aiter: 7d604afe v0.1.14rc1.dev240+g7d604afe5 (source: dev image preserved — [CI-AITER-CHECK] Dev/patched version detected → skipping rebuild)

Failed Tests

Test File	Test Function	Error	Log
sglang/multimodal_gen/test/server/test_server_2_gpu.py	TestDiffusionServerTwoGpu::test_diffusion_generation[flux2_modelopt_fp8_tp2_t2i]	RuntimeError: CUDA error: HIPBLAS_STATUS_NOT_SUPPORTED in torch._scaled_mm	Log
sglang/multimodal_gen/test/server/test_server_2_gpu.py	TestDiffusionServerTwoGpu::test_diffusion_generation[ideogram4_fp8_tp2_t2i]	ERROR (collection/setup): 403 Client Error — gated HF repo ideogram-ai/ideogram-4-fp8	Log

Final pytest tally: 1 failed, 9 passed, 1 skipped, 12 deselected, 8 warnings, 1 error.

Failure Summary

The 2-GPU diffusion server suite has two distinct problems. (1) flux2_modelopt_fp8_tp2_t2i fails: the FLUX.2 ModelOpt-FP8 transformer's first FP8 linear (to_q) raises RuntimeError: CUDA error: HIPBLAS_STATUS_NOT_SUPPORTED inside torch._scaled_mm during server warmup and again on the actual request. (2) ideogram4_fp8_tp2_t2i errors out because the model repo ideogram-ai/ideogram-4-fp8 is a gated HuggingFace repo returning 403 (not a code defect).

Regression Status

Known recurring failure (both items). The FLUX2 FP8 failure and the ideogram 403 error reproduce identically in every queryable scheduled run.

Recent history of test_server_2_gpu.py (test_diffusion_generation[flux2_modelopt_fp8_tp2_t2i]) in job multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 1) (branch=main, event=schedule, completed runs only):

Date	Run	Job	Test File Status	Failed Function	Error
Jun 23 13:03	28028218911	job	❌ Failed	flux2_modelopt_fp8_tp2_t2i	HIPBLAS_STATUS_NOT_SUPPORTED
Jun 23 07:20	28009346928	job	❌ Failed	flux2_modelopt_fp8_tp2_t2i	HIPBLAS_STATUS_NOT_SUPPORTED
Jun 23 00:51	27994433780	job	❌ Failed	flux2_modelopt_fp8_tp2_t2i	HIPBLAS_STATUS_NOT_SUPPORTED
Jun 22 13:56	27958083310	job	❌ Failed	flux2_modelopt_fp8_tp2_t2i	(same)
Jun 22 01:00	27923265639	job	❌ Failed	flux2_modelopt_fp8_tp2_t2i	(same)
Jun 21 12:45	27904738325	job	❌ Failed	flux2_modelopt_fp8_tp2_t2i	(same)

No green run for this test file found in the queryable window (lookback ≥ Jun 21 12:45 UTC, last 7 scheduled runs). First-observed failure is at or before the start of this window — treat as persistently red, not a fresh regression.

Failure Cluster

ROCm FP8 _scaled_mm HIPBLAS_STATUS_NOT_SUPPORTED (FLUX2 ModelOpt-FP8); plus a separate gated-repo 403 collection error.

Facts (verified observations, no interpretation)

• Verbatim error: RuntimeError: CUDA error: HIPBLAS_STATUS_NOT_SUPPORTED when calling \HIPBLAS_STATUS_NOT_SUPPORTED`` (28 occurrences in the current log).
• Stack trace (verified): flux_2.py:179 _get_qkv_projections → attn.to_q → linear.py:427 → modelopt_quant.py:425 apply → fp8_utils.py:1695 apply_fp8_linear → torch._scaled_mm.
• The error fires twice: during synthetic server warmup (Synthetic server warmup failed; continuing startup) and on the real request, returning 500 Internal Server Error → test FAILED.
• ideogram error is 403 Client Error … Access to model ideogram-ai/ideogram-4-fp8 is restricted and you are not in the authorized list — repo gated, server never started.
• aiter version is a preserved dev build (v0.1.14rc1.dev240+g7d604afe5), identical mechanism across runs; aiter is unchanged between passing-other-tests and this failure (9 other diffusion tests in the same job PASSED).
• Reproducibility: identical FLUX2 FP8 failure + ideogram error in all 6 prior completed scheduled runs checked (Jun 21–23).

Hypothesised Causes (with confidence, with disconfirming evidence)

• [HIGH] hipBLAS on the MI325 (gfx942) image does not support the FP8 e4m3fn torch._scaled_mm configuration emitted by apply_fp8_linear for the FLUX.2 ModelOpt-FP8 transformer.
  • Supporting: error is HIPBLAS_STATUS_NOT_SUPPORTED raised directly inside torch._scaled_mm (fp8_utils.py:1695); the 9 non-FP8 diffusion tests in the same job pass on the same GPUs; open fix PR #28889 explicitly targets "FP8 e4m3fn GEMM on ROCm … dequant->bf16 fallback in apply_fp8_linear".
  • Disconfirming: (none found) — the symptom, stack frame, and PR title all converge on the same FP8 GEMM path.
  • How to verify: on a branch with #28889 applied, re-run test_diffusion_generation[flux2_modelopt_fp8_tp2_t2i] on an MI325 2-GPU runner.
• [FACT] ideogram4_fp8_tp2_t2i errors because ideogram-ai/ideogram-4-fp8 is a gated HF repo returning 403 in CI.
  • Supporting: 403 Client Error … Access to model … is restricted and you are not in the authorized list verbatim in log.
  • Disconfirming: (none) — directly observable; not a code bug.
  • How to verify: grant the CI HF token access to the gated repo, or skip/deselect the test on AMD CI until access is provisioned.

In-flight Fix Check

• FLUX2 FP8: ✅ #28889 "[AMD] Fix FP8 e4m3fn GEMM on ROCm: dequant->bf16 fallback in apply_fp8_linear (fixes flux2 modelopt-fp8 HIPBLAS_STATUS_NOT_SUPPORTED)" (open since 2026-06-22, non-draft) — chase reviewers, do NOT open a duplicate. (Search: is:pr is:open HIPBLAS_STATUS_NOT_SUPPORTED.)
• ideogram 403: ❌ none found in open PRs — needs CI HF-token access provisioning or a test skip on AMD. (Search: is:pr is:open flux2 fp8, no access-related PR.)

Suggested Triage Steps

• Verify in-flight fix #28889 merges cleanly and re-run test_diffusion_generation[flux2_modelopt_fp8_tp2_t2i] on MI325 2-GPU to confirm it clears the HIPBLAS_STATUS_NOT_SUPPORTED failure.
• For ideogram4_fp8_tp2_t2i: grant the CI HuggingFace token access to the gated ideogram-ai/ideogram-4-fp8 repo, or deselect/skip this case on AMD CI until access is provisioned, so it stops surfacing as a recurring ERROR.

Status (factual, no priority assignment)

• Persistence: flux2_modelopt_fp8_tp2_t2i failing in every completed scheduled run from at least Jun 21 12:45 UTC through Jun 23 (≥7 runs, no green run in window). ideogram4_fp8_tp2_t2i ERROR present in all the same runs.
• Scope: job multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 1) in pr-test-amd.yml; one FP8 test fails, one gated-repo test errors, 9 other diffusion tests pass.
• Blocked work: FLUX.2 ModelOpt-FP8 (and ideogram-4-fp8) 2-GPU diffusion CI signal on AMD.
• In-flight fix: FLUX2 FP8 → ✅ #28889 (open since 2026-06-22); ideogram 403 → ❌ none.

stage-b-test-large-8-gpu-mi35x-disaggregation-amd (linux-mi35x-gpu-8.fabric) — failed step(s): Run test_disaggregation

Commit Info

• sglang: e0dc8b71
• aiter: 7d604afe5 (v0.1.14rc1.dev240+g7d604afe5, source: dev image — Dockerfile default matched, no override)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/disaggregation/test_disaggregation_basic.py	TestDisaggregationAccuracy.test_gsm8k	KVTransferError: ... timed out after 300.0s in KVPoll.WaitingForInput → server 500 → KeyError: 'answer' → retry() exceed maximum number of retries	Log
test/registered/amd/disaggregation/test_disaggregation_pp.py	TestDisaggregationDecodePPAccuracy.setUpClass	RuntimeError: The memory capacity is unbalanced. Some GPUs may be occupied by other processes. → Server process exited with code -9	Log
test/registered/amd/disaggregation/test_mori_transfer_engine_e2e.py	TestMoriTransferEngineE2E.setUpClass	torch.OutOfMemoryError: HIP out of memory. Tried to allocate 16.50 GiB. GPU 1 ... 271.29 GiB is allocated by PyTorch → Server process exited with code -9	Log

Failure Summary

The test_disaggregation step ran 3 test files; 0/3 passed. test_disaggregation_basic.py::test_gsm8k failed because every decode KV transfer timed out (300.0s in KVPoll.WaitingForInput), so the prefill→decode disaggregation never completed and the gsm8k eval got empty responses. The subsequent two files (test_disaggregation_pp.py, test_mori_transfer_engine_e2e.py) both crashed in setUpClass with GPU-memory errors (unbalanced memory / HIP OOM with ~271 GiB still held by PyTorch on GPU 1) before running any test.

Regression Status

• test_disaggregation_basic.py (test_gsm8k) — Known recurring failure (same KVTransfer timeout in the prior run).
• test_disaggregation_pp.py (setUpClass) — Known recurring failure (same "memory unbalanced" in the prior run).
• test_mori_transfer_engine_e2e.py (setUpClass) — New regression today (passed in the prior run → today OOM).

Recent history of job stage-b-test-large-8-gpu-mi35x-disaggregation-amd (linux-mi35x-gpu-8.fabric) (pr-test-amd.yml, branch=main, event=schedule, completed runs only):

Date	Run	basic test_gsm8k	pp setUpClass	mori setUpClass
Jun 23 19:xx	28048998470 (this run)	❌ KVTransfer timeout	❌ mem unbalanced	❌ HIP OOM
Jun 23 16:xx	28028218911	❌ KVTransfer timeout	❌ mem unbalanced	✅ Passed (1/3)
Jun 23 07:xx	28009346928	❌ (job failure)	—	—
Jun 23 00:xx	27994433780	❌ (job failure)	—	—

(Job has failed in every completed scheduled run in the lookback window. The basic/pp failures are persistent; mori is newly red as of this run. No green run found for test_gsm8k in the queryable window — lookback ≈ last 4 scheduled runs / ~24h.)

Failure Cluster

Two related clusters:

• A — "PD decode KV transfer timeout (KVPoll.WaitingForInput 300s) on MI35x" → test_gsm8k.
• B — "GPU memory not released → unbalanced/HIP-OOM in later disaggregation setUpClass" → test_disaggregation_pp.py + test_mori_transfer_engine_e2e.py.

Facts (verified observations)

• Every decode request in test_gsm8k failed identically: Decode transfer failed ... KVTransferError ... timed out after 300.0s in KVPoll.WaitingForInput (hundreds of occurrences, lines ~19960–24732), surfaced to the client as HTTP 500 Internal Server Error.
• Final basic-test traceback: KeyError: 'answer' in few_shot_gsm8k.py:123 → Exception: retry() exceed maximum number of retries (Ran 6 tests in 998.223s, FAILED (errors=1)).
• test_disaggregation_pp.py setUpClass: pre_model_load_memory=279.64, local_gpu_memory=1153.04 → "memory unbalanced ... GPUs may be occupied by other processes" → exit -9 (Ran 0 tests).
• test_mori_transfer_engine_e2e.py setUpClass: HIP OOM allocating the aiter workspace buffer (aiter_backend.py:274); GPU 1 reports 271.29 GiB already allocated by PyTorch out of 287.98 GiB before this server even loads → exit -9 (Ran 0 tests).
• Same test_gsm8k KVTransfer timeout and same pp "memory unbalanced" appeared in prior run 82961543942; mori passed there (1/3 passed) vs 0/3 now.
• aiter identical (dev image g7d604afe5) across this and the prior run; runner linux-mi35x-gpu-8.fabric (gfx950).

Hypothesised Causes (with confidence)

• [MEDIUM] Cluster A: the PD/KV-transfer path (mori transfer backend, --disaggregation-transfer-backend mori, RDMA) on MI35x fails to complete decode-side KV reception, so all rooms time out at the 300s KVPoll.WaitingForInput deadline.
  • Supporting: 100% of decode requests time out with the identical error; reproduces in two consecutive runs; this is the disaggregation transport, not model compute.
  • Disconfirming: no green run of test_gsm8k found in the lookback window, so I cannot pin a pass_sha..fail_sha; persistence suggests an environment/infra or longer-standing issue rather than a fresh code regression.
  • How to verify: bisect scheduled-run history further back to find the last green test_gsm8k; check RDMA fabric health on linux-mi35x-gpu-8.fabric; re-run with --disaggregation-transfer-backend alternative to isolate mori.
• [MEDIUM] Cluster B: the pp/mori setUpClass crashes are a cascade from GPU memory not being freed by the preceding disaggregation server processes — ~271 GiB remains allocated on GPU 1 before the next test loads, producing both "memory unbalanced" and HIP OOM.
  • Supporting: GPU 1 shows 271 GiB held by PyTorch at the start of the mori server; pp sees a partially-occupied GPU; mori was green in the prior run when basic left the GPU cleaner — i.e. mori's pass/fail tracks leftover memory, not its own code.
  • Disconfirming: pp also failed in the prior run, so pp's failure is not purely explained by today's heavier leak; pp may have an independent multi-GPU memory-balance issue.
  • How to verify: confirm kill_process_tree / GPU cleanup runs between files in the disaggregation suite; add a VRAM-clear/assert step between the 3 files; if mori passes in isolation, Cluster B is confirmed as cleanup-cascade.

In-flight Fix Check

• In-flight fix: ⚠️ partial — #27093 ("mori PD fix: nested state metadata, per-layer kv_item_len, and DSV4 pool page alignment") touches the mori PD transfer path and may be relevant to Cluster A; verify whether it addresses the MI35x KVPoll timeout before opening a new PR.
• In-flight fix: ❌ none found specifically matching the KVTransfer WaitingForInput timeout or the disaggregation GPU-cleanup cascade (searched open PRs for disaggregation+KVTransfer, disaggregation+timed+out, mori+disaggregation).

Suggested Triage Steps

• Bisect the scheduled-run history of this job further back to find the last green test_gsm8k and establish a pass_sha..fail_sha window (only ~24h checked here — all red).
• Inspect RDMA/fabric health on runner linux-mi35x-gpu-8.fabric; the uniform 300s WaitingForInput timeout across all rooms points at the decode-side KV reception/transport, not model compute.
• Verify #27093 applies to this MI35x mori path and re-run the disaggregation suite on a branch with it.
• For Cluster B: add/confirm GPU-memory teardown between the 3 disaggregation files (the ~271 GiB residual on GPU 1 indicates the basic-test servers aren't fully reaped); run test_mori_transfer_engine_e2e.py in isolation — if it passes, the OOM is a cleanup cascade, not a mori bug.

Status

• Persistence: test_gsm8k (KVTransfer timeout) and test_disaggregation_pp.py setUpClass have failed in every completed scheduled run checked (~24h); test_mori_transfer_engine_e2e.py first failed in this run (was green in the immediately prior run).
• Scope: 1 job (stage-b-test-large-8-gpu-mi35x-disaggregation-amd) on MI35x/gfx950; all 3 disaggregation test files (0/3 passed).
• Blocked work: blocks PD-disaggregation CI signal on AMD MI35x.
• In-flight fix: ⚠️ #27093 possibly relevant to mori PD (Cluster A); no direct fix found for the KVPoll timeout or the GPU-cleanup cascade.

multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 1) — failed step(s): Run diffusion server tests (2-GPU)

Commit Info

• sglang: 5e6d7c16
• aiter: 7d604afe5 (v0.1.14rc1.dev240+g7d604afe5, source: dev image build preserved — [CI-AITER-CHECK] Dev/patched version detected … skipping rebuild)

Failed Tests

Test File	Test Function	Error	Log
python/sglang/multimodal_gen/test/server/test_server_2_gpu.py	TestDiffusionServerTwoGpu::test_diffusion_generation[flux2_modelopt_fp8_tp2_t2i]	openai.InternalServerError: Internal Server Error (HTTP 500), caused server-side by RuntimeError: CUDA error: HIPBLAS_STATUS_NOT_SUPPORTED in torch._scaled_mm	Log

Result line: 1 failed, 10 passed, 1 skipped, 12 deselected … in 2660.14s. The other 10 scenarios in this shard passed; qwen_image_t2i_2_gpus was SKIPPED.

Failure Summary

The 2-GPU diffusion server test flux2_modelopt_fp8_tp2_t2i (FLUX.2-dev with a ModelOpt FP8 transformer, --tp-size 2) returned HTTP 500 to the OpenAI client. The server-side cause is a RuntimeError: CUDA error: HIPBLAS_STATUS_NOT_SUPPORTED raised at the FP8 GEMM during the denoising step. The same error fires during synthetic server warmup ("Synthetic server warmup failed; continuing startup") and again on the actual generation request, so the server starts but cannot produce output.

Regression Status

Known recurring failure (likely Never-passed on AMD for this scenario).

Recent history of test_server_2_gpu.py (test_diffusion_generation[flux2_modelopt_fp8_tp2_t2i]) in job multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 1) (branch main, event schedule, completed runs only):

Date	Run	Job	Test File Status	Failed Function	Error
Jun 24	28067315750	shard 1	❌ Failed	flux2_modelopt_fp8_tp2_t2i	HIPBLAS_STATUS_NOT_SUPPORTED
Jun 23	28028218911	shard 1	❌ Failed	flux2_modelopt_fp8_tp2_t2i	HIPBLAS_STATUS_NOT_SUPPORTED
Jun 23	28009346928	shard 1	❌ Failed	flux2_modelopt_fp8_tp2_t2i	HIPBLAS_STATUS_NOT_SUPPORTED
Jun 23	27994433780	shard 1	❌ Failed	flux2_modelopt_fp8_tp2_t2i	HIPBLAS_STATUS_NOT_SUPPORTED
Jun 22	27923265639	shard 1	❌ Failed	flux2_modelopt_fp8_tp2_t2i	HIPBLAS_STATUS_NOT_SUPPORTED
Jun 21	27897611658	shard 1	❌ Failed	flux2_modelopt_fp8_tp2_t2i	HIPBLAS_STATUS_NOT_SUPPORTED

Failing identically with the same HIPBLAS_STATUS_NOT_SUPPORTED symptom in every completed schedule run queried (Jun 21 → Jun 24). No passing run for this scenario was found in the lookback window. The other tests in the shard pass, so the shard-level "failure" is attributable solely to this one scenario.

Failure Cluster

HIPBLAS_STATUS_NOT_SUPPORTED in FP8 torch._scaled_mm (ROCm FLUX.2 ModelOpt-fp8)

Facts (verified observations, no interpretation)

• Server-side stack trace (verbatim, log):
  • …/multimodal_gen/runtime/layers/quantization/modelopt_quant.py:425, in apply → return apply_fp8_linear(
  • …/srt/layers/quantization/fp8_utils.py:1695, in apply_fp8_linear → output = torch._scaled_mm(
  • RuntimeError: CUDA error: HIPBLAS_STATUS_NOT_SUPPORTED when calling \HIPBLAS_STATUS_NOT_SUPPORTED``
• The error first appears at warmup: [denoising_step_0] Error during execution after 8842.9 ms: CUDA error: HIPBLAS_STATUS_NOT_SUPPORTED (log), then on the generation request → POST /v1/images/generations … 500 Internal Server Error.
• The failing GEMM is the FP8 rowwise torch._scaled_mm branch in fp8_utils.py (lines 1680–1688), invoked via modelopt_quant.py apply for the FLUX.2 ModelOpt-fp8 transformer (--transformer-path lmsys/flux2-dev-modelopt-fp8-sglang-transformer).
• Runner GPU arch: mi30x / gfx942 (MI325), from [CI-AITER-CHECK] markers.
• Reproducibility: 6/6 completed scheduled runs Jun 21–24, same test, same symptom; all other scenarios in the shard pass.
• aiter version unchanged (dev image v0.1.14rc1.dev240+g7d604afe5 preserved across runs).

Hypothesised Causes (with confidence, with disconfirming evidence)

• [HIGH] The FP8 path in apply_fp8_linear (fp8_utils.py:1695) calls torch._scaled_mm with a quantization/scale layout that hipBLAS(Lt) on gfx942 does not support for this model's FP8 (e4m3fn) weights, so the GEMM returns HIPBLAS_STATUS_NOT_SUPPORTED.
  • Supporting: stack trace lands exactly on torch._scaled_mm in fp8_utils.py:1695 ([FACT]); the in-flight fix PR #28889 targets this exact file and describes a dequant→bf16 fallback for the same HIPBLAS_STATUS_NOT_SUPPORTED on ROCm; failure is AMD-only and confined to the ModelOpt-fp8 scenario while bf16/other scenarios in the shard pass.
  • Disconfirming: not a regression introduced by a recent commit — the test has never been observed green in the queryable window, so this is a missing-ROCm-support gap rather than a code change that broke it. No A/B baseline is needed (not a scout run).
  • How to verify: check out PR #28889 on a branch and re-run test_diffusion_generation[flux2_modelopt_fp8_tp2_t2i] on an MI325 (gfx942) runner; confirm the dequant→bf16 fallback path is taken and the 500 disappears.

In-flight Fix Check

• In-flight fix: ✅ #28889 [AMD] Fix FP8 e4m3fn GEMM on ROCm: dequant->bf16 fallback in apply_fp8_linear (fixes flux2 modelopt-fp8 HIPBLAS_STATUS_NOT_SUPPORTED) — open since 2026-06-22, not draft, changes the single file python/sglang/srt/layers/quantization/fp8_utils.py. Directly matches this failure; chase reviewers, do not open a duplicate.
• (Searched repo:sgl-project/sglang is:pr is:open with keywords HIPBLAS scaled_mm (0 hits) and flux2 fp8 (3 hits, #28889 the relevant one).)

Suggested Triage Steps

• Verify in-flight fix #28889 resolves this: apply on a branch and re-run test_diffusion_generation[flux2_modelopt_fp8_tp2_t2i] on an MI325 (gfx942) runner in the CI container.
• While the fix is pending, this is a known persistent red on pr-test-amd shard 1 — it can be treated as a tracked failure (not a new regression) so it doesn't mask new breakage in the other 10 scenarios.
• If #28889 is confirmed, ping reviewers to land it; if it does not fully fix the rowwise FP8 case, inspect the fp8_utils.py:1680–1688 rowwise torch._scaled_mm branch for an additional gfx942 capability guard.

Status

• Persistence: Failing in every completed scheduled run from Jun 21 to Jun 24 (6/6 queried), same test, same HIPBLAS_STATUS_NOT_SUPPORTED symptom; no green run found in the lookback window (likely never-passed on AMD).
• Scope: 1 test scenario (flux2_modelopt_fp8_tp2_t2i) in multimodal-gen-test-2-gpu-amd (…, 1) of pr-test-amd.yml. The other 10 scenarios in the shard pass.
• Blocked work: FLUX.2 ModelOpt-FP8 (e4m3fn) 2-GPU diffusion serving on AMD ROCm/gfx942.
• In-flight fix: ✅ #28889 (open since 2026-06-22, awaiting merge) — touches fp8_utils.py, the exact failing file.

stage-b-test-large-8-gpu-mi35x-disaggregation-amd (linux-mi35x-gpu-8.fabric) — failed step(s): Run test_disaggregation

Commit Info

• sglang: 5e6d7c16
• aiter: 7d604afe (v0.1.14rc1.dev240+g7d604afe5, source: dev image preserved — AITER_COMMIT_OVERRIDE empty, dev/patched version detected so no rebuild)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/disaggregation/test_disaggregation_basic.py	test_gsm8k (TestDisaggregationAccuracy)	KeyError: 'answer' after retry() exceed maximum number of retries — all 200 GSM8K /generate calls returned HTTP 500 from KVTransferError … timed out after 300.0s in KVPoll.WaitingForInput	Log
test/registered/amd/disaggregation/test_disaggregation_pp.py	setUpClass (TestDisaggregationDecodePPAccuracy)	RuntimeError: Server process exited with code -9; scheduler raised RuntimeError: The memory capacity is unbalanced … + RDMA Failed to register memory … Cannot allocate memory [12]	Log

(test_mori_transfer_engine_e2e.py passed → 1/3 passed.)

Failure Summary

Two of three disaggregation test files failed on the MI35x 8-GPU runner. In test_disaggregation_basic.py::test_gsm8k, every one of the 200 prefill→decode requests timed out after 300 s in KVPoll.WaitingForInput (mooncake KV transfer never completed), so all /generate calls returned HTTP 500 and the harness raised KeyError: 'answer'. In test_disaggregation_pp.py::setUpClass, the decode/prefill servers were SIGKILLed (exit -9) after the scheduler hit a "memory capacity is unbalanced" check and the mooncake transfer engine failed RDMA memory registration with ENOMEM.

Regression Status

New regression (recurring since 2026-06-22 ~20:35 UTC). Both failing test files passed cleanly until run 27958083310 and have failed every completed run since.

Recent history of test_disaggregation_basic.py (test_gsm8k) and test_disaggregation_pp.py (setUpClass) in job stage-b-test-large-8-gpu-mi35x-disaggregation-amd (branch main, event schedule, completed runs only):

Date (UTC)	Run	Job	basic.py / pp.py Status	Detail
Jun 24 02:24	28067315750	job	❌ / ❌	KV transfer 300 s timeout · exit -9 (this run)
Jun 23 16:33	28028218911	job	❌ / ❌	same pattern (basic 1388s)
Jun 23 10:40	28009346928	job	❌ / ❌	same pattern (basic 1373s)
Jun 23 01:56	27994433780	job	❌ / ❌	same pattern (basic 1374s)
Jun 22 20:35	27977371862	job	❌ / ❌	first failing run (0/3, mori also failed)
Jun 22 14:15	27958083310	job	✅ / ✅	last passing run (3/3, basic 254s)
Jun 22 01:18	27923265639	job	✅ / ✅	3/3 passed

First observed failure: 2026-06-22 ~20:35 UTC. Last known pass: 2026-06-22 ~14:15 UTC. Regression window: 4923bb93..adc203df (13 commits). (Current run is the 5th consecutive completed failure.)

Failure Cluster

Mooncake PD KV-transfer timeout / RDMA memory-registration failure on MI35x disaggregation

Facts (verified observations)

• test_gsm8k ran the full 200/200 GSM8K progress bar in 10:00, every request ending in POST /generate HTTP/1.1 500 Internal Server Error; each carried KVTransferError(bootstrap_room=…): Request … timed out after 300.0s in KVPoll.WaitingForInput. Prefill side logged Failed due to an unknown reason from another rank. Final harness error is KeyError: 'answer' at few_shot_gsm8k.py:123 → test_disaggregation_basic.py:113.
• test_disaggregation_pp.py decode/prefill servers died with exit -9. Scheduler stack: model_runner.py:1287 init_torch_distributed → RuntimeError: The memory capacity is unbalanced. … pre_model_load_memory=45.41…, local_gpu_memory=279.64…/612.58…. Immediately followed by 4× rdma_context.cpp:243 Failed to register memory … Cannot allocate memory [12] (ENOMEM).
• Both PD servers used --disaggregation-transfer-backend mooncake --disaggregation-ib-device rdma0..rdma7, TP2 PP2.
• test_mori_transfer_engine_e2e.py passed in this run (and the last 3 runs); it failed only in the first failing run 27977371862.
• aiter version unchanged across passing and failing runs (dev image v0.1.14rc1.dev240+g7d604afe5 preserved; AITER_COMMIT_OVERRIDE empty).
• Regression window commits (13): the HEAD commit adc203df touches only the Rust gateway (pd_router.rs); commit b28e9901 rewrites server_args.py (~4625 lines) including all disaggregation_* arg definitions.

Hypothesised Causes (with confidence)

• [LOW] ServerArgs Annotated-style migration b28e9901 (#28919) subtly changed a disaggregation arg default/parse.
  • Supporting: lands inside the regression window; rewrites every disaggregation_* / mooncake_* field definition that this exact test exercises.
  • Disconfirming: the diff is mechanical and disaggregation_transfer_backend default is still "mooncake", disaggregation_ib_device default still None; no code-level evidence any default/value actually flipped, and the symptom is at the RDMA/transfer-engine layer, not arg parsing.
  • How to verify: git revert b28e9901 on a branch, re-run test_disaggregation_basic.py on an MI35x 8-GPU runner.
• [LOW] MI35x linux-mi35x-gpu-8.fabric runner environment issue — RDMA memory-registration ENOMEM [12] + "memory capacity unbalanced" point to a memlock ulimit, RDMA resource exhaustion, or leftover GPU processes rather than an sglang code path.
  • Supporting: failure surfaces in C++ transfer engine (rdma_context.cpp) and CUDA memory checks, not the Python stack; basic.py hangs ~1377 s and may leave the prefill/decode servers occupying GPU memory, so pp.py::setUpClass then sees pre_model_load_memory=45 GB and dies (cascade).
  • Disconfirming: the pass→fail transition is sharp at a single ~6 h window and persists across 5 consecutive completed runs, which argues against pure flake/transient infra.
  • How to verify: on the runner check ulimit -l, residual sglang.launch_server processes / GPU memory between jobs, and dmesg for RDMA/OOM; confirm whether pp.py passes when run in isolation.
• [LOW] PD-router change adc203df (#27430).
  • Supporting: it is the HEAD commit of the first failing run and is PD-routing related.
  • Disconfirming: it only modifies sgl-model-gateway/src/routers/http/pd_router.rs (Rust gateway), which this test's python load-balancer fixture does not use; cannot explain a mooncake KV-transfer RDMA timeout.

No hypothesis above LOW confidence. Needs a git bisect across 4923bb93..adc203df plus a runner-environment check to disambiguate code regression vs infra.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for disaggregation+mi35x, mooncake+register+memory, disaggregation+timeout+decode; matches are NIXL/feature PRs (e.g. #28945, #28515), none targeting this mooncake KV-transfer timeout regression.

Suggested Triage Steps

• Bisect the regression window 4923bb93..adc203df; prioritize reverting b28e9901 on a branch and re-running test_disaggregation_basic.py on an MI35x 8-GPU runner.
• In parallel, inspect the linux-mi35x-gpu-8.fabric runner for memlock ulimit, leftover sglang.launch_server processes / occupied GPU memory between jobs, and RDMA resource limits — the Cannot allocate memory [12] on ibv_reg_mr is environment-level.
• Determine whether pp.py::setUpClass is a cascade of basic.py not cleaning up: run test_disaggregation_pp.py standalone (fresh GPUs) and check if it passes.
• If bisect points at no sglang commit, treat as MI35x infra regression and engage runner ops.

Status

• Persistence: failing every completed schedule run since 2026-06-22 ~20:35 UTC — 5 consecutive completed failures (~1.5 days).
• Scope: this job only (stage-b-test-large-8-gpu-mi35x-disaggregation-amd), 2 of 3 disaggregation test files (test_disaggregation_basic.py, test_disaggregation_pp.py); test_mori_transfer_engine_e2e.py passes.
• Blocked work: MI35x PD-disaggregation (mooncake) CI signal.
• In-flight fix: ❌ none found in open PRs.

multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 1) — failed step(s): N/A

Commit Info

• sglang: 5e6d7c16
• aiter: N/A — the job log is unavailable (see below), so the [CI-AITER-CHECK] marker cannot be read. Dockerfile default for this branch is 7d604afe (docker/rocm.Dockerfile:34), but it could not be verified for this run.

Failed Tests

Test File	Test Function	Error	Log
python/sglang/multimodal_gen/test/server/test_server_1_gpu.py (partition 1 of ONE_GPU_CASES)	Unidentifiable — log & artifacts unavailable	Unknown — see Failure Summary	Job (log returns BlobNotFound)

⚠️ I could not pin the failing test function or error. Both the per-job log and the run-level log return BlobNotFound from the GitHub API, and no diffusion-failures artifacts were uploaded for this run (the "Upload diffusion failure artifacts" step never ran — see below). The same BlobNotFound occurs for every AMD job log I tried, including historical ones, so this is a systematic log-retention/access issue in this environment, not specific to this job.

Failure Summary

The job multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 1) failed during step 9, "Run diffusion server tests (1-GPU)" (run_suite.py --suite 1-gpu --partition-id 1 --total-partitions 4 -k "not flux_2"). Partition 1 runs one of the three LPT-balanced bins of ONE_GPU_CASES (T2I/TI2I/T2V/LoRA diffusion server tests, FLUX.2 excluded) from test_server_1_gpu.py. The job ran ~55 min (02:07:35→03:02:39 UTC), the longest of the four 1-GPU shards, with step 9 still showing in_progress at job conclusion failure — a hard-termination signature (process kill / runner loss / crash) rather than a clean pytest failure. The exact failing case and error are not recoverable because the log and artifacts are gone.

Regression Status

Flaky test (intermittent) — at the shard level.

Recent history of multimodal-gen-test-1-gpu-amd partition 1 (pr-test-amd.yml, branch main, event schedule, completed runs only):

Date (UTC)	Run	Job (part 1)	Shard Status	Notes
Jun 24 00:48	28067315750	job	❌ Failed	this run; ~55 min, hard kill
Jun 23 13:03	28028218911	job	✅ Passed	—
Jun 23 07:20	28009346928	job	✅ Passed	(part 3 failed this run)
Jun 23 00:51	27994433780	job	✅ Passed	—
Jun 22 19:10	27977371862	job	❌ Failed	log also BlobNotFound, no artifacts
Jun 22 13:56	27958083310	job	✅ Passed	—
Jun 22 08:09	27938642367	job	❌ Failed	log also BlobNotFound, no artifacts

Partition 1 failed in 3 of the last 7 completed scheduled runs (~43%), passing in between with the same kind of head SHAs — an intermittent pattern, not a clean regression. Caveat: partition contents are assigned by LPT estimated-time bin-packing, so the specific case in "part 1" can shift between runs if cases/estimates change; the shard-level verdict is reliable, but I cannot confirm the same case failed each time without logs.

Failure Cluster

GPU diffusion 1-GPU server shard hard-termination (suspected timeout/OOM during model warmup/inference)

Facts (verified observations)

• Step 9 "Run diffusion server tests (1-GPU)" is the failing step; it shows status=in_progress while the job's conclusion=failure (GitHub jobs API). Steps 10 "Upload diffusion failure artifacts" and 20 "Post Checkout" never started. [FACT]
• Job duration ~55 min; step timeout is timeout-minutes: 90 (pr-test-amd.yml:712), so the step did not hit its own timeout. [FACT]
• The other three 1-GPU shards in this run passed: part 0 (33 min), part 2 (42 min), part 3 (12 min). Only part 1 failed. [FACT]
• The per-job log, the two prior part-1 failure logs, and the run-level log ZIP all return BlobNotFound / 404; run/.../artifacts lists no diffusion-failures artifact for this run or the two prior part-1 failures. [FACT]
• run_suite.py retries "retryable" failures (OOM / TimeoutError / SafetensorError / FileNotFoundError / perf-assert) up to 6 times before failing (run_suite.py:494-548, 621-691); a multi-retry loop is consistent with the ~55 min runtime. [FACT]
• This suite has a documented timeout-flakiness history: test_update_weights_from_disk.py is disabled with the comment "24 timeout failures since 2026-04-09 across multimodal-gen-test-1-gpu" (gpu_cases.py:867-869). [FACT]
• Separately, multimodal-gen-test-2-gpu-amd partition 1 failed in all 6 recent completed scheduled runs + this run (a persistent, distinct failure — not this job). [FACT]

Hypothesised Causes (with confidence)

• [LOW] Intermittent timeout/OOM on a heavy case in the 1-GPU partition-1 bin (model warmup/inference exceeding limits and being killed), consistent with this suite's known timeout flakiness.
  • Supporting: longest shard runtime (~55 min) with step left in_progress (hard kill); documented "24 timeout failures since 2026-04-09" in this exact suite; retry loop targets OOM/timeout; shard passes ~57% of runs.
  • Disconfirming: no log/stack trace to confirm the kill signal or OOM; no artifacts written (a clean pytest timeout would normally still upload artifacts via if: always() — but the upload step never ran, suggesting the runner/process died abruptly, which could equally be infra).
  • How to verify: re-run the shard and capture the log live (run_suite.py streams pytest output); watch dmesg/free -h for OOM-killer; check runner health/ensure_vram_clear.
• [SPECULATION] Infrastructure/runner loss (agent disconnect on linux-mi325-1gpu-sglang) rather than a test fault.
  • Supporting: in_progress step + missing post-steps + total log/artifact absence are also classic runner-loss symptoms.
  • Disconfirming: the shard fails preferentially on part 1 (the heaviest bin), which argues for a workload-dependent cause, not random runner loss.
  • How to verify: check runner-side logs / GitHub runner diagnostics for this run; correlate with other jobs that shared the runner.

No hypothesis reaches MEDIUM — root cause cannot be established without the log. Needs a live re-run with log capture.

In-flight Fix Check

• In-flight fix: ❌ none found for the 1-GPU partition-1 flakiness specifically. Searches run: is:pr is:open multimodal-gen, diffusion flaky, diffusion timeout, multimodal timeout.
• Possibly-related (cannot confirm relevance without knowing the failing case): #28832 "[diffusion] Fix Qwen-Image-Layered latent shape" — only relevant if the failing case was qwen_image_layered_i2i (a ONE_GPU_CASES member). [SPECULATION].

Suggested Triage Steps

• First, fix log/artifact retrieval — every AMD job log returns BlobNotFound and no diffusion-failures artifacts exist, so no failure here can be triaged at the test level. Confirm whether AMD self-hosted runner logs are being uploaded/retained; without this, this report cannot be improved.
• Re-run multimodal-gen-test-1-gpu-amd part 1 at 5e6d7c16 and capture the streamed pytest output + dmesg/free -h to determine whether the kill is OOM, timeout, or runner loss.
• Once the failing case is known, check it against this suite's known timeout flakiness (gpu_cases.py:867) and decide whether to quarantine that case or raise its est-time/timeout.
• Separately escalate the persistent multimodal-gen-test-2-gpu-amd part 1 failure (red in all 6 recent completed runs) — that is a harder failure than this flaky 1-GPU shard and warrants its own investigation.

Status

• Persistence: Intermittent — 1-GPU part 1 failed 3 of last 7 completed scheduled runs (~43%), most recently this run (2026-06-24 00:48 UTC); passes in between.
• Scope: This shard (1-GPU part 1) only for this failure. Note a separate persistent failure in 2-GPU part 1 (all 6 recent runs).
• Blocked work: Intermittently reddens the AMD 1-GPU multimodal-gen CI signal; not consistently blocking.
• In-flight fix: ❌ none found for this flakiness; #28832 only relevant if the failing case is qwen_image_layered_i2i (unconfirmed).
• Investigation blocker: Log and artifacts unavailable (BlobNotFound, no artifacts) — test-level root cause not determinable for this run.

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-06-24 23:07 UTC)

[amd-bot]

pr-test-amd.yml — Cross-Run Pattern Summary (8 runs in lookback window)

Last scanned (UTC): 2026-06-24 23:10 UTC

Agent assessment

Persistent clusters

1. PD-disaggregation KV-transfer / RDMA fault on MI35x — stage-b-test-large-8-gpu-mi35x-disaggregation-amd

• Failed in all 8/8 runs (latest job, run 28067315750, run 27994433780). Red since ~2026-06-18.
• Tests: test/registered/amd/disaggregation/test_disaggregation_basic.py::TestDisaggregationAccuracy.test_gsm8k (KVTransferError … timed out 300s in KVPoll.WaitingForInput) and test_disaggregation_pp.py::TestDisaggregationDecodePPAccuracy.setUpClass (Server process exited -9; RDMA Failed to register memory: Cannot allocate memory [12]).
• Top hypothesis: [HIGH] faulty/resource-exhausted RDMA NIC rdma5 on linux-mi35x-gpu-8.fabric (infra, not code). Disconfirming: error 12 could also be an exhausted RLIMIT_MEMLOCK; no healthy-NIC A/B yet.
• In-flight fix: ✅ #29127 (DEBUG/infra-repro, "do not merge", open 2026-06-24) — chase IT/infra, do not duplicate.

2. FP8 torch._scaled_mm HIPBLAS_STATUS_NOT_SUPPORTED (FLUX.2 ModelOpt-fp8) — multimodal-gen-test-2-gpu-amd (…, 1)

• Failed in all 8/8 runs (latest job, run 28067315750); Never-passed in the queryable window (back to Jun 21).
• Test: python/sglang/multimodal_gen/test/server/test_server_2_gpu.py::test_diffusion_generation[flux2_modelopt_fp8_tp2_t2i] (other 10 scenarios pass).
• Top hypothesis: [MEDIUM] ROCm/gfx942 hipBLAS rejects the FLUX.2 e4m3fn fp8 rowwise shape at fp8_utils.py:1695 (torch._scaled_mm fallback). Disconfirming: aiter image byte-identical across runs, so not an aiter version change.
• In-flight fix: ❌ none found.

Regression candidates (latest-run-only) — both are infra, not code

• multimodal-gen-test-2-gpu-amd (…, 2) job 83273767527: NOT a real regression. Build failure at Install dependencies — mori build hit libabsl_strings_internal.so.20210324: invalid ELF header (corrupt system lib on the runner). Prior 3 completed runs passed this shard's build; diff d5c566e (profile cleanups) is unrelated. [MEDIUM] runner-image corruption. Triage: re-run on a fresh MI325 runner; if green, close as transient.

Flakes / intermittent

• multimodal-gen-test-1-gpu-amd (…, 1) — 3/8 runs, all on 2026-06-24 (28067315750, 28081940667, 28121318394). Abrupt step termination (unfinalized step-9, runner/process crash); all logs expired (BlobNotFound) so test file unknown. [LOW] infra vs [LOW] native crash — undetermined. Recurs on shard 1 only while siblings pass.
• stage-b-test-1-gpu-small-amd-nondeterministic — 2/8 runs (28099674087, 28121318394). test/registered/models/test_vlm_models.py::TestVLMModels.test_vlm_mmmu_benchmark TIMEOUT 1800s (Qwen2.5-VL-3B, 5.4× slower than last pass 331s; not a crash). 5 prior passes → likely flake/infra; window [5e6d7c16..d5c566e] (39 commits) if it reproduces.
• One-off flakes (1/8 each): stage-a-…-1-gpu-small 28048998470 (same mori/libabsl ELF infra fault); stage-b-…-1-gpu-small (…,11) 28067315750 (test_priority_scheduling.py::setUpClass 2400s hang in decode CUDA-graph capture); stage-c-…-8-gpu (…,0) 28009346928 (RCCL allreduce SIGSEGV) & (…,1) 28028218911 (runner lost comms).

Suggested next actions

1. Cluster 1 (largest, 8/8): chase infra on #29127 — rerun with rdma5 removed from --disaggregation-ib-device; check ulimit -l / orphaned sglang.launch_server procs. Resolving this clears the single biggest persistent failure.
2. Cluster 2 (8/8, Never-passed): decide whether to skip/quarantine flux2_modelopt_fp8_tp2_t2i on ROCm until the hipBLAS fp8-shape limitation is addressed; no code path in the window touches it.
3. mori/libabsl ELF corruption (hit 2 distinct jobs): quarantine/recycle the affected MI325 runner and verify image-pull integrity (registry fallback logged in the same runs).
4. 1-GPU diffusion shard-1 crash: re-run before logs expire and retain step-9 output — currently unattributable (BlobNotFound); check runner dmesg/rocm-smi for OOM/GPU reset 2026-06-24.
5. VLM MMMU timeout: re-run 28121318394; if it reproduces, bisect 5e6d7c16..d5c566e on test_vlm_models.py only.

---

Runs analysed

Run	Started (UTC)	Failed jobs
27994433780	2026-06-23 00:59	2
28009346928	2026-06-23 09:40	5
28028218911	2026-06-23 14:27	3
28048998470	2026-06-23 19:17	3
28067315750	2026-06-24 01:42	4
28081940667	2026-06-24 08:25	3
28099674087	2026-06-24 17:14	3
28121318394	2026-06-24 20:24	5

Persistent failures (in every run)

• multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 1) — in 8/8 runs (4433780, 9346928, 8218911, 8998470, 7315750, 1940667, 9674087, 1318394)
• stage-b-test-large-8-gpu-mi35x-disaggregation-amd (linux-mi35x-gpu-8.fabric) — in 8/8 runs (4433780, 9346928, 8218911, 8998470, 7315750, 1940667, 9674087, 1318394)

Latest-only failures (potential regressions)

• multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 2) — in 1/8 runs (1318394)

Flaky / intermittent failures

• multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 1) — in 3/8 runs (7315750, 1940667, 1318394)
• multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 3) — in 1/8 runs (9346928)
• multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 0) — in 1/8 runs (9346928)
• stage-a-test-1-gpu-small-amd (linux-mi325-1gpu-sglang) — in 1/8 runs (8998470)
• stage-b-test-1-gpu-small-amd (linux-mi325-1gpu-sglang, 11) — in 1/8 runs (7315750)
• stage-b-test-1-gpu-small-amd-nondeterministic (linux-mi325-1gpu-sglang) — in 2/8 runs (9674087, 1318394)
• stage-c-test-large-8-gpu-amd (linux-mi325-8gpu-sglang, 0) — in 1/8 runs (9346928)
• stage-c-test-large-8-gpu-amd (linux-mi325-8gpu-sglang, 1) — in 1/8 runs (8218911)

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-06-24 23:10 UTC)

[amd-bot]

pr-test-amd-rocm720.yml — 88 failure(s)

Run started (UTC): 2026-06-23 00:39 UTC
Last scanned (UTC): 2026-06-24 19:29 UTC
Commits: sglang d6aacd2

Summary

Counts: 88 failures · 3 clusters · 87 carrying over (≥3 days) · 1 new today

#	Cluster	Job	Test File	Test Function	Status	Hypothesis (confidence)
1	Concurrency cancellation (scheduled run superseded)	extra-a-test-1-gpu-large-rocm720	— (no test ran)	—	85 jobs · chronic (6–7/7 recent scheduled runs cancelled)	[HIGH] cancel-in-progress: true on schedule events; newer cron run preempts in-flight run
2	Concurrency cancellation (scheduled run superseded)	dsv4-flash-fp4-fp8-rocm720	— (no test ran)	—	part of 85 · chronic	[HIGH] newer schedule run created 1 s before mass cancel; shared concurrency group
3	Concurrency cancellation (runner starvation, 0 steps)	extra-a-test-1-gpu-small-rocm720	N/A	N/A	part of 85 · recurring	[MEDIUM] MI325 1-GPU runner-pool contention; 0 steps vs 8 steps when runner available
4	DeepSeek-V4-Pro slow weight-load → mid-load cancel	dsv4-pro-fp4-rocm720	test/registered/amd/test_deepseek_v4_pro_fp4.py	GSM8K (server never ready)	recurring timing-race (2 cancel / 2 success in window)	[HIGH] ~16 min load even on success + 6 h runner-queue delay leaves no margin before next cron cancels
5	DeepSeek-V4-Pro slow weight-load → mid-load cancel	dsv4-pro-fp4-rocm720	(server launch, not pytest)	DeepSeek-V4-Pro startup	recurring timing-race	[MEDIUM] cold/slow HF fp8 weight load; TP0/TP7 never finished Load weight end
6	mori build failure (corrupt libabsl ELF header)	extra-a-test-1-gpu-small-rocm720	N/A (build, step 5)	N/A	🆕 new today · 1/4 runs (sibling passed same SHA)	[MEDIUM] corrupt libabsl_time_zone.so.20210324 on the specific runner

Cluster: Concurrency cancellation (scheduled run superseded) — 85 jobs

• Affected: the overwhelming majority of the 88 failed jobs across runs 28046873327 (40/49 jobs cancelled at one instant), 28008999561, 27993948299, 27977093213. Spans every stage family: stage-a/b/c, multimodal-gen, dsv4-flash, extra-a, 8-GPU MI35x/MI325 jobs.
• Shared facts [FACT]: jobs conclude cancelled (not failure); log blobs return BlobNotFound; 0 steps; mass cancellation at a single timestamp coinciding (±1–2 s) with a newer scheduled run entering the same group pr-test-amd-rocm720-refs/heads/main. pr-test-amd-rocm720.yml:123 evaluates cancel-in-progress: true for schedule events.
• Top hypothesis [HIGH]: GitHub Actions concurrency control aborts the in-flight scheduled run when the next cron (0 */6 * * * + daily 30 17 * * *) fires on main. Disconfirming: a few jobs lingered past the cancel instant (runner-shutdown lag) — does not contradict. Runner starvation on linux-mi35x-gpu-8/linux-mi325-*gpu is a secondary contributor (0-step cancellations with no superseding run, [MEDIUM]).
• In-flight fix: ❌ none found (orchestration artifact, not a code defect).
• Suggested triage: inspect a completed (non-cancelled) scheduled run such as 28027779773 for real test signal; discuss with workflow owners whether to scope the schedule concurrency group per-run or stagger the two crons so the 6-hourly run isn't killed by the daily run (warn: increases CI resource usage).

Cluster: DeepSeek-V4-Pro slow weight-load → mid-load cancel — 2 jobs

• Affected: dsv4-pro-fp4-rocm720 (test/registered/amd/test_deepseek_v4_pro_fp4.py) and dsv4-pro-fp4-rocm720.
• Shared facts [FACT]: TP8 fp8 checkpoint; soft watchdog (300 s) fires repeatedly but is non-fatal; even successful runs take ~16 min (elapsed=976 s, 81 watchdog fires) to load; cancellation coincides with the next scheduled run queueing. Threads parked in _load_w13 (moe/fused_moe_triton/layer.py:503) H2D copies on a HIP runtime mutex.
• Top hypothesis [HIGH]: pathologically slow DSV4 weight load + long runner-queue delay leaves no margin before concurrency cancels the run. Disconfirming: cannot separate "slower-than-usual load" from "less time" — TP0/TP7 never logged Load weight end.
• In-flight fix: ✅ #28729 (open since 2026-06-19) speeds up DSV4 fp8 weight loading; also ✅ #25498 (CI weight-loader prefetch). Chase reviewers; do not duplicate.
• Suggested triage: pre-warm /sgl-data/hf-cache for deepseek-ai/DeepSeek-V4-Pro; verify #28729 brings load time well under the cron cadence; consider excluding this job from concurrency-based cancellation.

Cluster: mori build failure (corrupt libabsl ELF header) — 1 job, NEW today

• Affected: extra-a-test-1-gpu-small-rocm720 — failed in step 5 Install dependencies (mori build), step 6 skipped.
• Shared facts [FACT]: test_peer_ssd_eviction: error while loading shared libraries: /lib/x86_64-linux-gnu/libabsl_time_zone.so.20210324: invalid ELF header → result 127 (log); an earlier local-registry pull timeout on the same runner (line 1296); sibling extra-a-test-1-gpu-large passed on the identical SHA in the same run.
• Top hypothesis [MEDIUM]: corrupt/truncated system absl library on the specific runner/container. Disconfirming: (none found) — no sglang commit touches absl/mori; it's a base-image system lib.
• In-flight fix: ❌ none found (runner/environment corruption — infra ticket, not a code PR).
• Suggested triage: re-run the job (clean runner should pass); on the affected runner readelf -h /lib/x86_64-linux-gnu/libabsl_time_zone.so.20210324 and recreate the container/image layer if corrupt; correlate with the 10.245.143.50:5000 registry timeout.

Suggested triage order

Rank	Triage step	Targets	Why this first
1	Pull real signal from a completed (non-cancelled) scheduled run, e.g. 28027779773	cluster 1 (85 jobs)	85 of 88 "failures" carry zero test signal — this workflow's results are dominated by orchestration cancellation, not test faults
2	Discuss schedule-concurrency policy (per-run group / stagger crons) with workflow owners	cluster 1	Chronic: 6–7 of last 7 scheduled runs cancelled; fixing it restores CI signal across the whole matrix
3	Chase in-flight #28729 and pre-warm DSV4 weights	cluster 2 (rows 4–5)	Existing fix targets the slow-load root cause; reduces the timing-race cancellations
4	Re-run 83137794582; inspect/recreate corrupt absl lib on that runner	cluster 3 (row 6)	Isolated, new today; cheap to confirm as runner corruption (sibling passed same SHA)

---

Job	Failed Steps	Started
call-pr-test-amd-extra-rocm720 / extra-a-test-1-gpu-large-amd-rocm720 (linux-mi325-1gpu-sglang)	N/A	2026-06-23T18:10
dsv4-flash-fp4-fp8-amd-rocm720	N/A	2026-06-23T18:10
dsv4-pro-fp4-amd-rocm720	N/A	2026-06-23T18:10
call-pr-test-amd-extra-rocm720 / extra-a-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang)	N/A	2026-06-23T18:10
call-pr-test-amd-extra-rocm720 / extra-a-test-2-gpu-large-amd-rocm720 (linux-mi325-2gpu-sglang)	Start CI container	2026-06-23T18:39
stage-b-test-1-gpu-small-amd-nondeterministic-rocm720 (linux-mi325-1gpu-sglang)	N/A	2026-06-23T18:11
stage-b-test-1-gpu-large-amd-rocm720 (linux-mi325-1gpu-sglang, 0)	N/A	2026-06-23T18:11
stage-b-test-2-gpu-large-amd-rocm720 (linux-mi325-2gpu-sglang, 1)	N/A	2026-06-23T18:11
multimodal-gen-test-2-gpu-amd-rocm720 (linux-mi325-2gpu-sglang, 0)	N/A	2026-06-23T18:11
stage-b-test-large-8-gpu-mi35x-disaggregation-amd-rocm720 (linux-mi35x-gpu-8.fabric)	N/A	2026-06-23T18:11
stage-b-test-1-gpu-large-amd-rocm720 (linux-mi325-1gpu-sglang, 1)	N/A	2026-06-23T18:11
multimodal-gen-test-1-gpu-amd-rocm720 (linux-mi325-1gpu-sglang, 2)	N/A	2026-06-23T18:11
stage-c-test-large-8-gpu-amd-mi35x-rocm720 (linux-mi35x-gpu-8, 1)	N/A	2026-06-23T18:11
stage-a-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang)	N/A	2026-06-23T18:11
stage-c-test-large-8-gpu-amd-rocm720 (linux-mi325-8gpu-sglang, 2)	N/A	2026-06-23T18:11
multimodal-gen-test-1-gpu-amd-rocm720 (linux-mi325-1gpu-sglang, 0)	N/A	2026-06-23T18:11
stage-c-test-large-8-gpu-amd-rocm720 (linux-mi325-8gpu-sglang, 0)	N/A	2026-06-23T18:11
stage-c-test-large-8-gpu-amd-rocm720 (linux-mi325-8gpu-sglang, 3)	N/A	2026-06-23T18:11
stage-c-test-large-8-gpu-amd-rocm720 (linux-mi325-8gpu-sglang, 1)	N/A	2026-06-23T18:11
multimodal-gen-test-2-gpu-amd-rocm720 (linux-mi325-2gpu-sglang, 2)	N/A	2026-06-23T18:11
stage-c-test-large-8-gpu-amd-mi35x-rocm720 (linux-mi35x-gpu-8, 0)	N/A	2026-06-23T18:11
stage-b-test-2-gpu-large-amd-rocm720 (linux-mi325-2gpu-sglang, 0)	N/A	2026-06-23T18:11
multimodal-gen-test-2-gpu-amd-rocm720 (linux-mi325-2gpu-sglang, 1)	N/A	2026-06-23T18:11
multimodal-gen-test-1-gpu-amd-rocm720 (linux-mi325-1gpu-sglang, 3)	N/A	2026-06-23T18:11
stage-b-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang, 12)	N/A	2026-06-23T18:11
stage-b-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang, 1)	N/A	2026-06-23T18:11
stage-b-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang, 6)	N/A	2026-06-23T18:11
stage-b-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang, 3)	N/A	2026-06-23T18:11
stage-b-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang, 5)	N/A	2026-06-23T18:11
stage-b-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang, 2)	N/A	2026-06-23T18:11
multimodal-gen-test-1-gpu-amd-rocm720 (linux-mi325-1gpu-sglang, 1)	N/A	2026-06-23T18:11
stage-b-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang, 8)	N/A	2026-06-23T18:11
stage-b-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang, 7)	N/A	2026-06-23T18:11
stage-b-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang, 9)	N/A	2026-06-23T18:11
stage-b-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang, 4)	N/A	2026-06-23T18:11
stage-b-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang, 11)	N/A	2026-06-23T18:11
stage-b-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang, 10)	N/A	2026-06-23T18:11
stage-b-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang, 13)	N/A	2026-06-23T18:11
stage-b-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang, 0)	N/A	2026-06-23T18:11
stage-c-test-4-gpu-amd-rocm720	N/A	2026-06-23T18:40
dsv4-pro-fp4-amd-rocm720	Accuracy Test MI35x ROCm 7.2 (8-GPU DeepSeek-V4-Pro FP4)	2026-06-23T12:22
call-pr-test-amd-extra-rocm720 / extra-a-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang)	N/A	2026-06-23T00:39
call-pr-test-amd-extra-rocm720 / extra-a-test-1-gpu-large-amd-rocm720 (linux-mi325-1gpu-sglang)	N/A	2026-06-23T00:39
dsv4-flash-fp4-fp8-amd-rocm720	N/A	2026-06-23T19:17
dsv4-pro-fp4-amd-rocm720	N/A	2026-06-23T19:17
dsv4-pro-fp4-amd-rocm720	Accuracy Test MI35x ROCm 7.2 (8-GPU DeepSeek-V4-Pro FP4)	2026-06-24T06:33
call-pr-test-amd-extra-rocm720 / extra-a-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang)	Install dependencies	2026-06-24T08:18
call-pr-test-amd-extra-rocm720 / extra-a-test-2-gpu-large-amd-rocm720 (linux-mi325-2gpu-sglang)	Run test	2026-06-24T17:56
call-pr-test-amd-extra-rocm720 / extra-a-test-1-gpu-large-amd-rocm720 (linux-mi325-1gpu-sglang)	N/A	2026-06-24T18:08
call-pr-test-amd-extra-rocm720 / extra-a-test-2-gpu-large-amd-rocm720 (linux-mi325-2gpu-sglang)	N/A	2026-06-24T18:08
call-pr-test-amd-extra-rocm720 / extra-a-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang)	N/A	2026-06-24T18:08
dsv4-pro-fp4-amd-rocm720	N/A	2026-06-24T18:08
dsv4-flash-fp4-fp8-amd-rocm720	N/A	2026-06-24T18:08
stage-b-test-2-gpu-large-amd-rocm720 (linux-mi325-2gpu-sglang, 1)	N/A	2026-06-24T18:08
stage-a-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang)	N/A	2026-06-24T18:08
stage-b-test-2-gpu-large-amd-rocm720 (linux-mi325-2gpu-sglang, 0)	N/A	2026-06-24T18:08
stage-b-test-1-gpu-large-amd-rocm720 (linux-mi325-1gpu-sglang, 1)	N/A	2026-06-24T18:08
stage-b-test-1-gpu-large-amd-rocm720 (linux-mi325-1gpu-sglang, 0)	N/A	2026-06-24T18:08
multimodal-gen-test-2-gpu-amd-rocm720 (linux-mi325-2gpu-sglang, 2)	N/A	2026-06-24T18:08
stage-b-test-large-8-gpu-mi35x-disaggregation-amd-rocm720 (linux-mi35x-gpu-8.fabric)	N/A	2026-06-24T18:08
multimodal-gen-test-2-gpu-amd-rocm720 (linux-mi325-2gpu-sglang, 1)	N/A	2026-06-24T18:08
multimodal-gen-test-2-gpu-amd-rocm720 (linux-mi325-2gpu-sglang, 0)	N/A	2026-06-24T18:08
stage-b-test-1-gpu-small-amd-nondeterministic-rocm720 (linux-mi325-1gpu-sglang)	N/A	2026-06-24T18:08
stage-c-test-large-8-gpu-amd-mi35x-rocm720 (linux-mi35x-gpu-8, 0)	N/A	2026-06-24T18:08
multimodal-gen-test-1-gpu-amd-rocm720 (linux-mi325-1gpu-sglang, 1)	N/A	2026-06-24T18:08
stage-c-test-large-8-gpu-amd-rocm720 (linux-mi325-8gpu-sglang, 2)	N/A	2026-06-24T18:08
multimodal-gen-test-1-gpu-amd-rocm720 (linux-mi325-1gpu-sglang, 0)	N/A	2026-06-24T18:08
stage-c-test-large-8-gpu-amd-mi35x-rocm720 (linux-mi35x-gpu-8, 1)	N/A	2026-06-24T18:08
stage-c-test-large-8-gpu-amd-rocm720 (linux-mi325-8gpu-sglang, 1)	N/A	2026-06-24T18:08
stage-c-test-large-8-gpu-amd-rocm720 (linux-mi325-8gpu-sglang, 0)	N/A	2026-06-24T18:08
stage-c-test-large-8-gpu-amd-rocm720 (linux-mi325-8gpu-sglang, 3)	N/A	2026-06-24T18:08
multimodal-gen-test-1-gpu-amd-rocm720 (linux-mi325-1gpu-sglang, 3)	N/A	2026-06-24T18:08
stage-b-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang, 1)	N/A	2026-06-24T18:08
stage-b-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang, 0)	N/A	2026-06-24T18:08
stage-b-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang, 4)	N/A	2026-06-24T18:08
stage-b-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang, 6)	N/A	2026-06-24T18:08
stage-b-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang, 5)	N/A	2026-06-24T18:08
stage-b-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang, 13)	N/A	2026-06-24T18:08
stage-b-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang, 7)	N/A	2026-06-24T18:08
stage-b-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang, 9)	N/A	2026-06-24T18:08
multimodal-gen-test-1-gpu-amd-rocm720 (linux-mi325-1gpu-sglang, 2)	N/A	2026-06-24T18:08
stage-b-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang, 3)	N/A	2026-06-24T18:08
stage-b-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang, 8)	N/A	2026-06-24T18:08
stage-b-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang, 2)	N/A	2026-06-24T18:08
stage-b-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang, 10)	N/A	2026-06-24T18:08
stage-b-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang, 12)	N/A	2026-06-24T18:08
stage-b-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang, 11)	N/A	2026-06-24T18:08
stage-c-test-4-gpu-amd-rocm720	N/A	2026-06-24T18:35

Per-Job Analysis

call-pr-test-amd-extra-rocm720 / extra-a-test-1-gpu-large-amd-rocm720 (linux-mi325-1gpu-sglang) — failed step(s): N/A

Commit Info

• sglang: ed26a109eeed967e3a43e8f6bef5b6ec30484cf0
• aiter: N/A — job log blob no longer exists (BlobNotFound from the logs API), and no test step ever ran, so no [CI-AITER-CHECK] markers are available.

Failed Tests

Test File	Test Function	Error	Log
—	—	No test ran. Job conclusion = cancelled (not failure).	Job

This was not a test failure. The job extra-a-test-1-gpu-large-amd-rocm720 ran from 18:10:50Z and was cancelled at 18:40:52Z. The job has an empty steps array and no retrievable log, consistent with a runner-level cancellation rather than a test assertion or crash.

Failure Summary

The job did not fail a test — it was cancelled along with 39 other jobs in run 28046873327 (40 cancelled, 7 success, only the two *-finish aggregation gates marked failure because their upstreams were cancelled). The whole run's conclusion is cancelled. A newer scheduled run of the same workflow on main superseded it via GitHub Actions concurrency control.

Regression Status

Infrastructure issue (concurrency cancellation) — not a code regression.

Recent scheduled runs of pr-test-amd-rocm720.yml on main (event=schedule, completed runs only):

Date (UTC)	Run	Conclusion	Head SHA
06-23 18:40	28048657152	queued/in-flight [IN-FLIGHT]	e0dc8b7137
06-23 18:10	28046873327 (this run)	❌ cancelled	ed26a109
06-23 12:55	28027779773	✅ success	0460f277b7
06-23 07:13	28008999561	❌ cancelled	c4376aaa88
06-23 00:39	27993948299	❌ cancelled	ba9d5aed98
06-22 19:05	27977093213	❌ cancelled	adc203dfee
06-22 18:45	27975943807	❌ cancelled	43b1fb95f6

Scheduled runs of this workflow are cancelled very frequently (6 of the last 7 completed scheduled runs), indicating a recurring concurrency-cancellation pattern, not a per-commit regression.

Facts (verified observations)

• Job 83027236328 conclusion = "cancelled", status = "completed", empty steps array; started 2026-06-23T18:10:50Z, ended 2026-06-23T18:40:52Z. [FACT]
• Run 28046873327 conclusion = "cancelled"; of 49 jobs: 40 cancelled, 7 success, 2 *-finish gates failure. [FACT]
• Newer scheduled run 28048657152 (event = schedule, head_sha e0dc8b7137) created 2026-06-23T18:40:50Z — 2 s before this job's cancellation timestamp. [FACT]
• Both runs are event = schedule on branch = main → identical concurrency group pr-test-amd-rocm720-refs/heads/main. [FACT]
• pr-test-amd-rocm720.yml:123 sets cancel-in-progress: ${{ !inputs.run_all_tests && github.event_name != 'workflow_call' && github.event_name != 'workflow_dispatch' }}, which evaluates to true for schedule events. [FACT]
• Job log is unavailable: logs API returns BlobNotFound. [FACT]

Hypothesised Causes

• [HIGH] The job was cancelled by GitHub Actions concurrency control, because a newer scheduled run on main entered the same concurrency group with cancel-in-progress: true.
  • Supporting: superseding run 28048657152 created at 18:40:50Z (2 s before cancellation at 18:40:52Z); both runs share group pr-test-amd-rocm720-refs/heads/main; the entire run (40/49 jobs) was cancelled simultaneously — a hallmark of run-level concurrency cancellation, not per-job failure; workflow file confirms cancel-in-progress is true for schedule events (line 123).
  • Disconfirming: (none found) — no test step executed, no error log exists, and the timing/group alignment is exact.
  • How to verify: confirm in the cancelled run's UI that the cancellation reason reads "canceled because a new run was started" / superseded by run 28048657152.

No code-level (sglang or aiter) hypothesis applies — this is not a test or build failure.

In-flight Fix Check

This is not a code defect, so a code-fix PR is not the right lever. The relevant question is whether scheduled-run concurrency cancellation is being addressed.

• In-flight fix: ❌ none found — searched open PRs:
  • curl -s -H "Authorization: token $GH_PAT" "https://api.github.com/repos/sgl-project/sglang/search/issues?q=repo:sgl-project/sglang+is:pr+is:open+concurrency+cancel" → no matching open PR touching the rocm720 schedule concurrency group.

Suggested Triage Steps

• Treat this job result as not actionable for code — it is a superseded/cancelled scheduled run, not a test failure. Re-run analysis against a completed (non-cancelled) scheduled run such as 28027779773 to get real signal for this job.
• If frequent cancellation of scheduled runs is undesirable (6/7 recent scheduled runs cancelled), consider whether the schedule concurrency policy should keep cancel-in-progress: true. Triage option: scope the schedule concurrency group per-run (or set cancel-in-progress: false for schedule) so a later cron does not abort a still-running scheduled run — discuss with workflow owners before changing, since it increases CI resource usage.
• Verify whether the upstream cron cadence (0 */6 * * * plus the daily 30 17 * * *) is overlapping with run duration, causing the next cron to cancel the prior run.

Status (factual)

• Persistence: 6 of the last 7 completed scheduled runs of this workflow are cancelled; recurring concurrency pattern, not a new event.
• Scope: run-wide — 40 of 49 jobs in run 28046873327 cancelled, not isolated to this job.
• Blocked work: scheduled ROCm 7.2 AMD CI signal is lost for head_sha ed26a109; no test verdict was produced.
• In-flight fix: ❌ none found in open PRs.

dsv4-flash-fp4-fp8-amd-rocm720 — failed step(s): N/A

Commit Info

• sglang: ed26a109
• aiter: N/A — the job log blob does not exist (BlobNotFound), so no [CI-AITER-CHECK] markers are retrievable. Not a scout run.

Failed Tests

Test File	Test Function	Error	Log
N/A	N/A	Job cancelled mid-run; no test ever reported pass/fail	Job

This was not a test failure. The job dsv4-flash-fp4-fp8-amd-rocm720 was cancelled at 2026-06-23T18:40:51Z after running for ~30 min (started 18:10:37Z). Its log blob no longer exists (<Error><Code>BlobNotFound</Code>), and the job recorded zero steps.

Failure Summary

The run 28046873327 (schedule event, sha ed26a109) had 40 of 49 jobs cancelled simultaneously at 18:40:51Z. The target job was one of them. The two failure conclusions in the run (pr-test-amd-rocm720-finish, call-pr-test-amd-extra-rocm720 / pr-test-amd-extra-finish) are aggregator gate jobs that fail when their needed jobs are cancelled — they are a consequence, not a cause.

Regression Status

Infrastructure issue (concurrency cancellation) — not a code regression.

Recent pr-test-amd-rocm720.yml scheduled runs on main (completed only):

Date (UTC)	Run	Conclusion	Note
06-23 18:40	28048657152	queued/in-progress	The run that cancelled ours (created 18:40:50Z)
06-23 18:10	28046873327	❌ cancelled	This run
06-23 12:55	28027779773	✅ success	—
06-23 07:13	28008999561	cancelled	concurrency
06-23 00:39	27993948299	cancelled	concurrency
06-22 19:05	27977093213	cancelled	concurrency

Nearly every scheduled run is cancelled — a chronic pattern of overlapping scheduled runs cancelling one another via the shared concurrency group.

Failure Cluster

Concurrency cancellation of overlapping scheduled run (not a test failure)

Facts (verified observations, no interpretation)

• [FACT] Job conclusion is cancelled, not failure (GitHub API job object).
• [FACT] Job log blob returns <Error><Code>BlobNotFound</Code>; job has 0 recorded steps. No test output exists.
• [FACT] 40/49 jobs in the run were cancelled at the identical timestamp 2026-06-23T18:40:51Z; 7 succeeded, 2 (finish aggregators) failed.
• [FACT] A newer scheduled run 28048657152 was created at 2026-06-23T18:40:50Z — one second before the mass cancellation.
• [FACT] Both runs are schedule events on refs/heads/main, so they share concurrency group pr-test-amd-rocm720-refs/heads/main (workflow line 122).
• [FACT] cancel-in-progress evaluates to true for schedule events (workflow line 123).
• [FACT] All test matrices set fail-fast: false, so this is not a matrix fast-fail cascade.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [HIGH] The run was cancelled by GitHub Actions' cancel-in-progress concurrency policy when the next scheduled run (28048657152, sha e0dc8b71) started.
  • Supporting: newer run created 18:40:50Z ≡ cancellation 18:40:51Z (1 s gap); both share the same concurrency group + cancel-in-progress: true; cron schedules 30 17 * * * and 0 */6 * * * plus GitHub's scheduled-run delays cause overlapping firings on the same ref.
  • Disconfirming: a few jobs (jit-kernel-unit-test-amd-rocm720, sgl-kernel jobs) kept running until ~19:17 rather than dying at 18:40:51 — likely runner-shutdown lag, not contradicting the concurrency cancel.
  • How to verify: inspect run 28048657152 — confirm it is a schedule run on main that triggered the cancel of 28046873327.
• [SPECULATION] Genuine code/test problem in dsv4-flash-fp4-fp8-amd-rocm720.
  • Supporting: none.
  • Disconfirming: no log, no steps, conclusion is cancelled not failure; the test never executed to completion.

In-flight Fix Check

• In-flight fix: ❌ none found — this is a CI-orchestration artifact, not a defect with an open PR. Searched is not applicable (no failing test name / library to key on).

Suggested Triage Steps

• Re-check the target job in the superseding run 28048657152 (or the next completed scheduled run) — that is where real signal for ed26a109's dsv4-flash-fp4-fp8 test will appear.
• If a definitive accuracy result for this commit is needed, trigger a workflow_dispatch run (gets a unique concurrency group and won't be cancelled by the next schedule).
• Structural follow-up for maintainers: the two cron schedules (30 17 * * *, 0 */6 * * *) plus GitHub's scheduled-run delays repeatedly produce overlapping runs on main that cancel each other (nearly every recent scheduled run is cancelled). Consider separating their concurrency groups or staggering crons so the 6-hourly dsv4 run isn't killed by the daily run.

Status (factual, no priority assignment)

• Persistence: Not a persistent test failure. The cancellation pattern itself recurs — most scheduled pr-test-amd-rocm720.yml runs over the last ~3 days are cancelled by concurrency.
• Scope: 40/49 jobs in run 28046873327 cancelled together; no test signal produced for any of them.
• Blocked work: dsv4 fp4/fp8 accuracy signal for this commit is missing until a non-cancelled scheduled or dispatched run completes.
• In-flight fix: ❌ none found — orchestration issue, no code fix applicable.

dsv4-pro-fp4-amd-rocm720 — failed step(s): N/A

Commit Info

• sglang: ed26a109 — "[Fix] Return streaming logprobs when reasoning/tool parser is active (#28601)"
• aiter: N/A — the job never executed any steps (no runner allocated), so no [CI-AITER-CHECK] markers exist; the log blob returns BlobNotFound.

Failed Tests

Test File	Test Function	Error	Log
—	—	No test executed — job was cancelled while queued (0 steps ran)	Job

This was not a test failure. The job dsv4-pro-fp4-amd-rocm720 concluded as cancelled, not failure. It recorded zero steps — it sat queued waiting for a linux-mi35x-gpu-8 (8‑GPU MI35x) runner from 18:10:37Z and was cancelled at 18:40:51Z before any checkout/container/test step began. The log blob does not exist (BlobNotFound) precisely because nothing ran.

Failure Summary

The entire run 28046873327 (PR Test ROCm 7.2 (AMD), daily schedule) was cancelled, taking 40 of 49 jobs (including this one) to cancelled. The two failure jobs in the run are the aggregator *-finish jobs, which fail by design when upstream jobs are cancelled. The target dsv4 job never acquired its scarce 8‑GPU runner before the cancellation hit.

Regression Status

Infrastructure issue (run cancellation) — not a code regression. The dsv4 test passes when it actually gets to run.

Recent history of dsv4-pro-fp4-amd-rocm720 (same branch main, event=schedule, completed runs only):

Date (UTC)	Run	dsv4 Conclusion	Steps Ran	Note
Jun 23 16:47	28027779773	✅ success	8	Ran to completion
Jun 23 18:10	28046873327	⚪ cancelled	0	this run — never got runner
Jun 23 12:22	28008999561	⚪ cancelled	8	Cancelled mid-run
Jun 23 05:52	27993948299	✅ success	8	Ran to completion
Jun 22 21:52	27977093213	⚪ cancelled	0	Never got runner

No failure conclusion appears for this job in the queryable window — only success and cancelled. When it runs, it passes.

Failure Cluster

Scheduled run cancelled by concurrency cancel-in-progress while dsv4 job queued for scarce 8-GPU MI35x runner (no test executed).

Facts (verified observations, no interpretation)

• [FACT] Job conclusion is cancelled, run conclusion is cancelled (GitHub API job/run objects). Not a failure.
• [FACT] Target job recorded 0 steps; started_at=2026-06-23T18:10:37Z, completed_at=2026-06-23T18:40:51Z (~30 min queued, then cancelled).
• [FACT] The next scheduled run 28048657152 was created at 2026-06-23T18:40:50Z — one second before this job's cancellation at 18:40:51Z.
• [FACT] Workflow concurrency (pr-test-amd-rocm720.yml:122-123) keys schedule runs on group pr-test-amd-rocm720-refs/heads/main with cancel-in-progress: true for non-workflow_call/non-workflow_dispatch events. Any newer schedule run on main cancels the in-progress one.
• [FACT] dsv4 job requires runs-on: linux-mi35x-gpu-8 (pr-test-amd-rocm720.yml:1197); the sibling 1‑GPU stage-b-test-1-gpu-small-amd-mi35x-rocm720 did get a runner and succeeded at 18:30:58Z, so MI35x capacity existed but the 8‑GPU slot did not free up in time.
• [FACT] The only two failure jobs in the run are pr-test-amd-rocm720-finish ([83040560486]) and call-pr-test-amd-extra-rocm720 / pr-test-amd-extra-finish ([83033310987]) — aggregator gates that fail because upstream jobs were cancelled.
• [FACT] Log blob returns BlobNotFound (no log was produced).

Hypothesised Causes (with confidence, with disconfirming evidence)

• [HIGH] The run (and this job) was terminated by concurrency cancel-in-progress when the next scheduled run entered the shared refs/heads/main group.
  • Supporting: new run 28048657152 created 18:40:50Z, this job cancelled 18:40:51Z; concurrency config at pr-test-amd-rocm720.yml:122-123 sets cancel-in-progress: true for schedule events on group refs/heads/main.
  • Disconfirming: a few sibling jobs (e.g. jit-kernel-unit-test-amd-rocm720) completed success as late as 19:16:57Z, i.e. after the cancellation timestamp — so the cancellation cascade did not stop every job instantly. This does not change the conclusion for dsv4 (it had 0 steps and was killed at the cancel instant) but indicates the kill was not perfectly synchronous across the matrix.
  • How to verify: inspect run 28048657152's concurrency group; confirm it shares pr-test-amd-rocm720-refs/heads/main with this run.
• [MEDIUM] Scarce 8‑GPU MI35x runner availability is the reason the job was still queued (0 steps) and thus vulnerable to the cancel window.
  • Supporting: 30 min queued with no steps; runs that do get the 8‑GPU runner promptly (e.g. 27993948299, 28027779773) complete and pass.
  • Disconfirming: cannot directly observe runner-queue depth from the API; the 1‑GPU MI35x sibling did get scheduled, so the shortage is specific to the 8‑GPU pool (inferred, not logged).
  • How to verify: check MI35x 8‑GPU runner queue/availability dashboards for 18:10–18:40 UTC on 2026-06-23.

In-flight Fix Check

• In-flight fix: ❌ none found — not applicable. This is a scheduling/cancellation event, not a defect to fix. No code change is implicated.
• (Searched the failing commit's PR #28601 — it modifies streaming logprobs / reasoning-tool-parser code, entirely unrelated to dsv4 accuracy or runner scheduling.)

Suggested Triage Steps

• Treat this as a non-actionable cancellation, not a regression — no bisect needed. dsv4 passed at 27993948299 and 28027779773 on nearby SHAs.
• If repeated dsv4 cancellations are degrading signal, the overlap is structural: the daily 30 17 * * * and 6‑hourly 0 */6 * * * crons share the refs/heads/main concurrency group with cancel-in-progress: true, so closely-spaced runs cancel each other. Consider whether the dsv4 8‑GPU job should run under a dedicated concurrency group (it already gets a unique group under workflow_call/workflow_dispatch, but not under direct schedule).
• To get an on-demand green signal for this exact SHA, re-run via workflow_dispatch selecting stage dsv4-pro-fp4-amd-rocm720 (manual dispatch gets a unique concurrency group and won't be cancelled by the crons).
• If 8‑GPU MI35x queue waits routinely exceed the inter-cron interval, raise runner-capacity provisioning for the linux-mi35x-gpu-8 pool.

Status (factual, no priority assignment)

• Persistence: dsv4 has no failure conclusion in the queryable schedule window; it alternates success (when it gets the runner) and cancelled (when superseded). This specific run was cancelled with 0 steps.
• Scope: Run-wide cancellation — 40/49 jobs in run 28046873327 cancelled; the 2 failure jobs are aggregator *-finish gates reacting to the cancellation, not independent test failures.
• Blocked work: No dsv4 FP4 accuracy signal produced for SHA ed26a109 from this run; the same SHA can be validated via manual dispatch if a green signal is required.
• In-flight fix: ❌ none — not a code defect.

call-pr-test-amd-extra-rocm720 / extra-a-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang) — failed step(s): N/A

Commit Info

• sglang: ed26a109
• aiter: N/A — the job log blob no longer exists (GitHub returned BlobNotFound), and the job recorded zero steps, so no [CI-AITER-CHECK] marker is available.

Failed Tests

Test File	Test Function	Error	Log
N/A	N/A	Job cancelled, not failed — no test executed to completion	Job

This is not a test failure. The job conclusion is cancelled (not failure), it has no recorded steps, and its log was never persisted.

Failure Summary

Job extra-a-test-1-gpu-small-amd-rocm720 was cancelled ~30 min after starting (started 18:10:50Z, cancelled 18:40:52Z). All 40 in-progress jobs in run 28046873327 were cancelled within the same one-second window (18:40:51–18:40:52Z). The run's overall conclusion is cancelled. No test ran to a pass/fail verdict.

Regression Status

Infrastructure issue (orchestration cancellation) — not a code regression. The run was superseded by a newer scheduled run via the workflow's cancel-in-progress concurrency policy.

Failure Cluster

Workflow-level concurrency cancellation (superseded scheduled run)

Facts (verified observations, no interpretation)

• Job conclusion is cancelled, not failure (verified via /actions/jobs/83027236333). Steps array is empty. [FACT]
• Run 28046873327 overall conclusion is cancelled (event schedule, branch main, sha ed26a109). [FACT]
• 40 of 49 jobs cancelled simultaneously at 18:40:51–18:40:52Z; 7 succeeded, 2 failure (the pr-test-amd-extra-finish and pr-test-amd-rocm720-finish aggregator jobs, which fail by design when upstream jobs are cancelled). [FACT]
• A newer scheduled run 28048657152 (sha e0dc8b71) was created at 18:40:50Z — i.e. 1–2 seconds before every job in our run was cancelled. [FACT]
• Workflow concurrency policy (pr-test-amd-rocm720.yml:117-123): for a schedule event the group resolves to pr-test-amd-rocm720-refs/heads/main and cancel-in-progress evaluates to true (none of run_all_tests / workflow_call / workflow_dispatch apply). [FACT]
• The job log blob returns BlobNotFound (expired/never persisted for a cancelled-during-setup job). [FACT]

Hypothesised Causes (with confidence, with disconfirming evidence)

• [HIGH] The new scheduled run 28048657152 entered the shared concurrency group pr-test-amd-rocm720-refs/heads/main and, because cancel-in-progress: true for schedule events, cancelled the still-running run 28046873327.
  • Supporting: New run created at 18:40:50Z; all 40 in-progress jobs cancelled at 18:40:51–52Z (≤2 s later); both runs are schedule/main → identical group; cancel-in-progress is true for this path (pr-test-amd-rocm720.yml:123).
  • Disconfirming: (none found) — timing, group identity, and the policy expression all converge.
  • How to verify: Check run 28048657152's timeline/annotations for a "superseded the previous run" entry, or the original run's annotation Canceling since a higher priority waiting request … exists.

In-flight Fix Check

Not applicable — there is no test/code failure to fix. The cancellation is expected concurrency behavior. No PR search performed (no failing test name / error to key on).

Suggested Triage Steps

• No code action needed. Treat this job as cancelled-by-supersede, not a failure. Exclude it from failure trend/regression accounting (it never produced a pass/fail).
• If full signal is needed for sha ed26a109, inspect the superseding run 28048657152 (sha e0dc8b71) for the actual test results on current main.
• If these back-to-back schedule cancellations are undesirable (note the run at 18:10 was itself cancelled 30 min in by an 18:40 run — two cron schedules 30 17 * * * and 0 */6 * * * can crowd each other), consider whether the every-6h cron overlap with the in-progress daily run is intended.

Status (factual, no priority assignment)

• Persistence: Single occurrence; run-level cancellation, not a recurring test failure. Note the prior scheduled run 28008999561 and several earlier ones also show cancelled overall conclusions — consistent with routine concurrency supersede on the main schedule, not a persistent test break.
• Scope: All 40 in-progress jobs in run 28046873327; no test-level blast radius.
• In-flight fix: N/A — not a failure requiring a fix.

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-06-24 19:29 UTC)

[amd-bot]

pr-test-amd-rocm720.yml — Per-Job Analysis (continued 2/12)

call-pr-test-amd-extra-rocm720 / extra-a-test-2-gpu-large-amd-rocm720 (linux-mi325-2gpu-sglang) — failed step(s): Start CI container

Commit Info

• sglang: ed26a109
• aiter: N/A (job was cancelled during docker image pull, before amd_ci_install_dependency.sh ran — no [CI-AITER-CHECK] markers in log)

Failed Tests

Test File	Test Function	Error	Log
N/A	N/A	Job cancelled during docker image pull — no test executed	Log

This is not a test failure. The job was cancelled at 18:41:04Z while still pulling the rocm/sgl-dev:v0.5.13.post1-rocm720-mi30x-20260623 docker image (last layer downloaded at 18:40:57, cancellation at 18:41:04). Checkout completed, but dependency installation and tests never started.

Failure Summary

The job extra-a-test-2-gpu-large-amd-rocm720 never ran any test. It was cancelled mid docker-image-pull because a newer scheduled run of the same workflow superseded this one via the GitHub Actions concurrency group. No sglang or aiter code is implicated.

Regression Status

Infrastructure issue (concurrency-group cancellation) — not a code regression, not a flaky test.

Evidence (same branch main, same event schedule, completed runs only):

Date	Run	Status	Note
Jun 23 18:40	28048657152	queued (superseding)	Created 18:40:50Z — triggered the cancellation
Jun 23 18:10	28046873327	cancelled	This run — all 40 test jobs cancelled at 18:40:51Z
Jun 23 12:55	28027779773	✅ success	—
Jun 23 07:13	28008999561	cancelled	recurring pattern
Jun 23 00:39	27993948299	cancelled	recurring pattern

Failure Cluster

Concurrency-group cancellation of in-progress scheduled run (superseded by newer schedule run)

Facts (verified observations, no interpretation)

• [FACT] Run 28046873327 overall conclusion is cancelled; of 49 jobs: 7 success, 40 cancelled, 2 failure (the two are aggregator *-finish gate jobs that fail by design when upstream is cancelled).
• [FACT] All 40 cancelled jobs completed simultaneously at ~18:40:51Z (verified from jobs API completed_at), the signature of a single run-level cancellation event, not independent per-job failures.
• [FACT] A newer scheduled run 28048657152 was created at 18:40:50Z — one second before the mass cancellation — on a later head SHA e0dc8b71.
• [FACT] This job's log ends with ##[error]The operation was canceled. at line 1407 (log) immediately after the final docker layer 4631d8141576: Download complete at 18:40:57.
• [FACT] No dependency install ran (0 [CI-AITER-CHECK] markers) and no tests ran (0 pytest/collected markers) in the job log.
• [FACT] The scheduled-run history for pr-test-amd-rocm720.yml shows repeated cancelled conclusions (07:13, 00:39, and earlier), indicating runs frequently overlap and supersede each other.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [HIGH] The in-progress run was cancelled by the GitHub Actions concurrency group when the next scheduled run started ~30 min later, while this run's long-running 2-GPU jobs were still pulling images.
  • Supporting: newer run 28048657152 created at 18:40:50Z; all jobs in this run cancelled at 18:40:51Z; mass-simultaneous cancellation across 40 jobs; log shows generic The operation was canceled not a code error.
  • Disconfirming: (none found) — no test or build error appears anywhere in the log; the job hadn't reached any sglang/aiter code.
  • How to verify: inspect the workflow's concurrency: block in pr-test-amd-rocm720.yml and confirm cancel-in-progress: true keyed on a group shared by scheduled runs.

In-flight Fix Check

• In-flight fix: ❌ none found — this is a CI-orchestration timing artifact, not a code bug. Searched open PRs for scheduling/concurrency keywords:

  curl -s -H "Authorization: token $GH_PAT" "https://api.github.com/search/issues?q=repo:sgl-project/sglang+is:pr+is:open+pr-test-amd-rocm720+concurrency"
  

Suggested Triage Steps

• Treat this job as no signal, not a failure — the changed code was never exercised. Re-run is unnecessary unless fresh signal for ed26a109 is desired.
• If recurring cancellations harm CI signal: review the concurrency group in pr-test-amd-rocm720.yml. If scheduled runs share a cancel-in-progress: true group, consider keying scheduled runs by run_id (so each schedule completes) or widening the cron interval so a run finishes before the next is queued.
• Confirm a later scheduled run for a nearby SHA completes (e.g. the 12:55 run 28027779773 succeeded) to ensure the underlying test job itself is healthy.

Status (factual, no priority assignment)

• Persistence: Recurring orchestration pattern — multiple scheduled runs of pr-test-amd-rocm720.yml on Jun 23 ended cancelled (07:13, 00:39, this 18:10 run).
• Scope: Run-wide — 40 of 49 jobs in this run cancelled simultaneously; not specific to extra-a-test-2-gpu-large-amd-rocm720.
• Blocked work: No CI signal produced for this scheduled run; superseded by 28048657152.
• In-flight fix: ❌ none found — orchestration artifact, no code fix applicable.

stage-b-test-1-gpu-small-amd-nondeterministic-rocm720 (linux-mi325-1gpu-sglang) — failed step(s): N/A

Commit Info

• sglang: ed26a109
• aiter: N/A — the job log blob does not exist (returns BlobNotFound); the job was cancelled before any step ran, so no [CI-AITER-CHECK] markers were emitted.

Failed Tests

Test File	Test Function	Error	Log
N/A — no test executed	N/A	Job cancelled by GitHub Actions concurrency (cancel-in-progress), not a test failure	Job

Failure Summary

This job did not fail a test — its conclusion is cancelled. The job started at 18:11:00 UTC and was cancelled at 18:40:51 UTC with zero recorded steps (it never reached test execution). The entire parent run 28046873327 was cancelled (40 jobs cancelled, 7 success, only the 2 *-finish aggregator jobs failure — and those fail solely because their upstream needs were cancelled).

Regression Status

Infrastructure issue (concurrency cancellation) — not a code regression.

The cancellation timestamp (18:40:51 UTC) coincides exactly with the creation of the next scheduled run 28048657152 (event=schedule, branch=main, created 18:40:50 UTC). Both runs share the same concurrency group and the newer one preempted this one.

Recent pr-test-amd-rocm720.yml scheduled runs on main (completed only):

Created (UTC)	Run	Conclusion
06-23 18:40	28048657152	queued (superseding run)
06-23 18:10	28046873327	❌ cancelled (this run)
06-23 12:55	28027779773	✅ success
06-23 07:13	28008999561	cancelled
06-23 00:39	27993948299	cancelled

Most scheduled runs of this workflow on main are routinely cancelled by the same mechanism — only the 12:55 run completed successfully in the recent window.

Failure Cluster

Scheduled run cancelled by concurrency cancel-in-progress (newer schedule run preempted) — orchestration artifact, no test signal.

Facts (verified observations)

• Job conclusion is cancelled, not failure (GitHub API, job 83027275828).
• Job has 0 steps recorded and ran 18:11:00 → 18:40:51 UTC; the log blob is missing (BlobNotFound), consistent with a cancel before step execution.
• Parent run 28046873327 conclusion cancelled: 40 jobs cancelled, 7 success, 2 failure (both *-finish aggregators that need the cancelled jobs).
• Superseding run 28048657152 was created at 18:40:50 UTC (1 s before this job's cancellation), same event=schedule, same branch=main.
• .github/workflows/pr-test-amd-rocm720.yml:117-123: concurrency group pr-test-amd-rocm720-${{ ... || github.ref }} (→ …-refs/heads/main for schedule) with cancel-in-progress: ${{ !inputs.run_all_tests && github.event_name != 'workflow_call' && github.event_name != 'workflow_dispatch' }} → evaluates to true for schedule events. Two scheduled runs on main therefore share one group and the newer one cancels the older.

Hypothesised Causes

• [FACT] The job was cancelled because a newer scheduled run (28048657152) entered the shared concurrency group pr-test-amd-rocm720-refs/heads/main while this run was in progress, and cancel-in-progress=true for schedule events (workflow lines 117-123).
  • Supporting: cancellation timestamp (18:40:51) is 1 s after the superseding run's creation (18:40:50); both are schedule/main; job has no steps and no log.
  • Disconfirming: (none found) — no test ever executed, so there is no competing code-level explanation.
  • How to verify: already verified via API timestamps + concurrency config. No further action needed to confirm cancellation cause.
• Why two scheduled runs overlapped: this workflow has two crons (30 17 * * * daily and 0 */6 * * * every-6h). When a long daily run is still in progress as the next 6h-cron run launches, the 6h run preempts it. [MEDIUM] — the 18:00 6h-cron run plausibly preempted the still-running 17:30 daily run; exact cron-to-run mapping not confirmable from creation times alone.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for concurrency/cancel-in-progress changes to this workflow:
  • curl -s -H "Authorization: token $GH_PAT" "https://api.github.com/search/issues?q=repo:sgl-project/sglang+is:pr+is:open+pr-test-amd-rocm720+concurrency" → no matching open PR.
• No fix is strictly needed: this is expected concurrency behaviour, not a broken test.

Suggested Triage Steps

• Treat this job as no signal — it is a cancelled scheduled run, not a regression. Do not attribute any test failure to commit ed26a109.
• If a clean run is needed for this SHA, re-run the workflow via workflow_dispatch (which gets a unique concurrency group per github.run_id and is exempt from cancel-in-progress), so it cannot be preempted by the next cron.
• If frequent cancellation of scheduled runs is unwanted, consider whether the daily (30 17) and 6h (0 */6) crons should use distinct concurrency groups (e.g. include github.event.schedule in the group key) so they no longer cancel each other on main.

Status

• Persistence: Most recent scheduled runs on main are cancelled by this concurrency mechanism (only the 06-23 12:55 run completed in the recent window). Recurring orchestration artifact, not a sustained test failure.
• Scope: Whole-run cancellation — 40 jobs in run 28046873327, including this one. The 2 failure jobs are *-finish aggregators reflecting cancelled upstreams, not real failures.
• Blocked work: No test signal produced for SHA ed26a109 on this workflow for this run; coverage relies on the next non-preempted scheduled (or dispatched) run.
• In-flight fix: ❌ none found — not required; behaviour is by-design concurrency cancellation.

stage-b-test-1-gpu-large-amd-rocm720 (linux-mi325-1gpu-sglang, 0) — failed step(s): N/A

Commit Info

• sglang: ed26a109
• aiter: N/A — job log blob is unavailable (BlobNotFound from the logs API); logs are not uploaded for jobs killed by a concurrency cancel, so [CI-AITER-CHECK] markers cannot be read.

Failed Tests

Test File	Test Function	Error	Log
—	—	No test failure. Job stage-b-test-1-gpu-large-amd-rocm720 (linux-mi325-1gpu-sglang, 0) was cancelled (conclusion cancelled, zero steps recorded), not failed.	log unavailable (BlobNotFound)

Failure Summary

This is not a test failure — it is an infrastructure-level cancellation. The job was cancelled [FACT] (conclusion cancelled, no steps executed/recorded) at 18:40:51Z, together with all 40 other test jobs in run 28046873327, which itself has run-level conclusion cancelled. No test in this job reached a pass/fail verdict.

Regression Status

Infrastructure issue (concurrency cancellation) — not a code regression and not a flaky test.

Run history of pr-test-amd-rocm720.yml (event=schedule, branch=main, completed runs):

Created	Run	Conclusion	Note
Jun 23 18:40	28048657152	queued/in-flight	The superseding run — its creation cancelled our run
Jun 23 18:10	28046873327	cancelled	This run — superseded
Jun 23 12:55	28027779773	✅ success	—
Jun 23 07:13	28008999561	cancelled	—
Jun 23 00:39	27993948299	cancelled	—

(Note: several recent scheduled runs share the cancelled conclusion, consistent with overlapping schedule triggers repeatedly superseding each other under the shared refs/heads/main concurrency group.)

Failure Cluster

Concurrency cancellation — scheduled run superseded by a newer scheduled run (cancel-in-progress).

Facts (verified observations)

• [FACT] Job 83027875874 conclusion = cancelled, status = completed, zero steps recorded; started 18:11:00Z, ended 18:40:51Z.
• [FACT] Job-status distribution for run 28046873327: 7 success, 40 cancelled, 2 failure. The only 2 failure jobs are aggregators call-pr-test-amd-extra-rocm720 / pr-test-amd-extra-finish and pr-test-amd-rocm720-finish (they fail when upstream jobs are cancelled).
• [FACT] All 40 test jobs were cancelled at the same instant (18:40:51-18:40:52Z).
• [FACT] A newer scheduled run 28048657152 (event schedule, branch main) was created at 18:40:50Z — one second before the cancellation.
• [FACT] pr-test-amd-rocm720.yml:122-123 sets concurrency group: pr-test-amd-rocm720-…|| github.ref and cancel-in-progress: true for schedule events. Two scheduled runs on main share the same group, so the newer run cancels the older in-progress one.

Hypothesised Causes

• [FACT] Concurrency cancel-in-progress triggered by an overlapping scheduled run.
  • Supporting: newer schedule run created at the exact second of cancellation; concurrency block keyed on github.ref with cancel-in-progress: true for schedule events; all jobs cancelled simultaneously with no step output.
  • Disconfirming: (none found) — the timing and config are unambiguous.
  • How to verify: compare created_at of run 28048657152 (18:40:50Z) against the cancel timestamp of any job in 28046873327 (18:40:51Z); inspect pr-test-amd-rocm720.yml:117-123.

In-flight Fix Check

Not applicable — there is no test/code failure to fix. This is expected GitHub Actions concurrency behavior. No PR search warranted.

Suggested Triage Steps

• No code action needed. Treat this job as cancelled-by-superseding, not a failure; exclude it from failure trend/regression accounting.
• If these overlapping cancellations are noisy: the two crons (30 17 * * * and 0 */6 * * *) plus GitHub's schedule-delivery delay can produce closely-spaced scheduled runs on main that fall in the same concurrency group and cancel each other. A maintainer could decide whether that overlap is intended; this is a workflow-scheduling question, not a regression.
• To get real signal for this job, look at the most recent completed, non-cancelled scheduled run — 28027779773 (Jun 23 12:55, ✅ success).

Status

• Persistence: Single-run cancellation event; not a recurring code failure. Several recent scheduled runs share the cancelled conclusion due to the same overlapping-schedule concurrency behavior.
• Scope: Run-wide — all 40 test jobs in 28046873327 cancelled, not specific to this job.
• Blocked work: None — the superseding run 28048657152 provides the up-to-date signal.
• In-flight fix: N/A (no failure to fix).

stage-b-test-2-gpu-large-amd-rocm720 (linux-mi325-2gpu-sglang, 1) — failed step(s): N/A

Commit Info

• sglang: ed26a109
• aiter: N/A (job log blob already expired — BlobNotFound from the logs API — and no [CI-AITER-CHECK] markers are recoverable; aiter version cannot be determined)

Failed Tests

Test File	Test Function	Error	Log
N/A	N/A	Job cancelled mid-execution by workflow concurrency — no test failure	Job

No test failed. The job conclusion is cancelled, not failure. It ran 18:11:00Z → 18:40:51Z and was terminated while still executing.

Failure Summary

The job stage-b-test-2-gpu-large-amd-rocm720 (linux-mi325-2gpu-sglang, 1) did not fail a test — it was cancelled mid-run, along with 39 other jobs in run 28046873327 (40 cancelled, 7 success, 2 aggregator "finish" jobs marked failure as a downstream consequence). The cancellation was caused by the next scheduled run of the same workflow superseding this one under the workflow's cancel-in-progress concurrency policy.

Regression Status

Infrastructure issue (concurrency cancellation) — not a test regression. The job produced no test signal because it was killed by GitHub Actions concurrency before completing.

Run-level context for pr-test-amd-rocm720.yml (same branch main, event=schedule, completed runs only):

Created (UTC)	Run	Conclusion	Head SHA
2026-06-23 18:40:50	28048657152	queued/in-flight [IN-FLIGHT] (this run cancelled ours)	e0dc8b71
2026-06-23 18:10:06	28046873327	❌ cancelled (this job)	ed26a109
2026-06-23 12:55:42	28027779773	✅ success	0460f277
2026-06-23 07:13:14	28008999561	cancelled	c4376aaa
2026-06-23 00:39:01	27993948299	cancelled	ba9d5aed

Note the recurring cancelled conclusions across scheduled runs — consistent with overlapping cron triggers repeatedly cancelling each other (see Hypothesised Causes).

Failure Cluster

Scheduled run cancelled by concurrency cancel-in-progress (no test signal)

Facts (verified observations, no interpretation)

• Job 83027275896 conclusion = cancelled; started 2026-06-23T18:11:00Z, completed 2026-06-23T18:40:51Z (verified via jobs API).
• Run 28046873327 (event=schedule, branch main, attempt 1) conclusion = cancelled: 40 cancelled, 7 success, 2 failure. The only 2 failure jobs are aggregators — pr-test-amd-rocm720-finish (83040560486) and call-pr-test-amd-extra-rocm720 / pr-test-amd-extra-finish (83033310987) — both failed only at step "Check all dependent job statuses" (i.e. reporting the upstream cancellation).
• The next scheduled run 28048657152 was created at 2026-06-23T18:40:50Z — 1 second before our job's cancellation at 18:40:51Z.
• Workflow concurrency config (.github/workflows/pr-test-amd-rocm720.yml:122-123): group pr-test-amd-rocm720-${{ ... || github.ref }} (= ...-refs/heads/main for schedule events) with cancel-in-progress: ${{ !inputs.run_all_tests && github.event_name != 'workflow_call' && github.event_name != 'workflow_dispatch' }} → evaluates true for schedule events.
• Workflow has two crons (.github/workflows/pr-test-amd-rocm720.yml:8-9): 30 17 * * * (daily) and 0 */6 * * * (every 6h). Both fire on branch main and therefore share the same concurrency group.
• The job log blob is no longer retrievable (BlobNotFound from the logs API), so per-step/per-test data is unavailable. This does not affect the conclusion since the cancellation is established from job/run metadata.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [HIGH] The job was cancelled because a later scheduled run entered the same concurrency group and triggered cancel-in-progress.
  • Supporting: schedule events resolve the concurrency group to pr-test-amd-rocm720-refs/heads/main with cancel-in-progress=true (workflow:122-123); the next schedule run 28048657152 was created at 18:40:50Z and our job died at 18:40:51Z; the two crons (30 17, 0 */6) overlap on the same ref so back-to-back fires cancel each other; the same cancelled conclusion recurs across multiple prior scheduled runs.
  • Disconfirming: (none found) — no test/build error is present in the metadata, and the only failure jobs are downstream aggregators reporting the cancellation.
  • How to verify: inspect run 28048657152's concurrency group and confirm it cancelled 28046873327; confirm the 30 17 and 0 */6 crons both map to refs/heads/main.
• [LOW] A real test failure was masked by the cancellation.
  • Supporting: job ran ~30 min before being killed, so some tests may have executed.
  • Disconfirming: job conclusion is cancelled (not failure); the log blob is gone so there is no positive evidence of any test failure; the immediately prior completed scheduled run 28027779773 was a full success.
  • How to verify: check the equivalent stage-b-test-2-gpu-large-amd-rocm720 job in a non-cancelled scheduled run (e.g. the next completed run) for the same test set.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for concurrency/cancel-in-progress changes to this workflow:
  • curl -s -H "Authorization: token $GH_PAT" "https://api.github.com/search/issues?q=repo:sgl-project/sglang+is:pr+is:open+pr-test-amd-rocm720+concurrency"
  • This is an orchestration/scheduling artifact rather than a code defect; a "fix" (if desired) would be a workflow concurrency-group tweak, not a source change.

Suggested Triage Steps

• Treat this as non-actionable for code: the job carries no test signal. Re-check the same job in the next completed scheduled run for a real pass/fail verdict.
• If the recurring schedule-vs-schedule cancellation is undesirable, consider scoping the concurrency group to include github.run_id (or the cron expression) for schedule events so overlapping cron fires do not cancel each other — investigate in .github/workflows/pr-test-amd-rocm720.yml:122-123. (Investigation step, not a directive.)
• Confirm whether the 30 17 * * * and 0 */6 * * * crons are intended to overlap; the 18:00 fire of the 6h cron lands ~30 min after the (delayed) 17:30 daily fire, repeatedly killing the longer run.

Status (factual, no priority assignment)

• Persistence: This specific job shows cancelled, not a persistent test failure. Multiple recent scheduled runs of pr-test-amd-rocm720.yml share the cancelled conclusion (e.g. runs at 18:10, 07:13, 00:39 on 2026-06-23), consistent with repeated concurrency cancellation; the 12:55 run 28027779773 completed green.
• Scope: Entire run 28046873327 — 40 jobs cancelled in one shot; not isolated to this 2-GPU job.
• Blocked work: This scheduled run produced no AMD ROCm 7.2 test signal; signal is available from the adjacent completed run instead.
• In-flight fix: ❌ none found — not a code defect; would require a workflow concurrency adjustment if action is desired.

multimodal-gen-test-2-gpu-amd-rocm720 (linux-mi325-2gpu-sglang, 0) — failed step(s): N/A

Commit Info

• sglang: ed26a109
• aiter: N/A — the job never executed any steps, so no [CI-AITER-CHECK] markers were emitted and the log blob does not exist.

Failed Tests

Test File	Test Function	Error	Log
N/A (no test executed)	N/A	Job cancelled before any step ran	Job

No test ran. The job 83027275919 has zero executed steps — its log blob returns BlobNotFound. Its conclusion is cancelled, not failure.

Failure Summary

This is not a test failure. The entire workflow run 28046873327 (PR Test ROCm 7.2 (AMD), event=schedule) concluded cancelled: of 49 jobs, 40 were cancelled, 7 success, and 2 failure (both are downstream *-finish gate jobs that fail-by-design when upstream jobs are cancelled). The target job multimodal-gen-test-2-gpu-amd-rocm720 and its 3 matrix shards were cancelled while queued, before executing a single step.

Regression Status

Infrastructure issue (orchestration cancellation, not a code regression).

Status of multimodal-gen-test-2-gpu-amd-rocm720 across recent schedule runs on main (completed runs only):

Date (UTC)	Run	Job Status	Notes
Jun 23 18:10	28046873327	⏹️ Cancelled (3 shards, 0 steps)	whole run cancelled by concurrency
Jun 23 12:55	28027779773	⏭️ Skipped	change-detection guard; run succeeded
Jun 23 07:13	28008999561	⏭️ Skipped	run cancelled
Jun 23 00:39	27993948299	⏭️ Skipped	run cancelled
Jun 22 19:05	27977093213	⏭️ Skipped	run cancelled
Jun 22 13:52	27957777753	⏭️ Skipped	run cancelled

This job has never actually executed its multimodal-gen tests in the queryable schedule window — it is either skipped (change-detection guard) or cancelled. There is no green run with real test execution to regress from.

Failure Cluster

Run-level concurrency cancellation (no test executed)

Facts (verified observations, no interpretation)

• Job 83027275919 conclusion = cancelled; steps: [] (empty); log blob → BlobNotFound (verified via API).
• Run 28046873327 conclusion = cancelled; job breakdown: 40 cancelled, 7 success, 2 failure.
• The 2 failure jobs are call-pr-test-amd-extra-rocm720 / pr-test-amd-extra-finish (83033310987) and pr-test-amd-rocm720-finish (83040560486) — both terminal gate jobs that surface upstream cancellation as failure.
• Run created 2026-06-23T18:10:06Z, cancelled 2026-06-23T18:40:50Z. The next schedule run 28048657152 was created at 2026-06-23T18:40:50Z — the exact instant this run was cancelled.
• pr-test-amd-rocm720.yml:122-123: schedule runs share concurrency group pr-test-amd-rocm720-refs/heads/main with cancel-in-progress: true. A new schedule run on main therefore cancels any in-progress one.
• 10 of the 12 most recent schedule runs of this workflow concluded cancelled — the dominant outcome for this workflow.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [FACT] The job was cancelled by GitHub Actions concurrency cancel-in-progress, when the subsequent schedule run 28048657152 entered the shared group pr-test-amd-rocm720-refs/heads/main and superseded run 28046873327.
  • Supporting: concurrency config at .github/workflows/pr-test-amd-rocm720.yml:122-123; the superseding run's created_at equals this run's cancellation timestamp to the second; 10/12 recent schedule runs share the same cancelled fate.
  • Disconfirming: (none found) — the job has zero executed steps and no error log, fully consistent with a queue-time cancellation rather than a test/code fault.
  • How to verify: curl .../runs/28048657152 confirms it is the next schedule run on main (queued at 18:40:50Z); inspect concurrency block in the workflow file.

No code-level regression is implicated — the multimodal-gen test code never ran.

In-flight Fix Check

Not applicable — there is no test/code failure to fix. This is expected orchestration behavior (newer scheduled run preempts older). No fix PR is warranted; opening one would be a false positive.

Suggested Triage Steps

• Treat this as a non-failure. Cancelled runs caused by cancel-in-progress are expected and should be filtered out of failure tracking (only conclusion in {failure} with executed steps are real test failures).
• If full multimodal-gen coverage on every schedule tick is desired, consider whether the back-to-back schedule cadence (runs at 18:10 and 18:40) is intentional — the tight spacing guarantees the earlier run is killed before it finishes. Spacing the crons or relaxing cancel-in-progress for the daily full run would let it complete.
• To actually exercise this job's tests, trigger a workflow_dispatch/run_all_tests run (which gets a unique concurrency group and won't be preempted), or land a change that trips the change-detection guard so the matrix expands.

Status (factual, no priority assignment)

• Persistence: Cancellation is the dominant outcome — 10 of the last 12 schedule runs of pr-test-amd-rocm720.yml concluded cancelled.
• Scope: Run-wide, not job-specific — 40 of 49 jobs in run 28046873327 were cancelled by the same supersession event.
• Blocked work: No real signal lost from code's perspective; this job's multimodal-gen tests are not actually being exercised on schedule (skipped or cancelled every recent run).
• In-flight fix: N/A — not a code defect.

stage-b-test-large-8-gpu-mi35x-disaggregation-amd-rocm720 (linux-mi35x-gpu-8.fabric) — failed step(s): N/A

Commit Info

• sglang: ed26a109
• aiter: N/A — the target job never started any steps, so no [CI-AITER-CHECK] markers were emitted (no log produced; the log blob returns BlobNotFound).

Failed Tests

Test File	Test Function	Error	Log
N/A	N/A	No test failure — job conclusion is cancelled, not failure	Job (log blob expired/never written: BlobNotFound)

This is not a test failure. The job stage-b-test-large-8-gpu-mi35x-disaggregation-amd-rocm720 (linux-mi35x-gpu-8.fabric) was cancelled before it executed any steps (the GitHub API returns an empty steps array and the log download returns BlobNotFound).

Failure Summary

The entire run 28046873327 (PR Test ROCm 7.2 (AMD), schedule event, branch main) was cancelled. Of 49 jobs: 7 succeeded, 40 were cancelled simultaneously at 2026-06-23T18:40:51–52Z, and only the two *-finish gate jobs registered as failure (they fail by design when upstream jobs are cancelled). The target job never ran its test steps, so there is no [CI-AITER-CHECK] data, no pytest output, and no failing test function to report.

Regression Status

Infrastructure / workflow cancellation — not a code regression.

Recent history of job stage-b-test-large-8-gpu-mi35x-disaggregation-amd-rocm720 (pr-test-amd-rocm720.yml, branch main, event schedule, completed runs only):

Date	Run	Target Job Status	Notes
Jun 23 18:10	28046873327	⏹️ Cancelled	Superseded by newer scheduled run (this run)
Jun 23 12:55	28027779773	⏭️ Skipped	6-hour cron — dsv4-only, heavy mi35x jobs skipped
Jun 22 19:05	27977093213	⏭️ Skipped	—
Jun 22 13:52	27957777753	⏭️ Skipped	—
Jun 22 00:42	27922836455	⏭️ Skipped	—
Jun 21 18:31	27913598623	⏭️ Skipped	—

In the entire queryable window this job has never actually executed to completion — it was either skipped (6-hour dsv4-only cron) or cancelled. There is no green or red test baseline for it.

Failure Cluster

Workflow run cancelled by concurrency control (superseded scheduled run)

Facts (verified observations, no interpretation)

• Job conclusion = cancelled (not failure); steps array is empty; log download returns <Code>BlobNotFound</Code>. [FACT]
• Run-level conclusion = cancelled, event = schedule, branch = main, run_attempt = 1. [FACT]
• 40 of 49 jobs cancelled at the same instant 2026-06-23T18:40:51–52Z; the only two failure jobs are pr-test-amd-rocm720-finish and call-pr-test-amd-extra-rocm720 / pr-test-amd-extra-finish (aggregation gates). [FACT]
• A newer scheduled run 28048657152 (head_sha e0dc8b71) was created at 2026-06-23T18:40:50Z — 1 second before the mass cancellation. [FACT]
• Workflow concurrency block (.github/workflows/pr-test-amd-rocm720.yml:117-123): for schedule events the group resolves to pr-test-amd-rocm720-refs/heads/main with cancel-in-progress: true (true because not run_all_tests, not workflow_call, not workflow_dispatch). [FACT]
• Across recent scheduled runs, almost every run shows conclusion = cancelled, consistent with frequent crons (30 17 * * * and 0 */6 * * *) repeatedly superseding the prior in-progress run on the same main concurrency group. [FACT]

Hypothesised Causes (with confidence, with disconfirming evidence)

• [FACT] The job was cancelled by GitHub Actions concurrency control when the newer scheduled run 28048657152 entered the shared pr-test-amd-rocm720-refs/heads/main group with cancel-in-progress: true.
  • Supporting: newer run created 18:40:50Z, this run's 40 jobs cancelled 18:40:51Z (1 s later); concurrency config at .github/workflows/pr-test-amd-rocm720.yml:122-123 sets cancel-in-progress: true for schedule events; job has no steps and no log.
  • Disconfirming: (none found) — there is no test output, exit code, or stack trace indicating any code-level failure.
  • How to verify: open run 28048657152 and confirm it is the superseding schedule run on main; observe its head_sha differs (e0dc8b71 vs ed26a109).

In-flight Fix Check

Not applicable — there is no test/code failure to fix. No PR search performed because the cancellation is an expected consequence of the workflow's cancel-in-progress concurrency policy, not a defect.

Suggested Triage Steps

• No code action needed. Treat this job as cancelled/superseded, not failed. The failure status on the two *-finish gate jobs is a downstream artifact of the cancellation, not an independent failure.
• If full signal for the heavy mi35x disaggregation job is genuinely required, re-run only after confirming no newer scheduled run will supersede it, or trigger via workflow_dispatch (which gets a unique concurrency group per .github/workflows/pr-test-amd-rocm720.yml:122 and will not be auto-cancelled).
• If these schedule-vs-schedule cancellations are noisy in CI dashboards, consider whether the two crons (30 17 * * * and 0 */6 * * *) overlap near 18:00 UTC and self-cancel — a maintainer-level scheduling decision, out of scope for this PR.

Status (factual, no priority assignment)

• Persistence: Single cancelled occurrence in this run; the job has not executed to completion in the queryable window (only skipped or cancelled).
• Scope: Run-wide — 40 jobs in run 28046873327 cancelled together; this is not isolated to the target job.
• Blocked work: No real CI signal lost from a regression standpoint; the heavy mi35x disaggregation suite simply did not run this cycle.
• In-flight fix: N/A — not a defect.

stage-b-test-1-gpu-large-amd-rocm720 (linux-mi325-1gpu-sglang, 1) — failed step(s): N/A

Commit Info

• sglang: ed26a109 (#28601)
• aiter: N/A — job log blob does not exist (BlobNotFound), so no [CI-AITER-CHECK] markers are available.

Failed Tests

Test File	Test Function	Error	Log
N/A	N/A	Job cancelled (concurrency supersession), not a test failure	Job

The job ran ~30 min then was terminated. No test executed to completion; no log artifact was uploaded (BlobNotFound).

Failure Summary

This job did not fail a test — its conclusion is cancelled. The entire run 28046873327 was cancelled (40/49 jobs cancelled, 7 success, 2 failure — and the 2 failures are only the *-finish aggregator jobs that fail-fast when upstream jobs are cancelled). The job started 18:11 UTC and was cancelled at 18:40:51 UTC.

Regression Status

Infrastructure / orchestration cancellation — not a code regression.

The cancellation timestamp (18:40:51 UTC) coincides exactly with the queuing of the next scheduled run 28048657152 at 18:40:50 UTC. The workflow's concurrency config (.github/workflows/pr-test-amd-rocm720.yml:122-123) groups scheduled runs by github.ref (= refs/heads/main) with cancel-in-progress: true for schedule events, so the newer run superseded this one in the shared group pr-test-amd-rocm720-refs/heads/main.

Recent pr-test-amd-rocm720.yml scheduled runs on main (completed only):

Date (UTC)	Run	Conclusion	Note
06-23 18:40	28048657152	queued/in-flight	superseding run [IN-FLIGHT]
06-23 18:10	28046873327	❌ cancelled	this run — superseded
06-23 12:55	28027779773	✅ success	—
06-23 07:13	28008999561	cancelled	superseded
06-23 00:39	27993948299	cancelled	superseded

Cancellation-by-supersession is recurring for this workflow because back-to-back scheduled/dispatched runs on main collide in one concurrency group.

Failure Cluster

CI run cancelled by concurrency supersession (no test failure)

Facts (verified observations, no interpretation)

• Job 83027275938 conclusion = cancelled, steps = [], started 2026-06-23T18:11:00Z, completed 2026-06-23T18:40:51Z (GitHub API).
• Job log fetch returns <Error><Code>BlobNotFound</Code> — no log artifact exists.
• Run 28046873327 conclusion = cancelled; job breakdown: 7 success, 40 cancelled, 2 failure (both *-finish aggregators).
• Next scheduled run 28048657152 created 2026-06-23T18:40:50Z — 1 s before this job's cancel timestamp.
• Concurrency group pr-test-amd-rocm720-${{ github.ref }} with cancel-in-progress: true for schedule events (pr-test-amd-rocm720.yml:122-123).

Hypothesised Causes (with confidence, with disconfirming evidence)

• [HIGH] Run was cancelled because a newer scheduled run entered the same concurrency group and triggered cancel-in-progress.
  • Supporting: cancel timestamp (18:40:51) is 1 s after the next run was queued (18:40:50); concurrency group keyed on github.ref (= refs/heads/main, shared across all scheduled runs) with cancel-in-progress: true; 40 jobs cancelled simultaneously (run-wide, not job-specific).
  • Disconfirming: (none found) — no test error exists in any artifact to suggest a code-level failure.
  • How to verify: confirm run 28048657152 shares group pr-test-amd-rocm720-refs/heads/main; re-run this single job — it should pass if no superseding run is queued.

In-flight Fix Check

Not applicable — there is no test/code failure to fix. This is expected concurrency behavior. No fix-PR search performed.

Suggested Triage Steps

• Treat this job as non-actionable noise: cancelled by supersession, not a real failure. No bisect needed.
• If a clean signal is needed for commit ed26a109, re-run only this job/run when no newer scheduled run is pending, or wait for the superseding run 28048657152 to complete and read its result instead.
• (CI hygiene, maintainer call) If frequent supersession is hurting signal, consider keying the scheduled concurrency group on github.run_id (so scheduled runs don't cancel each other) — but this increases runner load; evaluate the trade-off.

Status (factual, no priority assignment)

• Persistence: One-off orchestration event; multiple recent scheduled runs of this workflow show the same cancelled pattern (06-23 00:39, 07:13, 18:10).
• Scope: Run-wide — 40 jobs in run 28046873327 cancelled together; not specific to this job or any test file.
• Blocked work: None — superseding run 28048657152 re-tests the same branch.
• In-flight fix: N/A — not a code failure.

multimodal-gen-test-1-gpu-amd-rocm720 (linux-mi325-1gpu-sglang, 2) — failed step(s): N/A

Commit Info

• sglang: ed26a109
• aiter: N/A — job log blob does not exist (BlobNotFound); the job was cancelled before logs were persisted, so no [CI-AITER-CHECK] markers are available.

Failed Tests

Test File	Test Function	Error	Log
N/A	N/A	No test failure — job cancelled (not failed)	Job

This job did not fail at the test level. Its conclusion is cancelled, it has zero recorded steps, and its log blob returns BlobNotFound. Nothing executed to a pass/fail verdict.

Failure Summary

The job multimodal-gen-test-1-gpu-amd-rocm720 (linux-mi325-1gpu-sglang, 2) was cancelled, not failed. The entire run 28046873327 was cancelled: of 49 jobs, 40 are cancelled, 7 success, and 2 failure (both are downstream finish-gate jobs — pr-test-amd-rocm720-finish and call-pr-test-amd-extra-rocm720 / pr-test-amd-extra-finish — which report failure when their upstream matrix is cancelled). This is an orchestration cancellation, not a code or test problem.

Regression Status

Infrastructure issue (concurrency cancellation) — not a test regression.

Recent pr-test-amd-rocm720.yml runs on main, event=schedule, completed runs only:

Created (UTC)	Run	Conclusion	Note
2026-06-23 18:40	28048657152	queued/in-progress	The run that superseded ours [IN-FLIGHT]
2026-06-23 18:10	28046873327	❌ cancelled	this run
2026-06-23 12:55	28027779773	✅ success
2026-06-23 07:13	28008999561	❌ cancelled	superseded
2026-06-23 00:39	27993948299	❌ cancelled	superseded
2026-06-22 19:05	27977093213	❌ cancelled	superseded

cancelled is the dominant outcome for this scheduled workflow — consistent with frequent overlapping schedule triggers cancelling earlier in-flight runs via the concurrency group.

Failure Cluster

Run-level concurrency cancellation (newer scheduled run superseded older in-flight run)

Facts (verified observations)

• Job conclusion = cancelled, status = completed; steps list is empty (verified via /actions/jobs/83027275971).
• Job log blob returns <Error><Code>BlobNotFound</Code></Error> — no log was persisted.
• Job started 2026-06-23T18:11:00Z, cancelled 2026-06-23T18:40:51Z (~30 min in).
• The next scheduled run 28048657152 was created 2026-06-23T18:40:50Z — 1 second before this job's cancellation timestamp.
• Run-wide: 40/49 jobs cancelled, 7 success, 2 failure (both downstream finish-gate aggregators).
• The same cancelled outcome recurs across the majority of recent event=schedule runs of this workflow.

Hypothesised Causes (with confidence)

• [HIGH] Concurrency-group cancellation: the newer scheduled run 28048657152 cancelled this in-flight run.
  • Supporting: next run created at 18:40:50Z, this job cancelled at 18:40:51Z (1 s apart); 40/49 jobs cancelled simultaneously across the run; this is the recurring outcome for scheduled runs of this workflow.
  • Disconfirming: (none found) — no test executed, no log evidence of any test or build error.
  • How to verify: inspect the concurrency: block in .github/workflows/pr-test-amd-rocm720.yml; confirm cancel-in-progress: true with a group key shared across schedule triggers.

In-flight Fix Check

Not applicable — this is a concurrency-cancellation artifact, not a code/test defect. No fix PR is warranted. (No keyword search performed because there is no failing test or error signature to match.)

Suggested Triage Steps

• Treat this job as non-actionable noise: it was cancelled by a superseding scheduled run, not failed.
• If these cancellations are polluting CI signal, review the concurrency group key for pr-test-amd-rocm720.yml — overlapping schedule triggers are cancelling earlier in-flight runs ~30 min in. Consider a less frequent schedule or a concurrency key that does not cancel scheduled runs mid-flight.
• For real signal on this matrix shard, look at the most recent completed success scheduled run 28027779773 or wait for the superseding run 28048657152 to complete.

Status

• Persistence: Recurring cancelled outcome across most recent scheduled runs; not a persistent test failure.
• Scope: Run-wide (40/49 jobs cancelled in run 28046873327); not specific to this multimodal-gen shard.
• Blocked work: None — no test or build defect; signal is simply absent for this superseded run.
• In-flight fix: N/A — not a code defect.

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-06-24 19:29 UTC)

[amd-bot]

pr-test-amd-rocm720.yml — Per-Job Analysis (continued 3/12)

stage-c-test-large-8-gpu-amd-mi35x-rocm720 (linux-mi35x-gpu-8, 1) — failed step(s): N/A

Commit Info

• sglang: ed26a109
• aiter: N/A (job was cancelled before any step ran; no [CI-AITER-CHECK] markers were emitted)

Failed Tests

Test File	Test Function	Error	Log
N/A	N/A	Job cancelled before any step executed — no test ran	Job

The job stage-c-test-large-8-gpu-amd-mi35x-rocm720 (linux-mi35x-gpu-8, 1) has conclusion: cancelled, an empty steps array, and a 404 BlobNotFound for its log — meaning it was terminated while queued/starting, before executing the setup, dependency-install, or test steps.

Failure Summary

This is not a test failure. The entire run 28046873327 (PR Test ROCm 7.2 (AMD), event: schedule, branch main) was cancelled: 40 jobs cancelled, 7 success, and 2 finish-gate failure jobs (which fail by design when upstream jobs are cancelled). The target 8-GPU MI35x job never ran a single step.

Regression Status

Infrastructure issue — concurrency cancellation (not a regression, not flaky).

Root cause is a concurrency-group supersession, verified by three independent facts:

1. [FACT] Timing is exact. The job was cancelled at 2026-06-23T18:40:51Z. The very next schedule run 28048657152 (sha e0dc8b71) was created at 2026-06-23T18:40:50Z — one second earlier.
2. [FACT] The concurrency config cancels in-progress schedule runs. In pr-test-amd-rocm720.yml:122-123, schedule runs share group pr-test-amd-rocm720-${{ github.ref }} (= ...-refs/heads/main) and cancel-in-progress evaluates to true (not run_all_tests, not workflow_call, not workflow_dispatch). So a newer scheduled run on main cancels the older in-progress one.
3. [FACT] This is the dominant pattern for this workflow. 8 of the last 10 scheduled runs are cancelled (only 28027779773 succeeded and one is still queued) — consistent with frequent scheduled runs (a 0 */6 * * * cron plus extra triggers) repeatedly superseding each other on the shared refs/heads/main group.

Recent history of this job (stage-c-test-large-8-gpu-amd-mi35x-rocm720, branch main, event=schedule, completed runs only):

Date	Run	Job Status	Note
Jun 23 18:10	28046873327	⚪ Cancelled	Superseded by next schedule run at 18:40:50

(The job produced no test-level signal because no step executed; there is nothing to compare at the test-file level.)

Failure Cluster

Concurrency-group cancellation of in-progress scheduled run (no steps executed)

Facts (verified observations, no interpretation)

• Job 83027275950: status=completed, conclusion=cancelled, started_at=2026-06-23T18:11:00Z, completed_at=2026-06-23T18:40:51Z, steps=[] (empty).
• Log fetch returns HTTP BlobNotFound — no log was ever written (job never reached a logging step).
• Run 28046873327: conclusion=cancelled, event=schedule, head_sha=ed26a109, run_attempt=1; 49 jobs total (7 success / 40 cancelled / 2 failure).
• Next schedule run 28048657152 created 2026-06-23T18:40:50Z, i.e. 1s before this job's cancel timestamp.
• Concurrency config: pr-test-amd-rocm720.yml:122-123, cancel-in-progress: true for schedule events.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [HIGH] The run was cancelled by GitHub Actions concurrency (cancel-in-progress) when the next scheduled run on main started.
  • Supporting: cancel timestamp 18:40:51Z vs next-run creation 18:40:50Z (1s apart); both share concurrency group pr-test-amd-rocm720-refs/heads/main; config at lines 122-123 enables cancel-in-progress for schedule events; 40/49 jobs in the run share the cancelled conclusion (run-wide cancel, not a single-job kill).
  • Disconfirming: (none found) — there is no log, error, or stack trace indicating a test/runtime failure, consistent with a clean cancellation rather than a fault.
  • How to verify: inspect run 28048657152's concurrency group in the Actions UI, or re-run 28046873327 at a time with no overlapping schedule trigger and confirm the 8-GPU MI35x job runs to completion.

In-flight Fix Check

Not applicable — there is no defect to fix; the cancellation is expected behavior of the concurrency design. No fix PR should be opened for this.

Suggested Triage Steps

• No code action needed. Treat this cancellation as noise from overlapping scheduled triggers, not a test regression.
• To obtain real CI signal for ed26a109 on this job, re-run the run 28046873327 (or trigger a workflow_dispatch, which gets a unique concurrency group per pr-test-amd-rocm720.yml:122 and will not be cancelled by schedule runs).
• If frequent self-cancellation of scheduled main runs is undesirable, maintainers could consider giving schedule runs a per-run concurrency group (as is already done for workflow_dispatch) — a workflow-design decision, not required for this commit.

Status (factual, no priority assignment)

• Persistence: Single cancelled run; the broader workflow shows 8/10 recent scheduled runs cancelled due to the same concurrency design.
• Scope: Run-wide — 40 of 49 jobs in run 28046873327 were cancelled together; this 8-GPU MI35x job is one of them.
• Blocked work: No test signal produced for commit ed26a109 on this job; needs a non-superseded re-run to validate.
• In-flight fix: N/A — not a defect.

stage-a-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang) — failed step(s): N/A

Commit Info

• sglang: ed26a109
• aiter: N/A — the job was cancelled and its log blob is gone (BlobNotFound); no [CI-AITER-CHECK] markers are recoverable.

Failed Tests

Test File	Test Function	Error	Log
N/A	N/A	No test failed — the job was cancelled, not failed	Job

This is not a test failure. The job conclusion is cancelled. Its log blob no longer exists (BlobNotFound), so no test-level data is recoverable — consistent with a job that was terminated mid-run rather than completing with a test failure.

Failure Summary

The entire run 28046873327 was cancelled at 2026-06-23T18:40:51Z: 40 of 49 jobs (including this one) ended at that same instant, and the only two failure jobs are the *-finish aggregation gates reacting to the cancel. No job ran to a test verdict. This is a CI-orchestration cancellation, not a code regression.

Regression Status

Infrastructure issue — run-level concurrency cancellation (not a regression, not a flake, not a never-passed test).

Recent schedule/main runs of pr-test-amd-rocm720.yml (completed only):

Created (UTC)	Run	Conclusion
2026-06-23 18:40	28048657152	queued — the superseding run
2026-06-23 18:10	28046873327	❌ cancelled (this run)
2026-06-23 12:55	28027779773	✅ success
2026-06-23 07:13	28008999561	❌ cancelled
2026-06-23 00:39	27993948299	❌ cancelled
2026-06-22 19:05	27977093213	❌ cancelled

Cancellation is a chronic, recurring pattern for scheduled runs of this workflow, not a one-off.

Failure Cluster

Scheduled run cancelled by colliding cron (concurrency cancel-in-progress)

Facts (verified observations)

• Job conclusion = cancelled; run conclusion = cancelled (verified via Jobs API). No steps array on the job, no log blob (BlobNotFound).
• [FACT] All 40 cancelled jobs share completion timestamp 18:40:51–18:40:52Z — a single external cancellation, not 40 independent failures.
• [FACT] The next scheduled run 28048657152 was created at 2026-06-23T18:40:50Z — within 1 second of this run's cancellation.
• [FACT] pr-test-amd-rocm720.yml defines two crons: 30 17 * * * (daily) and 0 */6 * * * (every 6h). The 06:00/12:00/18:00 6-hourly fire lands in the same hour as the delayed daily fire, so two scheduled runs of the same workflow overlap (.github/workflows/pr-test-amd-rocm720.yml:7-9).
• [FACT] For schedule events the concurrency group resolves to pr-test-amd-rocm720-refs/heads/main with cancel-in-progress: true (pr-test-amd-rocm720.yml:122-123). Any newer scheduled run on main therefore cancels the in-flight one.

Hypothesised Causes

• [HIGH] The run was cancelled by GitHub Actions concurrency, because the later scheduled run 28048657152 entered the same concurrency group (pr-test-amd-rocm720-refs/heads/main, cancel-in-progress: true) one second before this run died.
  • Supporting: matching 1-second timestamps; concurrency config at pr-test-amd-rocm720.yml:122-123; two overlapping crons at lines 7-9; chronic cancel pattern across recent scheduled runs.
  • Disconfirming: (none found) — no log blob to inspect, but all evidence is consistent and no test reached a verdict.
  • How to verify: open the superseding run's "cancelling" annotation, or compare concurrency-group keys of both runs in the Actions UI.

In-flight Fix Check

• In-flight fix: ⚠️ partial — #28987 "[CI] Fix AMD extra scheduled runs cancelling each other" (open since 2026-06-23) addresses the same class of bug but only edits .github/workflows/pr-test-amd-extra.yml, not pr-test-amd-rocm720.yml. It does not fix this workflow's collision.
• API used: search/issues?q=repo:sgl-project/sglang+is:pr+is:open+concurrency+cancel.

Suggested Triage Steps

• Confirm the cancel source: inspect run 28048657152 and the cancel annotation on 28046873327 in the Actions UI to verify they shared the concurrency group.
• Decide whether the two crons (30 17 daily + 0 */6) are intended to overlap; if not, consider giving each cron/purpose a distinct concurrency group (mirror the approach in #28987) so the 6-hourly dsv4-only run does not cancel the full daily run, or vice-versa.
• No code/test investigation is warranted — ed26a109 did not cause a test failure here. Re-trigger the workflow (or wait for the next non-colliding schedule) to obtain a clean signal for this SHA.

Status

• Persistence: Recurring — most recent scheduled main runs of this workflow are cancelled (e.g. 18:10, 07:13, 00:39 on 2026-06-23; multiple on 2026-06-22). The 12:55 run succeeded, so signal is intermittently available.
• Scope: Whole-run cancellation — all 40 in-flight jobs of run 28046873327, not just this job.
• Blocked work: Loses scheduled ROCm 7.2 CI signal whenever two crons collide in the same hour.
• In-flight fix: ⚠️ #28987 fixes the sibling pr-test-amd-extra.yml only; pr-test-amd-rocm720.yml still needs the same treatment.

stage-c-test-large-8-gpu-amd-rocm720 (linux-mi325-8gpu-sglang, 2) — failed step(s): N/A

Commit Info

• sglang: ed26a109
• aiter: N/A — job was cancelled before any setup step ran; no [CI-AITER-CHECK] markers exist (log blob BlobNotFound).

Failed Tests

Test File	Test Function	Error	Log
N/A	N/A	No test ran — job conclusion: cancelled, zero steps recorded	Job

This is not a test failure. The job stage-c-test-large-8-gpu-amd-rocm720 (linux-mi325-8gpu-sglang, 2) was cancelled (not failure), executed zero steps, and its log blob no longer exists.

Failure Summary

The job was preempted by GitHub Actions concurrency cancellation. A newer scheduled run of the same workflow started one second before this run's jobs were terminated, and the workflow's cancel-in-progress policy cancelled the older in-progress run. No test code was exercised.

Regression Status

Infrastructure issue (concurrency preemption) — not a code regression.

Run-level evidence (this run 28046873327, schedule event, branch main, completed):

Metric	Value
Target job conclusion	cancelled (0 steps)
Run conclusion	cancelled
Jobs: 49 total	7 success · 40 cancelled · 2 failure (only the *-finish aggregators, which fail because upstream was cancelled)
All 40 cancellations timestamp	identical 2026-06-23T18:40:51Z (single top-level cancel, not per-job)

Failure Cluster

Concurrency preemption — in-progress scheduled run cancelled by a newer scheduled run.

Facts (verified observations)

• [FACT] Job 83027275973 has conclusion: cancelled, steps: [] (API /actions/jobs/83027275973). It never started any test step.
• [FACT] Log fetch returns <Error><Code>BlobNotFound</Code> — no log was retained for this cancelled job.
• [FACT] All 40 cancelled jobs in run 28046873327 completed at the exact same instant 2026-06-23T18:40:51Z, the signature of one run-level cancellation.
• [FACT] A newer scheduled run 28048657152 was created at 2026-06-23T18:40:50Z (head e0dc8b71) — 1 second before the cancellation.
• [FACT] pr-test-amd-rocm720.yml:122-123 sets the concurrency group for schedule runs to github.ref (refs/heads/main) with cancel-in-progress: true when github.event_name is not workflow_call/workflow_dispatch and run_all_tests is false — i.e. true for schedule. Both runs share the same group ⇒ the new run cancels the old one.
• [FACT] The 2 failure jobs are aggregator *-finish jobs (83040560486), which report failure when their dependencies are cancelled — not independent test failures.

Hypothesised Causes

• [FACT] Concurrency preemption. Run 28046873327 (created 18:10:06) was cancelled by newer scheduled run 28048657152 (created 18:40:50) sharing concurrency group pr-test-amd-rocm720-refs/heads/main with cancel-in-progress: true.
  • Supporting: 1-second gap between new-run creation and the synchronized cancellation of all 40 jobs; concurrency config at pr-test-amd-rocm720.yml:122-123; both runs are schedule on main.
  • Disconfirming: (none found) — there is no evidence of any test executing or failing.
  • How to verify: already verified from API timestamps + workflow config; no further action needed.

In-flight Fix Check

Not applicable — no test/code failure to fix. This is expected behavior of the cancel-in-progress concurrency policy.

• In-flight fix: ❌ none needed — preemption is by design, not a bug.

Suggested Triage Steps

• No code triage required. This job did not test anything; treat it as a non-signal.
• For CI signal on commit ed26a109, consult the superseding run 28048657152 (head e0dc8b71) or the most recent completed schedule run 28027779773 (✅ success) instead.
• If preemption of scheduled runs is undesired, a maintainer could give schedule runs a per-run concurrency group (as is already done for workflow_dispatch/run_all_tests) — a workflow-config decision, not a code fix.

Status

• Persistence: Single occurrence; this run was preempted ~30 min in. Several adjacent scheduled runs also show cancelled (e.g. 28008999561, 27993948299), consistent with frequent schedule-on-schedule preemption rather than a test regression.
• Scope: Entire run cancelled (40 jobs), not specific to this job. No test signal produced.
• Blocked work: None — the next scheduled run supersedes this one and provides fresh signal.
• In-flight fix: ❌ none needed — preemption is by design.

multimodal-gen-test-1-gpu-amd-rocm720 (linux-mi325-1gpu-sglang, 0) — failed step(s): N/A

Commit Info

• sglang: ed26a109
• aiter: N/A — job log blob no longer exists (BlobNotFound); no [CI-AITER-CHECK] markers retrievable. Logs were never persisted because the job was cancelled mid-run.

Failed Tests

Test File	Test Function	Error	Log
N/A (no test failure)	N/A	Job cancelled mid-run — not a test failure	Job

This was not a test failure. The job's conclusion is cancelled, the whole run's conclusion is cancelled, and the log blob returns BlobNotFound (no logs persisted).

Failure Summary

The entire run 28046873327 was cancelled: of 49 jobs, 40 cancelled, 7 success, 2 failure (the two failure jobs are the pr-test-amd-rocm720-finish / pr-test-amd-extra-finish gate jobs, which fail because their upstream dependencies were cancelled). The target job multimodal-gen-test-1-gpu-amd-rocm720 (linux-mi325-1gpu-sglang, 0) was cancelled at 18:40:51Z after running ~30 min — it did not reach a test verdict.

Regression Status

Infrastructure issue (concurrency cancellation) — not a code regression.

Recent pr-test-amd-rocm720.yml scheduled runs on main (completed runs only):

Date (UTC)	Run	Conclusion
Jun 23 18:10	28046873327	❌ cancelled (this run)
Jun 23 12:55	28027779773	✅ success
Jun 23 07:13	28008999561	❌ cancelled
Jun 23 00:39	27993948299	❌ cancelled
Jun 22 19:05	27977093213	❌ cancelled
Jun 22 18:45	27975943807	❌ cancelled

Cancellation is the dominant outcome for this workflow's scheduled runs (only 1 of the last ~8 succeeded), consistent with a recurring concurrency collision rather than a per-test regression.

Failure Cluster

Whole-run cancellation by concurrency group (superseding scheduled run), no test executed to completion.

Facts (verified observations)

• [FACT] Job conclusion = cancelled; run conclusion = cancelled (from GitHub API).
• [FACT] Job started 2026-06-23T18:11:00Z, completed (cancelled) 2026-06-23T18:40:51Z.
• [FACT] Job log returns <Error><Code>BlobNotFound</Code></Error> — no log content to inspect.
• [FACT] Run breakdown: 40 cancelled, 7 success, 2 failure (the 2 failures are *-finish gate jobs).
• [FACT] The next scheduled run 28048657152 was created at 2026-06-23T18:40:50Z — 1 second before this job's cancellation timestamp (18:40:51Z).
• [FACT] pr-test-amd-rocm720.yml:122-123: schedule runs share concurrency group pr-test-amd-rocm720-${{ github.ref }} (= …-refs/heads/main for all schedule events) with cancel-in-progress: true.
• [FACT] Two cron triggers exist (pr-test-amd-rocm720.yml:8-9): 30 17 * * * (daily) and 0 */6 * * * (6-hourly). With GitHub's schedule-firing delays these overlap on main.

Hypothesised Causes

• [HIGH] This run was cancelled by GitHub Actions concurrency when the next scheduled run started.
  • Supporting: new run 28048657152 created 18:40:50Z, this job cancelled 18:40:51Z (1 s later); both are schedule events on main, sharing group pr-test-amd-rocm720-refs/heads/main with cancel-in-progress: true (pr-test-amd-rocm720.yml:122-123); the two crons (30 17 and 0 */6) plus firing delays produce overlapping runs.
  • Disconfirming: (none found) — timing, concurrency config, and the cancelled-run pattern all converge.
  • How to verify: confirm 28048657152 is on main (event=schedule) — already verified head_sha=e0dc8b71, branch main.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for pr-test-amd-rocm720 concurrency:

  curl -s -H "Authorization: token $GH_PAT" "https://api.github.com/search/issues?q=repo:sgl-project/sglang+is:pr+is:open+concurrency+rocm720"
  

  This is an infra/scheduling artifact, not a code defect requiring a fix PR.

Suggested Triage Steps

• No code action needed — there is no test failure here. Treat this job as cancelled/non-signal.
• If recurring scheduled-run cancellation is undesirable, consider whether the two cron triggers (30 17 * * * and 0 */6 * * *) should share a concurrency group, or scope the group per-cron, so the daily run is not killed by the 6-hourly run (or vice versa). This is a workflow-config decision for maintainers.
• To get a real signal for multimodal-gen-test-1-gpu-amd-rocm720, check the most recent completed/success scheduled run 28027779773 or re-run this workflow via workflow_dispatch (which gets a unique concurrency group per pr-test-amd-rocm720.yml:122 and won't be cancelled).

Status

• Persistence: Cancellation is the dominant outcome for this workflow's scheduled runs (≥6 of last ~8 cancelled); chronic concurrency collision, not a new event.
• Scope: Whole run — 40/49 jobs cancelled in run 28046873327. No single test implicated.
• Blocked work: No real test signal from this scheduled run; rely on the last successful run 28027779773 instead.
• In-flight fix: ❌ none found — no fix needed (infra/scheduling artifact).

stage-c-test-large-8-gpu-amd-rocm720 (linux-mi325-8gpu-sglang, 0) — failed step(s): N/A

Commit Info

• sglang: ed26a109
• aiter: N/A — the job was cancelled before running tests; log blob has expired (BlobNotFound) so no [CI-AITER-CHECK] markers are recoverable.

Failed Tests

Test File	Test Function	Error	Log
N/A	N/A	No test failed — job conclusion is cancelled (not failure)	Job

This was not a test failure. The job stage-c-test-large-8-gpu-amd-rocm720 (linux-mi325-8gpu-sglang, 0) was terminated mid-run by GitHub Actions concurrency cancellation. The job reports zero steps and conclusion: cancelled.

Failure Summary

The job did not fail a test — it was cancelled. The entire run 28046873327 (conclusion: cancelled) had 40/49 jobs cancelled, 7 succeeded, and only 2 "failures" — both of which are the downstream finish-aggregator jobs (pr-test-amd-rocm720-finish, call-pr-test-amd-extra-rocm720 / pr-test-amd-extra-finish) that are designed to fail when their upstream matrix is cancelled. No real test failure occurred.

Regression Status

Infrastructure issue (benign concurrency cancellation) — not a regression, not a flake.

The cancellation cascade is expected behavior by design. The workflow's concurrency block (.github/workflows/pr-test-amd-rocm720.yml:122-123) sets:

• group: pr-test-amd-rocm720-${{ ... || github.ref }} → for schedule events this resolves to pr-test-amd-rocm720-refs/heads/main
• cancel-in-progress: true for schedule events

Two scheduled crons fired close together on the same main ref:

Time (UTC)	Run	Cron	Outcome
18:10:06	28046873327 (this run)	30 17 * * * daily (delayed)	❌ Cancelled at 18:40:51
18:40:50	28048657152	0 */6 * * * 18:00 (delayed)	Superseded the prior run

The newer run was created at 18:40:50Z and this job was cancelled at 18:40:51Z — a 1-second gap that conclusively ties the cancellation to the concurrency supersede. This pattern is recurring: most scheduled pr-test-amd-rocm720.yml runs in the history show conclusion: cancelled for the same reason (two overlapping crons sharing one main-keyed concurrency group with cancel-in-progress: true).

Facts (verified observations, no interpretation)

• Job 83027276011 conclusion: cancelled, started_at: 18:11:00Z, completed_at: 18:40:51Z, zero steps recorded.
• Run 28046873327 conclusion: cancelled; job tally = 7 success, 40 cancelled, 2 failure (both finish-aggregators only).
• The next scheduled run 28048657152 was created_at: 18:40:50Z — 1 s before this job's cancellation timestamp.
• Concurrency config (.github/workflows/pr-test-amd-rocm720.yml:122-123): schedule runs share group pr-test-amd-rocm720-refs/heads/main with cancel-in-progress: true.
• Log blob is expired/unavailable (BlobNotFound), consistent with a cancelled job whose logs were never finalized.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [FACT] The job was cancelled by GitHub Actions concurrency supersede, not by any test failure.
  • Supporting: job conclusion: cancelled with zero steps; newer scheduled run 28048657152 created at 18:40:50Z vs this job cancelled at 18:40:51Z; concurrency group with cancel-in-progress: true for schedule events confirmed in workflow source.
  • Disconfirming evidence: (none found) — the only two non-cancelled non-success jobs are finish-aggregators that fail as a downstream consequence of the cancel, not independent test failures.
  • How to verify: re-read job 83027276011 metadata (conclusion: cancelled, no steps) and compare created_at of run 28048657152 to this job's completed_at.

In-flight Fix Check

Not applicable — this is not a test failure or product bug; no fix is warranted. The cancellation is the intended cancel-in-progress behavior. (No PR search performed because there is nothing to fix.)

Suggested Triage Steps

• No code action needed. This is an expected concurrency cancellation, not a CI failure to investigate.
• If these benign cancelled runs create noise in failure tracking, consider filtering scheduled-run analysis to conclusion in {success, failure} and excluding conclusion == cancelled runs (and excluding the *-finish aggregator jobs, which fail by design on upstream cancel).
• If full daily-cron signal is desired despite the overlapping 6-hour cron, the maintainer could de-duplicate the two crons' concurrency groups (e.g. key the group on the cron schedule) so the 18:00 6-hourly run no longer cancels the delayed 17:30 daily run — but this is a workflow-design choice, not a bug.

Status (factual, no priority assignment)

• Persistence: Recurring across most scheduled pr-test-amd-rocm720.yml runs (the majority of the last ~12 scheduled runs show conclusion: cancelled from the same overlapping-cron supersede).
• Scope: Run-wide — 40 jobs in run 28046873327 were cancelled together; not specific to this 8-GPU job.
• Blocked work: None — no test signal was lost that wasn't re-covered by the superseding run 28048657152.
• In-flight fix: N/A — not a failure requiring a fix.

stage-c-test-large-8-gpu-amd-rocm720 (linux-mi325-8gpu-sglang, 3) — failed step(s): N/A

Commit Info

• sglang: ed26a109
• aiter: N/A (job log blob expired — BlobNotFound — so no [CI-AITER-CHECK] markers are retrievable; the job never reached a reportable test stage)

Failed Tests

Test File	Test Function	Error	Log
N/A	N/A	Job cancelled, not failed — no test failure occurred	Job

This job did not fail a test. Its conclusion is cancelled. The run-level conclusion is also cancelled, and the per-job log blob has already been purged (BlobNotFound), which is normal for a job terminated mid-execution.

Failure Summary

The job stage-c-test-large-8-gpu-amd-rocm720 (linux-mi325-8gpu-sglang, 3) was cancelled (started 18:11:00Z, cancelled 18:40:52Z), not failed. It was one of 40 jobs in run 28046873327 that were all cancelled within the same one-second window (18:40:51–18:40:52Z). The only two failure conclusions in the entire run are the downstream aggregator jobs (pr-test-amd-rocm720-finish, call-pr-test-amd-extra-rocm720 / pr-test-amd-extra-finish), which report failure solely because their upstream jobs were cancelled.

Regression Status

Infrastructure issue (concurrency cancellation) — not a code regression and not a flake.

Recent pr-test-amd-rocm720.yml scheduled runs on main (completed runs only):

Date	Run	Event	Conclusion	Note
Jun 23 18:40	28048657152	schedule	cancelled	The run that superseded ours
Jun 23 18:10	28046873327	schedule	cancelled	This run (cancelled by the one above)
Jun 23 12:55	28027779773	schedule	success	Ran to completion (no overlap)
Jun 23 07:13	28008999561	schedule	cancelled	Same concurrency-cancel pattern
Jun 23 00:39	27993948299	schedule	cancelled	Same pattern

cancelled is the recurring norm for this workflow's schedule runs whenever two cron triggers overlap; the 12:55 run succeeded because no newer run started during its execution.

Failure Cluster

Concurrency-group cancellation by a superseding scheduled run (no test failure).

Facts (verified observations, no interpretation)

• Job conclusion = cancelled; run conclusion = cancelled (both verified via /jobs/83027276017 and /runs/28046873327).
• 49 jobs total in the run: 40 cancelled, 7 success, 2 failure (both the *-finish aggregators, which fail because upstream was cancelled).
• 40 jobs were cancelled simultaneously at 2026-06-23T18:40:51Z–18:40:52Z.
• A newer schedule run 28048657152 (sha e0dc8b71) was created at 2026-06-23T18:40:50Z — the exact instant the cancellations fired.
• Workflow concurrency config (.github/workflows/pr-test-amd-rocm720.yml:117-123): group pr-test-amd-rocm720-${{ ... || github.ref }} resolves to pr-test-amd-rocm720-refs/heads/main for schedule events, with cancel-in-progress: true (true because event is schedule, not workflow_call/workflow_dispatch/run_all_tests).
• The per-job log blob returns BlobNotFound (expired/purged); no test-level evidence is retrievable, consistent with a job killed before completing.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [FACT] The job was cancelled by GitHub Actions concurrency control when the next scheduled run entered the shared group pr-test-amd-rocm720-refs/heads/main.
  • Supporting: superseding run 28048657152 created_at = 18:40:50Z matches the mass-cancel timestamp 18:40:51Z to the second; concurrency block sets cancel-in-progress: true for schedule events; all 40 in-flight jobs cancelled together (a single signal, not per-test failures).
  • Disconfirming: (none found) — there is no failure conclusion among the actual test jobs, and no test log indicates a code error.
  • How to verify: already verified — compare the cancel timestamps against the superseding run's creation time (done above).

In-flight Fix Check

Not applicable — there is no code/test failure to fix. This is expected concurrency behaviour (the daily 30 17 * * * cron run overlapped with the 0 */6 * * * cron run, and the newer one cancelled the older). No PR search performed because no fix is warranted.

Suggested Triage Steps

• No action required for code. This is not a test failure; the job was cancelled by design when a newer scheduled run took over the concurrency group.
• If a full signal is needed for this commit (ed26a109), re-trigger the workflow at a time when no other schedule run will overlap (the 12:55 run 28027779773 succeeded for exactly this reason), or use workflow_dispatch (which gets a unique concurrency group full-{run_id} and is therefore not cancelled).
• If these overlapping-cron cancellations are considered noise in CI monitoring, consider staggering the two cron schedules (30 17 * * * vs 0 */6 * * *) so the daily run finishes before the 18:00 6-hour run begins.

Status (factual, no priority assignment)

• Persistence: Recurring whenever two cron triggers overlap; the immediately prior overlapping runs (07:13, 00:39 on Jun 23) show the same cancelled outcome. The 12:55 non-overlapping run succeeded.
• Scope: Run-wide — 40 of 49 jobs in run 28046873327 cancelled in the same event, not specific to this 8-GPU shard.
• Blocked work: None — the superseding run 28048657152 provides the up-to-date signal (it too was later cancelled, but by the subsequent 00:33 run 28066754977).
• In-flight fix: N/A — no code defect.

stage-c-test-large-8-gpu-amd-rocm720 (linux-mi325-8gpu-sglang, 1) — failed step(s): N/A

Commit Info

• sglang: ed26a109 — "[Fix] Return streaming logprobs when reasoning/tool parser is active (#28601)"
• aiter: N/A — the job's log blob no longer exists (BlobNotFound) and the job was cancelled before producing any [CI-AITER-CHECK] markers. The sister finish job's env shows AITER_COMMIT_OVERRIDE empty, i.e. Dockerfile default would have been used.

Failed Tests

Test File	Test Function	Error	Log
— (none)	—	Job cancelled ~29 min in; no test reached a pass/fail verdict	Job (log expired: BlobNotFound)

This is not a test failure. The target job stage-c-test-large-8-gpu-amd-rocm720 (linux-mi325-8gpu-sglang, 1) has conclusion: cancelled, zero recorded steps, and its log blob was never finalized.

Failure Summary

The job did not fail on any test — it was cancelled at 18:40:51 UTC, ~29 min into its run, before any test produced a result. All 40 test jobs in run 28046873327 were cancelled at the identical timestamp (18:40:51–18:40:52). The two jobs the API reports as failure — pr-test-amd-rocm720-finish and pr-test-amd-extra-finish — are aggregator gates that exit 1 whenever any upstream need is cancelled; their "failure" is a downstream artifact of the cancellation, not an independent fault.

Regression Status

Infrastructure issue (concurrency cancellation) — recurring.

Recent scheduled (event=schedule, branch=main, completed) runs of pr-test-amd-rocm720.yml:

Date (UTC)	Run	Conclusion	Note
Jun 24 00:33	28066754977	queued	[IN-FLIGHT] excluded
Jun 23 18:40	28048657152	cancelled	the run that cancelled ours
Jun 23 18:10	28046873327	cancelled	this run
Jun 23 12:55	28027779773	success	only green run in window
Jun 23 07:13	28008999561	cancelled
Jun 23 00:39	27993948299	cancelled
Jun 22 19:05	27977093213	cancelled
Jun 22 18:45	27975943807	cancelled

9 of the last 10 scheduled runs are cancelled — this test job has essentially no usable green/red signal from the scheduled cadence right now.

Failure Cluster

Run-level concurrency cancellation (cancel-in-progress supersession of in-progress scheduled run) — not a test fault.

Facts (verified observations)

• [FACT] Target job 83027276029: conclusion: cancelled, started 18:11:00Z, completed 18:40:51Z, no steps recorded.
• [FACT] All 40 test jobs in run 28046873327 were cancelled within the same 1-second window (18:40:51–18:40:52Z).
• [FACT] The next scheduled run 28048657152 (sha e0dc8b71) has created_at 18:40:50Z — one second before the mass cancellation.
• [FACT] pr-test-amd-rocm720.yml:117-123 defines concurrency group pr-test-amd-rocm720-${{ … || github.ref }} (→ pr-test-amd-rocm720-refs/heads/main for schedule events) with cancel-in-progress: true when github.event_name is not workflow_call/workflow_dispatch and run_all_tests is unset — i.e. true for schedule.
• [FACT] The finish jobs' logs show the gate logic: if result == "failure" || result == "cancelled": exit 1; 83040560486 reports multimodal-gen-test-1-gpu-amd-rocm720: cancelled → The above jobs failed.
• [FACT] Target job's log endpoint returns BlobNotFound (log never finalized due to cancellation).

Hypothesised Causes (with confidence)

• [HIGH] Our run 28046873327 was cancelled by GitHub's concurrency cancel-in-progress when the newer same-branch scheduled run 28048657152 entered the shared group pr-test-amd-rocm720-refs/heads/main.
  • Supporting: newer run created_at 18:40:50Z exactly precedes the 18:40:51Z mass cancel; both are schedule/main → identical concurrency group; cancel-in-progress evaluates true for schedule (pr-test-amd-rocm720.yml:123); all 40 jobs killed simultaneously (signature of an external run-level cancel, not a per-test failure).
  • Disconfirming: (none found) — the timing, shared group, and config are mutually consistent.
  • How to verify: GitHub UI shows the cancel reason on run 28046873327; confirm two schedule runs (18:10 and 18:40) landed within the same run window.
• [MEDIUM] Underlying trigger: two scheduled runs landed ~30 min apart (18:10 and 18:40) while the suite needs far longer to complete, so each new scheduled run cancels the prior in-progress one — explaining why 9/10 recent scheduled runs are cancelled.
  • Supporting: cron 30 17 * * * + 0 */6 * * * (pr-test-amd-rocm720.yml:8-9); recurring cancelled pattern in the run history above; the lone success (12:55) is a run that happened not to be superseded.
  • Disconfirming: the 18:10 vs 18:40 spacing doesn't cleanly match either cron expression (possible GitHub schedule delivery jitter or a delayed delivery), so the exact trigger of the second run is not fully pinned.
  • How to verify: inspect the trigger metadata of run 28048657152 and compare scheduled-run inter-arrival times vs median suite duration.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for concurrency/cancel changes to this workflow:

  curl -s -H "Authorization: token $GH_PAT" "https://api.github.com/search/issues?q=repo:sgl-project/sglang+is:pr+is:open+pr-test-amd-rocm720+concurrency"
  

  No open PR addresses the cancellation cadence. (This is an orchestration/scheduling matter, not a code bug, so a fix PR may not be the right vehicle.)

Suggested Triage Steps

• Treat this job as no-signal, not failed: there is no test regression to chase from run 28046873327. Look instead at the most recent completed (non-cancelled) scheduled run — 28027779773 (Jun 23 12:55, success) — for the real state of stage-c-test-large-8-gpu-amd-rocm720.
• If consistent scheduled signal is wanted, investigate why scheduled runs arrive close together and supersede each other (review cron cadence in pr-test-amd-rocm720.yml:8-9 vs typical suite wall-clock; consider whether schedule runs should use a per-run unique concurrency group like the workflow_dispatch/run_all_tests path already does at line 122).
• To confirm the cancellation cause, open run 28046873327 in the UI and read the per-job cancel reason ("superseded by a newer run").

Status (factual)

• Persistence: 9 of the last 10 scheduled runs (branch=main) are cancelled; only 28027779773 (Jun 23 12:55) completed successfully in the window.
• Scope: run-wide — all 40 test jobs in run 28046873327 cancelled together; both finish gate jobs consequently report failure.
• Blocked work: scheduled AMD ROCm 7.2 CI produces little usable pass/fail signal under the current cadence; genuine regressions could be masked by chronic cancellation.
• In-flight fix: ❌ none found.

multimodal-gen-test-2-gpu-amd-rocm720 (linux-mi325-2gpu-sglang, 2) — failed step(s): N/A

Commit Info

• sglang: ed26a109
• aiter: N/A — job log blob no longer exists (BlobNotFound), so [CI-AITER-CHECK] markers are unreadable.

Failed Tests

Test File	Test Function	Error	Log
N/A — no test failure	N/A	Job cancelled mid-run (not failed)	Job (log expired)

This job did not fail. Its conclusion is cancelled. The whole run 28046873327 has conclusion cancelled (40/49 jobs cancelled, 7 success, 2 aggregation "finish" jobs marked failure because their upstream was cancelled). No test ever reported a failure; the job was killed at the orchestration layer.

Failure Summary

The 2-GPU multimodal-gen job started at 18:11:00Z and was killed at 18:40:51Z — exactly 1 second after the next scheduled run 28048657152 was created at 18:40:50Z. The two share a concurrency group with cancel-in-progress, so the newer run terminated this one before its long-running multimodal job could finish. This is an infrastructure/scheduling collision, not a code or test regression. The job log blob has already expired (BlobNotFound), consistent with a job that was cancelled before logs were persisted.

Regression Status

Infrastructure issue (scheduled-run concurrency collision) — recurring, never completes.

This specific job (multimodal-gen-test-2-gpu-amd-rocm720 (linux-mi325-2gpu-sglang, 2)) only runs in the daily 30 17 * * * full-suite cron; the 0 */6 * * * cron skips it (it targets dsv4-only). Every time the job is present, it is cancelled — never a clean pass/fail in the queryable window:

Date (UTC)	Run	Job present?	Status	Cancelled by next run created
Jun 23 18:10	28046873327	✅ ran	❌ Cancelled @18:40:51	28048657152 @18:40:50
Jun 22 18:45	27975943807	✅ ran	❌ Cancelled	27977093213 @19:05
Jun 21 18:01	27912869737	✅ ran	❌ Cancelled	27913598623 @18:31
Jun 23 12:55	28027779773	⊘ skipped (*/6 cron)	✅ run succeeded	—

No green run of this exact job exists in the lookback window — it is systematically cancelled before completion.

Facts (verified observations)

• [FACT] Job conclusion is cancelled, not failure; steps: []; started 2026-06-23T18:11:00Z, completed 2026-06-23T18:40:51Z (GitHub API /jobs/83027276023).
• [FACT] Parent run 28046873327 conclusion is cancelled (event=schedule, branch=main).
• [FACT] Next scheduled run 28048657152 was created 2026-06-23T18:40:50Z — 1 s before this job's cancellation timestamp.
• [FACT] Workflow concurrency (pr-test-amd-rocm720.yml:117-123): group resolves to github.ref (= refs/heads/main) for schedule events; cancel-in-progress is true for schedule events.
• [FACT] Two crons fire ~30 min apart after GitHub's scheduling delay: 30 17 * * * (daily full suite, the run that includes this job) and 0 */6 * * * (18:00, dsv4-only). They collide in the same concurrency group every day.
• [FACT] Log blob returns BlobNotFound — no test output to attribute a failure to.

Hypothesised Causes

• [FACT] Concurrency-group cancellation by a superseding scheduled run. The daily 17:30 full-suite cron run is consistently cancelled by the 18:00 6-hourly cron run because both map to concurrency group refs/heads/main with cancel-in-progress: true.
  • Supporting: cancellation timestamp (18:40:51) is 1 s after the next run's creation (18:40:50); workflow concurrency config confirms group+cancel; identical pattern on Jun 21/22/23.
  • Disconfirming: (none found) — there is no test error in any artifact; the log never persisted.
  • How to verify: inspect concurrency group keys of runs 28046873327 vs 28048657152, or re-dispatch the daily run in isolation and confirm the 2-GPU multimodal job completes when no later cron collides.

In-flight Fix Check

• In-flight fix: ⚠️ partial — #28987 "[CI] Fix AMD extra scheduled runs cancelling each other" (open since 2026-06-23) addresses this exact class of bug, but only edits .github/workflows/pr-test-amd-extra.yml — it does not touch pr-test-amd-rocm720.yml, the workflow that hosts this job. The rocm720 collision is not yet covered.
• Search run: q=repo:sgl-project/sglang+is:pr+is:open+concurrency+amd+schedule (18 hits; #28987 is the only direct match).

Suggested Triage Steps

• Apply the same fix approach as #28987 to pr-test-amd-rocm720.yml — make the concurrency group cron-specific (e.g. include github.event.schedule / github.workflow + cron in the group key) so the daily full-suite run and the 6-hourly dsv4 run no longer cancel each other.
• Alternatively, stagger the two crons further apart (the 30 17 daily and 0 */6 18:00 jobs land within ~30 min after GitHub's scheduling delay), or split them into separate workflow files with independent concurrency groups.
• Confirm the multimodal-gen 2-GPU job actually passes when run without collision (re-dispatch the daily suite manually and let it finish) before assuming any code-level problem.

Status

• Persistence: This job has been cancelled (never completing) in every daily full-suite run observed Jun 21–23; it only runs in that one daily cron, which is killed daily by the subsequent 6-hourly run.
• Scope: Orchestration-level; affects all jobs of the daily 30 17 rocm720 run, not just multimodal-gen. Same root cause as the sibling pr-test-amd-extra workflow targeted by #28987.
• Blocked work: Daily ROCm 7.2 full-suite signal (including 2-GPU multimodal-gen) is effectively never collected on schedule.
• In-flight fix: ⚠️ #28987 fixes the sibling pr-test-amd-extra.yml only; pr-test-amd-rocm720.yml still needs the equivalent change.

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-06-24 19:29 UTC)

[amd-bot]

pr-test-amd-rocm720.yml — Per-Job Analysis (continued 4/12)

stage-c-test-large-8-gpu-amd-mi35x-rocm720 (linux-mi35x-gpu-8, 0) — failed step(s): N/A

Commit Info

• sglang: ed26a109
• aiter: N/A — job log blob does not exist (BlobNotFound); no [CI-AITER-CHECK] markers retrievable. Job was cancelled before/without producing a persisted log.

Failed Tests

Test File	Test Function	Error	Log
N/A	N/A	Job cancelled (not a test failure) — superseded by a newer scheduled run via cancel-in-progress	Job

Failure Summary

The job stage-c-test-large-8-gpu-amd-mi35x-rocm720 (linux-mi35x-gpu-8, 0) did not fail a test — its conclusion is cancelled. The entire run 28046873327 was cancelled: of 49 jobs, 40 were cancelled, 7 success, and only the 2 *-finish/gate jobs report failure (a downstream consequence of the cancellation, not a real test failure). All 40 cancelled jobs stopped at the same instant (2026-06-23T18:40:51Z).

Regression Status

Infrastructure / not a regression — concurrency cancellation.

This is not a test failure and there is nothing to bisect. Recent pr-test-amd-rocm720.yml schedule runs on main:

Date	Run	Conclusion	Note
Jun 23 18:40	28048657152	cancelled	The superseding run (created 18:40:50Z) that cancelled the target
Jun 23 18:10	28046873327	cancelled	Target run — superseded
Jun 23 12:55	28027779773	success	—

Failure Cluster

Run cancelled by concurrency supersession (cancel-in-progress)

Facts (verified observations)

• Target job 83027276049 conclusion = cancelled; started_at=2026-06-23T18:11:00Z, completed_at=2026-06-23T18:40:51Z. The job recorded no steps (steps: []). [FACT]
• Log fetch returns <Error><Code>BlobNotFound</Code> — no persisted log exists for this job. [FACT]
• Run-level conclusion = cancelled; job conclusion breakdown across the 49 jobs: 40 cancelled, 7 success, 2 failure (pr-test-amd-extra-rocm720 / pr-test-amd-extra-finish and pr-test-amd-rocm720-finish). [FACT]
• All 40 cancelled jobs share the identical completed_at=2026-06-23T18:40:51Z — characteristic of a single workflow-level cancel, not independent test failures. [FACT]
• A newer schedule run 28048657152 was created_at=2026-06-23T18:40:50Z on the same branch main, one second before the target's jobs were cancelled. [FACT]
• .github/workflows/pr-test-amd-rocm720.yml:122-123: concurrency group: pr-test-amd-rocm720-…${{ … || github.ref }} and cancel-in-progress: ${{ !inputs.run_all_tests && github.event_name != 'workflow_call' && github.event_name != 'workflow_dispatch' }}. For a schedule event all three conditions are true, so cancel-in-progress = true and the group resolves to pr-test-amd-rocm720-refs/heads/main. [FACT]

Hypothesised Causes

• [HIGH] The job was cancelled because the newer scheduled run 28048657152 entered the same concurrency group (pr-test-amd-rocm720-refs/heads/main) and cancel-in-progress: true aborted the older in-progress run 28046873327.
  • Supporting: superseding run created 18:40:50Z, target jobs cancelled 18:40:51Z; concurrency group + cancel-in-progress evaluate true for schedule events (workflow file lines 122-123); all 40 jobs cancelled at the same timestamp.
  • Disconfirming: (none found) — no test output, exit code, or stack trace exists that would indicate a code-level failure.
  • How to verify: confirm overlap — the 18:10 run was still running (started 18:11, ~30 min in) when the 18:40 schedule cron fired. Two schedule crons (30 17 daily and 0 */6) plus the every-6h cadence make back-to-back schedule runs on main collide in this group.

In-flight Fix Check

Not applicable — there is no test/product failure to fix. This is expected concurrency behavior (the newer run is meant to supersede the older). No PR search performed; nothing to remediate in sglang code.

Suggested Triage Steps

• No action needed for code. Treat this job as a non-failure (cancelled), not a regression. Exclude cancelled jobs from failure trend/regression accounting per the "only completed, actually-run jobs count" rule.
• If these supersession-cancellations are noisy in CI dashboards: review whether the schedule cadence (30 17 * * * daily + 0 */6 * * *) should use a per-run concurrency group (like the workflow_dispatch/run_all_tests path already does) so scheduled runs don't cancel each other on main.
• To see real signal for this job, check the next completed schedule run that was not superseded (e.g. the latest queued run 28066754977 once it completes).

Status (factual)

• Persistence: One-off cancellation; the target run was superseded ~30 min into execution by the next scheduled run.
• Scope: Entire run 28046873327 — 40 jobs cancelled together; the 2 failure jobs are downstream gate/finish jobs reacting to the cancel.
• Blocked work: None — no test executed to completion or failed; no product regression indicated.
• In-flight fix: N/A — not a defect.

stage-b-test-2-gpu-large-amd-rocm720 (linux-mi325-2gpu-sglang, 0) — failed step(s): N/A

Commit Info

• sglang: ed26a109 (#28601 "Return streaming logprobs when reasoning/tool parser is active")
• aiter: N/A — the job log blob has expired (BlobNotFound from the logs API), so no [CI-AITER-CHECK] markers are retrievable. This is not an aiter-relevant failure regardless (see below).

Failed Tests

None. This job ran no failing test — it was cancelled mid-execution before any test could be reported as failed.

Test File	Test Function	Error	Log
N/A — job cancelled	N/A	Run-level cancellation (concurrency)	Job

Failure Summary

The job stage-b-test-2-gpu-large-amd-rocm720 (linux-mi325-2gpu-sglang, 0) has conclusion = cancelled, not failure. It started at 18:11:00Z and was killed at 18:40:51Z. ~40 jobs in run 28046873327 were cancelled at the same second (18:40:51–18:40:52Z). The only failure-conclusion jobs in the run are the two aggregator jobs pr-test-amd-rocm720-finish (83040560486) and call-pr-test-amd-extra-rocm720 / pr-test-amd-extra-finish (83033310987), which fail by design when upstream jobs are cancelled. No test regression occurred.

Regression Status

Infrastructure issue (concurrency cancellation) — not a code/test regression.

Recent event=schedule, branch=main runs of pr-test-amd-rocm720.yml (completed only):

Date (UTC)	Run	Conclusion	Note
Jun 23 18:40	28048657152	cancelled	superseded the next-next run
Jun 23 18:10	28046873327	cancelled	← this run, cancelled by the 18:40 run
Jun 23 12:55	28027779773	success	—
Jun 23 07:13	28008999561	cancelled	—
Jun 23 00:39	27993948299	cancelled	—
Jun 22 19:05	27977093213	cancelled	—

10 of the last 12 scheduled runs concluded cancelled — this is a chronic structural pattern, independent of commit ed26a109.

Failure Cluster

Scheduled run cancelled by concurrency cancel-in-progress (superseded by next scheduled run)

Facts (verified observations)

• [FACT] This job's conclusion = cancelled (job API), with no recorded step results and an expired log blob.
• [FACT] ~40 jobs in the run were cancelled simultaneously at 18:40:51–18:40:52Z (job-list API), i.e. run-level cancellation, not per-job/matrix fast-fail (fast-fail would skip downstream jobs, not cancel already-running ones).
• [FACT] The next scheduled run 28048657152 was created at 18:40:50Z — the same second this run's jobs were cancelled — and is also event=schedule.
• [FACT] .github/workflows/pr-test-amd-rocm720.yml:122-123: concurrency group: pr-test-amd-rocm720-…|| github.ref and cancel-in-progress: true for schedule events. All main scheduled runs share the group pr-test-amd-rocm720-refs/heads/main, so a newer scheduled run cancels the in-progress older one.
• [FACT] The workflow has two schedule crons (30 17 * * * daily full; 0 */6 * * * 6h dsv4-only, .github/workflows/pr-test-amd-rocm720.yml:7-9); both land in the same concurrency group, so the 6h cron can cancel the in-progress daily run.
• [FACT] Only the two *-finish aggregator jobs have conclusion=failure; every other job is success or cancelled. No test failed.

Hypothesised Causes

• [FACT] Run cancelled by GitHub Actions concurrency cancel-in-progress when the next scheduled run started.
  • Supporting: superseding run 28048657152 created at the exact cancellation second (18:40:50Z); shared concurrency group + cancel-in-progress: true for schedule events; mass simultaneous cancel of ~40 jobs.
  • Disconfirming: (none found) — there is no failure-conclusion test job and no log evidence of any test error.
  • How to verify: re-run the workflow standalone (or via workflow_dispatch, which gets a unique concurrency group) and confirm the job completes without external cancellation.

In-flight Fix Check

This is not a bug requiring a code fix; it is concurrency behavior. No fix-PR search applies. (The structural pattern — overlapping daily + 6h schedule crons sharing one cancel-in-progress group — could be revisited by maintainers if full daily signal is being lost, but that is a workflow-design decision, not a regression fix.)

• In-flight fix: ❌ none applicable — cancellation is expected concurrency behavior, not a defect.

Suggested Triage Steps

• Treat this job as no signal, not a failure — the underlying tests never reported a result.
• If full daily-run signal is needed, re-trigger via workflow_dispatch (gets a unique concurrency group per .github/workflows/pr-test-amd-rocm720.yml:122 and will not be cancelled by the next schedule).
• If maintainers want scheduled runs to stop cancelling each other, consider giving the daily 30 17 full run and the 6h dsv4-only cron distinct concurrency groups so the lightweight 6h run no longer aborts the in-progress full run.

Status

• Persistence: chronic — 10 of the last 12 scheduled runs concluded cancelled; this specific run was cancelled 29 min into execution.
• Scope: run-level — ~40 jobs in run 28046873327 cancelled together; not isolated to this job.
• Blocked work: scheduled ROCm 7.2 full-test signal is lost for this run, but no PR is blocked (schedule-triggered).
• In-flight fix: ❌ none applicable — expected concurrency cancellation, not a code defect.

multimodal-gen-test-2-gpu-amd-rocm720 (linux-mi325-2gpu-sglang, 1) — failed step(s): N/A

Commit Info

• sglang: ed26a109
• aiter: N/A (job was cancelled before any setup step ran; no [CI-AITER-CHECK] markers — log blob does not exist)

Failed Tests

Test File	Test Function	Error	Log
N/A — no test executed	N/A	Job cancelled before any step ran (0 steps recorded; log blob BlobNotFound)	Job

Failure Summary

This is not a test failure. The job multimodal-gen-test-2-gpu-amd-rocm720 (linux-mi325-2gpu-sglang, 1) was cancelled at 2026-06-23T18:40:51Z with zero steps recorded and no log artifact (the GitHub API returns BlobNotFound for its log). It was cancelled together with all 40 matrix jobs of run 28046873327 at the same instant — a run-level cancellation, not a per-test crash.

Regression Status

Infrastructure issue — concurrency cancel-in-progress cancellation, not a code regression.

Run-level conclusion counts for run 28046873327 (event=schedule, branch=main, completed):

Conclusion	Count
cancelled	40 (all matrix jobs, incl. this one)
success	7
failure	2 (pr-test-amd-rocm720-finish, pr-test-amd-extra-finish — cascade from cancelled upstream)

The target shard (..., 1) only exists in the full daily (30 17) cron matrix; recent runs that expand it have repeatedly ended in cancelled (see ### Facts), so no clean executed baseline for this exact test was available in the queryable window.

Failure Cluster

Scheduled run cancelled by concurrency cancel-in-progress (newer schedule run superseded the daily run)

Facts (verified observations)

• Job state: status=completed, conclusion=cancelled, steps=[], started 18:11:00Z, completed 18:40:51Z (GitHub API jobs/83027276083).
• Log unavailable: GET jobs/83027276083/logs → <Error><Code>BlobNotFound</Code> — no log was ever written (job never reached an executing step).
• Mass cancellation: 40 of 49 jobs in run 28046873327 all cancelled within the same second (18:40:51–18:40:52Z).
• Superseding run: run 28048657152 (event=schedule) was created at 18:40:50Z — 1 second before this run's jobs were cancelled.
• Concurrency config (.github/workflows/pr-test-amd-rocm720.yml:117-123): for schedule events the group resolves to pr-test-amd-rocm720-refs/heads/main and cancel-in-progress evaluates to true (not run_all_tests, not workflow_call, not workflow_dispatch).
• Two crons share that group (pr-test-amd-rocm720.yml:8-10): 30 17 * * * (daily full matrix) and 0 */6 * * * (every-6h, dsv4-only). Our run (created 18:10:06Z) is the delayed daily cron; the 18:00 six-hour cron (run 28048657152) entered the same group and cancelled it.
• Recurring pattern: the last ~10 scheduled runs are overwhelmingly cancelled (28048657152, 28008999561, 27993948299, 27977093213, 27957777753, …), consistent with the daily run being routinely superseded by the 6h cron.

Hypothesised Causes (with confidence)

• [HIGH] The daily-cron run was cancelled by GitHub Actions cancel-in-progress when the next scheduled run entered the shared concurrency group pr-test-amd-rocm720-refs/heads/main.
  • Supporting: superseding schedule run 28048657152 created 18:40:50Z, exactly 1 s before all 40 jobs cancelled at 18:40:51Z; concurrency block sets cancel-in-progress: true for schedule events; both 30 17 and 0 */6 crons resolve to the same refs/heads/main group.
  • Disconfirming: (none found) — timing, config, and the run-wide simultaneous cancellation all converge.
  • How to verify: inspect the run's cancellation audit / GET runs/28046873327 (conclusion=cancelled, no failed executing step) and confirm 28048657152 shares the group key.
• [FACT] The two *-finish job failures are a cascade, not independent failures — they ran after their upstream matrix was cancelled and reported the cancellation as a failed aggregation step.

In-flight Fix Check

• In-flight fix: ❌ none found — searched repo:sgl-project/sglang is:pr is:open pr-test-amd-rocm720 concurrency; only unrelated PR #24651 (Qwen3.5 FP8 kernel) returned.

Suggested Triage Steps

• No code/test action needed for ed26a109 — the test never ran, so this run produces no signal for the multimodal-gen suite. Re-derive its health from a daily run that completes uncancelled.
• If the goal is to stop the daily full run from being killed by the 6h dsv4 cron: review the concurrency group key in pr-test-amd-rocm720.yml:122 so the 30 17 (full matrix) and 0 */6 (dsv4-only) schedule crons do not collide — e.g. include the cron identity / a per-cron suffix in the group, or set cancel-in-progress: false for schedule events.
• To get a clean run on this SHA's matrix, re-run via workflow_dispatch (which gets a unique full-<run_id> group and is exempt from cancel-in-progress).

Status (factual)

• Persistence: this specific job had a log/step-level failure for 0 runs — it was never executed. The cancellation pattern, however, recurs across most recent scheduled runs (see Facts).
• Scope: entire run 28046873327 — 40 cancelled jobs + 2 cascade failure finish jobs; root cause is one orchestration event affecting the whole run, not this job alone.
• Blocked work: the daily ROCm 7.2 full-matrix signal (multimodal-gen, stage-a/b/c, disaggregation) is lost for this cycle; no per-test data was produced.
• In-flight fix: ❌ none found in open PRs.

multimodal-gen-test-1-gpu-amd-rocm720 (linux-mi325-1gpu-sglang, 3) — failed step(s): N/A

Commit Info

• sglang: ed26a109 (#28601 — Return streaming logprobs when reasoning/tool parser is active)
• aiter: N/A — the job was cancelled before any step ran; no log blob exists, so no [CI-AITER-CHECK] markers are available.

Failed Tests

Test File	Test Function	Error	Log
N/A	N/A	Job cancelled (not failed). No test ever ran; no test-level failure exists.	Job (log blob BlobNotFound)

Failure Summary

The job multimodal-gen-test-1-gpu-amd-rocm720 (linux-mi325-1gpu-sglang, 3) did not fail — its conclusion is cancelled. It started at 18:11:00Z, ran ~30 min, and was cancelled at 18:40:52Z when a newer scheduled run entered the same concurrency group. The job has zero recorded steps and its log returns BlobNotFound, so there is no test-level failure to report.

Regression Status

Infrastructure issue (concurrency cancellation) — not a code regression.

Recent history of this matrix job in job multimodal-gen-test-1-gpu-amd-rocm720 (linux-mi325-1gpu-sglang, 3) (branch main, event=schedule, completed runs only — the job only matrix-expands on the daily 30 17 * * * cron; the 0 */6 * * * cron skips it):

Date	Run	Job conclusion	Started → Completed	Note
Jun 23	28046873327	⏹️ Cancelled	18:11:00Z → 18:40:52Z	cancelled by next scheduled run 28048657152 (created 18:40:50Z)
Jun 22	27975943807	⏹️ Cancelled	18:45:52Z → 19:05:27Z	same concurrency-collision pattern

No green (completed-success) run of this matrix job exists in the queryable scheduled-run window (Jun 18–23): every daily full-cron run in that window is cancelled. Lookback range searched: Jun 18 2026 → Jun 24 2026.

Failure Cluster

Scheduled run cancelled by concurrency collision (daily 17:30 full cron vs 18:00 6h cron, shared group)

Facts (verified observations, no interpretation)

• Job conclusion = cancelled, not failure (verified via /actions/jobs/83027276109). Steps array is empty.
• Log endpoint returns <Error><Code>BlobNotFound</Code> — no log was persisted for the cancelled job.
• Run 28046873327 ("PR Test ROCm 7.2 (AMD)", event=schedule) overall conclusion = cancelled: 7 success, 40 cancelled, 2 failure. The 2 failure jobs are both aggregator *-finish jobs (pr-test-amd-rocm720-finish, pr-test-amd-extra-finish) that fail-by-design when upstream jobs are cancelled.
• The target job was cancelled at 18:40:52Z, ~2s after the next scheduled run 28048657152 was created at 18:40:50Z.
• Workflow concurrency (.github/workflows/pr-test-amd-rocm720.yml:117-123): group for schedule triggers resolves to pr-test-amd-rocm720-refs/heads/main; cancel-in-progress = true for schedule events. → the newer scheduled run cancels the in-progress one.
• Schedule config (.github/workflows/pr-test-amd-rocm720.yml:7-9): two crons — 30 17 * * * (daily full run, expands multimodal-gen matrix) and 0 */6 * * * (every 6h, dsv4-only, skips multimodal-gen).
• Recurring pattern: every day Jun 18–23, two scheduled runs are created ~30 min apart (delayed 17:30 daily run, then delayed 18:00 6h run), and the daily run is cancelled by the 6h run.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [FACT] The job was terminated by GitHub Actions concurrency cancellation, not by any test or build error.
  • Supporting: job conclusion cancelled with empty steps and no log; cancel timestamp (18:40:52Z) coincides with creation of the next scheduled run (18:40:50Z); workflow has cancel-in-progress: true for schedule events on a github.ref-keyed group shared by both daily and 6h crons.
  • Disconfirming: (none found) — there is no log, stack trace, or failing test indicating a code problem.
  • How to verify: re-read /actions/jobs/83027276109 (conclusion=cancelled) and compare cancel time with created_at of run 28048657152.
• [MEDIUM] Root scheduling cause: the 30 17 * * * daily full run is delayed by GitHub past 18:00, so the 0 */6 * * * 18:00 run lands within the same window and cancels it via the shared concurrency group.
  • Supporting: identical two-runs-~30min-apart, daily-run-cancelled pattern on Jun 18/19/20/21/22/23; both crons share group pr-test-amd-rocm720-refs/heads/main with cancel-in-progress.
  • Disconfirming: GitHub schedule delays are not directly observable in the API beyond created_at; exact dispatch-vs-cron mapping is inferred from timestamps, not logged.
  • How to verify: inspect a week of event=schedule runs' created_at (done) and confirm the later run of each daily pair is the dsv4-only 6h variant.

In-flight Fix Check

• Searched open PRs for a concurrency/scheduling fix on this workflow:
  • curl ".../search/issues?q=repo:sgl-project/sglang+is:pr+is:open+pr-test-amd-rocm720+concurrency"
• In-flight fix: ❌ none found in open PRs — no open PR addresses the daily-vs-6h cron concurrency collision. This is a workflow-config matter, not a product-code fix.

Suggested Triage Steps

• Treat this job result as non-actionable for code: it is a cancellation, not a failure. Do not attribute it to commit ed26a109 or to aiter.
• If full daily multimodal-gen signal is desired, investigate the cron collision in .github/workflows/pr-test-amd-rocm720.yml:7-9,117-123: e.g. give the daily 17:30 run a distinct concurrency group from the 0 */6 * * * dsv4 run, or move the 6h cron off the 18:00 slot so it doesn't cancel the (delayed) daily run.
• To obtain real pass/fail data for multimodal-gen-test-1-gpu-amd-rocm720 (..., 3), re-run the daily full workflow via workflow_dispatch (which gets a unique full-<run_id> group and won't be cancelled by the 6h cron).

Status (factual, no priority assignment)

• Persistence: This specific matrix job has been cancelled (never completing) on the daily full cron every day observed, Jun 18–23 2026.
• Scope: Affects the entire daily 17:30 scheduled run of pr-test-amd-rocm720.yml (40/49 jobs cancelled in run 28046873327), not just this one job. The multimodal-gen matrix only runs on this daily cron, so it gets no completed signal.
• Blocked work: Daily ROCm 7.2 full-suite CI signal (including multimodal-gen) is effectively lost to the cron collision.
• In-flight fix: ❌ none found in open PRs.

stage-b-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang, 12) — failed step(s): N/A

Commit Info

• sglang: ed26a109 — [Fix] Return streaming logprobs when reasoning/tool parser is active (#28601)
• aiter: N/A — the target job's log blob no longer exists (BlobNotFound) and the job recorded zero steps, so no [CI-AITER-CHECK] markers are available. The run's finish job env shows AITER_COMMIT_OVERRIDE= (empty), i.e. Dockerfile default would have applied had the job run.

Failed Tests

Test File	Test Function	Error	Log
N/A	N/A	Job cancelled (concurrency supersession) — no test executed	Job

This is not a test failure. The job stage-b-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang, 12) has conclusion: cancelled with an empty steps[] array — it never reached test execution.

Failure Summary

The job did not fail on any test; it was cancelled along with 39 other jobs in run 28046873327 (whole-run conclusion: cancelled). The only failure-conclusion jobs in the run are the two aggregator gate jobs (pr-test-amd-rocm720-finish, call-pr-test-amd-extra-rocm720 / pr-test-amd-extra-finish), which exit 1 by design whenever any upstream job is cancelled. The cancellation was triggered by a newer scheduled run entering the same concurrency group.

Regression Status

Infrastructure issue (concurrency cancellation) — not a code regression, not a flaky test.

Recent pr-test-amd-rocm720.yml scheduled runs on main (completed only):

Date (UTC)	Run	Conclusion	Note
Jun 23 18:40	28048657152	cancelled	superseding run (also later cancelled)
Jun 23 18:10	28046873327	cancelled	this run
Jun 23 12:55	28027779773	success	—
Jun 23 07:13	28008999561	cancelled	—
Jun 23 00:39	27993948299	cancelled	—

11 of the last 12 scheduled runs concluded cancelled — a systemic pattern, not specific to this commit or job.

Failure Cluster

Scheduled run cancelled by concurrency supersession (not a test failure)

Facts (verified observations)

• Target job conclusion: cancelled, started_at: 18:11:00Z, completed_at: 18:40:52Z, steps: [] (no step ever executed). [FACT, from jobs API]
• Whole run 28046873327 conclusion: cancelled; of 49 jobs: 7 success, 40 cancelled, 2 failure (both aggregator *-finish gate jobs). [FACT]
• The finish gate job log shows it exiting 1 solely because an upstream job was cancelled (multimodal-gen-test-1-gpu-amd-rocm720: cancelled → The above jobs failed. → exit code 1). [FACT]
• A newer schedule-event run 28048657152 was created at 18:40:50Z; this run's job was cancelled at 18:40:52Z — a 2-second gap. [FACT]
• Workflow concurrency (.github/workflows/pr-test-amd-rocm720.yml:122-123): group resolves to pr-test-amd-rocm720-${{ github.ref }} = pr-test-amd-rocm720-refs/heads/main for schedule events (no inputs.pr_head_sha/inputs.ref), with cancel-in-progress: true for non-workflow_call/non-workflow_dispatch events. [FACT]
• The target job's log blob returns BlobNotFound (expired/never finalized). [FACT]

Hypothesised Causes (with confidence)

• [FACT] The run was cancelled because a newer scheduled run of the same workflow entered the shared concurrency group pr-test-amd-rocm720-refs/heads/main while this run was in progress, and cancel-in-progress: true cancelled the older run.
  • Supporting: .github/workflows/pr-test-amd-rocm720.yml:122-123 config; superseding run 28048657152 created 18:40:50Z vs this job cancelled 18:40:52Z; all schedule runs collapse to the same github.ref-based group.
  • Disconfirming: (none found) — the timing and config are unambiguous.
  • How to verify: already verified via jobs API timestamps + workflow concurrency config.
• [FACT] The two failure-conclusion jobs are aggregator gates, not real test failures.
  • Supporting: finish job log exits 1 on any upstream cancelled/failure.
  • Disconfirming: (none).

There is no code-level regression to attribute to ed26a109; the commit's code was never exercised by this cancelled job.

In-flight Fix Check

• In-flight fix: ❌ none needed — this is not a test/code defect. The recurring cancellation is a CI-orchestration characteristic: overlapping schedule triggers (30 17 * * * plus 0 */6 * * *) plus a github.ref-keyed concurrency group with cancel-in-progress: true mean closely-spaced scheduled runs routinely cancel each other. (Searched open PRs is not applicable to a per-run cancellation.)

Suggested Triage Steps

• No action required for this specific job — it ran no tests and produced no failure signal. Treat as cancelled, retry/ignore.
• If the recurring schedule-run cancellation is undesirable for CI signal: review the two overlapping cron schedules (30 17 * * * and 0 */6 * * *) and the concurrency group keying. Because both crons land near 18:00 UTC, back-to-back runs share the refs/heads/main group and cancel each other. Consider keying the concurrency group per-github.run_id (or per-schedule) for schedule events, or staggering the crons, so legitimate scheduled runs are not cancelled.
• To obtain a real CI signal for this commit's code path, re-run the workflow when no overlapping scheduled run is queued, or trigger a workflow_dispatch run (which gets a unique full-<run_id> concurrency group and won't be cancelled).

Status

• Persistence: 11 of the last 12 scheduled pr-test-amd-rocm720.yml runs on main concluded cancelled (Jun 21–24); only 28027779773 (Jun 23 12:55) succeeded.
• Scope: Run-level — 40 of 49 jobs in run 28046873327 cancelled; affects the entire workflow, not this single shard.
• Blocked work: No test signal produced by this scheduled run; the workflow's scheduled coverage is effectively lost on cancelled runs.
• In-flight fix: ❌ none — not a code defect; a CI-orchestration (concurrency/cron) review is the lever, not a code fix.

stage-b-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang, 1) — failed step(s): N/A

Commit Info

• sglang: ed26a109 — "[Fix] Return streaming logprobs when reasoning/tool parser is active (#28601)"
• aiter: N/A — job log blob has expired (BlobNotFound) and the job ran 0 steps, so no [CI-AITER-CHECK] markers are available.

Failed Tests

Test File	Test Function	Error	Log
none	none	No test ran — job was cancelled before execution	Job

This is not a test failure. The job stage-b-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang, 1) completed with conclusion cancelled and recorded 0 steps — no runner was ever assigned, no setup/checkout/test step ran.

Failure Summary

The entire run 28046873327 (PR Test ROCm 7.2 (AMD), event=schedule, branch main) was cancelled. Of 49 jobs: 7 succeeded, 40 were cancelled (39 with 0 steps), and 2 failure jobs are the *-finish gates whose only failing step is "Check all dependent job statuses" (i.e. they failed because their dependents were cancelled, not from any test). The target job is one of the 39 that never started.

Regression Status

Infrastructure issue (concurrency cancellation) — not a code regression.

Recent scheduled runs of pr-test-amd-rocm720.yml (same branch main, event=schedule, completed only):

Date (UTC)	Run	Conclusion	Note
2026-06-23 18:40	28048657152	cancelled	the run that cancelled ours
2026-06-23 18:10	28046873327	cancelled	← this run
2026-06-23 12:55	28027779773	✅ success
2026-06-23 07:13	28008999561	cancelled
2026-06-23 00:39	27993948299	cancelled

Cancellation is the dominant outcome for this scheduled workflow, consistent with chronic concurrency cancellation rather than per-test breakage.

Failure Cluster

Run-level concurrency cancellation (job never started; cancel-in-progress superseded by a newer scheduled run).

Facts (verified observations)

• [FACT] Job 83027276135 conclusion = cancelled, steps: [], runner_name empty, started 18:11:00Z, completed 18:40:51Z (GitHub API job object).
• [FACT] Run 28046873327 conclusion = cancelled (run-level), created 18:10:06Z.
• [FACT] The next scheduled run 28048657152 was created 18:40:50Z — the same second (±1s) our job was cancelled (18:40:51Z).
• [FACT] Both runs are event=schedule on main, so they share concurrency group pr-test-amd-rocm720-refs/heads/main with cancel-in-progress: true (.github/workflows/pr-test-amd-rocm720.yml:122-123).
• [FACT] The only failure jobs (pr-test-amd-rocm720-finish and the extra finish) fail solely on step "Check all dependent job statuses" — a cascade from the cancelled dependents, not a test.
• [FACT] The job log is no longer retrievable (BlobNotFound), so no stack trace or [CI-AITER-CHECK] data exists.

Hypothesised Causes

• [HIGH] The run was cancelled by GitHub Actions concurrency, not by any failing test.
  • Supporting: concurrency config at .github/workflows/pr-test-amd-rocm720.yml:122-123 sets group pr-test-amd-rocm720-refs/heads/main and cancel-in-progress: true for schedule events; the superseding run 28048657152 was created at the exact second (18:40:50Z) our job was cancelled (18:40:51Z); the target job recorded 0 steps (never executed).
  • Disconfirming: (none found) — no executed step, no test output, and no error log exist that could indicate a code-level failure.
  • How to verify: compare head_sha and concurrency group of runs 28046873327 and 28048657152 in the API (both schedule/main → same group); inspect run 28048657152's start time vs this run's cancel time.

No code-level hypothesis applies: with 0 steps executed there is nothing in commit ed26a109 to implicate.

In-flight Fix Check

Not applicable — there is no test failure or regression to fix. This is expected concurrency behavior. (No PR search performed, as there is no failing test name/error/library to key on.)

Suggested Triage Steps

• No action needed for code: re-running the test of interest is only possible via a run that actually executes. If signal is required for this SHA, re-run run 28046873327 (it will get a fresh concurrency context) or wait for the next non-cancelled scheduled run.
• If frequent cancellations are undesirable for the daily 30 17 cron, consider whether the overlapping 0 */6 cron and back-to-back schedule triggers should share a concurrency group — but this is a workflow-design question for maintainers, not a fix for this job.

Status (factual)

• Persistence: Single run-level cancellation; the immediately prior completed run (28027779773, 12:55) succeeded. Cancellation is the common outcome for this scheduled workflow (4 of the last 5 scheduled runs cancelled).
• Scope: Run-wide — 40 of 49 jobs in run 28046873327 cancelled together; not specific to this job or any test.
• Blocked work: No real CI signal produced for SHA ed26a109 on ROCm 7.2 from this run, but the next run supersedes it.
• In-flight fix: N/A — no failing test.

stage-b-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang, 6) — failed step(s): N/A

Commit Info

• sglang: ed26a109
• aiter: N/A — the job log blob no longer exists (BlobNotFound from the logs API), so no [CI-AITER-CHECK] markers are retrievable. The job was cancelled before producing any recorded steps.

Failed Tests

Test File	Test Function	Error	Log
—	—	No test failure. Job conclusion = cancelled; zero steps recorded; log blob deleted.	Job

This is not a test failure. The job stage-b-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang, 6) was cancelled mid-flight by the CI orchestrator, not by any failing assertion.

Failure Summary

The job started at 18:11:00Z and was cancelled at 18:40:52Z (conclusion: cancelled, no steps recorded). It was one of 40 jobs in run 28046873327 that were all cancelled at the identical instant (18:40:51–52Z) — 7 jobs had already succeeded, and the only 2 failure jobs are the downstream aggregation gates (pr-test-amd-rocm720-finish, call-pr-test-amd-extra-rocm720 / pr-test-amd-extra-finish) that report failure because their upstream jobs were cancelled. A synchronized mass cancellation like this is the signature of a workflow-level concurrency cancellation, not a code or test problem.

Regression Status

Infrastructure issue — concurrency-group cancellation. Not a code regression, not a flaky test.

Evidence chain (schedule runs of pr-test-amd-rocm720.yml on main, completed runs):

Run	Created	Conclusion	Note
28048657152	2026-06-23T18:40:50Z	cancelled	The superseding run — created at the exact moment ours was cancelled
28046873327	2026-06-23T18:10:06Z	cancelled	This run — cancelled 18:40:51-52Z
28027779773	2026-06-23T12:55:42Z	success	—
28008999561	2026-06-23T07:13:14Z	cancelled	superseded by next schedule run

Most schedule runs in the window show cancelled, consistent with overlapping cron schedules sharing one concurrency group.

Facts (verified observations)

• [FACT] Job conclusion = cancelled, status = completed, zero steps recorded (GitHub API /jobs/83027276137). Ran 18:11:00Z → 18:40:52Z.
• [FACT] Run 28046873327: 49 jobs total → 7 success, 40 cancelled, 2 failure. All 40 cancelled jobs completed at 18:40:51–52Z (single synchronized timestamp).
• [FACT] The 2 failure jobs are both *-finish aggregation gates that fail because upstream jobs were cancelled — not independent test failures.
• [FACT] The next schedule run 28048657152 was created at 18:40:50Z, one second before the cancellation timestamp of this run.
• [FACT] The job log blob is gone: logs API returns BlobNotFound, so no test/aiter detail is recoverable.
• [FACT] .github/workflows/pr-test-amd-rocm720.yml:122-123: for schedule events the concurrency group resolves to pr-test-amd-rocm720-refs/heads/main with cancel-in-progress: true.

Failure Cluster

Concurrency-group cancellation by superseding scheduled run (no test failure)

Hypothesised Causes

• [HIGH] This run was cancelled by the GitHub Actions concurrency group when the next scheduled run started.
  • Supporting: workflow concurrency group for schedule is pr-test-amd-rocm720-refs/heads/main with cancel-in-progress: true (.github/workflows/pr-test-amd-rocm720.yml:122-123); the superseding run 28048657152 was created at 18:40:50Z and all 40 in-flight jobs of this run were cancelled at 18:40:51–52Z; 7 jobs that finished earlier succeeded; no failing test step exists.
  • Disconfirming: (none found) — the timing match is exact and the mechanism is explicit in the workflow config.
  • How to verify: compare created_at of run 28048657152 (18:40:50Z) with the cancellation timestamps of run 28046873327's jobs (18:40:51-52Z) — already confirmed via API.

In-flight Fix Check

Not applicable — there is no test/code defect to fix. The cancellation is expected behavior of the concurrency config given overlapping cron schedules (30 17 * * * daily + 0 */6 * * * every 6h) sharing one concurrency group on refs/heads/main.

Suggested Triage Steps

• No code action required for ed26a109; this job carries no test signal.
• To recover real signal for this shard, look at the latest completed, non-cancelled schedule run — e.g. 28027779773 (success, 2026-06-23 12:55Z) — rather than this cancelled run.
• If frequent self-cancellation of scheduled runs is undesirable, the maintainer could consider giving overlapping schedule triggers distinct concurrency groups (e.g. keyed on github.run_id or the cron expression) so a new schedule run does not cancel an in-progress one. This is an optional CI-orchestration improvement, not a directive.

Status

• Persistence: Single cancelled run; this shard had a prior success at 2026-06-23 12:55Z (28027779773).
• Scope: Orchestration-level — all 40 in-flight jobs of run 28046873327 cancelled together; no individual test affected.
• Blocked work: None — no genuine failure to investigate.
• In-flight fix: N/A (no defect).

stage-b-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang, 3) — failed step(s): N/A

Commit Info

• sglang: ed26a109
• aiter: N/A — job executed 0 steps (cancelled before amd_ci_install_dependency.sh ran; no [CI-AITER-CHECK] markers; log blob expired)

Failed Tests

Test File	Test Function	Error	Log
N/A	N/A	Job cancelled (not failed) — superseded by a newer scheduled run via concurrency cancel-in-progress. Zero steps executed; no test ran.	Job

Failure Summary

The job stage-b-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang, 3) did not fail a test — its conclusion is cancelled. The parent run 28046873327 (event: schedule, created 18:10:06 UTC) was cancelled in its entirety (40 cancelled / 7 success / 2 failure) when a newer scheduled run 28048657152 (created 18:40:50 UTC) entered the shared concurrency group. This job recorded 0 steps, so no test code was ever exercised.

Regression Status

Infrastructure issue (orchestration cancellation) — not a test regression.

Recent scheduled runs of pr-test-amd-rocm720.yml (branch main, event schedule, completed only):

Date (UTC)	Run	Conclusion	Note
Jun 23 18:40	28048657152	cancelled	superseded in turn
Jun 23 18:10	28046873327	cancelled	this run — superseded by 18:40 run
Jun 23 12:55	28027779773	success	—
Jun 23 07:13	28008999561	cancelled	—
Jun 23 00:39	27993948299	cancelled	—

Cancellation is a recurring orchestration pattern on this workflow (multiple scheduled runs land close together — the 0 */6 * * * 6h cron plus the 30 17 * * * daily cron — and newer runs cancel in-progress ones in the same github.ref concurrency group). It is not specific to this job or this commit.

Failure Cluster

Scheduled run cancelled by concurrency cancel-in-progress (superseded by newer schedule run) — orchestration, not a test failure.

Facts (verified observations)

• Job 83027276143 conclusion = cancelled, started_at = 2026-06-23T18:11:00Z, completed_at = 2026-06-23T18:40:51Z, steps count = 0 (GitHub API job object).
• Parent run 28046873327: event = schedule, conclusion = cancelled; job breakdown across 49 jobs = {success: 7, cancelled: 40, failure: 2}.
• The 2 failure jobs are aggregation gates (pr-test-amd-rocm720-finish and call-pr-test-amd-extra-rocm720 / pr-test-amd-extra-finish 83033310987) — expected cascade when upstream jobs are cancelled.
• Superseding run 28048657152 was created at 18:40:50Z; this job cancelled at 18:40:51Z (1s later).
• Workflow concurrency (pr-test-amd-rocm720.yml:117-123): group: pr-test-amd-rocm720-${{ ... || github.ref }} and cancel-in-progress: ${{ !inputs.run_all_tests && github.event_name != 'workflow_call' && github.event_name != 'workflow_dispatch' }} → evaluates true for schedule events. Both runs share group pr-test-amd-rocm720-refs/heads/main.
• Log blob unavailable (BlobNotFound, expired) — consistent with a cancelled run that produced no step logs.

Hypothesised Causes

• [FACT] The job was cancelled by GitHub Actions concurrency, not by any test failure. The newer schedule run 28048657152 entered the same concurrency group (pr-test-amd-rocm720-refs/heads/main) with cancel-in-progress: true, cancelling the in-progress run 28046873327 one second after the newer run was queued.
  • Supporting: conclusion cancelled; 0 steps executed; 1-second gap between superseding run creation (18:40:50) and this job's cancellation (18:40:51); workflow concurrency config confirms cancel-in-progress is active for schedule.
  • Disconfirming: (none found) — no test ever ran, so no code-level failure is possible.
  • How to verify: re-run the workflow at a time with no overlapping schedule run, or inspect that the job log (if retained) shows only The operation was canceled.

In-flight Fix Check

Not applicable — this is not a test/code failure. No fix PR is warranted. (No keyword search run: there is no failing test or library to match.)

Suggested Triage Steps

• No action needed for code correctness — no test failed and the cited commit ed26a109 was never exercised by this job.
• If repeated schedule-run cancellations are reducing CI signal coverage, consider whether the overlapping 0 */6 * * * and 30 17 * * * crons plus close-spaced schedule triggers should share a cancel-in-progress group — but this is a CI-config policy decision, not a regression.
• For the actual ROCm 7.2 stage-b signal on this commit, look at the most recent completed (non-cancelled) scheduled run instead, e.g. 28027779773 (Jun 23 12:55, success).

Status

• Persistence: Single cancelled run; cancellation is a recurring orchestration pattern (several recent schedule runs also cancelled), not a persistent test failure.
• Scope: Entire run 28046873327 cancelled (40 jobs) by concurrency supersession — workflow-wide, not job-specific.
• Blocked work: No test signal lost permanently — a later/earlier completed run provides coverage for nearby commits.
• In-flight fix: N/A — not a code defect.

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-06-24 19:29 UTC)

[amd-bot]

pr-test-amd-rocm720.yml — Per-Job Analysis (continued 5/12)

stage-b-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang, 5) — failed step(s): N/A

Commit Info

• sglang: ed26a109 — "[Fix] Return streaming logprobs when reasoning/tool parser is active (#28601)"
• aiter: N/A — the job log blob has expired (BlobNotFound from the logs API), so [CI-AITER-CHECK] markers are not recoverable. Not determinable.

Failed Tests

Test File	Test Function	Error	Log
—	—	No test failure — job was cancelled mid-run	Job

This is not a test failure. The job stage-b-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang, 5) was cancelled (not failure).

Failure Summary

The job started at 18:11:00Z and was cancelled at 18:40:52Z, ~30 minutes in, before any test result was reported. It was not cancelled in isolation: all 49 jobs in run 28046873327 were cancelled within the same 1-second window (18:40:51–18:40:52Z), and the whole run's conclusion is cancelled. This is a workflow-level cancellation, not a per-test failure.

Regression Status

Infrastructure / concurrency cancellation — not a code regression. No test-file-level history is applicable because no test ran to completion or reported a status.

Failure Cluster

Run cancelled by concurrency (superseded by next scheduled run)

Facts (verified observations, no interpretation)

• Target job 83027276156: status=completed, conclusion=cancelled, started 18:11:00Z, completed 18:40:52Z. Steps list is empty in the API response.
• Run 28046873327 (PR Test ROCm 7.2 (AMD), event=schedule, head_sha ed26a109): conclusion=cancelled.
• Of 49 jobs: 7 success, 40 cancelled, 2 failure. The 2 failures are the finalizer/gate jobs (call-pr-test-amd-extra-rocm720 / pr-test-amd-extra-finish 83033310987 and pr-test-amd-rocm720-finish 83040560486) — they report failure because their upstream jobs were cancelled, not because of an independent fault.
• All 40 cancelled jobs completed at 18:40:51Z–18:40:52Z (simultaneous), the signature of an external cancel rather than per-job timeouts (which would be staggered).
• A newer scheduled run 28048657152 (event=schedule, head_sha e0dc8b71) started at 18:40:50Z — i.e. 1–2 seconds before our run's jobs were cancelled. It too ended cancelled (superseded in turn).
• Workflow concurrency config (.github/workflows/pr-test-amd-rocm720.yml:117-123): for schedule events the group resolves to pr-test-amd-rocm720-${{ github.ref }} (= pr-test-amd-rocm720-refs/heads/main) with cancel-in-progress: true (true whenever not run_all_tests/workflow_call/workflow_dispatch).
• The job log itself is no longer retrievable: https://api.github.com/.../jobs/83027276156/logs returns BlobNotFound.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [HIGH] The run was cancelled by GitHub Actions concurrency, because a newer scheduled run started while this one was in progress and they share the concurrency group pr-test-amd-rocm720-refs/heads/main with cancel-in-progress: true.
  • Supporting: newer scheduled run 28048657152 started 18:40:50Z, exactly when all 40 jobs of this run were cancelled (18:40:51–52Z); workflow concurrency block (lines 117-123) sets cancel-in-progress: true for schedule events and the group key is github.ref (identical for both scheduled runs on main); simultaneous cancel timestamps are the concurrency-cancel signature.
  • Disconfirming: (none found) — the log blob has expired so the in-job cancellation message could not be read directly, but every queryable signal is consistent with a concurrency cancel.
  • How to verify: open the run 28046873327 page — the banner reads "canceled because a higher priority run / concurrency"; confirm run 28048657152 is the superseding run on the same github.ref.
• [LOW] Manual cancellation by a maintainer.
  • Supporting: would also produce simultaneous cancel timestamps.
  • Disconfirming: the precise 1-second alignment with the start of a newer scheduled run is far better explained by concurrency than coincidental manual action; manual cancels rarely coincide to the second with a new run's start.
  • How to verify: check the run's activity/audit log for a user-initiated cancel.

In-flight Fix Check

Not applicable — this is a concurrency/scheduling artifact, not a defect requiring a code fix. No PR search performed (no failing test or library to key on).

Suggested Triage Steps

• No code action needed. This cancellation is expected behavior: two scheduled runs of pr-test-amd-rocm720.yml overlapped on main and the older was auto-cancelled. The two cron entries (30 17 * * * and 0 */6 * * *) plus GitHub's schedule-dispatch jitter make overlapping scheduled runs likely.
• To confirm: load 28046873327 and verify the cancellation banner attributes it to a higher-priority/superseding run.
• If recurrent overlapping scheduled cancellations are undesirable, consider (a maintainer decision, not a directive) tightening the cron cadence or scoping the concurrency group for scheduled runs differently so legitimate scheduled runs don't cancel each other.
• For actual CI signal on ed26a109, look at the next completed scheduled run rather than this cancelled one.

Status (factual, no priority assignment)

• Persistence: single-run event; run 28046873327 cancelled at 18:40:52Z. The immediately newer run 28048657152 was also cancelled (superseded in turn); the last clean signal was the success run 28027779773 at 12:55Z.
• Scope: entire run (40 jobs cancelled + 2 finalizer jobs reporting failure as a consequence). Not isolated to this shard.
• Blocked work: none from a code standpoint — no regression introduced; this run simply produced no usable CI signal.
• In-flight fix: N/A — no defect to fix.

stage-b-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang, 2) — failed step(s): N/A

Commit Info

• sglang: ed26a109
• aiter: N/A — the job log blob no longer exists (BlobNotFound) and the job recorded zero steps, so no [CI-AITER-CHECK] markers are available.

Failed Tests

Test File	Test Function	Error	Log
N/A	N/A	Job cancelled before any step executed — no test ran	Job

This was not a test failure. The job's conclusion is cancelled, it has no recorded steps, and the run-level conclusion is also cancelled.

Failure Summary

Job stage-b-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang, 2) was cancelled, not failed. It started at 18:11:00Z and was terminated at 18:40:51Z with no steps having run to completion. It was part of a mass cancellation: 40 of 49 jobs in run 28046873327 were cancelled at the same instant (18:40:51–52Z), and the two *-finish aggregator jobs reported failure as a downstream consequence of that cancellation.

Regression Status

Infrastructure issue (concurrency supersession) — not a code regression.

Recent pr-test-amd-rocm720.yml scheduled runs on main (completed runs only):

Created (UTC)	Run	Conclusion	Note
2026-06-23 18:40:50	28048657152	cancelled	The superseding run (then itself superseded)
2026-06-23 18:10:06	28046873327	cancelled	This run
2026-06-23 12:55:42	28027779773	✅ success	—
2026-06-23 07:13:14	28008999561	cancelled	—
2026-06-23 00:39:01	27993948299	cancelled	—

Most scheduled runs end in cancelled because each new schedule supersedes the in-progress one — this is the expected steady-state pattern for this workflow, not a per-test regression.

Failure Cluster

Run cancelled by concurrency cancel-in-progress (scheduled run superseded mid-flight) — no test executed.

Facts (verified observations)

• [FACT] Job 83027276157 conclusion is cancelled; the job API returns an empty steps array (no step ever reached a terminal conclusion).
• [FACT] The job log is unretrievable: GET .../jobs/83027276157/logs returns <Error><Code>BlobNotFound</Code></Error>.
• [FACT] Run 28046873327 conclusion is cancelled (event=schedule, branch=main, run_attempt=1).
• [FACT] Job tally for the run: 7 success, 40 cancelled, 2 failure. The 2 failures are aggregator jobs only: call-pr-test-amd-extra-rocm720 / pr-test-amd-extra-finish (83033310987) and pr-test-amd-rocm720-finish (83040560486).
• [FACT] 40 jobs were cancelled at the same timestamp 2026-06-23T18:40:51Z (a handful at 18:40:52Z), indicating a single run-level cancellation event rather than independent per-test failures.
• [FACT] A newer scheduled run 28048657152 was created at 2026-06-23T18:40:50Z — one second before the mass cancellation of this run.
• [FACT] .github/workflows/pr-test-amd-rocm720.yml:122-123: for a schedule event, cancel-in-progress evaluates to true and the concurrency group resolves to pr-test-amd-rocm720-${{ github.ref }} (= refs/heads/main), shared by all scheduled runs on main.

Hypothesised Causes

• [HIGH] The run (and therefore this job) was cancelled by GitHub Actions concurrency cancel-in-progress when the next scheduled run superseded it.
  • Supporting: newer scheduled run 28048657152 created 18:40:50Z; this run's jobs all cancelled 18:40:51Z; the workflow's concurrency group for schedule is keyed on github.ref with cancel-in-progress: true (pr-test-amd-rocm720.yml:122-123). Three independent evidence chains converge (timing + config + group key).
  • Disconfirming: (none found) — no job in the run shows a test-level failure; the only failures are post-cancellation aggregator jobs.
  • How to verify: re-query actions/runs/28048657152 creation time vs this run's job cancellation timestamps (already done); inspect the superseding run's own conclusion (also cancelled, by the run after it).

In-flight Fix Check

Not applicable — there is no code defect to fix. This is expected concurrency behavior for the scheduled workflow.

• Search run: curl ".../search/issues?q=repo:sgl-project/sglang+is:pr+is:open+pr-test-amd-rocm720+concurrency" → not run, because no fix is warranted for a benign supersession.

Suggested Triage Steps

• No action needed for code. To confirm this is benign, note that prior scheduled run 28027779773 (last not-superseded run) completed success.
• If these cancelled scheduled runs are creating noise in the failure tracker, consider filtering out runs whose conclusion is cancelled with zero failed test jobs (only aggregator *-finish jobs failed) at the harness level — they carry no test signal.
• To get a clean ROCm 7.2 signal at this SHA, trigger a workflow_dispatch run (which gets a unique concurrency group per pr-test-amd-rocm720.yml:122 and won't be superseded by the 6h cron).

Status

• Persistence: Single cancelled run; not a recurring test failure. The cancelled conclusion is the normal outcome for most scheduled runs of this workflow (each cron supersedes the previous in-progress one).
• Scope: Entire run 28046873327 (40 jobs) cancelled simultaneously; this target job is one of them. No test-level failure in any job.
• Blocked work: None — no genuine CI breakage. This run produced no usable ROCm 7.2 test signal for ed26a109; the prior run 28027779773 passed.
• In-flight fix: N/A — benign concurrency supersession, no fix required.

multimodal-gen-test-1-gpu-amd-rocm720 (linux-mi325-1gpu-sglang, 1) — failed step(s): N/A

Commit Info

• sglang: ed26a109
• aiter: N/A — job log blob has expired (GitHub returned BlobNotFound), and the [CI-AITER-CHECK] markers cannot be recovered.

Failed Tests

Test File	Test Function	Error	Log
—	—	No test failure — job was cancelled, not failed	Job

This job did not fail a test. Its conclusion is cancelled: it started at 2026-06-23T18:11:00Z, ran ~30 min, and was terminated at 2026-06-23T18:40:51Z with zero completed steps recorded.

Failure Summary

The job multimodal-gen-test-1-gpu-amd-rocm720 (linux-mi325-1gpu-sglang, 1) was cancelled, along with 39 other jobs in run 28046873327 (whole-run conclusion = cancelled). The cancellation timestamp (18:40:50–51Z) coincides to the second with the creation of the next scheduled run 28048657152 (created 2026-06-23T18:40:50Z), indicating a concurrency-group cancellation: the newer scheduled run superseded this in-progress one. The only two failure jobs in the run are the gate jobs pr-test-amd-rocm720-finish (83040560486) and call-pr-test-amd-extra-rocm720 / pr-test-amd-extra-finish (83033310987), which fail only because their "Check all dependent job statuses" step sees the cancelled upstreams — they are not independent failures.

Regression Status

Infrastructure issue (concurrency cancellation) — not a test regression.

Recent history of multimodal-gen-test-1-gpu-amd-rocm720 on pr-test-amd-rocm720.yml (branch main, event schedule, completed runs only):

Date	Run	Job Status	Note
Jun 23 18:10	28046873327	⏹️ Cancelled	This run — superseded by concurrency
Jun 23 18:40	28048657152	⏭️ Skipped	Job not selected (change-detection gate)
Jun 23 12:55	28027779773	⏭️ Skipped	Run succeeded overall
Jun 23 07:13	28008999561	⏭️ Skipped	—
Jun 23 00:39	27993948299	⏭️ Skipped	—

This job is gated (skipped in most scheduled runs) and actually executed only in this run, where it was cancelled before any step completed. There is no recent run in which this job ran to a pass/fail conclusion, so no green/red baseline for the multimodal-gen test itself exists in the queryable window.

Facts (verified observations)

• Job conclusion: cancelled; started_at=2026-06-23T18:11:00Z, completed_at=2026-06-23T18:40:51Z; steps: [] (no step ran to completion). [verified via jobs API]
• Run 28046873327: 49 jobs → 7 success, 40 cancelled, 2 failure (both *-finish gate jobs). [verified]
• Both failure jobs failed only on step 2 "Check all dependent job statuses" — i.e. cascade from cancelled deps, not real failures. [verified]
• Next scheduled run 28048657152 created 2026-06-23T18:40:50Z — same second this run was cancelled; it is also cancelled. [verified]
• The two scheduled runs are only ~30 min apart (18:10 vs 18:40), and the workflow's recent schedule history shows a chronic pattern of cancelled runs (8 of the last 10 completed scheduled runs are cancelled). [verified via workflow runs API]
• Job log blob is expired: GitHub API returns BlobNotFound for the job logs. [verified]

Hypothesised Causes (with confidence)

• [HIGH] Concurrency-group cancellation — the in-progress run was auto-cancelled when a newer schedule run was created.
  • Supporting: cancellation timestamp (18:40:50–51Z) matches the next run's creation time to the second; 40 jobs cancelled simultaneously; whole-run conclusion cancelled; only gate jobs show failure (cascade).
  • Disconfirming: cannot 100% confirm the workflow's concurrency: config sets cancel-in-progress: true from this data alone (would need to read the workflow YAML), but the timing is otherwise unexplained.
  • How to verify: check the concurrency block in .github/workflows/pr-test-amd-rocm720.yml; confirm whether overlapping schedule triggers are intended (the 30-min gap between 18:10 and 18:40 runs is suspicious).
• [LOW] Manual/admin cancellation — possible but not supported by any evidence; the exact-second alignment with a new run creation points to automated concurrency cancellation instead.

In-flight Fix Check

• In-flight fix: ❌ none found — this is not a code defect; no fix PR is applicable. (No keyword search performed — there is no failing test/library to key on.)

Suggested Triage Steps

• Inspect the concurrency: group in .github/workflows/pr-test-amd-rocm720.yml — confirm cancel-in-progress behavior and whether the chronic cancelled-run pattern (8/10 recent scheduled runs cancelled) is expected or a misconfiguration causing wasted CI.
• Investigate why two schedule runs fired only ~30 min apart (18:10 and 18:40). If the cron is intended to be less frequent, a duplicate/overlapping trigger may be needlessly cancelling prior runs before they finish.
• To get real multimodal-gen-test signal, re-run this job on a commit where the change-detection gate selects it, and ensure no newer scheduled run cancels it mid-flight.

Status (factual)

• Persistence: Single-run event; job cancelled before any step completed. The broader workflow shows 8/10 recent completed scheduled runs ending in cancelled.
• Scope: Whole run 28046873327 cancelled (40 jobs); not specific to this job. No test-level failure.
• Blocked work: No multimodal-gen ROCm 7.2 test signal produced by this run; the chronic cancellation pattern likely suppresses signal across this scheduled workflow.
• In-flight fix: ❌ none found — not a code defect.

stage-b-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang, 8) — failed step(s): N/A

Commit Info

• sglang: ed26a109
• aiter: N/A (job log returned BlobNotFound — logs expired/unavailable for this cancelled job, so no [CI-AITER-CHECK] markers could be read)

Failed Tests

Test File	Test Function	Error	Log
— (none)	—	Job cancelled before any step ran — no test executed	Job

This was not a test failure. The job stage-b-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang, 8) has conclusion: cancelled with zero steps recorded (no setup, no test step ever ran). The job log API returns BlobNotFound.

Failure Summary

The entire run 28046873327 (PR Test ROCm 7.2 (AMD), event schedule) was cancelled: of 49 jobs, 7 succeeded, 40 were cancelled, and 2 *-finish aggregation jobs failed only because their "Check all dependent job statuses" step saw cancelled dependencies. The target job recorded no steps and ran no tests. [FACT]

Regression Status

Infrastructure issue (concurrency cancellation) — not a code regression and not a flaky test.

Recent schedule runs of pr-test-amd-rocm720.yml on main (completed only):

Date (UTC)	Run	Conclusion	Note
06-24 00:33	28066754977	queued	in-flight
06-23 18:40	28048657152	cancelled	superseding run that cancelled ours
06-23 18:10	28046873327	cancelled	this run
06-23 12:55	28027779773	success	clean run
06-23 07:13	28008999561	cancelled	—
06-23 00:39	27993948299	cancelled	—

Most scheduled runs on main end up cancelled — a recurring orchestration artifact, not a per-test regression.

Facts (verified observations)

• Job 83027276191: status=completed, conclusion=cancelled, started 2026-06-23T18:11:00Z, completed 2026-06-23T18:40:52Z, steps array empty. [FACT]
• Run 28046873327: event schedule, conclusion cancelled; job breakdown = 7 success / 40 cancelled / 2 failure. [FACT]
• The 2 failures are both aggregation jobs — call-pr-test-amd-extra-rocm720 / pr-test-amd-extra-finish and pr-test-amd-rocm720-finish — failing on step "Check all dependent job statuses" because dependencies were cancelled. [FACT]
• The next scheduled run 28048657152 (event schedule, same branch main) was created 2026-06-23T18:40:50Z — 2 seconds before our job's cancellation timestamp 18:40:52Z. [FACT]
• Workflow concurrency config (.github/workflows/pr-test-amd-rocm720.yml:122-123): group resolves to pr-test-amd-rocm720-${{ github.ref }} (= refs/heads/main) for schedule events, with cancel-in-progress: true for schedule events. [FACT]
• Job log unavailable: BlobNotFound from the logs endpoint. [FACT]

Hypothesised Causes

• [HIGH] The run was cancelled by GitHub Actions concurrency: a newer schedule run on the same refs/heads/main concurrency group superseded it.
  • Supporting: both runs are schedule on main → identical concurrency group pr-test-amd-rocm720-refs/heads/main; cancel-in-progress=true for schedule (workflow file line 123); the superseding run 28048657152 was created at 18:40:50, our job cancelled at 18:40:52; 40 sibling jobs cancelled simultaneously; the only non-cancel failures are dependency-status aggregators.
  • Disconfirming: (none found) — no step ran, no log exists, so there is no evidence of a code/test failure.
  • How to verify: compare created_at of 28048657152 vs the cancellation time (already done); inspect the concurrency group rendering for both runs.

No code-level regression hypothesis applies — the changed-code commit ed26a109 was never exercised by this job.

In-flight Fix Check

Not applicable — no code defect to fix. This is expected concurrency behavior (two overlapping schedule crons, 30 17 * * * and 0 */6 * * *, both on main cancelling each other when timing overlaps). No PR search warranted.

Suggested Triage Steps

• No action needed for code. Treat this job as a cancellation, not a failure. Re-running the workflow on the same SHA, or consulting the clean run 28027779773 (06-23 12:55, success), gives the real test signal.
• If these recurring schedule cancellations create noise in the tracker, consider (orchestration-side, human decision) giving the two scheduled crons distinct concurrency groups so they stop cancelling each other — analogous to the existing workflow_dispatch/run_all_tests unique-group handling at .github/workflows/pr-test-amd-rocm720.yml:122.

Status

• Persistence: One-off cancellation of this run; the broader pattern (most schedule runs on main ending cancelled) is recurring and benign.
• Scope: Whole run — 40 of 49 jobs cancelled by the same concurrency event; no test-level impact.
• Blocked work: None — the changed code was not tested here, but a green schedule run (28027779773) exists nearby.
• In-flight fix: N/A (not a defect).

stage-b-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang, 7) — failed step(s): N/A

Commit Info

• sglang: ed26a109
• aiter: N/A — the job was cancelled before producing any log (the log blob returns BlobNotFound), so no [CI-AITER-CHECK] markers are available.

Failed Tests

Test File	Test Function	Error	Log
N/A	N/A	Job was cancelled (not failed) by workflow concurrency; no tests ran to completion	Job

This is not a test failure. The job stage-b-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang, 7) has conclusion: cancelled with zero recorded steps, and its log blob does not exist (BlobNotFound).

Failure Summary

The entire scheduled run 28046873327 was cancelled (run conclusion: cancelled): of 49 jobs, 40 were cancelled, 7 success, and 2 failure — and those 2 "failures" are only the pr-test-amd-extra-finish / pr-test-amd-rocm720-finish gate jobs reacting to the cancellation. The target job ran from 18:11:01 to 18:40:52 UTC and was terminated mid-flight. No test file or test function failed.

Regression Status

Infrastructure issue (concurrency cancellation) — not a code regression and not a flaky test.

This is the normal behavior of the workflow's concurrency group, not a per-test problem. Recent event=schedule, branch=main runs of pr-test-amd-rocm720.yml:

Date (UTC)	Run	Status	Conclusion	Head SHA
2026-06-24 00:33	28066754977	queued	—	9ef18307
2026-06-23 18:40	28048657152	completed	❌ cancelled	e0dc8b71
2026-06-23 18:10	28046873327 (this run)	completed	❌ cancelled	ed26a109
2026-06-23 12:55	28027779773	completed	✅ success	0460f277
2026-06-23 07:13	28008999561	completed	❌ cancelled	c4376aaa
2026-06-23 00:39	27993948299	completed	❌ cancelled	ba9d5aed

Most scheduled runs end as cancelled because overlapping schedule triggers on refs/heads/main repeatedly supersede one another.

Failure Cluster

Run cancelled by concurrency supersession (no test failure)

Facts (verified observations, no interpretation)

• [FACT] Job 83027276238 conclusion: cancelled, status: completed, zero steps recorded; started 2026-06-23T18:11:01Z, completed 18:40:52Z.
• [FACT] The job log blob does not exist — the API returns <Code>BlobNotFound</Code>. No test output was ever persisted.
• [FACT] Run 28046873327 overall conclusion: cancelled; 49 jobs = 40 cancelled / 7 success / 2 failure (the 2 failures are *-finish gate jobs).
• [FACT] The next scheduled run 28048657152 was created at 18:40:50Z — within ~2 s of this job's cancellation at 18:40:52Z.
• [FACT] pr-test-amd-rocm720.yml:122-123 sets the concurrency group to github.ref (refs/heads/main) for schedule events, with cancel-in-progress: true (true because run_all_tests is unset and the event is schedule, not workflow_call/workflow_dispatch).

Hypothesised Causes (with confidence, with disconfirming evidence)

• [FACT] The run was cancelled by GitHub Actions concurrency, not by any test error.
  • Supporting: pr-test-amd-rocm720.yml:122 groups schedule runs by refs/heads/main with cancel-in-progress: true (line 123); the newer schedule run 28048657152 started at 18:40:50Z and this job was cancelled at 18:40:52Z; the cancelled job has no steps and no log blob.
  • Disconfirming: (none found) — every observation is consistent with concurrency cancellation; there is no error message, stack trace, or test failure anywhere because the log was never written.
  • How to verify: compare created_at of the successor run vs completed_at of the cancelled job (already done — 2 s apart); inspect the workflow concurrency block (already done).

In-flight Fix Check

Not applicable — there is no test failure to fix. This is expected concurrency behavior. (No PR search performed because nothing is broken.)

Suggested Triage Steps

• No code action required. Confirm signal by inspecting the most recent completed, non-cancelled scheduled run — 28027779773 (✅ success, 2026-06-23 12:55) — for the true health of stage-b-test-1-gpu-small-amd-rocm720 shard 7.
• If repeated cancellation of scheduled runs is undesirable noise, consider whether the cancel-in-progress policy for schedule events on refs/heads/main should be relaxed (e.g. let scheduled runs queue rather than cancel) — a workflow-design decision for the maintainer, not a fix for this job.

Status (factual, no priority assignment)

• Persistence: One-off cancellation of run 28046873327; the immediately preceding completed run 28027779773 succeeded.
• Scope: Whole run (40 of 49 jobs cancelled), caused by a single concurrency event — not specific to this job or shard.
• Blocked work: None — this is not a test regression. The cancelled job carries no test signal.
• In-flight fix: N/A — no failure to fix.

stage-b-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang, 9) — failed step(s): N/A

Commit Info

• sglang: ed26a109
• aiter: N/A — the job log blob no longer exists (BlobNotFound) and the job ran zero steps, so no [CI-AITER-CHECK] markers are available.

Failed Tests

Test File	Test Function	Error	Log
N/A	N/A	No test ran — job was cancelled before executing any step	Job

This is not a test failure. The job stage-b-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang, 9) has conclusion: cancelled (not failure) and recorded zero steps — not even "Set up job". It was queued/assigned to a runner at 18:11:01Z and cancelled at 18:40:52Z without ever running a test.

Failure Summary

The job did not fail on any test; it was cancelled by GitHub Actions concurrency before execution. The entire run 28046873327 (PR Test ROCm 7.2 (AMD), event=schedule) has conclusion: cancelled: 40 jobs cancelled, 7 succeeded, and the 2 *-finish aggregator jobs "failed" only because their Check all dependent job statuses step saw cancelled dependents. The two reported "failures" are aggregator artifacts, not real test failures.

Regression Status

Infrastructure issue (orchestration / concurrency cancellation) — not a code regression. No test executed, so there is nothing to bisect.

Recent pr-test-amd-rocm720.yml scheduled runs on main (completed only):

Date	Run	Status	Note
Jun 23 18:40	28048657152	❌ cancelled	superseded by next run
Jun 23 18:10	28046873327	❌ cancelled	this run — superseded by 28048657152
Jun 23 12:55	28027779773	✅ success	completed normally
Jun 23 07:13	28008999561	❌ cancelled	superseded
Jun 23 00:39	27993948299	❌ cancelled	superseded

Most scheduled runs end as cancelled because each new scheduled run cancels the in-progress one (see Facts) — this is a chronic orchestration pattern, not a per-test regression.

Failure Cluster

"Scheduled run cancelled by concurrency cancel-in-progress (no test executed)"

Facts (verified observations)

• Job 83027276210: status=completed, conclusion=cancelled, steps=[] (zero steps), started 2026-06-23T18:11:01Z, completed 2026-06-23T18:40:52Z. [FACT]
• Log fetch returned BlobNotFound — logs were never produced / have expired (consistent with a job that ran no steps). [FACT]
• Run 28046873327 overall conclusion=cancelled: of 49 jobs → 7 success, 40 cancelled (39 with zero steps), 2 *-finish failures. [FACT]
• The 2 "failure" jobs are aggregators: pr-test-amd-rocm720-finish failed only at step 2 Check all dependent job statuses (steps 1 and 3 succeeded). [FACT]
• A newer scheduled run 28048657152 (event=schedule, head_sha e0dc8b71) was created at 2026-06-23T18:40:50Z — within ~16s of our run's jobs being cancelled (latest cancel completed 18:41:06Z). [FACT]
• Workflow concurrency config (pr-test-amd-rocm720.yml:122-123): group pr-test-amd-rocm720-${{ ... || github.ref }} (= ...refs/heads/main for schedule runs) with cancel-in-progress: true for schedule events. [FACT]

Hypothesised Causes (with confidence)

• [HIGH] Our run was cancelled by GitHub Actions concurrency cancel-in-progress when the next scheduled run 28048657152 entered the same group (pr-test-amd-rocm720-refs/heads/main).
  • Supporting: workflow sets cancel-in-progress: true for event=schedule with group keyed on github.ref; the superseding schedule run was created at 18:40:50Z, matching the cancellation timestamp (18:40:52Z / 18:41:06Z) to the second. Multiple consecutive scheduled runs show the same cancelled pattern.
  • Disconfirming: (none found) — timing and config converge; no test/build error appears anywhere in the run.
  • How to verify: compare creation time of the next same-group run vs the cancel time of any run in this series (already confirmed here).

In-flight Fix Check

Not applicable — there is no test/code failure to fix. This is expected orchestration behavior (a later scheduled run preempting an earlier overlapping one). If the overlap itself is considered undesirable, that is a workflow-scheduling tuning question, not a bug fix.

• In-flight fix: ❌ none needed — no code/test failure

Suggested Triage Steps

• No code action required. Treat this job result as a non-signal: cancelled before any step ran.
• If a real CI signal is needed for sglang ed26a109 on this workflow, look at a completed run instead — e.g. 28027779773 (✅ success) — or re-trigger via workflow_dispatch (which gets a unique concurrency group and won't be cancelled).
• If frequent cancellation of scheduled runs is reducing AMD CI coverage, consider whether the overlapping 30 17 * * * and 0 */6 * * * crons combined with cancel-in-progress on the shared refs/heads/main group is intended — this is a scheduling-policy discussion for the maintainers, not a per-commit triage step.

Status (factual)

• Persistence: Cancellation (not failure). Most recent scheduled runs on main end as cancelled due to concurrency preemption; one clean success at Jun 23 12:55 (28027779773).
• Scope: Run-wide orchestration event — 40 of 49 jobs in run 28046873327 cancelled, including this job. Not specific to stage-b-test-1-gpu-small or the mi325 runner.
• Blocked work: None — superseded by a newer scheduled run that proceeded in its place.
• In-flight fix: ❌ none needed — no code/test failure.

stage-b-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang, 4) — failed step(s): N/A

Commit Info

• sglang: ed26a109
• aiter: N/A (job was cancelled before any step ran; no [CI-AITER-CHECK] markers — the job log blob does not exist)

Failed Tests

Test File	Test Function	Error	Log
N/A — no test ran	N/A	Job cancelled by concurrency, 0 steps executed	Job

This is not a test failure. The job conclusion is cancelled (not failure), it recorded 0 steps, and its log blob returns BlobNotFound. The only two failure jobs in the run are the *-finish aggregator jobs (pr-test-amd-rocm720-finish, pr-test-amd-extra-finish), which fail by design when upstream jobs are cancelled.

Failure Summary

The job stage-b-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang, 4) was cancelled mid-run by GitHub Actions concurrency control. Run 28046873327 (the daily 30 17 * * * cron, delayed to 18:10:06) was superseded by run 28048657152 (the 6-hourly 0 */6 * * * dsv4 cron, delayed to 18:40:50). Both ran on refs/heads/main and share the concurrency group pr-test-amd-rocm720-refs/heads/main with cancel-in-progress: true, so the newer run cancelled all 40 in-progress jobs of the older run.

Regression Status

Infrastructure issue (CI orchestration / concurrency self-cancellation) — not a code regression. No test was executed, so there is no test-file-level regression to track.

Recent pr-test-amd-rocm720.yml scheduled runs on main (completed only):

Date (UTC)	Run	Event/Cron	Conclusion
Jun 23 18:40	28048657152	schedule (6h dsv4)	cancelled
Jun 23 18:10	28046873327	schedule (daily, this run)	cancelled
Jun 23 12:55	28027779773	schedule	✅ success
Jun 23 07:13	28008999561	schedule	cancelled
Jun 23 00:39	27993948299	schedule	cancelled
Jun 22 19:05	27977093213	schedule	cancelled
Jun 22 18:45	27975943807	schedule	cancelled

Cancellations recur frequently among scheduled runs — a structural symptom of multiple cron schedules colliding on the same github.ref concurrency group.

Failure Cluster

Scheduled run self-cancellation via shared concurrency group (cancel-in-progress on github.ref)

Facts (verified observations)

• [FACT] Job 83027276206 conclusion = cancelled, started_at 18:11:01Z, completed_at 18:40:52Z, 0 steps recorded (GitHub API job object).
• [FACT] Log blob returns <Error><Code>BlobNotFound</Code> — no execution log exists for the cancelled job.
• [FACT] Run 28046873327 created 18:10:06Z; run 28048657152 created 18:40:50Z — our job was cancelled at 18:40:52Z, 2 seconds after the newer run was created.
• [FACT] Both runs are event=schedule on head_branch=main.
• [FACT] Run 28048657152 is the 6h 0 */6 * * * cron: call-gate, check-changes, and all stage-*/sgl-kernel/jit-kernel jobs are skipped (guarded by if: github.event.schedule != '0 */6 * * *'); only dsv4-flash-fp4-fp8 and dsv4-pro-fp4 ran.
• [FACT] pr-test-amd-rocm720.yml:122-123 — concurrency group: pr-test-amd-rocm720-…|| inputs.pr_head_sha || inputs.ref || github.ref resolves to pr-test-amd-rocm720-refs/heads/main for schedule events; cancel-in-progress evaluates to true for schedule (!inputs.run_all_tests && event != 'workflow_call' && event != 'workflow_dispatch').
• [FACT] Run-level breakdown: 7 success, 40 cancelled, 2 failure (both *-finish aggregators).

Hypothesised Causes

• [FACT] Concurrency self-cancellation. The daily 30 17 * * * full run and the 6-hourly 0 */6 * * * dsv4 run both key their concurrency group on github.ref (= refs/heads/main) with cancel-in-progress: true. The two crons fire ~30 min apart (and GitHub delays bunch them closer), so the later 6h run cancels the in-progress daily run. This is directly observable from the timing (18:40:50 create → 18:40:52 cancel) and the workflow's concurrency block.
  • Supporting: 2-second create→cancel timing; identical github.ref; cancel-in-progress: true for schedule; recurring cancellation pattern across the run history.
  • Disconfirming: (none found) — no step ever executed, ruling out any test/code cause.
  • How to verify: give each schedule cron a distinct concurrency group (e.g. include github.run_id or the cron string), or set cancel-in-progress: false for schedule events; then confirm the daily run survives the next 6h cron.

In-flight Fix Check

• In-flight fix: ⚠️ partial — #28987 "[CI] Fix AMD extra scheduled runs cancelling each other" (open since 2026-06-23). It fixes the identical bug class (scheduled runs sharing a concurrency group omitting github.run_id, with cancel-in-progress cancelling each other) but only modifies pr-test-amd-extra.yml — it does NOT touch pr-test-amd-rocm720.yml. The ROCm 7.2 workflow analyzed here is still affected.
  • API: search/issues?q=repo:sgl-project/sglang+is:pr+is:open+cancel-in-progress+schedule → matched #28987.
• For pr-test-amd-rocm720.yml specifically: ❌ none found — needs its own fix (or extend #28987's approach to this workflow).

Suggested Triage Steps

• No code investigation needed — this is a CI-config issue, not a test/regression. Do not bisect ed26a109 or chase aiter.
• Extend the #28987 approach to pr-test-amd-rocm720.yml:117-123: make the schedule concurrency group unique per cron (include github.run_id or the github.event.schedule string) so the 6h dsv4 cron no longer cancels the daily full run.
• Alternatively, set cancel-in-progress: false for schedule events so an in-progress full run is allowed to complete.
• Optionally re-trigger the daily run (workflow_dispatch, which already gets a unique full-{run_id} group) to obtain a clean ROCm 7.2 signal for ed26a109.

Status

• Persistence: Recurring — the majority of pr-test-amd-rocm720.yml scheduled runs on main in the queryable window conclude cancelled; the daily 17:30 full run is reliably superseded by the 18:00 6h cron.
• Scope: Affects the whole run (40 jobs cancelled) whenever two schedule crons overlap; structural to pr-test-amd-rocm720.yml concurrency config. The sibling pr-test-amd-extra.yml has the same class of bug.
• Blocked work: The daily full ROCm 7.2 PR-test signal is lost on days the 6h cron collides with it (no stage-b/-a/-c coverage produced).
• In-flight fix: ⚠️ #28987 fixes the sibling pr-test-amd-extra.yml only; pr-test-amd-rocm720.yml is not yet covered.

stage-b-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang, 11) — failed step(s): N/A

Commit Info

• sglang: ed26a109 — [Fix] Return streaming logprobs when reasoning/tool parser is active (#28601)
• aiter: N/A (job was cancelled before any setup/test step ran; no [CI-AITER-CHECK] markers — the log blob does not exist)

Failed Tests

Test File	Test Function	Error	Log
N/A — no test ran	N/A	Job cancelled by concurrency control before any step executed	Job

This was not a test failure. The job stage-b-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang, 11) has conclusion: cancelled, recorded zero steps, and its log blob returns BlobNotFound — meaning it never produced output.

Failure Summary

The job did not fail on any test. It was cancelled at 2026-06-23T18:40:52Z, together with 39 other in-flight jobs of run 28046873327 that all cancelled simultaneously at 18:40:51-52Z. The mass simultaneous cancellation is the signature of GitHub Actions concurrency-group preemption, not a fast-fail cascade (a cascade would show a failure job first, then staggered cancellations).

Regression Status

Infrastructure / orchestration issue — not a code regression.

The cancellation is explained entirely by run orchestration:

• This run 28046873327 (event=schedule, branch=main) was created 2026-06-23T18:10:06Z.
• The next scheduled run 28048657152 was created 2026-06-23T18:40:50Z — the exact instant this run's jobs were cancelled (18:40:51-52Z).
• .github/workflows/pr-test-amd-rocm720.yml:122-123 sets concurrency.group: pr-test-amd-rocm720-${{ … || github.ref }} and cancel-in-progress: ${{ !inputs.run_all_tests && github.event_name != 'workflow_call' && github.event_name != 'workflow_dispatch' }}. For schedule events both conditions resolve so the group is pr-test-amd-rocm720-refs/heads/main and cancel-in-progress is true. The newer schedule run shared that group and preempted this one.

This is a recurring pattern across this workflow's schedule history — many recent scheduled runs (28048657152, 28008999561, 27993948299, 27977093213 …) also end in cancelled, each superseded by the next scheduled run.

Failure Cluster

Job cancelled by concurrency-group preemption (newer scheduled run superseded in-flight run)

Facts (verified observations, no interpretation)

• Job 83027276259: status=completed, conclusion=cancelled, started 18:11:01Z, completed 18:40:52Z, steps list empty.
• Log endpoint returns <Error><Code>BlobNotFound</Code></Error> — no log was ever written (job cancelled before/at setup).
• Run 28046873327: 7 success, 40 cancelled, 2 failure. The 2 failure jobs are the gate jobs pr-test-amd-rocm720-finish and call-pr-test-amd-extra-rocm720 / pr-test-amd-extra-finish, which fail because their upstream needs were cancelled.
• All 40 cancelled jobs share the same cancel timestamp window 18:40:51-52Z (single preemption event, not staggered).
• Newer scheduled run 28048657152 created 18:40:50Z, same branch main, same event schedule.
• Workflow concurrency config at pr-test-amd-rocm720.yml:117-123 enables cancel-in-progress for schedule events on group keyed by github.ref.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [HIGH] The job was cancelled because the newer scheduled run 28048657152 preempted this run via the shared concurrency group pr-test-amd-rocm720-refs/heads/main (cancel-in-progress: true for schedule events).
  • Supporting: timestamp coincidence (newer run created 18:40:50Z = cancellation instant 18:40:51-52Z); concurrency block in pr-test-amd-rocm720.yml:122-123 keys schedule runs to github.ref with cancel-in-progress; all 40 jobs cancelled simultaneously (preemption signature, not cascade); the same workflow's schedule history is littered with cancelled runs each superseded by the next.
  • Disconfirming: (none found) — no failure job preceded the cancellation that could indicate a fast-fail cascade; the only failure jobs are downstream gate jobs that fail as a result of the upstream cancellation.
  • How to verify: inspect run 28048657152's start time vs this run's cancel time (already aligned), or check the GitHub Actions UI annotation on the cancelled job which typically reads "Canceled — a newer run … in the same concurrency group".

In-flight Fix Check

Not applicable — there is no code failure to fix. The behavior is the intended effect of the workflow's cancel-in-progress concurrency policy.

• In-flight fix: ❌ N/A — no defect; cancellation is by design (concurrency preemption)

Suggested Triage Steps

• No code action needed: the commit under test ed26a109 did not cause this and no test executed.
• If a clean CI signal for this commit is required, re-check the immediately following completed run on main, or re-run this workflow at a time when no overlapping scheduled run is queued.
• Optional CI-hygiene investigation (orchestration, not a code bug): the 30 17 * * * and 0 */6 * * * schedule crons plus GitHub schedule delays can produce two schedule runs close in time that share the github.ref concurrency group and cancel each other; if these self-cancellations are undesirable, consider keying the schedule concurrency group by github.run_id (as already done for workflow_dispatch) — a maintainer decision, not a directive.

Status (factual, no priority assignment)

• Persistence: Single cancellation event for run 28046873327 at 18:40:51-52Z; self-cancellation of overlapping schedule runs recurs across this workflow's history.
• Scope: 40 jobs in this one run were cancelled by the same preemption; no test logic is implicated.
• Blocked work: This run produced no AMD ROCm 7.2 stage-b test signal for commit ed26a109; signal must come from another run.
• In-flight fix: ❌ N/A — cancellation is by design, no defect to fix.

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-06-24 19:29 UTC)

[amd-bot]

pr-test-amd-rocm720.yml — Per-Job Analysis (continued 6/12)

stage-b-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang, 10) — failed step(s): N/A

Commit Info

• sglang: ed26a109
• aiter: N/A — job log blob is expired/unavailable (BlobNotFound from the logs API), so no [CI-AITER-CHECK] markers could be read. aiter version is irrelevant here (see below — no test executed to completion).

Failed Tests

Test File	Test Function	Error	Log
N/A	N/A	Job cancelled mid-run (not a test failure)	Job

No test failed. The job conclusion is cancelled, not failure.

Failure Summary

This job did not fail a test — it was cancelled ~30 min into execution (started 18:11:01Z, cancelled 18:40:52Z). The entire run 28046873327 was cancelled (conclusion: cancelled): of 49 jobs, 40 are cancelled, 7 success, and only 2 failure — both of which are the downstream finish gate jobs (pr-test-amd-rocm720-finish, call-pr-test-amd-extra-rocm720 / pr-test-amd-extra-finish) that fail-by-design when upstream jobs are cancelled.

Regression Status

Infrastructure / orchestration cancellation — not a test regression.

The cancellation cause is concurrency cancel-in-progress, confirmed by two independent facts:

Fact	Evidence
Next scheduled run created exactly at cancel time	Run 28048657152 created_at = 2026-06-23T18:40:50Z; all 40 cancelled jobs in our run terminated at 18:40:51–18:40:52Z
Workflow cancels in-progress schedule runs	.github/workflows/pr-test-amd-rocm720.yml:122-123 — concurrency group for schedule resolves to pr-test-amd-rocm720-refs/heads/main with cancel-in-progress: true (true because run_all_tests is false and event is schedule)

The newer scheduled run shared the same concurrency group and evicted this run. [FACT]

Recent scheduled runs of pr-test-amd-rocm720.yml (same branch/event, completed only):

Created	Run	Conclusion
2026-06-23 18:40	28048657152	cancelled (superseded by the next one too)
2026-06-23 18:10	28046873327	cancelled (this run)
2026-06-23 12:55	28027779773	✅ success

Note: two schedule runs fired only 30 min apart (18:10 and 18:40), which is what produced the back-to-back self-cancellation.

Facts (verified observations)

• Job conclusion = cancelled, status = completed; job had zero steps recorded (cancelled before/while running steps). [FACT]
• Run-level conclusion = cancelled. Job census: 7 success, 40 cancelled, 2 failure (both finish gates). [FACT]
• All cancelled jobs share completion timestamp 18:40:51–52Z, coinciding with creation of run 28048657152 at 18:40:50Z. [FACT]
• Sister mi35x shard stage-b-test-1-gpu-small-amd-mi35x-rocm720 finished success at 18:30:58Z (before the cancellation window), so the test content itself was healthy on the comparable hardware. [FACT]
• Job log is unavailable (BlobNotFound), so per-test pass/fail counts inside this shard cannot be recovered. [FACT]

Hypothesised Causes

• [FACT] The job was cancelled by GitHub Actions concurrency cancel-in-progress when the subsequent scheduled run (created 18:40:50Z) claimed the shared concurrency group pr-test-amd-rocm720-refs/heads/main.
  • Supporting: exact timestamp coincidence + workflow concurrency config at lines 122-123.
  • Disconfirming: (none found) — no test failure signature exists; the 2 "failure" jobs are downstream finish gates reacting to the cancellation.
  • How to verify: already verified via run metadata; no further action needed.

In-flight Fix Check

Not applicable — this is a benign concurrency cancellation, not a defect. No fix PR is warranted. (No API search performed because there is no failing test/library to key on.)

Suggested Triage Steps

• No code action required. Treat this job as cancelled, not failed; exclude it from failure trend counts.
• If these self-cancellations are noisy, consider (maintainer judgement) whether the two close-spaced schedule triggers (18:10 and 18:40) are intended; the 18:40 cron evicted the 18:10 run before it could finish.
• For a real signal on commit ed26a109, look at the most recent completed: success scheduled run 28027779773 or wait for a non-superseded run.

Status

• Persistence: Single occurrence; this run was superseded by the next scheduled run.
• Scope: Run-wide — 40 jobs in run 28046873327 cancelled together by one concurrency eviction; not specific to shard 10.
• Blocked work: None — the cancellation is an orchestration artifact, no test regression to track.
• In-flight fix: N/A (not a defect).

stage-b-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang, 13) — failed step(s): N/A

Commit Info

• sglang: ed26a109
• aiter: N/A — the job log blob has expired (BlobNotFound), so no [CI-AITER-CHECK] markers are recoverable. This is moot: the job never reached the dependency-install step.

Failed Tests

Test File	Test Function	Error	Log
N/A	N/A	No test failure — job cancelled mid-run by concurrency group	Job

Failure Summary

The target job stage-b-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang, 13) did not fail a test. It was cancelled (conclusion: cancelled) at 2026-06-23T18:40:51Z after running ~30 min, with zero recorded steps and no runner assigned in the job record. Its log blob no longer exists. The entire run 28046873327 was cancelled (40 cancelled, 7 success, 2 -finish aggregator failures that are downstream of the cancellation).

Regression Status

Infrastructure issue (concurrency-group cancellation) — not a test regression. No code-level investigation is warranted.

The two *-finish jobs reported as failure (83033310987, 83040560486) are aggregator gates that fail because their upstream matrix jobs were cancelled — they are symptoms, not independent failures.

Failure Cluster

Scheduled run cancelled by concurrency group (cron collision on main)

Facts (verified observations, no interpretation)

• Run 28046873327: event=schedule, branch=main, conclusion=cancelled, created 2026-06-23T18:10:06Z.
• Target job 83027276278: conclusion=cancelled, started 18:11:01Z, completed 18:40:51Z, steps: [], runner_name: "".
• Job log fetch returns BlobNotFound (XML error blob) — log was never persisted / has expired.
• 40 of 49 jobs were cancelled at the identical timestamp 2026-06-23T18:40:51Z — a uniform, externally-triggered mass cancellation (not staggered per-job failures).
• Pre-test gates all succeeded before cancellation: call-gate / pr-gate (18:10:37Z), check-changes (18:11:00Z), and stage-b-test-1-gpu-small-amd-mi35x-rocm720 (18:30:58Z).
• A newer scheduled run 28048657152 (event=schedule, branch=main, sha e0dc8b71) was created 2026-06-23T18:40:50Z — one second before our run's jobs were cancelled at 18:40:51Z.
• Workflow concurrency config (.github/workflows/pr-test-amd-rocm720.yml:117-123): group resolves to pr-test-amd-rocm720-${{ github.ref }} = pr-test-amd-rocm720-refs/heads/main for schedule events, with cancel-in-progress: ${{ !inputs.run_all_tests && github.event_name != 'workflow_call' && github.event_name != 'workflow_dispatch' }} → true for schedule.
• Workflow has two cron triggers (.github/workflows/pr-test-amd-rocm720.yml:8-9): 30 17 * * * and 0 */6 * * *.

Hypothesised Causes (with confidence)

• [FACT] The job was cancelled by the workflow's own concurrency group when a newer scheduled run started.
  • Supporting: newer schedule run 28048657152 created at 18:40:50Z, our jobs cancelled 18:40:51Z; both runs are event=schedule on main → identical concurrency group pr-test-amd-rocm720-refs/heads/main; cancel-in-progress=true for schedule; cancellation is uniform across all 40 in-progress jobs at one timestamp.
  • Disconfirming: (none found) — every observation is consistent with a concurrency cancel; there is no test error, stack trace, or per-job failure signature anywhere in the run.
  • How to verify: already verified via the 1-second create/cancel correlation and the concurrency block; no further action needed.
• [HIGH] The two delayed cron triggers (30 17 * * * → fired ~18:10; 0 */6 * * * → fired ~18:40) collided. GitHub schedule events are routinely delayed under load, so the 17:30 and 18:00 crons landed ~30 min apart and the second cancelled the first.
  • Supporting: our run created 18:10:06 (delayed 17:30 cron), superseding run created 18:40:50 (delayed 18:00 cron); both share the main concurrency group.
  • Disconfirming: exact cron-to-run attribution is inferred from timing, not from a run-level cron field in the API response (hence HIGH not FACT).

In-flight Fix Check

• Searched open PRs for concurrency/cancel keywords:
  • In-flight fix: ❌ none found — curl -s -H "Authorization: token $GH_PAT" "https://api.github.com/search/issues?q=repo:sgl-project/sglang+is:pr+is:open+pr-test-amd-rocm720+concurrency" returns no matching open PR. This is expected: a cancelled-by-concurrency run is generally benign noise, not a tracked defect.

Suggested Triage Steps

• No code triage required — this is not a test failure. Treat the job as cancelled/no-signal, not red.
• If these cron-collision cancellations are noisy in the tracker, consider (maintainer decision) whether the two schedule crons (30 17 and 0 */6) should share a cancel-in-progress group at all — e.g. include the cron schedule or github.run_id in the schedule concurrency group so a 17:30 daily run and an 18:00 6-hourly run don't cancel each other. This is a workflow-config observation, not a directive.
• To get real signal for this matrix shard at commit ed26a109, re-run the stage on a dedicated dispatch (which gets a unique concurrency group and won't be cancelled).

Status (factual, no priority assignment)

• Persistence: single occurrence; the job was cancelled, never executed a test. Note that 5 of the last 8 scheduled runs of this workflow also show conclusion=cancelled, consistent with recurring cron-collision concurrency cancellations on main.
• Scope: entire run 28046873327 — 40 jobs cancelled simultaneously; not specific to this one shard.
• Blocked work: no test signal produced for this run; superseded by 28048657152, which carries the next signal.
• In-flight fix: ❌ none found (and likely none needed — benign cancellation, not a defect).

stage-b-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang, 0) — failed step(s): N/A

Commit Info

• sglang: ed26a109 — [Fix] Return streaming logprobs when reasoning/tool parser is active (#28601)
• aiter: N/A — job was cancelled; no log available (the log blob returns BlobNotFound), so no [CI-AITER-CHECK] markers can be read.

Failed Tests

Test File	Test Function	Error	Log
N/A	N/A	No test failure — job cancelled mid-run before any step recorded a failure	Job

Failure Summary

This job did not fail a test. Its conclusion is cancelled (not failure), it recorded zero failed steps, and it ran for ~30 min (18:11:01 → 18:40:51 UTC) before being terminated. The entire run 28046873327 was cancelled: 40 of 49 jobs were cancelled at the identical second (18:40:51–18:40:52Z), and the only two failure jobs are the pr-test-amd-extra-finish / pr-test-amd-rocm720-finish gate jobs, which fail by design when their dependents are cancelled.

Regression Status

Infrastructure issue (concurrency cancellation) — not a test regression.

Cause is a cancel-in-progress concurrency supersession, confirmed by run timing on the same workflow / branch / event (schedule, main):

Time (UTC)	Run	Event	Status	Note
2026-06-23 18:10:06	28046873327	schedule	❌ cancelled	this run — cancelled at 18:40:51
2026-06-23 18:40:50	28048657152	schedule	cancelled	superseding run, created the same second this one was cancelled; later itself superseded by the 00:33 run
2026-06-23 12:55:42	28027779773	schedule	✅ success	prior completed run

Failure Cluster

Run cancelled by concurrency supersession (cancel-in-progress).

Facts (verified observations)

• [FACT] Job 83027276302 conclusion = cancelled, steps: [], no step with conclusion=failure (GitHub jobs API).
• [FACT] 40/49 jobs in the run are cancelled, all at 18:40:51–18:40:52Z; 7 success; 2 failure (both *-finish gate jobs, failed step = Check all dependent job statuses).
• [FACT] Run 28046873327 conclusion = cancelled (run-level), event = schedule, branch = main, attempt 1.
• [FACT] A newer scheduled run 28048657152 was created at 2026-06-23T18:40:50Z — the same second this run's jobs were cancelled.
• [FACT] pr-test-amd-rocm720.yml:122-123: for schedule, concurrency group = pr-test-amd-rocm720-refs/heads/main and cancel-in-progress: true → a new schedule run on main cancels any in-progress one.
• [FACT] Job log is unavailable (BlobNotFound from the logs endpoint), so the set of tests that had run before cancellation cannot be determined.

Hypothesised Causes

• [FACT] The job was cancelled because a newer schedule run of the same workflow on main started while this run was still in progress, and the workflow's concurrency setting (cancel-in-progress: true) cancels in-progress runs in the refs/heads/main group.
  • Supporting: superseding run 28048657152 created at the exact cancellation second; concurrency config at pr-test-amd-rocm720.yml:122-123; mass simultaneous cancellation across 40 jobs; zero failed steps.
  • Disconfirming: (none found) — no evidence of any test/code failure; the two failure jobs are gate jobs reacting to the cancellation.
  • How to verify: not needed — this is directly observable. Note the broader pattern: scheduled runs take ~5-6 h but the cron fires more frequently, so scheduled runs chronically supersede each other (5 of the last 8 scheduled runs are cancelled).

In-flight Fix Check

Not applicable — there is no code/test failure to fix. The cancellation is expected behaviour of the workflow's concurrency policy. (No PR search performed, since there is no failing test/library keyword to query.)

Suggested Triage Steps

• Treat this job result as non-actionable noise — no test failed; the run was superseded by the next scheduled run.
• If reliable nightly signal on main is desired, consider that overlapping schedule runs continually cancel each other (cancel-in-progress: true on a per-ref group). A maintainer could review the cron cadence vs. typical ~5-6 h run duration so scheduled runs have time to complete before the next one starts.
• To inspect what actually ran (if needed), re-trigger the workflow manually (workflow_dispatch gets a unique concurrency group per pr-test-amd-rocm720.yml:122, so it won't be cancelled by schedule runs).

Status (factual)

• Persistence: Single cancelled run; 5 of the last 8 scheduled runs of this workflow are cancelled (chronic supersession), but no test failure is involved.
• Scope: Whole run 28046873327 — 40 cancelled jobs; not specific to this job or any test file.
• Blocked work: None — this is concurrency supersession, not a CI breakage.
• In-flight fix: N/A (no failure to fix).

stage-c-test-4-gpu-amd-rocm720 — failed step(s): N/A

Commit Info

• sglang: ed26a109 ("[Fix] Return streaming logprobs when reasoning/tool parser is active" #28601)
• aiter: N/A — the job never executed, so no [CI-AITER-CHECK] markers were emitted (job log blob is also expired/BlobNotFound).

Failed Tests

Test File	Test Function	Error	Log
—	—	No test ran. Job was cancelled before execution (started_at == completed_at == 2026-06-23T18:40:52Z).	Job

This is not a test failure. The job stage-c-test-4-gpu-amd-rocm720 was cancelled while queued and executed zero test steps.

Failure Summary

The job did not fail on any test — it was cancelled as part of a run-wide cancellation of run 28046873327. 40 of 49 jobs in this run were cancelled at the same instant (18:40:51–18:40:52Z). The two *-finish gate jobs subsequently reported failure because their upstream jobs were cancelled. The cancellation was caused by GitHub Actions concurrency-group preemption: a newer scheduled run superseded this one.

Regression Status

Infrastructure / orchestration issue (concurrency preemption) — not a code regression, and not a test failure.

Recent history of job stage-c-test-4-gpu-amd-rocm720 (workflow pr-test-amd-rocm720.yml, branch main, event schedule, completed runs only):

Date (UTC)	Run	Job Status	Notes
Jun 23 18:40	28048657152	⏭️ Skipped	(this run itself superseded ours)
Jun 23 18:10	28046873327	🚫 Cancelled	this job — preempted before running
Jun 23 12:55	28027779773	⏭️ Skipped	run overall success
Jun 23 07:13	28008999561	⏭️ Skipped	—

Note: in normal completed scheduled runs this job is consistently skipped (gated by stage/change-detection), so it never executes tests on the daily cron. The cancelled state here is purely an artifact of the run-wide preemption — there is no green/red test signal to track for it.

Failure Cluster

"Run-wide cancellation via concurrency-group preemption (newer scheduled run superseded older)"

Facts (verified observations)

• Job conclusion = cancelled; started_at == completed_at == 2026-06-23T18:40:52Z → zero execution time. [FACT]
• Run 28046873327 (event=schedule, sha ed26a109) overall conclusion = cancelled; job breakdown: 7 success, 40 cancelled, 2 failure (the two *-finish aggregator gates). [FACT]
• 40 jobs were cancelled within the same 1-second window (18:40:51–52Z). [FACT]
• A newer scheduled run 28048657152 (event=schedule, branch main, sha e0dc8b71) was created at 2026-06-23T18:40:50Z — 1 second before the mass cancellation. [FACT]
• Workflow concurrency config (.github/workflows/pr-test-amd-rocm720.yml:117-123): for a schedule event the group resolves to pr-test-amd-rocm720-${{ github.ref }} = pr-test-amd-rocm720-refs/heads/main, and cancel-in-progress evaluates to true (the event is not workflow_call/workflow_dispatch and run_all_tests is unset). [FACT]
• Two cron schedules overlap into the same window after GitHub scheduling delay: daily 30 17 * * * (→ our 18:10 run) and 6-hourly 0 */6 * * * (→ the 18:40 superseder). Both fire on main and share the same concurrency group. [FACT]
• The job log is BlobNotFound (expired/never produced), consistent with the job never running. [FACT]

Hypothesised Causes (with confidence)

• [HIGH] The run was cancelled because the newer scheduled run 28048657152 entered the shared concurrency group pr-test-amd-rocm720-refs/heads/main with cancel-in-progress=true, preempting the in-progress run 28046873327. Our job was still queued (waiting for a 4-GPU runner) and was cancelled before it could start or skip.
  • Supporting: superseder run created 18:40:50Z == cancellation 18:40:51Z (1s apart); both runs are schedule on main → identical concurrency group; cancel-in-progress true for schedule per workflow lines 117-123; 40 jobs cancelled simultaneously (signature of run-level cancel, not per-job failure).
  • Disconfirming: (none found) — the timing, shared group, and config all converge.
  • How to verify: GitHub run 28048657152 created_at vs run 28046873327 cancellation timestamp; re-read concurrency block in .github/workflows/pr-test-amd-rocm720.yml:117-123.
• [FACT] The two *-finish jobs reporting failure are a downstream consequence, not a root cause: they ran/failed at 18:41:09Z and 19:17:00Z, after their upstream jobs were cancelled.

No test-level root cause exists because no test ran.

In-flight Fix Check

This is expected, by-design concurrency behavior (overlapping scheduled crons on main cancel each other), not a bug requiring a fix. No fix-PR search is warranted. If maintainers want to stop the two crons from cancelling each other, the change would be in the concurrency group expression — searched for context:

• In-flight fix: ❌ none found / not applicable — this is orchestration-by-design, not a failing test.

Suggested Triage Steps

• Treat as a non-failure. No code investigation needed; commit ed26a109 is not implicated. Disregard for regression tracking.
• If the recurring run-wide cancellations are noisy in CI dashboards: consider whether the two schedule crons (30 17 * * * daily and 0 */6 * * * 6-hourly) should share a concurrency group on main. The 6h cron is intended to run only the dsv4 jobs (if: github.event.schedule != '0 */6 * * *' skips the gate for it), yet it still preempts the in-progress daily run via the shared github.ref group. A maintainer could differentiate the group by github.event.schedule so the dsv4-only cron does not cancel the full daily run.
• For the specific job: note that stage-c-test-4-gpu-amd-rocm720 is skipped in all normal scheduled runs, so it produces no real test signal on this cron regardless of cancellation.

Status (factual)

• Persistence: Single-occurrence cancellation in run 28046873327. Scheduled runs of this workflow are cancelled frequently (10 of the last 12 scheduled runs show cancelled), consistent with chronic cron-overlap preemption rather than test breakage.
• Scope: Run-wide — 40 jobs in this single run cancelled; no test job actually failed. Target job itself never executed (and is normally skipped on this cron).
• Blocked work: None attributable to code. The cron produces no actionable test signal when preempted.
• In-flight fix: Not applicable — by-design concurrency behavior, no failing test.

dsv4-pro-fp4-amd-rocm720 — failed step(s): Accuracy Test MI35x ROCm 7.2 (8-GPU DeepSeek-V4-Pro FP4)

Commit Info

• sglang: c4376aaa ("[Refactor] Remove dead out_cache_loc_swa buffers" #28968)
• aiter: 7d604afe5 — v0.1.14rc1.dev240+g7d604afe5 (source: dev image preserved — Dockerfile expects 7d604afe5…, image already had a dev build, so [CI-AITER-CHECK] reports "Dev/patched version detected → skipping rebuild")

Failed Tests

Test File	Test Function	Error	Log
N/A — server launch (not a pytest failure)	deepseek-ai/DeepSeek-V4-Pro server startup	Server never reached ready; job cancelled mid weight-load after ~33 min	Log

This is not a test-assertion failure. The GSM8K accuracy test (SGLANG_DSV4_ACCURACY_ONLY=1) never ran because the SGLang server for deepseek-ai/DeepSeek-V4-Pro (tp=8, --attention-backend dsv4, fp8) never finished initializing.

Failure Summary

The dsv4-pro-fp4-amd-rocm720 job launched a DeepSeek-V4-Pro server at 12:31. Model weight download/load was extremely slow: ranks TP1–TP6 only finished after elapsed=756–770 s (≈12.8 min), while TP0 and TP7 never printed Load weight end within the window. With weight load still incomplete, the GitHub step was cancelled at 12:55:55 (##[error]The operation was canceled.), ~33 min after the job started. The soft watchdog (300 s) fired repeatedly during this time but is non-fatal.

Regression Status

Flaky / intermittent (slow-startup cancellation) — NOT a code regression.

Recent history of dsv4-pro-fp4-amd-rocm720 (same branch main, event=schedule, completed runs only):

Date	Run	Job	Status	Note
Jun 23 12:55	28027779773	dsv4-pro-fp4-amd-rocm720	✅ Success	head 0460f277
Jun 23 07:13	28008999561	dsv4-pro-fp4-amd-rocm720	🚫 Cancelled (this run)	stuck in weight load
Jun 23 00:39	27993948299	dsv4-pro-fp4-amd-rocm720	✅ Success	—
Jun 22 19:05	27977093213	dsv4-pro-fp4-amd-rocm720	🚫 Cancelled	whole-run cancelled
Jun 22 13:52	27957777753	dsv4-pro-fp4-amd-rocm720	✅ Success	—

The job passes in most runs, including the very next one. The only other non-success job in this run was pr-test-amd-rocm720-finish (the aggregation gate), which failed solely because this job was cancelled.

Failure Cluster

Slow DeepSeek-V4-Pro fp8 weight download/load → server-startup timeout & job cancellation

Facts (verified observations)

• [CI-AITER-CHECK]: image aiter = v0.1.14rc1.dev240+g7d604afe5, dev build preserved (no rebuild).
• 12:31:50 TP7: [CI Download] Process 3209 ACQUIRED lock for deepseek-ai/DeepSeek-V4-Pro — TP7 became the sole downloader; other ranks block on the same filelock.
• Scheduler py-spy dumps (12:36–12:41) show TP2/TP3/TP5/TP6 idle in: _poll_until_acquired (filelock/_api.py:552) ← download_weights_from_hf (weight_utils.py:554) ← _prepare_weights (loader.py:472) ← load_weights (deepseek_v4.py:2106) ← load_model (model_runner.py:1423).
• Main process (PID 3044) idle in _wait_for_scheduler_ready (engine.py:1355) the entire time.
• 12:44:26–12:44:40: Load weight end. elapsed=756.70 s … 770.27 s for TP1–TP6. No Load weight end for TP0 or TP7.
• 12:46–12:51 dumps: surviving ranks then idle inside a c10d ProcessGroup collective at initialize (model_runner.py:693) — blocking on the lagging rank(s).
• 12:55:55: ##[error]The operation was canceled. Job wall-time 12:22:11 → 12:55:58 (~33.5 min).
• Soft watchdog (watchdog_timeout=300, self.soft=True) fired at 12:36, 12:41, 12:46, 12:51 — informational, did not kill the process.

Hypothesised Causes (with confidence)

• [MEDIUM] Cold / slow HuggingFace weight download+load of the very large DeepSeek-V4-Pro fp8 checkpoint exceeded the job's effective time budget, leaving TP0/TP7 still loading when the step was cancelled.
  • Supporting: serialized download under a single filelock held by TP7; finishing ranks took 756–770 s just for load; TP0/TP7 never completed; the next run (warm cache, head 0460f277) succeeded.
  • Disconfirming: TP0/TP7's exact stall point is not captured (they never logged Load weight end); cannot rule out a genuine post-load distributed-init stall at model_runner.py:693 rather than pure download slowness.
  • How to verify: re-run with a pre-warmed /sgl-data/hf-cache for deepseek-ai/DeepSeek-V4-Pro; compare load elapsed with a passing run; check the runner's disk/network during the failing window.
• [LOW] Distributed collective stall after load (one rank lagging at the model_runner.py:693 barrier).
  • Supporting: 12:46+ dumps show ranks in a c10d collective.
  • Disconfirming: this is the expected blocking behavior when TP0/TP7 simply hadn't finished loading yet — consistent with slow load, not a hang.
• [LOW] Commit c4376aaa (HEAD).
  • Supporting: it is the run's head commit.
  • Disconfirming: it only removes dead out_cache_loc_swa buffers and is unrelated to weight download/load; the same job passed on neighboring commits. Effectively exonerated.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs:
  • curl ".../search/issues?q=repo:sgl-project/sglang+is:pr+is:open+DeepSeek-V4-Pro+download" → no matching weight-download/timeout PR.
  • curl ".../search/issues?q=repo:sgl-project/sglang+is:pr+is:open+dsv4+watchdog" → none.
    This is an intermittent infra/slow-load symptom; a code fix may not be the right lever.

Suggested Triage Steps

• Confirm whether /sgl-data/hf-cache was cold on this runner — if so, pre-stage deepseek-ai/DeepSeek-V4-Pro weights (or rely on the existing CI download-lock cache) so load time stays well under the job timeout.
• Compare Load weight end elapsed in this run (756–770 s) vs the passing run 28027779773 to quantify the slowdown.
• If slow loads recur, consider raising the dsv4-pro job timeout, or parallelizing the per-rank weight read so TP0/TP7 don't lag the c10d barrier at model_runner.py:693.
• No bisect warranted — the job passes on surrounding commits; treat as flaky-by-infrastructure unless load time regresses consistently.

Status (factual)

• Persistence: Single cancelled occurrence in run 28008999561; job succeeded immediately before and after. Not persistent.
• Scope: This job only (dsv4-pro-fp4-amd-rocm720). The run's other "failure" (pr-test-amd-rocm720-finish) is a downstream gate that failed only because this job was cancelled.
• Blocked work: Blocks the pr-test-amd-rocm720 scheduled signal for this run only.
• In-flight fix: ❌ none found — likely infra/slow-download, not a code defect.

call-pr-test-amd-extra-rocm720 / extra-a-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang) — failed step(s): N/A

Commit Info

• sglang: ba9d5aed — "Fix nightly CI test for Kimi K2.5 INT4 + H200 (#28746)"
• aiter: N/A — the job was cancelled before any steps executed; no [CI-AITER-CHECK] markers were produced and the job log blob no longer exists (BlobNotFound).

Failed Tests

Test File	Test Function	Error	Log
N/A	N/A	Job cancelled before any step ran (0 steps executed) — no test failure	Job

No test ever ran. This is not a test failure — it is an orchestration/runner cancellation.

Failure Summary

The job extra-a-test-1-gpu-small-amd-rocm720 was cancelled (conclusion cancelled, not failure) 12 minutes after the runner picked it up (00:39:10 → 00:51:40 UTC) with zero steps recorded. Its sister job extra-a-test-1-gpu-large-amd-rocm720 was cancelled at the exact same instant (00:51:40). The only job reporting failure in the run is the aggregator gate pr-test-amd-extra-finish, which exit 1s solely because its upstream needs returned cancelled ([FACT]: finish log — extra-a-test-1-gpu-small-amd: cancelled → The above jobs failed. → ##[error]Process completed with exit code 1).

Regression Status

Infrastructure issue (cancellation, not a code regression). When this job actually acquires its MI325 runner it passes — it succeeded in 3 of the last 4 scheduled runs.

Recent history of extra-a-test-1-gpu-small-amd-rocm720 in workflow pr-test-amd-rocm720.yml (branch main, event schedule, completed runs only):

Date (UTC)	Run	Steps	Job Status	Note
Jun 23 19:58	28048657152	8	✅ Success	ran ~36 min
Jun 23 15:10	28027779773	8	✅ Success	ran ~37 min
Jun 23 09:49	28008999561	8	✅ Success	ran ~38 min
Jun 23 00:39	27993948299	0	⛔ Cancelled	this run — 0 steps
Jun 22 21:53	27977093213	0	⛔ Cancelled	2h+ queued then cancelled, 0 steps

The cancelled occurrences both have 0 steps (job never started executing). The passing occurrences run a full 8-step suite. This confirms the cancellations happen before the test body runs, i.e. during runner acquisition/setup — not in the tests themselves.

Failure Cluster

"AMD MI325 1-GPU job cancelled before steps run (runner-acquisition / orchestration cancellation)"

Facts (verified observations)

• [FACT] Job conclusion is cancelled, not failure; steps: 0 in the job API payload.
• [FACT] Both 1-GPU extra-a jobs (small id 82852085866, large id 82852085884) were cancelled at the identical timestamp 2026-06-23T00:51:40Z.
• [FACT] Job log blob returns BlobNotFound — no executable log was ever uploaded.
• [FACT] The run's overall conclusion is cancelled; job census = 17 skipped, 4 success, 2 cancelled, 1 failure (the finish gate).
• [FACT] pr-test-amd-extra-finish failed only because of the cancelled upstreams (verified verbatim in its log).
• [FACT] The extra workflow runs with continue_on_error: true (from finish-job inputs).
• [FACT] This run was not superseded by a newer scheduled run — the next schedule run was 6h+ later (28008999561, 07:13 UTC).
• [FACT] Multiple recent scheduled runs of this workflow also concluded cancelled (28048657152, 28046873327, 27993948299, 27977093213).

Hypothesised Causes (with confidence)

• [MEDIUM] MI325 1-GPU runner pool contention / availability — the job waited 12 min, executed 0 steps, then was cancelled; sibling job cancelled at the same instant; the same job passes whenever a runner is available.
  • Supporting: 0 steps in cancelled runs vs 8 steps + ~37 min in passing runs; simultaneous cancellation of both 1-GPU jobs; recurring cancelled-run pattern on the self-hosted AMD pool.
  • Disconfirming: cannot see the orchestration/runner logs (blob expired), so the exact cancel signal (manual cancel, group-level timeout, or runner reclaim) is not directly verifiable.
  • How to verify: inspect the run's orchestration timeline / self-hosted runner queue logs for linux-mi325-1gpu-sglang around 2026-06-23T00:39–00:51 UTC; check whether a concurrency group or a manual cancel fired.
• [LOW] Workflow-level / concurrency cancellation unrelated to runner supply.
  • Supporting: run-level conclusion is cancelled; both jobs cancelled at the same instant.
  • Disconfirming: no newer scheduled run within the window to trigger a concurrency supersede; other jobs in the same run (the dsv4 jobs) ran successfully ~5h later, so the run was not globally aborted.
• The commit under test ba9d5aed is not implicated — no test code ran, and the same SHA's extra-a suite is the one that passes in adjacent runs is not relevant since each scheduled run uses its own HEAD; the cancellation is pre-test.

In-flight Fix Check

• In-flight fix: ❌ none found — this is an infra/runner-capacity symptom, not a code defect; there is no PR to chase. (No code-level keyword applies; searching for a fix PR is not meaningful for a runner-acquisition cancellation.)

Suggested Triage Steps

• Treat this as an infrastructure cancellation, not a test regression — no code triage on ba9d5aed is warranted.
• Check the linux-mi325-1gpu-sglang self-hosted runner availability/queue depth around 2026-06-23 00:39–00:51 UTC to confirm runner starvation.
• Re-run the workflow (workflow_dispatch or wait for the next schedule) and confirm extra-a-test-1-gpu-small-amd-rocm720 reaches its 8 steps — adjacent runs show it passes when a runner is free.
• If scheduled-run cancellations on this workflow remain frequent, review the workflow's concurrency group / timeout settings and the MI325 pool capacity allocated to scheduled (vs PR) runs.

Status (factual)

• Persistence: Intermittent cancellation; this job has both passed (3×) and been cancelled (2×) across the last 5 completed scheduled runs in the last ~24h. Not a sustained failure.
• Scope: Both 1-GPU extra-a jobs (small + large) in this run; the cascade surfaces as the single pr-test-amd-extra-finish failure. The extra workflow is continue_on_error: true.
• Blocked work: No real test signal lost for the changed code — when a runner is available the suite runs green. The red mark is a cancellation cascade, not a code break.
• In-flight fix: ❌ none found — infra/runner-capacity issue, no fix PR applicable.

call-pr-test-amd-extra-rocm720 / extra-a-test-1-gpu-large-amd-rocm720 (linux-mi325-1gpu-sglang) — failed step(s): N/A

Commit Info

• sglang: ba9d5aed (run head, committed 2026-06-23 00:27:33 UTC)
• aiter: N/A — the job was cancelled while queued; no steps ran, no [CI-AITER-CHECK] markers exist in any log.

Failed Tests

Test File	Test Function	Error	Log
N/A — no test ran	N/A	Job cancelled while queued (0 steps executed)	Job page

This is not a test failure. Job 82852085884 was cancelled at 2026-06-23T00:51:40Z after sitting queued since 00:39:10Z (~12.5 min). It has zero recorded steps (never reached "Set up job"), and its log blob returns BlobNotFound — confirming no work executed.

Failure Summary

The scheduled run 27993948299 (6-hourly 0 */6 cron) was cancelled mid-flight. On this cron, call-gate and 17 of 24 jobs skip; only the two dsv4 jobs (which succeeded) and the chained AMD-extra suite run. The two extra test jobs — extra-a-test-1-gpu-large-amd-rocm720 (this job) and its sister extra-a-test-1-gpu-small-amd-rocm720 (82852085866) — were both cancelled at the same second while still waiting for an MI325 runner. The pr-test-amd-extra-finish aggregator (82853544459) then reported failure solely because its dependent jobs were cancelled (step "Check all dependent job statuses" → failure).

Regression Status

Infrastructure issue (run cancellation) — not a code regression. No test executed.

Recent pr-test-amd-rocm720.yml schedule runs on main (completed only):

Date (UTC)	Run	Conclusion
2026-06-23 18:40	28048657152	cancelled
2026-06-23 18:10	28046873327	cancelled
2026-06-23 12:55	28027779773	✅ success
2026-06-23 07:13	28008999561	cancelled
2026-06-23 00:39	27993948299	cancelled (this run)
2026-06-22 19:05	27977093213	cancelled
2026-06-22 18:45	27975943807	cancelled

5 of the last 8 schedule runs were cancelled — a recurring orchestration pattern, now addressed by a merged fix (see below).

Failure Cluster

Scheduled AMD-extra suite jobs cancelled by shared concurrency group (queue-time cancellation, no test executed)

Facts (verified observations)

• Job conclusion cancelled; started_at=2026-06-23T00:39:10Z, completed_at=2026-06-23T00:51:40Z; 0 steps in the job's steps[] array (verified via Jobs API).
• Log download returns <Error><Code>BlobNotFound</Code> — no log was ever produced.
• Sister job extra-a-test-1-gpu-small-amd-rocm720 (82852085866) cancelled at the identical timestamp 00:51:40Z, also 0 steps.
• Run-level conclusion is cancelled (run_attempt: 1).
• The chained pr-test-amd-extra.yml concurrency block at the run's SHA used a group that, for schedule, resolved to the constant pr-test-amd-extra-schedule-main-all (omitting github.run_id) with cancel-in-progress active for scheduled events. PR #28987 quotes the exact runner message: Canceling since a higher priority waiting request for pr-test-amd-extra-schedule-main-all exists.

Hypothesised Causes (with confidence)

• [HIGH] Shared concurrency group in pr-test-amd-extra.yml cancelled this in-progress scheduled run when a later scheduled invocation entered the same group.
  • Supporting: PR #28987/#28989 document the root cause — every 6-hourly scheduled invocation resolved to the same group string pr-test-amd-extra-schedule-main-all because github.run_id was omitted, so a newer waiting request cancelled the running one. The job's 0-step, queued-then-cancelled signature matches a concurrency cancellation exactly.
  • Disconfirming: the canceller could not be pinpointed from run timestamps alone (next visible schedule run is 07:13, well after the 00:51 cancel) — the cancelling request may have been a transient/duplicate scheduled invocation not retained in the runs list. This is the only gap; the documented root cause and the runner message in #28987 otherwise make this HIGH.
  • How to verify: confirmed by the merged fix below — runs after the fix landed should stop cancelling.

In-flight Fix Check

• In-flight fix: ✅ MERGED — #28989 "[AMD] Fix AMD extra scheduled runs cancelling each other" merged 2026-06-23T03:48:33Z (b42c79c4), ~3 h after this run started. The fix changes the schedule/dispatch concurrency group to format('full-{0}', github.run_id) so each scheduled run gets a unique group. This run predated the fix.
• Related draft predecessor: #28987 (still open as draft) — superseded by the merged #28989; no new PR needed.
• Searched: repo:sgl-project/sglang is:pr pr-test-amd-extra cancelled.

Suggested Triage Steps

• No code action needed for this job — the cancellation is fixed in b42c79c4 (#28989), already on main.
• Confirm the fix is effective: check the next few pr-test-amd-rocm720.yml schedule runs after 2026-06-23 03:48 UTC for whether the extra suite still cancels (the 12:55 run 28027779773 already shows ✅ success; the 18:10/18:40 cancels are worth a glance to confirm they are a different cause, since both fall after the fix merged).
• If post-fix cancellations recur, re-investigate the parent pr-test-amd-rocm720.yml concurrency group (pr-test-amd-rocm720-refs/heads/main, cancel-in-progress on schedule) rather than the now-fixed child workflow.

Status (factual)

• Persistence: 5 of the last 8 pr-test-amd-rocm720.yml schedule runs cancelled (2026-06-22 18:45 → 2026-06-23 18:40); this specific run cancelled at 2026-06-23 00:51:40 UTC.
• Scope: both AMD-extra test jobs in this run (large + small shards) plus the extra-finish aggregator; the base rocm720 required path (dsv4 jobs + finish) succeeded and was unaffected (continue_on_error: true, not wired to the base finish gate).
• Blocked work: scheduled AMD-extra-tier signal (best-effort) was lost for this run; no PR is gated by it.
• In-flight fix: ✅ merged #28989 (b42c79c4), 2026-06-23 03:48 UTC.

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-06-24 19:29 UTC)

[amd-bot]

pr-test-amd-rocm720.yml — Cross-Run Pattern Summary (8 runs in lookback window)

Last scanned (UTC): 2026-06-24 19:30 UTC

Agent assessment

Persistent cluster 1 — Concurrency-supersession cancellation (dominant; ~80 of ~88 job failures)

• Pattern: The parent workflow's schedule concurrency group pr-test-amd-rocm720-refs/heads/main with cancel-in-progress: true (.github/workflows/pr-test-amd-rocm720.yml:117-123), combined with the 0 */6 cron and multi-hour MI35x/MI325 runner queues, means each new scheduled run cancels the still-running prior one. Confirmed by 1-second timestamp alignment in multiple jobs.
• Affected: whole runs 28046873327 (40 cancelled) and 28119463964 (40 cancelled, superseded by 28121084080); plus single-job cancellations extra-a-2gpu-large @28099289606 (test was passing at cancel time) and extra-a-1gpu-small @27993948299 (0 steps).
• Top hypothesis: [FACT] concurrency cancel-in-progress on the schedule cron supersedes in-progress runs; disconfirming: (none — verified from run/job timestamps + the workflow concurrency block).
• In-flight fix: ✅ #28989 (merged 2026-06-23) fixed it for the child pr-test-amd-extra workflow; the parent pr-test-amd-rocm720.yml schedule group still cancels whole runs — ❌ no fix for the parent.
• Suggested triage: maintainer decision — exempt long-running 8-GPU/MI35x scheduled jobs from cancel-in-progress, give each scheduled run a unique group (as #28989 did for the child), or lengthen the 6h cron vs. MI35x capacity.

Persistent cluster 2 — DeepSeek-V4-Pro slow fp8 weight load → cancelled mid-load (dsv4-pro-fp4-amd-rocm720)

• Affected: dsv4-pro-fp4 @28066754977 and dsv4-pro-fp4 @28008999561 — server still in _load_w13 (moe/fused_moe_triton/layer.py:503) at ~25 min when the next scheduled run cancelled it. test_deepseek_v4_pro_fp4.py GSM8K never ran. Passes when load completes in time (e.g. 28027779773, load elapsed=976 s, 81 soft-watchdog fires — slow load is the norm).
• Top hypothesis: [HIGH] pathologically slow DSV4 fp8 weight load leaves no margin before the 6h-cron concurrency cancel; disconfirming: cannot separate "slower than usual load" from "less time" without a completed-load timestamp.
• In-flight fix: ✅ #28729 "Offload fp8 wo_a dequant to GPU to speed up weight loading" (open since 2026-06-19) — chase reviewers; also ✅ #25498 (CI weight-loader prefetch).
• Note: the mi35x-gpu-8 dsv4 jobs (dsv4-flash @28048657152, dsv4-pro @28048657152) also cancelled at 0 steps from pure runner starvation (5h+ queue) — same concurrency-enabler as cluster 1.

Regression candidates

(none) — the latest completed-and-meaningful run is the cancellation 28119463964, which ran no tests. No fresh per-commit regression is identifiable in this window.

One-off / isolated infra

• mori build failure — corrupt libabsl_time_zone.so (invalid ELF header): extra-a-1gpu-small @28081633329, new today, 1/4 runs. Sibling 1gpu-large passed on the same SHA in the same run → runner/image corruption, not code. [MEDIUM]; ❌ no in-flight fix (infra ticket, not an sglang PR).

Suggested next actions (by ROI)

1. Parent-workflow concurrency (cluster 1): apply the #28989-style unique-group fix to pr-test-amd-rocm720.yml schedule runs, or exempt 8-GPU/MI35x stages from cancel-in-progress — this would eliminate the bulk of the ~80 cancellation "failures".
2. Chase #28729 (cluster 2) to cut DSV4 weight-load time so it completes inside the schedule cadence.
3. MI35x/MI325 runner capacity: investigate the 5h+ queue depth feeding both clusters 1 and 2.
4. mori/libabsl: re-run 83137794582; if it recurs, file an infra ticket to evict the corrupt image layer.

---

Runs analysed

Run	Started (UTC)	Failed jobs
27993948299	2026-06-23 00:39	2
28008999561	2026-06-23 12:22	1
28046873327	2026-06-23 18:10	40
28048657152	2026-06-23 19:17	2
28066754977	2026-06-24 06:33	1
28081633329	2026-06-24 08:18	1
28099289606	2026-06-24 17:56	1
28119463964	2026-06-24 18:08	40

Persistent failures (in every run)

(none)

Latest-only failures (potential regressions)

(none)

Flaky / intermittent failures

• call-pr-test-amd-extra-rocm720 / extra-a-test-1-gpu-large-amd-rocm720 (linux-mi325-1gpu-sglang) — in 3/8 runs (3948299, 6873327, 9463964)
• call-pr-test-amd-extra-rocm720 / extra-a-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang) — in 4/8 runs (3948299, 6873327, 1633329, 9463964)
• call-pr-test-amd-extra-rocm720 / extra-a-test-2-gpu-large-amd-rocm720 (linux-mi325-2gpu-sglang) — in 3/8 runs (6873327, 9289606, 9463964)
• dsv4-flash-fp4-fp8-amd-rocm720 — in 3/8 runs (6873327, 8657152, 9463964)
• dsv4-pro-fp4-amd-rocm720 — in 5/8 runs (8999561, 6873327, 8657152, 6754977, 9463964)
• multimodal-gen-test-1-gpu-amd-rocm720 (linux-mi325-1gpu-sglang, 0) — in 2/8 runs (6873327, 9463964)
• multimodal-gen-test-1-gpu-amd-rocm720 (linux-mi325-1gpu-sglang, 1) — in 2/8 runs (6873327, 9463964)
• multimodal-gen-test-1-gpu-amd-rocm720 (linux-mi325-1gpu-sglang, 2) — in 2/8 runs (6873327, 9463964)
• multimodal-gen-test-1-gpu-amd-rocm720 (linux-mi325-1gpu-sglang, 3) — in 2/8 runs (6873327, 9463964)
• multimodal-gen-test-2-gpu-amd-rocm720 (linux-mi325-2gpu-sglang, 0) — in 2/8 runs (6873327, 9463964)
• multimodal-gen-test-2-gpu-amd-rocm720 (linux-mi325-2gpu-sglang, 1) — in 2/8 runs (6873327, 9463964)
• multimodal-gen-test-2-gpu-amd-rocm720 (linux-mi325-2gpu-sglang, 2) — in 2/8 runs (6873327, 9463964)
• stage-a-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang) — in 2/8 runs (6873327, 9463964)
• stage-b-test-1-gpu-large-amd-rocm720 (linux-mi325-1gpu-sglang, 0) — in 2/8 runs (6873327, 9463964)
• stage-b-test-1-gpu-large-amd-rocm720 (linux-mi325-1gpu-sglang, 1) — in 2/8 runs (6873327, 9463964)
• stage-b-test-1-gpu-small-amd-nondeterministic-rocm720 (linux-mi325-1gpu-sglang) — in 2/8 runs (6873327, 9463964)
• stage-b-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang, 0) — in 2/8 runs (6873327, 9463964)
• stage-b-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang, 1) — in 2/8 runs (6873327, 9463964)
• stage-b-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang, 10) — in 2/8 runs (6873327, 9463964)
• stage-b-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang, 11) — in 2/8 runs (6873327, 9463964)
• stage-b-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang, 12) — in 2/8 runs (6873327, 9463964)
• stage-b-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang, 13) — in 2/8 runs (6873327, 9463964)
• stage-b-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang, 2) — in 2/8 runs (6873327, 9463964)
• stage-b-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang, 3) — in 2/8 runs (6873327, 9463964)
• stage-b-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang, 4) — in 2/8 runs (6873327, 9463964)
• stage-b-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang, 5) — in 2/8 runs (6873327, 9463964)
• stage-b-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang, 6) — in 2/8 runs (6873327, 9463964)
• stage-b-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang, 7) — in 2/8 runs (6873327, 9463964)
• stage-b-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang, 8) — in 2/8 runs (6873327, 9463964)
• stage-b-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang, 9) — in 2/8 runs (6873327, 9463964)
• stage-b-test-2-gpu-large-amd-rocm720 (linux-mi325-2gpu-sglang, 0) — in 2/8 runs (6873327, 9463964)
• stage-b-test-2-gpu-large-amd-rocm720 (linux-mi325-2gpu-sglang, 1) — in 2/8 runs (6873327, 9463964)
• stage-b-test-large-8-gpu-mi35x-disaggregation-amd-rocm720 (linux-mi35x-gpu-8.fabric) — in 2/8 runs (6873327, 9463964)
• stage-c-test-4-gpu-amd-rocm720 — in 2/8 runs (6873327, 9463964)
• stage-c-test-large-8-gpu-amd-mi35x-rocm720 (linux-mi35x-gpu-8, 0) — in 2/8 runs (6873327, 9463964)
• stage-c-test-large-8-gpu-amd-mi35x-rocm720 (linux-mi35x-gpu-8, 1) — in 2/8 runs (6873327, 9463964)
• stage-c-test-large-8-gpu-amd-rocm720 (linux-mi325-8gpu-sglang, 0) — in 2/8 runs (6873327, 9463964)
• stage-c-test-large-8-gpu-amd-rocm720 (linux-mi325-8gpu-sglang, 1) — in 2/8 runs (6873327, 9463964)
• stage-c-test-large-8-gpu-amd-rocm720 (linux-mi325-8gpu-sglang, 2) — in 2/8 runs (6873327, 9463964)
• stage-c-test-large-8-gpu-amd-rocm720 (linux-mi325-8gpu-sglang, 3) — in 2/8 runs (6873327, 9463964)

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-06-24 19:30 UTC)

[amd-bot]

nightly-test-amd-rocm720.yml — Per-Job Analysis (continued 2/3)

nightly-8-gpu-mi35x-qwen35-rocm720 — failed step(s): Accuracy Test MI35x ROCm 7.2 (8-GPU Qwen 3.5)

Commit Info

• sglang: ed26a109
• aiter: N/A — no [CI-AITER-CHECK] markers in this log; AITER_COMMIT_OVERRIDE empty (regular nightly, not a scout). aiter is irrelevant to this failure (pure ServerArgs validation, no kernel involvement).

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi35x/test_qwen35_eval_mi35x.py	TestQwen35EvalMI35x.test_lm_eval	AssertionError: extra_buffer needs CUDA/MUSA/NPU (FLA).	Log

Failure Summary

test_lm_eval launches an sglang server for Qwen/Qwen3.5-397B-A17B (Qwen3_5MoeForConditionalGeneration, a hybrid mamba model) with --attention-backend aiter on AMD MI35x / ROCm 7.2. The server process exits with code 1 during ServerArgs.__post_init__ before serving any request, so popen_launch_server fails and the test errors after exhausting its single retry (Ran 1 test in 47.418s, 0/1 passed).

Regression Status

New regression — first observed 2026-06-19. The same test passed as recently as 2026-06-18.

Recent history of test_qwen35_eval_mi35x.py (test_lm_eval) in job nightly-8-gpu-mi35x-qwen35-rocm720 (branch=main, event=schedule, completed runs only):

Date	Run	Test File Status	Failed Function	Error
Jun 24	28046795462	❌ Failed	test_lm_eval	extra_buffer needs CUDA/MUSA/NPU (FLA)
Jun 22	27975850577	❌ Failed	test_lm_eval	same
Jun 21	27912828989	❌ Failed	test_lm_eval	same
Jun 20	27879239829	❌ Failed	test_lm_eval	same
Jun 19	27841393725	❌ Failed	test_lm_eval	same (first failure)
Jun 18	27780200720	✅ Passed	—	—
Jun 17	27710084518	✅ Passed	—	—

First observed failure: 2026-06-19. Last known passing: 2026-06-18.
Regression window: last-pass head f7632ef8 → first-fail head ca88b7f1 (compare, 43 commits).

Failure Cluster

ROCm mamba extra_buffer assertion at server startup (extra_buffer needs CUDA/MUSA/NPU (FLA))

Facts (verified observations)

• Server stack trace (verbatim): server_args.py:4386 _validate_mamba_extra_buffer → assert (is_cuda() or is_musa() or is_npu()), "extra_buffer needs CUDA/MUSA/NPU (FLA).", called via _handle_mamba_radix_cache (line 4415) ← _handle_model_specific_adjustments (line 4230) ← __post_init__ (line 2541).
• Crash happens twice (initial + 1 retry), both with the identical assertion, then the test errors with retry() exceed maximum number of retries.
• Launch command (verbatim): sglang serve --model-path Qwen/Qwen3.5-397B-A17B --tp 8 --attention-backend aiter --trust-remote-code ... --device cuda (--device cuda is the AMD CI convention; the platform is HIP/ROCm, so is_cuda() is False).
• The same (test_file, test_function) failed in 5 consecutive completed runs (Jun 19–24) and passed in the 2 prior completed runs (Jun 17–18). Verified by reading each job log at the test-file level.
• Current source (server_args.py:4441-4456): _support_mamba_cache_extra_buffer returns self.linear_attn_backend == "triton" for Qwen3_5MoeForConditionalGeneration; _handle_mamba_radix_cache resolves mamba_radix_cache_strategy == "auto" to "extra_buffer" when wants_overlap (overlap schedule on by default) and the model supports it — with no ROCm guard.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [HIGH] Commit 66a7fd5c "refactor: mamba radix cache server args initialize" (#28151) introduced the mamba_radix_cache_strategy="auto" path and removed the explicit # On ROCm, extra_buffer is unsupported. fallback that previously routed ROCm to no_buffer.
  • Supporting: (a) lands 2026-06-18 20:02 UTC, inside the regression window and after the last passing nightly (Jun 18 run started 18:17 UTC, head f7632ef8, which excludes the commit) and before the first failing nightly (Jun 19, head ca88b7f1); (b) the commit diff deletes the line # On ROCm, extra_buffer is unsupported. and rewrites the exact failing functions _handle_mamba_radix_cache / _validate_mamba_extra_buffer; (c) on ROCm with overlap-on + triton linear-attn, auto now resolves to extra_buffer, hitting the is_cuda() or is_musa() or is_npu() assertion. Three independent evidence chains converge.
  • Disconfirming: (none found) — the Jun 18 pass / Jun 19 fail boundary aligns exactly with the commit's merge time, and the removed ROCm guard is the precise behavior that previously avoided this assertion.
  • How to verify: on a branch, set --mamba-radix-cache-strategy no_buffer (or revert 66a7fd5c) and re-run test_qwen35_eval_mi35x.py on AMD MI35x / ROCm 7.2; the server should start.

In-flight Fix Check

• In-flight fix: ✅ #27141 "[ROCm] Support radix-cache with mamba-extra-buffer for Qwen3.5" (open since 2026-06-03) — adds ROCm/FLA support for mamba extra_buffer, which is exactly what the failing assertion blocks. Chase reviewers; do NOT open a duplicate. (Note: it predates the regression, so it is an enabling feature rather than a direct revert — confirm it actually relaxes the is_cuda()/is_musa()/is_npu() assertion for ROCm.)
• Searched mamba+extra_buffer+ROCm, mamba+radix+rocm, extra_buffer+FLA via the issues search API.

Suggested Triage Steps

• Quick unblock test: re-run the qwen35 job with --mamba-radix-cache-strategy no_buffer to confirm it bypasses the assertion (validates the hypothesis cheaply).
• Confirm culprit: revert 66a7fd5c (#28151) on a branch and re-run test_qwen35_eval_mi35x.py on AMD MI35x / ROCm 7.2.
• Preferred fix direction (hypothetical): restore a ROCm fallback in _handle_mamba_radix_cache so auto resolves to no_buffer when not is_cuda()/is_musa()/is_npu(), OR land #27141 to genuinely support extra_buffer/FLA on ROCm.

Status

• Persistence: Failing for 5 consecutive completed scheduled runs since 2026-06-19 (first fail run 27841393725); every completed run since has failed identically.
• Scope: At minimum the nightly-8-gpu-mi35x-qwen35-rocm720 job on nightly-test-amd-rocm720.yml. The sibling nightly-8-gpu-qwen35-rocm720 job also shows failure in the same runs and likely shares this cluster (not separately verified here). Any AMD/ROCm nightly running a hybrid-mamba model (Qwen3.5) through the auto radix-cache path is exposed.
• Blocked work: Qwen3.5 accuracy (lm_eval) signal on AMD MI35x / ROCm 7.2.
• In-flight fix: ✅ #27141 (open since 2026-06-03) — needs review/merge or confirmation it covers this assertion.

nightly-8-gpu-mi35x-deepseek-r1-hicache-rocm720 — failed step(s): Accuracy Test MI35x ROCm 7.2 (8-GPU DeepSeek-R1 HiCache)

Commit Info

• sglang: ed26a109
• aiter: 7d604afe (v0.1.14rc1.dev240+g7d604afe5; source: dev/patched image build preserved — [CI-AITER-CHECK] reported "Dev/patched version detected → skipping rebuild")

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/test_deepseek_r1_hicache_mi35x.py	(whole-file run; benchmark phase)	Memory access fault by GPU on all 8 nodes → scheduler crashed with exit code -6 (SIGABRT) → client hangs → TIMEOUT after 3600s	Log

Failure Summary

The DeepSeek-R1-0528 hicache benchmark on 8×MI35x (ROCm 7.2) started serving at 01:07. Within ~90 s, during a large chunked-prefill forward pass (GEMM shape M:31120, N:256, K:7168), all 8 GPUs raised Memory access fault by GPU node-N ... Reason: Unknown, the scheduler subprocess aborted with exit code -6, and the benchmark client then looped on connection errors ("Rate limit exception so wait and retry") until the 3600 s harness timeout fired (0/1 passed). The fault is FACT; the underlying cause is not established here.

Regression Status

Known recurring failure / Never-passed in queryable window. The job nightly-8-gpu-mi35x-deepseek-r1-hicache-rocm720 has failed on every completed scheduled run in the window, and the GPU-memory-access-fault symptom is present in every completed log inspected from 2026-06-12 onward. No green run was found in the queryable window (back to 2026-05-31, all failure/cancelled).

Recent history of test_deepseek_r1_hicache_mi35x.py in job nightly-8-gpu-mi35x-deepseek-r1-hicache-rocm720 (branch main, event=schedule, completed runs only):

Date	Run	Test File Status	Symptom
Jun 23	28046795462	❌ Failed	8× mem access fault → exit -6 → timeout
Jun 22	27975850577	❌ Failed	timeout (no mem-fault line in log)
Jun 21	27912828989	❌ Failed	8× mem access fault → exit -6 → timeout
Jun 20	27879239829	❌ Failed	8× mem access fault → exit -6 → timeout
Jun 14	27507292665	❌ Failed	8× mem access fault → exit -6 → timeout
Jun 13	27474496879	❌ Failed	8× mem access fault → exit -6 → timeout
Jun 12	27434236636	❌ Failed	8× mem access fault → exit -6 → timeout

(First observed in queryable window: ≤ 2026-06-12. Last known passing: none found back to 2026-05-31. Jun 19/18/17/16 runs were cancelled or timed out without a captured mem-fault line and are not clean comparisons.)

Failure Cluster

GPU memory access fault during DeepSeek-R1 fp8-blockscale prefill GEMM on MI35x.

Facts (verified observations)

• Verbatim: Memory access fault by GPU node-7 ... on address 0x7e6772617000. Reason: Unknown. (and 7 sibling nodes), followed by Fatal Python error: Aborted and Subprocess scheduler_0 (pid=3331) crashed with exit code -6.
• The crashing main-thread stack (faulthandler dump) is in the prefill forward path:
  • model_runner.py:3065 _forward_raw → eager_runner.py:404 _execute_extend → deepseek_v2.py:2767/2504/2138/1789 forward → deepseek_v2.py:1894 forward_core → forward_mha.py:315 forward_normal_core → linear.py:1540 forward → fp8.py:838 apply → fp8_utils.py:857 aiter_w8a8_block_fp8_linear → aiter triton gemm_a8w8_blockscale.py:105.
• Verified at HEAD: fp8_utils.py:857 is the gemm_a8w8_blockscale_op(...) call site (the fp8 block-scale GEMM).
• The fault occurred on a large chunked-prefill batch: the immediately preceding aiter log lines report GEMM M:31120, N:256, K:7168 dtype=bfloat16 ... will use default config! using torch solution:0 (no tuned config found).
• aiter is a dev/patched build baked into the ROCm-7.2 MI35x image (7d604afe), preserved without rebuild.
• Reproducibility: same test file, same 8-GPU mem-fault → exit-6 → timeout pattern in every completed log from Jun 12 to Jun 23 inspected.
• An active debug branch exists upstream: origin/bingxche/debug-dsv32-mi35x-mtp-mem-access-fault (seen in the checkout's branch list), indicating an AMD engineer is already investigating an MI35x mem-access-fault.

Hypothesised Causes (with confidence)

• [LOW] Illegal memory access inside the aiter fp8 block-scale GEMM path (gemm_a8w8_blockscale) for large prefill shapes (M≈31k) on gfx950/MI35x.
  • Supporting: the faulting forward pass is the last operation before the fault; the GEMM ran with default config / using torch solution:0 (no tuned config for this shape), and the same path appears in every failing log.
  • Disconfirming: the [aiter] ... using torch solution:0 print is one log line before the fault and is NOT proof of causation (log-line proximity ≠ causation); the fault may be in a subsequent kernel. aiter is unchanged across the failing window (same dev build), yet the failure persists — so an aiter delta does not explain onset.
  • How to verify: re-run test_deepseek_r1_hicache_mi35x.py on MI35x with attention_backend / fp8 GEMM runner forced to the CK path (vs triton) to isolate the GEMM backend; capture an ROCm coredump (GPU core dump failed here — enable AMD_LOG_LEVEL/HSA_ENABLE_COREDUMP).
• [LOW] Hierarchical-cache (hicache) specific code path interacting with the prefill (this is the only MI35x DeepSeek job that enables enable_hierarchical_cache=True, hicache_io_backend=kernel).
  • Supporting: cache-controller threads (cache_controller.py prefetch/backup) are live in the dump; the non-hicache DeepSeek MI35x jobs are not part of this failure signature.
  • Disconfirming: the faulting stack is in the attention/linear GEMM, not in cache-controller code; the cache threads are merely blocked on queue.get.
  • How to verify: compare against the same model run with hierarchical cache disabled on MI35x.
• No hypothesis above LOW. Because every completed run in the window already fails with the same symptom and aiter is unchanged, there is no clean pass_sha..fail_sha regression window inside the queryable range; this is a long-standing AMD-only failure rather than a fresh regression. Needs a coredump + backend A/B, not a bisect.

In-flight Fix Check

• In-flight fix: ❌ none found directly targeting the DeepSeek-R1 hicache MI35x memory-access fault. Searched open PRs for mi35x memory access fault, deepseek hicache mi35x, mi35x mem access — nearest matches are unrelated (#27650 PCG/aiter fused all-reduce+RMSNorm; #27209 gfx950 Dockerfile; #28787 RMSNorm batch-invariance). An upstream debug branch bingxche/debug-dsv32-mi35x-mtp-mem-access-fault exists but is not an open PR — chase that author for status before opening a new investigation.

Suggested Triage Steps

• Capture a real GPU coredump: the run printed GPU core dump failed; re-run with HSA_ENABLE_COREDUMP=1 / AMD_LOG_LEVEL=3 on MI35x to localize the faulting kernel beyond the Python frame.
• Backend A/B on the fp8 block-scale GEMM: force CK vs triton (fp8_gemm_runner_backend) and re-run test_deepseek_r1_hicache_mi35x.py; if CK is clean, the fault is in the aiter triton gemm_a8w8_blockscale path for large-M prefill.
• hicache A/B: re-run the same model with --enable-hierarchical-cache off to confirm whether the fault is hicache-specific or generic DeepSeek-R1 prefill on MI35x.
• Coordinate with the owner of bingxche/debug-dsv32-mi35x-mtp-mem-access-fault — an active debug effort for this exact symptom likely already has narrowing data.
• Consider quarantining/marking this nightly job as known-failing on AMD until a fix lands, so it stops masking other regressions in the rocm720 nightly signal.

Status

• Persistence: Failing on every completed scheduled run in the queryable window; GPU-memory-access-fault symptom present in every completed log from 2026-06-12 through 2026-06-23. No green run found back to 2026-05-31.
• Scope: 1 job (nightly-8-gpu-mi35x-deepseek-r1-hicache-rocm720) in nightly-test-amd-rocm720.yml. Symptom is specific to MI35x + DeepSeek-R1 hicache; not observed as the same cluster in other jobs here.
• Blocked work: Blocks DeepSeek-R1 hicache nightly accuracy/perf signal on MI35x / ROCm 7.2.
• In-flight fix: ❌ none found in open PRs; active debug branch bingxche/debug-dsv32-mi35x-mtp-mem-access-fault exists (not yet a PR).

nightly-8-gpu-mi35x-kimi-k26-rocm720 — failed step(s): Accuracy Test MI35x ROCm 7.2 (8-GPU Kimi-K2.6)

Commit Info

• sglang: ed26a109
• aiter: 7d604afe (v0.1.14rc1.dev240+g7d604afe5, source: dev image — [CI-AITER-CHECK] Dev/patched version detected → skipping rebuild)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi35x/test_kimi_k26_eval_mi35x.py	TestKimiK26EvalMI35x.test_kimi_k26_gsm8k_accuracy	TimeoutError: Server failed to start within the timeout period → file-level TIMEOUT after 7200s, exit 255	Log

Failure Summary

The single accuracy test test_kimi_k26_gsm8k_accuracy never ran its GSM8K eval because the sglang server for moonshotai/Kimi-K2.6 failed to start. The server launch was attempted twice (initial + 1 retry); each attempt hung during HuggingFace weight download and the file-level 7200s budget was exhausted, producing a TIMEOUT and exit code 255. No accuracy score was ever produced.

Regression Status

Known recurring failure / Infrastructure issue — failing every completed scheduled run in the queryable window with the identical download-hang symptom.

Recent history of test_kimi_k26_eval_mi35x.py (test_kimi_k26_gsm8k_accuracy) in job nightly-8-gpu-mi35x-kimi-k26-rocm720 (nightly-test-amd-rocm720.yml, event=schedule, branch=main, completed runs only):

Date	Run	Test File Status	Symptom
Jun 24	28046795462	❌ Failed	stuck in download_weights_from_hf filelock → 7200s timeout
Jun 22	27975850577	❌ Failed	same: Server failed to start + 7200s timeout
Jun 21	27912828989	❌ Failed	same: _poll_until_acquired / download_weights_from_hf + 7200s timeout
Jun 20	27879239829	❌ Failed	(job failed; same job)
Jun 16–19	runs	⏭️ Cancelled	run-level cancelled, not comparable

No passing run found in the queryable window (lookback Jun 12–24; Jun 16–19 runs were cancelled). First failure I could confirm with this exact symptom: Jun 20.

Failure Cluster

Server start timeout — Kimi-K2.6 HF weight download never completes (filelock wait)

Facts (verified observations, no interpretation)

• All 8 scheduler ranks (TP0–TP7) are blocked at the same frame in the pyspy dump: _poll_until_acquired (filelock/_api.py:552) ← acquire (filelock/_api.py:506) ← download_weights_from_hf (model_loader/weight_utils.py:554) ← stream_language_weights (kimi_k25.py:780) ← load_model (model_runner.py:1423).
• The main process (pid 3173) is idle in _wait_for_scheduler_ready (engine.py:1355) — it never receives a scheduler-ready signal.
• Load weight begin. avail mem=276.xx GB is printed for all ranks at 01:20:04; no Load weight end is ever printed in either attempt.
• HF was actively pulling files: [transformers] A new version of the following files was downloaded from https://huggingface.co/moonshotai/Kimi-K2.6 — i.e. weights were NOT pre-cached on the runner.
• Scheduler watchdog timeout (self.watchdog_timeout=300, self.soft=True) fires repeatedly (soft → log only, does not kill) throughout both attempts.
• Attempt 1: ~01:19:20 → TimeoutError at 02:49:31 (~5400s, the SERVER_LAUNCH_TIMEOUT). Attempt 2 (retry) ran 02:49 → killed by file-level 7200s timeout at 03:19:05.
• py-spy dump --native for one rank repeatedly fails (Failed to get stack traces) — consistent with a process blocked in a native/syscall (network/file I/O), not a Python deadlock.
• Test config (test_kimi_k26_eval_mi35x.py:35): SERVER_LAUNCH_TIMEOUT = 5400; suite invoked with --timeout-per-file 7200. Two 5400s launch attempts cannot both fit inside one 7200s file budget.
• AITER_COMMIT_OVERRIDE empty; this is a regular nightly run (not AITER Scout), so no baseline A/B applies.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [MEDIUM] Kimi-K2.6 weights are downloaded fresh from HuggingFace on each run and the download does not complete within the per-attempt 5400s / per-file 7200s budget (slow/degraded network or an extremely large checkpoint not pre-cached on the MI35x runner).
  • Supporting: all ranks parked in download_weights_from_hf filelock wait; transformers logs an active download of new files; no Load weight end; identical symptom on Jun 20/21/22/24; native py-spy fails (blocked in I/O).
  • Disconfirming: I did not capture a byte-rate / progress-bar line proving the download was advancing (vs. truly stalled), so a genuine stall on the filelock holder cannot be excluded.
  • How to verify: on the runner, time a standalone huggingface-cli download moonshotai/Kimi-K2.6 and check whether the local HF cache is persisted across runs; inspect for a stale *.lock in the HF cache dir.
• [LOW] Filelock contention/deadlock among the 8 TP ranks (lock holder died or lock is stale, so the others poll forever).
  • Supporting: every rank is in _poll_until_acquired; none visibly shows an active download frame.
  • Disconfirming: pyspy cannot see native frames, so the lock holder may simply be downloading inside a C call invisible to the dump; no "lock held by dead pid" evidence in the log.
  • How to verify: enable HF/filelock debug logging; check whether exactly one rank ever progresses past the lock.
• [SPECULATION] sglang code regression in the Kimi-K2.6 load path (kimi_k25.py / deepseek_weight_loader.py).
  • Disconfirming: the stack is stuck strictly inside the download step (before any sglang-specific weight parsing), and the symptom predates this commit (Jun 20+) across many different head SHAs — points to environment, not a single commit.

In-flight Fix Check

• Searched open PRs: kimi download (7 results, all unrelated — Kimi tool-call/video features, not CI download), and weight download filelock (0 results).
• In-flight fix: ❌ none found — needs new PR or investigation (no PR addressing Kimi-K2.6 download timeout / weight-cache on MI35x).

Suggested Triage Steps

• Confirm whether the MI35x ROCm 7.2 runner persists the HuggingFace cache between nightly runs; if not, pre-stage moonshotai/Kimi-K2.6 weights on the runner (or a shared mount) so the test does not download ~full checkpoint each run.
• On the runner, run huggingface-cli download moonshotai/Kimi-K2.6 standalone and measure wall-clock + throughput to decide whether 7200s is simply insufficient vs. a hard stall.
• If it is a genuine filelock stall, check the HF cache dir for a stale .lock and test with a single rank (--tp 1) to isolate inter-rank lock contention.
• Only if a stall (not slowness) is confirmed: capture an HF/filelock debug-logged run to see which rank holds the lock; consider whether all 8 ranks should contend on the same download lock.
• Tune the budget last (it masks, not fixes): the sister nightly-8-gpu-kimi-k26-rocm720 job shows the same download_weights_from_hf pattern, so a budget bump alone won't help if the download truly cannot finish.

Status (factual, no priority assignment)

• Persistence: failing every completed scheduled run with this symptom since at least Jun 20 2026 (Jun 20/21/22/24; Jun 16–19 runs were cancelled).
• Scope: affects both nightly-8-gpu-mi35x-kimi-k26-rocm720 and the sibling nightly-8-gpu-kimi-k26-rocm720 job in nightly-test-amd-rocm720.yml (both show the same download_weights_from_hf hang).
• Blocked work: no Kimi-K2.6 GSM8K accuracy signal on AMD MI35x ROCm 7.2 nightly.
• In-flight fix: ❌ none found in open PRs.

nightly-8-gpu-mi35x-deepseek-v4-pro-rocm720 — failed step(s): Accuracy + Performance Test MI35x ROCm 7.2 (8-GPU DeepSeek-V4-Pro FP8 + FP4, triton), Accuracy + Performance Test MI35x ROCm 7.2 (8-GPU DeepSeek-V4-Pro FP8 + FP4, unified_kv_triton)

Commit Info

• sglang: ed26a109
• aiter: 7d604afe (v0.1.14rc1.dev240+g7d604afe5; source: dev image preserved, matches Dockerfile default; AITER_COMMIT_OVERRIDE empty)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/test_deepseek_v4_pro_fp8.py	setUpClass (server launch / weight load)	Server never became ready — scheduler stuck in download_weights_from_hf; job cancelled at 6h wall-clock	Job

This is not a test-assertion failure. The job was cancelled (conclusion cancelled) after step 6 ("8-GPU DeepSeek-V4-Pro FP8 + FP4, unified_kv_triton") hung for ~1.5 h and hit the 6-hour wall-clock limit; step 7 (triton) was then cancelled too. The two preceding sub-tests in step 6 actually passed.

Failure Summary

Step 6 runs several DeepSeek-V4-Pro sub-tests sequentially. The FP8 (Accuracy: 0.953 at 02:41, 0.941 at 05:34) and FP4 CP-interleave (gsm8k_score=0.9467 at 06:17) sub-tests passed. The terminal failure is in the next sub-test (test_deepseek_v4_pro_fp8.py): at 06:18:19 CI cache validation deleted the entire local cache for sgl-project/DeepSeek-V4-Pro-FP8, forcing a from-scratch re-download. That re-download stalled in the HuggingFace xet_get backend and never completed, producing continuous watchdog timeouts from 06:22:50 until GitHub cancelled the job at 07:45:39 (##[error]The operation was canceled.).

Regression Status

Flaky / infrastructure (download stall) — not a code regression.

Recent history of this job (nightly-8-gpu-mi35x-deepseek-v4-pro-rocm720, branch main, event schedule, completed runs only):

Date	Run	Job	Status	Note
Jun 24	28046795462	job	⏹️ Cancelled	FP8 cache wiped → re-download stalled in xet_get → 6h timeout
Jun 22	27975850577	job	❌ Failure	Same FP8 download-hang symptom (531 watchdog timeouts), but the download eventually completed (Accuracy: 0.941); final failure elsewhere
Jun 21	27912828989	job	✅ Success	—
Jun 20	27879239829	job	✅ Success	—

The xet_get download stall on sgl-project/DeepSeek-V4-Pro-FP8 recurs (Jun 22 + Jun 24); whether the job passes depends on whether the multi-hundred-GB re-download finishes inside the 6h budget.

Failure Cluster

HF weight re-download stall (xet_get) after CI cache invalidation → job 6h-timeout/cancel

Facts (verified observations)

• Job conclusion cancelled; ran 01:45:26 → 07:45:41 UTC (6h). Step 6 failure, step 7 cancelled.
• 02:41 Accuracy: 0.953; 05:34 Accuracy: 0.941; 06:17 [METRIC] gsm8k_score=0.9467 (deepseek-ai/DeepSeek-V4-Pro) — three sub-tests passed.
• 06:18:19 UTC [TP7] Removing entire cache for sgl-project/DeepSeek-V4-Pro-FP8 at /sgl-data/hf-cache/hub/...; 06:18:21 Process 30189 ACQUIRED lock for sgl-project/DeepSeek-V4-Pro-FP8.
• Continuous watchdog timeouts begin 06:22:50 (TokenizerManager watchdog timeout, self.watchdog_timeout=300, self.soft=True) and repeat every ~5 min until cancellation.
• Terminal pyspy stacks (e.g. 07:43:04 TP3): lock holder in xet_get (huggingface_hub/file_download.py:566) ← ci_download_with_validation_and_retry (model_loader/ci_weight_validation.py:1955) ← download_weights_from_hf (weight_utils.py:593); all other TPs blocked at download_weights_from_hf (weight_utils.py:554) → get_lock/filelock acquire. (Confirmed against source: weight_utils.py:554 = with get_lock(...), :593 = ci_download_with_validation_and_retry.)
• 07:45:39 ##[error]The operation was canceled.
• No ConnectionError/HTTPError/429/HIP error in the log — the download is silently stalled, not erroring.

Hypothesised Causes (with confidence)

• [HIGH] Terminal cancellation is caused by a stalled from-scratch HF re-download of sgl-project/DeepSeek-V4-Pro-FP8 via the xet backend that did not finish within the 6h job budget.
  • Supporting: cache for that repo was explicitly deleted at 06:18:19; the lock-holder pyspy is parked in xet_get→select for ~1.5h with zero progress lines; all sibling TPs block on the same download lock; identical xet_get stall recurred Jun 22.
  • Disconfirming: the same repo loaded fine earlier the same run (05:34 Accuracy: 0.941), so the network path is not universally broken — points to an intermittent xet stall + an over-eager cache-invalidation that throws away a known-good cache.
  • How to verify: re-run the job and watch xet_get throughput; or set HF_HUB_DISABLE_XET=1 / HF_HUB_ENABLE_HF_TRANSFER=1 for the FP8 sub-test and confirm the download completes.
• [MEDIUM] CI cache-validation logic (model_loader/ci_weight_validation.py) wiping the entire FP8 cache mid-run is the trigger that converts a slow link into a fatal timeout.
  • Supporting: the cache was valid/used at 05:34, then "Removing entire cache" at 06:18 forced a full re-download instead of an incremental/cached load.
  • Disconfirming: validation deleting a genuinely-corrupt cache may be correct behaviour; the deletion itself isn't proven wrong, only costly.
  • How to verify: inspect why validation flagged the FP8 snapshot as invalid (requires_hf_quant_config path) right after a passing run; check for partial/corrupt snapshot.
• [LOW] aiter/sglang code change. Stack trace is entirely in HF download/filelock; no model-forward or kernel frame involved. Effectively excluded as a cause of the cancellation.

In-flight Fix Check

• In-flight fix: ❌ none directly matching — xet download-stall / FP8 cache-wipe. Closest related open PR is #23400 ("ci: prefetch HF snapshots and skip redundant metadata calls to survive 429s"), which improves HF download resilience but does not address xet stalls or the full-cache deletion. (Searched is:open PRs for xet+download, cache+validation+download, DeepSeek-V4-Pro+download, weight+download+hang.)

Suggested Triage Steps

• Re-run the job; if the FP8 sub-test downloads and passes, confirms intermittent xet/network stall (flaky infra).
• Inspect model_loader/ci_weight_validation.py to learn why sgl-project/DeepSeek-V4-Pro-FP8 is flagged invalid right after a passing run, and consider not deleting the entire cache (prefer incremental repair) so a transient validation miss can't force a multi-hundred-GB re-download.
• For this large-model nightly, consider HF_HUB_DISABLE_XET=1 or hf_transfer, and/or pre-staging the FP8 weights on the MI35x runner so step 6 never downloads at test time.
• Optionally raise the per-job wall-clock or split FP4/FP8 into separate jobs so a download stall in one sub-test doesn't cancel the rest.

Status

• Persistence: Download-stall symptom seen on this job Jun 22 and Jun 24; Jun 24 ended in cancellation, Jun 22 recovered. Jun 20–21 green.
• Scope: This job only (nightly-8-gpu-mi35x-deepseek-v4-pro-rocm720), step 6; step 7 collaterally cancelled. Sister MTP job passed Jun 22.
• Blocked work: DeepSeek-V4-Pro FP8/FP4 8-GPU MI35x ROCm 7.2 nightly accuracy/perf signal.
• In-flight fix: ❌ none directly matching; #23400 is adjacent (HF download resilience) but insufficient.

nightly-8-gpu-qwen35-rocm720 — failed step(s): Accuracy Test ROCm 7.2 (8-GPU Qwen 3.5)

Commit Info

• sglang: ed26a109
• aiter: N/A (no [CI-AITER-CHECK] markers in log; AITER_COMMIT_OVERRIDE empty → Dockerfile default, not surfaced in this nightly job)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi30x/test_qwen35_eval_amd.py	TestQwen35EvalAMD.setUpClass	AssertionError: extra_buffer needs CUDA/MUSA/NPU (FLA). (server exits code 1 → setUpClass raises)	Log

Failure Summary

The single registered test test_qwen35_eval_amd.py failed in setUpClass: the sglang serve subprocess for Qwen/Qwen3.5-397B-A17B (tp=8, --attention-backend aiter) exited with code 1 during server-arg validation. The server crashed inside ServerArgs.__post_init__ → _handle_mamba_radix_cache → _validate_mamba_extra_buffer with AssertionError: extra_buffer needs CUDA/MUSA/NPU (FLA).. The build (sgl-kernel, mori) completed fine; this is a runtime server-launch failure, not a build or accuracy failure.

Regression Status

Known recurring failure / Never-passed on AMD ROCm (no green run found in the queryable schedule window; earlier runs in the lookback were cancelled, not passing).

Recent history of test_qwen35_eval_amd.py (TestQwen35EvalAMD.setUpClass) in job nightly-8-gpu-qwen35-rocm720 (branch main, event schedule, completed runs only):

Date	Run	Job	Test File Status	Failed Function	Error
Jun 24	28046795462	qwen35-rocm720	❌ Failed	setUpClass	extra_buffer needs CUDA/MUSA/NPU (FLA).
Jun 22	27975850577	qwen35-rocm720	❌ Failed	setUpClass	extra_buffer needs CUDA/MUSA/NPU (FLA).
Jun 21	27912828989	qwen35-rocm720	❌ Failed	setUpClass	extra_buffer needs CUDA/MUSA/NPU (FLA).
Jun 20	27879239829	qwen35-rocm720	❌ Failed	setUpClass	(same job failing)

(The Jun 16–19 schedule runs were cancelled and excluded. Verified the identical extra_buffer needs CUDA/MUSA/NPU (FLA). assertion in the Jun 22 and Jun 21 logs.)

Failure Cluster

Mamba extra_buffer radix-cache assertion on ROCm — server launch aborts for hybrid-mamba Qwen3.5 because extra_buffer strategy is hard-gated to CUDA/MUSA/NPU.

Facts (verified observations)

• Server command (from log line 3802): sglang serve --model-path Qwen/Qwen3.5-397B-A17B --tp 8 --attention-backend aiter --trust-remote-code --model-loader-extra-config '{"enable_multithread_load": true}' --watchdog-timeout 1200 --device cuda --host 127.0.0.1 --port 11000.
• Verbatim stack (log lines 3787–3800): server_args.py:6524 from_cli_args → __init__ → server_args.py:2541 __post_init__ → _handle_model_specific_adjustments → server_args.py:4415 _handle_mamba_radix_cache → server_args.py:4386 _validate_mamba_extra_buffer → AssertionError: extra_buffer needs CUDA/MUSA/NPU (FLA).
• Source (python/sglang/srt/server_args.py:4386-4388): the assertion is assert (is_cuda() or is_musa() or is_npu()), "extra_buffer needs CUDA/MUSA/NPU (FLA).". On an AMD ROCm/HIP build is_cuda() is False, so the assert always fires even though the launch passes --device cuda.
• The auto selection in _handle_mamba_radix_cache (server_args.py:4403-4412) picks "extra_buffer" whenever _support_mamba_cache_extra_buffer(model_arch) is True and overlap/paging is wanted — this predicate is architecture-based, not platform-based, so ROCm gets routed into the unsupported extra_buffer path.
• Reproducibility: same test file + same setUpClass + same assertion in all 3 completed schedule runs in the window (Jun 21/22/24).
• Not an aiter-override / scout run; AITER_COMMIT_OVERRIDE empty.

Hypothesised Causes (with confidence)

• [HIGH] The mamba_radix_cache_strategy="auto" logic selects extra_buffer for Qwen3.5 hybrid-mamba on ROCm, but _validate_mamba_extra_buffer (server_args.py:4386) only permits CUDA/MUSA/NPU, so the server aborts before it can serve.
  • Supporting: full stack trace lands exactly on that assert; is_cuda() is False on ROCm; the model arch is mamba-hybrid; identical across 3 runs; the in-flight fix #29158 targets precisely this (fall back to no_buffer on ROCm).
  • Disconfirming: (none found) — the assertion message and code path are unambiguous and the failure is fully reproducible.
  • How to verify: launch the same command in container rocm/sgl-dev:v0.5.13.post1-rocm720-mi30x and confirm the assert; then apply #29158 (force no_buffer on ROCm) and re-run test_qwen35_eval_amd.py.

In-flight Fix Check

• In-flight fix: ✅ #29158 "[AMD] Fall back to mamba no_buffer radix cache on ROCm instead of asserting (fixes nightly qwen35)" — open (non-draft) since 2026-06-24. Directly addresses this failure; chase reviewers, do NOT open a duplicate.
• Related (alternative approach): #27141 "[ROCm] Support radix-cache with mamba-extra-buffer for Qwen3.5" — would instead enable extra_buffer on ROCm.
• Search: repo:sgl-project/sglang is:pr is:open extra_buffer / qwen35 amd (API queries above).

Suggested Triage Steps

• Verify #29158 merges cleanly and re-run nightly-8-gpu-qwen35-rocm720 (and the sister nightly-8-gpu-mi35x-qwen35-rocm720, which fails identically) to confirm it clears the assertion.
• Decide between the two approaches: fall back to no_buffer on ROCm (#29158) vs. enabling extra_buffer FLA on ROCm (#27141); the former unblocks CI immediately, the latter preserves overlap/paging performance.
• If neither lands soon, consider quarantining test_qwen35_eval_amd.py on AMD nightly to recover signal, since it has never passed in the queryable window.

Status (factual)

• Persistence: Failing in every completed schedule run in the lookback window (Jun 21, 22, 24); no green run found — likely never-passed on AMD ROCm since the Qwen3.5 nightly job was added.
• Scope: Both nightly-8-gpu-qwen35-rocm720 and its mi35x sibling nightly-8-gpu-mi35x-qwen35-rocm720 fail with this assertion in the same runs. NVIDIA unaffected (assert only triggers on non-CUDA builds).
• Blocked work: Blocks all Qwen3.5-397B accuracy signal on AMD ROCm 7.2 nightly.
• In-flight fix: ✅ #29158 (open since 2026-06-24) — awaiting merge.

nightly-accuracy-2-gpu-rocm720 — failed step(s): N/A

Commit Info

• sglang: d6aacd28
• aiter: 7d604afe5 (v0.1.14rc1.dev240+g7d604afe5; source: dev image preserved — [CI-AITER-CHECK] reports "Dev/patched version detected → skipping rebuild", so no override and no rebuild)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi30x/test_gsm8k_eval_amd.py	TestNightlyGsm8KEval.test_gsm8k_all_models (subTest mistralai/Mistral-7B-Instruct-v0.3)	AssertionError: The following models failed: — Mistral-7B GSM8K score 0.369 < threshold 0.47	Log

Failure Summary

The 2-GPU nightly GSM8K accuracy suite (test_gsm8k_all_models) ran 5 instruction-tuned models and asserts at the end (line 201) if any model is below its threshold. mistralai/Mistral-7B-Instruct-v0.3 scored 0.369 vs a 0.47 threshold across all 3 retry attempts (0.371 / 0.370 / 0.369), which raises the end-of-test AssertionError and fails the job. DeepSeek-Coder-V2-Lite (0.819≥0.81), gemma-2-27b-it (0.861≥0.86), and Llama-3.1-70B (0.929≥0.89) passed.

Note: the GitHub log for step 6 is only retrievable up to 20:36:21 (truncated mid-Mixtral-8x7B eval; the job actually completed at 20:53:13). mistralai/Mixtral-8x7B-Instruct-v0.1 (threshold 0.69) and Qwen/Qwen3-30B-A3B-Thinking-2507 final scores are not visible in the retrievable log. Based on the prior-run pattern (Mixtral scored 0.296–0.315 in every prior run), Mixtral very likely also failed today, but I cannot confirm it from this run's log.

Regression Status

Known recurring failure / Never-passed for Mistral-7B-Instruct-v0.3 in the queryable window. Stable score ~0.36–0.37, consistently ~0.10 below the 0.47 threshold — not a fresh regression.

Recent history of test_gsm8k_eval_amd.py (test_gsm8k_all_models, Mistral-7B-Instruct-v0.3 subTest) in job nightly-accuracy-2-gpu-rocm720 (branch main, event schedule, completed runs only):

Date	Run	Mistral-7B Score	Test Status	Error
Jun 24	28119401197	0.369	❌ Failed	< 0.47 threshold
Jun 23	28046795462	0.361	❌ Failed	< 0.47 threshold
Jun 22	27975850577	0.362	❌ Failed	< 0.47 threshold
Jun 21	27912828989	0.363	❌ Failed	< 0.47 threshold
Jun 20	27879239829	0.371	❌ Failed	< 0.47 threshold

No passing run found in the queryable window (Jun 20–24; Jun 16–19 runs were cancelled). First-observed-passing date is beyond this window — treat as Never-passed for at least 5 days.

Failure Cluster

GSM8K accuracy below threshold (Mistral/Mixtral chat-eval undercount on AMD)

Facts (verified observations)

• Mistral-7B GSM8K, today's 3 attempts: 0.3714, 0.3699, 0.3691 → reported 0.369 (verbatim: Score: 0.369 (threshold: 0.47) / Status: ❌ FAILED).
• Threshold source: MODEL_SCORE_THRESHOLDS["mistralai/Mistral-7B-Instruct-v0.3"] = 0.47 with comment "52.1% − 5%" (test_gsm8k_eval_amd.py:45). Thresholds are derived from reported 5-shot/CoT scores, but the AMD test runs the chat-completion eval (eval_name="gsm8k", test_gsm8k_eval_amd.py:265).
• Assertion site: check_model_scores raises AssertionError(f"The following models failed:...") (test_gsm8k_eval_amd.py:199-201).
• Score is stable across 5 consecutive runs (0.361–0.371), independent of the changing sglang HEAD SHA — argues against a code regression.
• aiter version unchanged (dev image v0.1.14rc1.dev240+g7d604afe5, no override) — not an aiter delta.
• Mixtral-8x7B (threshold 0.69) scored 0.296–0.315 in prior runs (also ❌); gemma-2-27b-it sits right on its 0.86 threshold (0.857–0.861, flips pass/fail run-to-run).

Hypothesised Causes (with confidence)

• [HIGH] The Mistral/Mixtral failures are a test-harness / eval-extraction + threshold-calibration issue, not a model or kernel regression.
  • Supporting: score is stable ~0.37 across 5 runs with different HEAD SHAs and unchanged aiter; the threshold (0.47) is back-derived from a 5-shot/CoT reference while the eval uses chat completions with last-number extraction; open PR #27757 documents the same root cause ("chat API with completion-style few-shot prompt and last-number extraction… Mistral-family responses continue into additional Q/A text, so last-number extraction is wrong"; rerun scored 0.417 even on FlashInfer).
  • Disconfirming: I cannot rule out a real numerical accuracy gap on AMD MI30x without an A/B vs an NVIDIA run of the same eval; PR #27757's own note says moving off FA3 alone did not recover accuracy, so part of the gap may be genuine.
  • How to verify: run the same test_gsm8k_all_models Mistral subTest on NVIDIA with identical eval settings; or apply PR #27757's prompt/extraction change and re-run on MI30x to see if score crosses 0.47.
• [LOW] Genuine AMD-backend accuracy degradation (aiter attention/MoE path) for Mistral/Mixtral.
  • Supporting: AMD-only suite; Mixtral is far below threshold (~0.30 vs 0.69).
  • Disconfirming: scores are stable across aiter-unchanged runs; PR #27757 attributes the gap mainly to eval extraction; passing models (Llama-70B 0.929, gemma 0.861) use the same aiter backend.

In-flight Fix Check

• In-flight fix: ✅ #27757 "Fix Mistral GSM8K chat eval" (open since 2026-06-10, updated 2026-06-22, not draft) — makes the GSM8K chat prompt explicit, prefers Answer: <number> extraction, plumbs stop sequences, and keeps Mistral/Mixtral off FA3 auto-default. Directly targets this Mistral/Mixtral chat-eval undercount. Chase reviewers; do NOT open a duplicate. (search: repo:sgl-project/sglang is:pr is:open Mistral gsm8k threshold)

Suggested Triage Steps

• Verify #27757 still merges cleanly against current main and re-run the Mistral/Mixtral subTests on MI30x to confirm scores cross thresholds (0.47 / 0.69).
• If #27757 does not fully recover Mistral to ≥0.47, run the same eval on NVIDIA to separate "eval extraction bug" from "genuine AMD accuracy gap"; if the gap is genuine, the maintainer can decide whether to re-calibrate the AMD thresholds (comment on line 46 already anticipates "lower if AMD scores differently").
• Because the full step-6 log is truncated in the API, pull the job's github_summary.md artifact or the GitHub step-summary table to confirm Mixtral-8x7B and Qwen3-30B final scores for this run.
• Consider whether the gemma-2-27b-it threshold (0.86) is too tight — it oscillates 0.857–0.861 and flips pass/fail across runs (a separate borderline issue, not today's blocker).

Status (factual)

• Persistence: Mistral-7B GSM8K below threshold for ≥5 consecutive completed scheduled runs (since at least 2026-06-20), score stable ~0.36–0.37. No green run in queryable window.
• Scope: nightly-accuracy-2-gpu-rocm720 job of nightly-test-amd-rocm720.yml. The same Mistral/Mixtral chat-eval pattern affects the GSM8K accuracy suite broadly on AMD; Mixtral-8x7B is a second persistent failer in the same test.
• Blocked work: AMD 2-GPU nightly GSM8K accuracy signal is red every night due to this test (the assertion fails the whole job even though 3 other models pass).
• In-flight fix: ✅ #27757 (open since 2026-06-10) — awaiting merge.

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-06-24 23:07 UTC)