[CI Monitor] Daily Report - 2026-07-08

[amd-bot]

Daily Cross-Workflow Summary — 2026-07-08

Snapshot: 2026-07-08 23:52 UTC · Only completed runs counted in trends · Auto-updated every 30 min

TL;DR

🔴 RED · ~11 active clusters · 1 🆕 dominates today (R251: unguarded #include <cooperative_groups.h> in the DSA fused top-k v2 JIT kernel → server crash on ROCm/gfx950 during CUDA-graph capture) · yesterday's R249 cutlass wave RESOLVED (merged #30374 2026-07-07 21:43 — zero cutlass errors in every Jul-08 run) · Zombie-KFD VRAM wave now 6th consecutive day and dominates raw counts.
👉 Today's ask: (1) Merge/expedite ✅ #30506 (open, non-draft, created 2026-07-08 08:28) — disables DSA fused top-k v2 on ROCm and resolves R251, which crashes DeepSeek-V3.2 + GLM-5.1 MTP/accuracy on both nightly-amd and nightly-rocm720 ([HIGH] trigger #30378). (2) Reboot the MI35x zf775 runner pool a 6th day — zombie-KFD aborts (~26 nightly jobs) still mask real MI35x signal (ROCm/aiter#2061). (3) Land ready fixes: mamba R195 ✅ #29965, FP8-HIPBLAS diffusion ✅ #28889.

Workflow status

Workflow	Runs (latest)	✅	❌	7d trend (completed real-code failures)	Δ vs yesterday
nightly-test-amd	2 (Jul-08 28964511688)	0	17 (12 zombie-KFD + 2 R251 + 2 R248 + 1 R195)	3·7·9·~6·~5 real (rest VRAM)	🆕 R251; R249 cutlass gone
nightly-test-amd-rocm720	1 (Jul-08 28964405353)	0	21 (14 zombie-KFD + 2 R251 + 1 R248 + 1 R252 + 1 R239 + 1 MTP-acc)	4·10·~4·9·~6 real (rest VRAM)	🆕 R251/R252; R249 gone
release-docker-amd-nightly	0	—	—	—	no failures reported
release-docker-amd-rocm720-nightly	0	—	—	—	no failures reported
nightly-amd-mi355x-disagg	1 (Jul-08 28917370988)	0	1 (R241 Kimi GSM8K)	10·9·cancel·cancel·1	fresh completed run (Jul-06 was cancelled)
amd-aiter-scout	0 (Mon/Thu cron; today = Wed)	—	—	—	no run this cycle
pr-test-amd	7 (latest Jul-08 28943113233)	0	latest 7 jobs → ~5 clusters	chronic	R249 gone from Jul-08; MORI-clone infra new
pr-test-amd-rocm720	2 (latest Jul-08 28964460204)	0	latest 7 jobs → ~5 clusters	15·15·15·15·7	R249 gone from Jul-08; 45-min timeout new

Notes: (1) R249 cutlass is the good news — merged #30374 (if not is_hip() guard) landed Jul-07 21:43 and every Jul-08 nightly + PR run is cutlass-free. Yesterday's ~29-job wave is gone. (2) R251 is today's story — the v2 fused top-k JIT path (topk_impl.cuh:27 unguarded #include <cooperative_groups.h>, no HIP variant) was newly reached on AMD MTP/accuracy by #30378 (bbc537035, merged Jul-07); v1 has an is_hip_runtime() AOT-op branch, v2 does not. (3) Zombie-KFD (6th consecutive day) still aborts ~26 MI35x nightly jobs pre-flight, inflating counts and masking real signal. (4) aiter unchanged in the regular nightlies (dev image 9127c94a v0.1.17.dev110); no scout run today.

🆕 NEW clusters today

R251 · 🆕 · JIT cooperative_groups.h not found — DSA fused top-k v2 crashes on ROCm/gfx950 — nightly-amd + nightly-rocm720 · ✅ #30506

• Status: NEW today, first executed occurrence 2026-07-08. 4 jobs across nightly-amd (2) + nightly-rocm720 (2). Server SIGKILL (-9) during decode CUDA-graph capture. Last green Jul-04/05; the Jul-06/07 runs were VRAM-gated/cancelled so today is the first run to actually exercise the post-#30378 path.
• Top hypothesis: [HIGH] #30378 (bbc537035 "[DSA] Re-enable fused top-k v2 for MTP", merged Jul-07) removes the allow_topk_v2 guard, routing AMD MTP/spec-verify decode into plan_topk_v2 → _jit_topk_v2_module(), which JIT-compiles topk_impl.cuh — line 27 does an unguarded #include <cooperative_groups.h> (CUDA-only; HIP needs <hip/hip_cooperative_groups.h>). [FACT] underlying latent defect: v2 JIT path has no HIP guard/AOT ROCm op, unlike v1. Disconfirming: the v2 path nominally serves plain decode too, yet the -basic sibling passed Jul-3/4 — could not be re-tested today (VRAM-gated), so #30378 may be what first exposed AMD to it rather than the sole trigger.
• In-flight fix: ✅ #30506 "[AMD] Disable DSA fused top-k v2 on ROCm for GLM-5.x / DeepSeek-V3.2" (open, non-draft, 2026-07-08) — directly disables the failing path. Chase reviewers; do NOT open a duplicate.
• Suggested triage: confirm #30506 covers DSV3.2 MTP decode dispatch (not just GLM-5.x), merge, re-run. Quick A/B: re-run with SGLANG_OPT_USE_TOPK_V2=0 on MI35x — expect the crash to vanish, pinning #30378.

Workflow	Job (shard)	Test File	Test Function	Error	Log
nightly-test-amd	nightly-2-gpu-mi35x-glm51-mxfp4	test/registered/amd/accuracy/mi35x/test_glm51_mxfp4_tp2_gsm8k_mi35x.py	setUpClass	cooperative_groups.h not found → exit -9	Log
nightly-test-amd	nightly-perf-8-gpu-mi35x-deepseek-v32-mtp	test/registered/amd/perf/mi35x/test_deepseek_v32_mtp_perf_mi35x.py	test_bench_one_batch	ninja status 1 → cooperative_groups.h → exit -9	Log
nightly-test-amd-rocm720	nightly-perf-8-gpu-mi35x-deepseek-v32-basic-rocm720	test/registered/amd/perf/mi35x/test_deepseek_v32_basic_perf_mi35x.py	test_bench_one_batch	cooperative_groups.h not found → exit -9	Log
nightly-test-amd-rocm720	nightly-accuracy-8-gpu-mi35x-deepseek-v32-rocm720	test/registered/amd/accuracy/mi35x/test_deepseek_v32_eval_mi35x.py	test_deepseek_v32_accuracy	Capture cuda graph failed → cooperative_groups.h → exit -9	Log

R252 · 🆕 · HIP graph-capture abort on MoE-EP all-reduce (Qwen3-235B-MXFP4) — nightly-rocm720 · ❌ none

• Status: first executed occurrence Jul-08 (prior 6 runs blocked by VRAM gate → never-passed in window). 1 job. Distinct from R251 (this is a graph-capture-time HIP stream error, not a JIT compile failure). Matches the "RCCL op not permitted when stream is capturing" symptom seen on scout Jul-07.
• Top hypothesis: [MEDIUM] MoE expert-parallel all_reduce falls back to torch.distributed.all_reduce inside the HIP capture stream → hipErrorStreamCaptureUnsupported → SIGABRT (-6). Disconfirming: no A/B disabling EP or CUDA-graph on this config in today's logs; could also be a symm-mem path gap. In-flight fix: ❌ none directly (#29946 symm-mem fallback is related but unconfirmed).
• Suggested triage: re-run with CUDA-graph disabled (or EP all-reduce forced to symm-mem) to isolate; if confirmed, needs a capture-safe all-reduce path on ROCm.

Workflow	Job (shard)	Test File	Test Function	Error	Log
nightly-test-amd-rocm720	nightly-8-gpu-mi35x-qwen3-235b-mxfp4-rocm720	test/registered/amd/test_qwen3_instruct_mxfp4.py	TestQwen3Instruct2507MXFP4.setUpClass	operation not permitted when stream is capturing → exit -6	Log

Top active carry-over clusters

🔴 · Zombie-KFD context — pre-flight VRAM-clear abort (MI35x fleet) — INFRA · ❌ node reboot (ops) / ⚠️ #30431

• Status: STILL ACTIVE Jul-08 (6th consecutive day). Dominates nightly-amd (12), nightly-rocm720 (14), + pr-test MI35x shards on both workflows. GPUs 0–3 stuck ~15–16% VRAM, no owning KFD PID (rocm-smi --showpids empty) → ensure_vram_clear.sh aborts after 3 attempts, test never runs. [HIGH] leaked KFD context (ROCm/aiter#2061); needs node reboot. In-flight fix: ⚠️ #30431 "[DO NOT MERGE][AMD] ci: self-heal driver-level VRAM leaks" (WIP draft) — not merge-ready.

Workflow	Job (shard)	Test / fn	Error	Log
nightly-test-amd	nightly-perf-8-gpu-mi35x-deepseek-v32-basic	N/A (infra)	VRAM cleanup unsuccessful after 3 attempts → exit 1	Log
nightly-test-amd-rocm720	nightly-8-gpu-mi35x-glm51-rocm720	N/A (infra)	GPUs 0–3 @15–16%, no holder	Log
pr-test-amd	stage-c-test-large-8-gpu-amd-mi35x (1)	N/A (infra)	zombie KFD, node reboot	Log
pr-test-amd-rocm720	stage-c-test-large-8-gpu-mi35x (0)	N/A (infra)	VRAM cleanup unsuccessful → exit 1	Log

(+~22 more MI35x jobs across all four AMD workflows — identical zombie-KFD signature.)

R248 · HF weight / Xet download stall → server-launch timeout — nightly-amd + rocm720 (+ pr-test) · ❌ none

• Status: recurring (Jul-06/07/08). [HIGH] transient HF Hub/Xet (xet_get) stall during weight fetch; multiple distinct models same night ⇒ host/network, not model-specific. In-flight fix: ❌ none (retry / pre-seed runner cache).

Workflow	Job (shard)	Test File	Error	Log
nightly-test-amd	nightly-8-gpu-mi35x-qwen35-triton-dcp	test_qwen3p5_triton_dcp.py	xet_get stall → server launch TimeoutError	Log
nightly-test-amd	nightly-8-gpu-mi35x-glm51	test_glm51_eval_mi35x.py	HF xet download hang → accuracy timeout	Log
nightly-test-amd-rocm720	nightly-8-gpu-mi35x-glm5-mxfp4-rocm720	test_glm5_mxfp4_eval_mi35x.py	weight download never completes → 5400s timeout	Log

R195 · Mamba extra_buffer needs CUDA/MUSA/NPU (FLA) — nightly-amd · ✅ #29965

• Status: recurring / never-passed for test_qwen35_eval_mi35x.py. [HIGH] mamba auto radix-cache selects extra_buffer; ROCm not in the CUDA/MUSA/NPU path. In-flight fix: ✅ #29965 (open, non-draft, fall back to no_buffer — ready to land); ⚠️ #30359 (draft, adds real support).

Workflow	Job (shard)	Test File	Test Function	Error	Log
nightly-test-amd	nightly-8-gpu-mi35x-qwen35	test_qwen35_eval_mi35x.py	test_lm_eval	extra_buffer needs CUDA/MUSA/NPU (FLA) → exit 1	Log

R241 · Kimi-K2.6 FP8 non-MTP 1P1D disagg GSM8K 0.886 < 0.92 — mi355x-disagg · 🟡 diag #30433

• Status: never-passed since job introduced Jul-05 (0.887 Jul-05, 0.886 Jul-08; stable across two docker images). MTP sibling clears 0.948 on same nodes. [MEDIUM] base (non-MTP) disagg decode path degraded (candidate: triton MLA decode / cross-node KV transfer). In-flight fix: 🟡 #30433 diagnostic recipes (DO NOT MERGE) — not a fix yet.

Workflow	Job (shard)	Test File	Test Function	Error	Log
nightly-amd-mi355x-disagg	nightly-mi355x-benchmark (kimik26-fp8-1k1k-1p1d)	scripts/ci/slurm/recipes/mi355x-fp8/kimik26/1k1k/1p1d.yaml (GSM8K gate)	GSM8K 8-shot gate	accuracy=0.886 threshold=0.92	Log

Other real / carry-over failures (lower blast radius) · click to expand

Cluster	Workflow	Rep. job	Test / fn	Status	In-flight fix
GPU mem access fault — DSR1 HiCache (gfx950) (R239)	nightly-rocm720	dsr1-hicache-rocm720	test_deepseek_r1_hicache_mi35x.py → mem fault → 3600s timeout	never-passed (Jul 3/5/8)	⚠️ #29275
MTP accept-length below threshold (bs=1)	nightly-rocm720	dsv4-pro-mtp-rocm720	test_deepseek_v4_pro_fp4_mtp.py::test_b_bs_1_speed (2.46 < 2.8)	new failure Jul-08	❌ none
Diffusion update_weights_from_disk 500 (InferenceMode inplace)	pr-test-amd + rocm720	multimodal-gen-1gpu (3)	test_update_weights_from_disk.py	never-passed since Jul-04	❌ none
Diffusion FP8 torch._scaled_mm HIPBLAS_STATUS_NOT_SUPPORTED (flux2)	pr-test-amd	multimodal-gen-2gpu (1)	test_server_2_gpu.py::test_diffusion_generation[flux2_modelopt_fp8_tp2_t2i]	never-passed AMD	✅ #28889
Diffusion server startup / HF-cache timeout (lingbot_world)	pr-test-amd + rocm720	multimodal-gen-1gpu (1)	test_server_1_gpu.py → 1200s / 60-min budget	never-passed since Jul-03	❌ none
Perf-threshold flakes (score-API / triton throughput)	pr-test-amd + rocm720	stage-b-1gpu-large (0)	test_bench_serving_1gpu_part1.py::test_offline_throughput_... (3156<3500)	never-passed / flaky	🟡 #26178 / #29584
MORI git-submodule clone network failure (GnuTLS -54)	pr-test-amd	stage-b-1gpu-small-nondeterministic	N/A (dep install)	infra, Jul-08 latest run	❌ none
PD-disagg KV-transfer 30s timeout (prefill dead)	pr-test-amd	stage-b-large-8gpu-mi35x-disaggregation (.fabric)	Qwen3-8B tp2/pp2 disagg	recurring	❌ none
stage-b-large shard-0 45-min Run-test timeout	pr-test-amd-rocm720	stage-b-1gpu-large (0)	N/A (job-level timeout)	new Jul-08 (was 41–43 min)	❌ none
GSM8K dataset download 429 (raw.githubusercontent.com)	pr-test-amd-rocm720	stage-b-1gpu-small (9)	test_mixed_chunked_prefill.py::test_gsm8k	single-occurrence flake	❌ none

Yesterday's clusters (R249, R245–R250) — status today · click to expand

• R249 (unguarded import cutlass → ROCm arch de-registration → KeyError: 'sglang'): RESOLVED — merged ✅ #30374 2026-07-07 21:43 (if not is_hip() guard). Zero cutlass errors in any Jul-08 nightly or PR run. (Jul-07 PR runs that predate the merge still show it, as expected.) Note #30410 NPU-only / #30413 kernel-selection are separate and do NOT guard the top-level import.
• R248 (HF weight/Xet download stall): STILL ACTIVE — standing carry-over cluster above.
• R195 (mamba extra_buffer): STILL ACTIVE — above; ✅ #29965 ready.
• R239 / R241 / R238 (DSR1 HiCache mem fault / Kimi disagg GSM8K / MI355X DSV4-Flash disagg): carry over; R239 & R241 have fresh Jul-08 signal.
• R250 (ViT pos-embed silent hang) & R246/R247 (DFLASH page256 hang, test_moe_align hang): did NOT recur Jul-08 → confirmed low-frequency flakes.
• Diffusion InductorError/layernorm on ROCm (fix #29673): did NOT recur in today's digested runs.

Workflow drill-down · click to expand

nightly-test-amd · latest 28964511688 · 17 failures (12 zombie-KFD + 2 R251 + 2 R248 + 1 R195)

Job (shard)	Test File	Test Function	Cluster	Error
nightly-2-gpu-mi35x-glm51-mxfp4	test_glm51_mxfp4_tp2_gsm8k_mi35x.py	setUpClass	R251	cooperative_groups.h → -9
nightly-perf-8-gpu-mi35x-deepseek-v32-mtp	test_deepseek_v32_mtp_perf_mi35x.py	test_bench_one_batch	R251	cooperative_groups.h → -9
nightly-8-gpu-mi35x-qwen35	test_qwen35_eval_mi35x.py	test_lm_eval	R195	extra_buffer FLA
nightly-8-gpu-mi35x-qwen35-triton-dcp	test_qwen3p5_triton_dcp.py	setUpClass	R248	HF xet_get stall
+12 mi35x jobs	N/A	N/A	Zombie-KFD	VRAM @15–16%

nightly-test-amd-rocm720 · latest 28964405353 · 21 failures (14 zombie-KFD + 2 R251 + 1 R248 + 1 R252 + 1 R239 + 1 MTP-acc)

Job (shard)	Test File	Test Function	Cluster	Error
dsv32-basic-perf-rocm720	test_deepseek_v32_basic_perf_mi35x.py	test_bench_one_batch	R251	cooperative_groups.h → -9
dsv32-accuracy-rocm720	test_deepseek_v32_eval_mi35x.py	test_deepseek_v32_accuracy	R251	graph-capture → cooperative_groups.h
qwen3-235b-mxfp4-rocm720	test_qwen3_instruct_mxfp4.py	setUpClass	R252	stream-capture HIP → -6
dsr1-hicache-rocm720	test_deepseek_r1_hicache_mi35x.py	—	R239	GPU mem fault → timeout
dsv4-pro-mtp-rocm720	test_deepseek_v4_pro_fp4_mtp.py	test_b_bs_1_speed	MTP-acc	2.46 < 2.8
glm5-mxfp4-rocm720	test_glm5_mxfp4_eval_mi35x.py	test_glm5_mxfp4_accuracy	R248	HF download timeout
+14 mi35x jobs	N/A	N/A	Zombie-KFD	VRAM @15–16%

pr-test-amd · latest 28943113233 · 7 jobs

Job (shard)	Test File	Test Function	Cluster	Error
stage-b-1gpu-small-nondeterministic	N/A (dep install)	N/A	MORI-clone	GnuTLS recv -54
stage-b-1gpu-small (12)	N/A (dep install)	N/A	MORI-clone	submodule clone fail
stage-b-large-8gpu-mi35x-disaggregation (.fabric)	Qwen3-8B tp2/pp2 disagg	—	PD-KV	prefill instance dead
multimodal-gen-1gpu (3)	test_update_weights_from_disk.py	multiple	Diffusion-500	InferenceMode inplace
multimodal-gen-1gpu (1)	test_server_1_gpu.py	lingbot	Diffusion-startup	server not ready 1200s
+2 mi35x jobs	N/A	N/A	Zombie-KFD	VRAM @16%

pr-test-amd-rocm720 · latest 28964460204 · 7 jobs

Job (shard)	Test File	Test Function	Cluster	Error
stage-b-1gpu-large (0)	test_bench_serving_1gpu_part1.py	test_offline_throughput_with_triton_attention_backend	Perf	3156 < 3500
multimodal-gen-1gpu (3)	test_update_weights_from_disk.py	multiple	Diffusion-500	InferenceMode inplace
multimodal-gen-1gpu (1)	N/A (60-min timeout)	N/A	Diffusion-startup	HF cold-cache
stage-b-1gpu-small (9)	test_mixed_chunked_prefill.py	test_gsm8k	GSM8K-429	rate-limited
+3 mi35x/dsv4 jobs	N/A	N/A	Zombie-KFD	VRAM @15–16%

How this report is generated

• Only status == "completed" runs counted in trends. nightly-amd/rocm720 Jul-08 runs completed; ~26 MI35x jobs aborted pre-flight (zombie-KFD, 6th day) so real code signal is partially masked. release-docker (both) = 0 failures; amd-aiter-scout had no run (Mon/Thu cron, today = Wed).
• 🆕 today: R251 (JIT cooperative_groups.h on DSA top-k v2, [HIGH] trigger #30378, fix ✅ #30506); R252 (qwen3-235b HIP stream-capture EP all-reduce, first executed). Yesterday's R249 cutlass resolved (merged ✅ #30374); R246/R247/R250 confirmed flakes.
• Ready fixes: R251 #30506, R195 #29965, FP8-HIPBLAS #28889.
• Infra blockers: zombie-KFD VRAM wave (6th consecutive day, ROCm/aiter#2061, WIP #30431) + MORI submodule clone network failures inflate counts and mask code signal.
• Confidence: FACT/HIGH/MEDIUM/LOW/SPECULATION. Bot does NOT assign Priority — engineers decide from cluster size + persistence + fix availability.

---

Generated by amd-bot · last updated 2026-07-08 23:52 UTC

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-07-08 23:52 UTC)

---

CI Monitor — 2026-07-08

Repo: sgl-project/sglang

Monitored Workflows:

• nightly-test-amd.yml
• nightly-test-amd-rocm720.yml
• release-docker-amd-nightly.yml
• release-docker-amd-rocm720-nightly.yml
• nightly-amd-mi355x-disagg.yml
• amd-aiter-scout.yml
• pr-test-amd.yml
• pr-test-amd-rocm720.yml

Per-workflow failure reports are appended as comments below; the Daily Cross-Workflow Summary is rendered above this section.

[amd-bot]

pr-test-amd-rocm720.yml — 18 failure(s)

Run started (UTC): 2026-07-07 18:19 UTC
Last scanned (UTC): 2026-07-08 23:50 UTC
Commits: sglang 8d2b66f

Summary

Counts: 18 failed jobs across 2 scheduled runs (pr-test-amd-rocm720.yml) · ~21 failing tests · 8 clusters · 5 carrying over (≥3 days) · Cluster A already fixed by merged #30374

Two runs are mixed in this batch: Run A 28888292440 (head 11cea29c, pre-#30374) = 11 jobs; Run B 28964460204 (head 8d2b66fd, contains #30374 fix — verified git merge-base) = 7 jobs. Every cutlass failure (Cluster A) is in Run A only; Run B has zero cutlass failures. [FACT]

#	Cluster	Job	Test File	Test Function	Status	Hypothesis (confidence)
1	Unguarded import cutlass on ROCm → Transformers fallback crash	dsv4-flash-fp4-fp8	test/registered/amd/test_deepseek_v4_flash_fp4.py	TestDeepseekV4Fp4.setUpClass	new Jul 7 (Run A)	[HIGH] fefc1743 (#25220)
2	Unguarded import cutlass on ROCm	dsv4-flash-fp4-fp8	test_deepseek_v4_flash_fp8.py (+_fp8_tbo)	setUpClass	new Jul 7	[HIGH] fefc1743 (#25220)
3	Unguarded import cutlass on ROCm	stage-c-8gpu-mi35x (1)	test/registered/amd/test_deepseek_r1_mxfp4_8gpu.py	TestDeepseekR1MXFP4.setUpClass	new Jul 7	[HIGH] fefc1743 (#25220)
4	Unguarded import cutlass on ROCm	stage-c-8gpu-mi35x (1)	test/registered/amd/test_kimi_k25_mxfp4.py	TestKimiK25MXFP4.setUpClass	new Jul 7	[HIGH] fefc1743 (#25220)
5	Unguarded import cutlass on ROCm	stage-b-small (5)	test/registered/unit/managers/test_mm_process_config.py	test_ernie45_vl_injects_images_kwargs	new Jul 7	[HIGH] fefc1743 (#25220)
6	Unguarded import cutlass on ROCm	stage-b-small (6)	test/registered/hicache/test_hicache_variants.py	TestHiCacheMLA.setUpClass	new Jul 8	[HIGH] fefc1743 (#25220)
7	Unguarded import cutlass on ROCm	stage-b-small (4)	test/registered/mla/test_mla_fp8.py	setUpClass (TestMLA)	new Jul 8	[HIGH] fefc1743 (#25220)
8	Zombie KFD VRAM leak (ensure_vram_clear.sh gate)	dsv4-pro-fp4	N/A (infra pre-check)	N/A	recurring Jul 4–8	[HIGH] leaked KFD ctx (infra)
9	Zombie KFD VRAM leak	stage-c-8gpu-mi35x (0)	N/A (infra pre-check)	N/A	new for shard 0	[HIGH] leaked KFD ctx (infra)
10	Zombie KFD VRAM leak	dsv4-flash-fp4-fp8	N/A (infra pre-check)	N/A	recurring Jul 4–8	[HIGH] leaked KFD ctx (infra)
11	Zombie KFD VRAM leak	dsv4-pro-fp4	N/A (infra pre-check)	N/A	recurring	[HIGH] leaked KFD ctx (infra)
12	Zombie KFD VRAM leak	stage-c-8gpu-mi35x (0)	N/A (infra pre-check)	N/A	new for shard 0	[HIGH] leaked KFD ctx (infra)
13	Diffusion update_weights_from_disk HTTP 500 (InferenceMode inplace)	multimodal-gen (3)	multimodal_gen/.../test_update_weights_from_disk.py	5 FAILED + 4 ERROR	never-passed (Jul 3–7)	[HIGH] unguarded copy_ on inference tensor (linear.py:575)
14	Diffusion update_weights_from_disk HTTP 500	multimodal-gen (3)	test_update_weights_from_disk.py	5 FAILED + 4 ERROR	never-passed	[HIGH] same as row 13
15	Triton-backend offline throughput below AMD floor	stage-b-large (0)	test/registered/perf/test_bench_serving_1gpu_part1.py	test_offline_throughput_with_triton_attention_backend	never-passed since Jul 2	[MEDIUM] 3500 floor miscalibrated for MI325
16	Triton-backend offline throughput below threshold	stage-b-small (0)	test/registered/attention/test_triton_attention_backend.py	test_latency	new Jul 7 (passed prior 3)	[LOW] runner noise vs regression
17	Score-API latency above AMD bound	stage-b-large (1)	test/registered/perf/test_bench_serving_1gpu_part2.py	test_score_api_batch_scaling	5/5 fail Jul 3–7	[MEDIUM] MI325 bounds too tight
18	Diffusion partition-1 step timeout (cold HF cache)	multimodal-gen (1)	N/A (60-min step timeout)	N/A	persistent since Jun 30	[HIGH] HF Hub downloads consume 60-min budget
19	stage-b-large shard-0 45-min timeout	stage-b-large (0)	N/A (45-min step timeout)	N/A	timeout mode new Jul 8	[FACT] cumulative test time > 45-min cap
20	GSM8K dataset download 429	stage-b-small (9)	test/registered/scheduler/test_mixed_chunked_prefill.py	test_gsm8k	transient today	[FACT] raw.githubusercontent.com 429

Cluster: Unguarded import cutlass on ROCm → Transformers-backend fallback crash (rows 1–7)

• Affected: 5 jobs / 8 tests, Run A only. Symptoms vary downstream (ModuleNotFoundError: cutlass / AutoModel _DeepseekV4ConfigAlias ValueError / KeyError: 'sglang' → exit -9) but all share one top frame: jit_kernel/dsa/cutedsl_paged_mqa_logits.py:15 import cutlass.
• Shared facts: import chain deepseek_v2.py:61 → dsa_indexer.py:11 → jit_kernel/dsa/__init__.py:1 → cutedsl_paged_mqa_logits.py:15; No module named 'cutlass' count went 0 (Jul 6 green) → 120–330 (fail). All hops added by fefc1743 (#25220), no disconfirming evidence found.
• In-flight fix: ✅ MERGED 40a68521c (#30374, wraps CuteDSL import in if not is_hip(), landed 2026-07-07 21:43 UTC). Verified 40a68521c is an ancestor of Run B's 8d2b66fd but NOT of Run A's 11cea29c — matching the observed zero-cutlass-failures in Run B.
• Suggested triage: Confirm the next scheduled run (head ≥ 40a68521c) is green for these 8 tests; then grep -rn "^import cutlass" python/sglang/jit_kernel for any other unguarded top-level cutlass imports reachable on ROCm. #30410 (NPU-only) and #30413 (kernel-selection) do not cover the ROCm import.

Cluster: Zombie KFD VRAM leak (rows 8–12)

• Affected: 5 infra pre-check failures on the linux-mi35x-gpu-8-zf775 scale set (both runs). GPUs 0–3 stuck at 15–16% VRAM, No KFD PIDs, no container/host owner; ensure_vram_clear.sh exits 1 before any test runs.
• Top hypothesis: [HIGH] driver-level zombie KFD context (per script hint → ROCm/aiter#2061); node needs reboot. Disconfirming: ephemeral runner suffixes differ, so single-physical-node not proven by name alone. Not a code issue (8d2b66fd/11cea29c never reached).
• In-flight fix: ⚠️ #30431 "[DO NOT MERGE][AMD] ci: self-heal driver-level VRAM leaks" (WIP). Suggested triage: reboot/drain the zf775 MI35x nodes; advance #30431 out of DO-NOT-MERGE.

Cluster: Diffusion update_weights_from_disk HTTP 500 (rows 13–14)

• Affected: test_update_weights_from_disk.py in multimodal-gen (3), both runs, identical 5 FAILED + 4 ERROR. Top hypothesis [HIGH]: initial apply hits AssertionError param_data.shape==loaded_weight.shape (linear.py:411), then rollback's in-place param_data.copy_() (linear.py:575) fails with Inplace update to inference tensor outside InferenceMode; 4 FLUX.2 ERRORs are a separate fixture-layout bug (_compute_checksum_from_disk:220). In-flight fix: ❌ none. Triage: repro [Qwen-Image] on MI325, inspect the shape mismatch and guard the rollback copy_.

Cluster: Perf-threshold misses (rows 15–17)

• Rows 15+16 are triton-backend throughput below hardcoded floors (3500, 153); row 17 is score-API avg latency above AMD bounds. Row 15 & 17 are never-passed / 5-of-5 on MI325 → likely miscalibrated thresholds [MEDIUM]; row 16 passed the prior 3 runs and shows high intra-run variance (147→105) → [LOW] noise-vs-regression, re-run first. In-flight fix: #29584 is ⚠️ partial for row 17 (only bumps batch-50 p95, not the batch-10/25 avg bounds that failed); ❌ none for rows 15/16.

Cluster: Timeouts (rows 18–19) & GSM8K 429 (row 20)

• Row 18: diffusion partition-1 60-min step timeout, persistent since Jun 30 — cold/incomplete HF cache [HIGH]; pre-warm /sgl-data/hf-cache for partition 1's model set. Row 19: stage-b-large shard-0 exceeds 45-min cap [FACT] (prior runs finished at 41–43 min with no slack) — rebalance shard or raise timeout-minutes. Row 20: transient raw.githubusercontent.com 429 [FACT] — re-run/quarantine. All three: in-flight fix ❌ none.

Suggested triage order

Rank	Triage step	Targets	Why this first
1	Confirm merged #30374 keeps cutlass green on next scheduled run; grep for other unguarded ROCm import cutlass	rows 1–7 (cluster A)	Largest cluster (8 tests); fix already merged and absent from Run B — cheap verification
2	Reboot/drain zf775 MI35x nodes; push #30431 to mergeable	rows 8–12	5 infra jobs, ≥5 days persistent, blocks all MI35x 8-GPU signal
3	Pre-warm HF cache for multimodal-gen (1) partition-1 models	row 18	Persistent since Jun 30; no code fix needed
4	Repro + guard diffusion rollback copy_; fix FLUX fixture dir assumption	rows 13–14	Never-passed, 2 jobs, concrete code path identified
5	Full-clone git log -S to date the 3500/latency floors; extend #29584; re-run row 16 to rule out noise	rows 15–17	Distinguishes miscalibrated thresholds from real ROCm-7.2 perf regression

---

Job	Failed Steps	Started
stage-c-test-large-8-gpu-amd-mi35x-rocm720 (linux-mi35x-gpu-8, 0)	Ensure VRAM is clear	2026-07-07T18:22
dsv4-pro-fp4-amd-rocm720	Ensure VRAM is clear	2026-07-07T18:19
stage-b-test-1-gpu-large-amd-rocm720 (linux-mi325-1gpu-sglang, 1)	Run test	2026-07-07T20:53
dsv4-flash-fp4-fp8-amd-rocm720	Accuracy Test MI35x ROCm 7.2 (8-GPU DeepSeek-V4-Flash FP4 + FP8)	2026-07-07T18:21
stage-c-test-large-8-gpu-amd-mi35x-rocm720 (linux-mi35x-gpu-8, 1)	Run test	2026-07-07T18:30
multimodal-gen-test-1-gpu-amd-rocm720 (linux-mi325-1gpu-sglang, 3)	Run diffusion server tests (1-GPU)	2026-07-07T20:49
stage-b-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang, 5)	Run test	2026-07-07T21:25
stage-b-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang, 0)	Run test	2026-07-07T21:13
stage-b-test-1-gpu-large-amd-rocm720 (linux-mi325-1gpu-sglang, 0)	Run test	2026-07-08T00:16
stage-b-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang, 6)	Run test	2026-07-08T01:32
stage-b-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang, 4)	Run test	2026-07-08T01:30
dsv4-flash-fp4-fp8-amd-rocm720	Ensure VRAM is clear	2026-07-08T18:04
stage-c-test-large-8-gpu-amd-mi35x-rocm720 (linux-mi35x-gpu-8, 0)	Ensure VRAM is clear	2026-07-08T18:11
dsv4-pro-fp4-amd-rocm720	Ensure VRAM is clear	2026-07-08T18:09
stage-b-test-1-gpu-large-amd-rocm720 (linux-mi325-1gpu-sglang, 0)	Run test	2026-07-08T19:45
multimodal-gen-test-1-gpu-amd-rocm720 (linux-mi325-1gpu-sglang, 3)	Run diffusion server tests (1-GPU)	2026-07-08T19:49
stage-b-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang, 9)	Run test	2026-07-08T22:38
multimodal-gen-test-1-gpu-amd-rocm720 (linux-mi325-1gpu-sglang, 1)	N/A	2026-07-08T22:22

Per-Job Analysis

stage-c-test-large-8-gpu-amd-mi35x-rocm720 (linux-mi35x-gpu-8, 0) — failed step(s): Ensure VRAM is clear

Commit Info

• sglang: 11cea29c (#30150 — diffusion/cache-dit adapter specs)
• aiter: N/A — job failed in a pre-test host step; no container was launched and no [CI-AITER-CHECK] markers were emitted. AITER_COMMIT_OVERRIDE is empty (log line 36).

Failed Tests

Test File	Test Function	Error	Log
N/A (infrastructure)	N/A	ensure_vram_clear.sh failed: VRAM cleanup unsuccessful after 3 attempts → exit code 1	Log

No test ran. The job aborted in the host-level pre-test step bash scripts/ci/amd/ensure_vram_clear.sh rocm, long before any docker container / pytest suite was started.

Failure Summary

The stage-c-test-large-8-gpu-amd-mi35x-rocm720 (linux-mi35x-gpu-8, 0) job failed during the mandatory VRAM-clear pre-check on runner node linux-mi35x-gpu-8-zf775-runner-ks97f. At job start, GPUs 0–3 already had 15–16% VRAM allocated (threshold is 5%), and three cleanup attempts (SIGTERM→SIGKILL + 30 s wait each) failed to release it. The step exited 1 and the job never reached the test suite.

Regression Status

Infrastructure issue (dirty runner node — not a code regression).

Recent history of this job stage-c-test-large-8-gpu-amd-mi35x-rocm720 (linux-mi35x-gpu-8, 0) (same branch main, event=schedule, completed runs only):

Date	Run	Job	Test File Status	Failed Function	Error
Jul 07	28888292440	shard 0	❌ Failed (infra)	—	VRAM clear failed after 3 attempts
Jul 06	28813230096	shard 0	✅ Passed	—	—
Jul 05	28749537427	shard 0	✅ Passed	—	—

Shard 0 passed on the two prior scheduled runs, so this VRAM-leak occurrence is new for this shard. Note shard 1 in the Jul 06 run (85446190504) also failed — consistent with an intermittent node-level VRAM-leak pattern rather than a per-commit regression.

Failure Cluster

Driver-level VRAM leak — ensure_vram_clear.sh pre-check fails (zombie KFD context, node needs reboot)

Facts (verified observations, no interpretation)

• Job start VRAM already dirty: GPU[0]=16%, GPU[1]=15%, GPU[2]=15%, GPU[3]=16%; GPU[4–7]=0% (log lines 133–164). This state existed before any work by this job.
• rocm-smi --showpids reported No KFD PIDs currently running on all 3 diagnostic passes (log lines 278, 387, 496) — no host process or container holds the GPUs, yet VRAM stays allocated.
• No docker containers found on host and no zombie processes across all 3 attempts (log lines 215/324/433, 270/379/488).
• 3 cleanup attempts each waited 30 s after SIGKILL; VRAM never dropped below threshold (log lines 264, 373, 482).
• Terminal message (log lines 542–546): if no host process / container holds the GPU but VRAM is still allocated, this is almost certainly a zombie KFD context (see ROCm/aiter#2061). The node will need to be rebooted…
• Exit: ##[error]Process completed with exit code 1 (log line 547).
• AITER_COMMIT_OVERRIDE: empty — not a scout run (log line 36).

Hypothesised Causes (with confidence, with disconfirming evidence)

• [FACT] The job failed because pre-existing VRAM (from a prior job on the same runner node) could not be cleared, per the ensure_vram_clear.sh guard.
  • Supporting: dirty VRAM present at job start with zero KFD PIDs / containers / host processes holding the GPUs; script itself diagnoses a zombie KFD context.
  • Disconfirming: (none — this is directly observable in the log).
• [FACT] This failure is not caused by sglang commit 11cea29c.
  • Supporting: the failing step runs on the host before checkout's code is exercised; the leaked VRAM predates this job; the same commit's other shards/jobs are unaffected by this step's logic.
  • Disconfirming: (none found).
• [LOW] The leaking prior job left the zombie KFD context (candidate: a preceding 8-GPU test on the same node that crashed without cleanly releasing device memory — cf. [Bug] Custom all-reduce IPC buffers use fixed VA, conflict with torch_memory_saver pause/resume → zombie processes on MI300X ROCm/aiter#2061).
  • Supporting: leak scoped to GPU 0–3 (the 4-GPU half likely used by a previous workload); intermittent across shards/nodes.
  • Disconfirming: the specific prior workload is not identifiable from this log alone.
  • How to verify: check the node's previous job on linux-mi35x-gpu-8-zf775-runner-ks97f for an unclean exit / crash.

In-flight Fix Check

• In-flight fix: ✅ #30431 — "[DO NOT MERGE][AMD] ci: self-heal driver-level VRAM leaks + graceful GPU cleanup on shutdown" (open since 2026-07-07, non-draft but marked DO NOT MERGE / WIP). Directly targets this exact driver-level VRAM-leak failure mode. Search: is:pr is:open VRAM clear (16 results; #30431 is the on-point one). Do NOT open a duplicate — track/advance this PR.

Suggested Triage Steps

• Reboot / drain runner node linux-mi35x-gpu-8-zf775-runner-ks97f to clear the zombie KFD context, then re-run the job — per the script's own hint ([Bug] Custom all-reduce IPC buffers use fixed VA, conflict with torch_memory_saver pause/resume → zombie processes on MI300X ROCm/aiter#2061).
• Advance in-flight PR #30431 (self-heal driver-level VRAM leaks) toward mergeable state; it addresses this cluster at the CI-harness level.
• No code bisect needed — the commit range is irrelevant to this host-level pre-check failure.
• If this node keeps recurring, correlate with the immediately-preceding job on the same runner to find the workload leaking the KFD context.

Status (factual, no priority assignment)

• Persistence: New for shard 0 (passed Jul 05 & Jul 06); shard 1 hit the same VRAM-clear failure on Jul 06 — intermittent node-level pattern.
• Scope: Host-level pre-test step on runner linux-mi35x-gpu-8-zf775-runner-ks97f; no test suite exercised. Not tied to any sglang/aiter code change.
• Blocked work: This shard's stage-c large 8-GPU MI35x ROCm 7.2 coverage did not run for commit 11cea29c.
• In-flight fix: ✅ #30431 (open since 2026-07-07) — CI self-heal for driver-level VRAM leaks.

dsv4-pro-fp4-amd-rocm720 — failed step(s): Ensure VRAM is clear

Commit Info

• sglang: 11cea29c
• aiter: N/A — no [CI-AITER-CHECK] markers in log; the job died at the pre-flight VRAM check before any container/dependency install ran.

Failed Tests

Test File	Test Function	Error	Log
N/A (infra pre-flight)	N/A	ensure_vram_clear.sh: VRAM cleanup unsuccessful after 3 attempts → exit 1	Log

No test ever ran. The failure is an infrastructure pre-flight step (bash scripts/ci/amd/ensure_vram_clear.sh rocm), which runs before the SGLang container starts. There is no test-level failure to report.

Failure Summary

The dsv4-pro-fp4-amd-rocm720 job failed on runner linux-mi35x-gpu-8-zf775-runner-hldbj because the pre-flight GPU VRAM guard could not clear residual memory. GPUs 0–3 were each pinned at 16% VRAM (threshold 5%) and stayed at 16% across all three SIGTERM/SIGKILL + 30 s-wait cleanup cycles, so the step exited 1 (line 547). This is a runner-environment problem, not an sglang or aiter code issue.

Regression Status

Infrastructure issue (recurring, runner-specific — not a code regression).

Recent history of the pre-flight VRAM guard in job dsv4-pro-fp4-amd-rocm720 (pr-test-amd-rocm720.yml, event=schedule, branch=main, completed runs only):

Date	Run	Job Status	Cause	Runner
Jul 7	28888292440	❌ Failed	VRAM cleanup failed (GPU0-3 @16%)	...-hldbj
Jul 6	28813230096	✅ Passed	—	—
Jul 5	28749537427	❌ Failed	Same VRAM cleanup failure (GPU0-3 @16%)	...-nszmq

The intervening Jul 6 pass and the fact that Jul 5 hit the identical symptom on a different runner confirm this is a per-runner stuck-VRAM condition, not a persistent regression tied to any sglang commit.

Failure Cluster

Pre-flight VRAM cleanup failure (zombie KFD context on AMD runner)

Facts (verified observations)

• Step bash scripts/ci/amd/ensure_vram_clear.sh rocm exited 1 (line 547); container/test steps never started.
• GPUs 0,1,2,3 reported GPU Memory Allocated (VRAM%): 16 at initial check (lines 170-173) and were still at 16 after attempts 1, 2, and 3 (lines 222-225, 331-334, 440-443). VRAM was completely unaffected by SIGKILL.
• Diagnostics show No KFD PIDs currently running (line 496), no docker containers, and no host process holding the GPU device files — i.e. VRAM is allocated with no owning process.
• The script's own hint attributes this to a zombie KFD context and points to ROCm/aiter#2061, stating the node must be rebooted (lines 542-546).
• No [CI-AITER-CHECK] markers present (0 matches) → aiter version is irrelevant to this failure.

Hypothesised Causes

• [FACT] Runner linux-mi35x-gpu-8-zf775-runner-hldbj had ~16% VRAM stuck on GPUs 0–3 with no owning process (zombie KFD context). The pre-flight guard is working as designed: it detected the dirty node and refused to run tests on it.
  • Supporting: VRAM unchanged (16%→16%) after 3× SIGKILL+wait; no KFD PIDs, no containers, no fuser/lsof holders; script's built-in diagnosis names the zombie-KFD condition.
  • Disconfirming: (none — the log is unambiguous that this is a host/driver state issue, not code).
  • How to verify: reboot the affected runner and re-run; check rocm-smi --showmemuse on the host shows 0% before job start.
• [SPECULATION] A prior job on this runner leaked the VRAM (the KFD context outliving its process is the mechanism in ROCm/aiter#2061).
  • Supporting: 16% residual is consistent with a partially-loaded model left behind by an earlier job.
  • Disconfirming: this job's log starts clean; the leaking job is not identifiable from this log alone.
  • How to verify: inspect the previous job scheduled on ...-hldbj for an unclean teardown / OOM / crash.

In-flight Fix Check

• In-flight fix: ✅ #30431 "[DO NOT MERGE][AMD] ci: self-heal driver-level VRAM leaks + graceful GPU cleanup on shutdown" (open, draft, created 2026-07-07 21:59 UTC) — directly targets driver-level VRAM leaks and graceful GPU cleanup. This is the matching effort; do NOT open a duplicate. (Searched is:pr is:open VRAM cleanup.)

Suggested Triage Steps

• Reboot / recycle runner linux-mi35x-gpu-8-zf775-runner-hldbj (and previously-affected ...-nszmq) to clear the zombie KFD context, then re-run the job — the pre-flight guard will pass once VRAM is genuinely free.
• Identify the job that ran on ...-hldbj immediately before this one and check its teardown for a VRAM leak (crash/OOM leaving a KFD context); that is the upstream source per ROCm/aiter#2061.
• Track #30431 as the systemic fix (self-heal driver-level VRAM leaks) and chase it out of [DO NOT MERGE]/draft state.

Status

• Persistence: Recurring but intermittent — failed Jul 5 and Jul 7, passed Jul 6; each occurrence tied to a specific dirty runner, not to a code change.
• Scope: Pre-flight infra guard on AMD ROCm 7.2 runners; observed on at least 2 distinct runners (...-hldbj, ...-nszmq). No tests executed, so no test signal was collected for this job on affected runs.
• In-flight fix: ✅ #30431 (open, draft) — self-heal driver-level VRAM leaks + graceful GPU cleanup.

stage-b-test-1-gpu-large-amd-rocm720 (linux-mi325-1gpu-sglang, 1) — failed step(s): Run test

Commit Info

• sglang: 11cea29c
• aiter: 9127c94a — v0.1.17.dev110+g9127c94a1 (source: dev/patched image detected, rebuild skipped; Dockerfile default was the same SHA)

Failed Tests

Test File	Test Function	Error	Log
test/registered/perf/test_bench_serving_1gpu_part2.py	TestBenchServing1GPUPart2.test_score_api_batch_scaling	AssertionError: 80.59 not less than 70 (1st try, batch 25 avg latency); retry 77.25 not less than 60 (batch 10 avg latency) → retry() exceed maximum number of retries	Log

Failure Summary

The single failing test file is test/registered/perf/test_bench_serving_1gpu_part2.py; the failing function is test_score_api_batch_scaling. The score-API average latency exceeded the AMD-relaxed assertLess bound at test_bench_serving_1gpu_part2.py:130 — first at batch_size=25 (80.59 ms vs bound 70), then on the retry at batch_size=10 (77.25 ms vs bound 60), so the retry wrapper exhausted its retries and the file returned exit code 1. No other test file failed in this job.

Regression Status

Known recurring failure / Never-passed in queryable window — test_bench_serving_1gpu_part2.py has failed in every run of this job in the lookback window (either latency-assertion failures or 1800 s timeouts). No green run found in the last 5 job runs; earlier scheduled runs had this job skipped (not in matrix), so they are not comparable.

Recent history of test/registered/perf/test_bench_serving_1gpu_part2.py (test_score_api_batch_scaling) in job stage-b-test-1-gpu-large-amd-rocm720 (linux-mi325-1gpu-sglang, 1) (branch=main, event=schedule, completed runs only):

Date	Run	Job	Test File Status	Failed Function	Error
Jul 7 (today)	28888292440	shard 1	❌ Failed (assert, completed 940 s)	test_score_api_batch_scaling	80.59 not less than 70 → retry 77.25 not less than 60
Jul 6	28813230096	shard 1	❌ Failed (TIMEOUT 1800 s)	file hung (test not reached to completion)	✗ TIMEOUT after 1800 seconds
Jul 5	28749537427	shard 1	❌ Failed (TIMEOUT 1800 s)	file hung	✗ TIMEOUT after 1800 seconds
Jul 4	28714582765	shard 1	❌ Failed (TIMEOUT 1800 s)	file hung	✗ TIMEOUT after 1800 seconds
Jul 3	28676345359	shard 1	❌ Failed (assert)	test_score_api_batch_scaling	93.09 not less than 90 (batch 50 avg); 85.66 not less than 80 (batch 25 p95)

First observed failure for this test file: at/before Jul 3 (earliest completed run in window). No last-known-passing date within the queryable window (lookback: last 12 scheduled runs, Jul 2–7; only Jul 3–7 actually ran this job).

Failure Cluster

Score-API latency threshold exceeded on AMD MI325 (perf / test-config). (Symptom-level grouping; the Jul 4–6 entries are the more severe timeout variant of the same file.)

Facts (verified observations)

• assertLess(res["avg_latency_ms"], avg_latency_bound) at test_bench_serving_1gpu_part2.py:130 failed. AMD bounds are {10: (60,65), 25: (70,80), 50: (80,90)} (test_bench_serving_1gpu_part2.py:122).
• Verbatim: AssertionError: 80.58770396700129 not less than 70 (log line ~16192, 1st attempt) and AssertionError: 77.24984764680266 not less than 60 (log line ~18113, retry) → Exception: retry() exceed maximum number of retries. → Ran 3 tests in 912.608s / FAILED (errors=1) → ✗ FAILED: …test_bench_serving_1gpu_part2.py returned exit code 1.
• The file completed today (elapsed=940) but timed out on Jul 4/5/6 (elapsed=1800, ✗ TIMEOUT after 1800 seconds, job then ##[error]The action 'Run test' has timed out after 45 minutes.).
• aiter version that ran: v0.1.17.dev110+g9127c94a1 (from [CI-AITER-CHECK] markers; dev image preserved, no rebuild). Unchanged relative to Dockerfile default.
• Reproducibility: 5/5 completed job runs in window failed on this file (2 assertion, 3 timeout).

Hypothesised Causes (with confidence)

• [FACT] The score-API average latency exceeded the AMD-relaxed thresholds: batch 25 = 80.59 ms (bound 70), batch 10 = 77.25 ms (bound 60).
  • Supporting: verbatim assertion messages in the log; test_bench_serving_1gpu_part2.py:122,130.
  • Disconfirming: (none — this is directly observed).
• [MEDIUM] AMD MI325 (gfx942, ROCm 7.2) score-API latency bounds are too tight / the runner is slower than the MI300X the bounds were tuned for — a test-config drift rather than a product bug.
  • Supporting: open PR #29584 explicitly states these bounds "were originally tuned for MI300X" and MI325 "need slightly more headroom"; every completed run in the window fails this file; overages are moderate (15–29%), consistent with a headroom/variance problem, not a crash.
  • Disconfirming: #29584 only relaxes the batch-50 p95 bound (90→95) and does not touch the batch-10/25 avg bounds that failed today, so a pure config-drift explanation is incomplete; a genuine ROCm-7.2 perf regression cannot be ruled out without a green baseline.
• [LOW] A real score/reranker perf regression on ROCm 7.2.
  • Supporting: no green run exists in the queryable window; the file also times out on 3 of 5 runs (worse than a marginal-threshold picture).
  • Disconfirming: no code-level evidence; cannot bound a pass_sha..fail_sha regression window because there is no passing run to anchor against; cannot cite a commit.
  • How to verify: bisect on MI325 once a passing point is found, or run the score benchmark manually and compare avg latency to bounds.

No hypothesis above MEDIUM confidence for a root cause; the FACT is the threshold breach, but whether it is config-drift vs. a perf regression is unresolved without a green baseline.

In-flight Fix Check

• In-flight fix: ⚠️ partial — #29584 ("ci(amd): fix three pre-existing MI325 CI failures", open since 2026-06-28, updated 2026-07-07) edits this exact test file but only bumps the batch-50 p95 bound (90→95) and default_bounds. It does NOT cover today's failing bounds (batch-10 avg 60, batch-25 avg 70), so it would not make today's run green. Do not treat it as a complete fix; ping to extend it.
• Also adjacent: #26178 ("[AMD] WIP - relaxing thres for mi300", still WIP).
• Search used: search/issues?q=repo:sgl-project/sglang+is:pr+is:open+test_bench_serving_1gpu_part2 and +score_api_batch_scaling.

Suggested Triage Steps

• Run the score benchmark on the MI325 runner (Qwen/Qwen3-Reranker-0.6B, batch sizes 10/25/50) and record actual avg/p95 latencies to decide whether the correct fix is to widen the batch-10/25 avg bounds (extend #29584) or to investigate a ROCm-7.2 perf regression.
• If bounds are the answer, extend #29584 to also relax batch-10/25 avg bounds for is_in_amd_ci() and re-run this job.
• Separately investigate the Jul 4–6 1800 s timeouts of the same file (more severe than a marginal threshold): check whether the server hangs or the reranker throughput collapses on MI325/ROCm 7.2, since a hang is not fixed by loosening latency bounds.

Status

• Persistence: test_bench_serving_1gpu_part2.py failing in every completed run of this job Jul 3–7 (5/5); mixed mode (2 latency-assertion, 3 timeout).
• Scope: job stage-b-test-1-gpu-large-amd-rocm720 (linux-mi325-1gpu-sglang, 1) in pr-test-amd-rocm720.yml; shard-1 only. Likely also affects the non-ROCm720 stage-b-test-1-gpu-large-amd sister job (same test, same MI325 bounds) — worth confirming.
• Blocked work: this file blocks the stage-b AMD (ROCm 7.2) signal, forcing PRs whose diffs trigger stage-b to see a red job (the motivation cited in #29584).
• In-flight fix: ⚠️ #29584 partial (batch-50 p95 only) — does not cover today's batch-10/25 avg breach.

dsv4-flash-fp4-fp8-amd-rocm720 — failed step(s): Accuracy Test MI35x ROCm 7.2 (8-GPU DeepSeek-V4-Flash FP4 + FP8)

Commit Info

• sglang: 11cea29c
• aiter: 9127c94a — v0.1.17.dev110+g9127c94a1 (source: dev/patched image detected, rebuild skipped; Dockerfile default is the same 9127c94a)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/test_deepseek_v4_flash_fp4.py	TestDeepseekV4Fp4.setUpClass	Exception: Server process exited with code -9 (server crash; scheduler ValueError: Unrecognized configuration class _DeepseekV4ConfigAlias for AutoModel)	Log
test/registered/amd/test_deepseek_v4_flash_fp8.py	TestDeepseekV4Fp8.setUpClass	Exception: Server process exited with code -9 (same _DeepseekV4ConfigAlias ValueError in scheduler)	Log
test/registered/amd/test_deepseek_v4_flash_fp8_tbo.py	TestDeepseekV4FlashFp8Tbo.setUpClass	Exception: Server process exited with code 1 (Rank 0 scheduler died during initialization, same ValueError)	Log

Test Summary: 0/3 passed.

Failure Summary

All three DeepSeek-V4-Flash tests fail in setUpClass because the server crashes during model load. The scheduler raises ValueError: Unrecognized configuration class <class 'sglang.srt.utils.hf_transformers.common._DeepseekV4ConfigAlias'> for this kind of AutoModel at python/sglang/srt/models/transformers.py:617 (AutoModel.from_config). The config parses fine and the native DeepseekV4ForCausalLM architecture is detected, but at model-load time sglang silently falls back to the generic HF Transformers backend because the native model module failed to import.

Regression Status

New regression (root-cause error first seen 2026-07-07).

Recent history of test_deepseek_v4_flash_fp4.py / test_deepseek_v4_flash_fp8.py in job dsv4-flash-fp4-fp8-amd-rocm720 (branch main, event schedule, completed runs only):

Date (UTC)	Run	Job	Test File Status	Note
Jul 07	28888292440	dsv4-flash…rocm720	❌ Failed (0/3)	_DeepseekV4ConfigAlias AutoModel ValueError
Jul 06	28813230096	dsv4-flash…rocm720	⚪ Not run	Job aborted at VRAM cleanup unsuccessful (infra); tests never executed
Jul 05	28749537427	dsv4-flash…rocm720	⚪ Not run	Same VRAM-cleanup infra abort
Jul 04	28714582765	dsv4-flash…rocm720	⚪ Not run	Same VRAM-cleanup infra abort
Jul 03	28676345359	dsv4-flash…rocm720	✅ Passed (2/2)	fp4 + fp8 both pass (fp8_tbo not yet added)
Jul 03	28630406422	dsv4-flash…rocm720	✅ Passed (2/2)	—

Last known passing (tests actually executed): 2026-07-03, head 7820dc60a7. First failing: 2026-07-07, head 11cea29c. The 07-04/05/06 runs are infra aborts (VRAM cleanup) where the tests never ran, so they are neither pass nor fail for these files. Regression window: 7820dc60a7..11cea29c.

Failure Cluster

ROCm import failure: "No module named 'cutlass'" → model registry fallback to Transformers backend → AutoModel config ValueError

Facts (verified observations)

• Scheduler stack trace (log ~line 5491-5522): transformers.py:617 AutoModel.from_config(...) → ValueError: Unrecognized configuration class ...._DeepseekV4ConfigAlias for this kind of AutoModel. [FACT]
• The failing model list included DeepseekV4Config as a known AutoModel type in the error dump, but the object passed is the sglang alias _DeepseekV4ConfigAlias (registered in python/sglang/srt/utils/hf_transformers/common.py:129-133), which is not in AutoModel's mapping — confirming this is the Transformers-backend fallback path, not native model loading. [FACT]
• The current log contains 1440 lines of Ignore import error when loading sglang.srt.models.<X>: No module named 'cutlass', covering 30 distinct model modules including deepseek_v4, deepseek_v4_nextn, deepseek_v2, glm4_moe, longcat_flash, kimi_k25, etc. [FACT]
• Because deepseek_v4 fails to import, DeepseekV4ForCausalLM is absent from the registry self.models; _normalize_archs appends TransformersForCausalLM as fallback (python/sglang/srt/models/registry.py:74-77). [FACT]
• The passing run of 2026-07-03 (job 84905881970) had zero No module named 'cutlass' errors. [FACT]
• Import chain (verified in source): models/deepseek_v4.py:121 imports models/deepseek_v2.py; deepseek_v2.py:61 does module-level from sglang.srt.layers.attention.dsa.dsa_indexer import Indexer; dsa_indexer.py:11 does module-level from sglang.jit_kernel.dsa import (...); jit_kernel/dsa/__init__.py:1 eagerly from .cutedsl_paged_mqa_logits import ...; cutedsl_paged_mqa_logits.py:15 does an unguarded import cutlass (no try/except). On the AMD ROCm 7.2 image the NVIDIA cutlass DSL package is not installed → ImportError. [FACT]
• At last-pass 7820dc60a7, dsa_indexer.py did not import jit_kernel.dsa/cutlass (it used a function-local from aiter.ops.triton.pa_mqa_logits import ...). [FACT]

Hypothesised Causes

• [HIGH] Commit fefc1743 ("Cute-DSL FP8 MQA logits", #25220, merged 2026-07-07 06:34 UTC) added the module-level from sglang.jit_kernel.dsa import (...) to dsa_indexer.py. jit_kernel/dsa/__init__.py eagerly imports cutedsl_paged_mqa_logits.py, whose unguarded top-level import cutlass cannot resolve on ROCm, cascading into import failure of every model that transitively imports deepseek_v2 (30 modules), including DeepseekV4ForCausalLM.
  • Supporting: direct code evidence (the exact import chain above); it is the only window commit touching cutedsl_paged_mqa_logits.py and it added the dsa_indexer.py module-level import; temporal fit (last-pass SHA lacks the import, current has it; commit landed 2026-07-07 before the 18:11 UTC run); log shows No module named 'cutlass' for deepseek_v4 + the whole deepseek_v2-importing family.
  • Disconfirming: (none found) — the passing run had zero cutlass errors and the import chain is unconditional/unguarded, so the failure is deterministic on ROCm.
  • How to verify: on the ROCm 7.2 image run python -c "import sglang.srt.models.deepseek_v4" at HEAD vs at 7820dc60a7; or wrap the import cutlass in cutedsl_paged_mqa_logits.py / the jit_kernel/dsa/__init__.py re-export in a try/except (or guard behind an SM100 check) and re-run test_deepseek_v4_flash_fp4.py.

In-flight Fix Check

• In-flight fix: 🟡 candidate — #30413 "Fix additional kernels being triggered in CuteDSL MQA logits path" (open, created 2026-07-07 17:00 UTC). Its body explicitly says "It's a bug after refactoring in #25220" and it modifies dsa_indexer.py and jit_kernel/dsa/paged_mqa_logits.py, including removing an eager import annotated "Preserve the original eager import behavior on non-ROCm platforms." It is not certain this PR's primary intent (dropped kernels / numerics) fully removes the unguarded ROCm import cutlass; the maintainer should confirm it eliminates the module-level cutlass import before relying on it. (Searched search/issues for cutlass import ROCm, cutedsl import, No module named cutlass.)
• No dedicated "guard cutlass import on ROCm" PR was found beyond #30413.

Suggested Triage Steps

• Verify #30413 actually removes/guards the module-level import cutlass reachable from deepseek_v2→dsa_indexer→jit_kernel.dsa; if so, chase reviewers rather than opening a duplicate.
• If #30413 does not cover ROCm import safety, guard the top-level import cutlass in python/sglang/jit_kernel/dsa/cutedsl_paged_mqa_logits.py:15 (and the eager re-export in python/sglang/jit_kernel/dsa/__init__.py:1) behind a try/except or an SM100/Blackwell capability check, mirroring the lazy pattern already used elsewhere.
• Reproduce cheaply on the ROCm 7.2 image: python -c "import sglang.srt.models.deepseek_v4" — expect ModuleNotFoundError: No module named 'cutlass' at HEAD.
• Note: the 07-04/05/06 VRAM cleanup unsuccessful infra aborts are a separate issue on this runner that masked the regression for 3 days; worth tracking independently.

Status

• Persistence: Root-cause error first observed 2026-07-07 (this run); prior 3 days' runs aborted on infra before reaching the tests, so this is the first completed run to actually execute and fail with the config error.
• Scope: All 3 DeepSeek-V4-Flash tests in this job. The underlying cutlass ImportError disables 30 native model modules on ROCm (deepseek_v2/v4, glm4_moe, longcat_flash, kimi_k25, etc.), so other AMD jobs exercising those models are likely affected too.
• Blocked work: DeepSeek-V4-Flash fp4/fp8/fp8_tbo CI signal on AMD ROCm 7.2 (and any ROCm model relying on the native deepseek_v2 import chain).
• In-flight fix: 🟡 candidate #30413 — verify it removes the unguarded ROCm cutlass import before relying on it.

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-07-08 23:50 UTC)

[amd-bot]

pr-test-amd-rocm720.yml — Per-Job Analysis (continued 2/4)

stage-c-test-large-8-gpu-amd-mi35x-rocm720 (linux-mi35x-gpu-8, 1) — failed step(s): Run test

Commit Info

• sglang: 11cea29c
• aiter: v0.1.17.dev110+g9127c94a1 (source: dev/patched image preserved; Dockerfile default 9127c94a — rebuild skipped)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/test_deepseek_r1_mxfp4_8gpu.py	TestDeepseekR1MXFP4.setUpClass	Server process exited with code -9; scheduler crashed with KeyError: 'sglang' (Transformers-backend fallback)	Log
test/registered/amd/test_kimi_k25_mxfp4.py	TestKimiK25MXFP4.setUpClass	Server process exited with code 1; ValueError: KimiK25ForConditionalGeneration has no SGlang implementation and the Transformers implementation is not compatible with SGLang.	Log

(test_qwen3_coder_next_8gpu.py — the third file in this shard — passed.)

Failure Summary

Two 8-GPU MXFP4 model tests failed at setUpClass because their server never came up. Both models (amd/DeepSeek-R1-MXFP4-Preview → DeepseekV3ForCausalLM, and amd/Kimi-K2.5-MXFP4 → KimiK25ForConditionalGeneration) silently fell back to the HuggingFace Transformers backend instead of using their native SGLang implementations. The DeepSeek fallback then crashed with KeyError: 'sglang', and the Kimi (multimodal) fallback raised a ValueError.

Regression Status

New regression (started 2026-07-07 run).

Recent history of test_deepseek_r1_mxfp4_8gpu.py + test_kimi_k25_mxfp4.py in shard stage-c-test-large-8-gpu-amd-mi35x-rocm720 (linux-mi35x-gpu-8, 1) (branch=main, event=schedule, completed runs only):

Date	Run	Job	Test File Status	Failed Function	Error
Jul 7	28888292440	shard 1	❌ Both Failed	setUpClass (both)	cutlass import → Transformers fallback
Jul 6	28813230096	shard 1	⚠️ N/A — infra	—	Job aborted at "VRAM still not clear" precheck; tests never ran
Jul 5	28749537427	shard 1	✅ Both Passed	—	deepseek 906s / kimi 615s, gsm8k eval OK
Jul 4	28714582765	shard 1	⚠️ N/A — infra	—	Job aborted at "VRAM still not clear" precheck; tests never ran

Last known good: Jul 5 (48ba79c11e). First failure: Jul 7 (11cea29c). Regression window: 48ba79c11e...11cea29c (67 commits). Jul 6 / Jul 4 were unrelated GPU-cleanup infra aborts, not test failures.

Failure Cluster

Native model import fails on ROCm (No module named 'cutlass') → silent Transformers-backend fallback → server launch crash.

Facts (verified observations)

• [FACT] Current log (Jul 7): Ignore import error when loading sglang.srt.models.deepseek_v2: No module named 'cutlass' (17 occurrences) and ...kimi_k25: No module named 'cutlass'. On the Jul 5 passing log both counts are 0 — deepseek_v2/kimi_k25 imported natively.
• [FACT] DeepSeek stack (job log): DeepseekV3ForCausalLM has no SGLang implementation, falling back to Transformers implementation → transformers.py:617 AutoModel.from_config → remote modeling_deepseek.py:1148 self.self_attn = ATTENTION_CLASSES[config._attn_implementation] → KeyError: 'sglang' → sigquit → server exit -9.
• [FACT] Kimi stack: resolve_transformers_arch (model_loader/utils.py:182) raises ValueError: KimiK25ForConditionalGeneration has no SGlang implementation ... → server exit 1.
• [FACT] Eager import chain exists in source: models/deepseek_v2.py:61 → from ...dsa.dsa_indexer import Indexer; dsa_indexer.py:11-13 → from sglang.jit_kernel.dsa import (..., cutedsl_paged_mqa_logits, ...); jit_kernel/dsa/cutedsl_paged_mqa_logits.py → top-level import cutlass. models/kimi_k25.py:30 → from sglang.srt.models.deepseek_v2 import DeepseekV3ForCausalLM (so Kimi inherits the same failure).
• [FACT] KimiK25ForConditionalGeneration and DeepseekV3ForCausalLM both have native SGLang implementations (models/kimi_k25.py:616, models/deepseek_v2.py) — so this is an import-time break, not missing support.
• [FACT] cutedsl_paged_mqa_logits eager import + the import cutlass file were introduced by fefc1743 (git blame + --diff-filter=A).

Hypothesised Causes

• [HIGH] fefc1743 (#25220 "Cute-DSL FP8 MQA logits") added an eager, unguarded import cutlass into the sglang.jit_kernel.dsa package (via cutedsl_paged_mqa_logits.py), which is imported at module load by dsa_indexer.py → deepseek_v2.py → kimi_k25.py. On the ROCm/MI35x image the NVIDIA cutlass Python package is not installed, so both native model modules now raise ModuleNotFoundError at registry-load time and silently fall back to Transformers.
  • Supporting: commit lands 2026-07-07 06:34 UTC, inside the 48ba79c11e...11cea29c window; it created the exact import cutlass file and the eager from sglang.jit_kernel.dsa import cutedsl_paged_mqa_logits line; log A/B shows deepseek_v2 imported fine Jul 5 (0 cutlass errors) and fails Jul 7 (17).
  • Disconfirming: the other window commit that added top-level import cutlass, e85ef548 (#30117), routes through unquant.py where the cutedsl import is lazy (inside initialize_bf16_gemm_config, gated on is_cutedsl()), so it does not fire at import time — it is a red herring, not the cause.
  • How to verify: on the MI35x ROCm container, python -c "import sglang.srt.models.deepseek_v2" should raise No module named 'cutlass' at HEAD and succeed at 48ba79c11e; or revert fefc1743 on a branch and re-run both test files.

In-flight Fix Check

• In-flight fix: ❌ none found addressing the ROCm import guard. Searched open PRs for cutlass import dsa, cutedsl paged mqa, No module named cutlass.
• Related (but NOT a fix): #30413 "Fix additional kernels being triggered in CuteDSL MQA logits path" (open, 2026-07-07) explicitly references #25220 and touches dsa_indexer.py, but it fixes a kernel-selection bug, not the unguarded import cutlass on non-CUDA backends.

Suggested Triage Steps

• Confirm the mechanism: python -c "import sglang.srt.models.deepseek_v2" inside the MI35x rocm720 container — expect ModuleNotFoundError: No module named 'cutlass'.
• If confirmed, the fix direction is to make the import cutlass in python/sglang/jit_kernel/dsa/cutedsl_paged_mqa_logits.py (and its re-export in jit_kernel/dsa/__init__.py) lazy / guarded (import inside the call site, or try/except ImportError), so dsa_indexer → deepseek_v2 → kimi_k25 stay importable on ROCm.
• Alternatively, bisect 48ba79c11e...11cea29c if the A/B check is inconclusive — but fefc1743 is the only eager top-level cutlass import reachable from deepseek_v2.
• Coordinate with #30413 author since it already edits the same DSA files.

Status

• Persistence: New as of the 2026-07-07 06:34 UTC merge of fefc1743; first observed in this Jul 7 scheduled run (Jul 5 was green; Jul 6/Jul 4 were infra aborts, not comparable).
• Scope: Both native model modules deepseek_v2 and kimi_k25 fail to import on the ROCm/MI35x image; blocks test_deepseek_r1_mxfp4_8gpu.py and test_kimi_k25_mxfp4.py on pr-test-amd-rocm720. Any other AMD test loading a DSA/DeepSeek-derived model is likely affected too. NVIDIA CI is unaffected (cutlass present).
• In-flight fix: ❌ none addressing the import guard; #30413 touches the same area for a different bug.

multimodal-gen-test-1-gpu-amd-rocm720 (linux-mi325-1gpu-sglang, 3) — failed step(s): Run diffusion server tests (1-GPU)

Commit Info

• sglang: 11cea29c
• aiter: 9127c94a (v0.1.17.dev110+g9127c94a1, source: dev image — Dockerfile default preserved, AITER_COMMIT_OVERRIDE empty)

Failed Tests

All failures are in python/sglang/multimodal_gen/test/single_test_file/test_update_weights_from_disk.py (job step 9, "Run diffusion server tests (1-GPU)"). Summary: 5 failed, 1 passed, 4 errors.

Test File	Test Function	Error	Log
test_update_weights_from_disk.py	TestUpdateWeightsFromDisk::test_update_weights_from_disk_default[FLUX.2-klein-base-4B]	ERROR at setup: AssertionError: No weights dir for transformer in /tmp/sglang_perturbed_vae_* (fixture diffusion_server_no_offload)	Log
test_update_weights_from_disk.py	TestUpdateWeightsFromDisk::test_update_weights_specific_modules[FLUX.2-klein-base-4B]	ERROR at setup (same fixture)	Log
test_update_weights_from_disk.py	TestUpdateWeightsFromDisk::test_update_weights_rejects_invalid_requests[FLUX.2-klein-base-4B]	ERROR at setup (same fixture)	Log
test_update_weights_from_disk.py	TestUpdateWeightsFromDisk::test_corrupted_weights_rollback[FLUX.2-klein-base-4B]	ERROR at setup (same fixture)	Log
test_update_weights_from_disk.py	TestUpdateWeightsFromDisk::test_update_weights_from_disk_default[Qwen-Image]	assert 500 == 200 — POST /update_weights_from_disk returns 500	Log
test_update_weights_from_disk.py	TestUpdateWeightsFromDisk::test_update_weights_specific_modules[Qwen-Image]	AssertionError: Update failed: {'success': False, 'message': 'Inplace update to inference tensor outside InferenceMode is not allowed...'} (500)	Log
test_update_weights_from_disk.py	TestUpdateWeightsFromDisk::test_corrupted_weights_rollback[Qwen-Image]	assert (500 == 200)	Log
test_update_weights_from_disk.py	TestUpdateWeightsFromDiskWithOffload::test_update_weights_with_offload_enabled[FLUX.2-klein-base-4B]	AssertionError: Expected 200, got 500	Log
test_update_weights_from_disk.py	TestUpdateWeightsFromDiskWithOffload::test_update_weights_with_offload_enabled[Qwen-Image]	AssertionError: Expected 200, got 500	Log

Failure Summary

The standalone test file test_update_weights_from_disk.py (partition 4/4 of multimodal-gen-test-1-gpu-amd-rocm720) failed with 5 failures + 4 setup errors. The 4 errors are all FLUX.2-klein setup failures in the diffusion_server_no_offload fixture (test_update_weights_from_disk.py:410 → _compute_checksum_from_disk at :220), which asserts a per-module weights subdir exists but finds none. The 5 failures are the /update_weights_from_disk server endpoint returning HTTP 500 (both Qwen-Image and FLUX.2-klein, offload and no-offload).

Regression Status

Known recurring failure — this test file failed identically in the previous scheduled run.

Recent history of test_update_weights_from_disk.py in job multimodal-gen-test-1-gpu-amd-rocm720 (branch main, event schedule, completed runs only):

Date	Run	Job	Test File Status	Failed Functions	Error
Jul 7	28888292440	multimodal-gen-1gpu (3)	❌ Failed	5 failed + 4 setup errors	shape mismatch → InferenceMode 500; FLUX setup errors
Jul 6	28813230096	multimodal-gen-1gpu (3)	❌ Failed	same pattern (Inplace update to inference tensor 500 + FLUX No weights dir setup errors)	identical

No green run for this test file found in the queryable window (2 scheduled runs, Jul 6–7). The multimodal_gen unit suite is a recent addition to AMD CI, so this may be a Never-passed AMD port issue rather than a fresh regression; a wider lookback is needed to confirm a last-passing date.

Failure Cluster

Two symptom clusters in this one file:

• Cluster A — update_weights_from_disk HTTP 500 (weight shape mismatch → InferenceMode rollback failure) (5 failures)
• Cluster B — Fixture setup error: missing per-module weights dir for FLUX.2-klein (4 errors)

Facts (verified observations)

• Cluster A chained traceback (verbatim from log, today's run):
  • weights_updater.py:339 _apply_weights → :139 _load_weights_into_module → :200 load_weights_into_model → linear.py:411 assert param_data.shape == loaded_weight.shape → AssertionError (initial weight-load shape mismatch).
  • "During handling of the above exception, another exception occurred" → rollback path: weights_updater.py:350 _apply_weights → :381 _rollback → :139 → :200 → linear.py:533 → linear.py:575 param_data.copy_(loaded_weight) → RuntimeError: Inplace update to inference tensor outside InferenceMode is not allowed.
  • Server logs it as Error executing request in scheduler event loop: Inplace update to inference tensor... and returns POST /update_weights_from_disk HTTP/1.1 500 Internal Server Error.
• Cluster B (verbatim): test_update_weights_from_disk.py:410 (in diffusion_server_no_offload) → :220 _compute_checksum_from_disk → assert os.path.exists(weights_dir) → AssertionError: No weights dir for transformer in /tmp/sglang_perturbed_vae_gkax0snz. Background-thread variants also assert No weights dir for vae/transformer against the pristine snapshot dirs for both FLUX.2-klein and Qwen-Image.
• Reproducibility: identical failures in the immediately-prior scheduled run (Jul 6, run 28813230096, job 85446190846), including the scheduler.py:229 _handle_update_weights_from_disk → Inplace update to inference tensor server error.
• Environment: AMD MI325 (gfx942/mi30x), ROCm 7.2 image rocm/sgl-dev:v0.5.14-rocm720-mi30x-20260707, aiter v0.1.17.dev110+g9127c94a1. No aiter override.
• A WARNING [server-test] ROCm GPU memory not fully cleared after 60s precedes several Qwen-Image failures, but the 500 responses are driven by the code-level tracebacks above, not OOM.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [HIGH] Cluster A is a bug in the diffusion update_weights_from_disk code path: the initial apply raises AssertionError at linear.py:411 (param_data.shape == loaded_weight.shape), and the rollback then fails because model params are inference-mode tensors and param_data.copy_() at linear.py:575 is not wrapped in torch.inference_mode()/cloned.
  • Supporting: full chained stack trace present verbatim in the log for every 500; the endpoint's own error message ("Inplace update to inference tensor... make a clone... before doing inplace update") points directly at the un-guarded in-place copy_. Reproduces identically across two runs.
  • Disconfirming: the first exception is a shape assertion, not the InferenceMode error — so there are potentially two distinct defects (a shape mismatch on apply, and an unguarded copy on rollback). Fixing only the InferenceMode wrap would surface the underlying shape mismatch. Cannot tell from the log alone whether the shape mismatch is AMD-specific or a general test/data issue.
  • How to verify: on an AMD MI325 ROCm 7.2 container, run test_update_weights_from_disk.py::TestUpdateWeightsFromDisk::test_update_weights_from_disk_default[Qwen-Image] with a breakpoint at linear.py:411 to inspect param_data.shape vs loaded_weight.shape; separately, check whether wrapping the _rollback/copy_ in torch.inference_mode(False) or cloning removes the secondary RuntimeError.
• [MEDIUM] Cluster B is a test-side layout assumption mismatch: _compute_checksum_from_disk expects a <model_path>/<module_name> subdir (e.g. transformer, vae) to exist, but the perturbed FLUX.2-klein clone dir (and the raw HF snapshot in the background-thread variant) does not contain those subdirs at the expected path.
  • Supporting: assertion fires at :220 for both the perturbed /tmp/sglang_perturbed_vae_* dir and the pristine models--black-forest-labs--FLUX.2-klein-base-4B/snapshots/... dir; the FLUX.2-klein model does load its components successfully earlier in the same log (text_encoder, transformer, vae all loaded), suggesting the module→subdir mapping used by the checksum helper differs from the actual on-disk layout for this model.
  • Disconfirming: only FLUX.2-klein hits the setup error; Qwen-Image gets past setup (its failures are Cluster A), so the checksum-dir assumption is not universally broken — it may be a FLUX.2-klein-specific directory structure the helper doesn't handle.
  • How to verify: inspect the directory tree of a downloaded FLUX.2-klein snapshot vs. what _clone_model_with_modified_module / maybe_download_model produce, and confirm whether transformer/vae live directly under the model path or under a nested dir.
• [LOW] aiter / ROCm interaction. Disconfirming: this run uses the Dockerfile-default aiter (no override), the failing frames are pure sglang diffusion weight-loading code (weights_updater.py, linear.py), and the errors are Python-level assertions/RuntimeErrors — not kernel/HIP faults. Not a promising direction.

In-flight Fix Check

• In-flight fix: ❌ none found. Searched open PRs with keywords update_weights_from_disk, InferenceMode, multimodal_gen update_weights (GET /search/issues?q=repo:sgl-project/sglang+is:pr+is:open+...). Matches returned (#19499, #19152) are unrelated (FP8 MoE EPLB hot-reload; diffusion offload/wake-up) — neither addresses the diffusion update_weights_from_disk shape-mismatch/InferenceMode-rollback path or the FLUX.2-klein checksum-dir setup error.

Suggested Triage Steps

• Reproduce Cluster A on AMD MI325 ROCm 7.2: run test_update_weights_from_disk.py::TestUpdateWeightsFromDisk::test_update_weights_from_disk_default[Qwen-Image] and inspect the shape mismatch at linear.py:411 — determine whether the mismatch is the primary bug (and the InferenceMode error only a secondary rollback failure).
• If the shape mismatch is expected/intended (perturbed weights), guard the rollback in-place write: wrap weights_updater._rollback / linear.py:575 param_data.copy_() so it does not attempt an in-place update on an inference-mode tensor (clone or enter a non-inference context), then re-run.
• For Cluster B, verify the FLUX.2-klein on-disk module layout vs. the <model_path>/<module_name> assumption in _compute_checksum_from_disk (test_update_weights_from_disk.py:220); the fixture may need model-specific subdir resolution.
• Widen the lookback beyond Jul 6–7 to establish whether this test file has ever passed on AMD; if never-passed, consider skipping/xfailing on AMD until the diffusion weight-update path is fixed, to unblock the rest of the multimodal-gen signal.

Status

• Persistence: Failing on both queryable scheduled runs (Jul 6 and Jul 7, 2026); no passing run for this test file observed in that window.
• Scope: test_update_weights_from_disk.py in multimodal-gen-test-1-gpu-amd-rocm720 (partition 4/4). Both parametrized models (Qwen-Image, FLUX.2-klein) and both fixtures (offload / no-offload) affected. Not observed to involve aiter/kernel code.
• Blocked work: diffusion update_weights_from_disk CI signal on AMD ROCm 7.2.
• In-flight fix: ❌ none found — needs new PR or investigation.

stage-b-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang, 5) — failed step(s): Run test

Commit Info

• sglang: 11cea29c
• aiter: 9127c94a (v0.1.17.dev110+g9127c94a1, source: dev image detected — rebuild skipped)

Failed Tests

Test File	Test Function	Error	Log
test/registered/unit/managers/test_mm_process_config.py	TestOverrideProcessorsConfigInjection.test_ernie45_vl_injects_images_kwargs	ModuleNotFoundError: No module named 'cutlass'	Log

Failure Summary

The unit suite test_mm_process_config.py failed (FAILED (errors=1), 10/11 tests OK) during shard 5 of stage-b-test-1-gpu-small-amd-rocm720. test_ernie45_vl_injects_images_kwargs errored at import time: importing the Ernie4.5-VL processor transitively pulls in deepseek_v2 → dsa_indexer → sglang.jit_kernel.dsa, whose module does an unconditional top-level import cutlass. cutlass (the NVIDIA CUTLASS Python DSL) is not installed in the ROCm image, so the import raises ModuleNotFoundError. The job exited with code 255 and was not retried (correctly classified as a test failure, not a network issue).

Regression Status

New regression — first failing in this run (2026-07-07).

Recent history of test/registered/unit/managers/test_mm_process_config.py in job stage-b-test-1-gpu-small-amd-rocm720 (same branch main, event=schedule, completed runs only):

Date	Run	Job	Test File Status	Failed Function	Error
Jul 7	28888292440	shard 5	❌ Failed	test_ernie45_vl_injects_images_kwargs	ModuleNotFoundError: No module named 'cutlass'
Jul 6	28813230096	shard 5	✅ Passed (10/10)	—	—

• First observed failure: 2026-07-07. Last known passing: 2026-07-06 18:14 UTC (3abdbab9bb, verified: shard 5 ran this file and reported 10/10 passed, no cutlass error).
• Regression window: 3abdbab9bb..11cea29c.

Failure Cluster

ModuleNotFoundError: No module named 'cutlass' (unconditional CUTLASS import reached via DSA indexer chain on ROCm)

Facts (verified observations)

• Verbatim traceback (log step 6): cutedsl_paged_mqa_logits.py, line 15 → import cutlass → ModuleNotFoundError: No module named 'cutlass'.
• Import chain (from log): test_mm_process_config.py:236 → multimodal/processors/ernie45_vl.py:15 → models/ernie45_vl.py:42 → models/ernie45_moe_vl.py:44 → models/deepseek_v2.py:61 → layers/attention/dsa/dsa_indexer.py:11 → jit_kernel/dsa/__init__.py:1 → jit_kernel/dsa/cutedsl_paged_mqa_logits.py:15.
• python/sglang/jit_kernel/dsa/cutedsl_paged_mqa_logits.py:14-17 does top-level import cutlass / import cutlass.cute / from cutlass.utils import HardwareInfo with no try/except or platform guard (verified by reading the source). The module docstring states it targets "Blackwell SM100" (NVIDIA-only).
• python/sglang/jit_kernel/dsa/__init__.py:1 unconditionally re-exports from that module, so any importer of sglang.jit_kernel.dsa triggers import cutlass.
• Commit fefc1743 (#25220 "Cute-DSL FP8 MQA logits", merged 2026-07-06 23:34 -0700) added both files and rewired dsa_indexer.py (+105/−34) to import from sglang.jit_kernel.dsa.
• Ancestry (verified): fefc1743 is NOT an ancestor of the last-passing SHA 3abdbab9bb, but IS an ancestor of the failing SHA 11cea29c.
• Reproducibility: single test file, single job/shard in this run's log. Not aiter-related (import error, aiter loaded fine).

Hypothesised Causes (with confidence)

• [HIGH] Commit fefc1743 (#25220) introduced an unconditional import cutlass in jit_kernel/dsa/cutedsl_paged_mqa_logits.py, reachable on ROCm through the DSA indexer import chain that the same commit added to dsa_indexer.py.
  • Supporting: (a) direct code evidence — the exact failing line 15 was added by this commit; (b) the same commit rewired dsa_indexer.py to import sglang.jit_kernel.dsa, creating the chain in the traceback; (c) temporal — fefc1743 merged 2026-07-06 23:34, after the last green scheduled run (18:14) and before the failing run; (d) A/B — test_mm_process_config.py passed 10/10 on the pre-commit SHA 3abdbab9bb and fails post-commit.
  • Disconfirming: (none found) — the commit, code path, and timing all converge.
  • How to verify: on a ROCm container, git revert fefc1743 (or wrap the import cutlass block in cutedsl_paged_mqa_logits.py in a try/except or an is_cuda/is_sm100_supported guard) and re-run python3 test/registered/unit/managers/test_mm_process_config.py.

In-flight Fix Check

• In-flight fix: ⚠️ partial — #30410 "[NPU] Fix npu cutlass import" (open since 2026-07-07) moves the from sglang.jit_kernel.dsa import (...) in dsa_indexer.py behind if not is_npu():. This fixes NPU only — on ROCm is_npu() is False, so the guarded import still runs and import cutlass still fails (verified from the PR diff). It does NOT resolve this ROCm failure.
• Searched: repo:sgl-project/sglang is:pr is:open cutlass import (38 hits; #30410 is the only DSA-cutlass-import one). No open PR guards the ROCm/HIP path.

Suggested Triage Steps

• Guard the cutlass imports at their source in python/sglang/jit_kernel/dsa/cutedsl_paged_mqa_logits.py:14-17 (try/except ImportError or gate behind is_sm100_supported()/_is_cuda), since the module is NVIDIA-Blackwell-only — this fixes both ROCm and NPU at once, rather than the caller-side if not is_npu() guard in #30410.
• Alternatively, extend #30410's guard to exclude non-CUDA backends (e.g. if not is_npu() and not is_hip() or if _is_cuda) so ROCm is also protected; then re-run shard 5 of pr-test-amd-rocm720.yml.
• To confirm the culprit: revert fefc1743 on a branch and run test_mm_process_config.py in the ROCm image rocm/sgl-dev:v0.5.14-rocm720-mi30x-20260707.

Status

• Persistence: New — first failing scheduled run 2026-07-07 18:11 UTC; previous scheduled run (2026-07-06) was green for this test file.
• Scope: test_mm_process_config.py on ROCm 7.2 (stage-b-test-1-gpu-small-amd-rocm720). Any ROCm/NPU code path importing sglang.jit_kernel.dsa (e.g. via deepseek_v2 / Ernie4.5-VL) is at risk; the blast radius is the unconditional CUTLASS import, not this one test.
• In-flight fix: #30410 exists but fixes NPU only — ROCm still broken; needs a broader guard or a source-level fix in cutedsl_paged_mqa_logits.py.

stage-b-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang, 0) — failed step(s): Run test

Commit Info

• sglang: 11cea29c
• aiter: 9127c94a (v0.1.17.dev110+g9127c94a1; source: dev image preserved — [CI-AITER-CHECK] Dev/patched version detected → skipping rebuild)

Failed Tests

Test File	Test Function	Error	Log
test/registered/attention/test_triton_attention_backend.py	test_latency	AssertionError: 105.46 not greater than 153 (retry exhausted; first attempt 147.56 not greater than 153)	Log

Failure Summary

test_latency in test/registered/attention/test_triton_attention_backend.py:43 measures offline-throughput for the Llama-3.1-8B triton attention backend with --enable-torch-compile --cuda-graph-max-bs-decode 4 and asserts output_throughput > 153. The measured throughput was 147.56 on the first attempt and 105.46 on the retry — both below the hardcoded 153 threshold — so the retry helper raised retry() exceed maximum number of retries and the file returned exit code 1.

Regression Status

New regression (throughput dropped below threshold; test passed the previous 3 completed scheduled runs).

Recent history of test_triton_attention_backend.py (test_latency) in job stage-b-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang, 0) — same branch (main), same event (schedule), suite partition 0/14, completed runs only:

Date	Run	Job	Test File Status	Measured throughput	Error
Jul 07	28888292440	job	❌ Failed	147.56 → 105.46	AssertionError: <153
Jul 06	28813230096	job	✅ Passed	185.29	—
Jul 05	28749537427	job	✅ Passed	187.88	—
Jul 04	28714582765	job	✅ Passed	197.26	—

First observed failure: 2026-07-07. Last known passing: 2026-07-06 (185.29). The job's overall conclusion was success in the prior 3 runs and the test consistently ran in partition 0, so the comparison is valid at the test-file level.

Failure Cluster

Triton-attention offline-throughput below latency threshold (perf regression / runner noise)

Facts (verified observations, no interpretation)

• Failing command (verbatim): python3 -m sglang.benchmark.offline_throughput --num-prompts 1 --dataset-name random --random-input-len 256 --random-output-len 256 --model-path meta-llama/Llama-3.1-8B-Instruct --attention-backend triton --enable-torch-compile --cuda-graph-max-bs-decode 4
• Threshold is hardcoded: self.assertGreater(output_throughput, 153) (test_triton_attention_backend.py:43), gated on is_in_ci().
• Today: first attempt 147.56 (~3.5% under threshold), retry 105.46 (~31% under). Prior 3 completed runs: 185.29 / 187.88 / 197.26 (all comfortably over 153).
• Large intra-run variance today (147.56 vs 105.46 between two back-to-back attempts) — decode gen throughput samples in the log ranged 3.19 → 112.16 → 165.74 → 147.56 within a single benchmark.
• aiter version unchanged from a dev image (v0.1.17.dev110+g9127c94a1); no rebuild performed. No aiter override in this workflow (pr-test-amd-rocm720.yml, not a scout run).
• 46 sglang commits in the regression window 3abdbab9..11cea29c.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [LOW] Runner-side performance noise / GPU contention rather than a code regression.
  • Supporting: 147.56 vs 105.46 across two identical back-to-back runs is a ~28% spread; per-decode throughput samples oscillate wildly (3.19 → 165.74) within one benchmark; the first attempt (147.56) is only marginally under the 153 threshold.
  • Disconfirming: prior 3 runs were tightly clustered high (185–197), so a drop to a 105–147 band is a real shift, not just jitter around the threshold.
  • How to verify: re-run the job on the same MI325 runner; if it recovers to >180, this was transient/contention.
• [LOW] A commit in 3abdbab9..11cea29c regressed the triton-backend + torch.compile decode path on ROCm. Candidates touching relevant paths: 52c6e27e ([AMD] disable layernorm torch.compile decorator in eager mode on ROCm, #29673 — this test uses --enable-torch-compile), 3cbb7568 (Full Cuda Graph Support for Prefill, #27988), 946804e0 (Disable FA3 sparse mask kernels by default, #30356).
  • Supporting: window contains several attention / torch.compile / cuda-graph changes; regression is clean between Jul-06 pass and Jul-07 fail.
  • Disconfirming: no code-level evidence any of these is on the Llama-3.1-8B triton decode hot path; #52c6e27e is a diffusion-focused change and only affects eager mode (this test compiles); the large intra-run variance points at noise rather than a deterministic slowdown.
  • How to verify: git bisect throughput across [3abdbab9..11cea29c] on an MI325 runner, running the failing command 3× per commit and comparing median throughput.
• No hypothesis reaches MEDIUM: the marginal first-attempt value plus high variance make noise and a real regression indistinguishable without a bisect/re-run.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for test_triton_attention_backend (1 unrelated hit: #30172, an XPU runner-health harness) and triton attention throughput (only generic feature PRs, none addressing this threshold/perf regression).

Suggested Triage Steps

• Re-run job 85694092762 (or the whole partition-0 suite) on an MI325 runner first — a single green re-run at >180 confirms this was transient runner noise and closes it out cheaply.
• If it stays red: bisect throughput across 3abdbab9..11cea29c with the exact failing command (3 runs/commit, compare medians), focusing on the torch.compile / cuda-graph / attention commits listed above.
• If a real slowdown is confirmed, fix in the implicated code path; if the drop reflects legitimate new runner baseline, consider whether the hardcoded 153 threshold in test_triton_attention_backend.py:43 needs re-tuning (maintainer call, not a bot directive).

Status

• Persistence: First failure 2026-07-07; passed the previous 3 completed scheduled runs (Jul 04–06). One occurrence so far.
• Scope: 1 test (test_latency) in 1 job/shard (partition 0) of pr-test-amd-rocm720.yml. The parent run 28888292440 was still queued/in-progress at snapshot time; other shards not assessed here.
• Blocked work: Blocks the ROCm 7.2 triton-attention latency signal on AMD.
• In-flight fix: ❌ none found — needs a re-run to rule out noise, then a bisect if it persists.

stage-b-test-1-gpu-large-amd-rocm720 (linux-mi325-1gpu-sglang, 0) — failed step(s): Run test

Commit Info

• sglang: 11cea29c
• aiter: 9127c94a (source: dev image preserved — [CI-AITER-CHECK] reports Dev/patched version detected: v0.1.17.dev110+g9127c94a1 → skipping rebuild)

Failed Tests

Test File	Test Function	Error	Log
N/A — job-level timeout	N/A	The action 'Run test' has timed out after 45 minutes	Log

This is not a test-assertion failure. The Run test step hit its 45-minute hard timeout at 01:14:23 while the shard was starting file 8/11 (test/registered/attention/unittests/gdn/test_torch_native.py, begun 01:14:10, only 13 s before the cutoff). No test in this shard reported an assertion/error before the timeout; the final Test Summary block was never emitted.

Failure Summary

Shard 0 of stage-b-test-1-gpu-large-amd-rocm720 runs 11 test files sequentially under a single 45-minute Run test timeout. Today the cumulative wall-time of the first 7 completed files exhausted the budget, so the job was killed at the start of file 8/11. This is a job-level timeout, not a specific test breaking.

Regression Status

New failure mode today (timeout). The job has failed 4 consecutive scheduled runs, but the reason changed:

Recent history of job stage-b-test-1-gpu-large-amd-rocm720 (linux-mi325-1gpu-sglang, 0) (branch main, event schedule, completed runs only):

Date	Run	Files reached	Failure mode
Jul 08	28888292440	8/11 (killed)	⏱️ 45-min timeout (new)
Jul 06	28813230096	11/11	exit 255 — only test_bench_serving_1gpu_part1.py failed (perf, exit 1); 10/11 passed; total 43:07
Jul 05	28749537427	11/11	exit 255 — same perf file; total 43:02
Jul 04	28714582765	11/11	exit 255 — same perf file; total 41:29

The prior 3 runs finished all 11 files in 41–43 minutes — a razor-thin margin under the 45-minute cap. Today they did not finish because several files ran a few minutes slower (see Facts). The recurring test_bench_serving_1gpu_part1.py perf failure is a separate, carrying-over issue and cannot be confirmed present today (the summary never ran).

Failure Cluster

"stage-b-large shard 0 exceeds 45-min Run-test timeout" (job-level timeout, symptom-level grouping).

Facts (verified observations)

• Timeout: ##[error]The action 'Run test' has timed out after 45 minutes at 2026-07-08T01:14:23 (log line 25006).
• Shard start Begin (0/11) at 00:29:23; wall-clock to timeout = exactly 45:00; only reached Begin (8/11).
• Per-file elapsed today vs the 3-run prior average (from filename=…elapsed= markers):

  File	Today	Prior avg	Δ
  perf/test_bench_serving_1gpu_part1.py	1117	~1066	+51
  sessions/test_streaming_session.py	514	~527	−13
  spec/eagle/test_adaptive_speculative.py	242	~181	+61
  cuda_graph/piecewise/test_piecewise_cuda_graph_support_1_gpu.py	462	~400	+62
  sessions/test_session_control.py	229	~174	+55
  attention/unittests/dense/test_triton.py	82	~61	+21

• Cumulative overrun today ≈ +230 s vs prior; applied to a shard that was already finishing at ~43 min, this tips it past the 45-min cap.
• test_piecewise_cuda_graph_support_1_gpu.py consistently overruns its own estimate (elapsed=462, estimated_time=180) in every run, not just today.
• No GPU fault, crash, or assertion appears before the timeout; only benign warnings (No module named 'cutlass', aiter RoPE precision notice, tl.where deprecation).

Hypothesised Causes (with confidence, with disconfirming evidence)

• [FACT] The shard's cumulative sequential test time exceeded the 45-minute Run test timeout.
  • Supporting: wall-clock reached exactly 45:00 and was killed mid-file 8/11; prior runs completed the identical file set in 41–43 min with no headroom.
  • Disconfirming: (none) — the timeout is directly observed.
• [LOW] Ordinary run-to-run duration variance (not a code regression) pushed this run over the edge.
  • Supporting: the slowdowns are spread across 4 unrelated files (spec, piecewise cuda graph, session control, triton attention), which points to shared runner/GPU/HF-cache variance rather than one culprit; no single file dominates the overrun.
  • Disconfirming: cannot fully exclude a broad perf regression in 3abdbab9..11cea29c since multiple files each slowed ~30%; a bisect would be needed to rule it in/out.
• [SPECULATION] A specific sglang commit slowed the tests.
  • Supporting: none — no code-level evidence tied to any of the slower files.
  • Disconfirming: the same slow files pass functionally; only their duration grew, and the shard was already at the cliff before today.

No hypothesis above LOW confidence for a code cause; the timeout itself is FACT. The structural cause (shard budget ≈ timeout budget) is the actionable finding.

In-flight Fix Check

• In-flight fix: ❌ none found for the stage-b-large shard-0 timeout budget. Closest related work is #29594 ("ci(amd): consolidate stage-b-test-1-gpu-small-amd 14→6 shards"), which rebalances a different shard family and does not touch this job. (Searched open PRs for stage-b amd timeout, test partition balance amd, test_bench_serving amd.)

Suggested Triage Steps

• Confirm the structural cause: sum of per-file estimated times for shard 0 already ≈ 46 min for just 8 of 11 files — the partition is provisioned above the 45-min step cap with zero slack. Rebalance shard 0 (move 1–2 files, e.g. the chronically-overrunning test_piecewise_cuda_graph_support_1_gpu.py, to shard 1) or raise the Run test timeout-minutes for stage-b-test-1-gpu-large-amd-rocm720.
• Investigate why test_piecewise_cuda_graph_support_1_gpu.py consistently runs 2.2–2.6× its 180 s estimate on MI325/ROCm 7.2, and refresh the stale estimated_time values feeding the balancer.
• Track the recurring perf/test_bench_serving_1gpu_part1.py (exit 1) failure separately — it is the real red in the Jul 4–6 runs and is independent of today's timeout; today's run could not confirm its status.
• If a broad perf regression is suspected, bisect 3abdbab9bb..11cea29c90 measuring test_adaptive_speculative.py / test_session_control.py wall-time on an MI325 ROCm 7.2 container.

Status (factual)

• Persistence: Job stage-b…large…(shard 0) red 4 consecutive scheduled runs (Jul 4→8). Timeout mode specifically is new on Jul 8; Jul 4–6 were exit-255 (perf test) with the job finishing at 41–43 min.
• Scope: This shard/job on pr-test-amd-rocm720.yml. The parent run 28888292440 is still queued/in-flight overall; this job itself has completed (timed out).
• Blocked work: stage-b-large 1-GPU AMD ROCm 7.2 signal for shard 0 (attention unittests in files 8–11 never ran today).
• In-flight fix: ❌ none found for this shard's timeout; #29594 rebalances a different shard family only.

stage-b-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang, 6) — failed step(s): Run test

Commit Info

• sglang: 11cea29c9
• aiter: 9127c94a1 (v0.1.17.dev110+g9127c94a1; source: dev/patched image detected, rebuild skipped — AITER_COMMIT_OVERRIDE empty)

Failed Tests

Test File	Test Function	Error	Log
test/registered/hicache/test_hicache_variants.py	TestHiCacheMLA.setUpClass	Exception: Server process exited with code -9 (root cause KeyError: 'sglang' during model load)	Log

The other 9 test files in this shard passed (Test Summary: 9/10 passed).

Failure Summary

TestHiCacheMLA.setUpClass in test/registered/hicache/test_hicache_variants.py failed to launch its server for deepseek-ai/DeepSeek-Coder-V2-Lite-Instruct (--trust-remote-code --enable-hierarchical-cache). The scheduler crashed during load_model with KeyError: 'sglang' raised inside the model's trust-remote-code modeling_deepseek.py (ATTENTION_CLASSES[config._attn_implementation]), after sglang's native DeepseekV2ForCausalLM failed to import and forced a fallback to the Transformers backend. The server process then exited with code -9.

Regression Status

New regression (first failing run is today's).

Recent history of test/registered/hicache/test_hicache_variants.py in workflow pr-test-amd-rocm720.yml (branch=main, event=schedule, completed runs only; shard number varies per run so matched by test file):

Date	Run	Job (shard)	Test File Status	Failed Function	Error
Jul 8	28888292440	shard 6	❌ Failed	TestHiCacheMLA.setUpClass	KeyError: 'sglang' → exit -9
Jul 6	28813230096	shard 6	✅ Passed (10/10, 583s)	—	—
Jul 5	28749537427	shard 0	✅ Passed (10/10, 557s)	—	—

First observed failure: 2026-07-08 (run 28888292440). Last known passing: 2026-07-06 (run 28813230096, head 3abdbab9b).

Failure Cluster

ROCm model import fails on missing cutlass → Transformers-backend fallback → KeyError: 'sglang'

Facts (verified observations)

• Failing-run log: Ignore import error when loading sglang.srt.models.deepseek_v2: No module named 'cutlass' appears 330 times across models (deepseek_v2, deepseek_v4, glm4_moe, bailing_moe, kimi, longcat, mistral_large_3, …). In the Jul 6 passing-run log the same string appears 0 times (verified via grep -c).
• Passing Jul 6 run loaded the model natively: Load weight end. elapsed=39.57 s, type=DeepseekV2ForCausalLM and scored mgsm_en_score=0.884. No "falling back to Transformers" message.
• Failing Jul 8 run: DeepseekV2ForCausalLM has no SGLang implementation, falling back to Transformers implementation → Using Transformers backend → traceback ending KeyError: 'sglang' at modeling_deepseek.py:1203, then Received sigquit from a child process, Server process exited with code -9.
• Import chain at failing SHA (verified against git show 11cea29c9:...): deepseek_v2.py:61 from ...dsa.dsa_indexer import Indexer → dsa_indexer.py:11 from sglang.jit_kernel.dsa import ... → jit_kernel/dsa/__init__.py line 1 unconditional from .cutedsl_paged_mqa_logits import CuteDSLPagedMQALogitsRunner, pick_dsl_expand → cutedsl_paged_mqa_logits.py top-level import cutlass (unavailable on ROCm).
• Regression window 3abdbab9b..11cea29c9 = 46 commits.

Hypothesised Causes (with confidence)

• [HIGH] Commit fefc1743a (#25220 "Cute-DSL FP8 MQA logits") introduced the unconditional top-level import cutlass into the jit_kernel/dsa import chain that deepseek_v2.py pulls in at module load, so on ROCm the model import raises ModuleNotFoundError: No module named 'cutlass', forcing the Transformers-backend fallback which then hits KeyError: 'sglang'.
  • Supporting: culprit landed 2026-07-06 23:34 PDT — after the last passing run head 3abdbab9b and before failing head 11cea29c9 (git merge-base --is-ancestor confirms it is NOT in the pass run); it added jit_kernel/dsa/__init__.py + cutedsl_paged_mqa_logits.py with unguarded import cutlass; cutlass-error count went 0→330 between the two runs.
  • Disconfirming: (none found) — the import chain, timing, and error-count delta all converge.
  • How to verify: on an MI3xx ROCm-7.2 container, python -c "import sglang.srt.models.deepseek_v2" at 11cea29c9 should raise the cutlass ImportError; it should succeed after applying #30374.
• [FACT] The trust-remote-code modeling_deepseek.py ATTENTION_CLASSES dict has no 'sglang' key, so the Transformers-backend fallback for this model is itself broken — a secondary latent issue exposed only once the fallback path is taken. (Not the trigger; the trigger is the cutlass import failure above.)

In-flight Fix Check

• In-flight fix: ✅ already MERGED — 40a68521c (#30374 "[AMD] Fix DeepSeekV4 server cutlass error"), landed 2026-07-07 21:43 UTC. It wraps the CuteDSL import in if not is_hip() in jit_kernel/dsa/__init__.py. This run (28888292440) was triggered 2026-07-07 18:11 UTC — ~3.5 h before the fix merged, so it ran without the fix.
• Search run: curl ".../search/issues?q=repo:sgl-project/sglang+is:pr+cutlass+hip" and git log -S"is_hip" -- jit_kernel/dsa/__init__.py both surface #30374.

Suggested Triage Steps

• No new PR needed — verify the already-merged fix 40a68521c (#30374) resolves this by confirming the next scheduled pr-test-amd-rocm720.yml run (head SHA ≥ 40a68521c) shows test_hicache_variants.py green.
• If it recurs after the fix, check for other model modules with unguarded top-level import cutlass reachable on ROCm: grep -rn "^from .cutedsl\|^import cutlass" python/sglang/jit_kernel python/sglang/srt.
• Separately consider hardening the Transformers-backend fallback so a trust-remote-code model without a 'sglang' entry in ATTENTION_CLASSES fails with a clear message rather than a bare KeyError (latent secondary issue).

Status

• Persistence: New today — 1 completed failing run (2026-07-08); passed in the two prior completed runs (Jul 5, Jul 6).
• Scope: test_hicache_variants.py (TestHiCacheMLA) on AMD ROCm 7.2. Blast radius is broad in principle (any ROCm test serving a model whose import chain reaches jit_kernel/dsa, e.g. DeepSeek MLA models), but this shard's other 9 tests passed.
• Blocked work: HiCache MLA CI signal on AMD ROCm 7.2 for this one scheduled run only.
• In-flight fix: ✅ already merged — 40a68521c (#30374); expected green on next scheduled run.

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-07-08 23:50 UTC)

[amd-bot]

nightly-test-amd.yml — 53 failure(s)

Run started (UTC): 2026-07-07 18:12 UTC
Last scanned (UTC): 2026-07-08 21:46 UTC
Commits: sglang 8d2b66f

Summary

Counts: 53 failures · 6 clusters

#	Cluster	Job	Test File	Test Function	Status	Hypothesis (confidence)
1	VRAM cleanup / zombie KFD (MI35x)	nightly-8-gpu-mi35x-qwen35	N/A (infra)	N/A	≥4 days persistent	[FACT] pre-existing VRAM leak on zf775 pool; not code
2	VRAM cleanup / zombie KFD (MI35x)	nightly-8-gpu-mi35x-deepseek-r1-mxfp4	N/A (infra)	N/A	3+ days persistent	[FACT] zombie KFD context
3	VRAM cleanup / zombie KFD (MI35x)	nightly-8-gpu-mi35x-deepseek-r1-mxfp4-kv-fp8	N/A (infra)	N/A	3 days persistent	[FACT] zombie KFD context
4	VRAM cleanup / zombie KFD (MI35x)	nightly-8-gpu-mi35x-grok1-int4	N/A (infra)	N/A	4 days persistent	[FACT] zombie KFD context
5	VRAM cleanup / zombie KFD (MI35x)	nightly-8-gpu-mi35x-qwen35-triton-dcp	N/A (infra)	N/A	3 days persistent	[FACT] zombie KFD context
6	VRAM cleanup / zombie KFD (MI35x)	nightly-8-gpu-mi35x-deepseek-r1-mxfp4-kv-fp8	N/A (infra)	N/A	recurring	[FACT] zombie KFD context
7	VRAM cleanup / zombie KFD (MI35x)	nightly-8-gpu-mi35x-deepseek-r1-mxfp4	N/A (infra)	N/A	3 days persistent	[FACT] zombie KFD context
8	VRAM cleanup / zombie KFD (MI35x)	nightly-8-gpu-mi35x-glm5-mxfp4	N/A (infra)	N/A	4/6 runs	[FACT] zombie KFD context
9	VRAM cleanup / zombie KFD (MI35x)	nightly-8-gpu-mi35x-deepseek-r1-hicache	N/A (infra)	N/A	3 days persistent	[FACT] zombie KFD context
10	VRAM cleanup / zombie KFD (MI35x)	nightly-8-gpu-mi35x-grok1-int4	N/A (infra)	N/A	4 days persistent	[FACT] zombie KFD context
11	VRAM cleanup / zombie KFD (MI35x)	nightly-8-gpu-mi35x-kimi-k26	N/A (infra)	N/A	intermittent	[FACT] zombie KFD context
12	VRAM cleanup / zombie KFD (MI35x)	nightly-perf-8-gpu-mi35x-deepseek-v32-basic	N/A (infra)	N/A	intermittent	[FACT] zombie KFD context
13	VRAM cleanup / zombie KFD (MI35x)	nightly-perf-8-gpu-mi35x-deepseek-v32-mtp	N/A (infra)	N/A	intermittent	[FACT] zombie KFD context
14	VRAM cleanup / zombie KFD (MI35x)	nightly-accuracy-8-gpu-mi35x-deepseek-v32	N/A (infra)	N/A	2/5 runs	[FACT] zombie KFD context
15	VRAM cleanup / zombie KFD (MI35x)	nightly-accuracy-8-gpu-mi35x-deepseek-v32	N/A (infra)	N/A	2 days	[FACT] zombie KFD context
16	VRAM cleanup / zombie KFD (MI35x)	[nightly-accuracy-8-gpu-mi35x-deepseek-v32-mtp](https://github.com/sgl-project/sglang/actions/runs/28964511688/job/85944149

---

Job	Failed Steps	Started
nightly-2-gpu-mi35x-deepseek-r1-mxfp4-tp2	Ensure VRAM is clear	2026-07-07T18:12
nightly-2-gpu-mi35x-glm51-mxfp4	Accuracy Test MI35x (2-GPU GLM-5.1-MXFP4 GSM8K)	2026-07-07T18:12
nightly-accuracy-8-gpu-mi35x	Accuracy Test MI35x (8-GPU GPT-OSS)	2026-07-07T18:48
nightly-8-gpu-mi35x-grok1-int4	Ensure VRAM is clear	2026-07-07T18:48
nightly-8-gpu-mi35x-deepseek-r1-mxfp4-ar-fusion	Accuracy Test MI35x (8-GPU DeepSeek-R1-MXFP4 AllReduce Fusion)	2026-07-07T18:36
nightly-accuracy-8-gpu-mi35x-deepseek-v32	Ensure VRAM is clear	2026-07-07T18:50
nightly-8-gpu-mi35x-deepseek-r1-mxfp4-tp4	Accuracy Test MI35x (DeepSeek-R1-MXFP4 TP4 MTP), Accuracy Test MI35x (DeepSeek-R1-MXFP4 TP4)	2026-07-07T18:32
nightly-perf-8-gpu-mi35x-deepseek-v32-mtp	Ensure VRAM is clear	2026-07-07T19:30
nightly-8-gpu-mi35x-deepseek-r1-mxfp4-kv-fp8	Ensure VRAM is clear	2026-07-07T18:48
nightly-8-gpu-mi35x-kimi-k26	Ensure VRAM is clear	2026-07-07T19:05
nightly-8-gpu-mi35x-deepseek-r1-hicache	Ensure VRAM is clear	2026-07-07T18:51
nightly-8-gpu-mi35x-glm51	Ensure VRAM is clear	2026-07-07T19:15
nightly-8-gpu-mi35x-qwen35-triton-dcp	Ensure VRAM is clear	2026-07-07T19:12
nightly-accuracy-8-gpu-mi35x-deepseek-v32-mtp	Accuracy Test MI35x (8-GPU DeepSeek-V3.2 TP+MTP)	2026-07-07T18:52
nightly-8-gpu-mi35x-qwen35	Ensure VRAM is clear	2026-07-07T19:19
nightly-perf-8-gpu-mi35x-deepseek-v32-basic	Ensure VRAM is clear	2026-07-07T19:03
nightly-8-gpu-mi35x-deepseek-r1-mxfp4	Ensure VRAM is clear	2026-07-07T19:17
nightly-8-gpu-mi35x-glm5-mxfp4	Ensure VRAM is clear	2026-07-07T19:21
nightly-8-gpu-mi35x-qwen3-235b-mxfp4	Accuracy Test + Performance Test MI35x (8-GPU Qwen3-235B-MXFP4)	2026-07-07T19:14
nightly-accuracy-2-gpu	Nightly Test (2-GPU)	2026-07-08T15:08
nightly-8-gpu-kimi-k26	N/A	2026-07-07T18:12
nightly-8-gpu-deepseek-v32	N/A	2026-07-07T18:12
nightly-accuracy-2-gpu-vlm	N/A	2026-07-07T18:12
nightly-8-gpu-grok2	N/A	2026-07-07T18:12
nightly-8-gpu-deepseek-v3-kv-fp8	N/A	2026-07-07T18:12
nightly-accuracy-8-gpu	N/A	2026-07-07T18:12
nightly-perf-2-gpu-text	N/A	2026-07-07T18:12
nightly-8-gpu-deepseek-v31	N/A	2026-07-07T18:12
nightly-perf-2-gpu-vlm	N/A	2026-07-07T18:12
nightly-8-gpu-deepseek-v32-mtp	N/A	2026-07-07T18:12
nightly-8-gpu-qwen3-235b	N/A	2026-07-07T18:12
nightly-8-gpu-grok1-int4	N/A	2026-07-07T18:12
nightly-8-gpu-glm51	N/A	2026-07-07T18:12
nightly-8-gpu-minimax-m27	N/A	2026-07-07T18:12
nightly-8-gpu-qwen35	N/A	2026-07-07T18:12
nightly-2-gpu-mi35x-glm51-mxfp4	Accuracy Test MI35x (2-GPU GLM-5.1-MXFP4 GSM8K)	2026-07-08T18:00
nightly-perf-8-gpu-mi35x-deepseek-v32-basic	Ensure VRAM is clear	2026-07-08T19:03
nightly-accuracy-8-gpu-mi35x-deepseek-v32-mtp	Ensure VRAM is clear	2026-07-08T18:55
nightly-8-gpu-mi35x-deepseek-r1-mxfp4-tp4	Accuracy Test MI35x (DeepSeek-R1-MXFP4 TP4 MTP), Ensure VRAM is clear	2026-07-08T19:06
nightly-8-gpu-mi35x-glm5-mxfp4	Ensure VRAM is clear	2026-07-08T19:08
nightly-8-gpu-mi35x-deepseek-r1-hicache	Ensure VRAM is clear	2026-07-08T18:58
nightly-8-gpu-mi35x-grok1-int4	Ensure VRAM is clear	2026-07-08T19:00
nightly-8-gpu-mi35x-qwen35	Accuracy Test MI35x (8-GPU Qwen 3.5)	2026-07-08T18:44
nightly-8-gpu-mi35x-kimi-k26	Ensure VRAM is clear	2026-07-08T20:04
nightly-8-gpu-mi35x-deepseek-r1-mxfp4	Ensure VRAM is clear	2026-07-08T19:59
nightly-accuracy-8-gpu-mi35x-deepseek-v32	Ensure VRAM is clear	2026-07-08T19:51
nightly-accuracy-8-gpu-mi35x	Ensure VRAM is clear	2026-07-08T19:56
nightly-8-gpu-mi35x-deepseek-r1-mxfp4-kv-fp8	Ensure VRAM is clear	2026-07-08T20:02
nightly-4-gpu-mi35x-minimax-m25	Ensure VRAM is clear	2026-07-08T19:54
nightly-8-gpu-mi35x-qwen3-235b-mxfp4	Ensure VRAM is clear	2026-07-08T20:07
nightly-perf-8-gpu-mi35x-deepseek-v32-mtp	Performance Test MI35x (8-GPU DeepSeek-V3.2 MTP)	2026-07-08T19:48
nightly-8-gpu-mi35x-qwen35-triton-dcp	Accuracy Test MI35x (8-GPU Qwen 3.5 Triton DCP)	2026-07-08T19:37
nightly-8-gpu-mi35x-glm51	Accuracy Test MI35x (8-GPU GLM-5.1 DSA)	2026-07-08T19:56

Per-Job Analysis

nightly-2-gpu-mi35x-deepseek-r1-mxfp4-tp2 — failed step(s): Ensure VRAM is clear

Commit Info

• sglang: 11cea29c
• aiter: N/A — the job failed at a pre-test host-cleanup step before any dependency install ran, so no [CI-AITER-CHECK] markers exist. (AITER_COMMIT_OVERRIDE was empty.)

Failed Tests

Test File	Test Function	Error	Log
N/A (infra)	N/A	VRAM cleanup unsuccessful after 3 attempts → exit code 1	Log

No test ran. The job aborted during the pre-test host hygiene step bash scripts/ci/amd/ensure_vram_clear.sh rocm, before checkout-of-deps, container start, or model load.

Failure Summary

The runner node linux-mi35x-gpu-2-98ncc-runner-nzklz began the job with GPU[0] already at 16% VRAM allocated (threshold is 5%). The cleanup script ran 3 kill+wait cycles; VRAM never dropped below threshold, so the step exited 1 (log lines 335–343). This is a host/runner infrastructure failure, not an sglang code or test failure.

Regression Status

Infrastructure issue (runner-specific VRAM leak / zombie KFD context). Recurring intermittently, tied to which physical node the job lands on — not to a code regression.

Recent history of job nightly-2-gpu-mi35x-deepseek-r1-mxfp4-tp2 (branch main, event schedule, completed runs only):

Date	Run	Job Status	Failure Type
Jul 7	28888348134	❌ Failed	VRAM cleanup / zombie KFD context (this run)
Jul 5	28749559519	✅ Passed	—
Jul 4	28714602438	✅ Passed	—
Jul 3	28676359018	❌ Failed	Same VRAM cleanup / zombie KFD context (verified in log)

Pattern: passes when the job lands on a clean node, fails when it lands on a node with a stuck KFD context. Same-symptom failure confirmed on Jul 3.

Failure Cluster

Pre-test VRAM cleanup failure (zombie KFD context on ROCm runner)

Facts (verified observations)

• GPU[0] : GPU Memory Allocated (VRAM%): 16 at job start (line 131) and unchanged through all 3 cleanup attempts (lines 169, 229, 287). GPU[1] stayed at 0%.
• Attempt 1 diagnostics showed one KFD process PID 2003480 PROCESS NAME UNKNOWN, VRAM USED 0 (line 200); attempts 2–3 showed No KFD PIDs currently running (lines 258, 316) — yet VRAM stayed at 16%. Classic orphaned/zombie KFD context: no host process holds the GPU but VRAM remains allocated.
• fuser/lsof found no device-file holders; no docker containers; no zombie host processes (lines 186–192, 246–252, 304–310).
• Script's own hint (lines 338–342): "almost certainly a zombie KFD context (see [Bug] Custom all-reduce IPC buffers use fixed VA, conflict with torch_memory_saver pause/resume → zombie processes on MI300X ROCm/aiter#2061). The node will need to be rebooted."
• Failed at ##[error]Process completed with exit code 1 (line 343).

Hypothesised Causes (with confidence)

• [FACT] Runner node started the job with 16% VRAM already allocated by a leaked/zombie KFD context that no live process owned; the cleanup script cannot reclaim driver-level allocations, so it correctly aborted.
  • Supporting: rocm-smi --showpids reports no live KFD PIDs while --showmemuse still reports 16% (lines 258/266, 316/324); fuser/lsof find no holders.
  • Disconfirming (for a code-cause): the same job passed Jul 4 and Jul 5 on the identical branch/matrix — a code regression would fail every run, not just runs that land on a dirty node.
  • How to verify: reboot / drain node linux-mi35x-gpu-2-98ncc-runner-nzklz and re-run; VRAM should read 0% on a clean boot.

In-flight Fix Check

• In-flight fix: ⚠️ #30431 "[DO NOT MERGE][AMD] ci: self-heal driver-level VRAM leaks + graceful GPU cleanup on shutdown" (open, created 2026-07-07 21:59 UTC — after this run). Directly targets this exact driver-level VRAM-leak class, but is currently marked DO NOT MERGE (WIP). Track it; do not open a duplicate.
• Searched: is:pr is:open VRAM cleanup → 11 hits, #30431 is the on-point one.

Suggested Triage Steps

• Reboot or cordon runner node linux-mi35x-gpu-2-98ncc-runner-nzklz; confirm rocm-smi --showmemuse reads 0% on GPU[0] before returning it to the pool.
• Re-run the job — expected to pass once it lands on a clean node (as it did Jul 4/Jul 5).
• Follow WIP #30431 for a permanent self-heal of driver-level VRAM leaks; nudge its author toward merge-readiness since this leak is recurring on the MI35x fleet.
• If the leak keeps recurring on the same physical node, investigate whether a prior job's crash (e.g. an earlier failing deepseek-r1-mxfp4 shard on the same host) is what orphans the KFD context — the showpids UNKNOWN process suggests a hard crash left the context behind (cf. [Bug] Custom all-reduce IPC buffers use fixed VA, conflict with torch_memory_saver pause/resume → zombie processes on MI300X ROCm/aiter#2061).

Status (factual)

• Persistence: Intermittent infra failure; same symptom on Jul 3 and Jul 7, with clean passes Jul 4–5. Node-dependent, not run-count-monotonic.
• Scope: This job / this runner node. Pre-test host-cleanup step — never reached the DeepSeek-R1 MXFP4 TP2 test workload. No sglang test signal produced by this run.
• Blocked work: DeepSeek-R1 MXFP4 TP2 nightly accuracy/perf coverage on MI35x is not exercised whenever the job lands on a dirty node.
• In-flight fix: ⚠️ #30431 (WIP, DO NOT MERGE) — driver-level VRAM leak self-heal.

nightly-2-gpu-mi35x-glm51-mxfp4 — failed step(s): Accuracy Test MI35x (2-GPU GLM-5.1-MXFP4 GSM8K)

Commit Info

• sglang: 11cea29c
• aiter: 9127c94a — v0.1.17.dev110+g9127c94a1 (source: dev image preserved, AITER_COMMIT_OVERRIDE empty)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi35x/test_gpt_oss_w4a8_mxfp4_eval_mi35x.py	test_gpt_oss_accuracy (TestGptOssW4A8Mxfp4EvalMI35x)	AssertionError: Failed models: ['amd/gpt-oss-120b-w-mxfp4-a-fp8'] (raised at line 247)	Log

The other 3 accuracy tests in this job (test_gpt_oss_eval_mi35x.py, test_qwen3_coder_next_eval_mi35x.py, test_qwen3_moe_eval_mi35x.py) all passed — job summary was 3/4 passed.

Failure Summary

The amd/gpt-oss-120b-w-mxfp4-a-fp8 server never became ready because its HuggingFace weight download stalled during startup. Rank TP4 (pid 12910) acquired the download filelock at 19:02:48 and got stuck inside the HF Hub Xet backend (xet_get); no Load weight end was ever emitted for this model. All 8 scheduler ranks then hit two consecutive rounds of soft watchdog timeouts, the server never served a request within the ~905 s launch window, and the test harness marked the model failed.

Regression Status

New failure today (intermittent infrastructure / network), not a code regression.

Recent history of test_gpt_oss_w4a8_mxfp4_eval_mi35x.py (test_gpt_oss_accuracy) in job nightly-2-gpu-mi35x-glm51-mxfp4 (branch main, event schedule, completed runs only):

Date	Run	Job	Test File Status	Failed Function	Error
Jul 7	28888348134	nightly-2-gpu-mi35x-glm51-mxfp4	❌ Failed	test_gpt_oss_accuracy	download stall → server not ready
Jul 5	28749559519	nightly-2-gpu-mi35x-glm51-mxfp4	✅ Passed	—	—
Jul 4	28714602438	nightly-2-gpu-mi35x-glm51-mxfp4	✅ Passed	—	—
Jul 3	28676359018	nightly-2-gpu-mi35x-glm51-mxfp4	⏭️ Not run	—	job failed at step 3 "Ensure VRAM is clear" (zombie KFD context, ROCm/aiter#2061); accuracy step skipped

First observed failure of this test file: Jul 7. Last known passing: Jul 5. The test passed the last two times it actually executed, so the code path is healthy; today's failure is at the weight-download layer. Lookback window: last 5 scheduled runs (Jun 30 – Jul 7); the Jun 30/Jul 1/Jul 6 runs were cancelled (excluded).

Failure Cluster

HuggingFace weight-download stall (HF Xet backend) during server startup → soft watchdog timeout → server never ready.

Facts (verified observations, no interpretation)

• AssertionError: Failed models: ['amd/gpt-oss-120b-w-mxfp4-a-fp8'] at test_gpt_oss_w4a8_mxfp4_eval_mi35x.py:247; Ran 1 test in 905.297s → FAILED (failures=1).
• TP4 [CI Download] Process 12910 ACQUIRED lock for amd/gpt-oss-120b-w-mxfp4-a-fp8 at 19:02:48; lock file /data2/models/huggingface/hub/.sglang_locks/download_c5adc0261df2da8f.lock.
• py-spy dumps show the lock-holder stuck at xet_get (huggingface_hub/file_download.py:568) ← snapshot_download ← ci_download_with_validation_and_retry (ci_weight_validation.py:1955) ← download_weights_from_hf (weight_utils.py:571/610); the other 7 ranks blocked at _poll_until_acquired (filelock/_api.py:568) inside download_weights_from_hf.
• No Load weight end line exists for amd/gpt-oss-120b-w-mxfp4-a-fp8 (the next Load weight end in the log belongs to the subsequent, passing Qwen3-Coder-Next test).
• Two rounds of soft watchdog timeouts (watchdog_timeout=300, soft=True): 19:07:33–19:07:37 (Tokenizer + 8 TP) and 19:12:33–19:12:37 (Tokenizer + 8 TP). soft=True only logs; it does not kill the process.
• The failing model used cache root /data2/models/huggingface/hub/ (fresh download required), whereas the passing openai/gpt-oss-20b and openai/gpt-oss-120b tests earlier in the job used /sgl-data/hf-cache/hub/ and downloaded/loaded normally.
• hf-xet version in the image: 1.5.1; huggingface_hub 1.22.0.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [MEDIUM] HuggingFace Xet download stall / network hang while fetching amd/gpt-oss-120b-w-mxfp4-a-fp8 weights (uncached on /data2/...), holding the download filelock so all ranks block and the server never warms up.
  • Supporting: lock-holder py-spy stack is parked in xet_get/snapshot_download; no Load weight end ever emitted; two full 300 s soft-watchdog cycles elapse with zero download progress; the other 3 models in the same job (already cached under /sgl-data/hf-cache) loaded fine.
  • Disconfirming: no explicit ConnectionError/ReadTimeout/HTTP status appears in the log (the download simply hangs), so a data-corruption or Xet-server-side issue can't be fully ruled out from logs alone.
  • How to verify: on an MI35x runner, HF_HUB_ENABLE_HF_TRANSFER/xet-enabled huggingface-cli download amd/gpt-oss-120b-w-mxfp4-a-fp8 --cache-dir /data2/models/huggingface/hub and time it; retry with HF_HUB_DISABLE_XET=1 to see if the non-Xet path completes.
• [LOW] The amd/gpt-oss-120b-w-mxfp4-a-fp8 weights were not pre-seeded in the /data2/... cache on this node, forcing a large fresh download that exceeded the launch budget.
  • Supporting: distinct cache root vs the passing (cached) gpt-oss models; fresh ACQUIRED lock with no prior "already cached" marker.
  • Disconfirming: the test passed on Jul 4/Jul 5, implying the model is normally available quickly — consistent with a transient stall rather than a permanent cache-miss.
  • How to verify: check whether /data2/models/huggingface/hub on the MI35x runner already contains a complete snapshot of this repo.
• Not code-related: the test passed on Jul 4 and Jul 5 on nearby SHAs and the failure is at the download layer, not in model/kernel code — a code bisect is unlikely to be productive here.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for xet+download, weight+download+timeout, download+hang+CI (via search/issues?q=repo:sgl-project/sglang+is:pr+is:open+...); the returned PRs (e.g. #29178, #28905, #29328) address Kimi/MXFP4 weight-loading logic, not the HF Xet download stall.

Suggested Triage Steps

• Re-run the job — if amd/gpt-oss-120b-w-mxfp4-a-fp8 loads on retry, confirm it as a transient HF Xet download stall and no code action is needed.
• Pre-seed / verify the amd/gpt-oss-120b-w-mxfp4-a-fp8 snapshot in the /data2/models/huggingface/hub cache on the MI35x runner so the accuracy step doesn't depend on a live large download.
• Reproduce the download in isolation on the runner (Xet enabled vs HF_HUB_DISABLE_XET=1) to determine whether the Xet backend is the stall point; if so, consider disabling Xet or adding a hard download timeout in ci_weight_validation.ci_download_with_validation_and_retry so a stalled download fails fast + retries instead of consuming the whole launch window.
• Consider a non-soft (hard) watchdog or a launch-time download timeout for CI so a hung download aborts and retries rather than silently timing out.

Status (factual, no priority assignment)

• Persistence: 1 completed run (Jul 7); the test passed the two prior runs where it actually executed (Jul 4, Jul 5). Not previously seen at the test-file level in the queryable window.
• Scope: 1 test (test_gpt_oss_w4a8_mxfp4_eval_mi35x.py) in 1 job (nightly-2-gpu-mi35x-glm51-mxfp4); the job's other 3 accuracy tests passed. No evidence of the same download stall in sibling jobs this run.
• Blocked work: blocks the GLM-5.1 MXFP4 2-GPU nightly accuracy signal for the amd/gpt-oss-120b-w-mxfp4-a-fp8 model only.
• In-flight fix: ❌ none found — needs a runner cache fix / download-timeout hardening or a simple re-run if transient.

nightly-accuracy-8-gpu-mi35x — failed step(s): Accuracy Test MI35x (8-GPU GPT-OSS)

Commit Info

• sglang: 11cea29c ([diffusion][cache-dit] add dual-transformer Cache-DiT adapter specs (#30150))
• aiter: 9127c94a → v0.1.17.dev110+g9127c94a1 (source: dev/patched CI image rocm/sgl-dev:v0.5.14-rocm700-mi35x-20260707, preserved; AITER_COMMIT_OVERRIDE empty). Not relevant to this failure.

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi35x/test_gpt_oss_w4a8_mxfp4_eval_mi35x.py	test_gpt_oss_accuracy (TestGptOssW4A8Mxfp4EvalMI35x)	AssertionError: Failed models: ['amd/gpt-oss-120b-w-mxfp4-a-fp8'] (line 247)	Job log

Suite result: 3/4 passed. The other three files passed: test_gpt_oss_eval_mi35x.py (acc 0.520/0.825), test_qwen3_coder_next_eval_mi35x.py (acc 0.965/0.970), test_qwen3_moe_eval_mi35x.py (acc 0.925).

Failure Summary

The amd/gpt-oss-120b-w-mxfp4-a-fp8 server never became ready because model-weight download stalled. Rank TP4 (PID 12910) acquired the HF download lock at 19:02:48 and then hung inside the HuggingFace Xet download path; the other 7 TP ranks blocked waiting on the same lock. All ranks tripped the 300 s soft watchdog (during model load, not forward), the server never served a request, and the eval harness recorded the model as failed after 905 s. This is a weight-provisioning/download stall, not an accuracy, kernel, or model-code failure.

Regression Status

New failure mode — first time this test actually executed in the queryable window. In the previous three completed scheduled runs, the nightly-accuracy-8-gpu-mi35x job died at the upstream "Ensure VRAM is clear" step (infra) and never reached the accuracy suite, so there is no comparable pass/fail for test_gpt_oss_w4a8_mxfp4_eval_mi35x.py in that window.

Recent history of test_gpt_oss_w4a8_mxfp4_eval_mi35x.py (test_gpt_oss_accuracy) in job nightly-accuracy-8-gpu-mi35x (branch=main, event=schedule, completed runs only):

Date	Run	Job Conclusion	Test File Status	Note
Jul 07	28888348134	❌ failure	❌ Failed	weight download (HF Xet) stalled → model failed
Jul 05	28749559519	❌ failure	⏭️ Did not run	job aborted at "Ensure VRAM is clear" (exit 1)
Jul 04	28714602438	❌ failure	⏭️ Did not run	job aborted at "Ensure VRAM is clear" (exit 1)
Jul 03	28676359018	❌ failure	⏭️ Did not run	job aborted at "Ensure VRAM is clear" (exit 1)

Lookback: last 4 scheduled nightly-test-amd.yml runs on main (Jul 06 run was cancelled, excluded). No green run of this test file is present in the window.

Failure Cluster

HF Xet weight-download stall during model load (server never ready → eval fails model)

Facts (verified observations)

• command=sglang serve --model-path amd/gpt-oss-120b-w-mxfp4-a-fp8 ... --tp 8 --attention-backend triton (verbatim from log).
• [CI Download] Process 12910 (TP4) ACQUIRED lock for amd/gpt-oss-120b-w-mxfp4-a-fp8 at 19:02:48, lock file /data2/models/huggingface/hub/.sglang_locks/download_c5adc0261df2da8f.lock.
• py-spy dump of PID 12910 at 19:12:37 (native stack): download_weights_from_hf (weight_utils.py:610) → ci_download_with_validation_and_retry (ci_weight_validation.py:1955) → snapshot_download (_snapshot_download.py:492) → hf_hub_download (file_download.py:1018) → _download_to_tmp_and_move (file_download.py:1925) → xet_get (file_download.py:568).
• Waiter ranks (e.g. TP5/PID 12911) stuck in download_weights_from_hf (weight_utils.py:571) → filelock _poll_until_acquired.
• Scheduler/TokenizerManager watchdog timeout (watchdog_timeout=300, self.soft=True) fired on all 8 ranks at 19:07:33–19:07:37 and again 19:12:37; Pyspy failed (py-spy dump --native --pid 12910). Error: Failed to get stack traces.
• Test end: Ran 1 test in 905.297s → FAILED (failures=1) at 19:17:18; elapsed=910, estimated_time=900.
• The three passing tests in the same suite downloaded from a different cache volume /sgl-data/hf-cache/hub/ and each ACQUIRED lock and proceeded normally; only the failing model used /data2/models/huggingface/hub/.

Hypothesised Causes (with confidence)

• [MEDIUM] The amd/gpt-oss-120b-w-mxfp4-a-fp8 weights were not pre-provisioned on the runner and a live HF Xet download stalled/was too slow, so the server never finished loading within the ~900 s budget.
  • Supporting: py-spy stack of the lock-holding rank is squarely inside xet_get/snapshot_download; the soft watchdog fires during model load, before any forward pass; no kernel fault, HIP error, or accuracy mismatch anywhere in the log; the model uses the /data2/models cache volume while the three passing models use /sgl-data/hf-cache and load fine.
  • Disconfirming: (none found) — no evidence of an sglang code error on the load path; the failure signature is purely download I/O.
  • How to verify: check whether amd/gpt-oss-120b-w-mxfp4-a-fp8 is present/complete under /data2/models/huggingface/hub/ on the MI35x runner; re-run the single file test_gpt_oss_w4a8_mxfp4_eval_mi35x.py after pre-seeding the weights, or with HF_XET_HIGH_PERFORMANCE/Xet disabled, and confirm the server reaches ready state.
• [LOW] Stale/contended lock on the shared /data2/models volume delayed acquisition.
  • Supporting: all 7 non-holder ranks sat in _poll_until_acquired for the whole run.
  • Disconfirming: TP4 did acquire the lock and was actively in the download path — the bottleneck was the download itself, not lock contention.

No hypothesis reaches HIGH; the evidence points to infrastructure (weight provisioning/network), not an sglang code regression.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for gpt-oss w4a8 download, xet download timeout, gpt_oss mxfp4 accuracy (via search/issues?q=repo:sgl-project/sglang+is:pr+is:open+...). Hits (#29178, #28905, #30050, #28584) are unrelated (Kimi-K2.6 MoE scale loading, MLX gpt-oss support, MI300 runner CI).

Suggested Triage Steps

• Verify weight presence on the runner: ls /data2/models/huggingface/hub/models--amd--gpt-oss-120b-w-mxfp4-a-fp8 on the MI35x host; if incomplete, pre-seed the cache so the nightly run doesn't download 120B weights live.
• Re-run only test_gpt_oss_w4a8_mxfp4_eval_mi35x.py on the MI35x runner with the weights pre-cached; if it passes, confirms a provisioning/download stall (not code).
• If a live download is unavoidable, consider bumping this model's server launch/health-check timeout above the download time, or disabling HF Xet (HF_HUB_DISABLE_XET=1) for this large model, and re-run to isolate whether Xet is the stall point.
• Separately, the recurring "Ensure VRAM is clear" abort on Jul 03–05 is a distinct infra issue on the same job worth tracking on its own.

Status

• Persistence: The nightly-accuracy-8-gpu-mi35x job has been red 4/4 recent completed runs, but for different reasons — Jul 03–05 aborted at VRAM-clear (infra), Jul 07 (today) is the first run to reach the suite and fails only on the gpt-oss w4a8 weight download. This specific test-level failure is newly observed today.
• Scope: 1 test file in 1 job (nightly-accuracy-8-gpu-mi35x); the other 3 accuracy tests in the suite passed. No other backend/job affected by this signature.
• In-flight fix: ❌ none found — needs weight-provisioning check or new investigation.

nightly-8-gpu-mi35x-grok1-int4 — failed step(s): Ensure VRAM is clear

Commit Info

• sglang: 11cea29c
• aiter: N/A — the job died at a host-side pre-flight step before dependency install / container start, so no [CI-AITER-CHECK] markers were emitted.

Failed Tests

Test File	Test Function	Error	Log
N/A (infra pre-flight)	N/A	FAILED: VRAM cleanup unsuccessful after 3 attempts → exit 1	Log line 537

No test ran. The job failed in the bash scripts/ci/amd/ensure_vram_clear.sh rocm pre-flight step (the step immediately after checkout), before any docker container, dependency install, or pytest was invoked.

Failure Summary

The runner's pre-flight VRAM cleanup gate failed: GPUs 0–3 were each pinned at 16% VRAM allocated (threshold is 5%) on entry, and three cleanup attempts (SIGTERM→SIGKILL of GPU-device holders + 30 s wait each) failed to release it. Critically, rocm-smi --showpids reported "No KFD PIDs currently running" and fuser/docker ps found no host process or container holding the GPUs, yet VRAM stayed allocated. The script itself flags this as an unreclaimable zombie KFD context (ref ROCm/aiter#2061) requiring a node reboot.

Regression Status

Known recurring infrastructure failure (not a code regression).

Recent history of the mi35x VRAM pre-flight gate in job nightly-8-gpu-mi35x-grok1-int4 (same branch main, event schedule, completed runs only):

Date	Run	Job	Test File Status	Failed Function	Error
Jul 7	28888348134	mi35x-grok1-int4	❌ Failed (infra)	N/A — VRAM gate	VRAM cleanup unsuccessful after 3 attempts
Jul 5	28749559519	mi35x-grok1-int4	❌ Failed (infra)	N/A — VRAM gate	same
Jul 4	28714602438	mi35x-grok1-int4	❌ Failed (infra)	N/A — VRAM gate	same
Jul 3	28676359018	mi35x-grok1-int4 (85050499038)	❌ Failed	N/A	(mi35x variant, failing)

Note: the non-mi35x sister job nightly-8-gpu-grok1-int4 passed on Jul 3/4/5, so this is specific to the mi35x runner pool, not to the grok1-int4 test or the sglang commit. First observed in the queryable window: Jul 3 2026; no passing mi35x run found Jul 3–7.

Failure Cluster

Pre-flight VRAM cleanup gate failure (zombie KFD context on mi35x runner pool)

Facts (verified observations, no interpretation)

• Failure step is ensure_vram_clear.sh rocm, exit code 1 (line 545); the log ends at post-job cleanup with no other steps executed.
• On entry, GPUs 0–3 each report GPU Memory Allocated (VRAM%): 16; GPUs 4–7 report 0 (lines 167–171).
• All 3 cleanup attempts report No docker containers found on host, No KFD PIDs currently running, empty fuser/zombie-process lists, yet VRAM remains at 16% (lines 262, 371, 480).
• Script's own diagnosis: almost certainly a zombie KFD context (see ROCm/aiter#2061). The node will need to be rebooted (lines 540–543).
• The three failing runs ran on three different runner instances in the same pool: zf775-runner-9bdzn (Jul 7), zf775-runner-sgl9n (Jul 5), zf775-runner-494d5 (Jul 4) — so this is not a single bad node but the linux-mi35x-gpu-8-zf775 pool.
• No [CI-AITER-CHECK], no pip/version output, no stack trace exist in the log (job never reached those steps).

Hypothesised Causes (with confidence, with disconfirming evidence)

• [FACT] The job failed because the pre-flight VRAM gate could not bring GPUs 0–3 below the 5% threshold (verbatim in log).
• [MEDIUM] A persistent driver-level / zombie-KFD VRAM leak on the zf775 mi35x runner pool leaves ~16% VRAM allocated with no owning host process, which the userspace kill-based cleanup cannot reclaim.
  • Supporting: no KFD PIDs, no containers, no fuser holders, yet VRAM persists across 3 SIGKILL+wait cycles; identical symptom on 3 distinct nodes over 3+ days; script explicitly names zombie KFD context per [Bug] Custom all-reduce IPC buffers use fixed VA, conflict with torch_memory_saver pause/resume → zombie processes on MI300X ROCm/aiter#2061.
  • Disconfirming: cannot inspect the prior job on each node from this log, so the exact leaking workload isn't proven from here; possible the leak is left by whatever job previously ran on each node rather than a systemic driver bug.
  • How to verify: reboot a zf775 node, confirm VRAM clears; then correlate which job last ran on each node before the failing run to see if a specific mi35x workload leaks.
• [LOW] The grok1-int4 mi35x workload itself leaks VRAM that survives container teardown.
  • Supporting: same test lineage repeatedly on the affected pool.
  • Disconfirming: the gate fails on entry (before this job's container starts), so any leak was inherited from a prior job, not caused by this run; the non-mi35x grok1-int4 job passes.

In-flight Fix Check

• In-flight fix: ✅ #30431 — "[DO NOT MERGE][AMD] ci: self-heal driver-level VRAM leaks + graceful GPU cleanup on shutdown" (open since 2026-07-07, currently marked DO NOT MERGE). Directly targets this exact symptom; track it rather than opening a duplicate.
• Search run: search/issues?q=repo:sgl-project/sglang+is:pr+is:open+VRAM+cleanup (11 hits; #30431 is the on-point one).

Suggested Triage Steps

• Reboot / drain the affected linux-mi35x-gpu-8-zf775 runner pool nodes to clear the zombie KFD contexts, then re-run 28888348134 — this is the immediate unblock per the script's own hint.
• Correlate the previous job on zf775-runner-9bdzn/-sgl9n/-494d5 before each failing run (via runner job history) to identify whether a specific mi35x workload is leaking VRAM on teardown.
• Chase reviewers on #30431 and clarify its DO NOT MERGE blocker, since it is scoped exactly at self-healing these driver-level VRAM leaks.
• If leaks recur post-reboot, consider a node-reboot/self-heal hook in ensure_vram_clear.sh when it detects allocated VRAM with zero KFD PIDs and zero device-file holders (the unreclaimable-zombie signature).

Status (factual, no priority assignment)

• Persistence: mi35x VRAM gate failing on every completed scheduled run in the window — Jul 3, 4, 5, and Jul 7 (Jul 6 run was cancelled). No passing mi35x run Jul 3–7.
• Scope: nightly-8-gpu-mi35x-grok1-int4 on the linux-mi35x-gpu-8-zf775 pool (≥3 distinct runner instances). Non-mi35x sister job nightly-8-gpu-grok1-int4 passes, so blast radius is the mi35x runner pool, not the grok1-int4 test.
• Blocked work: all mi35x nightly coverage on this pool is blocked at pre-flight; no test signal is produced.
• In-flight fix: ✅ #30431 (open 2026-07-07, DO NOT MERGE) — self-heal driver-level VRAM leaks.

nightly-8-gpu-mi35x-deepseek-r1-mxfp4-ar-fusion — failed step(s): Accuracy Test MI35x (8-GPU DeepSeek-R1-MXFP4 AllReduce Fusion)

Commit Info

• sglang: 11cea29c
• aiter: 9127c94a — v0.1.17.dev110+g9127c94a1 (source: dev image, preserved; [CI-AITER-CHECK] Dev/patched version detected → skipping rebuild)
• CI image: rocm/sgl-dev:v0.5.14-rocm700-mi35x-20260707

(Not a scout run — this is nightly-test-amd.yml, AITER_COMMIT_OVERRIDE empty. No Baseline A/B / Failure Origin section required.)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi35x/test_deepseek_r1_mxfp4_ar_fusion_eval_mi35x.py	test_deepseek_r1_mxfp4_ar_fusion_accuracy	AssertionError: Failed models: ['amd/DeepSeek-R1-MXFP4-Preview (ar-fusion)'] (line 242)	Log

Underlying crash (all 8 TP ranks): KeyError: 'sglang' raised during model load — Log.

Failure Summary

The server for amd/DeepSeek-R1-MXFP4-Preview crashed at startup. sglang logged DeepseekV3ForCausalLM has no SGLang implementation, falling back to Transformers implementation, then the trust-remote-code modeling_deepseek.py failed with KeyError: 'sglang' at ATTENTION_CLASSES[config._attn_implementation]. All TP schedulers hit sigquit, the server exited (code -9/-6), and the GSM8K accuracy test asserted failure.

Regression Status

New regression (this signature) — first observed today (Jul 7).

The native DeepseekV3ForCausalLM was NOT registered because sglang.srt.models.deepseek_v2 failed to import with No module named 'cutlass' (480 such errors in this job's log). This forced the Transformers-backend fallback, which sets _attn_implementation="sglang" (transformers.py:614); the remote model code has no "sglang" key in ATTENTION_CLASSES → KeyError.

Recent history of test_deepseek_r1_mxfp4_ar_fusion_eval_mi35x.py (test_deepseek_r1_mxfp4_ar_fusion_accuracy), job nightly-8-gpu-mi35x-deepseek-r1-mxfp4-ar-fusion (branch main, event=schedule, completed runs only):

Date	Run	Image	Test File Status	Failing Signature
Jul 7	28888348134	mi35x-20260707	❌ Failed	cutlass import → KeyError: 'sglang' (NEW)
Jul 6	28813271245	—	⏭️ Cancelled	excluded (not completed)
Jul 5	28749559519	mi35x-20260705	❌ Failed	different cause (0 cutlass, 0 KeyError; exit 1, cause not isolated)
Jul 4	28714602438	mi35x-20260704	✅ Passed	—

Note: the test file has been red since Jul 5, but via an unrelated failure. The cutlass/KeyError signature analyzed here is new to the Jul 7 image.

Failure Cluster

Missing cutlass package on mi35x-20260707 image → model-module import failure → Transformers fallback → KeyError: 'sglang'

Facts (verified observations)

• [CI-AITER-CHECK] markers: Dockerfile default AITER_COMMIT=9127c94a…; image carried dev build v0.1.17.dev110+g9127c94a1, preserved (no rebuild).
• job.log line 5453 / 5496: KeyError: 'sglang' at ATTENTION_CLASSES[config._attn_implementation] inside remote modeling_deepseek.py; same traceback on all 8 TP ranks.
• job.log line 5372+: DeepseekV3ForCausalLM has no SGLang implementation, falling back to Transformers implementation.
• sglang.srt.models.deepseek_v2 (which registers DeepseekV3ForCausalLM via EntryClass, deepseek_v2.py:2980) logged Ignore import error … deepseek_v2: No module named 'cutlass'.
• Image A/B (FACT): No module named 'cutlass' count = 480 on mi35x-20260707 (this job) vs 0 on mi35x-20260705 (Jul 5 kv-fp8 PASS) and 0 on mi35x-20260704 (Jul 4 ar-fusion PASS).
• Sibling A/B (FACT): nightly-8-gpu-mi35x-deepseek-r1-mxfp4-kv-fp8 passed on the 20260705 image (Jul 5) and failed on the 20260707 image (Jul 7) — same test, image is the only changed variable.
• Blast radius: ~19 of 22 mi35x jobs in run 28888348134 failed (deepseek, glm5*, kimi-k26, qwen35, grok1-int4, etc.), consistent with a broad import breakage.

Hypothesised Causes (with confidence)

• [HIGH] The mi35x-20260707 CI image is missing the cutlass (nvidia-cutlass-dsl) Python package that the 20260704/20260705 images had. sglang eagerly imports cutedsl kernels at module top level (e.g. jit_kernel/dsa/cutedsl_paged_mqa_logits.py, layers/quantization/nvfp4_gemm_swiglu_nvfp4_quant.py, jit_kernel/cutedsl_kda.py), and deepseek_v2's top-level import chain pulls one of these in; without cutlass the whole module fails to register.
  • Supporting: 480 vs 0 No module named 'cutlass' strictly tracks the image date; deepseek tests pass on AMD normally (so no unconditional cutlass need exists in the healthy state — it was satisfied by the package being present); kv-fp8 same-test A/B flips purely on image change; broad multi-model breakage.
  • Disconfirming: I did not pin the exact unguarded top-level import cutlass reachable from deepseek_v2 (the dsa/__init__.py cutedsl path is correctly is_hip()-guarded), so the precise import edge is unconfirmed — but the image-level cause is not in doubt.
  • How to verify: in the mi35x-20260707 container run python -c "import cutlass" and pip show nvidia-cutlass-dsl; compare pip freeze against the mi35x-20260705 image.
• [LOW] An sglang commit in 48ba79c1..11cea29c (67 commits) added a new unguarded top-level cutlass import into deepseek_v2's chain.
  • Supporting: several cutedsl commits landed in the window (e.g. #25220, #30117).
  • Disconfirming: strongly weakened by the image A/B — the same sglang code that fails on 20260707 would also fail on 20260705 if this were a source regression, but 20260705 had 0 cutlass errors. The differentiator is the image, not the source.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for cutlass+mi35x, No module named cutlass, mi35x image cutlass (via search/issues); the only hits (#28291, #15755, #22455) are unrelated. The image is built by the AMD Docker pipeline, so the fix likely lives in the image build, not an sglang source PR.

Suggested Triage Steps

• In the rocm/sgl-dev:v0.5.14-rocm700-mi35x-20260707 container, run python -c "import cutlass" and pip show nvidia-cutlass-dsl; diff pip freeze against …-20260705 to confirm the dropped package.
• If confirmed, restore the cutlass/nvidia-cutlass-dsl install in the mi35x image build recipe (or pin the image tag back to 20260705 for nightly until the image is fixed), then re-run the mi35x nightly matrix.
• Independently, consider guarding the top-level cutedsl imports reachable from deepseek_v2 behind is_hip()/try-except so a missing cutlass degrades gracefully instead of dropping native model registration (defense-in-depth; not the root cause).
• Separately triage the Jul 5 ar-fusion failure (job) — it is a distinct, still-unexplained failure on the 20260705 image and will likely persist after the cutlass image fix.

Status

• Persistence: This signature is new today (Jul 7, image 20260707). The test file itself has been red since Jul 5 via an unrelated cause; last green Jul 4.
• Scope: Broad — ~19/22 mi35x jobs in run 28888348134 fail on the same missing-cutlass import breakage (deepseek r1-mxfp4 ar-fusion/tp4/kv-fp8/hicache, deepseek-v32, glm5/glm51, kimi-k26, qwen35, grok1-int4, etc.).
• Blocked work: All mi35x nightly accuracy/perf signal on the 20260707 image.
• In-flight fix: ❌ none found — needs an image-build fix (restore cutlass wheel) or nightly image pin.

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-07-08 21:46 UTC)

[amd-bot]

nightly-test-amd.yml — Per-Job Analysis (continued 2/8)

nightly-accuracy-8-gpu-mi35x-deepseek-v32 — failed step(s): Ensure VRAM is clear

Commit Info

• sglang: 11cea29c
• aiter: N/A — the job failed at the pre-test host-side VRAM check, before any container/dependency install; no [CI-AITER-CHECK] markers exist in the log.

Failed Tests

Test File	Test Function	Error	Log
N/A (no test ran)	N/A	Pre-test step ensure_vram_clear.sh rocm failed: VRAM cleanup unsuccessful after 3 attempts → exit 1	Log

This is not a test failure. The job aborted during the runner-hygiene step (bash scripts/ci/amd/ensure_vram_clear.sh rocm), which runs before the Docker container starts and before any DeepSeek-V3.2 accuracy test executes.

Failure Summary

The AMD MI35x runner linux-mi35x-gpu-8-zf775-runner-f4wrn entered the job with GPUs 0–3 already holding 15–16% VRAM. The pre-test cleanup script tried 3 times (SIGTERM→SIGKILL→30 s wait) to clear it and failed each time; VRAM stayed pinned even though no host process, Docker container, or KFD PID held the devices. The job exited 1 before the accuracy suite ran.

Regression Status

Infrastructure issue (recurring, node-state — NOT a code regression).

Recent history of job nightly-accuracy-8-gpu-mi35x-deepseek-v32 (branch main, event schedule, completed runs only; current run 28888348134 is overall queued/in-flight but this job has completed=failure):

Date	Run	Job Status	Detail
Jul 7	28888348134	❌ Failed	VRAM cleanup failed (runner …-f4wrn)
Jul 5	28749559519	✅ Passed	—
Jul 4	28714602438	✅ Passed	—
Jul 3	28676359018	❌ Failed	Same VRAM cleanup failure (runner …-26vrd)
Jul 2	28611376093	✅ Passed	—

Failing in 2 of the last 5 completed runs, on two different physical runners (…-f4wrn today, …-26vrd on Jul 3), with identical symptoms — consistent with a driver/node-level VRAM leak rather than a per-commit code regression.

Failure Cluster

Pre-test VRAM cleanup failure (zombie KFD context on MI35x runner)

Facts (verified observations, no interpretation)

• Failing step: Run bash scripts/ci/amd/ensure_vram_clear.sh rocm; ##[error]Process completed with exit code 1. (log line 545).
• Initial state: GPU[0]:16 GPU[1]:15 GPU[2]:15 GPU[3]:16 VRAM%, GPUs 4–7 at 0% (lines 168–171).
• All 3 cleanup attempts diagnostics show: No KFD PIDs currently running, No docker containers found on host, no zombie processes, lsof not installed — nothing holds the GPUs yet VRAM persists.
• Final message: === FAILED: VRAM cleanup unsuccessful after 3 attempts === plus hint: "almost certainly a zombie KFD context ([Bug] Custom all-reduce IPC buffers use fixed VA, conflict with torch_memory_saver pause/resume → zombie processes on MI300X ROCm/aiter#2061). The node will need to be rebooted." (lines 537–543).
• AITER_COMMIT_OVERRIDE: empty (regular nightly, no scout override) (line 114).
• Jul 3 run 28676359018 reproduced the identical VRAM cleanup unsuccessful after 3 attempts on runner …-26vrd.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [FACT] The job was aborted by the runner-hygiene guard, not by any DeepSeek-V3.2 test. The VRAM check found 15–16% residual VRAM on GPUs 0–3 and failed the job by design.
  • Supporting: the error is emitted by ensure_vram_clear.sh; no test/pytest/server output appears anywhere in the log.
  • Disconfirming: (none — directly observable).
• [MEDIUM] The residual VRAM is a leaked/zombie KFD context left by a prior job on the same physical node, which the cleanup script cannot reclaim without a reboot.
  • Supporting: no host process, container, or KFD PID holds the devices across all 3 attempts, yet VRAM stays pinned; the script's own hint cites [Bug] Custom all-reduce IPC buffers use fixed VA, conflict with torch_memory_saver pause/resume → zombie processes on MI300X ROCm/aiter#2061; the same symptom recurred on a second node (Jul 3), which argues against a single-host hardware fault.
  • Disconfirming: the specific prior job that leaked the VRAM was not identified from this log alone (would need the previous job on …-f4wrn); GPUs 4–7 were clean, so the leak is partial.
  • How to verify: check the job that ran on runner …-f4wrn immediately before this one for an ungraceful exit/OOM/crash; confirm a node reboot clears the residual VRAM.

In-flight Fix Check

• In-flight fix: ✅ #30431 — "[DO NOT MERGE][AMD] ci: self-heal driver-level VRAM leaks + graceful GPU cleanup on shutdown" (open, created 2026-07-07). Directly targets this exact failure class (driver-level VRAM leaks + graceful GPU cleanup). Currently marked DO NOT MERGE, so it needs iteration before it lands — track it rather than opening a duplicate.
  • Search: repo:sgl-project/sglang is:pr is:open ensure_vram_clear OR zombie KFD → 6 results, #30431 is the on-topic one.

Suggested Triage Steps

• Reboot / drain runner linux-mi35x-gpu-8-zf775-runner-f4wrn to clear the zombie KFD context, then re-run the job — per the script's own hint.
• Inspect the previous job that executed on …-f4wrn before 85694309069 to identify what leaked VRAM on GPUs 0–3 (ungraceful shutdown / OOM candidate).
• Chase #30431 toward mergeable state — it aims to self-heal exactly this leak; confirm whether its cleanup path reclaims a KFD-context leak without requiring a reboot.
• If this keeps recurring across nodes, consider tracking VRAM-leak incidents against [Bug] Custom all-reduce IPC buffers use fixed VA, conflict with torch_memory_saver pause/resume → zombie processes on MI300X ROCm/aiter#2061.

Status (factual, no priority assignment)

• Persistence: Intermittent infra failure — 2 of last 5 completed scheduled runs (Jul 3 and Jul 7), on two different runners.
• Scope: This job (nightly-accuracy-8-gpu-mi35x-deepseek-v32) in this run; a host-level VRAM leak on an MI35x node — any 8-GPU MI35x job landing on the affected node is at risk. No sglang code path implicated.
• Blocked work: DeepSeek-V3.2 nightly accuracy signal on AMD MI35x for this run (no accuracy data produced).
• In-flight fix: ✅ #30431 (open since 2026-07-07, DO NOT MERGE) — CI self-heal for driver-level VRAM leaks.

nightly-8-gpu-mi35x-deepseek-r1-mxfp4-tp4 — failed step(s): Accuracy Test MI35x (DeepSeek-R1-MXFP4 TP4 MTP), Accuracy Test MI35x (DeepSeek-R1-MXFP4 TP4)

Commit Info

• sglang: 11cea29c
• aiter: 9127c94a — v0.1.17.dev110+g9127c94a1 (source: dev image detected in CI, rebuild skipped)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi35x/test_deepseek_r1_mxfp4_tp4_mi35x.py	TestDeepSeekR1MXFP4TP4MI35x.setUpClass	KeyError: 'sglang' (server exits -9 during model load)	Log
test/registered/amd/accuracy/mi35x/test_deepseek_r1_mxfp4_tp4_mtp_mi35x.py	TestDeepSeekR1MXFP4TP4MTPMI35x.setUpClass	KeyError: 'sglang' (server exits -9 during model load)	Log

Failure Summary

Both DeepSeek-R1-MXFP4 accuracy tests failed in setUpClass: the server crashed (exit -9) during model init. The scheduler raised KeyError: 'sglang' from the model's remote modeling_deepseek.py at ATTENTION_CLASSES[config._attn_implementation]. This happens because sglang loaded the model through the generic Transformers backend fallback (log: Using Transformers backend.) instead of its native DeepseekV3ForCausalLM.

Regression Status

New regression (fresh failure mode, appeared 2026-07-07).

Recent history of test_deepseek_r1_mxfp4_tp4_mi35x.py in job nightly-8-gpu-mi35x-deepseek-r1-mxfp4-tp4 (branch main, event=schedule, completed runs only):

Date	Run	Test File Status	Failed Function	Error
Jul 07	28888348134	❌ Failed	setUpClass	KeyError: 'sglang' (model won't load)
Jul 06	28813271245	—	—	run cancelled (excluded)
Jul 05	28749559519	✅ Passed	—	accuracy=0.942, loaded natively
Jul 04	28714602438	✅ Passed	—	accuracy=0.948, loaded natively
Jul 03	28676359018	✅ Passed	—	—

The MTP sister file (..._mtp_mi35x.py) was failing Jul 4/5 on a different symptom (accuracy 0.939/0.942 < 0.944 threshold); on Jul 7 its failure mode also switched to the same KeyError: 'sglang' load crash. First observed for the KeyError cluster: 2026-07-07. Last known good (native load): Jul 05.

Failure Cluster

DeepseekV3 falls back to Transformers backend → KeyError: 'sglang' (native deepseek_v2 module fails to import).

Facts (verified observations)

• Jul 7 log: Using Transformers backend. immediately precedes each crash (lines 4622, 4974, 5350, 5674); Jul 4 log has no such message and instead shows Enable Aiter AllReduce Fusion for DeepseekV3ForCausalLM (native load).
• Stack trace ends at remote modeling_deepseek.py:1148 → ATTENTION_CLASSES[config._attn_implementation] with _attn_implementation="sglang" (injected at python/sglang/srt/models/transformers.py:614). KeyError: 'sglang' at log lines 4667, 4710, 5017, 5391, 5719.
• Jul 7 log contains 480 occurrences of Ignore import error when loading sglang.srt.models.<X>: No module named 'cutlass', including deepseek_v2, deepseek_nextn, deepseek_ocr (lines 4482–4491). Jul 4 log: 0 such errors.
• Import chain (verified in source): deepseek_v2.py:61 from ...dsa.dsa_indexer import Indexer → dsa_indexer.py:11-17 from sglang.jit_kernel.dsa import (...) → jit_kernel/dsa/__init__.py:1 → cutedsl_paged_mqa_logits.py:15 import cutlass (unguarded, top-level). The is_sm100_supported() guard is at line 437, inside a function — too late to protect the module import.
• cutlass (NVIDIA CUTLASS Python DSL) is not installed in the ROCm CI image, so the top-level import raises ModuleNotFoundError, which the model registry swallows → DeepseekV3ForCausalLM never registers → Transformers-backend fallback.

Hypothesised Causes (with confidence)

• [HIGH] Commit fefc1743 ("Cute-DSL FP8 MQA logits", #25220) introduced an unguarded top-level import cutlass into deepseek_v2's import chain, breaking native DeepseekV3 loading on ROCm.
  • Supporting: (a) the commit added python/sglang/jit_kernel/dsa/cutedsl_paged_mqa_logits.py (with import cutlass at line 15) and rewired dsa_indexer.py to import sglang.jit_kernel.dsa at top level — both verified in the diff; (b) merged 2026-07-06 23:34 PDT (= 2026-07-07 06:34 UTC), inside the regression window 48ba79c1..11cea29c, after the last green run (Jul 05) and before this run (started 18:12 UTC Jul 07); (c) Jul 7 log shows deepseek_v2 among 480 No module named 'cutlass' import failures, absent Jul 4.
  • Disconfirming: (none found) — Jul 4 baseline loaded natively with zero cutlass errors on the same test, and the code-level chain is direct and unconditional.
  • How to verify: on an MI35x ROCm container, python -c "import sglang.srt.models.deepseek_v2" on 11cea29c → expect ModuleNotFoundError: No module named 'cutlass'; then guard the import cutlass behind is_sm100_supported()/lazy-import and re-run test_deepseek_r1_mxfp4_tp4_mi35x.py.

In-flight Fix Check

• In-flight fix: ❌ none found for the ROCm import break. #30413 ("Fix additional kernels being triggered in CuteDSL MQA logits path", open since 2026-07-07) is a follow-up to #25220 but touches paged_mqa_logits.py / dsa_indexer.py / dsa_backend.py — not cutedsl_paged_mqa_logits.py, so it does not fix the unguarded import cutlass. (Searched open PRs: cutlass import dsa, No module named cutlass, cutedsl paged mqa.)

Suggested Triage Steps

• Make the cutlass import in python/sglang/jit_kernel/dsa/cutedsl_paged_mqa_logits.py:15 lazy or guard it behind is_sm100_supported() so importing sglang.srt.models.deepseek_v2 does not require the NVIDIA CUTLASS DSL on ROCm.
• Verify by importing sglang.srt.models.deepseek_v2 in the MI35x ROCm image; confirm DeepseekV3ForCausalLM registers (no "Using Transformers backend" in server startup).
• Consider adding a CI import-smoke check that fails loudly when a core model module (e.g. deepseek_v2) is silently demoted to the Transformers backend on AMD.
• Note broad blast radius: the same top-level cutlass import likely breaks all DeepseekV3-family models (and other models sharing the DSA import chain) on any non-CUTLASS environment, not just this test.

Status

• Persistence: KeyError: 'sglang' load crash first seen 2026-07-07 (1 completed run); test loaded natively and passed through Jul 05.
• Scope: 2 test files in this job (test_deepseek_r1_mxfp4_tp4_mi35x.py, ..._mtp_mi35x.py); likely affects all DeepseekV3-family / DSA-importing models on ROCm (480 No module named 'cutlass' import failures observed this run).
• Blocked work: DeepSeek-R1-MXFP4 TP4 + MTP accuracy signal on MI35x nightly.
• In-flight fix: ❌ none found — needs a new PR to guard the import cutlass in cutedsl_paged_mqa_logits.py.

nightly-perf-8-gpu-mi35x-deepseek-v32-mtp — failed step(s): Ensure VRAM is clear

Commit Info

• sglang: 11cea29c (verified from log: HEAD is now at 11cea29)
• aiter: N/A — the job failed before any dependency install; no [CI-AITER-CHECK] markers exist (Setup docker / Install dependencies steps were skipped).

Failed Tests

Test File	Test Function	Error	Log
N/A (infrastructure)	N/A	ensure_vram_clear.sh: VRAM cleanup unsuccessful after 3 attempts → exit code 1	Log

No test ran. The job died in pre-flight step 3 (Ensure VRAM is clear). Steps 4–6 (Setup docker, Install dependencies, Performance Test MI35x (8-GPU DeepSeek-V3.2 MTP)) were all skipped.

Failure Summary

The nightly-perf-8-gpu-mi35x-deepseek-v32-mtp job failed in the pre-flight scripts/ci/amd/ensure_vram_clear.sh rocm guard step. At job start, GPUs 0–3 already showed 16% VRAM allocated (before this job did anything), and the cleanup routine failed to reclaim it after 3 attempts. This is a runner-node infrastructure problem, not a code or test failure.

Regression Status

Infrastructure issue (intermittent, node-state dependent). Not a code regression.

Recent history of the Ensure VRAM is clear pre-flight step on the mi35x 8-GPU DeepSeek-V3.2-MTP jobs (branch main, event=schedule, completed runs only):

Date	Run	Job	Step 3 (VRAM clear)	Note
Jul 7	28888348134	perf-…-mtp	❌ Failed	this run; GPUs 0–3 dirty at start
Jul 5	28749559519	perf-…-mtp	❌ Failed	sister accuracy-…-mtp ✅ passed same run
Jul 4	28714602438	perf-…-mtp	✅ Passed	sister accuracy-…-mtp ❌ failed same run
Jul 3	28676359018	perf-…-mtp	✅ Passed	both mtp jobs passed

The failure alternates between the perf and accuracy mtp jobs across runs and does not track the sglang commit — a hallmark of leftover node state, not a code change.

Failure Cluster

Zombie KFD context / VRAM leak on AMD mi35x 8-GPU runner (pre-flight ensure_vram_clear.sh fails).

Facts (verified observations)

• Step 3 Ensure VRAM is clear = failure; steps 4, 5, 6 = skipped (jobs API).
• Initial GPU state at job start: GPUs 0–3 at VRAM%: 16, GPUs 4–7 at 0 (log line 167).
• Cleanup ran 3 attempts (SIGTERM→SIGKILL on GPU-device-file holders, 30 s wait each); VRAM remained at 16% on GPUs 0–3 through all 3 attempts (lines 262, 371, 480).
• Diagnostics on final attempt: No KFD PIDs currently running, No docker containers, lsof not installed — i.e. no live process or container holds the memory, yet VRAM stays allocated (log tail).
• Script's own hint: almost certainly a zombie KFD context (see ROCm/aiter#2061). The node will need to be rebooted (line 542).
• No pytest, docker run, or dependency-install output anywhere in the log — nothing from this run's code executed on GPU.

Hypothesised Causes

• [HIGH] Leftover zombie KFD/HSA context on runner linux-mi35x-gpu-8-zf775-runner-j68fr from a prior job on the same node, holding VRAM on GPUs 0–3.
  • Supporting: GPUs already dirty at job start before any work; no process/container/KFD-PID holds the memory; cleanup script cannot reclaim it across 3 attempts; the guard itself points at [Bug] Custom all-reduce IPC buffers use fixed VA, conflict with torch_memory_saver pause/resume → zombie processes on MI300X ROCm/aiter#2061 and prescribes a node reboot.
  • Disconfirming: (none found) — every fact points to pre-existing node state, none to this run's code.
  • How to verify: reboot the runner node and re-run; confirm GPUs report 0% VRAM at the start of the next job.
• [SPECULATION] sglang commit 11cea29c involvement — ruled out: no code from this commit executed (test/docker/deps steps skipped), and the same step passed on this job Jul 3–4 across different commits.

In-flight Fix Check

• In-flight fix: ✅ #30431 [DO NOT MERGE][AMD] ci: self-heal driver-level VRAM leaks + graceful GPU cleanup on shutdown (open since 2026-07-07) — directly targets driver-level VRAM leaks + GPU cleanup on the AMD runners. Currently marked DO-NOT-MERGE (WIP); chase it to completion rather than opening a duplicate.
• Search: is:pr is:open VRAM clear (16 hits, #30431 the relevant one); ensure_vram_clear (0 hits).

Suggested Triage Steps

• Reboot / recycle runner node linux-mi35x-gpu-8-zf775-runner-j68fr to flush the zombie KFD context, then re-run the job (per the script's own hint and [Bug] Custom all-reduce IPC buffers use fixed VA, conflict with torch_memory_saver pause/resume → zombie processes on MI300X ROCm/aiter#2061).
• Track in-flight PR #30431 to landing — it aims to self-heal exactly this VRAM-leak pattern so a node reboot isn't needed.
• Investigate which prior job on this node leaked the context (inspect the previous job scheduled on zf775-runner-j68fr before 19:30 UTC Jul 7) — the leak originates upstream, not in this MTP perf job.
• No sglang bisect warranted: the failing step runs before any repo code executes, and the failure alternates between jobs independent of commit.

Status

• Persistence: Intermittent on the mi35x 8-GPU node — this perf-…-mtp job failed the VRAM-clear guard on Jul 5 and Jul 7; the sister accuracy-…-mtp job failed it on Jul 4. Not every run (both passed Jul 3).
• Scope: Pre-flight infra guard on runner linux-mi35x-gpu-8-zf775-runner-j68fr; blocks all downstream perf/accuracy testing for the mi35x DeepSeek-V3.2-MTP jobs when it triggers. No test signal produced.
• In-flight fix: ✅ #30431 (open, WIP/DO-NOT-MERGE) — self-heal driver-level VRAM leaks on AMD runners.

nightly-8-gpu-mi35x-deepseek-r1-mxfp4-kv-fp8 — failed step(s): Ensure VRAM is clear

Commit Info

• sglang: 11cea29c
• aiter: N/A — the job failed at the ensure_vram_clear.sh pre-flight step, before the container/dependency install runs, so no [CI-AITER-CHECK] markers were emitted.

Failed Tests

Test File	Test Function	Error	Log
N/A (infrastructure — no test ran)	N/A	=== FAILED: VRAM cleanup unsuccessful after 3 attempts === → exit code 1	Log

This is not a test failure. The job aborted in the setup step bash scripts/ci/amd/ensure_vram_clear.sh rocm, which runs immediately after checkout and before any model/test execution.

Failure Summary

The runner linux-mi35x-gpu-8-zf775-runner-9mv88 started this job with all 8 GPUs already 75–91% VRAM-occupied by orphaned processes. The pre-flight VRAM-clear guard ran 3 cleanup attempts (SIGTERM → SIGKILL → 30s wait each), all failed to release VRAM, and the job exited 1. No sglang test code was reached.

Regression Status

Known recurring failure — infrastructure (dirty runner), not a code regression.

Recent history of the VRAM-clear pre-flight step in job nightly-8-gpu-mi35x-deepseek-r1-mxfp4-kv-fp8 (same branch main, event=schedule, completed runs only):

Date	Run	Job Status	Failure Step	Error
Jul 7	28888348134	❌ Failed	ensure_vram_clear.sh	VRAM cleanup unsuccessful after 3 attempts
Jul 5	28749559519	✅ Passed	—	—
Jul 4	28714602438	❌ Failed	ensure_vram_clear.sh	VRAM cleanup unsuccessful after 3 attempts (verified in log)
Jul 3	28676359018	❌ Failed	ensure_vram_clear.sh	VRAM cleanup unsuccessful after 3 attempts (verified in log)

The identical VRAM-cleanup failure recurred on Jul 3, Jul 4, and Jul 7, with a clean run on Jul 5. Across the mi35x-mxfp4 matrix, which job fails rotates run-to-run (a different subset fails each night), which is the signature of shared-runner VRAM contention rather than a specific test being broken.

Failure Cluster

Runner VRAM not clear at job start — zombie KFD context on mi35x 8-GPU runner (pre-flight ensure_vram_clear.sh abort).

Facts (verified observations, no interpretation)

• Failing step is bash scripts/ci/amd/ensure_vram_clear.sh rocm (log line 110); no test/container step ran afterward.
• At job start GPUs read VRAM% = 80/89/91/91/75/75/75/75 across GPU[0–7] (log lines 168–175), all above the 5% threshold.
• rocm-smi --showpids shows persistent KFD processes named UNKNOWN holding ~231–232 GB each on GPU 1 (PIDs 875199, 875200, 875201, 875203, 875214, 875215, 875234, 875252) — VRAM figures unchanged across all 3 cleanup attempts (log lines 285–311, 425–450, 564–589).
• fuser/lsof/zombie-process/docker checks all found no host process or container holding the GPUs (log lines 272–278, 411–417, 550–556) — the VRAM is held by contexts not owned by any visible host process.
• The script's own hint (log lines 634–637): "almost certainly a zombie KFD context (see [Bug] Custom all-reduce IPC buffers use fixed VA, conflict with torch_memory_saver pause/resume → zombie processes on MI300X ROCm/aiter#2061). The node will need to be rebooted."
• AITER_COMMIT_OVERRIDE is empty (log line 34/114) — this is a plain nightly run, not a scout override.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [HIGH] Dirty runner: a prior job on linux-mi35x-gpu-8-zf775-runner-9mv88 leaked GPU memory / left zombie KFD contexts that survive process kill, so this job's pre-flight guard correctly refused to start.
  • Supporting: no host process/container holds the GPUs yet ~232 GB/GPU remains allocated and is unchanged across 3 SIGKILL+wait cycles; script hint explicitly names zombie KFD context (ROCm/aiter#2061); the same step failed Jul 3/4 and passed Jul 5, and the failing job rotates across the matrix — all consistent with node-state contamination, not code.
  • Disconfirming: (none found) — no evidence points to the sglang commit; the failure occurs before any sglang code executes.
  • How to verify: reboot the runner node, then re-run the job; confirm VRAM reads ~0% at start.
• [LOW] The sglang commit 11cea29c is implicated.
  • Supporting: it is the checked-out HEAD.
  • Disconfirming: the job aborts before any sglang code runs; the same step fails on unrelated commits on other nights. Effectively ruled out.

In-flight Fix Check

• In-flight fix: ✅ #30431 "[DO NOT MERGE][AMD] ci: self-heal driver-level VRAM leaks + graceful GPU cleanup on shutdown" (opened 2026-07-07 21:59 UTC, open, non-draft but marked DO-NOT-MERGE — still WIP). Directly targets this class of driver-level VRAM leak. Track this PR; do not open a duplicate.
• Search used: repo:sgl-project/sglang is:pr is:open VRAM cleanup.

Suggested Triage Steps

• Reboot / recycle the runner node linux-mi35x-gpu-8-zf775-runner-9mv88 to clear the zombie KFD contexts, then re-run job 85694309200 — the script hint states a reboot is required.
• Identify which prior job on this runner leaked the ~232 GB/GPU allocations (check the runner's job history immediately preceding 18:48 UTC on Jul 7) to find the leaking workload.
• Follow / review #30431 as the durable self-heal fix for driver-level VRAM leaks; nudge it past DO-NOT-MERGE status.
• Consider tracking this as an infra/runner-health item (per [Bug] Custom all-reduce IPC buffers use fixed VA, conflict with torch_memory_saver pause/resume → zombie processes on MI300X ROCm/aiter#2061) rather than an sglang code regression.

Status (factual, no priority assignment)

• Persistence: VRAM-clear guard failed for this job on Jul 3, Jul 4, and Jul 7 (passed Jul 5) — intermittent-recurring over the last 5 scheduled runs.
• Scope: Infrastructure — affects any mi35x 8-GPU nightly job scheduled onto the contaminated runner; the specific failing job in the mxfp4 matrix rotates each night. No sglang test signal was produced by this run.
• Blocked work: DeepSeek-R1 MXFP4 KV-FP8 nightly accuracy/perf signal on mi35x is not collected while the runner stays dirty.
• In-flight fix: ✅ #30431 (open since 2026-07-07, WIP/DO-NOT-MERGE) — self-heal driver-level VRAM leaks.

nightly-8-gpu-mi35x-kimi-k26 — failed step(s): Ensure VRAM is clear

Commit Info

• sglang: 11cea29c
• aiter: N/A (docker setup was skipped — the job died in the pre-flight Ensure VRAM is clear step before any dependency install; no [CI-AITER-CHECK] markers in log)

Failed Tests

Test File	Test Function	Error	Log
N/A — infrastructure failure, no test ran	N/A	VRAM cleanup unsuccessful after 3 attempts → Process completed with exit code 1	Log L537

This is not a test failure. The Accuracy Test MI35x (8-GPU Kimi-K2.6) step (step 6) was skipped — the job never reached it. It failed in step 3, the scripts/ci/amd/ensure_vram_clear.sh rocm pre-flight guard.

Failure Summary

The job ran on AMD runner linux-mi35x-gpu-8-zf775-runner-dqqwv. Before any container/test setup, the VRAM-clear pre-flight guard found GPUs 0–3 each holding 16% allocated VRAM (L167-171). After 3 cleanup attempts (kill SGLang procs, SIGTERM/SIGKILL device-file holders, 30s waits), VRAM never dropped below the 5% threshold, so the guard exited 1 and all downstream steps (docker setup, install, accuracy test) were skipped.

Regression Status

Recurring infrastructure failure (not a code regression).

Recent history of job nightly-8-gpu-mi35x-kimi-k26 (workflow nightly-test-amd.yml, event=schedule, branch=main, completed runs only):

Date	Run	Job	Test File Status	Failed Function	Error
Jul 07	28888348134	nightly-8-gpu-mi35x-kimi-k26	❌ Infra (step 3)	— (test skipped)	VRAM clear failed, GPU0-3 @16%
Jul 05	28749559519	nightly-8-gpu-mi35x-kimi-k26	❌ Infra (step 3)	— (test skipped)	VRAM clear failed (same guard, runner ...zf775-runner-2jrjh)
Jul 04	28714602438	nightly-8-gpu-mi35x-kimi-k26	✅ Passed	—	—
Jul 03	28676359018	nightly-8-gpu-mi35x-kimi-k26	✅ Passed	—	—
Jul 02	28611376093	nightly-8-gpu-mi35x-kimi-k26	✅ Passed	—	—

The Kimi-K2.6 accuracy test itself last passed Jul 04; it has not actually executed since. Both Jul 05 and Jul 07 failures are the identical VRAM-clear pre-flight guard on the same zf775 MI35x-8GPU runner pool — the test logic was never exercised in either.

Failure Cluster

Pre-flight VRAM-clear guard failure on MI35x 8-GPU runner (stuck/leaked VRAM, suspected zombie KFD context)

Facts (verified observations, no interpretation)

• Step 3 Ensure VRAM is clear failed; steps 4–6 (docker setup, install deps, accuracy test) were skipped (verified from job steps API).
• GPUs 0–3 each reported GPU Memory Allocated (VRAM%): 16 at initial check and after all 3 cleanup attempts; GPUs 4–7 were ~0–1% (L438-441).
• Every diagnostic showed no live holders: fuser/lsof empty, No KFD PIDs currently running, No docker containers, no zombie processes (L481-496). VRAM is allocated but no process owns it.
• The script's own hint: "almost certainly a zombie KFD context (see [Bug] Custom all-reduce IPC buffers use fixed VA, conflict with torch_memory_saver pause/resume → zombie processes on MI300X ROCm/aiter#2061). The node will need to be rebooted" (L540-543).
• AITER_COMMIT_OVERRIDE empty; this is a regular nightly run, not an AITER Scout run — no aiter baseline A/B applies.
• Same failure signature on the prior completed run (Jul 05, job 85246183424), same runner pool linux-mi35x-gpu-8-zf775-runner.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [HIGH] Environmental: a leaked/zombie KFD GPU context from a prior job on the zf775 MI35x-8GPU runner left ~16% VRAM allocated on GPUs 0–3; the node requires a reboot the cleanup script cannot perform.
  • Supporting: VRAM allocated but zero live holders (fuser/lsof/KFD/docker all empty); persists unchanged across 3 kill+wait cycles; script explicitly diagnoses zombie KFD context; same signature recurred Jul 05 on the same runner pool.
  • Disconfirming: (none found) — no evidence pointing at sglang code; the job never ran any sglang code past checkout, and the test passed Jul 02–04 on the same workflow/model with no relevant intervening infra change visible in the log.
  • How to verify: reboot/recycle runner linux-mi35x-gpu-8-zf775-runner-*, then re-run the job; confirm rocm-smi --showmemuse reports GPUs 0–3 at 0% on a fresh boot.
• [LOW] A prior Kimi-K2.6 (or other 8-GPU) job on this runner crashed mid-run and orphaned VRAM without a reapable process.
  • Supporting: GPUs 0–3 (the low-index set an 8-GPU / TP job would grab first) are the ones stuck; matches a partial teardown.
  • Disconfirming: cannot identify the prior job from this log alone; the previous job on the same runner isn't recorded here.
  • How to verify: inspect the runner's prior job assignment and its exit state.

In-flight Fix Check

• In-flight fix: ✅ #30431 "[DO NOT MERGE][AMD] ci: self-heal driver-level VRAM leaks + graceful GPU cleanup on shutdown" (open, non-draft, created 2026-07-07 — same day as this failure) directly targets driver-level VRAM leaks + GPU cleanup. Marked DO-NOT-MERGE (WIP). Track it; do not open a duplicate. (Search: search/issues?q=repo:sgl-project/sglang+is:pr+is:open+zombie+KFD.)

Suggested Triage Steps

• Recycle/reboot the affected runner: linux-mi35x-gpu-8-zf775-runner-dqqwv (and check sibling ...-2jrjh from Jul 05) — per the script hint, a zombie KFD context needs a host reboot; the in-job cleanup cannot recover it.
• After reboot, re-run job nightly-8-gpu-mi35x-kimi-k26 on a clean node to confirm the Kimi-K2.6 accuracy test itself is healthy (it passed Jul 04, so no code regression is implicated).
• Follow #30431 toward mergeable state — it aims to self-heal exactly this leak class; ping the author on the DO-NOT-MERGE status.
• If the zf775 pool keeps leaking, consider a scheduled node reboot / KFD-context reaper between nightly runs as a stop-gap.

Status (factual, no priority assignment)

• Persistence: 2 of the last 3 completed scheduled runs failed at this pre-flight guard (Jul 05 and Jul 07); the guard last passed Jul 04. Failures are intermittent and tied to runner state, not a continuous break.
• Scope: This is a runner-level condition. In today's run it's visible on this job; the zf775 MI35x-8GPU pool is shared, so other 8-GPU MI35x jobs landing on the same leaked node are at risk. The Kimi-K2.6 accuracy test has been effectively un-run since Jul 04.
• Blocked work: Nightly Kimi-K2.6 (8-GPU MI35x) accuracy signal — no result produced Jul 05 or Jul 07.
• In-flight fix: ✅ #30431 (open, WIP/DO-NOT-MERGE) — self-heal driver-level VRAM leaks.

nightly-8-gpu-mi35x-deepseek-r1-hicache — failed step(s): Ensure VRAM is clear

Commit Info

• sglang: 11cea29c
• aiter: N/A — the job failed at the host-side VRAM pre-flight before any container started, so no [CI-AITER-CHECK] markers were emitted. (AITER_COMMIT_OVERRIDE empty → not a scout run.)

Failed Tests

Test File	Test Function	Error	Log
N/A (infra pre-flight)	scripts/ci/amd/ensure_vram_clear.sh rocm	FAILED: VRAM cleanup unsuccessful after 3 attempts → exit 1	Log

No DeepSeek-R1 HiCache test ran. The job never reached the container/test stage — it aborted at the runner's GPU-cleanliness gate.

Failure Summary

The job failed at the host pre-flight step ensure_vram_clear.sh on runner linux-mi35x-gpu-8-zf775-runner-7wln5. GPUs 0–3 each reported 16% VRAM still allocated (threshold is 5%) at job start. Three cleanup attempts (SIGTERM→SIGKILL + 30 s wait each) all failed to release the memory, so the step exited 1 and the job aborted before any test executed. This is an infrastructure/runner-state failure, not a code failure.

Regression Status

Infrastructure issue (intermittent) — not a code regression.

Recent history of job nightly-8-gpu-mi35x-deepseek-r1-hicache (branch main, event=schedule, completed runs only):

Date	Run	Job	Failure Point	Detail
Jul 7	28888348134	job	❌ VRAM pre-flight	GPUs 0–3 stuck @16%, exit 1 (this run)
Jul 5	28749559519	job	⚠️ tests ran, failed	pre-flight passed; test_deepseek_r1_hicache_mi35x.py failures, exit 255
Jul 4	28714602438	job	⚠️ tests ran, failed	pre-flight passed; test failures, exit 255
Jul 3	28676359018	job	❌ VRAM pre-flight	GPUs 0–3 stuck @16%, exit 1 (same signature as this run)

The identical VRAM pre-flight failure occurred Jul 3 and Jul 7 — the same 16% residual on GPUs 0–3 on the same runner class, indicating a recurring stuck-VRAM condition on the MI35X node. On Jul 4/5 the gate passed and the job progressed to the HiCache tests (which then failed for a separate, test-level reason not in scope here). No pre-flight-passing green run for this job in the queryable window (last 8 scheduled runs; earlier ones were cancelled).

Failure Cluster

Host VRAM not clear at job start (residual GPU allocation / suspected zombie KFD context on MI35X runner)

Facts (verified observations)

• ERROR: VRAM usage exceeds threshold (5%) on some GPUs: GPU[0..3] = 16% at job start (log lines 167–171); GPUs 4–7 at 0–1%.
• Three cleanup attempts each ran Killing SGLang processes → SIGTERM → SIGKILL → 30 s wait → re-check; all three re-reported GPUs 0–3 @16% (lines 219–223, 328–332, 437–441).
• Diagnostics on every attempt show no holder: rocm-smi --showpids → "No KFD PIDs currently running"; docker ps → no containers; zombie-process scan empty; fuser/lsof found nothing (lines 264–318, 372–427, 481–536).
• Final: === FAILED: VRAM cleanup unsuccessful after 3 attempts === then ##[error]Process completed with exit code 1. (lines 537, 545).
• The script's own hint (lines 540–543): "if no host process / container holds the GPU but VRAM is still allocated, this is almost certainly a zombie KFD context (see ROCm/aiter#2061). The node will need to be rebooted."
• Container/tests never started → no aiter version, no test output produced.

Hypothesised Causes (with confidence)

• [MEDIUM] Zombie KFD context / driver-level VRAM leak on the MI35X runner linux-mi35x-gpu-8-zf775-runner-7wln5 holding ~16% on GPUs 0–3.
  • Supporting: VRAM allocated but every diagnostic (KFD PIDs, docker, fuser, zombies) shows no owning process — the classic leaked-context signature; the cleanup script's own hint names this and links ROCm/aiter#2061; identical residual (exactly 16% on GPUs 0–3) recurred on Jul 3 and Jul 7, consistent with a persistent node state a user-space kill cannot clear.
  • Disconfirming: no direct KFD context dump in the log proves a leaked context specifically (the check only proves "VRAM allocated, no host holder"); a prior job on the same node leaving allocations is also possible.
  • How to verify: reboot/drain the node and re-run the job; inspect dmesg/kfd state on the runner; confirm the same node hosted the Jul 3 failure.
• [LOW] Prior CI job on the same runner left the GPUs dirty (not cleaned on its own teardown).
  • Supporting: only GPUs 0–3 affected (a subset), consistent with a previous partial-node workload; runners are reused across jobs.
  • Disconfirming: user-space SIGKILL + 30 s waits did not reclaim it, which points to kernel/driver-level state rather than a lingering user process.
  • How to verify: correlate the previous job scheduled on zf775-runner-7wln5 and its exit path.

In-flight Fix Check

• In-flight fix: ✅ #30431 "[DO NOT MERGE][AMD] ci: self-heal driver-level VRAM leaks + graceful GPU cleanup on shutdown" — open (draft-quality, marked DO NOT MERGE), created 2026-07-07. Directly targets this exact driver-level VRAM-leak/cleanup failure. Track it; do not open a duplicate.
• Search run: search/issues?q=repo:sgl-project/sglang+is:pr+is:open+VRAM+cleanup+KFD → 2 hits (#30431 relevant; #28534 is unrelated HiCache functionality).

Suggested Triage Steps

• Reboot or drain runner linux-mi35x-gpu-8-zf775-runner-7wln5 to clear the leaked GPU 0–3 allocation, then re-run this job to confirm the gate passes.
• Confirm whether the Jul 3 pre-flight failure (run 28676359018) occurred on the same physical node — if so, the node is chronically leaking and should be pulled from the pool until fixed.
• Track #30431 (self-heal VRAM leaks) toward a mergeable state — it is the standing fix for this cluster.
• Separately, the underlying HiCache test (test_deepseek_r1_hicache_mi35x.py) was failing on Jul 4/5 when the gate passed — that is a distinct test-level issue worth its own investigation once the runner is healthy.

Status

• Persistence: VRAM pre-flight failure seen on 2 of the last 4 completed scheduled runs (Jul 3 and Jul 7); identical 16%-on-GPU-0–3 signature on the MI35X runner. The job has not had a fully-green (pre-flight + tests) run in the queryable window.
• Scope: This job only in this run; the pre-flight gate is shared infra so any MI35X job landing on the affected node would hit the same wall.
• Blocked work: DeepSeek-R1 HiCache nightly signal on MI35X — tests do not execute when the gate fails.
• In-flight fix: ✅ #30431 (open, DO NOT MERGE) — self-heal driver-level VRAM leaks.

nightly-8-gpu-mi35x-glm51 — failed step(s): Ensure VRAM is clear

Commit Info

• sglang: 11cea29c
• aiter: N/A — the job failed at the pre-test VRAM check (step 3), before docker setup / dependency install, so no [CI-AITER-CHECK] markers were ever emitted.

Failed Tests

Test File	Test Function	Error	Log
N/A (no test ran)	N/A	Infra pre-check failure: FAILED: VRAM cleanup unsuccessful after 3 attempts	Log

Not a test failure. Step 3 "Ensure VRAM is clear" (scripts/ci/amd/ensure_vram_clear.sh rocm) failed, so steps 4-7 (Setup docker, Install dependencies, Accuracy Test MI35x 8-GPU GLM-5.1 DSA, Performance Test MI35x 8-GPU GLM-5.1) were all skipped. The GLM-5.1 tests never executed.

Failure Summary

The runner linux-mi35x-gpu-8-zf775-runner-9rjn5 entered the job with GPUs 0-3 each holding 16% VRAM already allocated. The cleanup script ran 3 attempts (SIGTERM→SIGKILL→30s wait each) but VRAM never dropped below the 5% threshold, so the job aborted with exit code 1 (line 545). This is a host/driver state problem on the runner node, independent of the sglang commit under test.

Regression Status

Infrastructure issue (recurring, not a code regression).

Recent history of job nightly-8-gpu-mi35x-glm51 (nightly-test-amd.yml, event=schedule, branch=main, completed runs only):

Date	Run	Job	Test File Status	Failed Step	Error
Jul 7	28888348134	nightly-8-gpu-mi35x-glm51	⚠️ Skipped (infra)	step 3 Ensure VRAM is clear	VRAM cleanup failed ×3
Jul 5	28749559519	nightly-8-gpu-mi35x-glm51	⚠️ Skipped (infra)	step 3 Ensure VRAM is clear	VRAM cleanup failed ×3
Jul 4	28714602438	nightly-8-gpu-mi35x-glm51	✅ Passed	—	—
Jul 3	28676359018	nightly-8-gpu-mi35x-glm51	✅ Passed	—	—

The same VRAM-cleanup failure also hit the sister job nightly-2-gpu-mi35x-glm51-mxfp4 in run 28676359018 (Jul 3), i.e. it is a runner-fleet problem, not job-specific.

Failure Cluster

Zombie KFD context — VRAM not clear on mi35x runner (pre-test infra abort)

Facts (verified observations)

• [FACT] Entry-state VRAM: GPU[0-3] at 16% each, GPU[4-7] at 0% (line 168-171).
• [FACT] All 3 cleanup attempts failed identically; VRAM stayed at 16% on GPU 0-3 across attempts 1-3 (lines 262, 371, 480).
• [FACT] No host process, docker container, or KFD PID held the GPUs during diagnostics — No KFD PIDs currently running, No docker containers found on host, empty fuser/zombie lists (lines 276, 322, 486). VRAM allocated with no owning process ⇒ classic zombie KFD context.
• [FACT] The script's own hint attributes this to a zombie KFD context and points to ROCm/aiter#2061, recommending a node reboot (lines 540-543).
• [FACT] AITER_COMMIT_OVERRIDE is empty (line 34) — this is a plain nightly run, not an AITER Scout run, so no Baseline A/B check applies.
• [FACT] Tests never ran (steps 4-7 skipped), so the sglang commit under test is not implicated.

Hypothesised Causes (with confidence)

• [HIGH] Zombie KFD (kernel fusion driver) context leaked by a previous job on the same runner node left VRAM allocated on GPU 0-3, which the userspace cleanup script cannot reclaim.
  • Supporting: 16% VRAM allocated with zero owning processes/containers/KFD PIDs across all diagnostics; matches the exact signature in ROCm/aiter#2061 that the script's own hint calls out; recurs on the mi35x fleet (Jul 5 same job, Jul 3 sister job) — a per-node persistent state, not a per-commit event.
  • Disconfirming: (none found) — no evidence points to the sglang commit; downstream tests never executed.
  • How to verify: reboot node linux-mi35x-gpu-8-zf775-runner-9rjn5 and re-run; VRAM should return to 0% and the job proceed.

In-flight Fix Check

• In-flight fix: ✅ #30431 "[DO NOT MERGE][AMD] ci: self-heal driver-level VRAM leaks + graceful GPU cleanup on shutdown" (open since 2026-07-07, non-draft but marked DO NOT MERGE) — directly targets driver-level VRAM leaks / GPU cleanup. Track this PR rather than opening a duplicate. (Search: is:pr is:open VRAM KFD and ensure_vram_clear.)

Suggested Triage Steps

• Reboot the affected runner node linux-mi35x-gpu-8-zf775-runner-9rjn5 (and audit other mi35x-8gpu nodes) to clear the zombie KFD context, then re-run 28888348134.
• Identify the prior job that ran on this node before this one and leaked the context — a crashed/killed 8-GPU run that didn't tear down its KFD context cleanly is the likely leaker.
• Follow / help land #30431 (self-healing driver-level VRAM cleanup) to make the runner recover automatically instead of hard-failing the job.
• Track upstream ROCm/aiter#2061 for the driver-side root fix.

Status

• Persistence: Intermittent infra failure on the mi35x fleet — this job failed Jul 5 and Jul 7 (passed Jul 3-4); sister job nightly-2-gpu-mi35x-glm51-mxfp4 also hit it Jul 3.
• Scope: Runner-node-level (zombie KFD context). Affects any job scheduled onto an mi35x node with a leaked context; no sglang/aiter code involvement.
• Blocked work: Nightly GLM-5.1 8-GPU accuracy + performance signal on MI35x is missing for this run.
• In-flight fix: ✅ #30431 (open since 2026-07-07) — self-heal driver-level VRAM leaks.

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-07-08 21:46 UTC)

[amd-bot]

nightly-test-amd.yml — Per-Job Analysis (continued 3/8)

nightly-8-gpu-mi35x-qwen35-triton-dcp — failed step(s): Ensure VRAM is clear

Commit Info

• sglang: 11cea29c
• aiter: N/A — no [CI-AITER-CHECK] markers in the log; the job failed at the pre-flight VRAM check before dependency install / container start.

Failed Tests

Test File	Test Function	Error	Log
N/A (infrastructure — no test executed)	N/A	VRAM cleanup unsuccessful after 3 attempts → exit code 1	Log

This is not a test failure. The job aborted in the Ensure VRAM is clear pre-flight step (scripts/ci/amd/ensure_vram_clear.sh rocm) before any SGLang test ran.

Failure Summary

The job failed during the Ensure VRAM is clear setup step: GPUs 0–3 each reported 16% VRAM allocated (above the 5% threshold), and three cleanup attempts (SIGTERM/SIGKILL + 30 s waits) failed to release it. The step exited 1 and no tests were executed. The runner's own diagnostics show no KFD PIDs, no docker containers, and no host processes holding the GPUs, yet VRAM stayed allocated.

Regression Status

Infrastructure issue (recurring, intermittent — runner-pool GPU state, not an SGLang code regression).

Recent history of the Ensure VRAM is clear step in job nightly-8-gpu-mi35x-qwen35-triton-dcp (nightly-test-amd.yml, branch main, event schedule, completed runs only):

Date	Run	Job	Step Status	Runner	Error
Jul 7	28888348134	…qwen35-triton-dcp	❌ Failed	…runner-ssc8r	VRAM stuck at 16% on GPU 0–3
Jul 5	28749559519	…qwen35-triton-dcp	✅ Passed	…runner-5gz6w	—
Jul 4	28714602438	…qwen35-triton-dcp	❌ Failed	…runner-ldtqx	Same VRAM cleanup failure
Jul 3	28676359018	…qwen35-triton-dcp	❌ Failed	…runner-kjdjw	Same VRAM cleanup failure

First observed in this window: Jul 3. Last passing: Jul 5. The failure recurs across different runner instances in the linux-mi35x-gpu-8-zf775-runner-* MI35x 8-GPU pool (ssc8r, ldtqx, kjdjw), so it is not a single stuck node — it is a pool-wide, intermittent pre-existing-VRAM-leak condition.

Failure Cluster

Pre-flight VRAM cleanup failure — zombie KFD context holds VRAM on MI35x 8-GPU runners.

Facts (verified observations, no interpretation)

• ERROR: VRAM usage exceeds threshold (5%) on some GPUs — GPU[0..3] each GPU Memory Allocated (VRAM%): 16; GPU[4..7] at 0% (log).
• Three cleanup attempts all ended ✗ VRAM still not clear (log); final === FAILED: VRAM cleanup unsuccessful after 3 attempts === → Process completed with exit code 1 (log).
• Diagnostics at each attempt: No KFD PIDs currently running, No docker containers found on host, no zombie host processes — GPUs held VRAM with no visible owner.
• The script's own hint attributes this to a zombie KFD context (references ROCm/aiter#2061) and states the node needs a reboot (log).
• AITER_COMMIT_OVERRIDE: is empty; this is a regular nightly run, not an AITER Scout run.
• The failure state (VRAM allocated by no visible process) is independent of the checked-out SGLang commit — same failure occurred at Jul 3/4 on different commits and different runners.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [FACT] The job was aborted by the ensure_vram_clear.sh pre-flight gate because GPUs 0–3 held VRAM the cleanup could not release. This is directly observable in the log.
• [HIGH] Root condition is a driver-level / zombie KFD VRAM leak on the MI35x runner pool left by a previous job on that GPU, not by this run's code.
  • Supporting: ROCm SMI shows no KFD PIDs, no containers, no host processes, yet 16% VRAM remains on 4 GPUs; the script's own diagnostic names zombie KFD context; recurs across three distinct runner pods on different SGLang commits.
  • Disconfirming: I did not inspect the prior job on runner ssc8r to confirm which workload leaked the VRAM, so the exact leak source is unproven (none found that weakens the "not-this-commit" conclusion).
  • How to verify: check the job that ran immediately before on the same physical MI35x node; confirm VRAM clears after a node reboot.
• [SPECULATION] A specific SGLang commit in the checkout window caused the leak — no evidence; the step fails before any SGLang code from this checkout runs, and the leak predates the checkout.

In-flight Fix Check

• In-flight fix: ✅ #30431 — [DO NOT MERGE][AMD] ci: self-heal driver-level VRAM leaks + graceful GPU cleanup on shutdown. Directly targets this exact driver-level VRAM-leak condition; marked DO NOT MERGE (still in development). Track/chase it rather than opening a duplicate.
• Search run: is:pr is:open VRAM cleanup (11 hits) and is:pr is:open ensure_vram_clear (0 hits).

Suggested Triage Steps

• Reboot / recycle the affected MI35x runner pods (linux-mi35x-gpu-8-zf775-runner-*) to clear the zombie KFD VRAM contexts, then re-run this job — the script hint explicitly states a reboot is required.
• Identify the job that ran immediately before on the same physical node to find the leaking workload (candidate source of the zombie KFD context).
• Advance in-flight #30431 (self-heal VRAM leaks + graceful cleanup on shutdown) toward mergeable state; it is the standing fix for this class of failure.
• Consider tracking upstream ROCm/aiter#2061 for the driver-level root cause.

Status (factual, no priority assignment)

• Persistence: Recurring on nightly-test-amd.yml — failed Jul 3, Jul 4, Jul 7 (passed Jul 5) at the same Ensure VRAM is clear pre-flight step. Intermittent, tied to runner GPU state.
• Scope: nightly-8-gpu-mi35x-qwen35-triton-dcp across ≥3 distinct runner pods in the MI35x 8-GPU pool; no SGLang tests executed on failing runs.
• Blocked work: Blocks the entire Qwen3.5 Triton DCP 8-GPU MI35x nightly signal whenever a runner has leaked VRAM.
• In-flight fix: ✅ #30431 (DO NOT MERGE / in development) — self-heal driver-level VRAM leaks.

nightly-accuracy-8-gpu-mi35x-deepseek-v32-mtp — failed step(s): Accuracy Test MI35x (8-GPU DeepSeek-V3.2 TP+MTP)

Commit Info

• sglang: 11cea29c
• aiter: 9127c94a (v0.1.17.dev110+g9127c94a1, source: dev image preserved — same as Dockerfile default, not overridden)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi35x/test_deepseek_v32_mtp_eval_mi35x.py	TestDeepseekV32TPMTP.setUpClass	Exception: Server process exited with code -9 (server crash during model load)	Log

Failure Summary

The DeepSeek-V3.2 MTP accuracy server failed to launch: setUpClass in test_deepseek_v32_mtp_eval_mi35x.py:75 called popen_launch_server, which raised Server process exited with code -9. The underlying scheduler crash is a ValueError: Unrecognized configuration class '_DeepseekV32ConfigAlias' for AutoModel, raised after the model was routed to the Transformers fallback backend (DeepseekV32ForCausalLM has no SGLang implementation, falling back to Transformers implementation). Ran 0 tests — the eval never started.

Regression Status

New failure signature (this run) on an otherwise intermittent job. This specific config-resolution crash has not appeared in the queryable window; the job flips between pass and several different failure modes day-to-day.

Recent history of test_deepseek_v32_mtp_eval_mi35x.py (TestDeepseekV32TPMTP) in job nightly-accuracy-8-gpu-mi35x-deepseek-v32-mtp (branch main, event=schedule, completed runs only):

Date	Run	Test File Status	Failed Function	Error
Jul 7	28888348134	❌ Failed	setUpClass	_DeepseekV32ConfigAlias unrecognized by AutoModel → exit -9 (crash @86s)
Jul 5	28749559519	✅ Passed	—	Accuracy 0.965, native SGLang load
Jul 4	28714602438	❌ Failed	(setup)	VRAM cleanup unsuccessful — infra/runner (different signature)
Jul 3	28676359018	✅ Passed	—	—
Jul 2	28611376093	❌ Failed	setUpClass	ran ~46 min then exit -9 (different signature)

Last known good for this test: Jul 5 (48ba79c11). First observed with the config-alias signature: Jul 7 (today). Note the Jul 2 / Jul 4 failures are distinct signatures, so they are not the same regression.

Failure Cluster

Transformers-fallback config crash — Unrecognized configuration class _DeepseekV32ConfigAlias for AutoModel

Facts (verified observations)

• Server launch command (log): sglang serve --model-path deepseek-ai/DeepSeek-V3.2 --trust-remote-code --tp 8 --dsa-prefill-backend tilelang --dsa-decode-backend tilelang --speculative-algorithm EAGLE --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 --mem-fraction-static 0.7 ...
• Model snapshot identical to the passing run: a7e62ac04ecb2c0a54d736dc46601c5606cf10a6.
• transformers version is 5.12.1 in BOTH the failing (image ...20260707) and passing (image ...20260705) runs — verified from Requirement already satisfied: transformers ... (5.12.1) in each log.
• On Jul 5 (transformers 5.12.1, sglang 48ba79c11) the model loaded natively (no fallback message, accuracy 0.965). Today, same transformers, sglang 11cea29c, it fell back to Transformers and crashed.
• Stack trace (verbatim top frames): model_runner.py:1512 load_model → loader.py:320 _initialize_model → transformers.py:617 → AutoModel.from_config(...) → ValueError: Unrecognized configuration class <class 'sglang.srt.utils.hf_transformers.common._DeepseekV32ConfigAlias'>.
• DeepseekV32ForCausalLM does have a native SGLang class (python/sglang/srt/models/deepseek_v2.py:2955, in EntryClass), so the fallback itself is the anomaly.
• The config alias _DeepseekV32ConfigAlias (python/sglang/srt/utils/hf_transformers/common.py:126) subclasses HF DeepseekV3Config; transformers' AutoModel mapping only knows the native DeepseekV32Config, not this alias — so once the Transformers path is taken, the crash is guaranteed.
• common.py (where the alias is registered) was NOT modified in the 48ba79c11..11cea29c window (67 commits) — alias behavior is unchanged between the passing and failing runs.
• Config/architecture-resolution files that WERE modified in the window: python/sglang/srt/utils/hf_transformers/config.py and python/sglang/srt/configs/model_config.py.

Hypothesised Causes

• [LOW] A commit in the 48ba79c11..11cea29c window changed config/architecture resolution so DeepseekV32ForCausalLM is no longer detected as natively supported (is_native_supported=False in model_loader/utils.py:217), routing it to the Transformers fallback where the alias is rejected. Most direct suspect: e339c83f (#30275) rewrote HfModelConfigParser.parse to call _try_load_longcat_config (PretrainedConfig.get_config_dict(...)) before AutoConfig.from_pretrained(..., trust_remote_code=...).
  • Supporting: it is the only window commit that edits the config parse entry point; the failing frame is config→AutoModel resolution; timing fits (merged Jul 7, first failing run Jul 7).
  • Disconfirming: for DeepSeek-V3.2 the architecture does not match _LONGCAT_ARCHS, so _try_load_longcat_config returns None and falls through to the identical AutoConfig.from_pretrained(trust_remote_code=...) call — no obvious behavioral change for this model. This makes a clean causal link weak.
  • How to verify: on sglang 11cea29c, launch deepseek-ai/DeepSeek-V3.2 --trust-remote-code --tp 8 on MI35x and inspect whether get_model_architecture returns the native class; then git revert e339c83f on a branch and re-test.
• [LOW] 6279805962 (#27867, DSv4 loading-time weight dequant) also modifies model_config.py in the window and could perturb DeepSeek config post-load handling.
  • Supporting: touches model_config.py, in-window.
  • Disconfirming: targeted at DSv4, not V3.2; no evidence it changes the V3.2 architecture list.
  • How to verify: include in the bisect below.
• [SPECULATION] Non-deterministic AutoConfig.register ordering (native transformers deepseek_v32 vs sglang alias) — but disfavored because common.py is unchanged and Jul 5 used the same transformers 5.12.1 yet loaded natively.

No hypothesis reaches MEDIUM. Needs git bisect between 48ba79c11..11cea29c.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for DeepseekV32+AutoModel, DeepseekV32ConfigAlias, deepseek v32 transformers fallback, DeepseekV32 trust-remote-code (via search/issues); no matching PR. The two hits for the last query (#22872, #16151) are unrelated (Elastic EP, W4AFP8).

Suggested Triage Steps

• Bisect 48ba79c11..11cea29c with a minimal AutoConfig + get_model_architecture repro for deepseek-ai/DeepSeek-V3.2 --trust-remote-code (no GPU needed to reproduce the fallback decision) — focus first on e339c83f (#30275) and 6279805962 (#27867).
• Add a debug print of architectures and is_native_supported in model_loader/utils.py:217 for a passing (Jul 5 SHA) vs failing (today SHA) checkout to pinpoint where native detection flips.
• If confirmed as a config-alias gap, one fix direction is to also register _DeepseekV32ConfigAlias in the AutoModel mapping (so the fallback path can construct it) — but the preferred fix is to keep DeepseekV32 on the native SGLang path; decide after bisect.
• Because the job is intermittent with multiple signatures, re-run this job once to confirm the config-alias crash is reproducible (vs a one-off), and keep the Jul 2 / Jul 4 failures tracked separately (46-min exit -9; VRAM-cleanup infra).

Status

• Persistence: First occurrence of this config-alias signature (Jul 7). The job itself has failed 3 of the last 5 completed scheduled runs, but with 3 different signatures.
• Scope: 1 job (nightly-accuracy-8-gpu-mi35x-deepseek-v32-mtp), 1 test file. The MTP throughput/serving job nightly-8-gpu-deepseek-v32-mtp passed in recent runs; not observed elsewhere.
• Blocked work: DeepSeek-V3.2 MTP accuracy signal on MI35x (eval never runs — 0 tests executed).
• In-flight fix: ❌ none found — needs new PR or investigation.

nightly-8-gpu-mi35x-qwen35 — failed step(s): Ensure VRAM is clear

Commit Info

• sglang: 11cea29c ([diffusion][cache-dit] add dual-transformer Cache-DiT adapter specs (#30150))
• aiter: N/A — job failed at the pre-flight VRAM cleanup step before dependency install; no [CI-AITER-CHECK] markers in the log.

Failed Tests

Test File	Test Function	Error	Log
N/A (no test ran)	N/A	ensure_vram_clear.sh: === FAILED: VRAM cleanup unsuccessful after 3 attempts === → exit code 1	Log

This is not a test failure. The job failed in the Run bash scripts/ci/amd/ensure_vram_clear.sh rocm pre-flight step and never reached the SGLang container / test phase.

Failure Summary

The nightly-8-gpu-mi35x-qwen35 job failed during the AMD VRAM pre-flight cleanup on runner linux-mi35x-gpu-8-zf775-runner-chpkn. At job start, GPUs 0–3 each reported 16% VRAM already allocated (GPUs 4–7 at 0%). The cleanup loop ran all 3 attempts and could not clear it, so the step exited 1 and the job aborted before any qwen35 test executed.

Regression Status

Known recurring failure — infrastructure (not a code regression).

The failing unit is the pre-flight ensure_vram_clear.sh step, not a test file. History of the nightly-8-gpu-mi35x-qwen35 job at this step (branch main, event=schedule, completed runs only):

Date	Run	Job	Step Status	Runner	Error
Jul 7	28888348134	qwen35	❌ Failed	...runner-chpkn	VRAM cleanup unsuccessful (GPUs 0-3 @16%)
Jul 5	28749559519	qwen35	❌ Failed	...runner-sp75s	VRAM cleanup unsuccessful (GPUs 0-3 @16%)
Jul 4	28714602438	qwen35	❌ Failed	...runner-rvtf9	VRAM cleanup unsuccessful (GPUs 0-3 @16%)
Jul 3	28676359018	qwen35	❌ Failed	...runner-rpgj7	VRAM cleanup unsuccessful (GPUs 0-3 @16%)

(Jul 6 run 28813271245 was cancelled → excluded.) Same pre-flight failure for 4 consecutive completed scheduled runs (Jul 3–7). No passing run for this step in the queryable window. Different runner container names each time, but all from the linux-mi35x-gpu-8-zf775 node pool — and the value is identically 16% on GPUs 0-3 every run.

Failure Cluster

Stuck/zombie KFD VRAM allocation on mi35x node pool — pre-flight VRAM cleanup fails (ensure_vram_clear.sh, 3 attempts).

Facts (verified observations)

• Step: Run bash scripts/ci/amd/ensure_vram_clear.sh rocm (log group opens at line 110); final failure === FAILED: VRAM cleanup unsuccessful after 3 attempts === at log line 537; ##[error]Process completed with exit code 1. at line 545.
• Initial GPU status (line 131-144): GPU[0-3] = 16% VRAM, GPU[4-7] = 0%. Same after all 3 cleanup attempts.
• Diagnostics on every failed attempt found no owner: No KFD PIDs currently running, No docker containers found on host, no zombie processes, fuser/lsof found no device-file holders (lsof not installed, line 266).
• scripts/ci/amd/ensure_vram_clear.sh:257-261 — the script's own final hint: "if no host process / container holds the GPU but VRAM is still allocated, this is almost certainly a zombie KFD context (see [Bug] Custom all-reduce IPC buffers use fixed VA, conflict with torch_memory_saver pause/resume → zombie processes on MI300X ROCm/aiter#2061). The node will need to be rebooted before subsequent jobs can succeed."
• Reproducibility: identical failure signature across 4 runs / 4 distinct runner hostnames in the linux-mi35x-gpu-8-zf775 pool. Many sibling mi35x jobs in the same runs also failed, consistent with a shared node-pool condition rather than a per-job issue.

Hypothesised Causes (with confidence)

• [HIGH] Zombie/orphaned KFD VRAM context on the mi35x node pool holds ~16% VRAM on GPUs 0-3 with no reclaimable owner, tripping the 5% pre-flight threshold.
  • Supporting: no host process, container, KFD PID, or device-fd holder found after aggressive SIGTERM/SIGKILL cleanup, yet VRAM stays at exactly 16% across all 3 attempts; the script explicitly documents this exact signature (zombie KFD context, [Bug] Custom all-reduce IPC buffers use fixed VA, conflict with torch_memory_saver pause/resume → zombie processes on MI300X ROCm/aiter#2061) and states the node must be rebooted.
  • Disconfirming: exactly-16%-on-GPUs-0-3-every-run across different runner names could alternatively be a reserved/baseline carveout or an SMI reporting quirk on MI35X rather than a leaked context — but that would still be an infra/threshold issue, not sglang code.
  • How to verify: reboot (or drain + power-cycle) the affected linux-mi35x-gpu-8-zf775 node(s); re-check rocm-smi --showmemuse at idle. If it returns to 0%, confirmed zombie context; if it stays at 16%, it's a firmware/driver baseline vs. the 5% threshold.
• [FACT] The failure is unrelated to sglang commit 11cea29c. The step runs before checkout-dependent test code executes any product logic, and the same step failed identically on 3 earlier commits.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for ensure_vram_clear (0 results) and VRAM mi35x (5 results, all unrelated feature PRs, none touching CI VRAM cleanup). API calls: search/issues?q=repo:sgl-project/sglang+is:pr+is:open+ensure_vram_clear and ...+VRAM+mi35x.

Suggested Triage Steps

• Infra owner: reboot / power-cycle the linux-mi35x-gpu-8-zf775 node pool to clear the stuck KFD context (per the script hint and [Bug] Custom all-reduce IPC buffers use fixed VA, conflict with torch_memory_saver pause/resume → zombie processes on MI300X ROCm/aiter#2061), then rerun nightly-test-amd.yml.
• If VRAM stays at 16% after reboot, investigate whether MI35X reports a non-zero idle VRAM baseline and, if so, revisit the 5% threshold in scripts/ci/amd/check_vram_clear.sh for this GPU family.
• No sglang code bisect is warranted — this is not a code regression.

Status

• Persistence: Pre-flight VRAM cleanup failing for ≥4 consecutive completed scheduled runs (since Jul 3 2026), including today.
• Scope: nightly-8-gpu-mi35x-qwen35, plus numerous sibling mi35x jobs failing in the same runs — points to a shared linux-mi35x-gpu-8-zf775 node-pool condition, not a single job.
• Blocked work: mi35x nightly qwen35 signal is fully blocked; no test coverage produced.
• In-flight fix: ❌ none found — needs infra action (node reboot), not a code PR.

nightly-perf-8-gpu-mi35x-deepseek-v32-basic — failed step(s): Ensure VRAM is clear

Commit Info

• sglang: 11cea29c
• aiter: N/A — job failed at the pre-flight VRAM gate before dependency install; no [CI-AITER-CHECK] markers exist in the log. AITER_COMMIT_OVERRIDE was empty (log line 34/114).

Failed Tests

Test File	Test Function	Error	Log
N/A (no test ran)	N/A	FAILED: VRAM cleanup unsuccessful after 3 attempts → exit code 1	Log L537

This is not a test failure. The job died in the bash scripts/ci/amd/ensure_vram_clear.sh rocm pre-flight step (L110), which is a host-environment gate that runs before the container, dependency install, and any DeepSeek-V3.2 perf test.

Failure Summary

On runner node linux-mi35x-gpu-8-zf775-runner-v9bhl, GPUs 0–3 each held 15–16% allocated VRAM at job start (L167-171). The cleanup script ran 3 attempts (SIGTERM→SIGKILL→30 s wait each), but VRAM never dropped below the 5% threshold and the job exited 1 (L537). Notably, rocm-smi --showpids reported "No KFD PIDs currently running" and no host processes/containers held the GPUs (L276, L488) — VRAM was allocated with no owning process.

Regression Status

Known recurring infrastructure failure (not a code regression). The DeepSeek-V3.2 perf test itself never executed, so this reflects runner-pool health, not sglang code.

Recent history of job nightly-perf-8-gpu-mi35x-deepseek-v32-basic (nightly-test-amd.yml, branch=main, event=schedule, completed runs only):

Date	Run	Job	Status	Failure	Runner node
Jul 7	28888348134	job	❌ Failed	VRAM cleanup gate	...zf775-runner-v9bhl
Jul 6	28813271245	—	⚪ Cancelled	(run cancelled, excluded)	—
Jul 5	28749559519	job	❌ Failed	VRAM cleanup gate	...zf775-runner-nxk88
Jul 4	28714602438	job	✅ Passed	—	—
Jul 3	28676359018	job	✅ Passed	—	—

First observed: Jul 5 2026. Last passing: Jul 4 2026. Failing intermittently (2 of last 4 completed runs) and correlated with the mi35x-gpu-8-zf775 runner pool, not with the head SHA. Note: the parent run 28888348134 was still queued at the run level at snapshot time (other jobs in-flight); this specific job has completed with failure.

Failure Cluster

Zombie-KFD-context VRAM leak on mi35x-8GPU runner (pre-flight ensure_vram_clear.sh gate fails)

Facts (verified observations)

• Gate step ensure_vram_clear.sh rocm failed after 3 cleanup attempts; exit code 1 (L537-545).
• GPUs 0–3 held 15–16% VRAM; GPUs 4–7 at 0% (L167-171).
• No KFD PIDs currently running and no host processes/docker containers held the GPU device files (L264-278).
• The script's own hint explicitly attributes this to a zombie KFD context and points at ROCm/aiter#2061, stating the node must be rebooted (L540-543).
• Same symptom + same runner pool (different node nxk88) on Jul 5 (L537 of prev job).
• No test, no container, no aiter/torch versions were reached — the failure is entirely on the host before workload start.

Hypothesised Causes (with confidence)

• [HIGH] Host-level leaked/zombie KFD GPU context on the mi35x-gpu-8-zf775 runner pool leaves VRAM allocated on GPUs 0–3 with no owning process; the pre-flight gate correctly refuses to run and the node needs a reboot.
  • Supporting: VRAM allocated but No KFD PIDs and no process/container holds the device files; the script's built-in diagnostic explicitly names this scenario (ROCm/aiter#2061); reproduced on a second node in the same pool (Jul 5).
  • Disconfirming: (none found) — no evidence pointing at the sglang HEAD SHA; Jul 3/4 passed on the same code line and workload.
  • How to verify: reboot / drain the affected runner node(s) and re-run; confirm GPUs 0–3 report 0% VRAM at job start.
• [LOW] A prior job on the same node crashed without releasing its HIP context (e.g. an earlier 8-GPU perf run OOM/segfault), leaving the leak.
  • Supporting: only GPUs 0–3 (a subset) are dirty, consistent with a prior partial-teardown workload.
  • Disconfirming: not observable from this log — the leaking job ran in a previous session; needs host-side journald/dmesg from the node.
  • How to verify: inspect the previous job scheduled on ...runner-v9bhl and check node dmesg for GPU faults around that time.

In-flight Fix Check

• In-flight fix: ✅ [#30431](https://github.com/sgl-project/sglang/pull/30431) ("[DO NOT MERGE][AMD] ci: self-heal driver-level VRAM leaks + graceful GPU cleanup on shutdown", open since 2026-07-07) — directly targets this driver-level VRAM-leak/self-heal scenario. Currently marked DO NOT MERGE; track it rather than opening a duplicate. (Query: search/issues?q=repo:sgl-project/sglang+is:pr+is:open+VRAM+cleanup.)

Suggested Triage Steps

• Reboot or drain the affected mi35x-gpu-8-zf775 runner nodes (...runner-v9bhl, ...runner-nxk88); confirm all 8 GPUs read 0% VRAM before re-enabling, then re-run the job.
• Track #30431 and, if its self-heal approach is validated, push it out of DO-NOT-MERGE state so the gate can recover the node instead of hard-failing.
• If leaks recur, capture host dmesg/journald + rocm-smi --showpids from the node immediately after a failing gate to identify the workload that leaks the KFD context (candidate: the preceding 8-GPU perf job).
• Because this is a runner-health gate failure, no sglang code bisect is warranted — do not attribute to any commit in the range.

Status

• Persistence: Recurring since 2026-07-05 (failed Jul 5 and Jul 7; Jul 6 run cancelled; Jul 3–4 passed). Intermittent, tied to node state rather than every run.
• Scope: Pre-flight VRAM gate on the mi35x-gpu-8-zf775 8-GPU runner pool; observed on ≥2 distinct nodes. Blocks the DeepSeek-V3.2 8-GPU perf nightly (and would block any other job scheduled to a dirtied node in this pool).
• Blocked work: DeepSeek-V3.2 basic 8-GPU perf signal on MI35x is unavailable on affected runs.
• In-flight fix: ✅ #30431 (open since 2026-07-07, DO NOT MERGE) — self-heal driver-level VRAM leaks.

nightly-8-gpu-mi35x-deepseek-r1-mxfp4 — failed step(s): Ensure VRAM is clear

Commit Info

• sglang: 11cea29c
• aiter: N/A — job failed at the pre-flight VRAM check before Docker/dependency install ran, so no [CI-AITER-CHECK] markers were emitted. AITER_COMMIT_OVERRIDE was empty (this is a regular nightly run, not an AITER Scout run).

Failed Tests

Test File	Test Function	Error	Log
N/A (infrastructure — no test ran)	N/A	FAILED: VRAM cleanup unsuccessful after 3 attempts → exit code 1	Log

No test executed. The failure is in step 3 "Ensure VRAM is clear" (scripts/ci/amd/ensure_vram_clear.sh rocm). Steps 4–6 (Setup docker, Install dependencies, Accuracy Test MI35x (8-GPU DeepSeek-R1-MXFP4), Performance Test) were all skipped.

Failure Summary

The pre-flight VRAM guard aborted the job because GPUs 0–3 each held 16% allocated VRAM (threshold is 5%) on runner linux-mi35x-gpu-8-zf775-runner-qq66f. Three cleanup attempts (SIGTERM→SIGKILL + 30 s waits) failed to reclaim it, yet diagnostics showed no KFD PIDs, no Docker containers, and no host processes holding the GPUs. The DeepSeek-R1-MXFP4 accuracy/perf tests never ran.

Regression Status

Infrastructure issue (recurring) — not a code regression. The commit SHA is irrelevant; the leaked VRAM is left over from a prior job on the same physical runner pool.

Recent history of Ensure VRAM is clear step in job nightly-8-gpu-mi35x-deepseek-r1-mxfp4 (branch main, event schedule, completed runs only):

Date	Run	Runner	Step Status	Error
Jul 7	28888348134	zf775…qq66f	❌ Failed	VRAM 16% on GPU0-3, cleanup failed ×3
Jul 6	28813271245	—	⚪ Run cancelled (excluded)	—
Jul 5	28749559519	zf775…45s7t	❌ Failed	same VRAM leak
Jul 4	28714602438	zf775…jsc42	❌ Failed	same VRAM leak
Jul 3	28676359018	zf775…t2txx	✅ Passed	—

First observed failure for this job: Jul 4 2026. Last known passing: Jul 3 2026. The same Ensure VRAM is clear failure also hit sibling mi35x jobs (-ar-fusion, -kv-fp8, -tp2) across the zf775 and 98ncc runner pools in the same runs — this is a pool-wide runner health problem, not specific to the deepseek-r1-mxfp4 job.

Failure Cluster

Pre-flight VRAM guard abort — leaked/zombie KFD VRAM on mi35x runners (no live GPU holder)

Facts (verified observations, no interpretation)

• Verbatim: ERROR: VRAM usage exceeds threshold (5%) on some GPUs: GPU[0..3] GPU Memory Allocated (VRAM%): 16 (log lines 167–171).
• All three cleanup attempts printed ✗ VRAM still not clear after attempt N; final === FAILED: VRAM cleanup unsuccessful after 3 attempts === (line 537), then ##[error]Process completed with exit code 1 (line 545).
• Diagnostics on every attempt: No KFD PIDs currently running (lines 276, 385, 494), No docker containers found on host (line 213), zombie-process list empty (lines 268, 377, 486).
• The script itself prints the hypothesis: this is almost certainly a zombie KFD context (see ROCm/aiter#2061). The node will need to be rebooted (lines 541–543).
• GPUs 4–7 were clean (0–1% VRAM); only GPUs 0–3 leaked — consistent with a prior 4-GPU tenant leaving state behind.
• Steps 4–6 skipped (verified via job steps API); zero test execution.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [HIGH] Leaked VRAM from a prior job on the same runner (zombie KFD context) — the guard is behaving correctly by refusing to start.
  • Supporting: 16% VRAM allocated on 4 GPUs with no KFD PID / no container / no host process holding them; identical symptom recurs across multiple runners in the zf775/98ncc mi35x pools and across Jul 4/5/7; the script's own hint cites [Bug] Custom all-reduce IPC buffers use fixed VA, conflict with torch_memory_saver pause/resume → zombie processes on MI300X ROCm/aiter#2061.
  • Disconfirming: (none found) — no evidence points to the checked-out sglang commit; the tests never ran, so no sglang/aiter code path was exercised.
  • How to verify: reboot / drain runner linux-mi35x-gpu-8-zf775-runner-qq66f, then re-run the job; confirm VRAM returns to <5% at pre-flight.
• [LOW] The ensure_vram_clear.sh cleanup is insufficient (can't evict a zombie KFD context via SIGTERM/SIGKILL).
  • Supporting: SIGKILL of GPU-device holders did not release the memory across 3 attempts.
  • Disconfirming: this is a mitigation gap, not the originating cause; a driver-level context is not killable from userspace by design.
  • How to verify: see in-flight PR #30431 below, which targets driver-level self-heal.

In-flight Fix Check

• In-flight fix: ✅ #30431 "[DO NOT MERGE][AMD] ci: self-heal driver-level VRAM leaks + graceful GPU cleanup on shutdown" (open since 2026-07-07, currently marked DO-NOT-MERGE/WIP) — directly addresses driver-level VRAM leaks + GPU cleanup on shutdown. Track this PR; do not open a duplicate.
• Search runs: search/issues?q=repo:sgl-project/sglang+is:pr+is:open+VRAM+clear (16 hits, #30431 top), ...+KFD+zombie (2 hits, #30431 top), ...+ensure_vram_clear (0 hits).

Suggested Triage Steps

• Reboot or drain the affected mi35x runners (zf775 + 98ncc pools; today's node …qq66f) to clear the zombie KFD VRAM, then re-run this job.
• Chase reviewers on #30431 and move it out of DO-NOT-MERGE — it is the standing fix for this cluster.
• Track upstream [Bug] Custom all-reduce IPC buffers use fixed VA, conflict with torch_memory_saver pause/resume → zombie processes on MI300X ROCm/aiter#2061 (the driver/KFD context leak referenced by the cleanup script) for a root-cause driver fix.
• Consider adding a post-job GPU-cleanup / node-cordon-on-leak step so a leaking job doesn't poison the next scheduled job on the same runner.

Status (factual, no priority assignment)

• Persistence: Ensure VRAM is clear has failed this job on Jul 4, Jul 5, and Jul 7 (3 of the last 3 completed scheduled runs it appeared in; passed Jul 3). Recurring for ~4 days.
• Scope: Pool-wide on mi35x runners — same failure in sibling jobs -ar-fusion, -kv-fp8, -tp2 across the zf775 and 98ncc runner pools in the same nightly runs. No test signal produced for 8-GPU DeepSeek-R1-MXFP4 accuracy/perf on those affected runs.
• Blocked work: DeepSeek-R1-MXFP4 nightly accuracy + performance coverage on MI35x whenever a run lands on a leaked runner.
• In-flight fix: ✅ #30431 (open since 2026-07-07, DO-NOT-MERGE) — self-heal driver-level VRAM leaks.

nightly-8-gpu-mi35x-glm5-mxfp4 — failed step(s): Ensure VRAM is clear

Commit Info

• sglang: 11cea29c
• aiter: N/A — the job failed in the ensure_vram_clear.sh pre-flight step, before dependency install; no [CI-AITER-CHECK] markers were emitted.

Failed Tests

Test File	Test Function	Error	Log
N/A (pre-test infra step)	N/A	VRAM cleanup unsuccessful after 3 attempts → exit code 1	Log

This is not a test failure. The job never reached any pytest/test suite. It failed in the workflow's setup step bash scripts/ci/amd/ensure_vram_clear.sh rocm, which is a runner-health precondition check that runs before the sglang container starts.

Failure Summary

The nightly-8-gpu-mi35x-glm5-mxfp4 job aborted during the pre-test VRAM-clear step on runner linux-mi35x-gpu-8-zf775-runner-tkk7x. GPUs 0–3 each reported 16% VRAM allocated that could not be freed across 3 cleanup attempts (SIGTERM/SIGKILL + 30s waits), despite rocm-smi --showpids reporting no KFD processes running and fuser finding no GPU device-file holders. The step exited 1 and the job never started the sglang container or any GLM5-mxfp4 test.

Regression Status

Known recurring infrastructure failure (runner-level, not code).

Recent history of the VRAM-clear pre-test step in job nightly-8-gpu-mi35x-glm5-mxfp4 (branch main, event schedule, completed runs only):

Date	Run	Job	Step Status	Failure	Error
Jul 7	28888348134	nightly-8-gpu-mi35x-glm5-mxfp4	❌ Failed	VRAM-clear	VRAM cleanup unsuccessful after 3 attempts (GPU0-3 stuck at 16%)
Jul 5	28749559519	nightly-8-gpu-mi35x-glm5-mxfp4	✅ Passed	—	—
Jul 4	28714602438	nightly-8-gpu-mi35x-glm5-mxfp4	✅ Passed	—	—
Jul 3	28676359018	nightly-8-gpu-mi35x-glm5-mxfp4	❌ Failed	VRAM-clear	VRAM cleanup unsuccessful after 3 attempts (verified: same infra pattern)

This is intermittent and runner-bound, not commit-bound: the same GLM5-mxfp4 job passed Jul 4 & 5 on different commits, and the identical VRAM-clear failure struck Jul 3. It tracks which physical runner the job lands on, not the sglang code state.

Failure Cluster

Zombie KFD context — stuck VRAM on MI35x runner (pre-test VRAM-clear step fails, exit 1).

Facts (verified observations)

• Failing step: Run bash scripts/ci/amd/ensure_vram_clear.sh rocm (log lines 110–545).
• GPUs 0,1,2,3 each report GPU Memory Allocated (VRAM%): 16; GPUs 4–7 at 0% (line 168-171).
• All 3 cleanup attempts fail identically; VRAM never drops below the 5% threshold.
• rocm-smi --showpids → No KFD PIDs currently running (line 276); fuser/lsof find no device-file holders; no docker containers, no zombie processes.
• The script itself diagnoses the cause: "almost certainly a zombie KFD context (see [Bug] Custom all-reduce IPC buffers use fixed VA, conflict with torch_memory_saver pause/resume → zombie processes on MI300X ROCm/aiter#2061). The node will need to be rebooted" (line 540-543).
• AITER_COMMIT_OVERRIDE: is empty (line 34) — not an AITER Scout run; no Baseline A/B check applies.
• Jul 3 failure (85050499098) confirmed to be the identical VRAM cleanup unsuccessful after 3 attempts pattern.

Hypothesised Causes (with confidence)

• [HIGH] Stale/zombie KFD (kernel-fusion-driver) GPU context left allocated on the runner by a previous job, holding ~16% VRAM on GPUs 0-3 that no live process owns.
  • Supporting: VRAM is allocated but rocm-smi --showpids shows no KFD PIDs and fuser finds no holders — the classic signature of a driver-level leak that user-space SIGKILL cannot reclaim ([Bug] Custom all-reduce IPC buffers use fixed VA, conflict with torch_memory_saver pause/resume → zombie processes on MI300X ROCm/aiter#2061). Only GPUs 0-3 affected, consistent with a prior partial-TP job's leftover.
  • Disconfirming: (none found) — the log evidence is internally consistent and the script's own diagnostic reaches the same conclusion.
  • How to verify: reboot/drain runner linux-mi35x-gpu-8-zf775-runner-tkk7x and re-run; confirm VRAM returns to 0%.
• [FACT] This is not caused by sglang commit 11cea29c — the failure occurs in the pre-flight runner check before any sglang code executes, and the same job passed on other commits Jul 4/5.

In-flight Fix Check

• In-flight fix: ✅ #30431 "[DO NOT MERGE][AMD] ci: self-heal driver-level VRAM leaks + graceful GPU cleanup on shutdown" (open, non-draft, created 2026-07-07) — directly targets driver-level VRAM leak self-healing. Currently marked DO NOT MERGE; track it rather than opening a duplicate. (Search: search/issues?q=repo:sgl-project/sglang+is:pr+is:open+ensure_vram_clear+OR+VRAM+zombie, 7 hits.)

Suggested Triage Steps

• Reboot/drain the affected runner linux-mi35x-gpu-8-zf775-runner-tkk7x to clear the zombie KFD context, then re-run the job — this is the remediation the script itself recommends.
• Track #30431 and [Bug] Custom all-reduce IPC buffers use fixed VA, conflict with torch_memory_saver pause/resume → zombie processes on MI300X ROCm/aiter#2061 as the systemic self-heal fix; if it stalls, chase reviewers.
• If the leak recurs on the same node, correlate with the preceding job scheduled on zf775 to find which workload leaves the 16% VRAM behind on GPUs 0-3 (likely a partial-TP / crashed job that didn't tear down its KFD context).

Status (factual)

• Persistence: Intermittent, runner-bound. Failed Jul 3 and Jul 7; passed Jul 4 & 5 on the same job with different commits. Correlates with runner state, not commit state.
• Scope: This job/runner instance. The pre-test VRAM-clear guard is shared across all AMD MI35x nightly jobs, so any job landing on a leaked node hits the same failure.
• Blocked work: GLM5-mxfp4 8-GPU nightly coverage on this run; no test signal was produced.
• In-flight fix: ✅ #30431 (open, DO NOT MERGE) — driver-level VRAM leak self-heal.

nightly-8-gpu-mi35x-qwen3-235b-mxfp4 — failed step(s): Accuracy Test + Performance Test MI35x (8-GPU Qwen3-235B-MXFP4)

Commit Info

• sglang: 11cea29c
• aiter: 9127c94a — v0.1.17.dev110+g9127c94a1 (source: dev image; [CI-AITER-CHECK] reports "Dev/patched version detected → skipping rebuild", matches Dockerfile default for gfx950)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/test_qwen3_instruct_mxfp4.py	N/A (server never became ready — no test body reached)	TIMEOUT after 3600s → exit code 255	Log

Failure Summary

The single-file test suite test/registered/amd/test_qwen3_instruct_mxfp4.py timed out after 3600s and the job exited 255. The sglang server for amd/Qwen3-235B-A22B-Instruct-2507-mxfp4 (tp=4, ep=2, --attention-backend aiter) launched at 19:23:23 but never reached readiness; every scheduler rank remained blocked in HuggingFace weight download for the entire hour. No accuracy assertion or forward pass was ever executed.

Regression Status

Never-passed (no green run found in the queryable window) — but today's symptom is new for this job.

Recent history of test_qwen3_instruct_mxfp4.py in job nightly-8-gpu-mi35x-qwen3-235b-mxfp4 (branch=main, event=schedule, completed runs only):

Date	Run	Test File Status	Failure signature
Jul 7	28888348134	❌ Failed	TIMEOUT 3600s — HF weight download hang (this run)
Jul 6	28813271245	⏹️ Cancelled	(run cancelled — excluded)
Jul 5	28749559519	❌ Failed	VRAM cleanup unsuccessful after 3 attempts (exit 1)
Jul 4	28714602438	❌ Failed	VRAM cleanup unsuccessful after 3 attempts (exit 1)
Jul 2	28611376093	❌ Failed	Rank 0 scheduler died during initialization (exit code: -6)

This job has failed every completed nightly since at least Jul 2, but with three distinct signatures — today's download-timeout is not the same failure as the prior VRAM-cleanup or scheduler-died crashes. First observed failure of this download-hang signature: Jul 7. No known passing date for this test file in the window.

Failure Cluster

Model weight download hang (HF Xet) → 3600s test timeout

Facts (verified observations)

• Server args (log 19:23:23): model_path='amd/Qwen3-235B-A22B-Instruct-2507-mxfp4', tp_size=4, ep_size=2, attention_backend='aiter', soft_watchdog_timeout=300.
• All scheduler ranks (TP0–TP3, EP0/EP1) plus TokenizerManager fired soft watchdog timeouts every ~5 min from 19:28:26 through 20:18:38 — 10+ consecutive cycles, all still in the same state.
• py-spy dumps (e.g. log line ~7620 and 4899/5167) show the blocking stack is identical every time:

  snapshot_download (huggingface_hub/_snapshot_download.py:492)
  ... ci_download_with_validation_and_retry (model_loader/ci_weight_validation.py:1955)
  download_weights_from_hf (model_loader/weight_utils.py:610)
  ... load_weights (qwen3_moe.py:1124) ... load_model (model_runner.py:1512)
  ThreadPoolExecutor-1_0 (active): xet_get (huggingface_hub/file_download.py:568)
  

  The worker thread is actively in xet_get (HF Xet/hf-xet backend, v1.5.1) — i.e. the download was in progress but never completed within the budget.
• Main server process was parked in _wait_for_scheduler_ready (engine.py:1380) the whole time — the model never finished loading.
• Timeout enforced by the runner: ✗ TIMEOUT: ...test_qwen3_instruct_mxfp4.py after 3600 seconds (log line 7654); harness reported "Exit code 255 indicates test failure (not network issue), not retrying".
• Container image pull earlier hit a transient local-registry failure and fell back to public registry successfully (log 1436–1437) — unrelated to the model download.

Hypothesised Causes (with confidence)

• [MEDIUM] HuggingFace Xet weight download stalled/too slow for the ~120GB+ Qwen3-235B mxfp4 checkpoint, consuming the full 3600s test budget before the server could start.
  • Supporting: py-spy shows the process wedged in xet_get for the entire hour with no progress; server never left _wait_for_scheduler_ready; download is the only active code path.
  • Disconfirming: prior runs of the same job (Jul 2/4/5) got past download and failed later (scheduler init / VRAM), so weight fetch usually succeeds on this runner — pointing to a transient network/Xet-backend slowness today rather than a permanent config break. The harness's "not a network issue" note is heuristic (based on exit-code 255) and is contradicted by the stack trace.
  • How to verify: re-run the job; check runner egress bandwidth to the HF Xet CAS endpoint; pre-stage amd/Qwen3-235B-A22B-Instruct-2507-mxfp4 into a local cache/--download-dir; try HF_HUB_DISABLE_XET=1 to fall back to the classic transfer backend and compare download time.
• [LOW] 3600s per-file budget is too tight for a 235B download + load + accuracy eval when the model is not pre-cached on the runner.
  • Supporting: the entire hour was spent in download alone, leaving zero time for load/eval.
  • Disconfirming: other 235B jobs (nightly-8-gpu-qwen3-235b) succeed in the same nightly, suggesting the model is normally cached and the budget is usually sufficient.
  • How to verify: confirm whether the mi35x runner has a warm HF cache for this repo; measure a cold-cache download time.

Root cause is not attributable to aiter or an sglang code change: this is a nightly (nightly-test-amd.yml, not a scout run), the aiter version is the Dockerfile default dev build, and the failure occurs entirely in the HF download path before any model/kernel code runs.

In-flight Fix Check

• In-flight fix: ❌ none found — no open PR matches the download-timeout for this test. (Searched qwen3 235b mxfp4 → only #29419 "ci(amd): make sgl-data mount opt-in" and #27209 gfx950 Dockerfile, neither addresses this; snapshot_download timeout → no open PRs.)

Suggested Triage Steps

• Re-run nightly-8-gpu-mi35x-qwen3-235b-mxfp4 alone; if it passes, treat today's failure as a transient HF Xet download stall.
• Pre-stage the model on the mi35x runner (or set --download-dir/warm HF cache) so nightly runs don't re-download 235B weights inside the 3600s window.
• Try HF_HUB_DISABLE_XET=1 for this case and compare download throughput — isolates whether the Xet CAS backend is the bottleneck.
• Separately, note this job has been red since ≥ Jul 2 with rotating symptoms (download hang → VRAM cleanup → scheduler init -6); the mxfp4 235B case likely needs dedicated stabilization beyond the download issue.

Status

• Persistence: nightly-8-gpu-mi35x-qwen3-235b-mxfp4 failing every completed nightly since ≥ 2026-07-02; today's download-hang signature first seen 2026-07-07. Test file has no known passing run in the queryable window.
• Scope: 1 job in nightly-test-amd.yml (mi35x 8-GPU). The sibling nightly-8-gpu-qwen3-235b (non-mxfp4) passed in the same runs.
• Blocked work: Qwen3-235B mxfp4 accuracy signal on AMD mi35x — no coverage while red.
• In-flight fix: ❌ none found — needs new investigation.

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-07-08 21:46 UTC)

[amd-bot]

nightly-test-amd-rocm720.yml — 61 failure(s)

Run started (UTC): 2026-07-07 18:10 UTC
Last scanned (UTC): 2026-07-08 20:50 UTC
Commits: sglang 8d2b66f

Summary

Counts: 61 failures · 9 clusters · 44 infra failures carrying over (≥3 days) · 5 code failures first-run today · 9 cutlass-fallback (Run A only) resolved by merged #30374

Spans two scheduled runs of nightly-test-amd-rocm720.yml: Run A 28888218691 (Jul 07, sglang 11cea29c) and Run B 28964405353 (Jul 08). Two infra clusters (zombie-KFD VRAM gate + MI325 runner starvation) account for 44/61 failures.

#	Cluster	Job (representative)	Test File	Test Function	Status	Hypothesis (confidence)
1	Zombie-KFD VRAM gate fail	deepseek-r1-mxfp4 (+27 more)	N/A (ensure_vram_clear.sh pre-test gate)	N/A	28 jobs, ≥6d persistent	[FACT] driver/KFD VRAM leak, no owning process — ROCm/aiter#2061
2	MI325 runner starvation	accuracy-8-gpu (+15 more)	N/A (job cancelled, 0 steps)	N/A	16 jobs, recurring	[FACT] mi325 pool never assigned runner in ~24h → concurrency cancel-in-progress
3	cutlass → Transformers fallback → KeyError 'sglang'	deepseek-r1-mxfp4-tp2	test/registered/amd/accuracy/mi35x/test_deepseek_r1_mxfp4_tp2_mi35x.py	setUpClass	9 jobs, Run A only — resolved	[HIGH] No module named 'cutlass' → registry fallback; fixed by merged #30374
4	cooperative_groups.h JIT compile fail (DSA topk_v2, gfx950)	accuracy-deepseek-v32	test/registered/amd/accuracy/mi35x/test_deepseek_v32_eval_mi35x.py	test_deepseek_v32_accuracy	2 jobs, new today	[HIGH] DSA fused top-k v2 not HIP-guarded → ninja compile fail in graph capture; fix #30506
5	HF Hub download failure	nightly-test-1-gpu-unit	test/registered/lora/test_lora_openai_compatible.py	TestLoRAOpenAICompatible.setUpClass	2 jobs, likely flaky	[MEDIUM] RemoteProtocolError / weight-download timeout during server startup
6	mamba extra_buffer unsupported on ROCm	nightly-8-gpu-mi35x-qwen35	test/registered/amd/accuracy/mi35x/test_qwen35_eval_mi35x.py	TestQwen35EvalMI35x.test_lm_eval	1 job	[HIGH] extra_buffer needs CUDA/MUSA/NPU (FLA); fixes #30359/#29965
7	HIP graph-capture abort on MoE-EP all-reduce	qwen3-235b-mxfp4	test/registered/amd/test_qwen3_instruct_mxfp4.py	TestQwen3Instruct2507MXFP4.setUpClass	1 job, new today	[MEDIUM] EP-group all_reduce falls to non-capturable torch.distributed.all_reduce → hipErrorStreamCaptureUnsupported
8	GPU memory access fault (DSR1 FP8 blockscale prefill)	deepseek-r1-hicache	test/registered/amd/test_deepseek_r1_hicache_mi35x.py	TestDeepSeekR1HiCacheMI35x	1 job, new today	[LOW] Memory access fault on 8 TP ranks (gfx950); candidate #29275/#24476
9	MTP acceptance-length below threshold	deepseek-v4-pro-mtp	test/registered/amd/test_deepseek_v4_pro_fp4_mtp.py	test_b_bs_1_speed	1 job, new today	[LOW] acc_length 2.46 < 2.8 all retries — threshold/perf, not a crash

Cluster: Zombie-KFD VRAM gate fail (28 jobs)

• Affected: all linux-mi35x jobs across both runs — Run A: deepseek-r1-mxfp4/hicache, grok1-int4, grok2, accuracy/perf-deepseek-v32, qwen3-235b-mxfp4, qwen35-triton-dcp, deepseek-v4-pro/flash, glm51, glm5-mxfp4, minimax-m25; Run B: same family + kv-fp8/tp4/ar-fusion/mtp variants, qwen35.
• Shared facts: ensure_vram_clear.sh exits 1 after 3 attempts — VRAM held (~15–16%) with no owning process; node needs reboot. Self-diagnosed as ROCm/aiter#2061. Pre-test gate, so no test ran.
• In-flight fix: ⚠️ #30431 "[DO NOT MERGE][AMD] ci: self-heal driver-level VRAM leaks" — draft; chase it forward, do not duplicate.
• Suggested triage: reboot/drain affected MI35x nodes; advance #30431 out of draft.

Cluster: MI325 runner starvation (16 jobs)

• Affected: all linux-mi325-2gpu/8gpu-sglang jobs in Run A (accuracy-8/2-gpu ±vlm, deepseek-v3-kv-fp8, deepseek-v31/v32/v32-mtp, grok1-int4/grok2, qwen35, qwen3-235b, kimi-k26, glm51, minimax-m27, perf-2-gpu-text/vlm).
• Shared facts: conclusion: cancelled, empty runner_name, 0 steps, BlobNotFound; queued ~24h then cancelled by next run's concurrency (cancel-in-progress on schedule). Jul 06 sister run passed after ~21h queue → pool is chronically backlogged, not code.
• In-flight fix: ❌ none — infra/fleet capacity, not a PR.
• Suggested triage: investigate linux-mi325-8gpu-sglang idle count vs queued jobs; add per-job queue timeout so starvation surfaces explicitly.

Cluster: cutlass → Transformers fallback (9 jobs, Run A only)

• Affected: Run A mi35x model-load jobs — deepseek-r1-mxfp4 (kv-fp8/tp2/tp4/ar-fusion), glm51-mxfp4, deepseek-v32-mtp (accuracy+perf, AutoModel ValueError), deepseek-v4-pro-mtp, kimi-k26.
• Shared facts: ModuleNotFoundError: No module named 'cutlass' drops native registry → Transformers backend → KeyError: 'sglang' / unrecognized-config crash (exit -9).
• In-flight fix: ✅ #30374 MERGED 2026-07-07 21:43 UTC (~3.5h after Run A started, so absent at 11cea29c). Run B mi35x model-load jobs no longer show cutlass — confirms resolution (they now hit the VRAM gate / cooperative_groups instead).
• Suggested triage: none — verify green once VRAM-gate cluster clears.

Cluster: cooperative_groups.h JIT compile fail (2 jobs, new today)

• Affected: accuracy + perf deepseek-v32 mi35x (Run B). ninja exited with status 1 → 'cooperative_groups.h' file not found during CUDA-graph capture on gfx950.
• In-flight fix: ✅ #30506 "[AMD] Disable DSA fused top-k v2 on ROCm for GLM-5.x / DeepSeek-V3.2" (open, non-draft) — chase reviewers.

Cluster: HF Hub download failure (2 jobs)

• Affected: LoRA OpenAI-compat (Run A, RemoteProtocolError), glm5-mxfp4 (Run B, weight-download never completes → GSM8K timeout). Likely transient/flaky; In-flight fix: ❌ none.
• Suggested triage: re-run; a green retry confirms an HF Hub flake.

Cluster: mamba extra_buffer / HIP MoE-EP / GPU mem fault / MTP acc (1 job each)

• mamba (qwen35): fixes #30359/#29965 open — chase reviewers.
• HIP MoE-EP all-reduce (qwen3-235b-mxfp4): first CI execution today (prior 6 runs blocked by VRAM gate, so no regression window). ❌ no direct fix; #29946 related-but-non-covering. Verify with --disable-cuda-graph.
• GPU mem fault (deepseek-r1-hicache): candidates #29584/#24476/#29275 — bpreshuffle scale-layout path. [LOW], needs repro.
• MTP acc-length (deepseek-v4-pro-mtp): 2.46 < 2.8 — perf/threshold, ❌ no fix.

Suggested triage order

Rank	Triage step	Targets	Why this first
1	Reboot/drain MI35x nodes + advance #30431 out of draft	Cluster 1 (28 jobs)	Largest cluster; masks real test signal on mi35x
2	Investigate linux-mi325-8gpu-sglang capacity; add queue timeout	Cluster 2 (16 jobs)	Second-largest; ~24h queue → concurrency cancel, no logs
3	Chase merge of #30506	Cluster 4 (rows: v32 accuracy+perf)	Fix ready (non-draft), unblocks DeepSeek-V3.2 on gfx950
4	Chase #30359/#29965	Cluster 6 (qwen35)	Fixes exist and are on-point
5	Repro MoE-EP all-reduce with --disable-cuda-graph	Cluster 7 (qwen3-235b-mxfp4)	Confirms graph-capture root cause; no fix yet

---

Job	Failed Steps	Started
nightly-test-1-gpu-unit-rocm720	Nightly Unit Test ROCm 7.2 (1-GPU)	2026-07-07T20:37
nightly-2-gpu-mi35x-glm51-mxfp4-rocm720	Accuracy Test MI35x ROCm 7.2 (2-GPU GLM-5.1-MXFP4 GSM8K)	2026-07-07T18:10
nightly-8-gpu-mi35x-deepseek-r1-mxfp4-kv-fp8-rocm720	Accuracy Test MI35x ROCm 7.2 (8-GPU DeepSeek-R1-MXFP4 KV FP8)	2026-07-07T18:10
nightly-2-gpu-mi35x-deepseek-r1-mxfp4-tp2-rocm720	Accuracy Test MI35x ROCm 7.2 (2-GPU DeepSeek-R1-MXFP4 TP2)	2026-07-07T18:10
nightly-8-gpu-mi35x-deepseek-r1-mxfp4-rocm720	Ensure VRAM is clear	2026-07-07T18:10
nightly-8-gpu-mi35x-grok2-rocm720	Ensure VRAM is clear	2026-07-07T18:10
nightly-accuracy-8-gpu-mi35x-deepseek-v32-rocm720	Ensure VRAM is clear	2026-07-07T18:12
nightly-8-gpu-mi35x-deepseek-r1-hicache-rocm720	Ensure VRAM is clear	2026-07-07T18:19
nightly-accuracy-8-gpu-mi35x-rocm720	Ensure VRAM is clear	2026-07-07T18:13
nightly-perf-8-gpu-mi35x-deepseek-v32-basic-rocm720	Ensure VRAM is clear	2026-07-07T18:28
nightly-8-gpu-mi35x-grok1-int4-rocm720	Ensure VRAM is clear	2026-07-07T18:22
nightly-perf-8-gpu-mi35x-deepseek-v32-mtp-rocm720	Performance Test MI35x ROCm 7.2 (8-GPU DeepSeek-V3.2 MTP)	2026-07-07T18:15
nightly-8-gpu-mi35x-deepseek-r1-mxfp4-ar-fusion-rocm720	Accuracy Test MI35x ROCm 7.2 (8-GPU DeepSeek-R1-MXFP4 AllReduce Fusion)	2026-07-07T18:10
nightly-8-gpu-mi35x-qwen3-235b-mxfp4-rocm720	Ensure VRAM is clear	2026-07-07T18:15
nightly-accuracy-8-gpu-mi35x-deepseek-v32-mtp-rocm720	Accuracy Test MI35x ROCm 7.2 (8-GPU DeepSeek-V3.2 TP+MTP)	2026-07-07T18:21
nightly-8-gpu-mi35x-deepseek-r1-mxfp4-tp4-rocm720	Accuracy Test MI35x ROCm 7.2 (DeepSeek-R1-MXFP4 TP4 MTP), Accuracy Test MI35x ROCm 7.2 (DeepSeek-R1-MXFP4 TP4)	2026-07-07T18:36
nightly-8-gpu-mi35x-deepseek-v4-pro-rocm720	Accuracy + Performance Test MI35x ROCm 7.2 (8-GPU DeepSeek-V4-Pro FP8 + FP4, triton), Ensure VRAM is clear	2026-07-07T18:26
nightly-8-gpu-mi35x-deepseek-v4-flash-rocm720	Accuracy + Performance Test MI35x ROCm 7.2 (8-GPU DeepSeek-V4-Flash FP8 + FP4, triton), Ensure VRAM is clear	2026-07-07T18:17
nightly-8-gpu-mi35x-glm51-rocm720	Ensure VRAM is clear	2026-07-07T18:34
nightly-8-gpu-mi35x-glm5-mxfp4-rocm720	Ensure VRAM is clear	2026-07-07T18:24
nightly-8-gpu-mi35x-qwen35-rocm720	Accuracy Test MI35x ROCm 7.2 (8-GPU Qwen 3.5)	2026-07-07T18:24
nightly-8-gpu-mi35x-kimi-k26-rocm720	Accuracy Test MI35x ROCm 7.2 (8-GPU Kimi-K2.6)	2026-07-07T18:26
nightly-8-gpu-mi35x-qwen35-triton-dcp-rocm720	Ensure VRAM is clear	2026-07-07T19:24
nightly-4-gpu-mi35x-minimax-m25-rocm720	Ensure VRAM is clear	2026-07-07T18:46
nightly-8-gpu-mi35x-deepseek-v4-pro-mtp-rocm720	Accuracy + MTP Accept Length Test MI35x ROCm 7.2 (8-GPU DeepSeek-V4-Pro FP8 MTP, triton), Accuracy + MTP Accept Length Test MI35x ROCm 7.2 (8-GPU DeepSeek-V4-Pro FP8 MTP, unified_kv_triton)	2026-07-07T18:39
nightly-accuracy-2-gpu-rocm720	N/A	2026-07-07T18:10
nightly-8-gpu-deepseek-v3-kv-fp8-rocm720	N/A	2026-07-07T18:10
nightly-perf-2-gpu-vlm-rocm720	N/A	2026-07-07T18:10
nightly-8-gpu-deepseek-v31-rocm720	N/A	2026-07-07T18:10
nightly-accuracy-8-gpu-rocm720	N/A	2026-07-07T18:10
nightly-accuracy-2-gpu-vlm-rocm720	N/A	2026-07-07T18:10
nightly-8-gpu-grok1-int4-rocm720	N/A	2026-07-07T18:10
nightly-8-gpu-deepseek-v32-rocm720	N/A	2026-07-07T18:10
nightly-8-gpu-grok2-rocm720	N/A	2026-07-07T18:10
nightly-perf-2-gpu-text-rocm720	N/A	2026-07-07T18:10
nightly-8-gpu-qwen35-rocm720	N/A	2026-07-07T18:10
nightly-8-gpu-deepseek-v32-mtp-rocm720	N/A	2026-07-07T18:10
nightly-8-gpu-kimi-k26-rocm720	N/A	2026-07-07T18:10
nightly-8-gpu-glm51-rocm720	N/A	2026-07-07T18:10
nightly-8-gpu-qwen3-235b-rocm720	N/A	2026-07-07T18:10
nightly-2-gpu-mi35x-glm51-mxfp4-rocm720	Ensure VRAM is clear	2026-07-08T17:58
nightly-8-gpu-minimax-m27-rocm720	N/A	2026-07-07T18:10
nightly-2-gpu-mi35x-deepseek-r1-mxfp4-tp2-rocm720	Ensure VRAM is clear	2026-07-08T17:58
nightly-8-gpu-mi35x-grok1-int4-rocm720	Ensure VRAM is clear	2026-07-08T18:01
nightly-accuracy-8-gpu-mi35x-rocm720	Ensure VRAM is clear	2026-07-08T17:59
nightly-perf-8-gpu-mi35x-deepseek-v32-mtp-rocm720	Ensure VRAM is clear	2026-07-08T18:06
nightly-8-gpu-mi35x-deepseek-r1-mxfp4-kv-fp8-rocm720	Ensure VRAM is clear	2026-07-08T18:14
nightly-8-gpu-mi35x-deepseek-r1-mxfp4-tp4-rocm720	Accuracy Test MI35x ROCm 7.2 (DeepSeek-R1-MXFP4 TP4 MTP), Ensure VRAM is clear	2026-07-08T18:19
nightly-8-gpu-mi35x-deepseek-r1-mxfp4-ar-fusion-rocm720	Ensure VRAM is clear	2026-07-08T18:22
nightly-accuracy-8-gpu-mi35x-deepseek-v32-mtp-rocm720	Ensure VRAM is clear	2026-07-08T18:16
nightly-4-gpu-mi35x-minimax-m25-rocm720	Ensure VRAM is clear	2026-07-08T18:42
nightly-8-gpu-mi35x-qwen35-triton-dcp-rocm720	Ensure VRAM is clear	2026-07-08T18:44
nightly-8-gpu-mi35x-qwen3-235b-mxfp4-rocm720	Accuracy Test + Performance Test MI35x ROCm 7.2 (8-GPU Qwen3-235B-MXFP4)	2026-07-08T18:22
nightly-8-gpu-mi35x-qwen35-rocm720	Ensure VRAM is clear	2026-07-08T18:49
nightly-8-gpu-mi35x-deepseek-v4-pro-rocm720	Accuracy + Performance Test MI35x ROCm 7.2 (8-GPU DeepSeek-V4-Pro FP8 + FP4, triton), Ensure VRAM is clear	2026-07-08T18:47
nightly-perf-8-gpu-mi35x-deepseek-v32-basic-rocm720	Performance Test MI35x ROCm 7.2 (8-GPU DeepSeek-V3.2 Basic)	2026-07-08T17:58
nightly-8-gpu-mi35x-glm51-rocm720	Ensure VRAM is clear	2026-07-08T18:52
nightly-accuracy-8-gpu-mi35x-deepseek-v32-rocm720	Accuracy Test MI35x ROCm 7.2 (8-GPU DeepSeek-V3.2)	2026-07-08T18:24
nightly-8-gpu-mi35x-glm5-mxfp4-rocm720	Accuracy Test MI35x ROCm 7.2 (8-GPU GLM-5-MXFP4)	2026-07-08T18:16
nightly-8-gpu-mi35x-deepseek-r1-hicache-rocm720	Accuracy Test MI35x ROCm 7.2 (8-GPU DeepSeek-R1 HiCache)	2026-07-08T18:45
nightly-8-gpu-mi35x-deepseek-v4-pro-mtp-rocm720	Accuracy + MTP Accept Length Test MI35x ROCm 7.2 (8-GPU DeepSeek-V4-Pro FP8 MTP, triton)	2026-07-08T18:51

Per-Job Analysis

nightly-test-1-gpu-unit-rocm720 — failed step(s): Nightly Unit Test ROCm 7.2 (1-GPU)

Commit Info

• sglang: 11cea29c
• aiter: 9127c94a (source: dev image preserved — log: Dev/patched version detected: v0.1.17.dev110+g9127c94a1 → skipping rebuild, matches Dockerfile default)

Failed Tests

Test File	Test Function	Error	Log
test/registered/lora/test_lora_openai_compatible.py	TestLoRAOpenAICompatible.setUpClass	Exception: Server process exited with code -9 (underlying httpx.RemoteProtocolError: Server disconnected without sending a response)	Log

The other 9 test files in this job step passed (Test Summary: 9/10 passed).

Failure Summary

The unit-test batch step ran 10 registered test files; test/registered/lora/test_lora_openai_compatible.py was the only failure. Its setUpClass could not launch the server (popen_launch_server → server exited -9). The scheduler child died during weight loading with httpx.RemoteProtocolError: Server disconnected without sending a response while fetching the safetensors index file for meta-llama/Llama-3.2-1B-Instruct from the HuggingFace Hub.

Regression Status

New failure (for this test file / this job).

Recent history of test/registered/lora/test_lora_openai_compatible.py in job nightly-test-1-gpu-unit-rocm720 (branch main, event schedule, completed runs only):

Date	Run	Job	Test File Status	Failed Function	Error
Jul 07	28888218691	nightly-test-1-gpu-unit-rocm720	❌ Failed	setUpClass	RemoteProtocolError (HF Hub)
Jul 05	28749501468	nightly-test-1-gpu-unit-rocm720	✅ Passed	—	—
Jul 04	28714556066	nightly-test-1-gpu-unit-rocm720	✅ Passed	—	—
Jul 03	28676295311	nightly-test-1-gpu-unit-rocm720	✅ Passed	—	—

(The job concluded success on Jul 3–5, so the test file passed in each. First observed failure: Jul 07 2026. Last known passing: Jul 05 2026. Jul 06 run was cancelled — excluded. Today's overall run is still queued for its remaining jobs, but this job completed.)

Failure Cluster

HF Hub network disconnect during LoRA-server startup weight download (RemoteProtocolError)

Facts (verified observations, no interpretation)

• Exception: Server process exited with code -9. Check server logs for errors. raised in popen_launch_server at test_utils.py:1029, from test_lora_openai_compatible.py:62 (setup_class → popen_launch_server).
• Scheduler child traceback ends in httpx.RemoteProtocolError: Server disconnected without sending a response, thrown from weight_utils.py:639 download_safetensors_index_file_from_hf → hf_hub_download → get_hf_file_metadata.
• The LoRA test disables offline mode: log shows CI_OFFLINE: LoRA enabled, skip offline mode - meta-llama/Llama-3.2-1B-Instruct and CI_OFFLINE: Launching server HF_HUB_OFFLINE=0. Non-LoRA tests in the same job ran with HF_HUB_OFFLINE=1 and passed.
• Model weights themselves were already cached locally: Found local HF snapshot for meta-llama/Llama-3.2-1B-Instruct ... skipping download — the network call that failed was the safetensors index-file metadata HEAD to huggingface.co, not the weight blobs.
• Same test file + job passed in the previous 3 completed scheduled runs (Jul 3/4/5).
• Not a scout run (nightly-test-amd-rocm720.yml, AITER_COMMIT_OVERRIDE empty) — no aiter A/B attribution applies.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [HIGH] Transient HuggingFace Hub connectivity failure on the MI30x runner (infrastructure, not a code regression).
  • Supporting: the terminal exception is a network-layer disconnect (RemoteProtocolError: Server disconnected without sending a response) inside an httpx request to huggingface.co; it occurred only in the one test that forces HF_HUB_OFFLINE=0; the same test passed the prior 3 days on nearby commits; weights were already local so only a metadata HEAD hit the network.
  • Disconfirming: none found — no code in the weight-download path changed in a way visible from the trace; the error is squarely in the HTTP client, not sglang logic.
  • How to verify: re-run the job; if green, confirmed flake. If it recurs, check runner egress to huggingface.co and whether the LoRA test can use offline mode / a locally-cached index.
• [LOW] A code change in the offline-mode gating causing the LoRA path to unnecessarily reach the network.
  • Supporting: only the offline-disabled test failed.
  • Disconfirming: this LoRA test has always disabled offline mode (by design, to allow HF adapter fetch) and passed on Jul 3–5 with the same behavior, so gating is not new.
  • How to verify: git log --oneline on python/sglang/srt/model_loader/weight_utils.py and the CI_OFFLINE helper between Jul 5 and Jul 7; confirm no relevant change.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for lora openai offline (2 unrelated PRs: #29468, #28315) and HF_HUB_OFFLINE LoRA (0 results). No PR targets this HF-Hub-download flake.

Suggested Triage Steps

• Re-run nightly-test-1-gpu-unit-rocm720 for run 28888218691; a green re-run confirms a transient HF Hub flake.
• If it recurs, check MI30x runner egress/DNS to huggingface.co and any HF Hub rate-limiting during the run window (~21:02 UTC Jul 7).
• Consider making test_lora_openai_compatible.py resilient to the metadata fetch: pre-cache the safetensors index for meta-llama/Llama-3.2-1B-Instruct (and the LoRA adapter codelion/Llama-3.2-1B-Instruct-tool-calling-lora) so the LoRA path can run offline, matching the other tests in this job.

Status (factual, no priority assignment)

• Persistence: 1 completed run (first seen 2026-07-07 ~21:02 UTC); passed the previous 3 scheduled runs (Jul 3–5).
• Scope: 1 test file in 1 job (nightly-test-1-gpu-unit-rocm720); other 9 test files in the same step passed. Overall run 28888218691 still queued for remaining jobs at snapshot time.
• Blocked work: LoRA OpenAI-compatible endpoint coverage on AMD ROCm 7.2 nightly signal for this run.
• In-flight fix: ❌ none found — needs re-run to confirm flake, or a new PR to pre-cache the index for offline LoRA testing.

nightly-2-gpu-mi35x-glm51-mxfp4-rocm720 — failed step(s): Accuracy Test MI35x ROCm 7.2 (2-GPU GLM-5.1-MXFP4 GSM8K)

Commit Info

• sglang: 11cea29c
• aiter: v0.1.17.dev110+g9127c94a1 (source: dev image preserved — [CI-AITER-CHECK] reports Dockerfile default AITER_COMMIT=9127c94a18e4398e1eba91f6639e910f0994ad02, dev/patched version detected → rebuild skipped)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi35x/test_deepseek_r1_mxfp4_tp2_mi35x.py	TestDeepSeekR1MXFP4TP2MI35x.setUpClass	Exception: Server process exited with code -9 (root: KeyError: 'sglang' in scheduler)	Log

Failure Summary

The DeepSeek-R1-MXFP4-TP2 accuracy test failed in setUpClass because the server process died during startup (exit code -9). The scheduler crashed while loading the model: amd/DeepSeek-R1-MXFP4-Preview (a trust_remote_code model) was routed to the Transformers fallback backend because the native DeepseekV3ForCausalLM module failed to import (No module named 'cutlass'), and the fallback then raised KeyError: 'sglang'. Ran 0 tests in 76.214s / FAILED (errors=1).

Note: the failing CI step is labelled "Accuracy Test MI35x ROCm 7.2 (2-GPU GLM-5.1-MXFP4 GSM8K)", but the log (line 3844) shows it actually executed test_deepseek_r1_mxfp4_tp2_mi35x.py loading amd/DeepSeek-R1-MXFP4-Preview. Step-name vs executed-test mismatch — worth a maintainer glance, but does not affect the root cause below. The near-identical sister job nightly-2-gpu-mi35x-deepseek-r1-mxfp4-tp2-rocm720 (85693869924) also failed in this run.

Regression Status

New regression (first failing run 2026-07-07). This same test file passed in the 3 prior completed scheduled runs.

Recent history of test_deepseek_r1_mxfp4_tp2_mi35x.py (TestDeepSeekR1MXFP4TP2MI35x) on nightly-test-amd-rocm720.yml (branch main, event=schedule, completed runs only; tracked via the job that runs this file):

Date	Run	Job	Test File Status	Failed Function	Error
Jul 7	28888218691	glm51-mxfp4	❌ Failed	setUpClass	KeyError: 'sglang' → server -9
Jul 6	28813124510	—	⏭️ Cancelled (excluded)	—	—
Jul 5	28749501468	deepseek-r1-mxfp4-tp2	✅ Passed	—	—
Jul 4	28714556066	deepseek-r1-mxfp4-tp2	✅ Passed (1/1)	—	—
Jul 3	28676295311	deepseek-r1-mxfp4-tp2	✅ Passed (1/1)	—	—

First observed failure: 2026-07-07. Last known passing: 2026-07-05. This is broad — nearly every mi35x rocm720 job in the Jul 7 run failed (only nightly-test-1-gpu-mi35x passed), consistent with a shared model-import regression.

Failure Cluster

No module named 'cutlass' → Transformers-backend fallback → KeyError: 'sglang' at model load

Facts (verified observations)

• Direct log evidence (verbatim): KeyError: 'sglang' at modeling_deepseek.py:1148 → self.self_attn = ATTENTION_CLASSES[config._attn_implementation] (line 4022-4023); preceded by DeepseekV3ForCausalLM has no SGLang implementation, falling back to Transformers implementation (line 3973) and Using Transformers backend (line 3982).
• Registry import failures: Ignore import error when loading sglang.srt.models.deepseek_v2: No module named 'cutlass' (line 4085) — plus glm4_moe, kimi_linear, longcat_flash, mistral_large_3, and many others in the same run.
• Final: Exception: Server process exited with code -9 at test_utils.py:1015 popen_launch_server (line 4209); Test Summary: 0/1 passed.
• A/B contrast (same test file, same branch/event): prior passing jobs 85153561702 (Jul 4) and 85050304365 (Jul 3) had 0 No module named 'cutlass', 0 Transformers fallbacks, 0 KeyError: 'sglang', test PASSED. The current run is flooded with all three. So cutlass was importable in the image on Jul 3-5 and is not on Jul 7.
• Confirmed top-level (module-scope) import chain in current source: deepseek_v2.py:61 → dsa/dsa_indexer.py:11 → jit_kernel/dsa/__init__.py:1 → jit_kernel/dsa/cutedsl_paged_mqa_logits.py:15 (import cutlass, unguarded). cutlass = the NVIDIA-only nvidia-cutlass-dsl package, not present on the AMD ROCm mi35x (gfx950) image.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [HIGH] Commit fefc1743 (#25220 "Cute-DSL FP8 MQA logits", merged 2026-07-07 06:34 UTC) introduced an unguarded top-level import cutlass reachable from the DeepSeek model-load path.
  • Supporting: (a) it lands inside the regression window 48ba79c1..11cea29c, immediately before the failing 18:10 UTC run; (b) it created jit_kernel/dsa/cutedsl_paged_mqa_logits.py with bare import cutlass and added the top-level from sglang.jit_kernel.dsa import (...) to dsa_indexer.py, completing the chain deepseek_v2 → dsa_indexer → jit_kernel/dsa/__init__ → cutedsl_paged_mqa_logits; (c) the error message is exactly No module named 'cutlass' for deepseek_v2 and every other model that top-imports the dsa indexer.
  • Disconfirming: none material. (The crash's final KeyError: 'sglang' is a secondary effect of the Transformers-backend fallback at transformers.py:614 setting config._attn_implementation = "sglang", which the model's custom remote code does not accept — this fallback+remote-code interaction is a pre-existing sharp edge, but it is only reached because the native import now fails.)
  • How to verify: on the mi35x rocm720 container, python -c "import sglang.srt.models.deepseek_v2" — expect ModuleNotFoundError: No module named 'cutlass'. Then guard the top-level import (try/except or is_sm100_supported() gate) in jit_kernel/dsa/__init__.py / cutedsl_paged_mqa_logits.py and re-run test_deepseek_r1_mxfp4_tp2_mi35x.py.
• [LOW] CI image dependency change (rocm720 image dropped nvidia-cutlass-dsl).
  • Supporting: cutlass was importable Jul 3-5 and is not Jul 7.
  • Disconfirming: cutlass is an NVIDIA package that would not normally be installed on a ROCm image at all; the far simpler explanation is that the import path was previously lazy/guarded and #25220 made it unconditional at module scope. No image-bump commit was identified in the window.
  • How to verify: pip show nvidia-cutlass-dsl inside the Jul 7 vs a Jul 4 image; diff the Dockerfile/requirements between the two image builds.

In-flight Fix Check

• In-flight fix: ⚠️ partial — #30413 "Fix additional kernels being triggered in CuteDSL MQA logits path" (open, 2026-07-07) is a follow-up to #25220, but it fixes a functional bug (extra kernels dropped) and touches dsa_indexer.py / paged_mqa_logits.py — it does not add a guard to jit_kernel/dsa/__init__.py or cutedsl_paged_mqa_logits.py, so it likely does not resolve the AMD No module named 'cutlass' fallback.
• In-flight fix (import guard): ❌ none found — no open PR guards the top-level import cutlass for non-NVIDIA platforms (searched cutlass+import+dsa, cutedsl+paged+mqa, import+cutlass+guard).

Suggested Triage Steps

• Confirm the import break directly: run python -c "import sglang.srt.models.deepseek_v2" in the mi35x rocm720 CI container; expect ModuleNotFoundError: No module named 'cutlass'.
• If confirmed, gate the CuteDSL imports behind a platform/SM check (e.g. is_sm100_supported()) or a try/except ImportError in python/sglang/jit_kernel/dsa/__init__.py and cutedsl_paged_mqa_logits.py, so AMD/ROCm keeps the native DeepSeek path instead of falling back to Transformers.
• Verify whether #30413 incidentally restores a lazy import; if not, a new guard PR is needed.
• Secondary hardening: make the Transformers fallback fail loudly (or skip) when the remote-code model's ATTENTION_CLASSES lacks "sglang", instead of a bare KeyError (transformers.py:614).
• Because the same import chain hits glm/kimi/longcat/mistral models across nearly all mi35x rocm720 jobs, one guard fix should clear the whole cluster — re-run the workflow after landing it.

Status

• Persistence: First failing run 2026-07-07 (1 completed run); passed 2026-07-05 and earlier. New today.
• Scope: Broad — nearly all mi35x rocm720 nightly jobs in run 28888218691 failed with the same cutlass/fallback signature (deepseek-r1-mxfp4 variants, glm5/glm51, qwen3-235b/qwen35, grok, kimi-k26, minimax-m25, deepseek-v4). Only nightly-test-1-gpu-mi35x passed.
• Blocked work: DeepSeek/GLM/Qwen/Kimi MXFP4 accuracy & perf signal on AMD MI35x ROCm 7.2.
• In-flight fix: ⚠️ #30413 is a related #25220 follow-up but likely does not fix the import guard; no dedicated guard PR found.

nightly-8-gpu-mi35x-deepseek-r1-mxfp4-kv-fp8-rocm720 — failed step(s): Accuracy Test MI35x ROCm 7.2 (8-GPU DeepSeek-R1-MXFP4 KV FP8)

Commit Info

• sglang: 11cea29c
• aiter: 9127c94a — v0.1.17.dev110+g9127c94a1 (source: dev/patched image build preserved; [CI-AITER-CHECK] Dev/patched version detected → skipping rebuild)
• CI image: rocm/sgl-dev:v0.5.14-rocm720-mi35x-20260707

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi35x/test_deepseek_r1_mxfp4_kv_fp8_eval_mi35x.py	test_deepseek_r1_mxfp4_kv_fp8_accuracy	AssertionError: Failed models: ['amd/DeepSeek-R1-MXFP4-Preview (kv-fp8)'] (server never started: KeyError: 'sglang' during model load)	Log

Failure Summary

The single test test_deepseek_r1_mxfp4_kv_fp8_accuracy failed because the SGLang server crashed at model-load time before serving any request. All 8 TP ranks reported No module named 'cutlass' when importing sglang.srt.models.deepseek_v2, so the DeepseekV3ForCausalLM architecture was not registered and the loader fell back to the generic Transformers backend, which then raised KeyError: 'sglang'. The test asserts on the empty result set and fails; Test Summary: 0/1 passed, process exit 255.

Regression Status

New regression (first observed today 2026-07-07). This exact test passed 2026-07-05 with accuracy=0.950 threshold=0.93 ✅ loading the model natively.

Recent history of test_deepseek_r1_mxfp4_kv_fp8_eval_mi35x.py (test_deepseek_r1_mxfp4_kv_fp8_accuracy) in job nightly-8-gpu-mi35x-deepseek-r1-mxfp4-kv-fp8-rocm720 (branch main, event schedule, completed runs only):

Date	Run	Job	Test File Status	Failed Function	Error
Jul 07	28888218691	job	❌ Failed	test_deepseek_r1_mxfp4_kv_fp8_accuracy	KeyError: 'sglang' → AssertionError: Failed models
Jul 05	28749501468	job	✅ Passed	—	accuracy=0.950
Jul 04	28714556066	job	⚠️ N/A (infra)	—	VRAM cleanup unsuccessful after 3 attempts — test never ran (different failure)
Jul 03	28676295311	job	✅ Passed	—	—

Last known passing run: Jul 05 (28749501468, head 48ba79c1). Jul 06 run was cancelled (excluded). Jul 04 was an unrelated infra VRAM-cleanup failure, not this test.

Failure Cluster

cutlass ImportError breaks native DeepSeek load on AMD → Transformers-backend fallback KeyError 'sglang'

Facts (verified observations, no interpretation)

• Today's log: No module named 'cutlass' appears 480× (Ignore import error when loading sglang.srt.models.deepseek_v2/deepseek_v4/deepseek_v4_nextn/deepseek_vl2). Jul 05 passing log: 0×.
• Today's log: DeepseekV3ForCausalLM has no SGLang implementation, falling back to Transformers implementation appears 60×. Jul 05: 0× — Jul 05 loaded natively: Load weight end … type=DeepseekV3ForCausalLM, quant=quark.
• Verified stack trace (today), scheduler crash:
  • python/sglang/srt/model_loader/loader.py:320 _initialize_model → model_class(**kwargs)
  • python/sglang/srt/models/transformers.py:617 AutoModel.from_config(...), immediately preceded at transformers.py:614 by self.text_config._attn_implementation = "sglang"
  • trust-remote-code …/DeepSeek-R1-MXFP4-Preview/…/modeling_deepseek.py:1148 self.self_attn = ATTENTION_CLASSES[config._attn_implementation]( → KeyError: 'sglang'
• On AMD/ROCm, import cutlass (NVIDIA CUTLASS Python DSL) is not an installable/valid package; e.g. python/sglang/srt/layers/quantization/nvfp4_gemm_swiglu_nvfp4_quant.py:23 and layers/attention/linear/kernels/kda_blackwell/*.py do unguarded top-level import cutlass.
• Same AITER dev version in both passing (Jul 05) and failing (Jul 07) runs (v0.1.17.dev110+g9127c94a1). Image tag changed: …mi35x-20260705 (pass) → …mi35x-20260707 (fail).
• 67 sglang commits in the regression window 48ba79c1..11cea29c.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [MEDIUM] A top-level (unguarded) import cutlass entered deepseek_v2's module-level import chain during the window, breaking native DeepSeek registration on AMD (where cutlass DSL is unavailable), which forces the Transformers-backend fallback and the KeyError: 'sglang'.
  • Supporting: Jul 05 imported deepseek_v2 cleanly (0 cutlass errors, native load); today it fails with 480 cutlass ImportErrors — the import chain behavior changed, not the model or aiter. The proximate KeyError chain is FACT-verified end to end.
  • Disconfirming: I inspected deepseek_v2's direct top-level quant imports (fp8, fp8_kernel, mxfp4_flashinfer_trtllm_moe, unquant) — none contain an unguarded top-level import cutlass; the files that do (nvfp4_gemm_swiglu_nvfp4_quant.py, kda_blackwell/*) are imported only lazily (function-scope) by deepseek_v2. So I could not pin the exact culprit import via static reading. (Local git is a shallow clone — git log -- <file> is unreliable for bisecting, so the window could not be narrowed by file history.)
  • How to verify: in the …20260707 container at HEAD 11cea29c, run python -c "import sglang.srt.models.deepseek_v2" and read the full traceback to find the exact module doing import cutlass; then git log 48ba79c1..11cea29c -- <that_file> on a full clone to find the introducing commit.
• [LOW] Environment/image regression: the …20260707 daily image dropped a cutlass (nvidia-cutlass-dsl) package that …20260705 happened to ship, exposing a previously-satisfied import.
  • Supporting: only variable that provably changed alongside the code is the image tag (20260705→20260707).
  • Disconfirming: cutlass DSL is NVIDIA-only and would not normally be installed in a ROCm image; if the import had been in the top-level chain on Jul 05 with the package absent, Jul 05 would have shown the same errors — it did not. This points to the chain itself changing (code), not the package.
  • How to verify: diff pip list between the two image tags for nvidia-cutlass-dsl/cutlass.
• [SPECULATION] aiter change — ruled out: identical aiter dev build in passing and failing runs.

In-flight Fix Check

• In-flight fix: ❌ none found targeting this specific failure. Searched open PRs: attn_implementation+sglang (0 results); No+module+named+cutlass (2 results — #15755, #22455 — both unrelated); cutlass+import+deepseek (13 results, none address the AMD Transformers-fallback KeyError: 'sglang'; closest but unrelated: #30426 "[Tiny] Fix Import Error for Pure TP config with flashinfer_mxfp4").

Suggested Triage Steps

• Reproduce the import failure directly: in container rocm/sgl-dev:v0.5.14-rocm720-mi35x-20260707 at 11cea29c, run python -c "import sglang.srt.models.deepseek_v2" and capture the traceback to identify the exact module performing the top-level import cutlass.
• Bisect the window 48ba79c1..11cea29c on a full clone, testing only the one-liner import above (fast, no GPU needed) to find the introducing commit.
• If confirmed as a code regression, the fix direction is to guard the offending top-level import cutlass behind a try/except ImportError or a CUDA/SM100 capability check so ROCm can import deepseek_v2 natively (mirroring how nvfp4_gemm_swiglu_nvfp4_quant is imported lazily elsewhere).
• If the import one-liner instead passes on …20260705 and fails on …20260707 at the same SHA, treat it as an image regression and diff the two images' pip manifests for the cutlass DSL package.
• Cross-check scope: the other nightly-8-gpu-mi35x-deepseek-r1-mxfp4-* jobs and any AMD DeepSeek job in today's run — if they also show the 480× cutlass ImportError, this breaks all native DeepSeek loading on AMD, not just kv-fp8.

Status (factual, no priority assignment)

• Persistence: First failing completed run is today 2026-07-07 (regression since last pass 2026-07-05); Jul 04 was a separate infra failure, Jul 06 cancelled.
• Scope: At minimum this job. Likely broader — the No module named 'cutlass' → Transformers fallback pattern would affect every native DeepSeek load on this AMD image; verify against sibling deepseek-r1-mxfp4-* jobs in the same run.
• Blocked work: DeepSeek-R1-MXFP4 KV-FP8 accuracy signal on MI35x / ROCm 7.2.
• In-flight fix: ❌ none found in open PRs — needs new PR or investigation.

nightly-2-gpu-mi35x-deepseek-r1-mxfp4-tp2-rocm720 — failed step(s): Accuracy Test MI35x ROCm 7.2 (2-GPU DeepSeek-R1-MXFP4 TP2)

Commit Info

• sglang: 11cea29c
• aiter: 9127c94a — v0.1.17.dev110+g9127c94a1 (source: dev image detected, rebuild skipped; AITER_COMMIT_OVERRIDE empty)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi35x/test_deepseek_r1_mxfp4_tp2_mi35x.py	TestDeepSeekR1MXFP4TP2MI35x.setUpClass	Exception: Server process exited with code -9 (root: KeyError: 'sglang' → ModuleNotFoundError: No module named 'cutlass')	Log

Failure Summary

The server for amd/DeepSeek-R1-MXFP4-Preview crashed during startup (exit code -9), so setUpClass raised and 0 tests ran. On ROCm, sglang.srt.models.deepseek_v2 failed to import with No module named 'cutlass' (log line 3915/4085), so DeepseekV3ForCausalLM was never registered and SGLang fell back to the Transformers backend, which then hit KeyError: 'sglang' in the model's remote-code modeling_deepseek.py (log line 4196).

Regression Status

New regression (first failing 2026-07-07; passed the prior three completed scheduled runs).

Recent history of test_deepseek_r1_mxfp4_tp2_mi35x.py (setUpClass) in job nightly-2-gpu-mi35x-deepseek-r1-mxfp4-tp2-rocm720 (branch=main, event=schedule, completed runs only):

Date	Run	Job	Test File Status	Failed Function	Error
Jul 07	28888218691	job	❌ Failed	setUpClass	exit -9 / KeyError 'sglang'
Jul 06	28813124510	—	⚪ Cancelled	—	run cancelled (excluded)
Jul 05	28749501468	job	✅ Passed	—	—
Jul 04	28714556066	job	✅ Passed	—	—
Jul 03	28676295311	job	✅ Passed	—	—

Last passing SHA (Jul 05): 48ba79c1; first failing SHA (Jul 07): 11cea29c. Regression window 48ba79c11e..11cea29c = 67 commits.

Failure Cluster

ROCm cutlass ImportError → Transformers-backend fallback → KeyError 'sglang'

Facts (verified observations)

• Log line 3915: Ignore import error when loading sglang.srt.models.deepseek_v2: No module named 'cutlass'. 120 such lines across ~30 MoE models in this run; 0 in the passing Jul 05 run (log diff verified). [FACT]
• Log line ~4151: DeepseekV3ForCausalLM has no SGLang implementation, falling back to Transformers implementation. [FACT]
• Log line 4157–4196: Scheduler hit an exception … ATTENTION_CLASSES[config._attn_implementation] … KeyError: 'sglang' in remote-code modeling_deepseek.py:1148. [FACT]
• Log line 4205–4209: setUpClass … Exception: Server process exited with code -9; Ran 0 tests … FAILED (errors=1). [FACT]
• Code path: model_loader/utils.py:216-220 falls back to resolve_transformers_arch when is_native_supported is False (arch not in registry because the model module import was swallowed). transformers.py:613 sets _attn_implementation = "sglang"; transformers.py:327 registers "sglang" only into transformers' global ALL_ATTENTION_FUNCTIONS, which the remote-code module's local ATTENTION_CLASSES dict does not consult. [FACT]
• At 11cea29c, python/sglang/jit_kernel/dsa/__init__.py:1 is an unguarded top-level from .cutedsl_paged_mqa_logits import ...; that module does top-level import cutlass (NVIDIA-only). deepseek_v2.py reaches it via dsa_indexer.py:11 (from sglang.jit_kernel.dsa import ...). [FACT]

Hypothesised Causes

• [HIGH] Commit fefc1743 "Cute-DSL FP8 MQA logits" (#25220) added jit_kernel/dsa/__init__.py with an eager, unguarded import cutlass chain, imported transitively by every DSA/MoE model module (deepseek_v2, glm4_moe, kimi_*, longcat, etc.).
  • Supporting: fefc1743 is in the regression window and its __init__.py (verified via git show fefc1743:…) contains the unguarded import; cutlass is unavailable on ROCm; 30 models simultaneously start failing to import with exactly No module named 'cutlass'; Jul 05 (before this commit) had 0 such errors.
  • Disconfirming: none found — the subsequent fix targets precisely this file/path.
  • How to verify: already confirmed — the follow-up fix 40a68521 (#30374) wraps the import in if not is_hip():.

In-flight Fix Check

• In-flight fix: ✅ MERGED 40a68521 "[AMD] Fix DeepSeekV4 server cutlass error" (#30374), merged 2026-07-07 21:43 UTC — ~3.5 h after this run started (18:10 UTC), so it was not present at 11cea29c. Adds the if not is_hip() guard in jit_kernel/dsa/__init__.py and dsa_indexer.py. Search: curl .../search/issues?q=…+cutlass+deepseek.

Suggested Triage Steps

• Confirm the next scheduled nightly-test-amd-rocm720.yml run (head SHA ≥ 40a68521) shows deepseek_v2 importing cleanly and test_deepseek_r1_mxfp4_tp2_mi35x.py passing — expected green now that #30374 is merged.
• If it still fails, grep the run log for No module named 'cutlass' to check for any other unguarded top-level import cutlass reachable from model modules (candidates: layers/attention/cute_utils/, layers/attention/linear/kernels/{gdn,kda}_blackwell/).
• No aiter action needed — this is a pre-existing sglang ROCm import bug, independent of the aiter dev build.

Status

• Persistence: New regression, first (and only) completed failure on 2026-07-07; passed Jul 03–05.
• Scope: Broad on ROCm — the same cutlass ImportError disables ~30 MoE model modules; this job is the observed accuracy-test casualty. Not applicable to NVIDIA (cutlass present).
• Blocked work: DeepSeek-R1 MXFP4 TP2 accuracy signal on MI35x/ROCm 7.2.
• In-flight fix: ✅ merged #30374 — should self-resolve on the next scheduled run.

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-07-08 20:50 UTC)

[amd-bot]

nightly-test-amd-rocm720.yml — Per-Job Analysis (continued 2/10)

nightly-8-gpu-mi35x-deepseek-r1-mxfp4-rocm720 — failed step(s): Ensure VRAM is clear

Commit Info

• sglang: 11cea29c
• aiter: N/A — the job died in the pre-test host VRAM check; the container was never started and dependency install never ran, so no [CI-AITER-CHECK] markers exist. AITER_COMMIT_OVERRIDE is empty (regular nightly, not a scout run).

Failed Tests

Test File	Test Function	Error	Log
N/A (no test ran)	N/A	Host pre-check scripts/ci/amd/ensure_vram_clear.sh rocm exited 1: VRAM cleanup unsuccessful after 3 attempts	Log

This is not a test failure. The job failed on the runner host before the SGLang container launched, so zero DeepSeek-R1 MXFP4 tests executed.

Failure Summary

The job nightly-8-gpu-mi35x-deepseek-r1-mxfp4-rocm720 failed at the ensure_vram_clear.sh step. GPUs 0–3 each reported 16% VRAM still allocated, above the 5% threshold, and three cleanup attempts (SIGTERM/SIGKILL of GPU-holding processes + 30 s waits) failed to reclaim it. No host process, docker container, or KFD PID was holding the GPUs (No KFD PIDs currently running, No docker containers found), so the allocation is orphaned at the driver level.

Regression Status

Recurring infrastructure failure (host-side, runner-specific — not a code regression).

Recent history of job nightly-8-gpu-mi35x-deepseek-r1-mxfp4-rocm720 (nightly-test-amd-rocm720.yml, main, event=schedule, completed runs only):

Date	Run	Job	Test File Status	Failed Function	Error
Jul 07	28888218691	job	❌ Failed (infra)	— (no test ran)	VRAM cleanup unsuccessful ×3
Jul 06	28813124510	—	⚪ Run cancelled	—	not comparable
Jul 05	28749501468	job	✅ Passed	—	—
Jul 04	28714556066	job	❌ Failed (infra)	— (no test ran)	VRAM cleanup unsuccessful ×3
Jul 03	28676295311	job	❌ Failed (infra)	— (no test ran)	VRAM cleanup unsuccessful ×3
Jul 02	28611268103	job	✅ Passed	—	—

The Jul 04 and Jul 03 failures were verified (log tail) to be the identical VRAM cleanup unsuccessful after 3 attempts pre-check failure — same cluster, not a different test. Pattern is intermittent (fails Jul 03–04, passes Jul 05, fails Jul 07), consistent with a host that accumulates orphaned VRAM until rebooted.

Failure Cluster

Stuck VRAM / zombie KFD context on AMD mi35x runner (pre-test ensure_vram_clear.sh gate).

Facts (verified observations, no interpretation)

• Exit: ##[error]Process completed with exit code 1 at step 3 (ensure_vram_clear.sh rocm), line 545.
• Runner: linux-mi35x-gpu-8-zf775-runner-lflrd (same physical node across the recurring failures).
• VRAM state: GPU[0]–GPU[3] = 16% allocated at all three checkpoints; GPU[4]=0%, GPU[5–7]=1%. Values unchanged across all 3 cleanup attempts (18:10:57 → 18:11:32 → 18:12:08 → 18:12:43), i.e. cleanup had zero effect.
• No holders: No KFD PIDs currently running, No docker containers found on host, fuser/zombie-process diagnostics empty. lsof not installed on the runner.
• The script itself prints the diagnosis at line 542: "almost certainly a zombie KFD context (see [Bug] Custom all-reduce IPC buffers use fixed VA, conflict with torch_memory_saver pause/resume → zombie processes on MI300X ROCm/aiter#2061). The node will need to be rebooted."
• No SGLang code path was exercised — failure is entirely host-side, upstream of the container.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [HIGH] Orphaned/zombie KFD (kernel fusion driver) context on the mi35x runner node is holding ~16% VRAM on GPUs 0–3 that no live process owns, so the cleanup script cannot reclaim it.
  • Supporting: VRAM stays fixed at 16% across 3 SIGKILL+wait cycles; no KFD PID, docker container, or fuser holder present; the script's own hint and [Bug] Custom all-reduce IPC buffers use fixed VA, conflict with torch_memory_saver pause/resume → zombie processes on MI300X ROCm/aiter#2061 describe exactly this signature; the same runner shows the same failure on Jul 03 and Jul 04.
  • Disconfirming: (none found) — every fact points to a host driver-level leak, not sglang code. The 11cea29c HEAD commit is a diffusion/Cache-DiT doc change unrelated to GPU memory and was never executed.
  • How to verify: reboot node linux-mi35x-gpu-8-zf775-runner-lflrd, then re-run the job; VRAM should read 0% at the pre-check and the job should proceed to the tests.
• [LOW] A prior job on the same node leaked the context (crash / OOM-kill that left a KFD queue mapped).
  • Supporting: intermittent pattern (passes when the node happens to be clean) fits a per-node leak accumulated by an earlier job.
  • Disconfirming: no cross-job evidence gathered here identifying the leaking predecessor job.
  • How to verify: inspect the previous job scheduled on this runner before 18:10 UTC for a crash/OOM signature.

In-flight Fix Check

• In-flight fix: ❌ none found — is:pr is:open ensure_vram_clear returns 0 results; a VRAM mi35x search returns only unrelated AMD feature PRs (e.g. #29328, #28534). No open PR addresses the zombie-KFD cleanup. This is an infra/ops issue rather than a code fix; the tracking reference is [Bug] Custom all-reduce IPC buffers use fixed VA, conflict with torch_memory_saver pause/resume → zombie processes on MI300X ROCm/aiter#2061.

Suggested Triage Steps

• Reboot / drain runner linux-mi35x-gpu-8-zf775-runner-lflrd to clear the orphaned KFD context, then re-run 28888218691 — this is the action the script's own hint prescribes.
• Check the job scheduled on this node immediately before 18:10 UTC Jul 07 for a crash/OOM that could leave a zombie KFD queue; if a specific test consistently precedes these leaks, that test is the candidate leak source.
• Consider adding an automated node-reboot / cordon on ensure_vram_clear.sh failure so a single stuck node stops silently failing nightly jobs across days (ops decision).
• Track against [Bug] Custom all-reduce IPC buffers use fixed VA, conflict with torch_memory_saver pause/resume → zombie processes on MI300X ROCm/aiter#2061; if a driver/runtime fix lands, confirm it is deployed to the mi35x fleet.

Status (factual, no priority assignment)

• Persistence: Intermittent on this job — failed Jul 03, Jul 04, and Jul 07 (all identical VRAM-clear infra failures); passed Jul 02 and Jul 05; Jul 06 run cancelled. Not every completed run, but recurring on the same runner.
• Scope: Host-level, runner linux-mi35x-gpu-8-zf775-runner-lflrd. Any job dispatched to this node while VRAM is stuck will fail at the same pre-check regardless of code under test. GPUs 0–3 affected.
• Blocked work: Blocks the DeepSeek-R1 MXFP4 (8-GPU) nightly signal on AMD ROCm 7.2 whenever this node is in the leaked state; no test coverage produced.
• In-flight fix: ❌ none found — needs an ops reboot of the runner; tracked upstream as [Bug] Custom all-reduce IPC buffers use fixed VA, conflict with torch_memory_saver pause/resume → zombie processes on MI300X ROCm/aiter#2061.

nightly-8-gpu-mi35x-grok2-rocm720 — failed step(s): Ensure VRAM is clear

Commit Info

• sglang: 11cea29c ([diffusion][cache-dit] add dual-transformer Cache-DiT adapter specs #30150)
• aiter: N/A — the job failed in the pre-test ensure_vram_clear.sh step before any container was launched, so no [CI-AITER-CHECK] markers were emitted.

Failed Tests

Test File	Test Function	Error	Log
N/A (pre-test infra step)	N/A	VRAM cleanup unsuccessful after 3 attempts → exit code 1	Log

No test ran. The failure is entirely in the runner-setup step bash scripts/ci/amd/ensure_vram_clear.sh rocm, which runs before the SGLang container starts.

Failure Summary

The job aborted during the pre-test VRAM-clear guard on runner linux-mi35x-gpu-8-zf775-runner-fxcn4. On job start, all 8 GPUs already reported 75–91% VRAM allocated (threshold is 5%), held by orphaned KFD processes shown as UNKNOWN (e.g. PIDs 875234/875252/… each holding ~232 GB on GPU 1). Three cleanup attempts (SIGTERM → SIGKILL → 30 s wait) each failed to free the memory, and the step exited 1. Neither the grok2 model tests nor any SGLang process from this run executed.

Regression Status

Infrastructure issue (recurring, node-specific — not a code regression).

Recent history of the VRAM-clear pre-test step in job nightly-8-gpu-mi35x-grok2-rocm720 (same branch main, same event schedule, completed runs only):

Date	Run	Job	Step Status	Failure	Error
Jul 7	28888218691	job	❌ Failed	VRAM guard	VRAM cleanup unsuccessful after 3 attempts
Jul 5	28749501468	job	✅ Passed	—	—
Jul 4	28714556066	job	❌ Failed	VRAM guard	VRAM cleanup unsuccessful after 3 attempts (identical)
Jul 3	28676295311	job	✅ Passed	—	—

Intermittent: fails Jul 4 & Jul 7, passes Jul 3 & Jul 5. Correlates with runner state (leaked VRAM from a prior tenant), not with the sglang commit under test.

Failure Cluster

Pre-test VRAM guard abort — zombie KFD context holding GPU memory on shared AMD runner.

Facts (verified observations, no interpretation)

• Step ensure_vram_clear.sh rocm reported ERROR: VRAM usage exceeds threshold (5%) on entry: GPU0 80%, GPU1 89%, GPU2 91%, GPU3 91%, GPU4–7 75% (log lines 167–175).
• rocm-smi --showpids lists multiple UNKNOWN KFD processes each holding ~2.3e11 bytes (~232 GB) on GPU 1, e.g. PID 875234 = 232594653184 bytes (log line 286).
• No docker containers found on host and no host processes were killable — VRAM stayed pinned across all 3 SIGTERM/SIGKILL + 30 s cleanup cycles (lines 217, 270, 409, 548).
• Step's own diagnostic hint: "almost certainly a zombie KFD context (see [Bug] Custom all-reduce IPC buffers use fixed VA, conflict with torch_memory_saver pause/resume → zombie processes on MI300X ROCm/aiter#2061). The node will need to be rebooted" (lines 634–637).
• Exit code 1 at line 639. No test framework (pytest/unittest) output appears anywhere in the log.
• The Jul 4 failing job (85153561774) shows the byte-identical VRAM cleanup unsuccessful after 3 attempts signature.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [FACT] The job failed because the runner started with GPUs already 75–91% full and the cleanup script could not free them (verbatim in log).
• [HIGH] The leaked VRAM is held by a zombie KFD context left by a previous job on this shared node, not by anything this run created.
  • Supporting: rocm-smi --showpids shows UNKNOWN processes with no reapable host PID / container; SIGKILL cannot release them; the script's own diagnostic points to a zombie KFD context and node reboot ([Bug] Custom all-reduce IPC buffers use fixed VA, conflict with torch_memory_saver pause/resume → zombie processes on MI300X ROCm/aiter#2061).
  • Disconfirming: (none found) — no evidence any process from commit 11cea29c ran; the VRAM was already allocated at job entry before any container launch.
• [FACT] Not caused by the commit under test — the failing step precedes checkout-independent test execution, and the same signature occurred on Jul 4 under a different commit (854b46be).

In-flight Fix Check

• In-flight fix: ✅ #30431 ([AMD] ci: self-heal driver-level VRAM leaks + graceful GPU cleanup on shutdown, open since 2026-07-07, currently marked [DO NOT MERGE]) — directly targets driver-level VRAM leaks and GPU cleanup; track this rather than opening a duplicate. (Searched is:pr is:open VRAM cleanup and ensure_vram_clear.)

Suggested Triage Steps

• Reboot / drain runner node linux-mi35x-gpu-8-zf775-runner-fxcn4 to clear the zombie KFD contexts, then re-run the job — per the script's own hint ([Bug] Custom all-reduce IPC buffers use fixed VA, conflict with torch_memory_saver pause/resume → zombie processes on MI300X ROCm/aiter#2061).
• Correlate with the previous job scheduled on this node to find which workload leaks the KFD context on exit (candidate for the graceful-shutdown fix in #30431).
• Follow #30431 to Ready for review; it aims to self-heal exactly this leak.
• No sglang source bisect is warranted — the failure is pre-test infra and independent of the commit under test.

Status

• Persistence: Intermittent on this node — failed 2026-07-07 and 2026-07-04; passed 2026-07-05 and 2026-07-03. Fires when the runner starts with leaked VRAM.
• Scope: Runner-level infra affecting nightly-8-gpu-mi35x-grok2-rocm720 on node linux-mi35x-gpu-8-zf775-runner-fxcn4; any 8-GPU job scheduled onto a node in this state would hit the same guard. No test coverage produced this run.
• Blocked work: grok2 FP8 nightly signal on MI35x ROCm 7.2 for this run — no accuracy/perf data collected.
• In-flight fix: ✅ #30431 (open since 2026-07-07, [DO NOT MERGE]) — self-heal driver-level VRAM leaks + graceful GPU cleanup.

nightly-accuracy-8-gpu-mi35x-deepseek-v32-rocm720 — failed step(s): Ensure VRAM is clear

Commit Info

• sglang: 11cea29c
• aiter: N/A — the job died in the pre-container Ensure VRAM is clear step (step 3); no docker image was built and no [CI-AITER-CHECK] markers were emitted.

Failed Tests

Test File	Test Function	Error	Log
N/A (infra — pre-test)	N/A	VRAM cleanup unsuccessful after 3 attempts → exit code 1	Log

No test ran. The intended test (test/registered/amd/accuracy/mi35x/test_deepseek_v32_eval_mi35x.py, step 6) was skipped because the runner-hygiene gate failed first.

Failure Summary

The job failed in the Ensure VRAM is clear step (scripts/ci/amd/ensure_vram_clear.sh rocm) before any container or test started. On arrival, GPUs 0–3 each held 15–16% VRAM with no host process, KFD PID, or docker container owning it. Three cleanup attempts (SIGTERM → SIGKILL → 30s wait) all failed to release the memory, so the script exited 1 and steps 4–6 were skipped.

Regression Status

Known recurring infrastructure failure (pool-wide zombie KFD VRAM leak on the mi35x-gpu-8-zf775 runner pool).

Recent history of job nightly-accuracy-8-gpu-mi35x-deepseek-v32-rocm720 (same branch main, same event schedule, completed runs only):

Date	Run	Runner	Test File Status	Failure
Jul 7	28888218691	…zf775-runner-xm7pp	⚠️ Never ran	VRAM cleanup fail (infra) — this run
Jul 6	28813124510	—	—	Run cancelled (excluded)
Jul 5	28749501468	…zf775-runner-ln67j	⚠️ Never ran	VRAM cleanup fail (infra)
Jul 4	28714556066	…zf775-runner-njh2x	⚠️ Never ran	VRAM cleanup fail (infra)
Jul 3	28676295311	…zf775-runner-mnqdt	❌ Failed	Real test failure: test_deepseek_v32_eval_mi35x.py exit 1

The VRAM-cleanup infra failure has recurred on 3 of the last 4 completed runs (Jul 4, 5, 7), each on a different runner in the same linux-mi35x-gpu-8-zf775-runner-* pool. Note: the underlying accuracy test itself has not had a green run in this window — Jul 3 reached the test but failed it for a separate (accuracy) reason. This report concerns today's failure, which is purely infra.

Failure Cluster

VRAM cleanup failure — zombie KFD context on mi35x runner pool (pre-test hygiene gate)

Facts (verified observations)

• On job start, rocm-smi reported GPU[0]=16%, GPU[1]=15%, GPU[2]=15%, GPU[3]=16% VRAM allocated; GPUs 4–7 at 0% (log).
• Diagnostics on every attempt showed No KFD PIDs currently running, no docker containers, and no fuser/lsof holders — yet VRAM stayed allocated (log).
• All 3 cleanup attempts left VRAM unchanged (16/15/15/16%); final === FAILED: VRAM cleanup unsuccessful after 3 attempts === then exit code 1 (log).
• The script itself attributes this to a zombie KFD context and states the node needs a reboot, citing ROCm/aiter#2061 (log line 542).
• Same failure signature confirmed verbatim in Jul 5 (job) and Jul 4 (job) logs, on different runners.
• AITER_COMMIT_OVERRIDE is empty; this is nightly-test-amd-rocm720.yml (not an AITER Scout run), so no Baseline A/B check applies.

Hypothesised Causes (with confidence)

• [FACT] The job failure itself is the VRAM-clear gate returning exit 1 — directly observable in the log. This is an infrastructure/runner-state failure, not a code regression in commit 11cea29c. The checked-out sglang code was never exercised.
• [MEDIUM] A prior job on the same runner leaked a driver-level (KFD) GPU context that the cleanup script cannot reap without a node reboot.
  • Supporting: VRAM allocated with zero owning processes/containers/KFD-PIDs; symptom matches the documented zombie-KFD pattern (ROCm/aiter#2061); reproduces across 3 different runners in the same pool.
  • Disconfirming: no direct evidence of which prior job/model leaked; the leak's root cause (a specific test/model teardown) is not visible from this log alone.
  • How to verify: inspect the previous job scheduled on runners xm7pp/ln67j/njh2x and check whether a specific model teardown leaves KFD contexts; reboot the affected node and confirm the next job passes the gate.
• [LOW] Runner pool provisioned without automatic reboot/health-gating between jobs, letting leaks accumulate.
  • Supporting: 3 distinct runners in one pool hit the identical state within 4 days.
  • Disconfirming: could equally be one leak-prone model reused across nodes rather than a provisioning gap.

In-flight Fix Check

• In-flight fix: ✅ #30431 [DO NOT MERGE][AMD] ci: self-heal driver-level VRAM leaks + graceful GPU cleanup on shutdown (open, non-draft, created 2026-07-07 21:59 UTC — hours after this run). Directly targets driver-level VRAM leaks + GPU cleanup on shutdown. Currently marked DO NOT MERGE (WIP) — track it rather than opening a duplicate.
• Search performed: is:pr is:open VRAM clear (16 hits) and is:pr is:open zombie KFD (2 hits), both surfacing #30431.

Suggested Triage Steps

• Reboot / drain the affected linux-mi35x-gpu-8-zf775-runner-* nodes to clear the zombie KFD contexts so the accuracy job can run.
• Track WIP #30431 (self-heal + graceful GPU cleanup on shutdown); its shutdown-side cleanup is the candidate structural fix for the leak source.
• Identify the leak source: inspect the immediately-preceding job on each affected runner to find a model/test whose teardown leaves an unreaped KFD context.
• Separately, once the gate is healthy, re-run to confirm whether the underlying accuracy test passes — note it failed for a real (non-infra) reason on Jul 3 (job) and has no green run in this window.

Status (factual)

• Persistence: VRAM-cleanup infra failure on 3 of last 4 completed scheduled runs (Jul 4, 5, 7); Jul 3 reached the test.
• Scope: nightly-accuracy-8-gpu-mi35x-deepseek-v32-rocm720 on the mi35x-gpu-8-zf775 runner pool; affects ≥3 distinct runners (xm7pp, ln67j, njh2x). Blocks all DeepSeek-V3.2 accuracy signal on ROCm 7.2 MI35x nightly.
• In-flight fix: ✅ #30431 (open, DO NOT MERGE / WIP).

nightly-8-gpu-mi35x-deepseek-r1-hicache-rocm720 — failed step(s): Ensure VRAM is clear

Commit Info

• sglang: 11cea29c
• aiter: N/A — job aborted at the pre-flight VRAM guard, before dependency install ran, so no [CI-AITER-CHECK] markers exist.

Failed Tests

Test File	Test Function	Error	Log
N/A (no test ran)	N/A	Pre-flight infra step ensure_vram_clear.sh failed: === FAILED: VRAM cleanup unsuccessful after 3 attempts === → exit code 1	Job log

This is not a test failure. The job never reached the test phase. It failed at the Run bash scripts/ci/amd/ensure_vram_clear.sh rocm pre-flight step on runner linux-mi35x-gpu-8-zf775-runner-m8chn. The intended test (had it run) is test/registered/amd/test_deepseek_r1_hicache_mi35x.py (suite nightly-amd-8-gpu-mi35x-deepseek-r1-hicache).

Failure Summary

Before any container/test setup, the CI pre-flight guard checked GPU VRAM and found all 8 GPUs already at 75–91% allocated (threshold is 5%). It attempted cleanup 3 times (SIGTERM → SIGKILL → 30s wait), but VRAM never released, so the job exited 1. The runner GPUs were dirty from a prior workload; this job was the victim, not the cause.

Regression Status

Infrastructure issue (runner GPU contamination) — not a code regression.

Recent history of job nightly-8-gpu-mi35x-deepseek-r1-hicache-rocm720 (branch main, event schedule, completed runs only):

Date	Run	Job Conclusion	Failure Mode	Reached the test?
Jul 7	28888218691	❌ failure	VRAM cleanup unsuccessful (this failure)	No
Jul 5	28749501468	❌ failure	Fatal Python error: Aborted during test (server crash)	Yes — different failure
Jul 4	28714556066	❌ failure	Same VRAM cleanup unsuccessful	No
Jul 3	28676295311	❌ failure	py-spy/server hang during test	Yes — different failure
Jul 2	28611268103	❌ failure	py-spy/server hang during test	Yes — different failure

The specific VRAM-cleanup infra failure recurs intermittently (Jul 4 and Jul 7). On the other days the job progressed into the test and failed for an unrelated reason (server abort/hang). No clean green run for this job in the queryable window.

Failure Cluster

Unreclaimable driver/KFD-level VRAM leak — pre-flight ensure_vram_clear.sh abort (zombie KFD context)

Facts (verified observations)

• Pre-flight step Run bash scripts/ci/amd/ensure_vram_clear.sh rocm reported ERROR: VRAM usage exceeds threshold (5%) on some GPUs with GPU0=80%, GPU1=89%, GPU2=91%, GPU3=91%, GPU4–7=75% (log line 167).
• rocm-smi --showpids lists KFD processes with PROCESS NAME = UNKNOWN holding ~232 GB each on GPU 1 (PIDs 875199, 875200, 875201, 875203, 875214, 875215, 875234, 875252). These are leftover from a prior workload, not this job.
• Cleanup ran 3 attempts (SIGTERM, then SIGKILL, 30s wait each); VRAM never dropped. Final: === FAILED: VRAM cleanup unsuccessful after 3 attempts === then ##[error]Process completed with exit code 1.
• The script's own diagnostic states the diagnosis verbatim: "this is almost certainly a zombie KFD context (see [Bug] Custom all-reduce IPC buffers use fixed VA, conflict with torch_memory_saver pause/resume → zombie processes on MI300X ROCm/aiter#2061). The node will need to be rebooted before subsequent jobs can succeed."
• No [CI-AITER-CHECK] markers in the log (aborted before dependency install); no test process was ever started.

Hypothesised Causes (with confidence)

• [FACT] The runner linux-mi35x-gpu-8-zf775-runner-m8chn entered this job with GPUs already 75–91% full from a previous workload, and the leaked VRAM has no owning killable process (KFD zombie context, ROCm/aiter#2061).
  • Supporting: rocm-smi --showpids shows only UNKNOWN KFD holders that survive SIGKILL; VRAM unchanged across all 3 cleanup attempts; the script's built-in hint names this exact signature.
  • Disconfirming: (none found) — the failure is fully explained by pre-existing GPU state; no code in this commit is exercised.
  • How to verify: reboot the node and re-run; confirm VRAM returns to ~0% at job start.
• [SPECULATION] The leaking prior workload may be an earlier hicache/DeepSeek-R1 test that crashed without releasing GPU contexts (cf. the Jul 5 Fatal Python error: Aborted on this same job), leaving the zombie KFD state the next job trips over.
  • Supporting: same job family repeatedly crashes mid-test on this runner; a crash-on-shutdown is a plausible leak source.
  • Disconfirming: cannot identify which prior job/container leaked from this log alone; the holder PIDs are unattributable (UNKNOWN).
  • How to verify: correlate the leaking PIDs' start time with the prior job scheduled on this runner.

In-flight Fix Check

• In-flight fix: ✅ #30431 — [DO NOT MERGE][AMD] ci: self-heal driver-level VRAM leaks + graceful GPU cleanup on shutdown (open, created 2026-07-07). Its summary explicitly targets "unreclaimable driver/KFD-level VRAM leak … the ROCm/aiter#2061 signature" — the exact failure here. It roots the cause in the ci_sglang container never being gracefully stopped and running without --init. Chase this PR (currently marked DO NOT MERGE); do NOT open a duplicate.
• Search run: search/issues?q=repo:sgl-project/sglang+is:pr+is:open+VRAM+cleanup (11 results; #30431 is the direct match).

Suggested Triage Steps

• Reboot / drain runner linux-mi35x-gpu-8-zf775-runner-m8chn to clear the zombie KFD contexts, then re-run this job — unblocks the immediate signal.
• Track #30431 toward merge-ready (it is the self-heal fix for exactly this leak signature); verify it graceful-stops the ci_sglang container with --init so docker stop SIGTERM reaches PID 1.
• Separately, investigate the source of the leak: the Jul 5 Fatal Python error: Aborted during test_deepseek_r1_hicache_mi35x.py on this same runner is a candidate leaker — a crash that doesn't release GPU contexts would produce this KFD zombie state. This is a distinct issue from the pre-flight guard.

Status

• Persistence: Recurring infra failure on this job — same VRAM-cleanup abort on Jul 4 and Jul 7; no green run in the last 5 completed scheduled runs.
• Scope: Node-local to runner linux-mi35x-gpu-8-zf775-runner-m8chn; any job landing on this node fails until reboot. Blocks the DeepSeek-R1 HiCache MI35x ROCm 7.2 nightly signal.
• In-flight fix: ✅ #30431 (open since 2026-07-07, marked DO NOT MERGE) — self-heal driver-level VRAM leak + graceful GPU cleanup.

nightly-accuracy-8-gpu-mi35x-rocm720 — failed step(s): Ensure VRAM is clear

Commit Info

• sglang: 11cea29c — "[diffusion][cache-dit] add dual-transformer Cache-DiT adapter specs (#30150)"
• aiter: N/A — the job failed in the pre-flight host-side VRAM cleanup step, before any container/dependency setup ran, so no [CI-AITER-CHECK] markers were emitted.

Failed Tests

Test File	Test Function	Error	Log
N/A (infrastructure failure)	N/A	FAILED: VRAM cleanup unsuccessful after 3 attempts → exit 1	Log L537

No test ran. The job failed in the bash scripts/ci/amd/ensure_vram_clear.sh rocm pre-flight step immediately after checkout.

Failure Summary

The job never reached any test. The pre-flight host-side GPU cleanup script (scripts/ci/amd/ensure_vram_clear.sh) detected GPUs 0–3 each stuck at 16% VRAM allocated, exceeding the 5% threshold. After 3 kill+wait cleanup attempts (SIGTERM/SIGKILL to GPU-holding processes, 30s waits), VRAM never cleared, so the script exited 1. Diagnostics show no host process, no docker container, and no KFD PIDs holding the GPUs — the script itself attributes this to a zombie KFD context requiring a node reboot.

Regression Status

Infrastructure issue (runner/host state) — NOT a code regression from the commit.

Recent history of the pre-flight VRAM cleanup failure in job nightly-accuracy-8-gpu-mi35x-rocm720 (same branch main, same event schedule, completed runs only):

Date	Run	Job Conclusion	Failure Mode (this test-level view)
Jul 7	28888218691	❌ Failed	VRAM cleanup pre-flight (infra) — exit 1, 562-line log, no test ran
Jul 5	28749501468	❌ Failed	Different — tests ran, exit 255 (~22k-line log)
Jul 4	28714556066	❌ Failed	Different — tests ran, exit 255 (~22k-line log)
Jul 3	28676295311	❌ Failed	VRAM cleanup pre-flight (infra) — exit 1, 562-line log, no test ran
Jul 2	28611268103	❌ Failed	Different — tests ran, exit 255 (~25k-line log)

The VRAM-cleanup infra failure is intermittent: it hit Jul 7 and Jul 3, while Jul 2/4/5 got past cleanup and failed inside the actual accuracy tests (a separate, unrelated failure not analyzed here). First observed (in window): Jul 3. This is host-node state, independent of the sglang commit.

Failure Cluster

Pre-flight VRAM cleanup failure (zombie KFD context, MI35x node)

Facts (verified observations, no interpretation)

• Failing step: Run bash scripts/ci/amd/ensure_vram_clear.sh rocm, run right after actions/checkout and before any container/dependency setup (log L110–L545).
• GPUs 0, 1, 2, 3 each report GPU Memory Allocated (VRAM%): 16; GPUs 4–7 report 0 (log L167–L171, repeated across all 3 attempts).
• Threshold is 5%: ERROR: VRAM usage exceeds threshold (5%) on some GPUs (log L167).
• 3 cleanup attempts, each: kill SGLang procs → SIGTERM/SIGKILL GPU holders → wait 30s → recheck. All failed (log L210, L319, L428; final verdict L537).
• Diagnostics on every attempt: No KFD PIDs currently running (L276, L385, L494), no docker containers (L270, L379, L488), no zombie host processes (L268, L377, L486), lsof not installed (L266).
• Script's own hint (L540–L543): "almost certainly a zombie KFD context (see [Bug] Custom all-reduce IPC buffers use fixed VA, conflict with torch_memory_saver pause/resume → zombie processes on MI300X ROCm/aiter#2061). The node will need to be rebooted."
• AITER_COMMIT_OVERRIDE: is empty (L34, L114) — this is a regular scheduled nightly, not an AITER Scout override run.
• Runner: linux-mi35x-gpu-8-zf775-runner-cml6s (L2).

Hypothesised Causes (with confidence, with disconfirming evidence)

• [HIGH] Zombie KFD/GPU context left on the MI35x runner from a prior job leaked ~16% VRAM on GPUs 0–3; no live process holds it, so the cleanup script cannot reclaim it and the node needs a reboot.
  • Supporting: 4 GPUs pinned at exactly 16% with zero read/write activity; No KFD PIDs, no host procs, no containers holding them across all 3 attempts; the script's own diagnostic explicitly names this pattern and points to ROCm/aiter#2061.
  • Disconfirming: (none found) — the failure occurs before any sglang/aiter code is loaded, so no code path in this repo can be the proximate cause; the initial GPU state (GPUs already at 16% before any cleanup) proves the leak pre-dates this job.
  • How to verify: reboot / drain the linux-mi35x-gpu-8-zf775-runner-cml6s node and re-run; check the immediately-preceding job on that runner for a crash that left a KFD context behind.
• [LOW] Commit 11cea29c related.
  • Supporting: it is the checked-out SHA.
  • Disconfirming: the failure is in a host pre-flight shell step that runs before any repo code executes; the same failure occurred Jul 3 on a different SHA (7820dc60). Effectively ruled out.

In-flight Fix Check

• In-flight fix: ✅ #30431 — "[DO NOT MERGE][AMD] ci: self-heal driver-level VRAM leaks + graceful GPU cleanup on shutdown" directly targets this exact driver-level VRAM-leak / GPU-cleanup class. Currently marked DO NOT MERGE (WIP) — chase its status rather than opening a duplicate. (Searched: is:pr is:open VRAM cleanup and KFD zombie.)

Suggested Triage Steps

• Reboot or cordon-and-drain the runner linux-mi35x-gpu-8-zf775-runner-cml6s to clear the zombie KFD context, then re-run this job — a passing re-run confirms it was pure host state.
• Inspect the job that ran immediately before this one on the same runner for an ungraceful exit / crash that could have leaked the KFD context (correlate with ROCm/aiter#2061).
• Track in-flight PR #30431; if its self-heal cleanup is viable, it would prevent this class of failure without manual reboots.
• Do NOT attempt to bisect sglang commits for this failure — it is host-side and pre-dates any repo code execution.

Status

• Persistence: Intermittent host-node failure; observed at the VRAM pre-flight step on Jul 7 and Jul 3 (2 of the last 5 scheduled runs of this job). The other 3 runs got past cleanup and failed later inside the accuracy tests (separate issue).
• Scope: This job / this MI35x runner (linux-mi35x-gpu-8-zf775-runner-cml6s). No sglang test executed. Any other job scheduled onto the same node will also fail until it is rebooted.
• Blocked work: Blocks the entire nightly-accuracy-8-gpu-mi35x-rocm720 accuracy signal for this run (and any co-scheduled job on the affected node).
• In-flight fix: ✅ #30431 (open, marked DO NOT MERGE) — driver-level VRAM-leak self-heal.

nightly-perf-8-gpu-mi35x-deepseek-v32-basic-rocm720 — failed step(s): Ensure VRAM is clear

Commit Info

• sglang: 11cea29c [diffusion][cache-dit] add dual-transformer Cache-DiT adapter specs (#30150)
• aiter: N/A — the job failed at the pre-container VRAM guard step; no [CI-AITER-CHECK] markers were emitted (Docker/deps never installed).

Failed Tests

Test File	Test Function	Error	Log
N/A (pre-flight infra step)	N/A	=== FAILED: VRAM cleanup unsuccessful after 3 attempts === → exit code 1	Log L639

No tests ran. The job aborted during the bash scripts/ci/amd/ensure_vram_clear.sh rocm pre-flight guard (immediately after checkout), before the container was started or any DeepSeek-V3.2 perf test executed.

Failure Summary

The MI35x 8-GPU ROCm 7.2 runner (linux-mi35x-gpu-8-zf775-runner-gbbfr) entered the job with GPUs already 75–91% VRAM-allocated. The ensure_vram_clear.sh guard ran 3 cleanup attempts (SIGTERM→SIGKILL + 30 s wait each); VRAM never dropped below the 5% threshold, so the guard failed the job with exit code 1. This is a runner-environment (leaked GPU state) failure, not a defect in the sglang commit under test.

Regression Status

Infrastructure issue (intermittent) — not a code regression.

Recent history of job nightly-perf-8-gpu-mi35x-deepseek-v32-basic-rocm720 (workflow nightly-test-amd-rocm720.yml, branch main, event schedule, completed runs only):

Date	Run	Job Status	Cause
Jul 07	28888218691	❌ Failed	VRAM guard failed (node gbbfr)
Jul 05	28749501468	✅ Passed	—
Jul 04	28714556066	✅ Passed	—
Jul 03	28676295311	❌ Failed	Same VRAM guard failure (node zlrt4)

Note: run 28888218691 is still queued overall, but this job itself completed (exit 1). The Jul 03 failure hit the same guard on a different runner node, confirming the pattern is node-state-driven and not tied to a single machine or the current commit.

Failure Cluster

Pre-flight VRAM guard abort — stuck/zombie KFD VRAM on MI35x 8-GPU ROCm runner.

Facts (verified observations)

• Step that failed: Run bash scripts/ci/amd/ensure_vram_clear.sh rocm (L110); final abort at L631–L639.
• Initial GPU VRAM at job start: GPU0=80% GPU1=89% GPU2=91% GPU3=91% GPU4-7=75% (L131–162), all well above the 5% threshold (L167).
• rocm-smi --showpids lists KFD processes as UNKNOWN holding ~232 GB each on GPU 1 (PIDs 875199–875252, L286–303) — but no docker containers (CONTAINER ID … STATUS empty, L278) and no host GPU device-file holders (fuser empty, lsof not installed, L272–274).
• After 3 cleanup attempts (SIGTERM/SIGKILL + 30 s), VRAM was unchanged (identical percentages across all 3 attempts, L223/L362/L501) → orphaned GPU allocations with no reapable owner.
• The guard's own hint (L634–637): almost certainly a zombie KFD context (see ROCm/aiter#2061). The node will need to be rebooted before subsequent jobs can succeed.

Hypothesised Causes

• [HIGH] Zombie/leaked KFD GPU contexts on the runner from a prior force-killed AMD job, holding VRAM with no live host process or container to reap.
  • Supporting: identical VRAM % across all 3 cleanup attempts despite SIGKILL; rocm-smi --showpids shows KFD PIDs as UNKNOWN with huge VRAM while fuser/docker show no holders; the guard script itself references ROCm/aiter#2061 and prescribes a node reboot.
  • Disconfirming: none found — every diagnostic in-log points to orphaned GPU state, and the same guard passes cleanly on other nights on healthy nodes.
  • How to verify: reboot node gbbfr (drain runner) and re-run the job; VRAM should return to ~0% at start.
• [FACT] The sglang commit under test is not implicated — the failure occurs before any repo code runs (only checkout + a static CI shell guard executed). No hypothesis assigns cause to 11cea29c.

In-flight Fix Check

• In-flight fix: ⚠️ #30431 [DO NOT MERGE][AMD] ci: self-heal driver-level VRAM leaks + graceful GPU cleanup on shutdown — directly targets this exact leaked-VRAM class but is currently marked DO NOT MERGE (WIP). Track it rather than opening a duplicate. (Query: search/issues?q=…+ensure_vram_clear+OR+zombie+KFD, 6 results.)

Suggested Triage Steps

• Drain and reboot runner node linux-mi35x-gpu-8-zf775-runner-gbbfr to clear the zombie KFD contexts, then re-run this job to confirm green (the guard prescribes exactly this).
• Cross-check node zlrt4 (Jul 03 failure) — if the same nodes recur, they may need proactive reboot/health-gating in the MI35x runner pool.
• Follow ROCm/aiter#2061 and chase #30431 reviewers, since a host-level self-heal is the durable fix for this cluster.
• No sglang code investigation warranted for this failure — the commit did not execute.

Status

• Persistence: Intermittent runner-state failure. Failed Jul 07 and Jul 03 (different nodes); passed Jul 04 & Jul 05. Not continuous.
• Scope: This job / the MI35x 8-GPU ROCm 7.2 runner pool; recurs across at least 2 distinct nodes (gbbfr, zlrt4).
• Blocked work: Blocks the nightly DeepSeek-V3.2 basic perf signal on MI35x ROCm 7.2 whenever a leaked-VRAM node is scheduled.
• In-flight fix: ⚠️ #30431 (DO NOT MERGE / WIP) — self-heal driver-level VRAM leaks.

nightly-8-gpu-mi35x-grok1-int4-rocm720 — failed step(s): Ensure VRAM is clear

Commit Info

• sglang: 11cea29c
• aiter: N/A — the job failed at the host-side ensure_vram_clear.sh pre-flight step; the Docker container was never started, so no [CI-AITER-CHECK] markers were emitted. AITER_COMMIT_OVERRIDE was empty (log line 34).

Failed Tests

Test File	Test Function	Error	Log
N/A (infra pre-flight)	N/A	VRAM cleanup unsuccessful after 3 attempts → exit code 1	Log

No tests ran. The failure is entirely in the runner-preparation step bash scripts/ci/amd/ensure_vram_clear.sh rocm, which runs before the test container is even pulled.

Failure Summary

The nightly-8-gpu-mi35x-grok1-int4-rocm720 job failed during the host-side VRAM-clear pre-flight check on runner linux-mi35x-gpu-8-zf775-runner-6fsrk. GPUs 0–3 each reported 16% VRAM allocated and stayed there across all 3 cleanup attempts (SIGTERM/SIGKILL + 30 s wait each). No host process, container, or KFD PID held the GPUs (No KFD PIDs currently running, no docker containers, no fuser holders), yet VRAM never dropped below the 5% threshold. The script exited 1 before any model test executed.

Regression Status

Recurring infrastructure failure (node-level, not a code regression).

Recent history of the VRAM pre-flight step in job nightly-8-gpu-mi35x-grok1-int4-rocm720 (branch main, event schedule, completed runs only):

Date	Run	Job	Step Status	Failure	Error
Jul 07	28888218691	nightly-8-gpu-mi35x-grok1-int4-rocm720	❌ Failed	VRAM clear	VRAM still not clear after 3 attempts (GPU 0–3 @16%)
Jul 06	28813124510	—	⚪ Cancelled	—	run cancelled (excluded)
Jul 05	28749501468	nightly-8-gpu-mi35x-grok1-int4-rocm720	✅ Passed	—	—
Jul 04	28714556066	nightly-8-gpu-mi35x-grok1-int4-rocm720	❌ Failed	VRAM clear	VRAM still not clear after 3 attempts
Jul 03	28676295311	nightly-8-gpu-mi35x-grok1-int4-rocm720	❌ Failed	VRAM clear	VRAM still not clear after 3 attempts

Same failure signature on Jul 03, Jul 04, and Jul 07 — intermittent (Jul 05 passed). Note this is specific to the mi35x runner: the sister nightly-8-gpu-grok1-int4-rocm720 (non-mi35x) job passed in every one of these runs, confirming the fault is tied to the mi35x node's GPU state, not to the sglang commit.

Failure Cluster

Stuck VRAM on mi35x runner — zombie KFD context blocking pre-flight cleanup.

Facts (verified observations, no interpretation)

• Pre-flight step ensure_vram_clear.sh rocm failed with exit code 1 (log line 545).
• GPUs 0–3 each reported GPU Memory Allocated (VRAM%): 16 on the initial check and after all 3 cleanup attempts; GPUs 4–7 were at 0% (log lines 168–171, 438–441).
• Cleanup ran 3 attempts, each: SIGTERM → 5 s → SIGKILL → 30 s wait, then re-check. VRAM unchanged every time.
• Diagnostics show no holders: No KFD PIDs currently running (line 494), No docker containers found (line 431), no fuser/zombie processes (lines 482–488). lsof not installed.
• The script's own hint (lines 540–543): "if no host process / container holds the GPU but VRAM is still allocated, this is almost certainly a zombie KFD context (see ROCm/aiter#2061). The node will need to be rebooted."
• Runner: linux-mi35x-gpu-8-zf775-runner-6fsrk (log line 2).
• Same signature verified in the Jul 03 and Jul 04 job logs (both exit 1 after "VRAM still not clear after attempt 3").

Hypothesised Causes (with confidence, with disconfirming evidence)

• [HIGH] Zombie KFD/HSA context on the mi35x runner leaked residual VRAM (GPU 0–3 @16%) from a prior job that the standard SIGTERM/SIGKILL sweep cannot reclaim; the node requires a reboot.
  • Supporting: VRAM stuck at a constant 16% across all 3 attempts despite zero live holders (no KFD PID, no container, no fuser process); the script itself diagnoses exactly this pattern and points to ROCm/aiter#2061; the sister non-mi35x job passed on the same commits (node-specific, not code-specific).
  • Disconfirming: (none found) — no code path in the sglang commit is exercised before this host-side script, so a code regression is ruled out.
  • How to verify: reboot / drain runner linux-mi35x-gpu-8-zf775-runner-6fsrk, then re-run the job; VRAM should read 0% on the next pre-flight.
• [LOW] A previous nightly job on this node crashed mid-run and left the allocation.
  • Supporting: only GPU 0–3 (a single prior 4-GPU workload footprint) are affected; 4–7 are clean.
  • Disconfirming: cannot confirm which prior job without the node's earlier job history; not actionable from this log alone.
  • How to verify: inspect the previous job scheduled on this runner for an abnormal exit.

In-flight Fix Check

• In-flight fix: ✅ #30431 — "[DO NOT MERGE][AMD] ci: self-heal driver-level VRAM leaks + graceful GPU cleanup on shutdown" (open since 2026-07-07, non-draft, marked DO NOT MERGE). Directly targets driver-level VRAM leaks + graceful GPU cleanup — the exact symptom here. Track this PR rather than opening a duplicate.
• Search run: search/issues?q=repo:sgl-project/sglang+is:pr+is:open+KFD and +VRAM+cleanup.

Suggested Triage Steps

• Reboot or drain the affected runner linux-mi35x-gpu-8-zf775-runner-6fsrk to clear the stuck GPU 0–3 VRAM, then re-run run 28888218691; a green pre-flight (VRAM 0%) confirms the zombie-context hypothesis.
• Review/land the self-heal work in #30431 (currently DO NOT MERGE) — pursue with its author, since it addresses driver-level VRAM leaks that the current 3-attempt SIGKILL sweep cannot recover.
• Follow ROCm/aiter#2061 for the upstream zombie-KFD-context root cause.
• Not a code issue: no bisect needed. The sglang commit 11cea29c is not exercised before the failing host script.

Status (factual, no priority assignment)

• Persistence: Recurring on the mi35x runner — same VRAM-clear failure on 2026-07-03, 2026-07-04, and 2026-07-07 (intermittent; 2026-07-05 passed, 2026-07-06 cancelled).
• Scope: Isolated to job nightly-8-gpu-mi35x-grok1-int4-rocm720 on runner linux-mi35x-gpu-8-zf775-runner-6fsrk. The sister non-mi35x grok1-int4 job passed on all these commits.
• Blocked work: Blocks the mi35x Grok1-INT4 nightly signal on ROCm 7.2; no model test in this job executes until the node's VRAM is cleared.
• In-flight fix: ✅ #30431 (open since 2026-07-07, DO NOT MERGE) — CI self-heal for driver-level VRAM leaks.

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-07-08 20:50 UTC)

[amd-bot]

nightly-test-amd-rocm720.yml — Per-Job Analysis (continued 3/10)

nightly-perf-8-gpu-mi35x-deepseek-v32-mtp-rocm720 — failed step(s): Performance Test MI35x ROCm 7.2 (8-GPU DeepSeek-V3.2 MTP)

Commit Info

• sglang: 11cea29c
• aiter: 9127c94a (v0.1.17.dev110+g9127c94a1, source: dev/patched image build preserved — [CI-AITER-CHECK] Dev/patched version detected → skipping rebuild; AITER_COMMIT_OVERRIDE empty)
• CI image: rocm/sgl-dev:v0.5.14-rocm720-mi35x-20260707

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/perf/mi35x/test_deepseek_v32_mtp_perf_mi35x.py	test_bench_one_batch	Exception: Server process exited with code -9 (root: ValueError: Unrecognized configuration class _DeepseekV32ConfigAlias for AutoModel)	Log

Failure Summary

The single perf test test_bench_one_batch (DeepSeek-V3.2 MTP, TP8) failed to launch the server on both the offline and online launch attempts. The deepseek-ai/DeepSeek-V3.2 model (arch DeepseekV32ForCausalLM) was routed to the generic Transformers backend, whose AutoModel.from_config rejected SGLang's internal _DeepseekV32ConfigAlias, crashing every TP scheduler and killing the server process (exit -9).

Regression Status

New regression (first observed today 2026-07-07). Last known passing run: 2026-07-05.

Recent history of test_deepseek_v32_mtp_perf_mi35x.py (test_bench_one_batch) in job nightly-perf-8-gpu-mi35x-deepseek-v32-mtp-rocm720 (branch main, event=schedule, completed runs only):

Date	Run	Job	Test File Status	Failed Function	Error
Jul 07	28888218691	perf-v32-mtp-rocm720	❌ Failed	test_bench_one_batch	ValueError _DeepseekV32ConfigAlias → server -9
Jul 06	28813124510	—	⚪ Run cancelled	—	(excluded)
Jul 05	28749501468	perf-v32-mtp-rocm720	✅ Passed (1/1)	—	—
Jul 04	28714556066	perf-v32-mtp-rocm720	❌ Failed (different cause)	test_bench_one_batch	VRAM cleanup unsuccessful after 3 attempts (infra)
Jul 03	28676295311	perf-v32-mtp-rocm720	✅ Passed	—	—

First observed (this signature): 2026-07-07. Last passing: 2026-07-05 (48ba79c1). Note the Jul-04 failure is a distinct infra symptom (VRAM cleanup), not this signature.

Failure Cluster

cutlass ImportError drops model registry → DeepseekV32 Transformers fallback → AutoModel ValueError

Facts (verified observations)

• FACT — During model registry population, 30+ model modules fail to import identically with No module named 'cutlass', including deepseek_v2, deepseek_v4, deepseek_nextn, glm4_moe, kimi_k25, ernie4, longcat_flash, bailing_moe_linear, pixtral (job log ~lines 4065-4073, 4587-4608): Ignore import error when loading sglang.srt.models.deepseek_v2: No module named 'cutlass'.
• FACT — Because deepseek_v2 (which defines DeepseekV32ForCausalLM, EntryClass at python/sglang/srt/models/deepseek_v2.py:2980) is dropped from ModelRegistry, the loader reports DeepseekV32ForCausalLM has no SGLang implementation, falling back to Transformers implementation (log line 4340). In model_loader/utils.py:217, is_native_supported becomes False, forcing resolve_transformers_arch.
• FACT — Transformers backend then calls AutoModel.from_config(...) (python/sglang/srt/models/transformers.py:617) and raises ValueError: Unrecognized configuration class <class 'sglang.srt.utils.hf_transformers.common._DeepseekV32ConfigAlias'> for this kind of AutoModel: AutoModel (log lines 4378, 4412, 4905). All 8 TP schedulers crash → Received sigquit → server exits -9 (log line 4930).
• FACT — Last passing run 2026-07-05 used image ...-mi35x-20260705: zero deepseek_v2 import errors, cutlass referenced 5×, test 1/1 passed (elapsed 1650s). Today's run uses image ...-mi35x-20260707.
• FACT — _DeepseekV32ConfigAlias is defined at python/sglang/srt/utils/hf_transformers/common.py:126 (model_type = "deepseek_v32"), and DeepseekV32ForCausalLM is a registered EntryClass at the CI commit — so the native path exists; it is only unreachable because the module import fails.
• FACT — Unguarded top-level from cutlass import ... exists in shared Blackwell helper python/sglang/srt/layers/attention/cute_utils/__init__.py:4, and unguarded import cutlass in nvfp4_gemm_swiglu_nvfp4_quant.py:23 and jit_kernel/cutedsl_bf16_gemm.py:32.

Hypothesised Causes (with confidence + disconfirming evidence)

• [HIGH] The 20260707 ROCm MI35x CI image is missing the cutlass Python package that the 20260705 image contained. A shared module in many models' top-level import chain does an unguarded import cutlass; with cutlass absent, deepseek_v2 (and 30+ other model modules) silently drop out of the registry, so DeepseekV32ForCausalLM can no longer resolve natively.
  • Supporting: 30+ unrelated model modules fail with the identical No module named 'cutlass'; the only material difference from the last green run is the image tag (20260705 had cutlass + passed, 20260707 does not); the failure occurs at import/registry time, before any GPU kernel runs.
  • Disconfirming: image contents were not inspected directly (inferred from log diff); a code change that newly pulls cutlass into the top-level chain cannot be 100% excluded — but that would not explain 30+ non-DeepSeek modules failing simultaneously.
  • How to verify: docker run --rm rocm/sgl-dev:v0.5.14-rocm720-mi35x-20260707 python -c "import cutlass" vs the same on ...-20260705; diff the two image build manifests.
• [LOW] A code change in the 67-commit window 48ba79c1..11cea29c newly introduced a top-level cutlass import into the shared chain (candidate: e85ef548 "Support Cutedsl BF16 GEMM JIT kernel", #30117).
  • Supporting: e85ef548 adds a top-level from ...quantization.unquant import get_bf16_gemm_backend to deepseek_v2.py and a cutedsl code path.
  • Disconfirming: that new import does not load cutlass at module-import time (the cutedsl_bf16_gemm import is lazy, inside get_bf16_gemm_backend()); and it cannot explain the 30+ other modules failing. Weakens this to LOW.

In-flight Fix Check

• In-flight fix (partial, related): #30410 "[NPU] Fix npu cutlass import" (open since 2026-07-07) — guards import cutlass for the NPU path (1 file). Same class of fix (unguarded cutlass import) but NPU-scoped; it will not restore the missing package in the ROCm image nor guard all 30+ affected modules. Search: search/issues?q=repo:sgl-project/sglang+is:pr+is:open+cutlass.
• In-flight fix (image): ❌ none found addressing a missing-cutlass package in the ROCm MI35x nightly image.

Suggested Triage Steps

• Confirm the environment hypothesis: docker run --rm rocm/sgl-dev:v0.5.14-rocm720-mi35x-20260707 python -c "import cutlass; print(cutlass.__file__)" and compare against ...-20260705. If it fails only on 20260707, the image build dropped cutlass — fix the image build (re-add the package) and re-run this job.
• If the image is the culprit, also treat the mass No module named 'cutlass' import failures across 30+ models as a broader blast-radius signal — other model tests on this image will silently fall back to Transformers.
• Harden the code path regardless: the unguarded top-level from cutlass import ... in cute_utils/__init__.py:4 (and the cutlass kernels) should be import-guarded so a missing cutlass degrades a single feature rather than dropping whole model classes from the registry. This makes ROCm robust to cutlass absence.
• If needed, bisect 48ba79c1..11cea29c for a newly-added top-level cutlass import — but prioritize the image check first (higher-probability cause).

Status

• Persistence: New today (2026-07-07); last green 2026-07-05 (Jul-06 run cancelled; Jul-04 failed on an unrelated VRAM-cleanup infra issue).
• Scope: 1 failing test in this job. However the underlying No module named 'cutlass' import failure hit 30+ model modules on this image, so blast radius likely extends to other DeepSeek/GLM/Kimi/Ernie model tests running on ...-mi35x-20260707.
• In-flight fix: #30410 addresses the cutlass-import class of bug for NPU only; no PR yet for the ROCm image gap.

nightly-8-gpu-mi35x-deepseek-r1-mxfp4-ar-fusion-rocm720 — failed step(s): Accuracy Test MI35x ROCm 7.2 (8-GPU DeepSeek-R1-MXFP4 AllReduce Fusion)

Commit Info

• sglang: 11cea29c
• aiter: 9127c94a (v0.1.17.dev110+g9127c94a1, source: dev image preserved — [CI-AITER-CHECK] Dev/patched version detected → skipping rebuild). Not implicated — this is a pure sglang Python-import regression, not an attention-kernel issue.

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi35x/test_deepseek_r1_mxfp4_ar_fusion_eval_mi35x.py	TestDeepSeekR1MXFP4ArFusionEvalMI35x.test_deepseek_r1_mxfp4_ar_fusion_accuracy	AssertionError: Failed models: ['amd/DeepSeek-R1-MXFP4-Preview (ar-fusion)']	Log

Failure Summary

The accuracy test fails because the server never starts: on all 8 TP workers the Scheduler crashes during model init with KeyError: 'sglang'. The model amd/DeepSeek-R1-MXFP4-Preview (arch DeepseekV3ForCausalLM) was loaded through the Transformers fallback backend (python/sglang/srt/models/transformers.py) instead of sglang's native DeepSeek model, because 30 native model modules — including deepseek_v2 — failed to import with No module named 'cutlass' and were silently dropped from the model registry. The process exits -9, the launch retry also fails, and the test reports exit code 255.

Regression Status

New regression (cutlass-fallback cluster). First failure: Jul 7. Last pass for this test file: Jul 5.

Recent history of test_deepseek_r1_mxfp4_ar_fusion_eval_mi35x.py (test_deepseek_r1_mxfp4_ar_fusion_accuracy) in job nightly-8-gpu-mi35x-deepseek-r1-mxfp4-ar-fusion-rocm720 (branch=main, event=schedule, completed runs only):

Date	Run	Test File Status	Failure Cause
Jul 7	28888218691	❌ Failed	KeyError: 'sglang' / cutlass import fallback (this cluster)
Jul 6	28813124510	⏭️ Cancelled	excluded (run cancelled)
Jul 5	28749501468	✅ Passed	native DeepseekV3ForCausalLM load, 0 import errors
Jul 4	28714556066	✅ Passed	—
Jul 3	28676295311	❌ Failed	different cause — VRAM cleanup unsuccessful (runner/infra), not this cluster
Jul 2	28611268103	✅ Passed	—

Regression window (last cutlass-clean pass → first fail): 48ba79c1..11cea29c (67 commits).

Failure Cluster

ModuleNotFoundError: 'cutlass' → native model registry fallback → KeyError 'sglang' in Transformers backend

Facts (verified observations)

• FACT — 8× TP workers crash identically: KeyError: 'sglang' at remote modeling_deepseek.py:1148, self.self_attn = ATTENTION_CLASSES[config._attn_implementation](...) (log line 4382). Stack goes through models/transformers.py:617 AutoModel.from_config(..., trust_remote_code=True).
• FACT — Fallback banner (log line 4288): DeepseekV3ForCausalLM has no SGLang implementation, falling back to Transformers implementation.
• FACT — 30 native model modules fail to import, only with No module named 'cutlass' (480 log occurrences, no other import-error reason): deepseek_v2, deepseek_v4, deepseek_nextn, glm4_moe, kimi_k25, longcat_flash, mistral_large_3, pixtral, ernie4, … (log line 4031+).
• FACT — models/transformers.py:614 unconditionally sets self.text_config._attn_implementation = "sglang"; the remote modeling_deepseek.py ATTENTION_CLASSES dict has no "sglang" key → KeyError.
• FACT — models/registry.py:104-109 (import_model_classes) swallows import errors (Ignore import error when loading {name}: {e}), so a broken import silently removes the arch from ModelRegistry.get_supported_archs(); model_loader/utils.py:217 then computes is_native_supported=False and routes to resolve_transformers_arch.
• FACT — Jul 5 passing run: image rocm/sgl-dev:v0.5.14-rocm720-mi35x-20260705, 0 cutlass import errors, model loaded natively (Enable Aiter AllReduce Fusion for DeepseekV3ForCausalLM, log line 3783). Jul 7 failing run: image ...-20260707.
• FACT — Import chain (all top-level, verified file-by-file): models/deepseek_v2.py:61 → layers/attention/dsa/dsa_indexer.py:11 (from sglang.jit_kernel.dsa import …) → jit_kernel/dsa/__init__.py:1 → jit_kernel/dsa/cutedsl_paged_mqa_logits.py:15,20 → jit_kernel/cutedsl_fp8_paged_mqa_logits.py:60 → import cutlass (unguarded; is_sm100_supported is imported only at line 21, after the cutlass import).
• FACT — jit_kernel/dsa/__init__.py and cutedsl_fp8_paged_mqa_logits.py are new files added by commit fefc1743 (#25220 "Cute-DSL FP8 MQA logits"), merged 2026-07-06 23:34 PDT (≈ Jul 7 06:34 UTC) — inside the regression window, absent from the Jul 5 tree.

Hypothesised Causes

• [HIGH] Commit fefc1743 (#25220) added an unguarded top-level import cutlass into jit_kernel/dsa/__init__.py's dependency chain, which is imported at module-load time by dsa_indexer.py → deepseek_v2.py (and 29 other MoE models). cutlass (NVIDIA nvidia-cutlass-dsl) is not installed on the ROCm image, so those modules fail to import, drop out of the model registry, and DeepseekV3ForCausalLM falls back to the Transformers backend, which then hits KeyError: 'sglang'.
  • Supporting: (a) full top-level import chain verified in source; (b) the two files carrying the unguarded import cutlass were created by this commit; (c) commit lands between the Jul 5 pass and Jul 7 fail and is absent from the Jul 5 tree; (d) failing log shows exactly the 30 affected modules failing on the single reason No module named 'cutlass', plus the DeepseekV3ForCausalLM … falling back to Transformers banner.
  • Disconfirming: none found. (A sibling commit e85ef548 / #30117 also added a cutedsl file with top-level import cutlass — cutedsl_bf16_gemm.py — but that one is imported lazily inside functions (deepseek_v2.py:1987, unquant.py:96), so it does not break module import; fefc1743 is the commit that wired it into a top-level __init__ chain.)
  • How to verify: on the Jul 7 ROCm image, run python -c "import sglang.srt.models.deepseek_v2" — expect ModuleNotFoundError: No module named 'cutlass' from cutedsl_fp8_paged_mqa_logits.py:60. A/B: git revert fefc1743 (or guard the import cutlass behind an SM100/CUDA check) on a branch and re-run this nightly job.
• [FACT, secondary] Two independent pre-existing defects turn this import failure into a silent accuracy failure rather than a clear error: (1) registry.py:109 swallows all import errors, so a broken native model degrades to the Transformers fallback with only a warning; (2) transformers.py:614 sets _attn_implementation="sglang" on a trust_remote_code model whose remote ATTENTION_CLASSES cannot honor it. These are contributing factors, not the trigger.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for cutlass import dsa, cutedsl import error, No module named cutlass, cutedsl_paged_mqa_logits; no matching PR guards the top-level import cutlass regression.

Suggested Triage Steps

• Reproduce the import in the ROCm CI container: python -c "import sglang.srt.models.deepseek_v2" on image rocm/sgl-dev:v0.5.14-rocm720-mi35x-20260707 — confirm the traceback terminates at cutedsl_fp8_paged_mqa_logits.py:60 import cutlass.
• A/B confirm the culprit: revert fefc1743 (#25220) on a branch and re-run nightly-8-gpu-mi35x-deepseek-r1-mxfp4-ar-fusion-rocm720; expect native DeepseekV3ForCausalLM load to return.
• If confirmed, the fix direction is to make the CuteDSL import cutlass lazy / SM100-gated (guard the imports in jit_kernel/dsa/__init__.py, cutedsl_paged_mqa_logits.py, cutedsl_fp8_paged_mqa_logits.py behind a CUDA/is_sm100_supported check or a try/except ImportError), so ROCm/AMD import paths don't require the NVIDIA cutlass package.
• Hardening (separate follow-ups): consider making registry.py import failures louder for first-party models so a broken native model doesn't silently degrade to the Transformers fallback; and reconsider unconditionally forcing _attn_implementation="sglang" in transformers.py for trust_remote_code models that can't support it.

Status

• Persistence: First failure of this cluster on the Jul 7 run 28888218691; this test file last passed Jul 5. (The Jul 3 failure was an unrelated VRAM-cleanup/infra issue.)
• Scope: Broad — 30 native MoE/quant model modules fail to import on this ROCm image (deepseek_v2/v4, glm4_moe, kimi_k25, longcat_flash, mistral_large_3, pixtral, ernie4, …). Any AMD nightly/PR job that loads a DSA-chain model via the native path is at risk of the same silent Transformers fallback. Blocks the DeepSeek-R1-MXFP4 ar-fusion accuracy signal on MI35x ROCm 7.2.
• In-flight fix: ❌ none found in open PRs — needs a new PR (lazy/gated cutlass import) or a revert of #25220.

nightly-8-gpu-mi35x-qwen3-235b-mxfp4-rocm720 — failed step(s): Ensure VRAM is clear

Commit Info

• sglang: 11cea29c
• aiter: N/A — the job failed at the host-level pre-flight VRAM check before any container start or dependency install, so no [CI-AITER-CHECK] markers were emitted. AITER_COMMIT_OVERRIDE was empty (regular nightly, not a Scout run).

Failed Tests

Test File	Test Function	Error	Log
N/A (no test ran)	N/A	Pre-flight step Ensure VRAM is clear (scripts/ci/amd/ensure_vram_clear.sh rocm) failed: FAILED: VRAM cleanup unsuccessful after 3 attempts → exit code 1	Log

This is not a test failure. The job died in the setup step that guards the runner before the model server / accuracy test is even launched.

Failure Summary

The nightly-8-gpu-mi35x-qwen3-235b-mxfp4-rocm720 job failed during the host pre-flight Ensure VRAM is clear step. At job start, GPUs 0–3 each reported 16% VRAM Allocated (lines 168–171). The cleanup script ran 3 attempts (SIGTERM → SIGKILL to GPU device-file holders, then a 30 s wait) but VRAM never dropped below the 5% threshold, so it exited 1 (line 537). No Qwen3-235B-MXFP4 model server or accuracy test ran.

Regression Status

Infrastructure issue — persistent, environmental (not a code regression).

Recent history of the Ensure VRAM is clear pre-flight step in job nightly-8-gpu-mi35x-qwen3-235b-mxfp4-rocm720 (same branch main, event=schedule, completed runs only):

Date	Run	Job	Step Status	Failure	Error
Jul 7	28888218691	job	❌ Failed	VRAM cleanup	FAILED: VRAM cleanup unsuccessful after 3 attempts
Jul 6	28813124510	—	⏭️ Cancelled	(run cancelled)	—
Jul 5	28749501468	job	❌ Failed	VRAM cleanup	FAILED: VRAM cleanup unsuccessful after 3 attempts
Jul 4	28714556066	job	❌ Failed	VRAM cleanup	FAILED: VRAM cleanup unsuccessful after 3 attempts
Jul 3	28676295311	job	❌ Failed	VRAM cleanup	FAILED: VRAM cleanup unsuccessful after 3 attempts

First observed (in queryable window): Jul 3 2026. No passing run of this step found in the last 4 completed scheduled runs. Notably, each run landed on a different runner pod (6mn6j, zhrmt, vms6f, 4d64n) — all in the linux-mi35x-gpu-8-zf775 pool — so this is a pool-wide stuck-VRAM state, not one bad node.

Failure Cluster

Unreclaimable driver/KFD-level VRAM leak on MI35x runner pool (VRAM allocated with no owning process — ROCm/aiter#2061 signature)

Facts (verified observations)

• Pre-flight step reports GPUs 0–3 at 16% VRAM Allocated at job start; GPUs 4–7 at 0–1% (lines 168–171).
• 3 cleanup attempts each failed; between attempts, diagnostics show no host processes, no docker containers, no KFD PIDs holding the GPUs (No KFD PIDs currently running, No docker containers found on host, empty fuser output — lines 264–317). VRAM stayed at 16% throughout.
• The script itself prints the diagnosis: "if no host process / container holds the GPU but VRAM is still allocated, this is almost certainly a zombie KFD context (see [Bug] Custom all-reduce IPC buffers use fixed VA, conflict with torch_memory_saver pause/resume → zombie processes on MI300X ROCm/aiter#2061). The node will need to be rebooted" (lines 540–543).
• Same step failed identically in the 3 prior completed runs (Jul 3/4/5) on 3 different runner pods in the same pool.
• Failure occurred before container launch → no aiter/torch/model code executed.

Hypothesised Causes (with confidence)

• [FACT] The job was blocked by a driver/KFD-level VRAM leak on the runner: 16% VRAM allocated on GPUs 0–3 with no owning process, container, or KFD PID. The guard step correctly detected it and can only fail (by design) — it does not reboot the node.
  • Supporting: Direct log evidence — VRAM allocated but all process/container/KFD holder diagnostics are empty; matches the [Bug] Custom all-reduce IPC buffers use fixed VA, conflict with torch_memory_saver pause/resume → zombie processes on MI300X ROCm/aiter#2061 signature the script names explicitly.
  • Disconfirming: (none found) — this is an environmental state, verifiable directly in the log; not attributable to any sglang commit.
  • How to verify: Reboot the affected linux-mi35x-gpu-8-zf775 pool nodes and re-run; VRAM should return to baseline and the job should proceed past the guard step.
• [LOW] A prior job on these nodes (e.g. an earlier crashed/OOM'd 8-GPU MXFP4 run) left the leaked KFD context. Not confirmable from this log alone (guard runs first, before any workload).
  • Supporting: The leak predates this job (present at checkout); the 8-GPU MXFP4 workloads are memory-heavy and prone to hard crashes that can strand KFD contexts.
  • Disconfirming: No cross-job scheduling data captured here to identify the leaking predecessor.
  • How to verify: Correlate node reboot/allocation history for the zf775 pool against prior failing 8-GPU jobs.

In-flight Fix Check

• In-flight fix: ✅ #30431 "[DO NOT MERGE][AMD] ci: self-heal driver-level VRAM leaks + graceful GPU cleanup on shutdown" (open, created 2026-07-07) — directly targets this [Bug] Custom all-reduce IPC buffers use fixed VA, conflict with torch_memory_saver pause/resume → zombie processes on MI300X ROCm/aiter#2061 KFD-level VRAM leak so the guard step can self-heal instead of only failing. Currently marked [DO NOT MERGE]; chase the author/reviewers rather than opening a duplicate. (Searched repo:sgl-project/sglang is:pr is:open VRAM clear.)

Suggested Triage Steps

• Reboot the linux-mi35x-gpu-8-zf775 runner-pool nodes (or drain + power-cycle) to clear the zombie KFD contexts — the guard step's own hint calls a reboot the only reliable recovery. This unblocks the nightly regardless of sglang code.
• Track #30431 to completion — it makes the guard self-heal so future leaks don't require manual reboots; verify it moves off [DO NOT MERGE].
• Once the pool is clean, re-run run 28888218691 to confirm the Qwen3-235B-MXFP4 test itself is healthy — it has never actually executed in the last 4 completed runs, so its pass/fail state is currently unknown and masked by this infra block.
• Consider investigating which upstream 8-GPU job strands the KFD context (heavy MXFP4 runs are candidates) to address the leak at the source.

Status

• Persistence: Ensure VRAM is clear step failing in every completed scheduled run since Jul 3 2026 (4 consecutive completed runs; Jul 6 run was cancelled).
• Scope: Pool-wide on linux-mi35x-gpu-8-zf775 (4 distinct runner pods affected). This report covers one job; other 8-GPU MI35x-rocm720 nightly jobs landing on the same pool are likely similarly blocked.
• Blocked work: Entirely blocks the Qwen3-235B-MXFP4 rocm720 nightly accuracy signal — the test never runs.
• In-flight fix: ✅ #30431 (open, 2026-07-07) — self-heal for driver-level VRAM leaks; still [DO NOT MERGE].

nightly-accuracy-8-gpu-mi35x-deepseek-v32-mtp-rocm720 — failed step(s): Accuracy Test MI35x ROCm 7.2 (8-GPU DeepSeek-V3.2 TP+MTP)

Commit Info

• sglang: 11cea29c
• aiter: 9127c94a — v0.1.17.dev110+g9127c94a1 (source: dev/patched version detected in CI image, rebuild skipped)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi35x/test_deepseek_v32_mtp_eval_mi35x.py	TestDeepseekV32TPMTP.setUpClass	Exception: Server process exited with code -9 (server crash during launch)	Log

Underlying scheduler crash (root of the -9):
ValueError: Unrecognized configuration class _DeepseekV32ConfigAlias for this kind of AutoModel: AutoModel (log).

Failure Summary

The DeepSeek-V3.2 TP=8 + EAGLE/MTP accuracy server never came up: setUpClass aborted because the server process exited with code -9 (SIGKILL after a sigquit from a crashed child). The child (TP5 scheduler) crashed inside load_model because a DeepSeek-V3.2 model instance was routed into the generic HuggingFace transformers fallback path (sglang/srt/models/transformers.py), which then called AutoModel.from_config() on _DeepseekV32ConfigAlias — a config class HF's AutoModel does not recognize. The main model itself resolved natively (Enable Aiter AllReduce Fusion for DeepseekV3ForCausalLM, log), so the crash is on a secondary model instance (the EAGLE/MTP draft, speculative_draft_model_path=deepseek-ai/DeepSeek-V3.2).

Regression Status

New regression. The accuracy job passed on the three most recent comparable completed runs; Jul 6 was cancelled (not comparable).

Recent history of test_deepseek_v32_mtp_eval_mi35x.py (TestDeepseekV32TPMTP) in job nightly-accuracy-8-gpu-mi35x-deepseek-v32-mtp-rocm720 (branch main, event schedule, completed runs only):

Date	Run	Test File Status	Failed Function	Error
Jul 7	28888218691	❌ Failed	setUpClass	Server exit -9 / Unrecognized config _DeepseekV32ConfigAlias
Jul 6	28813124510	⏭️ Cancelled	—	run cancelled (not comparable)
Jul 5	28749501468	✅ Passed	—	—
Jul 4	28714556066	✅ Passed	—	—
Jul 3	28676295311	✅ Passed	—	—

First observed failure: Jul 7 2026. Last known pass: Jul 5 2026 (pass_sha 48ba79c11e). Regression window: 48ba79c11e..11cea29c (67 commits).

Failure Cluster

DeepSeek-V3.2 MTP model falls into transformers fallback → AutoModel config not recognized → server crash

Facts (verified observations)

• server_args: model_impl='auto', speculative_algorithm='EAGLE', speculative_draft_model_path='deepseek-ai/DeepSeek-V3.2', tp_size=8, attention_backend='dsa' (log).
• Crash stack (verbatim): model_runner.py:1512 load_model → loader.py:761 → loader.py:320 _initialize_model → model_class(**kwargs) → transformers.py:1143 → transformers.py:617 AutoModel.from_config → ValueError (log). The resolved model_class was TransformersForCausalLM, i.e. the generic fallback, not a native class.
• Code fact (python/sglang/srt/model_loader/utils.py:217,221): the transformers fallback is taken when is_native_supported is False — i.e. none of hf_config.architectures are in the native registry. DeepseekV32ForCausalLM, DeepseekV3ForCausalLM, and DeepseekV3ForCausalLMNextN are ALL natively registered (deepseek_v2.py:2980 EntryClass, deepseek_nextn.py:279). Therefore at load time the model instance's hf_config.architectures must have been empty or non-native for the fallback to trigger.
• The main target model resolved natively to DeepseekV3ForCausalLM; the crashing instance is a secondary one (EAGLE/MTP draft) — the failure is specific to the MTP/draft config path.
• _DeepseekV32ConfigAlias is defined in python/sglang/srt/utils/hf_transformers/common.py:126 (subclass of HF DeepseekV3Config, model_type="deepseek_v32"); it is intentionally NOT registered with HF AutoModel, so any transformers-fallback attempt on it necessarily raises this ValueError.
• This is nightly-test-amd-rocm720.yml (not AITER Scout); aiter version is unchanged (dev image preserved), so aiter is not implicated.

Hypothesised Causes (with confidence)

• [MEDIUM] The regression is in sglang config/architecture resolution: for the DeepSeek-V3.2 MTP draft model, hf_config.architectures no longer contains a natively-supported entry at load time, so get_model_architecture falls through to the transformers backend (utils.py:221) and crashes on the unregistered _DeepseekV32ConfigAlias.
  • Supporting: crash stack lands in the transformers fallback; both the V3.2 and NextN native archs are registered, so only an empty/unrecognized architectures list explains the fallback; failure is fresh (Jul 5 → Jul 7) and only affects the MTP/EAGLE path.
  • Disconfirming: I could not confirm which commit empties/mis-sets architectures; _config_draft_model (model_config.py:533-541, which rewrites the V3.2 draft arch to DeepseekV3ForCausalLMNextN) is unchanged in the window.
  • How to verify: on an MI35x runner, launch deepseek-ai/DeepSeek-V3.2 with the exact test args and print model_config.hf_config.architectures for both target and draft just before get_model_architecture; confirm the draft list is empty/non-native.
• [LOW] Config-handling refactors merged Jul 6 inside the window: #30297 (c8618967, "Resolve config declarations onto server_args at end of __post_init__", 504-line server_args.py change) and #30151 (92a1f6e0, "Reorder ServerArgs sections; inline arch tuples").
  • Supporting: both touch server_args/config resolution and landed exactly between the last pass and first fail.
  • Disconfirming: neither diff visibly touches hf_config.architectures population or the draft-model arch rewrite; no code-level link established.
  • How to verify: bisect the window; if suspected, revert on a branch and re-run the test.
• [LOW] MTP-adjacent commits in window: #29787 (16372b4c, touches deepseek_nextn.py) and #30333 (9a6f8e59, DeepSeek V4 MTP).
  • Supporting: touch NextN/MTP code paths.
  • Disconfirming: #29787's deepseek_nextn.py change is topk-anchoring logic (not arch/config resolution); #30333 only edits deepseek_v4_compress_state.py (V4, not V3.2).
  • How to verify: included in the bisect range above.

No hypothesis reaches HIGH; the mechanism (transformers fallback due to empty/non-native architectures) is FACT, but the culprit commit needs a git bisect between 48ba79c11e..11cea29c.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for DeepSeek V3.2 MTP and Unrecognized configuration; no PR addresses this transformers-fallback/config regression (matches were unrelated: HiCache MTP pool, DCP support, generic trust_remote_code compat).

Suggested Triage Steps

• Reproduce on MI35x and dump hf_config.architectures for target vs draft right before get_model_architecture (model_loader/utils.py:216) to confirm the draft list is empty/non-native — this pinpoints whether the bug is in config parsing or draft-arch rewrite.
• Bisect 48ba79c11e..11cea29c, running test_deepseek_v32_mtp_eval_mi35x.py on an 8×MI35x runner; start with the config/server_args refactors #30297 and #30151.
• If a culprit is confirmed, revert it on a branch and re-run the accuracy job to A/B confirm.
• Cross-check the sister non-MTP DeepSeek-V3.2 accuracy job in the same run to confirm the regression is MTP/draft-specific (the non-MTP target resolves natively per the log).

Status

• Persistence: First and only failure on Jul 7 2026 (1 completed run); passed on the prior 3 comparable completed runs (Jul 3–5).
• Scope: 1 job (nightly-accuracy-8-gpu-mi35x-deepseek-v32-mtp-rocm720) in nightly-test-amd-rocm720.yml. Blocks DeepSeek-V3.2 MTP accuracy signal on MI35x/ROCm 7.2. Likely also affects the sibling perf/1-node MTP jobs if they share the draft-load path (not verified here).
• In-flight fix: ❌ none found — needs new PR or investigation.

nightly-8-gpu-mi35x-deepseek-r1-mxfp4-tp4-rocm720 — failed step(s): Accuracy Test MI35x ROCm 7.2 (DeepSeek-R1-MXFP4 TP4 MTP), Accuracy Test MI35x ROCm 7.2 (DeepSeek-R1-MXFP4 TP4)

Commit Info

• sglang: 11cea29c
• aiter: 9127c94a (v0.1.17.dev110+g9127c94a1; source: dev image detected → rebuild skipped, per [CI-AITER-CHECK] markers). Not implicated — failure is a Python cutlass package import, not an aiter kernel.

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi35x/test_deepseek_r1_mxfp4_tp4_mi35x.py	TestDeepSeekR1MXFP4TP4MI35x.setUpClass (L124)	Exception: Server process exited with code -9	step 6
test/registered/amd/accuracy/mi35x/test_deepseek_r1_mxfp4_tp4_mtp_mi35x.py	TestDeepSeekR1MXFP4TP4MTPMI35x.setUpClass (L141)	Exception: Server process exited with code -9	step 7

Both server steps (job page) died during model load, so Ran 0 tests for each — the accuracy assertions never ran.

Failure Summary

Both DeepSeek-R1-MXFP4 accuracy steps failed at setUpClass because the server subprocess crashed (SIGKILL, exit -9) during model load. The scheduler raised KeyError: 'sglang': DeepseekV3ForCausalLM fell back to the HuggingFace Transformers backend (because sglang.srt.models.deepseek_v2 failed to import with No module named 'cutlass'), and the model's remote code then did ATTENTION_CLASSES[config._attn_implementation] where _attn_implementation == 'sglang' is not a key.

Regression Status

New failure mode today (Jul 7) for a test file that has never passed in the queryable window. Prior runs failed for different reasons — the server used to start natively (no cutlass errors) and then hit accuracy or infra failures. Today the server does not even start.

Recent history of test_deepseek_r1_mxfp4_tp4_mtp_mi35x.py (setUpClass) in job nightly-8-gpu-mi35x-deepseek-r1-mxfp4-tp4-rocm720 (branch main, event=schedule, completed runs only):

Date	Run	Test File Status	Failure signature
Jul 7	28888218691	❌ Failed	Server crash -9 (cutlass→Transformers fallback→KeyError: 'sglang') ← new
Jul 6	28813124510	⏭️ cancelled	excluded (not completed)
Jul 5	28749501468	❌ Failed	Accuracy AssertionError: 0.840 < 0.944 (server started, native load, 0 cutlass errors)
Jul 4	28714556066	❌ Failed	Infra: VRAM cleanup unsuccessful after 3 attempts (test never ran)
Jul 3	28676295311	❌ Failed	Accuracy AssertionError: 0.849 < 0.944
Jul 2	28611268103	❌ Failed	Infra: VRAM cleanup unsuccessful

Lookback window Jul 2–7: no passing run found for this test file. The non-MTP file (test_deepseek_r1_mxfp4_tp4_mi35x.py) shows the same new crash today.

Failure Cluster

cutlass import failure → Transformers fallback → KeyError('sglang') server crash

Facts (verified observations)

• [FACT] Server crash chain (log lines 5112–5153): Scheduler hit an exception … model_runner.py:1512 load_model → loader.py:320 _initialize_model → transformers.py:617 AutoModel.from_config → modeling_deepseek.py:1148 ATTENTION_CLASSES[config._attn_implementation] → KeyError: 'sglang', then Received sigquit from a child process, then Server process exited with code -9.
• [FACT] DeepseekV3ForCausalLM has no SGLang implementation, falling back to Transformers implementation printed on all 4 TP ranks (log L4082–4102) — the native SGLang DeepSeek path was not used.
• [FACT] Ignore import error when loading sglang.srt.models.deepseek_v2: No module named 'cutlass' (L3968–3971), plus deepseek_v4, deepseek_nextn, etc. Because deepseek_v2 (which defines DeepseekV3ForCausalLM) failed to import, the model is unregistered → Transformers fallback.
• [FACT] 480 No module named 'cutlass' messages today vs 0 in the Jul 5 run (85246031306) — a broad, environment-wide change across dozens of models (glm4_moe, kimi, longcat, mistral, pixtral, deepseek…), not one model.
• [FACT] On Jul 5 the same test reached the accuracy assertion (0.840 not >= 0.944), proving the server started and DeepSeek loaded natively 2 days ago.
• [FACT] No sglang commit in window 48ba79c1..11cea29c (67 commits) touched nvfp4_gemm_swiglu_nvfp4_quant.py, layers/moe/__init__.py, layers/moe/utils.py, or cute_utils/. The two nvfp4_gemm_swiglu_nvfp4_quant importers (deepseek_v2.py:303, modelopt_quant.py:1623) are both deferred (inside functions), so they are not the module-load cause.

Hypothesised Causes (with confidence)

• [MEDIUM] The cutlass (NVIDIA CUTLASS DSL) Python package became unavailable/uninstalled in the ROCm 7.2 CI image between Jul 5 and Jul 7.
  • Supporting: 0→480 cutlass import errors with no sglang code change to the known import chain in the window; the breakage is uniform across all models, characteristic of a package/env change, not a code diff. Window contains a Docker-layer change #29394 (MoRI v1.2.1) that may have altered image contents.
  • Disconfirming: cutlass is an NVIDIA/Blackwell-SM100 DSL package that would not normally be present on a ROCm image at all; if it was genuinely gone, the native deepseek_v2 path would need a top-level cutlass dependency to fail at import — I could not locate one (all import cutlass sites in models/-reachable code appear deferred). This weakens the "package removed" story and suggests the import chain itself changed.
  • How to verify: on the ROCm 7.2 image at SHA 11cea29c, run python -c "import cutlass" and python -c "import sglang.srt.models.deepseek_v2"; diff pip list against the Jul 5 image.
• [LOW] A commit in the window moved/added a top-level cutlass import onto a module transitively imported by every model at load time.
  • Supporting: the failure is a module-import-time ImportError, which requires an eager (top-level) cutlass import somewhere on the deepseek_v2 import path.
  • Disconfirming: git log on the obvious shared modules shows no such change in the window.
  • How to verify: python -X importtime -c "import sglang.srt.models.deepseek_v2" on the CI image to find which module raises, then git log -p that file across 48ba79c1..11cea29c.
• [LOW] Secondary defect: SGLang's Transformers-backend fallback sets config._attn_implementation = "sglang", which this model's remote code (modeling_deepseek.py) does not recognize (ATTENTION_CLASSES has no 'sglang' key) → crash instead of a graceful error. This is a fallback-robustness bug that turns any deepseek_v2 import failure into a hard SIGKILL.
  • How to verify: inspect transformers.py:617 fallback path; the crash is a consequence, not the root trigger.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for cutlass+transformers+fallback, DeepseekV3ForCausalLM+transformers, attn_implementation+sglang+KeyError (via search/issues); the 5 hits (e.g. #27916, #20549) are unrelated MoE/NVFP4 work. Needs new investigation.

Suggested Triage Steps

• On the ROCm 7.2 CI image at 11cea29c, reproduce with python -c "import sglang.srt.models.deepseek_v2" and python -X importtime … to pin the exact module raising No module named 'cutlass'.
• Diff the ROCm 7.2 image contents (pip list, cutlass presence) between the Jul 5 and Jul 7 images; inspect Docker-layer PR #29394 for unintended package changes.
• If the import chain (not the package) changed, bisect 48ba79c1..11cea29c on the offending module found above.
• Separately, harden the Transformers-backend fallback so an unrecognized _attn_implementation='sglang' fails gracefully instead of SIGKILL — the fallback should not be triggered for a model that has a native SGLang implementation (deepseek_v2) that merely failed to import.
• Note: even before the crash, this test's underlying MTP accuracy was failing (0.84–0.85 < 0.944 on Jul 3/5); fixing the load crash will surface a pre-existing accuracy regression that also needs triage.

Status

• Persistence: Test file has no passing run in the queryable window (Jul 2–7, every completed schedule run failed). The specific server-crash signature is new today (Jul 7); Jul 3/5 failed on accuracy, Jul 2/4 on VRAM-cleanup infra.
• Scope: Both DeepSeek-R1-MXFP4 steps (TP4 and TP4-MTP) in this job. The underlying cutlass import failure is image-wide (480 occurrences, ~all models) so other AMD ROCm 7.2 model jobs relying on native DeepSeek/NVFP4 paths are likely affected too.
• Blocked work: DeepSeek-R1-MXFP4 accuracy signal on MI35x ROCm 7.2 (native path unreachable; running via slow Transformers fallback that crashes).
• In-flight fix: ❌ none found — needs new PR or investigation.

nightly-8-gpu-mi35x-deepseek-v4-pro-rocm720 — failed step(s): Accuracy + Performance Test MI35x ROCm 7.2 (8-GPU DeepSeek-V4-Pro FP8 + FP4, triton), Ensure VRAM is clear

Commit Info

• sglang: 11cea29c
• aiter: N/A — no [CI-AITER-CHECK] markers; the test container (ci_sglang) never started, so no aiter version was ever resolved.

Failed Tests

Test File	Test Function	Error	Log
N/A (infra pre-flight, not a test)	N/A	ensure_vram_clear.sh: VRAM usage exceeds 5% threshold on GPUs 0-3; cleanup unsuccessful after 3 attempts → exit 1	Log

No test ran. The job aborted in the bash scripts/ci/amd/ensure_vram_clear.sh rocm pre-flight guard. The subsequent run_suite.py --suite nightly-amd-8-gpu-mi35x-deepseek-v4-pro step then exited immediately (Error response from daemon: No such container: ci_sglang), so zero tests in the DeepSeek-V4-Pro suite were executed.

Failure Summary

The MI35x 8-GPU runner started the job with GPUs 0-3 already holding 15-16% VRAM (threshold is 5%). The ensure_vram_clear.sh guard ran 3 cleanup attempts (SIGTERM→SIGKILL on GPU-device holders + 30 s wait each) but VRAM never dropped, so it failed the job before the container launched. This is a host/infrastructure condition, not an sglang or test regression.

Regression Status

Flaky / recurring infrastructure issue (intermittent on the linux-mi35x-gpu-8 pool).

Recent history of the VRAM pre-flight guard for job nightly-8-gpu-mi35x-deepseek-v4-pro-rocm720 (branch main, event schedule, completed runs only):

Date	Run	Guard Status	Detail
Jul 7	28888218691	❌ Failed	GPUs 0-3 @15-16% VRAM; cleanup failed ×3 (this run)
Jul 5	28749501468	✅ Passed	Guard cleared; suite ran (job succeeded)
Jul 4	28714556066	❌ Failed	Same VRAM-cleanup failure, GPUs 0-3 @15-16%
Jul 3	28676295311	✅ Passed (guard)	Container pulled & suite ran (177k-line log); any failure there is a separate test issue, not this guard

First observed (in window): Jul 4. Cleared Jul 5, reappeared Jul 7 — the physical MI35x node leaks VRAM across jobs regardless of the ephemeral runner pod name (Jul 4 = ...b24pr, Jul 5 = ...p4sx5, Jul 7 = ...jf758; all on the zf775 node pool).

Failure Cluster

"ROCm VRAM pre-flight guard failure (zombie KFD context on MI35x runner)"

Facts (verified observations, no interpretation)

• ERROR: VRAM usage exceeds threshold (5%) on some GPUs: GPU[0]=16 GPU[1]=15 GPU[2]=15 GPU[3]=16 (log lines 167-171, repeated at 219-223, 328-332, 437-441). GPUs 4-7 were at 0%.
• 3 cleanup attempts, each: kill SGLang procs → SIGTERM/SIGKILL GPU-device holders → wait 30 s → recheck. All 3 failed (lines 262, 371, 480).
• Diagnostics during failure: No KFD PIDs currently running, No docker containers found on host, no zombie host processes (lines 268, 276, 322, 385). i.e. nothing on the host holds the GPU, yet VRAM stays allocated.
• Final: === FAILED: VRAM cleanup unsuccessful after 3 attempts === → ##[error]Process completed with exit code 1 (lines 537, 545).
• Guard's own hint: almost certainly a zombie KFD context (see ROCm/aiter#2061). The node will need to be rebooted (lines 540-543).
• Test step never started: Error response from daemon: No such container: ci_sglang (line 561); Exit code 1 indicates test failure ... not retrying (line 563).

Hypothesised Causes (with confidence, with disconfirming evidence)

• [HIGH] Host-level GPU-memory leak (zombie KFD context) on the MI35x node from a prior job that ran on the same physical hardware; the VRAM survives process teardown so the pre-flight guard cannot reclaim it.
  • Supporting: nothing on the host holds the GPUs (no KFD PIDs, no containers, no zombie procs) yet GPUs 0-3 stay pinned at 15-16% across all 3 cleanup cycles; identical symptom on Jul 4 on a different runner pod of the same node pool; matches the documented [Bug] Custom all-reduce IPC buffers use fixed VA, conflict with torch_memory_saver pause/resume → zombie processes on MI300X ROCm/aiter#2061 zombie-KFD pattern the guard itself calls out.
  • Disconfirming: (none found) — the fault is entirely pre-container and there is no sglang code path in the stack; the same commit's guard passed Jul 5, showing the sglang side is not implicated.
  • How to verify: reboot / drain the linux-mi35x-gpu-8-zf775 node and re-run; correlate with which suite ran on that node immediately before this job (the leaking predecessor).
• [LOW] A concurrent job was still using GPUs 0-3 when this job's guard ran (contention rather than a true leak).
  • Supporting: only 4 of 8 GPUs were occupied, consistent with another 4-GPU workload.
  • Disconfirming: the guard reports No KFD PIDs and No docker containers on the host, so no live consumer was detected; VRAM was static (0% read/write activity) across 90+ s.
  • How to verify: check the runner's scheduling logs for overlapping jobs on the same node during 18:26-18:28 UTC.

In-flight Fix Check

• In-flight fix: ✅ #30431 "[DO NOT MERGE][AMD] ci: self-heal driver-level VRAM leaks + graceful GPU cleanup on shutdown" — opened 2026-07-07 (same day), directly targets driver-level VRAM leaks / zombie KFD cleanup. Currently marked DO NOT MERGE (WIP) — track it rather than opening a duplicate.
• Search: is:pr is:open with keywords VRAM cleanup (11 hits), zombie KFD (2 hits, both surface #30431); ensure_vram_clear → 0 hits.

Suggested Triage Steps

• Reboot or cycle the linux-mi35x-gpu-8-zf775 physical node to clear the zombie KFD context, then re-run this job (per the guard's own hint and [Bug] Custom all-reduce IPC buffers use fixed VA, conflict with torch_memory_saver pause/resume → zombie processes on MI300X ROCm/aiter#2061).
• Identify the job that ran on this node immediately before 18:26 UTC on Jul 7 (and before the Jul 4 failure) — the predecessor that leaks GPUs 0-3 VRAM is the real culprit to fix.
• Follow / help land #30431; if its self-heal approach works, it should reclaim these zombie contexts without a manual reboot.
• No sglang code bisect is warranted — the failure is pre-container infra and reproduces without any test executing.

Status (factual, no priority assignment)

• Persistence: Intermittent — failed Jul 4 and Jul 7, passed Jul 5, in a 4-run window on main (schedule).
• Scope: Host/infra on the linux-mi35x-gpu-8-zf775 node pool; affects any 8-GPU MI35x nightly job scheduled onto a node with leaked VRAM. This run's DeepSeek-V4-Pro suite got zero coverage.
• In-flight fix: ✅ #30431 (open, DO NOT MERGE / WIP) — self-heal for driver-level VRAM leaks.

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-07-08 20:50 UTC)

[amd-bot]

nightly-test-amd-rocm720.yml — Per-Job Analysis (continued 4/10)

nightly-8-gpu-mi35x-deepseek-v4-flash-rocm720 — failed step(s): Accuracy + Performance Test MI35x ROCm 7.2 (8-GPU DeepSeek-V4-Flash FP8 + FP4, triton), Ensure VRAM is clear

Commit Info

• sglang: 11cea29c
• aiter: N/A — job aborted at the pre-test Ensure VRAM is clear step; the dependency-install step (which emits [CI-AITER-CHECK]) never ran, and this is not a scout run (AITER_COMMIT_OVERRIDE empty).

Failed Tests

Test File	Test Function	Error	Log
N/A (infra)	N/A	Ensure VRAM is clear step failed: driver/KFD-level VRAM leak (16% on GPU[0-3], threshold 5%), cleanup unsuccessful after 3 attempts → exit 1	Log

No test failed. The suite nightly-amd-8-gpu-mi35x-deepseek-v4-flash (run_suite.py --suite nightly-amd-8-gpu-mi35x-deepseek-v4-flash) never executed — the job died at the pre-test host-cleanup gate.

Failure Summary

The job failed before any test ran. The scripts/ci/amd/ensure_vram_clear.sh rocm pre-check found 16% VRAM already allocated on GPU[0-3] of runner linux-mi35x-gpu-8-zf775-runner-v5kbp, with no owning process (No KFD PIDs currently running, no containers, fuser/lsof show no holders). Three cleanup+wait cycles failed to reclaim it, so the step exited 1 and the subsequent run_suite.py step was aborted.

Regression Status

Infrastructure issue (recurring runner VRAM leak) — not a code regression.

Recent history of the Ensure VRAM is clear gate for job nightly-8-gpu-mi35x-deepseek-v4-flash-rocm720 (same branch main, event=schedule, completed runs only):

Date	Run	Job	Gate Status	Failure	Error
Jul 7	28888218691	v4-flash-rocm720	❌ Failed	VRAM-clear gate	zombie KFD VRAM leak, node ...-v5kbp
Jul 5	28749501468	v4-flash-rocm720	❌ Failed	VRAM-clear gate	identical leak, node ...-dvfd9
Jul 4	28714556066	v4-flash-rocm720	✅ Passed	—	—
Jul 3	28676295311	v4-flash-rocm720	✅ Passed	—	—

First observed failure of this gate for this job: Jul 5 2026 (verified — Jul 5 log shows the same === FAILED: VRAM cleanup unsuccessful after 3 attempts ===). Last passing: Jul 4 2026. (Jul 6 run 28813124510 was cancelled, excluded.)

Failure Cluster

Driver/KFD-level VRAM leak blocking pre-test GPU cleanup (ROCm/aiter#2061 signature)

Facts (verified observations)

• ERROR: VRAM usage exceeds threshold (5%) on some GPUs: GPU[0..3] : GPU Memory Allocated (VRAM%): 16 — repeated across all 3 cleanup attempts (log lines 167-171, 219-223, 328-332, 437-441).
• Leak has no owning process: No KFD PIDs currently running, No docker containers found on host, fuser/lsof list no GPU-device holders (lines 264-278, 373-387, 482-496).
• Final verdict at line 537: === FAILED: VRAM cleanup unsuccessful after 3 attempts ===; the script itself points to ROCm/aiter#2061 and states "The node will need to be rebooted" (lines 542-543). ##[error]Process completed with exit code 1 at line 545.
• The run_suite.py step then reports Error response from daemon: No such container: ci_sglang and Exit code 1 indicates test failure ... not retrying (lines 561-564) — i.e. the container was never started; no test ran.
• Same failure signature in the Jul 5 run on a different ephemeral runner name in the same node pool (zf775-runner-dvfd9 vs today's zf775-runner-v5kbp).

Hypothesised Causes (with confidence)

• [FACT] The job failed because of an unreclaimable driver/KFD-level VRAM leak on the MI35x runner host, not because of any code under test. The affected GPUs held VRAM with no owning process, and the cleanup script cannot recover this state without a node reboot.
  • Supporting: verbatim log evidence above; the script's own diagnostic names the [Bug] Custom all-reduce IPC buffers use fixed VA, conflict with torch_memory_saver pause/resume → zombie processes on MI300X ROCm/aiter#2061 zombie-KFD-context signature; the test suite never started.
  • Disconfirming: (none found) — there is no test output, stack trace, or model error anywhere in the log to suggest an sglang code fault.
  • How to verify: reboot/recycle the zf775 MI35x node, then re-run the job; expect a green VRAM gate.
• [LOW] The leak may be seeded by a prior job on the same node that crashed without gracefully stopping its ci_sglang container, leaving a zombie KFD context.
  • Supporting: in-flight PR #30431's root-cause description matches ("ci_sglang container was never gracefully stopped"); many sibling mi35x jobs also fail this window.
  • Disconfirming: this job's own log cannot identify the seeding job — attribution requires the previous job's log on the same physical node.
  • How to verify: correlate the node hostname across consecutive jobs and inspect the preceding job's shutdown path.

In-flight Fix Check

• In-flight fix: ✅ #30431 — "[DO NOT MERGE][AMD] ci: self-heal driver-level VRAM leaks + graceful GPU cleanup on shutdown" (open, created 2026-07-07). Directly targets this exact leak (cites [Bug] Custom all-reduce IPC buffers use fixed VA, conflict with torch_memory_saver pause/resume → zombie processes on MI300X ROCm/aiter#2061; adds self-heal + graceful ci_sglang shutdown). Currently marked DO NOT MERGE → track it, do NOT open a duplicate. (Search: search/issues?q=repo:sgl-project/sglang+is:pr+is:open+VRAM+cleanup.)

Suggested Triage Steps

• Reboot / recycle the affected MI35x node (zf775 pool) to clear the zombie KFD context, then re-run run 28888218691 — the VRAM gate should pass and let the deepseek-v4-flash suite actually run.
• Chase review/landing of #30431 (once un-flagged from DO-NOT-MERGE) — it converts this hard-fail into self-heal + graceful cleanup, addressing the whole cluster of mi35x nightly failures.
• Identify the job that ran on this node immediately before and check whether its ci_sglang container exited cleanly; if not, that is the leak source.

Status (factual)

• Persistence: VRAM-clear gate failing for this job on ≥2 completed scheduled runs (Jul 5 and Jul 7); passed Jul 3-4. Recurring, node-dependent.
• Scope: Node/pool-level — affects any job landing on the leaked zf775 MI35x node; numerous sibling *-mi35x-*-rocm720 jobs fail in the same window. The specific gate failure is host cleanup, upstream of all tests.
• Blocked work: The entire nightly-amd-8-gpu-mi35x-deepseek-v4-flash suite gets zero signal until the node is cleared.
• In-flight fix: ✅ #30431 (open 2026-07-07, DO-NOT-MERGE) — self-heal VRAM leak + graceful GPU cleanup.

nightly-8-gpu-mi35x-glm51-rocm720 — failed step(s): Ensure VRAM is clear

Commit Info

• sglang: 11cea29c
• aiter: N/A — job died at the pre-flight VRAM check before dependency install; no [CI-AITER-CHECK] markers in the log.

Failed Tests

Test File	Test Function	Error	Log
N/A (no test ran)	N/A	FAILED: VRAM cleanup unsuccessful after 3 attempts → exit 1	Log

This is not a test failure. The job failed in the very first host-side step — bash scripts/ci/amd/ensure_vram_clear.sh rocm — before the container, dependencies, or any test were started.

Failure Summary

The pre-flight VRAM sanity check on runner linux-mi35x-gpu-8-zf775-runner-r655q found GPUs 0–3 already holding 15–16% VRAM at job start (threshold is 5%). The cleanup script killed GPU-holding processes and waited 30s three times, but VRAM never dropped. No host processes, no docker containers, and no KFD PIDs were found holding the memory, so the script exited 1 and the job aborted. No sglang code was exercised.

Regression Status

Known recurring infrastructure failure (host/runner-level, node-pool zf775), intermittent.

Recent history of the nightly-8-gpu-mi35x-glm51-rocm720 job (branch main, event schedule, completed runs only):

Date	Run	Job Status	Runner	Cause
Jul 7	28888218691	❌ Failed	zf775-...-r655q	VRAM cleanup unsuccessful (this run)
Jul 5	28749501468	❌ Failed	zf775-...-87mr8	Same VRAM cleanup unsuccessful
Jul 4	28714556066	✅ Passed	zf775-...-nzbcz	—
Jul 3	28676295311	❌ Failed	zf775-...-ntwbh	Same VRAM cleanup unsuccessful
Jul 2	28611268103	✅ Passed	zf775-...-k2z2w	—

The failure toggles run-to-run and is tied to the physical GPU node the ephemeral runner lands on, not to any sglang commit. Confirmed ([FACT]) that the Jul 3 and Jul 5 glm51 failures hit the identical === FAILED: VRAM cleanup unsuccessful after 3 attempts === message.

Failure Cluster

Pre-flight VRAM cleanup failure (residual/zombie KFD VRAM on mi35x zf775 node).

Facts (verified observations)

• Log line 167: ERROR: VRAM usage exceeds threshold (5%) on some GPUs; GPUs 0=16%, 1=15%, 2=15%, 3=16% at job start (before any workload).
• Three cleanup attempts (lines 210–536): each sends SIGTERM/SIGKILL to GPU-device holders and waits 30s; VRAM stays at 15–16% on GPUs 0–3.
• Diagnostics show no holders: No KFD PIDs currently running, No docker containers found on host, empty fuser, no zombie processes (lines 264–318, 372–427, 481–536).
• Final exit at line 545: ##[error]Process completed with exit code 1.
• The script's own hint (lines 540–543): "if no host process / container holds the GPU but VRAM is still allocated, this is almost certainly a zombie KFD context (see ROCm/aiter#2061). The node will need to be rebooted."
• Same failure signature verified in Jul 5 and Jul 3 glm51 jobs.

Hypothesised Causes

• [HIGH] Zombie KFD/driver-level VRAM context left on physical node zf775 by a prior job, which the userspace cleanup script cannot reclaim (only a node reboot clears it).
  • Supporting: 15–16% VRAM present on GPUs 0–3 at start with zero identifiable holders (no KFD PIDs, no containers, no fuser hits); script's built-in hint points to exactly this scenario and cites ROCm/aiter#2061; reproduced across three separate runs on the same zf775 node pool.
  • Disconfirming: (none found) — no evidence implicates any sglang commit; the passing Jul 2/Jul 4 runs used the same job on the same pool, differing only by which node the ephemeral runner landed on.
  • How to verify: reboot / drain node linux-mi35x-gpu-8-zf775 and re-run; confirm VRAM reads 0% at next job start.

No hypothesis implicates the sglang commit under test — this is a host/runner-state failure, not a code regression.

In-flight Fix Check

• In-flight fix: ✅ #30431 "[DO NOT MERGE][AMD] ci: self-heal driver-level VRAM leaks + graceful GPU cleanup on shutdown" (open, draft, created 2026-07-07) — directly targets this driver-level VRAM-leak class. Still WIP/DO-NOT-MERGE; chase it rather than opening a duplicate.
• Search used: repo:sgl-project/sglang is:pr is:open VRAM clear (16 hits; #30431 is the direct match).

Suggested Triage Steps

• Reboot or drain the physical node behind the linux-mi35x-gpu-8-zf775-runner-* pool to clear the stuck KFD VRAM context, then re-run this job.
• Track ROCm/aiter#2061 and in-flight #30431 as the durable fix for driver-level VRAM self-heal.
• Consider adding an automated node-reboot/quarantine hook when ensure_vram_clear.sh exhausts its 3 attempts with no identifiable holder, so subsequent jobs don't keep landing on a poisoned node.
• No sglang code investigation needed — commit 11cea29c is not implicated.

Status

• Persistence: Intermittent on the zf775 mi35x node pool — failed Jul 3, Jul 5, Jul 7; passed Jul 2, Jul 4. Depends on which ephemeral node the runner lands on.
• Scope: Host/runner-level; any 8-GPU mi35x job scheduled onto the poisoned zf775 node is at risk (the pool also backs many other nightly-8-gpu-mi35x-*-rocm720 jobs). No test coverage lost due to code — signal is lost due to infra.
• Blocked work: GLM-5.1 nightly accuracy/perf signal on AMD MI35x ROCm 7.2 is blocked whenever the runner lands on the affected node.
• In-flight fix: ✅ #30431 (draft, opened 2026-07-07) — driver-level VRAM self-heal; not yet mergeable.

nightly-8-gpu-mi35x-glm5-mxfp4-rocm720 — failed step(s): Ensure VRAM is clear

Commit Info

• sglang: 11cea29c
• aiter: N/A — the job failed at a host-level pre-test step (ensure_vram_clear.sh) before the container was ever started, so no [CI-AITER-CHECK] markers were emitted. AITER_COMMIT_OVERRIDE was empty (not a scout run).

Failed Tests

Test File	Test Function	Error	Log
N/A (infrastructure — no test ran)	N/A	VRAM cleanup unsuccessful after 3 attempts → exit code 1	Log

This is not a test failure. The job died in the very first host step (bash scripts/ci/amd/ensure_vram_clear.sh rocm); the SGLang container was never launched and no test suite executed.

Failure Summary

The runner linux-mi35x-gpu-8-zf775-runner-rvm97 entered the job with 75–91% VRAM already allocated on all 8 GPUs (before checkout even finished). The pre-test VRAM-clear guard ran 3 cleanup attempts (SIGTERM → SIGKILL → 30 s wait, each attempt), none of which freed the memory, and the step exited 1. The diagnostics show the memory is held by leftover KFD contexts with no owning host process, i.e. zombie GPU contexts — the script itself points to ROCm/aiter#2061 and notes the node must be rebooted.

Regression Status

Known recurring infrastructure failure (not a code regression).

Recent history of job nightly-8-gpu-mi35x-glm5-mxfp4-rocm720 (workflow nightly-test-amd-rocm720.yml, branch main, event schedule, completed runs only):

Date	Run	Test/Step Status	Cause	Runner
Jul 7	28888218691	❌ Failed	VRAM cleanup failed (zombie KFD)	...runner-rvm97
Jul 5	28749501468	✅ Passed	—	—
Jul 4	28714556066	✅ Passed	—	—
Jul 3	28676295311	❌ Failed	Same VRAM cleanup failure (zombie KFD)	...runner-2k82h

The same ensure_vram_clear.sh failure hit this job on Jul 3 (verified: === FAILED: VRAM cleanup unsuccessful after 3 attempts ===), on a different zf775 runner (...2k82h). The job passed on Jul 4 and Jul 5. This is intermittent runner/node contamination, not tied to the sglang commit under test — the head SHA changes every run but the failure is a pre-existing GPU state left on the physical node.

Failure Cluster

VRAM cleanup failure on AMD runner — zombie KFD contexts (node needs reboot)

Facts (verified observations, no interpretation)

• Step that failed: Run bash scripts/ci/amd/ensure_vram_clear.sh rocm (log lines 110–639); ##[error]Process completed with exit code 1 at line 639.
• Initial GPU state (before any job work), lines 131–162: VRAM allocated GPU0 80%, GPU1 89%, GPU2 91%, GPU3 91%, GPU4–7 75% — already above the 5% threshold.
• 3 cleanup attempts each did SIGTERM → SIGKILL → 30 s wait; VRAM unchanged across all three (attempt-1 lines 214–270, attempt-2 353–409, attempt-3 492–548).
• rocm-smi --showpids (lines 285–310, repeated) lists persistent PIDs 875199, 875200, 875201, 875203, 875214, 875215, 875234, 875252 all on GPU 1, each holding ~231–232 GB VRAM, PROCESS NAME = UNKNOWN, CU OCCUPANCY = 0.
• fuser shows no GPU device-file holders (lines 272–273), lsof not installed, no host processes, and No docker containers found on host (lines 217, 356, 495). Memory is allocated but unowned by any live host process → zombie KFD contexts.
• Script's own hint (lines 634–637): "if no host process / container holds the GPU but VRAM is still allocated, this is almost certainly a zombie KFD context (see [Bug] Custom all-reduce IPC buffers use fixed VA, conflict with torch_memory_saver pause/resume → zombie processes on MI300X ROCm/aiter#2061). The node will need to be rebooted."
• Runner: linux-mi35x-gpu-8-zf775-runner-rvm97 (line 2). Jul 3 recurrence was on linux-mi35x-gpu-8-zf775-runner-2k82h.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [FACT] The job failed because the AMD MI35x node had leftover VRAM from zombie KFD contexts that the cleanup script cannot reclaim (kill signals have no effect on unowned KFD contexts; only a node reboot clears them).
  • Supporting: all 8 GPUs pre-allocated 75–91%; showpids lists UNKNOWN-name PIDs holding ~232 GB each on GPU 1; no host process/container owns them; SIGKILL had no effect across 3 attempts.
  • Disconfirming: (none found) — the evidence is directly in the log.
• [FACT] This is unrelated to the sglang commit under test.
  • Supporting: the failure occurs before the container/test runs; the same job passed on Jul 4/Jul 5 with different head SHAs; the identical failure recurred Jul 3 on a different runner.
  • Disconfirming: (none found).
• [LOW] The zombie context was left behind by a previous job on the same physical node (a prior SGLang run that crashed without releasing GPU memory).
  • Supporting: the zf775 node pool shows this repeatedly; [Bug] Custom all-reduce IPC buffers use fixed VA, conflict with torch_memory_saver pause/resume → zombie processes on MI300X ROCm/aiter#2061 documents this leak pattern.
  • Disconfirming: this log does not contain the prior job's output, so the specific leaking job cannot be identified from here.

In-flight Fix Check

• In-flight fix: ⚠️ [#30431](https://github.com/sgl-project/sglang/pull/30431) "[DO NOT MERGE][AMD] ci: self-heal driver-level VRAM leaks + graceful GPU cleanup on shutdown" — directly targets this driver-level VRAM-leak/cleanup class, but is currently marked DO NOT MERGE (WIP). Track it rather than opening a duplicate.
• Searched via search/issues?q=...+is:pr+is:open+VRAM+cleanup and ...+KFD.

Suggested Triage Steps

• Reboot / cordon the affected node(s) linux-mi35x-gpu-8-zf775-runner-rvm97 (and ...2k82h seen Jul 3) to clear the zombie KFD contexts, per the script hint and ROCm/aiter#2061.
• Identify the previous job that ran on zf775-runner-rvm97 before this run and check whether it crashed without releasing GPU memory — that upstream crash is the likely leak source.
• Follow / help land #30431 (self-heal VRAM leaks + graceful GPU cleanup on shutdown), which is the structural fix for this recurring zf775-pool contamination.
• If node reboots are not automatable, consider adding a reboot-on-detect hook when ensure_vram_clear.sh detects unowned KFD contexts, so a poisoned node auto-recovers instead of failing every subsequent scheduled job.

Status (factual, no priority assignment)

• Persistence: Intermittent — failed Jul 7 and Jul 3 (same VRAM-cleanup step, different zf775 runners); passed Jul 4 and Jul 5. Node-state dependent, not every run.
• Scope: This job (nightly-8-gpu-mi35x-glm5-mxfp4-rocm720) on the affected node; any job scheduled onto a poisoned zf775 MI35x node is at risk until the node is rebooted.
• Blocked work: The GLM5 MXFP4 8-GPU ROCm 7.2 nightly signal is missing for this run (no test executed).
• In-flight fix: ⚠️ #30431 (DO NOT MERGE / WIP) — structural self-heal for driver-level VRAM leaks.

nightly-8-gpu-mi35x-qwen35-rocm720 — failed step(s): Accuracy Test MI35x ROCm 7.2 (8-GPU Qwen 3.5)

Commit Info

• sglang: 11cea29c
• aiter: N/A (no [CI-AITER-CHECK] markers in log; aiter loaded from the pre-built image rocm/sgl-dev:v0.5.14-rocm720-mi35x-20260707, AITER_COMMIT_OVERRIDE empty)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi35x/test_qwen35_eval_mi35x.py	TestQwen35EvalMI35x.test_lm_eval	AssertionError: extra_buffer needs CUDA/MUSA/NPU (FLA). (server exits code 1 at startup → retry() exceed maximum number of retries)	Log

Failure Summary

The single test test_lm_eval in test_qwen35_eval_mi35x.py errored because the sglang serve subprocess for Qwen/Qwen3.5-397B-A17B exited with code 1 during argument resolution — before the server ever came up. The crash is an AssertionError: extra_buffer needs CUDA/MUSA/NPU (FLA). raised in ServerArgs.__post_init__ while resolving the mamba radix cache for this hybrid-mamba model. The test harness retried once, hit the same crash, and reported FAILED (errors=1).

Regression Status

Never-passed (no green run found in queryable window) — this is a standing ROCm feature gap, not a fresh regression.

Recent history of test_qwen35_eval_mi35x.py (test_lm_eval) in job nightly-8-gpu-mi35x-qwen35-rocm720 (branch main, event schedule, completed runs only):

Date	Run	Job	Test File Status	Failed Function	Error
Jul 07	run	qwen35-rocm720	❌ Failed	test_lm_eval	extra_buffer needs CUDA/MUSA/NPU (FLA)
Jul 05	run	qwen35-rocm720	❌ Failed	test_lm_eval	same assertion
Jul 02	run	qwen35-rocm720	❌ Failed	test_lm_eval	same assertion
Jun 29	run	qwen35-rocm720	❌ Failed	test_lm_eval	same assertion

Lookback window searched: 2026-06-20 → 2026-07-07 (all completed scheduled runs; 07-01, 06-30 were cancelled and excluded). Every completed run in this window failed with the identical assertion on the same test file. First observed failure in window: 2026-06-20 or earlier; no passing run found.

Failure Cluster

mamba radix cache extra_buffer unsupported on ROCm/HIP (assertion at server startup)

Facts (verified observations, no interpretation)

• Verbatim crash (job log, 2026-07-07 18:33:29): AssertionError: extra_buffer needs CUDA/MUSA/NPU (FLA).
• Crash call chain (from log): server_args.py:2734 __post_init__ → :4247 _handle_model_specific_adjustments → :4318 _handle_mamba_radix_cache → :4289 _validate_mamba_extra_buffer.
• The failing assertion is assert (is_cuda() or is_musa() or is_npu()), "extra_buffer needs CUDA/MUSA/NPU (FLA)." at python/sglang/srt/server_args.py:4289-4291.
• is_cuda() is defined as torch.cuda.is_available() and torch.version.cuda is not None (python/sglang/srt/utils/common.py:144-145). On a ROCm build torch.version.cuda is None (torch.version.hip is set), so is_cuda() → False; is_musa()/is_npu() are also False on MI35x → assertion fails. [FACT]
• Launch command (from log): sglang serve --model-path Qwen/Qwen3.5-397B-A17B --tp 8 --attention-backend aiter ... --device cuda — no explicit mamba-cache-strategy flag, so the default resolves to extra_buffer.
• Sister Qwen3.5 job nightly-8-gpu-mi35x-qwen35-triton-dcp-rocm720 PASSED in the same runs (Jul 05 job 85246031452 ✅, Jul 02 job 84844074402 ✅) — a differently-configured Qwen3.5 run on the same hardware does not hit the assertion.
• Reproducibility: identical failure in 4/4 completed scheduled runs checked (Jun 29, Jul 02, Jul 05, Jul 07). The build step and model download succeeded; the failure is purely arg-validation at server startup.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [HIGH] The extra_buffer mamba-radix-cache strategy is not implemented for ROCm/HIP; Qwen/Qwen3.5-397B-A17B is a hybrid-mamba model whose default cache resolution selects extra_buffer, and the platform guard at server_args.py:4289 intentionally rejects it on non-CUDA/MUSA/NPU platforms.
  • Supporting: assertion text + exact code location + is_cuda() returning False on ROCm are all directly verified above; multiple open PRs explicitly add ROCm support for this exact path (see In-flight Fix Check).
  • Disconfirming: none found — this is a platform capability gap, not a code regression. (The passing qwen35-triton-dcp sister job strengthens, rather than weakens, this: it avoids the default extra_buffer path.)
  • How to verify: launch the same command on MI35x with an explicit no-buffer strategy (e.g. the flag the triton-dcp variant uses) and confirm the server starts; or apply one of the in-flight PRs and re-run.
• [LOW] Test-config gap: the test does not pin a ROCm-compatible mamba cache strategy, so it inherits the CUDA-oriented extra_buffer default.
  • Supporting: the sibling qwen35-triton-dcp test on identical hardware passes with a different config.
  • Disconfirming: even if the test pinned no_buffer, the underlying feature (ROCm extra_buffer) would still be missing — this is a workaround, not the root cause.
  • How to verify: diff the two test configs (test_qwen35_eval_mi35x.py vs the triton-dcp variant) to see which mamba flags differ.

In-flight Fix Check

Search: is:pr is:open extra_buffer and is:pr is:open mamba extra_buffer rocm on sgl-project/sglang. Directly-matching open PRs exist:

• In-flight fix: ✅ #30359 — "[ROCm] Support mamba extra buffer for Qwen3.5" (open, created 2026-07-07). Most on-point; chase reviewers, do NOT open a duplicate.
• In-flight fix: ✅ #27141 — "[ROCm] Support radix-cache with mamba-extra-buffer for Qwen3.5" (open since 2026-06-03, last updated 2026-06-29).
• In-flight fix: ✅ #29965 — "[AMD] Fix mamba radix cache auto strategy crashing on ROCm (fall back to no_buffer)" (open since 2026-07-02) — addresses the crash directly via a no_buffer fallback.
• Related: #29832 "[AMD] Enable mamba extra_buffer on ROCm".

Suggested Triage Steps

• Verify whether #30359 (or #27141) fully covers Qwen/Qwen3.5-397B-A17B on MI35x, apply on a branch, and re-run test_qwen35_eval_mi35x.py in container rocm/sgl-dev:v0.5.14-rocm720-mi35x-20260707.
• As an interim signal-unblock, evaluate #29965's no_buffer fallback: it would let the server start on ROCm even before the full extra_buffer port lands.
• Compare test_qwen35_eval_mi35x.py against the passing qwen35-triton-dcp variant to see which mamba-cache flag the passing config sets; consider pinning the same ROCm-safe strategy in this test until the feature ports.

Status (factual, no priority assignment)

• Persistence: Failing every completed scheduled run in the queryable window (2026-06-20 → 2026-07-07); no green run found — effectively never-passed on ROCm 7.2 MI35x.
• Scope: nightly-8-gpu-mi35x-qwen35-rocm720 (this job). A sibling job nightly-8-gpu-qwen35-rocm720 also fails in the same runs; the nightly-8-gpu-mi35x-qwen35-triton-dcp-rocm720 Qwen3.5 variant passes.
• Blocked work: Blocks Qwen3.5-397B-A17B accuracy signal on ROCm 7.2 MI35x with the default (aiter + extra_buffer) config.
• In-flight fix: ✅ #30359 / #27141 (ROCm extra_buffer support) and #29965 (no_buffer fallback) — all open, awaiting merge.

nightly-8-gpu-mi35x-kimi-k26-rocm720 — failed step(s): Accuracy Test MI35x ROCm 7.2 (8-GPU Kimi-K2.6)

Commit Info

• sglang: 11cea29c
• aiter: 9127c94a — v0.1.17.dev110+g9127c94a1 (source: dev/patched version detected in CI image, rebuild skipped; AITER_COMMIT_OVERRIDE empty)
• CI image: rocm/sgl-dev:v0.5.14-rocm720-mi35x-20260707 (mi35x / gfx950, rebuilt 2026-07-07)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi35x/test_kimi_k26_eval_mi35x.py	TestKimiK26EvalMI35x.test_kimi_k26_gsm8k_accuracy	Exception: Server process exited with code 1 → retry() exceed maximum number of retries (server launch aborted by ValueError: KimiK25ForConditionalGeneration has no SGlang implementation)	Log

Failure Summary

The single test test_kimi_k26_gsm8k_accuracy errored because the SGLang server for moonshotai/Kimi-K2.6 (--tp 8) could not launch. During tokenizer/model init the loader raised ValueError: KimiK25ForConditionalGeneration has no SGlang implementation and the Transformers implementation is not compatible with SGLang. The launch was retried 4×, all with the identical error, then the test failed (0/1 passed, exit code 255).

Regression Status

New regression (first observed 2026-07-07) — but note the intervening runs were blocked by a separate infra failure, so the last time this test actually executed it passed.

Recent history of test_kimi_k26_eval_mi35x.py (test_kimi_k26_gsm8k_accuracy) in job nightly-8-gpu-mi35x-kimi-k26-rocm720 (branch main, event schedule, completed runs only):

Date	Run	Job	Test File Status	Failed Function	Error
Jul 7	28888218691	job	❌ Failed	test_kimi_k26_gsm8k_accuracy	KimiK25...has no SGlang implementation (cutlass import)
Jul 6	28813124510	—	⏭️ Run cancelled	—	(excluded — not completed to test)
Jul 5	28749501468	job	⚠️ Failed pre-test	—	VRAM usage exceeds threshold (infra; test never ran)
Jul 4	28714556066	job	⚠️ Failed pre-test	—	VRAM usage exceeds threshold (infra; test never ran)
Jul 3	28676295311	job	✅ Passed	—	—

• First observed failure (this cutlass symptom): 2026-07-07.
• Last known passing (test actually executed): 2026-07-03 (85050304358).
• Jul 4 & Jul 5 were a different failure (runner VRAM cleanup) that stopped the job before model loading; Jul 6's run was cancelled.

Failure Cluster

cutlass import failure → model arch not registered → server launch abort (mi35x rocm720 20260707 image)

Facts (verified observations)

• [FACT] Server abort trace (job log line 3969): ValueError: KimiK25ForConditionalGeneration has no SGlang implementation and the Transformers implementation is not compatible with SGLang, raised in python/sglang/srt/model_loader/utils.py:182 (resolve_transformers_arch). Same error on all 4 launch retries.
• [FACT] The job log contains 164 occurrences of Ignore import error when loading … No module named 'cutlass', including sglang.srt.models.kimi_k25 (line 3929) and sglang.srt.models.deepseek_v2 (line 3915), plus the whole deepseek/kimi/bailing/ernie family.
• [FACT] Code chain: python/sglang/srt/models/kimi_k25.py:30 imports DeepseekV3ForCausalLM from deepseek_v2; kimi_k25.py:885 registers EntryClass = [KimiK25ForConditionalGeneration]. If kimi_k25/deepseek_v2 fail to import during the registry scan (models/registry.py:105-109 swallows the error), the arch is never added to ModelRegistry.
• [FACT] get_model_architecture (model_loader/utils.py:216) sets is_native_supported = any(arch in supported_archs …); with KimiK25ForConditionalGeneration absent it falls through to resolve_transformers_arch, which raises the observed ValueError.
• [FACT] A/B across images: the passing mi35x job nightly-8-gpu-mi35x-deepseek-v4-pro-rocm720 on 2026-07-05 (image …-mi35x-20260705, 157,445-line log, 85246031426) had 0 No module named 'cutlass' errors; today's …-mi35x-20260707 image has 164.
• [FACT] cutlass is imported at module top level in python/sglang/srt/layers/quantization/nvfp4_gemm_swiglu_nvfp4_quant.py:23 (and blackwell kernel modules). The nvfp4 imports inside deepseek_v2.py:303 / modelopt_quant.py:1623 are lazy, so the module-load failure originates from a top-level import cutlass reached transitively during the registry scan.
• [FACT] Not an aiter-triggered run: AITER_COMMIT_OVERRIDE empty; workflow is nightly-test-amd-rocm720.yml, not the AITER Scout.

Hypothesised Causes (with confidence)

• [HIGH] The mi35x ROCm 7.2 image rebuilt 2026-07-07 (rocm/sgl-dev:v0.5.14-rocm720-mi35x-20260707) is missing / has a broken cutlass Python package. This makes import cutlass fail, cascading through the registry scan so the entire DeepSeek/Kimi model family (incl. KimiK25ForConditionalGeneration) is dropped from ModelRegistry, and Kimi-K2.6 can no longer be served.
  • Supporting: 0 cutlass import errors on the passing 20260705 image vs 164 on today's 20260707 image; the error message names the exact arch that lives in the un-importable kimi_k25 module; the same sglang code passed this test on Jul 3.
  • Disconfirming: (none found) — the failing code path is fully explained by a missing runtime dependency, and the last executing run (Jul 3, older sglang, older image) passed, pointing at the image env rather than sglang source.
  • How to verify: in the 20260707 mi35x container run python -c "import cutlass" (expect ModuleNotFound) and python -c "from sglang.srt.models.registry import ModelRegistry as R; print('KimiK25ForConditionalGeneration' in R.get_supported_archs())" (expect False); compare pip show nvidia-cutlass-dsl / pip list | grep -i cutlass between the 20260705 and 20260707 images.
• [LOW] An sglang commit in 11cea29c (or the Jul3→Jul7 window) moved a cutlass import to a hot top-level path.
  • Supporting: a top-level import cutlass exists in nvfp4_gemm_swiglu_nvfp4_quant.py.
  • Disconfirming: the passing 20260705 job exercised the same model modules with 0 cutlass errors, so the import path itself was satisfiable when the package was present — the differentiator is the image, not the code.

In-flight Fix Check

• In-flight fix: ❌ none found targeting "mi35x image missing cutlass". Related but not this issue: #30410 "[NPU] Fix npu cutlass import" (NPU-specific), #30448 "Refactor FP4 quantization and remove deprecated JIT kernels", #30443. Searched repo:sgl-project/sglang is:pr is:open cutlass (118 hits) and …+mi35x+cutlass (1 hit, unrelated). Root cause is the CI image build, which is not necessarily fixed by an sglang PR.

Suggested Triage Steps

• Confirm the environment regression: exec into the …-mi35x-20260707 image and run the two import checks above; diff pip list vs the …-mi35x-20260705 image to find what dropped cutlass.
• Inspect the mi35x/gfx950 image Dockerfile & build logs for 2026-07-07 to see why the cutlass (NVFP4 CUTLASS DSL) package is absent/broken; restore it or repin the image tag.
• Defensive sglang-side option to evaluate: guard the top-level import cutlass in nvfp4_gemm_swiglu_nvfp4_quant.py (and blackwell kernels) so a missing NVIDIA-only cutlass on ROCm does not cascade to drop unrelated model archs from the registry.
• Once the image is fixed, re-run nightly-8-gpu-mi35x-kimi-k26-rocm720; also spot-check other mi35x model jobs that reached model load today.

Status

• Persistence: First occurrence of this cutlass symptom on 2026-07-07 (1 completed run). The test last executed successfully on 2026-07-03; Jul 4–5 were blocked by an unrelated VRAM-cleanup infra failure and Jul 6 was cancelled.
• Scope: Confirmed for this Kimi-K2.6 mi35x job. Potentially affects any mi35x model test that reaches model loading on the 20260707 image (DeepSeek/Kimi family import blocked), but this cannot be confirmed fleet-wide from today's run because many sibling mi35x jobs failed earlier at the VRAM-cleanup step and never loaded a model.
• Blocked work: Kimi-K2.6 accuracy signal on mi35x ROCm 7.2.
• In-flight fix: ❌ none found — needs CI image investigation (see In-flight Fix Check).

nightly-8-gpu-mi35x-qwen35-triton-dcp-rocm720 — failed step(s): Ensure VRAM is clear

Commit Info

• sglang: 11cea29c
• aiter: N/A (job failed in the pre-flight host cleanup step before any container/dependency setup; no [CI-AITER-CHECK] markers in log)

Failed Tests

Test File	Test Function	Error	Log
N/A (no test ran)	N/A	Pre-flight step scripts/ci/amd/ensure_vram_clear.sh rocm failed: VRAM cleanup unsuccessful after 3 attempts → exit 1	Log

Failure Summary

The job never started any test. It failed at the very first workflow step (Run bash scripts/ci/amd/ensure_vram_clear.sh rocm), which is a host-level pre-flight guard that requires all GPUs to be near-idle before the container launches. GPUs 0–3 each reported a stuck GPU Memory Allocated (VRAM%): 16 that persisted across all three cleanup attempts (SIGTERM → SIGKILL → 30 s wait, ×3), so the guard aborted with exit 1.

Regression Status

Infrastructure issue (recurring, node/runner-pool level — not a code regression).

Recent history of the pre-flight VRAM guard in job nightly-8-gpu-mi35x-qwen35-triton-dcp-rocm720 (branch main, event schedule, completed runs only):

Date	Run	Job	Step Status	Failure	Runner
Jul 7	28888218691	job	❌ Failed	VRAM cleanup unsuccessful (GPUs 0-3 @ 16%)	...zf775-runner-rfvwf
Jul 5	28749501468	job	✅ Passed	—	—
Jul 4	28714556066	job	❌ Failed	VRAM cleanup unsuccessful (GPUs 0-3 @ 16%)	...zf775-runner-cmblx
Jul 3	28676295311	job	❌ Failed	VRAM cleanup unsuccessful (GPUs 0-3 @ 16%)	...zf775-runner-dblhz

First observed (this window): Jul 3. Last passing: Jul 5. The failure recurs intermittently across different node names, all within the same linux-mi35x-gpu-8-zf775 runner pool.

Failure Cluster

Pre-flight VRAM guard abort — stuck 16% VRAM on MI35x GPUs 0-3 (zombie KFD context)

Facts (verified observations)

• The failing step is the host guard scripts/ci/amd/ensure_vram_clear.sh rocm, line 110; it exits 1 at line 545.
• GPUs 0,1,2,3 each read exactly GPU Memory Allocated (VRAM%): 16; GPUs 4-7 read 0. This is stable across all 3 attempts and ~1.5 min elapsed.
• No process is holding the memory: rocm-smi --showpids → No KFD PIDs currently running; fuser/lsof find no GPU device-file holders; No docker containers found on host; no zombie processes listed.
• The script's own hint (line 542): "almost certainly a zombie KFD context (see [Bug] Custom all-reduce IPC buffers use fixed VA, conflict with torch_memory_saver pause/resume → zombie processes on MI300X ROCm/aiter#2061). The node will need to be rebooted before subsequent jobs can succeed."
• Same signature reproduced on 3 distinct nodes (rfvwf Jul 7, cmblx Jul 4, dblhz Jul 3), all in the zf775 MI35x-8GPU pool.
• No [CI-AITER-CHECK] markers and no test output exist — execution never reached container/dependency setup.

Hypothesised Causes (with confidence)

• [HIGH] Zombie KFD context / driver-level VRAM leak on the MI35x runner node left GPUs 0-3 with unreclaimable memory from a prior job; the pre-flight guard correctly refused to launch on a dirty node.
  • Supporting: VRAM is allocated but zero processes/containers/KFD-PIDs hold it (verified via showpids, fuser, docker ps); the script's own diagnostics and hint point to ROCm/aiter#2061; identical stuck-16%-on-4-GPUs pattern on multiple nodes.
  • Disconfirming: (none found) — no evidence any sglang commit is involved; the code path is a host guard that runs before checkout-independent test code.
  • How to verify: reboot node linux-mi35x-gpu-8-zf775-runner-rfvwf (and any pool node showing residual VRAM) and re-run; confirm GPUs return to 0%.
• [LOW] sglang commit regression.
  • Supporting: none.
  • Disconfirming: failure is in a host pre-flight step that never runs sglang test code; passes/fails alternate on the same commit-independent guard across different nodes (Jul 5 passed).
  • How to verify: not actionable — no code path implicated.

In-flight Fix Check

• In-flight fix: ✅ #30431 "[DO NOT MERGE][AMD] ci: self-heal driver-level VRAM leaks + graceful GPU cleanup on shutdown" (open since 2026-07-07, currently marked DO NOT MERGE) — directly targets this driver-level VRAM-leak class. Track this PR; do not open a duplicate.
• Search run: search/issues?q=repo:sgl-project/sglang+is:pr+is:open+VRAM+clear (16 hits; #30431 is the on-point one).

Suggested Triage Steps

• Reboot the affected MI35x node(s) in the linux-mi35x-gpu-8-zf775 pool to clear the zombie KFD context, then re-run the job — this is the remediation the guard's own hint recommends.
• Track in-flight PR #30431 (self-heal VRAM leaks); help move it out of DO-NOT-MERGE if this failure is blocking nightly signal.
• Cross-reference ROCm/aiter#2061 for the underlying KFD-context leak; consider whether a prior job on these nodes is leaving GPUs 0-3 dirty (the consistent 16%-on-4-GPUs pattern suggests a specific 4-GPU workload leaking on teardown).
• No sglang code bisect is warranted — the failure is pre-test infrastructure.

Status

• Persistence: Intermittent infra failure; failed Jul 3, Jul 4, Jul 7 (passed Jul 5) on the same pre-flight guard across 3 different pool nodes.
• Scope: linux-mi35x-gpu-8-zf775 runner pool; blocks the nightly-8-gpu-mi35x-qwen35-triton-dcp-rocm720 job whenever it lands on a node with leaked VRAM. Other jobs scheduled on affected nodes would hit the same guard.
• Blocked work: Qwen3.5 Triton DCP nightly signal on MI35x ROCm 7.2 when a dirty node is picked.
• In-flight fix: ✅ #30431 (open 2026-07-07, DO NOT MERGE) — track/advance.

nightly-4-gpu-mi35x-minimax-m25-rocm720 — failed step(s): Ensure VRAM is clear

Commit Info

• sglang: 11cea29c
• aiter: N/A — job died in the pre-test host-side VRAM check before any container/dependency install; no [CI-AITER-CHECK] markers emitted. AITER_COMMIT_OVERRIDE was empty.

Failed Tests

Test File	Test Function	Error	Log
N/A (infra pre-check)	N/A	FAILED: VRAM cleanup unsuccessful after 3 attempts → exit 1	Log L537

No test executed. The job failed in the bash scripts/ci/amd/ensure_vram_clear.sh rocm step, which runs before the container starts and any MiniMax-M25 test is launched.

Failure Summary

The MiniMax-M25 4-GPU job never reached its test phase. At job start, GPUs 0–3 on the runner already had 15–16% VRAM allocated by a prior workload. ensure_vram_clear.sh ran three SIGTERM/SIGKILL + 30s-wait cleanup cycles; VRAM stayed pinned at the same 16/15/15/16% the entire time, so the guard aborted with exit 1. Notably, rocm-smi --showpids reports "No KFD PIDs currently running" and fuser/docker/zombie checks find no holder — the allocation has no owning process.

Regression Status

Known recurring infrastructure failure (not a code regression).

Recent history of the VRAM pre-check in job nightly-4-gpu-mi35x-minimax-m25-rocm720 (same branch main, event=schedule, completed runs only):

Date	Run	Job	Test File Status	Failed Function	Error
Jul 7	28888218691	minimax-m25	❌ Failed (infra)	VRAM pre-check	VRAM cleanup unsuccessful (runner ...c9ppr)
Jul 5	28749501468	minimax-m25	❌ Failed (infra)	VRAM pre-check	VRAM cleanup unsuccessful (runner ...rvjqx)
Jul 4	28714556066	minimax-m25	❌ Failed (infra)	VRAM pre-check	VRAM cleanup unsuccessful (runner ...4gkfd)
Jul 3	28676295311	minimax-m25	❌ Failed (infra)	VRAM pre-check	VRAM cleanup unsuccessful (runner ...whgk6)
Jul 2	28611268103	minimax-m25	✅ Passed	—	—

First observed failure: Jul 3 2026. Last known passing: Jul 2 2026. Jul 1 / Jun 30 runs were cancelled at the run level (not comparable). Note the four failures span four different runner nodes (whgk6, 4gkfd, rvjqx, c9ppr) — this is a fleet-level condition on the linux-mi35x-gpu-8-zf775 pool, not a single bad box.

Failure Cluster

Pre-test VRAM cleanup abort — orphaned KFD VRAM allocation on mi35x ROCm 7.2 runners.

Facts (verified observations, no interpretation)

• [FACT] Failure is in step Run bash scripts/ci/amd/ensure_vram_clear.sh rocm, exit code 1 (L545).
• [FACT] GPUs 0–3 held 16/15/15/16% VRAM at job start (L168–171) and remained unchanged after all 3 cleanup attempts (L438–441).
• [FACT] No owning process found: rocm-smi --showpids = "No KFD PIDs currently running" (L276); fuser empty, no docker containers, no zombies.
• [FACT] The script's own hint attributes this to a zombie KFD context and links ROCm/aiter#2061, stating the node must be rebooted (L540–543).
• [FACT] Identical failure signature in the Jul 3/4/5 runs (verified: each log contains one FAILED: VRAM cleanup unsuccessful after 3 attempts).

Hypothesised Causes (with confidence, with disconfirming evidence)

• [HIGH] Orphaned/zombie KFD VRAM allocation left by a prior job on the mi35x runner that no live process owns, so SIGTERM/SIGKILL cleanup cannot reclaim it — only a node reboot clears it.
  • Supporting: VRAM pinned at a fixed 15–16% with zero KFD PIDs, zero fuser holders, zero containers; the guard script itself documents this exact scenario and points to ROCm/aiter#2061.
  • Disconfirming: none found — the evidence is consistent with a driver-level leak; there is no test code or sglang commit in the failing path (the failure precedes checkout of any test).
  • How to verify: reboot/recycle the affected runner nodes and confirm rocm-smi shows 0% VRAM; re-run the job. The prior job on each node likely leaked — cross-reference the previous job scheduled on ...c9ppr to find the leaker.
• [LOW] A specific upstream test (possibly a large 4-GPU model job scheduled just before) crashes without releasing VRAM, seeding the leak.
  • Supporting: the leak is on GPUs 0–3 (the 4-GPU set this job would use), suggesting a prior 4-GPU workload.
  • Disconfirming: not visible in this job's log; requires inspecting the prior job on the same node.
  • How to verify: audit which job ran immediately before on each of the four affected runners.

In-flight Fix Check

• In-flight fix: ✅ #30431 "[DO NOT MERGE][AMD] ci: self-heal driver-level VRAM leaks + graceful GPU cleanup on shutdown" — directly targets driver-level VRAM leaks and GPU cleanup on shutdown; still marked DO-NOT-MERGE. Chase this PR; do NOT open a duplicate. (Search: is:pr is:open VRAM clear.)

Suggested Triage Steps

• Recycle/reboot the affected linux-mi35x-gpu-8-zf775 runner nodes (...c9ppr, and previously ...rvjqx/...4gkfd/...whgk6) to clear the zombie KFD contexts, per the script hint and ROCm/aiter#2061.
• Identify the leaking predecessor: inspect the job that ran immediately before this one on runner ...c9ppr to find which workload exits without freeing GPU 0–3 VRAM.
• Progress #30431 (self-heal VRAM leak) out of DO-NOT-MERGE state, since manual node reboots are not scaling (4 nodes affected in 4 days).
• This is infra, not code — no bisect of the sglang commit range is warranted; the failing step runs before any sglang/test code executes.

Status (factual, no priority assignment)

• Persistence: Failing every completed scheduled run since Jul 3 2026 (4 consecutive: Jul 3, 4, 5, 7). Last green Jul 2.
• Scope: 4 distinct runner nodes in the linux-mi35x-gpu-8-zf775 pool; blocks the MiniMax-M25 4-GPU ROCm 7.2 nightly signal. Likely also affects sibling mi35x jobs sharing the pool (not verified in this analysis).
• Blocked work: No MiniMax-M25 nightly accuracy/perf coverage on AMD ROCm 7.2 while the pool leaks VRAM.
• In-flight fix: ✅ #30431 (DO NOT MERGE) — self-heal driver-level VRAM leaks.

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-07-08 20:50 UTC)

[amd-bot]

nightly-test-amd-rocm720.yml — Per-Job Analysis (continued 5/10)

nightly-8-gpu-mi35x-deepseek-v4-pro-mtp-rocm720 — failed step(s): Accuracy + MTP Accept Length Test MI35x ROCm 7.2 (8-GPU DeepSeek-V4-Pro FP8 MTP, triton), Accuracy + MTP Accept Length Test MI35x ROCm 7.2 (8-GPU DeepSeek-V4-Pro FP8 MTP, unified_kv_triton)

Commit Info

• sglang: 11cea29c
• aiter: 9127c94a1 (source: dev image — [CI-AITER-CHECK] reports v0.1.17.dev110+g9127c94a1, "Dev/patched version detected → skipping rebuild"; Dockerfile default was the same SHA)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/test_deepseek_v4_pro_fp4_mtp.py	TestDeepseekV4ProFp4MTP.setUpClass	Exception: Server process exited with code -9 (server crash during model load)	Log

Failure Summary

The deepseek-v4-pro-mtp server failed to launch during setUpClass. The scheduler crashed while loading the model: sglang.srt.models.deepseek_v4 could not be imported (No module named 'cutlass'), so DeepseekV4ForCausalLM was not registered and sglang fell back to the Transformers backend, which then rejected the config: ValueError: Unrecognized configuration class _DeepseekV4ConfigAlias for this kind of AutoModel. The server process exited with code -9 and the test recorded 0/1 passed.

Regression Status

New regression (new today). This is a distinct failure from the Jul 4/Jul 5 job failures, which were pre-test VRAM-cleanup infrastructure failures where the test binary never ran.

Recent history of test/registered/amd/test_deepseek_v4_pro_fp4_mtp.py (TestDeepseekV4ProFp4MTP.setUpClass) in job nightly-8-gpu-mi35x-deepseek-v4-pro-mtp-rocm720 (branch=main, event=schedule, completed runs only):

Date	Run	Test File Status	Failed Function	Error
Jul 7	28888218691	❌ Failed	setUpClass	cutlass ImportError → Transformers fallback → exit -9
Jul 5	28749501468	⚠️ Infra (test never ran)	—	VRAM cleanup unsuccessful after 3 attempts (different failure)
Jul 4	28714556066	⚠️ Infra (test never ran)	—	VRAM cleanup unsuccessful after 3 attempts (different failure)
Jul 3	28676295311	✅ Passed	—	1/1 passed (elapsed 2492s); zero cutlass import errors in log

First observed (this failure mode): 2026-07-07. Last known passing (test-level): 2026-07-03. Regression window: 7820dc60a7..11cea29c.

Failure Cluster

cutlass ImportError → model registry fallback to Transformers → server crash (exit -9)

Facts (verified observations)

• [FACT] Log: Ignore import error when loading sglang.srt.models.deepseek_v4: No module named 'cutlass' (also deepseek_v4_nextn, deepseek_v2, and ~28 other models; 968 total No module named 'cutlass' lines this run).
• [FACT] Log: DeepseekV4ForCausalLM has no SGLang implementation, falling back to Transformers implementation → ValueError: Unrecognized configuration class ...._DeepseekV4ConfigAlias for this kind of AutoModel → Server process exited with code -9.
• [FACT] sglang source does contain DeepseekV4ForCausalLM (python/sglang/srt/models/deepseek_v4.py:2144); the fallback happened only because the module failed to import at load time.
• [FACT] Jul 3 passing run log has 0 occurrences of No module named 'cutlass' and the test passed 1/1; Jul 7 has 968. The cutlass import failures are entirely new.
• [FACT] Import chain to the failing test: models/deepseek_v4.py:121 → models/deepseek_v2.py:61 (from ...dsa.dsa_indexer import Indexer) → dsa_indexer.py:18 (from sglang.jit_kernel.dsa import (..., cutedsl_paged_mqa_logits, ...)) → jit_kernel/dsa/cutedsl_paged_mqa_logits.py:15 (import cutlass). All four import statements are module-level / unconditional.
• [FACT] cutlass here is the NVIDIA-only nvidia-cutlass-dsl Python package (import cutlass.cute), not present on the AMD ROCm 7.2 image.

Hypothesised Causes (with confidence)

• [HIGH] Commit fefc1743 ("Cute-DSL FP8 MQA logits", #25220, merged 2026-07-07 06:34 UTC) introduced the new sglang.jit_kernel.dsa package (with top-level import cutlass in cutedsl_paged_mqa_logits.py) and added the unconditional top-level from sglang.jit_kernel.dsa import (...) to dsa_indexer.py:18. On AMD (no cutlass package) this makes dsa_indexer — and therefore deepseek_v2/deepseek_v4 and every model importing them — fail to import.
  • Supporting: exact import chain verified in source (above); the modifying diff of fefc1743 adds line dsa_indexer.py:18; commit lands in the regression window and merged ~12h before this run; Jul 3 (pre-fefc1743) had zero cutlass errors and passed; broad ~30-model breakage matches a widely-imported module (dsa_indexer).
  • Disconfirming: (none found) — the in-flight fix #30410 targets this exact import line, corroborating the diagnosis.
  • How to verify: on AMD MI35x ROCm 7.2 container, python -c "import sglang.srt.models.deepseek_v4" should raise ModuleNotFoundError: cutlass; then apply an is_hip()/is_cuda() guard around dsa_indexer.py:18 and re-run test/registered/amd/test_deepseek_v4_pro_fp4_mtp.py.

In-flight Fix Check

• In-flight fix: ⚠️ partial — #30410 "[NPU] Fix npu cutlass import" (open, created 2026-07-07) moves the exact from sglang.jit_kernel.dsa import (...) block in dsa_indexer.py behind if not is_npu():. However it only guards NPU, not AMD/ROCm — on AMD (is_hip()), the import would still execute and still fail. As written, #30410 will not fix this MI35x failure; the guard needs to also exclude ROCm (e.g. gate on is_cuda()).
• Search run: search/issues?q=repo:sgl-project/sglang+is:pr+is:open+cutlass+import (38 hits; #30410 is the on-point one).

Suggested Triage Steps

• Confirm the ImportError locally in the AMD ROCm 7.2 container: python -c "import sglang.srt.layers.attention.dsa.dsa_indexer" — expect No module named 'cutlass'.
• Ping #30410 reviewers to extend the guard to ROCm (gate the sglang.jit_kernel.dsa import on is_cuda() rather than only not is_npu()), since AMD MI35x is affected identically to NPU.
• If a faster unblock is needed, revert or guard fefc1743's top-level dsa_indexer.py:18 import on a branch and re-run test_deepseek_v4_pro_fp4_mtp.py on the MI35x runner.
• Do NOT chase aiter — aiter is unchanged (dev image, same SHA as Dockerfile default) and the failure is a pure Python import error.

Status

• Persistence: First occurrence of this failure mode is today's run (2026-07-07); the test last passed 2026-07-03 (Jul 4/5 failures were unrelated VRAM-cleanup infra issues, so the test itself last actually executed on Jul 3).
• Scope: Broad — deepseek_v4, deepseek_v4_nextn, deepseek_v2, and ~28 other models fail to import on the ROCm 7.2 image this run (968 No module named 'cutlass' lines). Any AMD nightly test that loads a model importing dsa_indexer is at risk.
• In-flight fix: ⚠️ #30410 exists but currently guards only NPU, not AMD/ROCm — needs extension before it resolves this job.

nightly-accuracy-2-gpu-rocm720 — failed step(s): N/A

Commit Info

• sglang: 11cea29c
• aiter: N/A — the job never started (0 steps, no runner assigned), so no [CI-AITER-CHECK] markers were emitted. The log blob does not exist (BlobNotFound).

Failed Tests

Test File	Test Function	Error	Log
N/A (job never executed)	N/A	Job cancelled while queued — no linux-mi325-2gpu-sglang runner became available; 0 steps ran	Job (log unavailable: BlobNotFound)

This is not a test failure. The job nightly-accuracy-2-gpu-rocm720 sat queued for ~23h48m (from 2026-07-07T18:10:37Z to 2026-07-08T17:58:43Z) with no runner ever assigned, then was cancelled.

Failure Summary

The job was never scheduled onto a runner: its required runner label linux-mi325-2gpu-sglang had no available/online runner for the entire lifetime of run 28888218691. It ran 0 steps and was cancelled ~24h later when the next day's nightly was created and took over the concurrency group. This is an infrastructure (runner-fleet availability) issue, not a code regression.

Regression Status

Infrastructure issue — runner pool unavailability, not a code/test regression.

Recent history of job nightly-accuracy-2-gpu-rocm720 (workflow nightly-test-amd-rocm720.yml, branch main, event schedule, completed runs only):

Date	Run	Job runner	Job outcome	Notes
Jul 07	28888218691	⚠️ none assigned	❌ Cancelled (0 steps)	linux-mi325-2gpu-sglang never came online
Jul 06	28813124510	linux-mi325-2gpu-sglang-...	❌ Failure (7 steps)	runner available; job ran & failed on tests
Jul 05	28749501468	linux-mi325-2gpu-sglang-...	❌ Failure (7 steps)	runner available
Jul 04	28714556066	linux-mi325-2gpu-sglang-...	❌ Failure (8 steps)	runner available
Jul 03	28676295311	linux-mi325-2gpu-sglang-...	❌ Failure (8 steps)	runner available
Jul 02	28611268103	linux-mi325-2gpu-sglang-...	❌ Failure (7 steps)	runner available

The runner-unavailability cancellation is new to this run (Jul 07). In the prior 5 scheduled runs the linux-mi325-2gpu-sglang runner was consistently available and the job executed (though it was failing on its actual tests — a separate, pre-existing issue outside this cancellation).

Failure Cluster

Runner pool unavailable — linux-mi325-2gpu-sglang / linux-mi325-8gpu-sglang jobs cancelled while queued (never scheduled).

Facts (verified observations, no interpretation)

• [FACT] Job 85693869945 has conclusion=cancelled, steps=[] (0 steps), runner_name="" — it never started. (GitHub API job object)
• [FACT] Required runner label for the job is linux-mi325-2gpu-sglang. (API labels field)
• [FACT] All 16 cancelled jobs in this run have 0 steps and required either linux-mi325-2gpu-sglang (4 jobs) or linux-mi325-8gpu-sglang (12 jobs). No other label pool had a cancelled-unscheduled job.
• [FACT] The pools that did schedule jobs in this run: linux-mi325-1gpu-sglang (3), linux-mi325-4gpu-sglang (1), linux-mi35x-gpu-1 (1), linux-mi35x-gpu-2 (2), linux-mi35x-gpu-8 (22). Neither linux-mi325-2gpu-sglang nor linux-mi325-8gpu-sglang appears among runners that ran.
• [FACT] Run started 2026-07-07T18:10:14Z, cancelled 2026-07-08T17:58:49Z (~23h48m). Next scheduled nightly 28964405353 was created at 2026-07-08T17:58:42Z — the same instant, indicating concurrency-group cancellation of the still-queued prior run.
• [FACT] Log blob returns BlobNotFound (expected — no steps executed, no log produced).

Hypothesised Causes (with confidence, with disconfirming evidence)

• [HIGH] The linux-mi325-2gpu-sglang runner pool (and linux-mi325-8gpu-sglang) was offline/at-zero-capacity for the duration of this run; the job could not be scheduled and was ultimately cancelled by the next nightly's concurrency group.
  • Supporting: 0 steps + empty runner_name (never assigned); ALL 16 unscheduled-cancelled jobs map exclusively to these two labels; sibling pools (1gpu/4gpu-sglang, all mi35x) scheduled fine in the same run; the same job scheduled successfully on linux-mi325-2gpu-sglang in all 5 prior scheduled runs.
  • Disconfirming: (none found) — the label-partitioned cancellation pattern is unambiguous; no evidence any 2gpu/8gpu-sglang job ran.
  • How to verify: check the AMD self-hosted runner fleet dashboard / ARC (Actions Runner Controller) scale-set status for linux-mi325-2gpu-sglang and linux-mi325-8gpu-sglang around 2026-07-07 → 2026-07-08 UTC; confirm zero online runners in those scale sets during the window.
• [LOW] The cancellation itself was a workflow/queue timeout rather than concurrency.
  • Supporting: run lasted ~24h, near GitHub's max.
  • Disconfirming: cancellation timestamp exactly matches the next nightly's creation time, which is the concurrency-cancel signature; timeout would not align to the second.
  • How to verify: inspect the concurrency group config in nightly-test-amd-rocm720.yml.

In-flight Fix Check

• In-flight fix: ❌ none found — this is a runner-fleet availability/infra matter, not a code fix. (Searched open PRs for runner/infra keywords; no code change would address an offline self-hosted runner pool.)

curl -s -H "Authorization: token $GH_PAT" "https://api.github.com/search/issues?q=repo:sgl-project/sglang+is:pr+is:open+linux-mi325-2gpu-sglang"

Suggested Triage Steps

• Check the AMD self-hosted runner fleet / ARC scale-set health for linux-mi325-2gpu-sglang and linux-mi325-8gpu-sglang during 2026-07-07T18:10Z–2026-07-08T17:58Z; confirm the pools were at zero capacity (node drain, crash, or autoscaler failure).
• Verify whether the two mi325 multi-GPU scale sets have since recovered by checking the current nightly run 28964405353 once it completes — do its 2gpu/8gpu-sglang jobs get scheduled?
• Consider a queue/runner-availability timeout so unscheduled jobs fail fast (with a clear "no runner available" signal) instead of sitting ~24h until concurrency-cancelled, which produces misleading "cancelled" CI noise.
• Separately (not part of this cancellation): when the runner is healthy, nightly-accuracy-2-gpu-rocm720 has been failing on its actual tests in the prior 5 runs — that pre-existing test failure needs its own investigation once the pool is back online.

Status (factual, no priority assignment)

• Persistence: Runner-unavailability cancellation is new to run 28888218691 (Jul 07); the pool was healthy in the prior 5 scheduled runs (Jul 02–06).
• Scope: 16 of 46 jobs in this run cancelled unscheduled — all on linux-mi325-2gpu-sglang (4 jobs) and linux-mi325-8gpu-sglang (12 jobs). Entire mi325 2-GPU and 8-GPU ROCm 7.2 nightly coverage was lost for this run.
• Blocked work: No accuracy/perf/8-GPU model signal on linux-mi325-2gpu-sglang / linux-mi325-8gpu-sglang for the Jul 07 nightly.
• In-flight fix: ❌ none found — infra/runner-fleet issue, not a code fix.

nightly-8-gpu-deepseek-v3-kv-fp8-rocm720 — failed step(s): N/A

Commit Info

• sglang: 11cea29c
• aiter: N/A — the job was cancelled before any step executed; no container was ever started, so no [CI-AITER-CHECK] markers exist (the log blob returns BlobNotFound).

Failed Tests

Test File	Test Function	Error	Log
— (no test ran)	—	Job cancelled while queued — never acquired a runner (0 steps executed)	Job

This is not a test failure. The job nightly-8-gpu-deepseek-v3-kv-fp8-rocm720 recorded conclusion: cancelled, steps: 0, and sat queued from 2026-07-07T18:10:37Z to 2026-07-08T17:58:43Z (~23 h 48 m) without ever starting. No DeepSeek-V3 KV-FP8 test code was executed.

Failure Summary

The job never ran. It waited in the queue for a linux-mi325-8gpu-sglang (MI30x/MI325 8-GPU) runner for almost 24 hours and was then cancelled when the next nightly scheduled run started. This is a runner-availability / queue-timeout infrastructure issue on the MI325 8-GPU pool, not a code or test defect. The DeepSeek-V3 KV-FP8 suite itself has been passing on every run where it actually got a runner.

Regression Status

Infrastructure issue (not a code regression). The test suite is healthy.

Recent history of job nightly-8-gpu-deepseek-v3-kv-fp8-rocm720 (branch main, event schedule, completed runs only):

Date	Run	Job Outcome	Ran?	Queue/Run detail
Jul 7	28888218691	❌ Cancelled	Never ran (0 steps)	Queued 18:10:37Z → cancelled 17:58:43Z next day (~23h48m)
Jul 6	28813124510	✅ Success	Yes	Queued ~20h, then ran Jul 7 14:17→15:15Z, passed
Jul 5	28749501468	✅ Success	Yes	Ran 17:48→18:22Z, passed
Jul 4	28714556066	✅ Success	Yes	Ran 19:12→19:36Z, passed
Jul 3	28676295311	✅ Success	Yes	Ran 22:00→22:22Z, passed

The job passed in all 4 prior scheduled runs whenever it obtained a runner. Note the Jul 6 run already showed a ~20 h queue wait before running — early warning of the same MI325 pool backlog.

Failure Cluster

MI325 8-GPU runner starvation — nightly jobs cancelled while queued (never scheduled)

Facts (verified observations, no interpretation)

• [FACT] Job conclusion is cancelled with steps: 0; log blob returns BlobNotFound (job never produced logs). Verified via GET /actions/jobs/85693870107 and .../logs.
• [FACT] Job was queued 2026-07-07T18:10:37Z and cancelled 2026-07-08T17:58:43Z (~23h48m wait). Verified via job started_at/completed_at.
• [FACT] Target job's runner label is linux-mi325-8gpu-sglang (.github/workflows/nightly-test-amd-rocm720.yml:878).
• [FACT] 16 jobs in run 28888218691 share the identical cancel signature (steps: 0, queued 18:10:37Z → cancelled 17:58:43Z). All 16 are non-mi35x (MI30x/MI325-pool) jobs; every mi35x (linux-mi35x-gpu-8) job did get a runner and ran.
• [FACT] The workflow sets cancel-in-progress: true for schedule events (nightly-test-amd-rocm720.yml:133), keyed on github.ref (refs/heads/main).
• [FACT] The next scheduled run 28964405353 was created 2026-07-08T17:58:42Z — 1 second before this run's cancellation at 17:58:43Z.
• [FACT] This job passed on Jul 3/4/5/6 scheduled runs (see Regression Status table).

Hypothesised Causes (with confidence, with disconfirming evidence)

• [HIGH] The linux-mi325-8gpu-sglang (MI30x/MI325 8-GPU) runner pool was saturated/backlogged; the job (and 15 MI325-pool siblings) never got a runner and was cancelled by the next nightly cron via cancel-in-progress: true.
  • Supporting: 16 MI325-pool jobs all cancelled with 0 steps and identical queue timestamps, while every MI35x-pool job ran; next scheduled run created 1 s before cancellation; Jul 6 run already showed a ~20 h queue wait on this exact job.
  • Disconfirming: (none found) — the test suite itself passed on every run where it obtained a runner, ruling out a code-side failure.
  • How to verify: check the MI325 8-GPU runner group's online/busy count and queue depth around 2026-07-07 18:10 → 2026-07-08 18:00 UTC; confirm no MI325 runner was idle during the window.

In-flight Fix Check

• In-flight fix: ❌ none found directly addressing MI325 8-GPU runner starvation / nightly queue-timeout. Search repo:sgl-project/sglang is:pr is:open mi325 runner (25 hits) and ... runner capacity (54 hits) surfaced only tangential runner-migration work — e.g. #30221 "[AMD] Test MI300 runners" and #28584 "WIP ci: test AMD PR workflow on new MI300 runners" — not a capacity/scheduling fix. Needs an infra action, not a code PR.

Suggested Triage Steps

• Inspect the linux-mi325-8gpu-sglang runner group utilization for the 2026-07-07 18:10 → 2026-07-08 18:00 UTC window (online count, busy%, queue depth) to confirm starvation and quantify the shortfall.
• Consider staggering the ROCm 7.2 nightly cron or reducing MI325 8-GPU job concurrency so the pool can drain within the ~24 h before the next nightly cancels it.
• If MI325 8-GPU capacity cannot be increased, evaluate raising runner count or moving some 8-GPU nightly suites onto the (evidently-available) MI35x pool.
• No sglang code change is warranted for this job — the DeepSeek-V3 KV-FP8 suite is green whenever scheduled.

Status (factual, no priority assignment)

• Persistence: This job first failed to run on 2026-07-07 (cancelled while queued); the same MI325 backlog was already visible Jul 6 (~20 h queue wait, but it still ran and passed). Not previously cancelled in the Jul 3–5 window.
• Scope: 16 non-mi35x (MI325/MI30x-pool) jobs in run 28888218691 were cancelled without running; the entire run concluded cancelled. MI35x-pool jobs were unaffected.
• Blocked work: All MI30x/MI325 8-GPU + 2-GPU accuracy/perf nightly signal for ROCm 7.2 is missing for 2026-07-07 (no data produced).
• In-flight fix: ❌ none found — requires runner-infra remediation, not a code PR.

nightly-perf-2-gpu-vlm-rocm720 — failed step(s): N/A

Commit Info

• sglang: 11cea29c
• aiter: N/A — the job never acquired a runner, so the Install dependencies step (which emits [CI-AITER-CHECK]) never ran. No log blob exists (BlobNotFound).

Failed Tests

Test File	Test Function	Error	Log
N/A (no test executed)	N/A	Job cancelled while queued — 0 steps ran, no runner assigned	Job

This was not a test failure. The job was cancelled, never failure. It never got past queueing.

Failure Summary

The job nightly-perf-2-gpu-vlm-rocm720 was cancelled, not failed. It sat in the queue for ~23h48m waiting for a linux-mi325-2gpu-sglang self-hosted runner that was never assigned (runner_name empty, steps: []), then was cancelled at 2026-07-08T17:58:43Z when the next scheduled nightly started and the workflow's cancel-in-progress concurrency rule tore down the still-in-progress run. 15 sibling jobs — all requiring the shared linux-mi325-{2,4,8}gpu-sglang large-runner pools — were cancelled at the exact same instant with identical started/completed timestamps.

Regression Status

Infrastructure issue (runner starvation → concurrency cancellation). NOT a code regression.

Execution history of job nightly-perf-2-gpu-vlm-rocm720 (same branch main, event=schedule, completed runs only):

Date	Run	Got runner?	Job Result	Duration
Jul 07	28888218691	❌ never assigned	⏹️ Cancelled (queued 24h)	—
Jul 06	28813124510	✅	✅ Success	~19 min
Jul 05	28749501468	✅	❌ Failure (test-level, separate)	~23 min
Jul 04	28714556066	✅	❌ Failure (test-level, separate)	~24 min
Jul 03	28676295311	✅	❌ Failure (test-level, separate)	~25 min
Jul 02	28611268103	✅	✅ Success	~21 min

In every prior run the job acquired a linux-mi325-2gpu-sglang runner within hours and completed in ~20-25 min. The "never scheduled onto a runner in 24h" state is unique to this run. (The earlier Jul 03-05 failures are real test-level failures but are a separate issue — out of scope here since today's job never executed a test.)

Failure Cluster

Runner starvation on linux-mi325-*gpu-sglang pool → job cancelled by next-schedule cancel-in-progress.

Facts (verified observations, no interpretation)

• Job conclusion is cancelled (not failure); runner_name is empty and steps: [] — the job never executed a single step. [Job API]
• Job window: started 2026-07-07T18:10:37Z → completed 2026-07-08T17:58:43Z (~23h48m). The started_at equals the run's run_started_at; it reflects queue-enter time, not execution.
• The job's own timeout-minutes: 180 (nightly-test-amd-rocm720.yml:422) never fired — that timer counts execution time, and the job never executed.
• The next scheduled run 28964405353 was created 2026-07-08T17:58:42Z — 1 second before this run's cancellation at 17:58:43Z.
• Workflow concurrency (nightly-test-amd-rocm720.yml:132-133): group nightly-test-amd-rocm720-refs/heads/main, cancel-in-progress: true for schedule events → a new scheduled run cancels the prior in-progress one.
• 16 jobs in run 28888218691 were cancelled with the identical 18:10:37Z → 17:58:43Z window and 0 runner assignment; all require the shared linux-mi325-{2,4,8}gpu-sglang large runners (2/4/8-GPU). The -mi35x- jobs (a different runner pool) ran and completed within minutes.
• Same-pattern cancellations recurred: run 28813124510 (Jul 06) and 28537991905 (Jul 01) also ran the full ~24h and were cancelled by the following schedule — i.e. the ROCm 7.2 nightly is not draining its queue within its 24h cadence.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [HIGH] The run did not complete within its 24h cadence because the linux-mi325-*gpu-sglang self-hosted runner pool was saturated; this job (2-GPU) never got a runner and was cancelled when the next scheduled run triggered cancel-in-progress.
  • Supporting: 16 jobs cancelled at the identical instant with no runner assigned, all on the shared large-runner pools; next scheduled run created 1s before cancellation; concurrency config confirms schedule runs cancel in-progress; two other recent runs (Jul 01, Jul 06) show the same full-24h-then-cancelled pattern.
  • Disconfirming: (none found) — the concurrency timing and empty runner_name/steps are direct evidence; the only open question is why the pool was saturated (see below).
• [LOW] Long-running or hung sibling jobs (e.g. the 8-GPU accuracy/perf jobs with timeout-minutes: 180) held runners and back-pressured the 2-GPU queue.
  • Supporting: many -sglang jobs were simultaneously queued and starved; the run took >24h overall.
  • Disconfirming: cannot confirm without per-runner occupancy logs, which are unavailable; the 2-GPU and 8-GPU pools may be distinct label pools rather than one shared pool.
  • How to verify: inspect runner-pool occupancy / autoscaler metrics for linux-mi325-*gpu-sglang over 2026-07-07 18:00Z .. 2026-07-08 18:00Z.

In-flight Fix Check

• In-flight fix: ❌ none found targeting the nightly runner-starvation / 24h-cadence overrun directly. Related capacity work exists but does not fix this: #28584 "test AMD PR workflow on new MI300 runners" and #30221 "Test MI300 runners" (searched is:pr is:open with nightly rocm720 timeout, runner capacity, cancel-in-progress nightly).

Suggested Triage Steps

• Confirm runner-pool saturation: pull runner occupancy / autoscaler metrics for linux-mi325-{2,4,8}gpu-sglang across the Jul 07 18:00Z → Jul 08 18:00Z window; check whether 8-GPU jobs held runners the full 180 min.
• Assess cadence vs capacity: the ROCm 7.2 nightly (cron 30 17 * * *) is not draining within 24h (Jul 01, 06, 07 all cancelled). Options to evaluate: reduce the nightly job matrix, split into staggered schedules, or add runner capacity.
• If full signal is needed for this specific job, re-run just nightly-perf-2-gpu-vlm-rocm720 via workflow_dispatch (which gets a unique concurrency group and won't be cancelled by the schedule).
• Separately, triage the real test-level failures of this job on Jul 03-05 (out of scope here) — those runs did execute and failed on tests.

Status (factual, no priority assignment)

• Persistence: This job was cancelled-while-queued only in the Jul 07 run; it executed normally in the 5 prior runs. The broader "run cancelled at 24h by next schedule" pattern has hit 3 of the last 8 scheduled runs (Jul 01, 06, 07).
• Scope: 16 jobs in run 28888218691 cancelled identically, all on shared linux-mi325-*gpu-sglang pools; the whole ROCm 7.2 nightly produced no signal for those 16 jobs.
• Blocked work: No perf/accuracy/large-model signal from the affected 2/4/8-GPU -sglang jobs for the Jul 07 nightly.
• In-flight fix: ❌ none found for the runner-starvation/cadence issue.

nightly-8-gpu-deepseek-v31-rocm720 — failed step(s): N/A

Commit Info

• sglang: 11cea29c
• aiter: N/A — job never executed, no [CI-AITER-CHECK] markers exist (log blob does not exist)

Failed Tests

Test File	Test Function	Error	Log
N/A	N/A	Job cancelled while queued — never assigned a runner, ran zero steps	Job

This is not a test failure. The job nightly-8-gpu-deepseek-v31-rocm720 never ran any test.

Failure Summary

The job was cancelled with 0 steps executed and no runner assigned (runner_name empty). It sat queued from 2026-07-07T18:10:37Z and was cancelled 2026-07-08T17:58:43Z — ~23h48m later — without ever starting. The GitHub log blob returns BlobNotFound because no log was ever produced. This is an infrastructure/runner-availability event, not a code or test regression.

Regression Status

Infrastructure issue (runner starvation + concurrency cancellation). The test job itself is healthy — it passed on each of the three prior completed runs where it obtained a runner.

Recent history of job nightly-8-gpu-deepseek-v31-rocm720 (branch main, event schedule, completed runs only):

Date	Run	Job Status	Steps	Runner	Note
Jul 7	28888218691	❌ Cancelled	0	none	queued ~24h, never ran (this run)
Jul 6	28813124510	✅ Success	9	…runner-jplzf	ran normally
Jul 5	28749501468	✅ Success	9	…runner-4pvqk	ran normally
Jul 4	28714556066	✅ Success	9	…runner-c9bxl	ran normally
Jul 1	28537991905	❌ Cancelled	0	none	same 24h-queue-then-cancel pattern

Failure Cluster

Runner starvation on linux-mi325-8gpu-sglang pool → jobs queued ~24h → concurrency-cancelled by next nightly

Facts (verified observations)

• [FACT] Target job: conclusion=cancelled, steps=[], runner_name="", labels=["linux-mi325-8gpu-sglang"], started_at=2026-07-07T18:10:37Z, completed_at=2026-07-08T17:58:43Z (GitHub jobs API).
• [FACT] Job log endpoint returns BlobNotFound — no log was ever written (job never started).
• [FACT] 16 of 46 jobs in run 28888218691 were cancelled, all with the identical started_at=2026-07-07T18:10:37Z and completed_at=2026-07-08T17:58:43Z. This batch is the non-mi35x set (8-gpu deepseek/grok/glm/qwen/kimi/minimax, accuracy, and perf jobs) — i.e. those targeting the linux-mi325-8gpu-sglang / mi325 runner pool.
• [FACT] The mi35x-pool jobs in the same run did obtain runners and ran (26 completed with failure, 4 success) — a separate concern, not part of this cancellation.
• [FACT] The next scheduled run 28964405353 was created 2026-07-08T17:58:42Z — 1 second before the 16 jobs were cancelled at 17:58:43Z.
• [FACT] The same job passed with 9 steps on Jul 4/5/6 when a linux-mi325-8gpu-sglang runner was available.

Hypothesised Causes (with confidence)

• [HIGH] The linux-mi325-8gpu-sglang runner pool was saturated/unavailable, so all mi325-targeted jobs queued for ~24h; the concurrency group then cancelled them when the next nightly (28964405353) started.
  • Supporting: all 16 cancelled jobs share the mi325 label and share the exact completion timestamp, which is 1s after the next scheduled run was created (classic cancel-in-progress concurrency behavior); zero steps and no runner ever assigned; the identical pattern recurred on Jul 1 (28537991905).
  • Disconfirming: (none found) — no evidence of any code/test execution; the same job is green when it lands on a runner.
  • How to verify: check MI325 8-GPU runner fleet availability/utilization for 2026-07-07 18:10 → 2026-07-08 18:00 UTC; inspect the workflow's concurrency: block for cancel-in-progress: true.

In-flight Fix Check

• In-flight fix: ❌ N/A — this is a runner-fleet/queue capacity + concurrency-cancel issue, not a code defect. No source fix applies. If desired, capacity is an infra/ops action, not a PR.

Suggested Triage Steps

• Confirm MI325 8-GPU runner availability during the 24h queue window (ops/fleet dashboard); the nightly cannot complete if the pool cannot serve all 8-GPU + accuracy + perf jobs before the next day's run.
• Consider whether the nightly workflow's concurrency should NOT cancel a still-running previous nightly, or whether queued-job timeouts should be shortened so stuck jobs fail fast (hours, not 24h) instead of blocking the matrix.
• Re-run nightly-8-gpu-deepseek-v31-rocm720 on demand once mi325 capacity is confirmed — the test itself is expected to pass (green Jul 4/5/6).
• Treat the separate mi35x-pool failure jobs in this run as a distinct investigation; they are unrelated to this cancellation.

Status

• Persistence: Job cancelled-while-queued on 2026-07-07 run; identical infra pattern also on 2026-07-01. Intervening runs (Jul 4/5/6) succeeded — intermittent, tied to runner-pool capacity.
• Scope: 16 of 46 jobs in run 28888218691, i.e. the entire linux-mi325-8gpu-sglang mi325 job set (8-gpu model, accuracy, and perf jobs). The mi35x pool was unaffected by the cancellation.
• Blocked work: No nightly signal for DeepSeek-V3.1 and the other 15 mi325 jobs on ROCm 7.2 for 2026-07-07.
• In-flight fix: N/A — infrastructure capacity / concurrency-config matter, not a code fix.

nightly-accuracy-8-gpu-rocm720 — failed step(s): N/A

Commit Info

• sglang: 11cea29c
• aiter: N/A — the job never ran; no log blob exists (BlobNotFound), so no [CI-AITER-CHECK] markers are available.

Failed Tests

Test File	Test Function	Error	Log
N/A (no test ran)	N/A	Job cancelled while queued — never assigned a runner, 0 steps executed	Job

This is not a test failure. The job nightly-accuracy-8-gpu-rocm720 (runs-on: linux-mi325-8gpu-sglang) never acquired a runner and executed zero steps before being cancelled.

Failure Summary

The job was queued at 2026-07-07 18:10:37 UTC but was never picked up by a linux-mi325-8gpu-sglang runner. It sat in queue for ~23h48m and was then cancelled at 2026-07-08 17:58:43 UTC. The log blob does not exist (BlobNotFound) because no runner ever started the job. Every other job in this run targeting the linux-mi325-2gpu-sglang / linux-mi325-8gpu-sglang pools shows the identical signature (0 steps, runner_name empty, conclusion: cancelled), while jobs on the separate linux-mi35x-gpu-* pool did run.

Regression Status

Infrastructure issue (runner-pool saturation) — not a code regression.

Recent history of job nightly-accuracy-8-gpu-rocm720 (nightly-test-amd-rocm720.yml, branch main, event=schedule, completed runs only):

Date	Run	Job Outcome	Runner acquired?	Detail
Jul 07	28888218691	❌ cancelled	No (0 steps, no runner)	Queued ~24h, cancelled by next run's concurrency
Jul 06	28813124510	✅ success	Yes (after ~21h queue)	Ran 9 steps, passed on linux-mi325-8gpu-sglang-lmwqt-runner-8n7ls

The most recent run in which this job could obtain a runner (Jul 06) passed — confirming the test/code path is healthy. The Jul 06 job itself only started at 2026-07-07 15:17 UTC (≈21h after being queued), showing the linux-mi325-8gpu-sglang pool is chronically backlogged.

Facts (verified observations)

• Target job conclusion: cancelled, runner_name: "" (empty), steps: [] (zero), started_at: 2026-07-07T18:10:37Z, completed_at: 2026-07-08T17:58:43Z (verified via job API).
• Log fetch returns <Error><Code>BlobNotFound</Code> — no log was ever produced.
• ALL cancelled jobs in run 28888218691 target linux-mi325-2gpu-sglang or linux-mi325-8gpu-sglang, each with 0 steps and no runner. Jobs on linux-mi35x-gpu-8 (a different pool) executed normally (ran to failure/success).
• The next scheduled run 28964405353 was created at 2026-07-08T17:58:42Z — 1 second before the target job's cancellation at 17:58:43Z.
• Workflow concurrency (.github/workflows/nightly-test-amd-rocm720.yml:127-133): group: …-${{ … || github.ref }} with cancel-in-progress: true for schedule events. The two runs share group refs/heads/main, so the newer run cancels the older run's still-queued jobs.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [FACT] The job was cancelled by the concurrency cancel-in-progress policy when the next day's scheduled run started, because it was still stuck in queue.
  • Supporting: next run 28964405353 created 1s before cancellation; concurrency block at nightly-test-amd-rocm720.yml:132-133 keys on github.ref with cancel-in-progress on for schedule events.
  • Disconfirming: (none found) — timing and config are unambiguous.
• [HIGH] Root cause is linux-mi325-8gpu-sglang runner-pool capacity/availability — the job could not obtain a runner within ~24h.
  • Supporting: 0 steps + empty runner_name = never scheduled; the Jul 06 sister job only started after ~21h in queue on the same pool; every linux-mi325-*-sglang job in this run was cancelled while every linux-mi35x-gpu-* job ran.
  • Disconfirming: The pool is not fully dead — the Jul 06 run eventually did place jobs on it. So this is saturation/backlog rather than a total outage.
  • How to verify: check the linux-mi325-8gpu-sglang self-hosted runner fleet status/idle count during the 2026-07-07 18:10 → 2026-07-08 18:00 window; compare queued-job count vs available runners.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs repo:sgl-project/sglang is:pr is:open mi325 runner (25 results, all about MI300/MI325 runner testing/migration, e.g. #30221, #28584; none address 8-GPU MI325 pool saturation / queued-job cancellation). This is an infra-capacity matter, not a code fix.

Suggested Triage Steps

• Inspect the linux-mi325-8gpu-sglang runner fleet: how many runners were online/idle vs the number of queued 8-GPU jobs during 2026-07-07 18:10 → 2026-07-08 18:00 UTC. The Jul 06 ~21h queue wait suggests the pool is a standing bottleneck.
• Consider whether the nightly matrix's 8-GPU MI325 jobs exceed the pool's throughput within a 24h cron cadence (many linux-mi325-8gpu-sglang jobs compete for the same scarce runners), causing systematic queue overflow that concurrency then cancels.
• Optionally add a per-job queue timeout (fail fast if no runner in N hours) so these surface as explicit "runner unavailable" rather than day-later concurrency cancellations that produce no log.
• Re-run only nightly-accuracy-8-gpu-rocm720 on-demand once the pool has idle capacity to confirm the test itself still passes (it did on Jul 06).

Status (factual)

• Persistence: Job could not get a runner on the 2026-07-07 run; cancelled after ~24h queue. Last time it acquired a runner (2026-07-06 run) it passed. Prior runs (Jul 01–06) show recurring cancelled run conclusions, indicating an intermittent-but-ongoing MI325 capacity problem.
• Scope: Affects ALL linux-mi325-2gpu-sglang and linux-mi325-8gpu-sglang jobs in run 28888218691 (≈19 jobs cancelled with no runner). linux-mi35x-gpu-* jobs unaffected.
• Blocked work: No nightly accuracy/8-GPU MI325 ROCm 7.2 signal produced for 2026-07-07; the failure is environmental, so no sglang code change is implicated.
• In-flight fix: ❌ none found — needs runner-fleet capacity investigation, not a code PR.

nightly-accuracy-2-gpu-vlm-rocm720 — failed step(s): N/A

Commit Info

• sglang: 11cea29c
• aiter: N/A — the job never started (0 steps executed), so no [CI-AITER-CHECK] markers were emitted and no dependency install ran.

Failed Tests

Test File	Test Function	Error	Log
N/A (job never ran)	N/A	Job cancelled with 0 steps executed — never acquired a runner	Job page

The runtime log blob returns BlobNotFound and the GitHub API reports zero steps for this job. The job's conclusion is cancelled, not failure.

Failure Summary

The job nightly-accuracy-2-gpu-vlm-rocm720 did not fail a test — it was cancelled before executing any step because it never got a runner. It sat queued for ~24h (started_at 2026-07-07T18:10:37Z → completed_at 2026-07-08T17:58:43Z) and was killed when the parent run overran into the next scheduled nightly cycle. This is an infrastructure / runner-starvation issue on the ROCm 7.2 pool, not a code regression.

Regression Status

Infrastructure issue (this run) — with a separate, pre-existing test-level failure on the days the job actually ran.

Recent history of job nightly-accuracy-2-gpu-vlm-rocm720 (branch main, event schedule, completed runs only):

Date	Run	Job outcome	Steps	Test File Status	Detail
Jul 7	28888218691	⚪ Cancelled	0	⚪ Never ran	No runner; run cancelled by concurrency (this run)
Jul 6	28813124510	⚪ Cancelled	0	⚪ Never ran	Same infra pattern — run overran ~24h
Jul 5	28749501468	❌ Failure	8	❌ Failed	test_vlms_mmmu_eval_amd.py::test_mmmu_vlm_models — 5 VLMs below MMMU threshold
Jul 4	28714556066	❌ Failure	8	❌ Failed	Same job structure (test executed, failed)

No green run for this job found in the last 4 scheduled runs (lookback Jul 4–Jul 7). When it runs, it fails on accuracy; when it doesn't run, it's cancelled for lack of a runner.

Failure Cluster

"Job cancelled — no runner (ROCm 7.2 pool starvation, run overran 24h)"

Facts (verified observations, no interpretation)

• Job conclusion = cancelled, steps = [] (0 steps). Log blob returns <Code>BlobNotFound</Code>. [FACT]
• Job queued 2026-07-07T18:10:37Z, cancelled 2026-07-08T17:58:43Z — ~23h48m, with no step ever starting. [FACT]
• Parent run 28888218691 conclusion = cancelled; the next scheduled run 28964405353 was created 2026-07-08T17:58:42Z — 7 seconds before this run's cancellation timestamp. [FACT — concurrency cancellation]
• In this run, 16 of 46 jobs were cancelled with 0 steps (all queued, never scheduled); 26 failed, 4 succeeded. [FACT]
• The prior scheduled run 28813124510 (Jul 6) shows the identical pattern: cancelled ~24h later exactly when this run started. Runs Jul 2–5 completed within ~6–15h. [FACT]
• Underlying test on Jul 5: test/registered/amd/accuracy/mi30x/test_vlms_mmmu_eval_amd.py:380 AssertionError: The following models failed: — Qwen2-VL-7B (0.290<0.30), Qwen2.5-VL-7B (0.290<0.33), Qwen3-VL-30B-A3B (0.260<0.29), MiniCPM-o-2_6 (0.280<0.32), deepseek-vl2-small (0.280<0.31). 6 passed / 5 failed. [FACT]

Hypothesised Causes (with confidence, with disconfirming evidence)

• [HIGH] This job's cancellation is a concurrency kill caused by ROCm 7.2 runner starvation, not a code fault. Multiple independent evidence chains converge: 0 steps executed, ~24h queue time, next scheduled run created 7s before cancellation, and 16 sibling jobs cancelled with the identical 0-step signature.

  • Supporting: timestamps above; 16/46 jobs never scheduled; Jul 6 run shows the same overrun→concurrency-cancel pattern.
  • Disconfirming: (none found) — no step ran, so no code path could have failed within this job.
  • How to verify: check nightly-test-amd-rocm720.yml concurrency: block and the available ROCm 7.2 runner count during Jul 7–8; confirm the run was cancelled by the incoming schedule rather than a manual cancel (Actions run timeline).

• [FACT] Separate, pre-existing accuracy failure in the underlying test (test_mmmu_vlm_models) on the days it ran (Jul 4, Jul 5). This is independent of the cancellation and will recur whenever the job gets a runner.

  • Supporting: Jul 5 log shows 5 models below MMMU threshold; scores cluster tightly at ~0.26–0.32 across nearly all models (including some that pass by low thresholds), which is near-chance for MMMU — pattern is consistent with a systematic eval/answer-parsing issue rather than 5 independent model regressions.
  • Disconfirming: no green baseline in the queried window to prove the thresholds were ever met on ROCm 7.2; could be miscalibrated thresholds rather than a code regression. Treat the "systematic parsing" idea as [LOW] until logprob/answer-extraction output is inspected.
  • How to verify: when a runner is available, re-run test_vlms_mmmu_eval_amd.py and inspect per-question MMMU answer extraction; bisect against the last run where these models met threshold.

In-flight Fix Check

• Runner capacity: In-flight fix: ❌ none found for ROCm 7.2 nightly runner starvation. Related but not matching: #30221 "[AMD] Test MI300 runners" and #28584 target the PR workflow's MI300 runners, not the rocm720 nightly pool. (searched is:pr is:open rocm720 runner.)
• VLM MMMU accuracy: In-flight fix: ❌ none found matching the MMMU VLM threshold failure (searched is:pr is:open MMMU VLM accuracy, 19 unrelated hits).

Suggested Triage Steps

• Confirm the cancellation cause in the run timeline: verify 28888218691 was cancelled by the incoming scheduled run 28964405353 (concurrency), and inspect the concurrency: group in nightly-test-amd-rocm720.yml.
• Investigate ROCm 7.2 runner availability for Jul 6–8 (why 16/46 jobs never got a runner and the whole run overran 24h). This is the actionable item for this job's cancellation.
• Consider whether the nightly matrix exceeds pool capacity (queue backpressure); if so, sharding/prioritization or a longer concurrency window may prevent 24h overruns.
• Separately, for the underlying accuracy failure: re-run test/registered/amd/accuracy/mi30x/test_vlms_mmmu_eval_amd.py once a runner is free; inspect MMMU answer-extraction for the ~0.26–0.32 score clustering; bisect the affected VLMs against the last threshold-meeting run.

Status (factual, no priority assignment)

• This job: cancelled, never executed — infrastructure (no runner) for ~24h, then concurrency-cancelled.
• Persistence (infra): 2 consecutive scheduled runs (Jul 6, Jul 7) overran ~24h and were concurrency-cancelled with jobs still queued.
• Scope (infra): 16 of 46 jobs in this run cancelled with 0 steps; blast radius is the whole nightly-test-amd-rocm720 matrix, not this one job.
• Underlying test: test_mmmu_vlm_models failed the last 2 times it actually ran (Jul 4, Jul 5); no green run in the lookback window.
• In-flight fix: ❌ none found for either the runner starvation or the MMMU accuracy failure.

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-07-08 20:50 UTC)

[amd-bot]

pr-test-amd.yml — 50 failure(s)

Run started (UTC): 2026-07-07 00:45 UTC
Last scanned (UTC): 2026-07-08 18:54 UTC
Commits: sglang 108a183

Summary

This is a large file with 50 job blocks. Let me delegate parsing to a subagent to extract structured data efficiently.

---

Note: agent timed out after 300s; showing the last assistant response captured before the kill. Re-run the bot command to retry.

---

Job	Failed Steps	Started
stage-b-test-large-8-gpu-mi35x-disaggregation-amd (linux-mi35x-gpu-8.fabric)	Ensure VRAM is clear	2026-07-07T18:43
multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 3)	Run diffusion server tests (1-GPU)	2026-07-07T21:26
multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 1)	N/A	2026-07-07T21:19
stage-c-test-large-8-gpu-amd-mi35x (linux-mi35x-gpu-8, 1)	Ensure VRAM is clear	2026-07-07T19:27
stage-c-test-large-8-gpu-amd-mi35x (linux-mi35x-gpu-8, 0)	Run test	2026-07-07T19:28
stage-b-test-1-gpu-small-amd (linux-mi325-1gpu-sglang, 6)	Run test	2026-07-07T21:34
stage-a-test-1-gpu-small-amd (linux-mi325-1gpu-sglang)	Run test	2026-07-07T15:17
stage-b-test-1-gpu-large-amd (linux-mi325-1gpu-sglang, 1)	Run test	2026-07-07T15:40
multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 3)	Run diffusion server tests (1-GPU)	2026-07-07T15:25
stage-b-test-1-gpu-small-amd (linux-mi325-1gpu-sglang, 4)	Run test	2026-07-07T15:51
stage-b-test-1-gpu-small-amd (linux-mi325-1gpu-sglang, 5)	Run test	2026-07-07T16:04
stage-b-test-1-gpu-small-amd (linux-mi325-1gpu-sglang, 6)	Run test	2026-07-07T15:55
stage-c-test-large-8-gpu-amd (linux-mi325-8gpu-sglang, 1)	Run test	2026-07-07T19:23
stage-c-test-large-8-gpu-amd-mi35x (linux-mi35x-gpu-8, 1)	Run test	2026-07-07T12:52
stage-c-test-large-8-gpu-amd (linux-mi325-8gpu-sglang, 3)	Run test	2026-07-07T19:23
stage-c-test-large-8-gpu-amd-mi35x (linux-mi35x-gpu-8, 0)	Run test	2026-07-07T12:52
multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 3)	Run diffusion server tests (1-GPU)	2026-07-07T11:56
multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 1)	Run diffusion server tests (2-GPU)	2026-07-07T16:57
multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 1)	N/A	2026-07-07T11:53
stage-b-test-1-gpu-small-amd (linux-mi325-1gpu-sglang, 5)	Run test	2026-07-07T12:07
stage-b-test-1-gpu-small-amd (linux-mi325-1gpu-sglang, 4)	Run test	2026-07-07T12:50
stage-b-test-1-gpu-small-amd (linux-mi325-1gpu-sglang, 6)	Run test	2026-07-07T12:59
stage-c-test-large-8-gpu-amd-mi35x (linux-mi35x-gpu-8, 0)	Run test	2026-07-07T07:44
stage-c-test-large-8-gpu-amd-mi35x (linux-mi35x-gpu-8, 1)	Ensure VRAM is clear	2026-07-07T07:42
multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 1)	Run diffusion server tests (2-GPU)	2026-07-07T09:50
multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 3)	Run diffusion server tests (1-GPU)	2026-07-07T03:03
stage-b-test-large-8-gpu-mi35x-disaggregation-amd (linux-mi35x-gpu-8.fabric)	Ensure VRAM is clear	2026-07-07T00:45
multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 1)	Post Checkout code, Run diffusion server tests (1-GPU), Upload diffusion failure artifacts	2026-07-07T03:34
stage-c-test-large-8-gpu-amd-mi35x (linux-mi35x-gpu-8, 1)	Run test	2026-07-07T01:32
multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 3)	Run diffusion server tests (1-GPU)	2026-07-08T01:45
stage-b-test-1-gpu-small-amd (linux-mi325-1gpu-sglang, 4)	Run test	2026-07-08T01:38
stage-b-test-1-gpu-small-amd (linux-mi325-1gpu-sglang, 5)	Run test	2026-07-08T01:38
stage-b-test-1-gpu-large-amd (linux-mi325-1gpu-sglang, 0)	Run test	2026-07-08T03:23
multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 1)	N/A	2026-07-08T05:02
stage-b-test-1-gpu-large-amd (linux-mi325-1gpu-sglang, 1)	Run test	2026-07-08T05:35
stage-c-test-large-8-gpu-amd-mi35x (linux-mi35x-gpu-8, 1)	Ensure VRAM is clear	2026-07-08T06:54
stage-b-test-large-8-gpu-mi35x-disaggregation-amd (linux-mi35x-gpu-8.fabric)	Run test_disaggregation	2026-07-08T06:54
call-pr-test-amd-extra / extra-a-test-1-gpu-small-amd (linux-mi325-1gpu-sglang)	Run test	2026-07-08T09:33
stage-b-test-1-gpu-small-amd-mi35x (linux-mi35x-gpu-1)	Ensure VRAM is clear	2026-07-08T12:37
stage-c-test-large-8-gpu-amd-mi35x (linux-mi35x-gpu-8, 1)	Ensure VRAM is clear	2026-07-08T12:48
stage-b-test-large-8-gpu-mi35x-disaggregation-amd (linux-mi35x-gpu-8.fabric)	Run test_disaggregation	2026-07-08T12:37
multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 3)	Run diffusion server tests (1-GPU)	2026-07-08T13:14
multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 0)	Run diffusion server tests (1-GPU)	2026-07-08T12:34
stage-b-test-1-gpu-large-amd (linux-mi325-1gpu-sglang, 1)	Run test	2026-07-08T13:26
multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 1)	N/A	2026-07-08T13:23
stage-b-test-1-gpu-small-amd (linux-mi325-1gpu-sglang, 10)	Run test	2026-07-08T14:06
stage-b-test-1-gpu-small-amd-nondeterministic (linux-mi325-1gpu-sglang)	Install dependencies	2026-07-08T15:37
multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 3)	Run diffusion server tests (1-GPU)	2026-07-08T15:35
multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 0)	Run diffusion server tests (1-GPU)	2026-07-08T15:37
stage-b-test-1-gpu-small-amd (linux-mi325-1gpu-sglang, 12)	Install dependencies	2026-07-08T18:13

Per-Job Analysis

stage-b-test-large-8-gpu-mi35x-disaggregation-amd (linux-mi35x-gpu-8.fabric) — failed step(s): Ensure VRAM is clear

Commit Info

• sglang: 11cea29c
• aiter: N/A (deps never installed — job failed at pre-flight VRAM check before the container/dependency step; no [CI-AITER-CHECK] markers in log)

Failed Tests

Test File	Test Function	Error	Log
N/A (infra pre-flight)	N/A	FAILED: VRAM cleanup unsuccessful after 3 attempts → exit code 1	Log

No tests ran. The job failed at step 3 "Ensure VRAM is clear" (scripts/ci/amd/ensure_vram_clear.sh rocm). Steps 5–10 (Start Container, Install dependencies, Run test_disaggregation, etc.) were all skipped.

Failure Summary

The pre-flight VRAM guard on runner linux-mi35x-gpu-8.fabric-64dpd-runner-jhrbj detected GPUs 0–3 each holding 16% VRAM before the job started, exceeding the 5% threshold. The cleanup script ran 3 attempts (SIGTERM→SIGKILL→30s wait each) but VRAM never cleared. Diagnostics confirm no host process, no docker container, no KFD PIDs, and no zombie processes hold the GPUs — the memory is stranded at the driver level. The job aborted before any disaggregation test executed.

Regression Status

Infrastructure issue (recurring on this node) — NOT a code regression.

Recent history of the VRAM-clear pre-flight step in job stage-b-test-large-8-gpu-mi35x-disaggregation-amd (linux-mi35x-gpu-8.fabric) (branch=main, event=schedule, completed runs only):

Date (UTC)	Run	VRAM-clear step	Notes
Jul 7 18:43	28890204713	❌ Failed	This run — VRAM stuck 16% on GPU 0–3
Jul 6 18:40	28814834996	❌ Failed	Same VRAM cleanup unsuccessful after 3 attempts
Jul 6 13:18	28794501292	✅ Passed	VRAM-clear passed; job failed later on a different cause — KVTransferError in test_disaggregation_basic.py/test_disaggregation_pp.py (separate cluster, not this infra failure)
Jul 6 07:38	28775611300	✅ Passed	Job succeeded

The VRAM-clear infra failure first appeared Jul 6 18:40 UTC and recurred in this Jul 7 18:43 run (the two most recent completed schedule runs where this step failed). Newer runs (Jul 7 00:44 / 07:15 / 12:52) were still queued at snapshot time and are excluded.

Failure Cluster

Driver-level stranded VRAM / zombie KFD context — pre-flight ensure_vram_clear.sh abort on MI35x-8gpu node

Facts (verified observations)

• Step 3 "Ensure VRAM is clear" is the only failing step; steps 5–10 are skipped; job exit code 1 (log lines 539–547).
• GPUs 0–3 report GPU Memory Allocated (VRAM%): 16 on the initial check and after all 3 cleanup attempts; GPUs 4–7 are at 0% (log lines 170–173, 439–443).
• Diagnostics show No KFD PIDs currently running, No docker containers found on host, empty zombie-process list, and fuser found no GPU device-file holders (log lines 266–280, 484–498).
• The script's own hint attributes this to a "zombie KFD context (see [Bug] Custom all-reduce IPC buffers use fixed VA, conflict with torch_memory_saver pause/resume → zombie processes on MI300X ROCm/aiter#2061)" and states "The node will need to be rebooted" (log lines 542–546).
• Runner: linux-mi35x-gpu-8.fabric-64dpd-runner-jhrbj (log line 2).
• This is pr-test-amd.yml with AITER_COMMIT_OVERRIDE: empty (not a scout run) — no aiter A/B analysis applies.

Hypothesised Causes (with confidence)

• [FACT] Environmental/infrastructure failure: a prior job on this physical MI35x node leaked GPU memory at the driver/KFD level and exited without releasing it, leaving ~16% VRAM stranded on GPUs 0–3. The current job's pre-flight guard correctly refused to run on a dirty node.
  • Supporting: cleanup finds no host process/container/KFD PID owning the memory, yet VRAM stays at 16% across all 3 attempts; the guard script itself diagnoses a zombie KFD context and recommends a reboot.
  • Disconfirming: (none found) — there is no evidence of a code fault; the job never reached sglang/test code.
  • How to verify: reboot linux-mi35x-gpu-8 and re-run; confirm VRAM returns to 0% at pre-flight.
• [LOW] The leak was seeded by the immediately-preceding disaggregation test on this node (the Jul 6 13:18 run showed disaggregation tests crashing with KVTransferError, and disaggregation multi-process teardown is a plausible source of stranded KFD contexts).
  • Supporting: same node runs the disaggregation suite; a crashing prefill/decode process can leave an orphaned KFD context.
  • Disconfirming: no direct evidence links a specific prior job to the stranded memory in this run; the 16%-on-4-GPUs pattern is not tied to a captured PID.
  • How to verify: correlate the node's job timeline; check whether the job that ran just before this one on the same physical host crashed during GPU teardown.

In-flight Fix Check

• In-flight fix: ✅ #30431 "[DO NOT MERGE][AMD] ci: self-heal driver-level VRAM leaks + graceful GPU cleanup on shutdown" (open, created 2026-07-07 21:59 UTC) — directly targets this exact driver-level VRAM-leak scenario. Currently marked DO NOT MERGE; track it rather than opening a duplicate. (Searched repo:sgl-project/sglang is:pr is:open ensure_vram_clear OR KFD VRAM.)

Suggested Triage Steps

• Reboot the physical node behind linux-mi35x-gpu-8 (or cycle the runner) to clear the stranded KFD context, then re-run this job — per the script's own hint and [Bug] Custom all-reduce IPC buffers use fixed VA, conflict with torch_memory_saver pause/resume → zombie processes on MI300X ROCm/aiter#2061.
• Follow #30431 — if its self-heal/graceful-cleanup approach lands, it should prevent this pre-flight abort from recurring; ping reviewers to move it out of DO NOT MERGE.
• Investigate the preceding job on the same host: the Jul 6 disaggregation KVTransferError crashes are a candidate source of leaked KFD contexts on this node — check whether disaggregation-test teardown reliably frees GPU memory.
• This is not actionable via a code revert; no sglang commit in the window is implicated.

Status

• Persistence: VRAM-clear pre-flight failure seen in 2 of the last 3 completed schedule runs of this job (Jul 6 18:40 and Jul 7 18:43 UTC); the Jul 6 07:38 run passed, so this is intermittent node contamination rather than continuous.
• Scope: This single job / single node (linux-mi35x-gpu-8). No tests executed; disaggregation-large CI signal on MI35x is blocked while the node stays dirty.
• In-flight fix: ✅ #30431 (open, DO NOT MERGE) — self-heal driver-level VRAM leaks.

multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 3) — failed step(s): Run diffusion server tests (1-GPU)

Commit Info

• sglang: 11cea29c
• aiter: 9127c94a (v0.1.17.dev110+g9127c94a1, source: dev image preserved — [CI-AITER-CHECK] Dev/patched version detected → skipping rebuild; equals Dockerfile default for gfx942)

Failed Tests

Job multimodal-gen-test-1-gpu-amd (…, 3) — all in python/sglang/multimodal_gen/test/single_test_file/test_update_weights_from_disk.py. Result: 5 failed, 1 passed, 4 errors.

Test File	Test Function	Error	Log
test_update_weights_from_disk.py	TestUpdateWeightsFromDisk::test_update_weights_from_disk_default[Qwen-Image]	assert 500 == 200 (line 432)	Log
test_update_weights_from_disk.py	TestUpdateWeightsFromDisk::test_update_weights_specific_modules[Qwen-Image]	Update failed: Inplace update to inference tensor outside InferenceMode is not allowed → assert 500 == 200 (line 454)	Log
test_update_weights_from_disk.py	TestUpdateWeightsFromDisk::test_corrupted_weights_rollback[Qwen-Image]	assert (500 == 200) (line 535)	Log
test_update_weights_from_disk.py	TestUpdateWeightsFromDiskWithOffload::test_update_weights_with_offload_enabled[FLUX.2-klein-base-4B]	Expected 200, got 500 (line 638)	Log
test_update_weights_from_disk.py	TestUpdateWeightsFromDiskWithOffload::test_update_weights_with_offload_enabled[Qwen-Image]	Expected 200, got 500 (line 638)	Log
test_update_weights_from_disk.py	TestUpdateWeightsFromDisk::test_update_weights_from_disk_default[FLUX.2-klein-base-4B]	ERROR at setup: AssertionError: No weights dir for transformer in /tmp/sglang_perturbed_vae_* (fixture line 410 → helper line 220)	Log
test_update_weights_from_disk.py	TestUpdateWeightsFromDisk::test_update_weights_specific_modules[FLUX.2-klein-base-4B]	ERROR at setup: No weights dir for transformer (fixture line 410)	Log
test_update_weights_from_disk.py	TestUpdateWeightsFromDisk::test_update_weights_rejects_invalid_requests[FLUX.2-klein-base-4B]	ERROR at setup: No weights dir for transformer (fixture line 410)	Log
test_update_weights_from_disk.py	TestUpdateWeightsFromDisk::test_corrupted_weights_rollback[FLUX.2-klein-base-4B]	ERROR at setup: No weights dir for transformer (fixture line 410)	Log

Failure Summary

The entire test_update_weights_from_disk.py suite (weight-update-from-disk for the diffusion runtime) fails on AMD MI325. Two symptom groups: (1) the Qwen-Image cases plus both *WithOffload cases return HTTP 500 from POST /update_weights_from_disk (assert 500 == 200); (2) the four FLUX.2-klein-base-4B no-offload cases error at fixture setup because the perturbed model directory has no transformer/vae weights subdir. 1 test passes (test_update_weights_rejects_invalid_requests[Qwen-Image]).

Regression Status

Known recurring failure (≥4 days persistent). Identical signature (5 failed, 1 passed, 4 errors, same Inplace update to inference tensor RuntimeError) in every completed scheduled run of this shard from Jul 4 through today. No green run of this test file found in the queryable window (Jul 1 shard-3 log did not run this file, so it is not a comparable pass).

Recent history of test_update_weights_from_disk.py in job multimodal-gen-test-1-gpu-amd (…, 3) (main, event=schedule, completed runs only):

Date	Run	Job	Test File Status	Failed Functions	Error
Jul 7	28890204713	shard 3	❌ Failed	5 failed / 4 errors	InferenceMode 500 + fixture setup
Jul 6	28814834996	shard 3	❌ Failed	5 failed / 4 errors	same
Jul 6	28794501292	shard 3	❌ Failed	5 failed / 4 errors	same
Jul 6	28775611300	shard 3	❌ Failed	5 failed / 4 errors	same
Jul 4	28689607720	shard 3	❌ Failed	5 failed / 4 errors	same

First observed failure (this window): Jul 4 2026. Last known passing: none found in queryable window. Lookback searched: Jul 1–Jul 7 scheduled runs.

Failure Cluster

Two symptom-level clusters in one test file:

• A — "500 on /update_weights_from_disk: InferenceMode inplace-update rejected" (5 FAILED: all Qwen-Image + both offload cases)
• B — "Fixture setup AssertionError: perturbed model dir missing weights subdir" (4 ERROR: FLUX.2-klein-base-4B no-offload cases)

Facts (verified observations)

• Cluster A stack trace (weights_updater.py:339 → :139 → :200 → linear.py:411 assert param_data.shape == loaded_weight.shape → AssertionError → rollback weights_updater.py:381 → linear.py:533 → :575 param_data.copy_(loaded_weight) → RuntimeError: Inplace update to inference tensor outside InferenceMode is not allowed). The initial shape assertion fails, rollback is triggered, and the rollback copy_ then trips the InferenceMode guard, so the endpoint returns 500. Verbatim from log lines 5914–5955.
• Many warnings Skipping weight update for FSDPQwen3ForCausalLM: parameter 'model.layers.N…' not found after mapping precede the failure — the text_encoder/transformer weight-name mapping is not matching the loaded checkpoint (log ~5511+).
• Cluster B: AssertionError: No weights dir for transformer in /tmp/sglang_perturbed_vae_* and .../vae raised inside the diffusion_server_no_offload fixture at test_update_weights_from_disk.py:410, calling _compute_checksum_from_disk at line 220 (assert os.path.exists(.../transformer)). Also raised in background threads (PytestUnhandledThreadExceptionWarning, Thread-2/3/8/9). Log lines 12730–12872.
• Reproducibility: identical 5 failed, 1 passed, 4 errors counts and identical InferenceMode RuntimeError in 5 consecutive completed scheduled runs (Jul 4–7), shard 3.
• Environment: aiter v0.1.17.dev110+g9127c94a1 (dev image, unchanged); triton mismatch warning (AITER_USE_SYSTEM_TRITON=1: require triton>=3.6.0, found 3.4.0) present but non-fatal.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [MEDIUM] Cluster A: the diffusion weight-update path (weights_updater.py + multimodal_gen/runtime/layers/linear.py) is broken for the transformer/DiT module weights — the shape assertion at linear.py:411 fails (checkpoint↔param shape/name mismatch), and the rollback code path performs an in-place copy_ on inference-mode tensors (linear.py:575), which PyTorch forbids.
  • Supporting: stack trace directly names these files/lines; the "parameter not found after mapping" warnings show the weight-name mapping is failing before the assertion; identical across 5 runs.
  • Disconfirming: cannot confirm it is AMD-specific — no NVIDIA baseline for this diffusion suite was checked; the 2-GPU shard 0 of the same workflow is sometimes green, so the break may be model/param-specific rather than backend-specific.
  • How to verify: run test_update_weights_from_disk.py locally on MI325 in the CI image; inspect whether param_data is under torch.inference_mode() at rollback; check if the module should wrap rollback in torch.no_grad()/clone before copy_.
• [MEDIUM] Cluster B: test-fixture/data issue — the perturbed_vae_model_dir built by the diffusion_server_no_offload fixture does not contain per-module weight subdirs (transformer/, vae/), so _compute_checksum_from_disk asserts on a missing path. This looks like a test-helper/layout assumption that does not hold for FLUX.2-klein-base-4B on this runner.
  • Supporting: assertion is inside test helper code (test_update_weights_from_disk.py:220), not product code; only affects the FLUX.2 no-offload param.
  • Disconfirming: could be downstream of a failed/partial model download rather than a test bug (a HF-cache path resolution error appears later in the log at line 12677).
  • How to verify: inspect the perturbed-dir construction in the fixture; confirm the FLUX.2 snapshot on /sgl-data/hf-cache has transformer/ and vae/ subdirs.
• [LOW] Regression introduced by a specific commit.
  • Supporting: n/a — failure predates the queryable window.
  • Disconfirming: no green run exists in the searched window, and the local clone is shallow (1 commit), so a pass_sha..fail_sha bisect window cannot be derived here.
  • How to verify: query the sister nightly-test-amd history / older pr-test-amd runs (before Jul 1) to find the last green run of this test file, then bisect.

No hypothesis reaches HIGH; needs a local repro or a green baseline to bisect against.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for update_weights_from_disk (12 hits, all unrelated: RL /pull_weights, MoE quant, etc.), Inplace+update+inference+tensor (11 hits, all unrelated), InferenceMode+weight (0). No open PR targets the multimodal_gen weight-update-from-disk InferenceMode/fixture failure.

Suggested Triage Steps

• Find the last green run of test_update_weights_from_disk.py in older pr-test-amd.yml (before Jul 1) or nightly-test-amd.yml history, then bisect pass_sha..fail_sha for changes under python/sglang/multimodal_gen/runtime/post_training/ and .../runtime/layers/linear.py.
• Cluster A: reproduce locally on MI325 in the CI image; check whether the rollback path (weights_updater.py:381) must clone()/exit inference_mode before param_data.copy_(loaded_weight) at linear.py:575, and why the initial shape assertion at linear.py:411 fails (weight-name mapping for FSDPQwen3ForCausalLM).
• Cluster B: inspect the diffusion_server_no_offload fixture's perturbed-dir builder and confirm FLUX.2-klein-base-4B snapshot layout on /sgl-data/hf-cache includes transformer/ and vae/ subdirs; if it is a test-data assumption bug, fix the helper rather than the product.
• If confirmed AMD-only, compare against an NVIDIA run of the same diffusion suite to localize backend vs. product logic.

Status

• Persistence: Failing every completed scheduled run since Jul 4 2026 (≥4 days), shard 3, identical 5 failed / 1 passed / 4 errors signature.
• Scope: multimodal-gen-test-1-gpu-amd shard 3 in pr-test-amd.yml. Other multimodal-gen shards (0/1/2) also show failures in these runs but were not confirmed at test-file level here; this analysis covers shard 3 only.
• Blocked work: diffusion runtime weight-update-from-disk CI signal on AMD.
• In-flight fix: ❌ none found in open PRs.

multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 1) — failed step(s): N/A

Commit Info

• sglang: 11cea29c
• aiter: N/A for this run — the current job's log blob is unavailable (BlobNotFound; the run is still in-flight, status=queued), so [CI-AITER-CHECK] markers cannot be read. The most recent completed shard-1 run on a near-identical sglang tree used v0.1.17.dev110+g9127c94a1 (dev image preserved; Dockerfile default 9127c94a). aiter is not implicated (see below).

Failed Tests

Test File	Test Function	Error	Log
test/../multimodal_gen/test/server/test_server_1_gpu.py	TestDiffusionServerOneGpu::test_diffusion_generation[lingbot_world_realtime_plastic_beach]	TimeoutError: Server not ready within 1200.0s (setup ERROR) — inferred, not confirmed for this job	job

⚠️ Coverage caveat: the current job's log is unavailable, so the exact failing test cannot be confirmed for this specific run. The row above is the failure that has hit shard (…, 1) in every completed run Jul 5–7. See Regression Status.

Failure Summary

Job multimodal-gen-test-1-gpu-amd (…, 1) ran ~61 min (21:19:58→22:21:00 UTC) then ended with conclusion failure while step 9 "Run diffusion server tests (1-GPU)" was still in_progress and step 10 pending — the signature of the runner being terminated mid-test, not a clean pytest exit. The step's own timeout is 90 min (pr-test-amd.yml:712), so the 61-min kill was external. Across all completed shard-1 runs this partition fails on one test — test_diffusion_generation[lingbot_world_realtime_plastic_beach] — whose diffusion server never becomes ready in 1200 s because it stalls re-downloading its model from HF Hub.

Regression Status

Never-passed on AMD (recurring since test introduction). The test case was added 2026-07-04 in b7c3709f (#29903); no green run has been observed since.

Recent history of test_server_1_gpu.py (test_diffusion_generation[lingbot_world_realtime_plastic_beach]) in job multimodal-gen-test-1-gpu-amd (…, 1) (branch=main, event=schedule, completed runs only):

Date (UTC)	Run	Test Status	Failed Function	Error
Jul 7 ~22:21	28890204713	⚠️ Unconfirmed (log unavailable)	—	job killed at 61 min, step in_progress
Jul 7 ~02:21	28814834996	❌ ERROR	[lingbot_world_realtime_plastic_beach]	TimeoutError: Server not ready within 1200.0s
Jul 6 ~19:26	28794501292	❌ ERROR	[lingbot_world_realtime_plastic_beach]	TimeoutError: Server not ready within 1200.0s
Jul 6 ~15:31	28775611300	❌ Not reached	(step timed out at 90 min before lingbot)	action … timed out after 90 minutes
Jul 5 ~19:52	28750542672	❌ ERROR	[lingbot_world_realtime_plastic_beach]	TimeoutError: Server not ready within 1200.0s

Note: shard/partition assignment is not stable across runs (the same run showed shards with 11 vs 13 items), so the verdict is tracked at the test-function level, not the partition index. Lookback window: Jul 5–7 (5 completed schedule runs); no PASS of this test found.

Failure Cluster

Diffusion server startup timeout — model snapshot re-download stall (HF cache "incomplete")

Facts (verified observations)

• Verbatim error (run 28814834996, test_server_utils.py:492 in _wait_for_ready): E TimeoutError: Server not ready within 1200.0s.
• Server startup sequence for lingbot_world_realtime_plastic_beach (model robbyant/lingbot-world-fast-diffusers, pipeline LingBotWorldCausalDMDPipeline): server logs Model found in cache but incomplete, will download from HF Hub → Downloading model snapshot from HF Hub … (attempt 1/5) → Fetching 34 files progresses to ~88% (30/34) then the 1200 s ready-deadline expires. First pytest attempt ran 1:03:18 and produced 11 passed, 1 skipped, 19 deselected, 1 error.
• The model robbyant/lingbot-world-fast-diffusers is not in the workflow's pre-cache diagnostic list (only Wan2.1, HunyuanVideo, Qwen-Image, FLUX.1, FLUX.2 are checked — pr-test-amd.yml:698), so it is fetched fresh each run.
• Cascading retry failure: on pytest retry the server dies immediately with RuntimeError: Scheduler port 5555 is unavailable and --strict-ports is enabled (server_args.py:788) — the scheduler from the first (hung, then killed) server had not released port 5555. This is a secondary effect of the primary hang, not an independent bug.
• Reproducibility: identical TimeoutError: Server not ready within 1200.0s for this test in 3/3 completed shard-1 runs where it was reached (Jul 5, Jul 6 19:26, Jul 7 02:21); the 4th completed run (Jul 6 15:31) hit the 90-min step timeout before reaching it.
• aiter/torch are unchanged confounders: the failure is a model-download stall before any inference; the [aiter] using torch solution / triton-version warnings in the log are pre-inference startup noise.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [HIGH] The lingbot_world_realtime_plastic_beach diffusion server cannot reach ready state within the 1200 s deadline because its 34-file model snapshot (robbyant/lingbot-world-fast-diffusers) is not persistently cached and re-downloads on every run, stalling past the deadline.
  • Supporting: three independent completed runs show the same TimeoutError: Server not ready within 1200.0s preceded by Model found in cache but incomplete, will download from HF Hub and a Fetching 34 files bar that never completes; model absent from the pre-cache list.
  • Disconfirming: cannot rule out that the underlying HF download is failing for network reasons (mirror 10.44.14.109:5000 context-deadline errors appear elsewhere in the log) rather than pure slowness; a fully-populated cache would need one clean download to confirm this is a cache-persistence issue vs. a persistently slow/failing fetch.
  • How to verify: pre-seed /sgl-data/hf-cache/hub/models--robbyant--lingbot-world-fast-diffusers on the runner (or add it to the pre-cache diagnostic + warm step) and re-run shard 1; if it then passes, the cause is confirmed as cache non-persistence.
• [LOW] The current job (85700654705) failed for a different reason (runner infra loss) than the recurring lingbot stall.
  • Supporting: it was killed at 61 min (< 90 min step timeout) with step still in_progress — atypical of the clean setup-ERROR seen in other runs, which completed at 73–99 min.
  • Disconfirming: every completed shard-1 run is red on the same test; a 61-min kill is also consistent with the runner/harness terminating a job stuck in the 20-min lingbot stall.
  • How to verify: re-read this job's log once the run completes and the blob is available (/actions/jobs/85700654705/logs).

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for lingbot, diffusion server not ready, strict-ports scheduler. Matches are feature PRs (e.g. #30361 support LingBot-World 2.0, #28065), none address model-cache persistence or the 1200 s startup timeout for this test on AMD CI.

Suggested Triage Steps

• Confirm the current job's actual failure once the run completes: re-fetch /actions/jobs/85700654705/logs and grep for Server not ready within 1200.0s vs. a runner-infra message.
• Warm/pre-cache robbyant/lingbot-world-fast-diffusers into the persistent /sgl-data/hf-cache (and add it to the "Diagnose HF cache" list at pr-test-amd.yml:698); re-run shard 1 to see if the stall clears.
• If the cache is already present but reported "incomplete", investigate the HF Hub download path / internal mirror (10.44.14.109:5000 deadline errors in the log) and consider raising the per-server ready deadline for this model or splitting it into its own longer-timeout shard.
• Independently, the retry Scheduler port 5555 unavailable cascade suggests the port isn't released after a killed hung server — worth ensuring kill_process_tree frees the scheduler port before retry.

Status (factual)

• Persistence: test_diffusion_generation[lingbot_world_realtime_plastic_beach] has failed in every completed shard-1 run it was reached in, Jul 5–7 2026; never observed passing since introduction on 2026-07-04 (#29903).
• Scope: shard (…, 1) of multimodal-gen-test-1-gpu-amd in pr-test-amd.yml. The unstable partitioning means the same stall can also blow the 90-min step budget for whichever shard the test lands in.
• This job specifically: root cause unconfirmed — log blob unavailable while run is in-flight; killed at 61 min with the test step in_progress.
• In-flight fix: ❌ none found — needs a new CI cache-warm PR or investigation.

stage-c-test-large-8-gpu-amd-mi35x (linux-mi35x-gpu-8, 1) — failed step(s): Ensure VRAM is clear

Commit Info

• sglang: 11cea29c
• aiter: N/A — the job failed at the pre-flight VRAM-cleanup step before dependency install; no [CI-AITER-CHECK] markers were emitted.

Failed Tests

Test File	Test Function	Error	Log
N/A (no test ran)	N/A	Pre-flight step ensure_vram_clear.sh rocm failed: VRAM not clear after 3 attempts → exit 1	Log

No tests executed. The job died in the Run bash scripts/ci/amd/ensure_vram_clear.sh rocm step (the first step after checkout), so the container/dependencies were never set up and no pytest suite ran.

Failure Summary

The stage-c-test-large-8-gpu-amd-mi35x (linux-mi35x-gpu-8, 1) job failed during the pre-flight GPU VRAM-cleanup guard on runner linux-mi35x-gpu-8-zf775-runner-dc4rc. At job start GPUs 0–3 each already held 16% VRAM left over from a previous workload, and three cleanup attempts (SIGTERM → SIGKILL → 30 s wait) could not free it, so the guard exited 1. This is an environment/runner-state failure, not a test or code failure.

Regression Status

Infrastructure issue (recurring on this runner) — not a code regression.

Recent history of this pre-flight VRAM guard in job stage-c-test-large-8-gpu-amd-mi35x (linux-mi35x-gpu-8, 1) (branch main, event schedule, completed runs only):

Date	Run	Job	Step Status	Failure	Error
Jul 7 19:27	28890204713	shard 1	❌ Failed	VRAM guard	VRAM cleanup unsuccessful after 3 attempts (GPUs 0-3 @16%)
Jul 6 20:10	28814834996	shard 1	❌ Failed	VRAM guard	VRAM cleanup unsuccessful after 3 attempts (same symptom)
Jul 6 13:18	28794501292	shard 1	✅ Passed	—	—
Jul 6 07:38	28775611300	shard 1	✅ Passed	—	—

The identical guard failure occurred in the previous scheduled run on the same shard; the two runs before that passed. The failure tracks the runner node's residual GPU state, not the sglang commit under test.

Failure Cluster

Pre-flight VRAM-cleanup guard failure (zombie KFD context / driver-level VRAM leak on AMD mi35x-gpu-8 runner)

Facts (verified observations, no interpretation)

• Failing step: bash scripts/ci/amd/ensure_vram_clear.sh rocm, exit code 1 (log line 547).
• At job start (before any sglang work) GPUs 0, 1, 2, 3 each reported GPU Memory Allocated (VRAM%): 16; threshold is 5% (log lines 169–173).
• 3 cleanup attempts, each SIGTERM → 5 s → SIGKILL → 30 s wait, all left GPUs 0-3 at 16% (attempts at lines 212, 321, 430; final FAIL at line 539).
• rocm-smi --showpids reported No KFD PIDs currently running on every attempt (lines 278, 387, 496); fuser/lsof/zombie/docker diagnostics all empty — no host process or container held the GPUs.
• Script's own diagnosis: almost certainly a zombie KFD context (see ROCm/aiter#2061). The node will need to be rebooted (lines 542–545).
• AITER_COMMIT_OVERRIDE: is empty (line 36) — this is a regular scheduled pr-test-amd.yml run, not an AITER Scout run.
• Script exists in repo at scripts/ci/amd/ensure_vram_clear.sh (verified locally).

Hypothesised Causes (with confidence, with disconfirming evidence)

• [FACT] The job failed because the runner's GPUs 0-3 held 16% residual VRAM at job start and could not be freed by the cleanup guard.
  • Supporting: log lines 169–173, 539–547 show the guard detecting >5% VRAM and exiting 1 after 3 attempts.
  • Disconfirming: (none — directly observable in the log).
• [HIGH] Root cause is a driver-level "zombie" KFD context (allocation with no owning host process) on runner linux-mi35x-gpu-8-zf775-runner-dc4rc, requiring a node reboot — i.e. contaminated runner state left by a prior job, unrelated to commit 11cea29c.
  • Supporting: VRAM allocated but No KFD PIDs currently running and no process/container holds any GPU device file — allocation persists with no owner; matches the exact zombie-KFD signature the script warns about ([Bug] Custom all-reduce IPC buffers use fixed VA, conflict with torch_memory_saver pause/resume → zombie processes on MI300X ROCm/aiter#2061). Two of the last four scheduled runs of this shard passed on the same code paths, and the failure recurs across different sglang HEAD SHAs (Jul 6 52c6e27e and Jul 7 11cea29c).
  • Disconfirming: (none found) — no evidence points to the code under test; the guard runs before any sglang code executes.
• [LOW] The prior job on this runner leaked VRAM on exit (rather than the driver itself), and a more thorough shutdown cleanup would have prevented the leak.
  • Supporting: in-flight PR #30431 targets "graceful GPU cleanup on shutdown", implying leaks originate from prior-job shutdown.
  • Disconfirming: No KFD PIDs + empty fuser/lsof means the leaking process is already gone, so the leak is at the driver/KFD layer and not reclaimable by the next job's userspace cleanup — consistent with needing a reboot.

In-flight Fix Check

• In-flight fix: ⚠️ [#30431](https://github.com/sgl-project/sglang/pull/30431) — "[DO NOT MERGE][AMD] ci: self-heal driver-level VRAM leaks + graceful GPU cleanup on shutdown". Directly targets this exact failure mode (driver-level VRAM leaks). Currently marked DO NOT MERGE — track its progress rather than opening a duplicate.
• API queried: search/issues?q=repo:sgl-project/sglang+is:pr+is:open+VRAM+zombie+KFD (2 hits; the other, #28534, is unrelated HiCache work).

Suggested Triage Steps

• Reboot / cordon the runner linux-mi35x-gpu-8-zf775-runner-dc4rc to clear the zombie KFD context, then re-run the job — the guard itself states the node needs a reboot before subsequent jobs can succeed.
• Identify the prior workload on that node (the job immediately before 28890204713 on zf775-runner-dc4rc) that left GPUs 0-3 at 16%; that job's shutdown path is the likely leak source.
• Track in-flight PR #30431 and, once it moves past "DO NOT MERGE", evaluate whether its self-heal step (e.g. driver reset) belongs in ensure_vram_clear.sh so the guard can recover without a manual reboot.
• Cross-reference ROCm/aiter#2061 for driver-side mitigation.

Status (factual, no priority assignment)

• Persistence: Failed in the 2 most recent completed scheduled runs of this shard (Jul 6 20:10 and Jul 7 19:27 UTC); the 2 runs before that (Jul 6 07:38 & 13:18) passed. Intermittent, tied to runner-node state.
• Scope: Pre-flight VRAM guard on the AMD mi35x 8-GPU runner (linux-mi35x-gpu-8-zf775-runner-dc4rc). Blocks the entire stage-c-test-large-8-gpu-amd-mi35x shard-1 suite from running — 0 tests execute when it triggers.
• Blocked work: All stage-c large-8-GPU mi35x tests in shard 1 for this run get no CI signal.
• In-flight fix: ⚠️ #30431 (marked DO NOT MERGE) — self-heal driver-level VRAM leaks; track, do not duplicate.

stage-c-test-large-8-gpu-amd-mi35x (linux-mi35x-gpu-8, 0) — failed step(s): Run test

Commit Info

• sglang: 11cea29c
• aiter: 9127c94a (source: dev image v0.1.17.dev110+g9127c94a1, matches Dockerfile default → no rebuild)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/test_kimi_k25_mxfp4_bcg_mi35x.py	TestKimiK25MXFP4BcgMI35x.test_bcg_gsm8k	Exception: Server process exited with code 1 → root: ValueError: KimiK25ForConditionalGeneration has no SGlang implementation	Log

(The sister file in the same job, test/registered/cuda_graph/breakable/test_breakable_cuda_graph.py, passed. Job result was 1/2 passed → exit code 255.)

Failure Summary

The Kimi-K2.5 MXFP4 server failed to launch on MI35x: architecture resolution raised ValueError: KimiK25ForConditionalGeneration has no SGlang implementation and the Transformers implementation is not compatible with SGLang (log:4543). The underlying reason is that the kimi_k25 model module (and the whole DeepSeek-family import graph it depends on) failed to import at model-registry time with ModuleNotFoundError: No module named 'cutlass', so KimiK25ForConditionalGeneration was never registered. The test aborted at server launch after ~43s (vs ~2290s full-eval in the last passing run).

Regression Status

New regression.

Recent history of test_kimi_k25_mxfp4_bcg_mi35x.py (test_bcg_gsm8k) in job stage-c-test-large-8-gpu-amd-mi35x (branch main, event schedule, completed runs only):

Date (UTC)	Run	Job (shard)	Test File Status	Failed Function	Error
Jul 7 18:43	28890204713	shard 0	❌ Failed	test_bcg_gsm8k	has no SGlang implementation
Jul 6 18:40	28814834996	shard 0	✅ Passed	—	passed:true, elapsed=2290s
Jul 6 13:18	28794501292	shard 0	✅ Passed	—	passed:true, elapsed=2237s

• First observed failure: Jul 7 18:43 UTC run (this run). Last known pass: Jul 6 18:40 UTC (52c6e27e).
• The 3 intervening scheduled runs (28833386129, 28848583109, 28867568883) are still queued [IN-FLIGHT] and excluded.

Failure Cluster

ModuleNotFoundError: No module named 'cutlass' poisons DeepSeek-family model registration on ROCm (server launch fails "has no SGlang implementation").

Facts (verified observations)

• Passing run 85451537137: 0 occurrences of No module named 'cutlass'; KimiK25ForConditionalGeneration loaded successfully (Load weight end … type=KimiK25ForConditionalGeneration, quant=quark).
• Failing run 85700655616: 71 occurrences of No module named 'cutlass', including Ignore import error when loading sglang.srt.models.deepseek_v2 … No module named 'cutlass' and …kimi_k25 … No module named 'cutlass' (log:4479, 4498).
• Import chain (verified at failing SHA 11cea29c): models/deepseek_v2.py:61 from …dsa.dsa_indexer import Indexer → dsa_indexer.py:11 from sglang.jit_kernel.dsa import (…) → jit_kernel/dsa/__init__.py:1 eager from .cutedsl_paged_mqa_logits import … → top-level import cutlass (layers/quantization/nvfp4_gemm_swiglu_nvfp4_quant.py / cutedsl kernels). kimi_k25.py:30 imports DeepseekV3ForCausalLM from deepseek_v2, so it fails too.
• Stack top (verified): model_loader/utils.py:182 resolve_transformers_arch → raise ValueError(...) (log:4541-4543).
• cutlass is the NVIDIA CUTLASS Python DSL; it is not installed in the ROCm CI image, so any eager import cutlass on ROCm raises ModuleNotFoundError.

Hypothesised Causes (with confidence)

• [HIGH] Commit fefc1743 (PR #25220, "Cute-DSL FP8 MQA logits") introduced an unguarded eager import cutlass into the model import graph — it added jit_kernel/dsa/__init__.py with a top-level from .cutedsl_paged_mqa_logits import … and wired from sglang.jit_kernel.dsa import (…) into dsa_indexer.py (imported at deepseek_v2.py top level). On ROCm this breaks registration of every DeepSeek-derived model, including kimi_k25.
  • Supporting: passing run has 0 cutlass errors + KimiK25 loaded; failing run has 71 + KimiK25 unregistered (A/B log evidence). fefc1743 is an ancestor of failing SHA 11cea29c but merged (Jul 7 06:34 UTC) after the last passing run's SHA 52c6e27e (Jul 6 18:40 UTC). Exact import chain verified in source at the failing SHA.
  • Disconfirming: (none found) — the fix below directly guards this exact code path and confirms the mechanism.
  • How to verify: on MI35x, python3 -c "import sglang.srt.models.deepseek_v2" at 11cea29c reproduces ModuleNotFoundError: No module named 'cutlass'; guarding the eager import removes it.

In-flight Fix Check

• Already merged: ✅ #30374 "[AMD] Fix DeepSeekV4 server cutlass error" (40a68521), merged 2026-07-07T21:43:06Z — ~3h after this run started (18:43 UTC), so it is NOT in the failing SHA (verified: not an ancestor of 11cea29c). It wraps the eager cutedsl import in jit_kernel/dsa/__init__.py and dsa_indexer.py with if not is_hip():. (API call: search/issues?q=repo:sgl-project/sglang+is:pr+cutlass.)

Suggested Triage Steps

• No new fix needed — confirm #30374 resolves this by checking the next completed scheduled pr-test-amd.yml run whose head SHA includes 40a68521; expect test_kimi_k25_mxfp4_bcg_mi35x.py green again.
• If it recurs, re-check for other unguarded top-level import cutlass sites reachable from model modules (e.g. layers/attention/linear/kernels/kda_blackwell/kernel_h.py, nvfp4_gemm_swiglu_nvfp4_quant.py) and extend the is_hip() guard pattern.

Status

• Persistence: First failing completed run (Jul 7 18:43 UTC); last completed pass Jul 6 18:40 UTC.
• Scope: Affects all DeepSeek-family model registrations on ROCm (deepseek_v2/v4, kimi_k25, glm4_moe, longcat, etc. show the same No module named 'cutlass'); this job surfaces it via the Kimi-K2.5 launch. Likely also breaks other AMD Kimi/DeepSeek server tests in the same window.
• In-flight fix: ✅ merged #30374 at 2026-07-07T21:43:06Z (post-dates this run).

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-07-08 18:54 UTC)

[amd-bot]

pr-test-amd.yml — Per-Job Analysis (continued 2/9)

stage-b-test-1-gpu-small-amd (linux-mi325-1gpu-sglang, 6) — failed step(s): Run test

Commit Info

• sglang: 11cea29c
• aiter: 9127c94a — v0.1.17.dev110+g9127c94a1 (source: dev image; [CI-AITER-CHECK] reports "Dev/patched version detected → skipping rebuild", Dockerfile default is the same 9127c94a)

Failed Tests

Test File	Test Function	Error	Log
test/registered/hicache/test_hicache_variants.py	TestHiCacheMLA.setUpClass	Exception: Server process exited with code -9 (server crash: KeyError: 'sglang')	Log

Suite result: Test Summary: 9/10 passed — test_hicache_variants.py is the sole failure (exit code 1). The other 9 test files in this shard passed.

Failure Summary

TestHiCacheMLA launches a server for deepseek-ai/DeepSeek-Coder-V2-Lite-Instruct (--trust-remote-code --enable-hierarchical-cache). The scheduler crashed during model load with KeyError: 'sglang', so the server exited with code -9 and setUpClass raised. The crash occurs because SGLang's native deepseek_v2 model failed to import (No module named 'cutlass') and the loader fell back to the Transformers backend, which the model's remote code does not support.

Regression Status

New regression. Recent history of test/registered/hicache/test_hicache_variants.py (TestHiCacheMLA.setUpClass) in stage-b-test-1-gpu-small-amd, branch main, event=schedule, completed runs only (shard number varies per run — verified at the test-file level):

Date (UTC)	Run	Job (shard w/ hicache)	Test File Status	Failed Function	Error
Jul 7 18:43	28890204713	shard 6	❌ Failed	TestHiCacheMLA.setUpClass	KeyError: 'sglang' → exit -9
Jul 6 18:40	28814834996	shard 6	✅ Passed	—	10/10 passed, elapsed 649s

• First observed failure: run 28890204713 (Jul 7 18:43 UTC). Last known pass: Jul 6 18:40 UTC (52c6e27e).
• The three intervening scheduled runs (Jul 7 00:44 / 07:15 / 12:52) are still queued/in-flight per the API and are excluded [IN-FLIGHT].
• Regression window: 52c6e27e..11cea29c (45 commits).

Failure Cluster

deepseek_v2 import fails on ROCm (No module named 'cutlass') → Transformers fallback KeyError 'sglang'

Facts (verified observations)

• Log (fail run): Ignore import error when loading sglang.srt.models.deepseek_v2: No module named 'cutlass', then DeepseekV2ForCausalLM has no SGLang implementation, falling back to Transformers implementation.
• Crash trace (fail run): models/transformers.py:617 AutoModel.from_config(...) → remote modeling_deepseek.py:1203 ATTENTION_CLASSES[config._attn_implementation] → KeyError: 'sglang'. transformers.py:614 sets self.text_config._attn_implementation = "sglang", but the model's remote code has no 'sglang' entry in its ATTENTION_CLASSES dict.
• A/B temporal evidence: Jul 6 passing shard log (85451536783) had zero occurrences of the cutlass import error, zero KeyError: 'sglang', zero Transformers fallback — deepseek_v2 imported natively and the test passed 10/10. The fail run has all three.
• Import chain (verified in source, all unguarded/unconditional and reached on ROCm):
  • models/deepseek_v2.py:61 → from ...attention.dsa.dsa_indexer import Indexer
  • attention/dsa/dsa_indexer.py:11 → from sglang.jit_kernel.dsa import (... cutedsl_paged_mqa_logits ...)
  • jit_kernel/dsa/__init__.py:1 → from .cutedsl_paged_mqa_logits import CuteDSLPagedMQALogitsRunner, pick_dsl_expand
  • jit_kernel/dsa/cutedsl_paged_mqa_logits.py:15 → import cutlass (NVIDIA CuTe DSL, not installed on ROCm) → ModuleNotFoundError.

Hypothesised Causes

• [HIGH] Commit fefc1743 ("Cute-DSL FP8 MQA logits", #25220) added the unguarded import cutlass chain that breaks deepseek_v2 import on ROCm.
  • Supporting: git show fefc1743 confirms it added the unconditional from sglang.jit_kernel.dsa import (... cutedsl_paged_mqa_logits ...) in dsa_indexer.py and created jit_kernel/dsa/__init__.py that eagerly imports cutedsl_paged_mqa_logits (which does top-level import cutlass). Committed 2026-07-07 06:34 UTC — inside the window, after the Jul 6 18:40 pass. deepseek_v2 imports dsa_indexer unconditionally at module top level, so any non-CUDA platform (ROCm, NPU) now fails to import it.
  • Disconfirming: (none material). The other cute-dsl commit in the window, e85ef548 (#30117), also touched deepseek_v2.py and was considered, but its cutlass/cutedsl imports (via unquant.py and fused_a_gemm.py) are all lazy/function-scoped or behind if _is_cuda: guards, so it does NOT break the ROCm import path — ruled out.
  • How to verify: On an AMD MI325 container, python3 -c "import sglang.srt.models.deepseek_v2" should raise ModuleNotFoundError: No module named 'cutlass' at HEAD, and succeed after reverting fefc1743 (or after making jit_kernel/dsa/__init__.py's cutlass import lazy/guarded for non-CUDA).

In-flight Fix Check

• In-flight fix: ⚠️ partial — #30410 "[NPU] Fix npu cutlass import" (open since 2026-07-07 15:42 UTC) edits the same dsa_indexer.py but only guards if not is_npu(). On ROCm is_npu() is False, so the from sglang.jit_kernel.dsa import ... still runs and still triggers import cutlass (the PR does not touch jit_kernel/dsa/__init__.py). This PR will NOT fix the AMD/ROCm failure — it needs to also exclude HIP (e.g. if not is_npu() and not is_hip()) or make the cutlass import in the jit_kernel/dsa package lazy.
• API queried: search/issues?q=repo:sgl-project/sglang+is:pr+is:open+cutlass+import.

Suggested Triage Steps

• Reproduce: run python3 -c "import sglang.srt.models.deepseek_v2" in the AMD CI container at 11cea29c — expect ModuleNotFoundError: No module named 'cutlass'.
• Confirm culprit: revert fefc1743 on a branch and re-run test/registered/hicache/test_hicache_variants.py (TestHiCacheMLA) on AMD MI325.
• Fix direction (hypothetical): make the import cutlass in python/sglang/jit_kernel/dsa/cutedsl_paged_mqa_logits.py / jit_kernel/dsa/__init__.py lazy or CUDA-only, and/or extend #30410's guard to cover ROCm (is_hip()), so deepseek_v2 imports natively on AMD again.
• Coordinate with #30410 author — the NPU-only guard is the same bug class; a combined non-CUDA guard fixes both platforms.

Status

• Persistence: First failing completed run Jul 7 18:43 UTC (28890204713); last pass Jul 6 18:40 UTC. New as of this run (intervening runs in-flight).
• Scope: Breaks native import of deepseek_v2 (and any model that unconditionally imports dsa_indexer) on all non-CUDA platforms lacking cutlass — AMD/ROCm here, likely NPU too. Concretely fails TestHiCacheMLA in stage-b-test-1-gpu-small-amd; other Transformers-fallback-incompatible DeepSeek tests on AMD are at risk.
• In-flight fix: #30410 open but NPU-only — does not cover AMD/ROCm; a HIP-inclusive fix is still needed.

stage-a-test-1-gpu-small-amd (linux-mi325-1gpu-sglang) — failed step(s): Run test

Commit Info

• sglang: e339c83f
• aiter: 9127c94a (v0.1.17.dev110+g9127c94a1) — source: dev image preserved ([CI-AITER-CHECK] Dev/patched version detected → skipping rebuild; Dockerfile default was the same commit, no AITER_COMMIT_OVERRIDE)

Failed Tests

Test File	Test Function	Error	Log
test/registered/models/test_vit_pos_embed_interpolate.py	TestViTPosEmbedInterpolate.test_moss_vl_vectorized_matches_loop	Hang → per-test TIMEOUT after 1200 seconds, then step timed out after 30 minutes	TIMEOUT · step error

Failure Summary

The stage-a-test-1-gpu-small-amd suite runs 6 tests sequentially. The first test, test_vit_pos_embed_interpolate.py, hung during test_moss_vl_vectorized_matches_loop: it printed the [CI Test Method] marker and reached Automatically detected platform rocm, then produced no further output for ~19.5 minutes until the harness killed it at the 1200s per-test timeout. The runner (--continue-on-error) advanced to the second test (test_wave_attention_kernels.py), but the overall "Run test" step hit its 30-minute cap and failed. No stack trace, assertion, or error message was emitted — this is a silent hang, not a numerical/logic failure.

Regression Status

New failure (first observed in this run) — likely infrastructure hang / flake pending confirmation.

Recent history of test_vit_pos_embed_interpolate.py (test_moss_vl_vectorized_matches_loop) in job stage-a-test-1-gpu-small-amd (linux-mi325-1gpu-sglang) (branch main, event schedule, completed runs only):

Date	Run	Job	Test File Status	Failed Function	Error
Jul 07 12:52	28867568883	stage-a	❌ Hung/Timeout	test_moss_vl_vectorized_matches_loop	1200s TIMEOUT
Jul 07 01:20	28814834996	stage-a	✅ Passed	—	Ran 2 tests in 11.591s
Jul 06 13:18	28794501292	stage-a	✅ Passed	—	—
Jul 06 07:38	28775611300	stage-a	✅ Passed	—	—
Jul 06 00:44	28760739527	stage-a	✅ Passed	—	—

First observed failure: 2026-07-07 15:27 UTC. Last known pass: 2026-07-07 01:20 UTC (run 28814834996, ~11.6s). The two intervening scheduled runs (28833386129, 28848583109) are still queued and were excluded.

Failure Cluster

Silent GPU-init hang / test timeout on AMD MI325 stage-a suite

Facts (verified observations)

• Timeline (verbatim from log, step 6): 15:26:51 suite enables 6 tests; 15:26:51 Begin (0/5) for test_vit_pos_embed_interpolate.py; 15:27:05 [CI Test Method] TestViTPosEmbedInterpolate.test_moss_vl_vectorized_matches_loop; 15:27:09 Automatically detected platform rocm; ~19.5 min of silence; 15:46:51 kill_process_tree called: parent_pid=3175 … pid=2794; 15:46:56 ✗ TIMEOUT: …test_vit_pos_embed_interpolate.py after 1200 seconds; 15:56:55 ##[error]The action 'Run test' has timed out after 30 minutes.
• The hang occurs after rocm platform detection and before any assertion. test_qwen3_vl_vectorized_matches_loop (the sibling method) was never reached.
• In the last passing run both methods completed in 11.591s (OK, elapsed=36s incl. startup) — the test's normal runtime is ~12s vs. the 1200s cap.
• The test is a lightweight bit-exact CPU/CUDA math unit test (nn.Embedding lookups + arithmetic, est_time=20s); it constructs no server and loads no model weights.
• Source unchanged in window: test/registered/models/test_vit_pos_embed_interpolate.py last modified by #29903 (b7c3709f); python/sglang/srt/models/moss_vl.py had no commits in the last-pass→fail window 52c6e27e..e339c83f (44 commits total).
• aiter version identical to passing runs (dev image 9127c94a, no override).

Hypothesised Causes (with confidence)

• [LOW] Infrastructure / GPU-init hang on the MI325 runner (linux-mi325-1gpu-sglang-57dnj-runner-4mwb8).
  • Supporting: hang is silent (no traceback/assertion), occurs immediately after rocm platform detection during the first GPU op / model import of the suite's first test; the exact test code passed in 4 consecutive prior completed runs in ~12s; test + moss_vl.py unchanged; aiter unchanged. Silent multi-minute stalls at first-GPU-touch are a classic runner/driver symptom.
  • Disconfirming: only one completed occurrence so far — cannot yet distinguish infra flake from a newly-introduced deadlock; the two runs between last-pass and this one are still queued, so a persistent pattern can't be ruled out.
  • How to verify: re-run the job on the same suite; if green, flake confirmed. If it recurs, attach py-spy dump/coredump to the hung pid to locate the stall frame.
• [LOW] A commit in 52c6e27e..e339c83f altered rocm/aiter init or import-time behavior reachable from moss_vl import. Candidate (unverified) commits touching init/graph/nccl paths: efdf02a3 (--pre-warm-nccl), 3cbb7568 (full CUDA graph prefill), 9ddea8d9 (MORI-EP decode graph).
  • Supporting: temporal correlation only — they land in the regression window.
  • Disconfirming: none of these modify the pure-math embedding-interpolation path this test exercises; the test does not build a server, capture graphs, or use NCCL. No code-level link to the failing path.
  • How to verify: not actionable until the hang reproduces; if it does, bisect 52c6e27e..e339c83f.

No hypothesis above LOW confidence. The silent hang with green history strongly points to infra/flake, but a single completed data point cannot exclude a regression.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for test_vit_pos_embed_interpolate (0 results), moss_vl (only #30008 "Set REQUEST_TIMEOUT=30 for AMD to deflake multimodal tests" — unrelated to this pure-math test), and stage-a-test-1-gpu-small-amd (no matching hang fix). API: search/issues?q=repo:sgl-project/sglang+is:pr+is:open+<kw>.

Suggested Triage Steps

• Re-run job stage-a-test-1-gpu-small-amd on run 28867568883 (or wait for the two queued scheduled runs to complete) to confirm flake vs. persistent regression before deeper investigation.
• If it recurs: run test/registered/models/test_vit_pos_embed_interpolate.py alone in container rocm/sgl-dev:v0.5.14-rocm700-mi30x-20260707 on an MI325 and capture py-spy dump on the hung PID to pinpoint whether the stall is in moss_vl import, rocm platform init, or the first .to("cuda") op.
• If reproduced and isolated to init, bisect the 44-commit window 52c6e27e..e339c83f, prioritizing the init/graph/nccl candidates above.
• If confirmed as a runner-level GPU hang, treat as infra (drain/recycle the runner); no sglang code change indicated.

Status

• Persistence: 1 completed run affected (2026-07-07 15:27 UTC); previous 4 completed runs green. Not yet confirmed persistent (2 intervening runs still queued).
• Scope: single job stage-a-test-1-gpu-small-amd (linux-mi325-1gpu-sglang) in pr-test-amd.yml; one test file. Job failed via the 30-min "Run test" step cap after the first test consumed 1200s.
• Blocked work: blocks the stage-a 1-GPU small AMD signal for this scheduled run; downstream tests in the suite after the hang did not complete.
• In-flight fix: ❌ none found — needs a re-run to classify, then new investigation if persistent.

stage-b-test-1-gpu-large-amd (linux-mi325-1gpu-sglang, 1) — failed step(s): Run test

Commit Info

• sglang: e339c83f
• aiter: 9127c94a (v0.1.17.dev110+g9127c94a1; source: dev image preserved — [CI-AITER-CHECK] Dev/patched version detected → skipping rebuild)

Failed Tests

Test File	Test Function	Error	Log
test/registered/perf/test_bench_serving_1gpu_part2.py	test_score_api_latency_throughput	AssertionError: 50.64538265578449 not less than 50 (p95 latency)	Log

All other 6 files in this shard passed (Test Summary: 6/7 passed).

Failure Summary

The perf test test_score_api_latency_throughput asserts res["p95_latency_ms"] < 50 at line 95. On this MI325 run the score-API p95 latency came in at 50.34 ms (first try) and 50.65 ms (retry), both marginally over the 50 ms bound (~1.3% overshoot), so retry() exhausted and the file returned exit code 1.

Regression Status

Function-level: New today for the p95 < 50 assertion. File-level: Known recurring AMD perf-threshold flakiness — the same file tips over a different borderline assertion on most recent runs.

Recent history of test_bench_serving_1gpu_part2.py (test_score_api_latency_throughput) in job stage-b-test-1-gpu-large-amd (linux-mi325-1gpu-sglang, 1) (branch=main, event=schedule, completed runs only):

Date (UTC)	Run	Job	This-Function Status	What failed in the file	Error
Jul 7 12:52	28867568883	shard 1	❌ Failed	test_score_api_latency_throughput	50.645 not less than 50 (p95)
Jul 6 18:40	28814834996	shard 1	✅ Passed	test_embeddings_api_batch_scaling	100.03 not less than 90
Jul 6 07:38	28775611300	shard 1	✅ Passed	test_vlm_online_latency	263.42 not less than 150 (ttft)

(Runs 28833386129, 28848583109, 28890204713 are still queued → excluded per completed-runs-only rule. Note: test files are dynamically partitioned across shards, so "shard 1" runs a different file set each run — comparison above is at the test-file level, not by shard identity.)

Failure Cluster

AMD MI325 perf-threshold marginally exceeded (test config, NVIDIA-tuned bound)

Facts (verified observations)

• Verbatim log (job.log): AssertionError: 50.337416667025536 not less than 50 then on retry AssertionError: 50.64538265578449 not less than 50 → Exception: retry() exceed maximum number of retries. → FAILED (errors=1) → ✗ FAILED: .../test_bench_serving_1gpu_part2.py returned exit code 1.
• The failing assertion self.assertLess(res["p95_latency_ms"], 50) (line 95) and assertLess(res["avg_latency_ms"], 48) (line 94) are not wrapped in is_in_amd_ci() — the same 48/50 ms bounds apply to NVIDIA and AMD. Every sibling assertion in this file (test_vlm_offline_throughput, test_vlm_online_latency, test_score_api_batch_scaling, both embeddings tests) has an explicit # relax for mi300x AMD branch; test_score_api_latency_throughput does not.
• aiter is unchanged (dev image preserved, no AITER_COMMIT_OVERRIDE); NVIDIA is unaffected by these AMD-only bounds.
• Reproducibility: 1 job in this run; the file fails intermittently on adjacent runs but on different borderline sub-assertions (embeddings batch-scaling, VLM ttft), consistent with noisy latency near a too-tight bound rather than a single code regression.

Hypothesised Causes (with confidence)

• [HIGH] The p95_latency_ms < 50 / avg_latency_ms < 48 thresholds in test_score_api_latency_throughput are NVIDIA-tuned and marginally too tight for MI325; the measured p95 (50.34 / 50.65 ms) exceeds 50 by ~1.3%.
  • Supporting: assertion has no is_in_amd_ci() relaxation while all siblings do (code, line 93-96); overshoot is tiny and non-deterministic across retries; NVIDIA unaffected; open PR #26178 already adds an AMD branch to exactly this assertion (if is_in_amd_ci(): assertLess(avg_latency_ms, 55); assertLess(p95_...)).
  • Disconfirming: this specific assertion passed in the two prior completed runs I checked, so a genuine perf regression landing in e339c83f's window cannot be fully excluded without a p95 time-series (step-summary values are not in stdout).
  • How to verify: re-run test/registered/perf/test_bench_serving_1gpu_part2.py::test_score_api_latency_throughput on MI325 several times and record p95 distribution; if it clusters ~48-52 ms, the bound is the cause.
• [LOW] A latency regression from a commit in this run's window.
  • Supporting: function newly failing today at function-level.
  • Disconfirming: overshoot is sub-2% and varies between retries (50.34 → 50.65); no code path in the score/serving hot path identified; other shards green.
  • How to verify: bisect only if repeated MI325 reruns show p95 consistently ≫ 50.

In-flight Fix Check

• In-flight fix: ✅ #26178 "[AMD] WIP - relaxing thres for mi300" (open since 2026-05-23) — directly adds an is_in_amd_ci() branch to test_score_api_latency_throughput relaxing avg_latency_ms/p95_latency_ms. Marked WIP; chase to completion rather than opening a duplicate.
• In-flight fix: ⚠️ partial #29584 "ci(amd): fix three pre-existing MI325 CI failures blocking #29275" (open since 2026-06-28) — relaxes test_score_api_batch_scaling batch-50 bound (90→95) in the same file but does not touch the p95 < 50 latency assertion; would fix the Jul-6-style failures, not today's.
• Search: curl .../search/issues?q=repo:sgl-project/sglang+is:pr+is:open+p95_latency_ms+score and +test_bench_serving_1gpu_part2.

Suggested Triage Steps

• Confirm the bound hypothesis: re-run test_score_api_latency_throughput on an MI325 (linux-mi325-1gpu-sglang) runner ~5×, record the p95 distribution around the 50 ms bound.
• If p95 clusters near 48-52 ms, adopt/finish #26178's AMD relaxation for this assertion (align with the pattern already used by every sibling test in the file), and consider folding it into #29584 which already relaxes the batch-scaling sibling.
• If p95 is consistently ≫ 50, bisect the commit window ending at e339c83f for a score-API/serving latency regression before relaxing the bound.

Status

• Persistence: test_score_api_latency_throughput p95 assertion first observed failing this run (Jul 7 12:52 UTC); the parent file test_bench_serving_1gpu_part2.py fails intermittently on borderline AMD perf bounds across most recent scheduled runs (Jul 6 07:38, Jul 6 18:40, Jul 7 12:52 — different sub-assertion each time).
• Scope: 1 job this run (stage-b-test-1-gpu-large-amd (linux-mi325-1gpu-sglang, 1)), AMD MI325 only; NVIDIA unaffected (bounds shared but NVIDIA meets them).
• In-flight fix: ✅ #26178 (WIP, targets this exact assertion); ⚠️ #29584 fixes a sibling assertion only.

multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 3) — failed step(s): Run diffusion server tests (1-GPU)

Commit Info

• sglang: e339c83f
• aiter: v0.1.17.dev110+g9127c94a1 (source: dev image preserved — [CI-AITER-CHECK] Dev/patched version detected → skipping rebuild; Dockerfile default was 9127c94a)

Failed Tests

Test file: python/sglang/multimodal_gen/test/single_test_file/test_update_weights_from_disk.py — final result 5 failed, 1 passed, 4 errors. All in step 9 "Run diffusion server tests (1-GPU)" (job log).

Test File	Test Function	Error	Log
test_update_weights_from_disk.py	TestUpdateWeightsFromDisk::test_update_weights_from_disk_default[Qwen-Image]	assert 500 == 200 (line 432)	Log
test_update_weights_from_disk.py	TestUpdateWeightsFromDisk::test_update_weights_specific_modules[Qwen-Image]	AssertionError: Update failed: {'success': False, 'message': 'Inplace update to inference tensor outside InferenceMode is not allowed...'} → 500 (line 454)	Log
test_update_weights_from_disk.py	TestUpdateWeightsFromDisk::test_corrupted_weights_rollback[Qwen-Image]	assert (500 == 200) (line 535)	Log
test_update_weights_from_disk.py	TestUpdateWeightsFromDiskWithOffload::test_update_weights_with_offload_enabled[FLUX.2-klein-base-4B]	AssertionError: Expected 200, got 500 (line 638)	Log
test_update_weights_from_disk.py	TestUpdateWeightsFromDiskWithOffload::test_update_weights_with_offload_enabled[Qwen-Image]	AssertionError: Expected 200, got 500 (line 638)	Log
test_update_weights_from_disk.py	TestUpdateWeightsFromDisk::test_update_weights_from_disk_default[FLUX.2-klein-base-4B]	ERROR (fixture): AssertionError: No weights dir for vae/transformer in _compute_checksum_from_disk (test line 220)	Log
test_update_weights_from_disk.py	TestUpdateWeightsFromDisk::test_update_weights_specific_modules[FLUX.2-klein-base-4B]	ERROR (same fixture)	Log
test_update_weights_from_disk.py	TestUpdateWeightsFromDisk::test_update_weights_rejects_invalid_requests[FLUX.2-klein-base-4B]	ERROR (same fixture)	Log
test_update_weights_from_disk.py	TestUpdateWeightsFromDisk::test_corrupted_weights_rollback[FLUX.2-klein-base-4B]	ERROR (same fixture)	Log

Failure Summary

The diffusion update_weights_from_disk server endpoint returns HTTP 500, so every [Qwen-Image] test that asserts status_code == 200 fails. The 500 originates in the server's weight-loading path: an initial shape assertion at linear.py:411 triggers a rollback, and the rollback then raises RuntimeError: Inplace update to inference tensor outside InferenceMode is not allowed at linear.py:575. Separately, the four [FLUX.2-klein-base-4B] cases ERROR out during fixture setup because _compute_checksum_from_disk cannot find per-module weight subdirs. All failures are confined to test_update_weights_from_disk.py.

Regression Status

Known recurring failure — identical 5 failed / 1 passed / 4 errors pattern with the same InferenceMode error in every completed run checked.

Recent history of test_update_weights_from_disk.py (matrix shard 3) in job multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 3) (branch main, event schedule, completed runs only):

Date	Run	Job	Test File Status	Failed Functions	Error
Jul 7	28867568883	shard 3	❌ Failed	5 failed + 4 errors	500 / InferenceMode
Jul 6	28814834996	shard 3	❌ Failed	same 5+4	same
Jul 6	28794501292	shard 3	❌ Failed	same 5+4	same
Jul 6	28775611300	shard 3	❌ Failed	same 5+4	same
Jul 4	28689607720	shard 3	❌ Failed	same InferenceMode	same

First observed failure: at least 2026-07-04 (earliest verified; lookback window covered Jul 4–7, no green run for this test file found). No passing run of this test file exists in the queryable window. The test file was last refactored 2026-06-27 by #28762. Note the overall run 28867568883 is still queued at snapshot time, but this specific job has conclusion=failure (completed) — safe to analyze.

Failure Cluster

HTTP 500 in diffusion update_weights_from_disk (inference-tensor inplace copy + weight shape assertion)

Facts (verified observations)

• Endpoint returns 500: INFO: "POST /update_weights_from_disk HTTP/1.1" 500 Internal Server Error (server log within step 9).
• Stack trace 1 (verbatim): File ".../runtime/layers/linear.py", line 411, in weight_loader → assert param_data.shape == loaded_weight.shape → AssertionError.
• Stack trace 2 (verbatim): File ".../runtime/layers/linear.py", line 575, in weight_loader → param_data.copy_(loaded_weight) → RuntimeError: Inplace update to inference tensor outside InferenceMode is not allowed. Call chain: scheduler.py:229 _handle_update_weights_from_disk → scheduler_post_training_mixin.py:11 → gpu_worker_post_training_mixin.py:38 → weights_updater.py:286 update_weights_from_disk → _apply_weights:350 → _rollback:381 → _load_weights_into_module:139 → load_weights_into_model:200 → linear.py:533/575.
• Many benign Skipping weight update for FSDPQwen3ForCausalLM: parameter '...' not found after mapping warnings precede the crash (name-mapping mismatches, not the fatal error).
• FLUX ERROR cause (verbatim): AssertionError: No weights dir for transformer in /sgl-data/hf-cache/hub/models--black-forest-labs--FLUX.2-klein-base-4B/.../ raised in _compute_checksum_from_disk (test line 220), invoked from fixture diffusion_server_no_offload (test line 410).
• aiter version is identical (v0.1.17.dev110+g9127c94a1) across the failing runs; unchanged, so not a variable here.
• Reproducibility: identical failure set in ≥4 consecutive completed runs (Jul 4–7).

Hypothesised Causes (with confidence)

• [HIGH] The 500 is caused by the weight-update path doing an inplace param_data.copy_() (and shape-asserted assignment) on model parameters that are inference tensors, which PyTorch forbids outside InferenceMode.
  • Supporting: the error message itself names the mechanism; stack trace points to the exact line linear.py:575 param_data.copy_(loaded_weight); the multimodal_gen models are heavily wrapped in @torch.no_grad()/inference contexts (runtime/models/**), which is how params become inference tensors. The endpoint returns 500 and all Qwen-Image tests assert 200.
  • Disconfirming: the very first failure is actually a shape AssertionError at linear.py:411 (a different line) which triggers the rollback that then hits the InferenceMode error — so at least two distinct defects are stacked; fixing the InferenceMode copy alone may still leave the shape assertion for some modules.
  • How to verify: run test_update_weights_from_disk.py::TestUpdateWeightsFromDisk::test_update_weights_specific_modules[Qwen-Image] locally; wrap the copy in torch.inference_mode(False) / clone param before copy_ in weights_updater.py and re-run.
• [MEDIUM] The shape AssertionError at linear.py:411 reflects a param↔checkpoint layout mismatch in the diffusion weight-update mapping (the preceding not found after mapping warnings suggest the name/shape mapping for these modules is incomplete).
  • Supporting: assert param_data.shape == loaded_weight.shape fires before the rollback path; many Skipping weight update ... not found after mapping lines immediately precede it.
  • Disconfirming: could be a downstream symptom of an already-partially-loaded module rather than an independent bug.
  • How to verify: log param_data.shape vs loaded_weight.shape for the first failing param.
• [FACT] The four [FLUX.2-klein-base-4B] cases are test-fixture errors (No weights dir for transformer/vae), independent of the server 500 — the checksum helper expects per-module subdirs that don't exist for the FLUX snapshot layout.

Not aiter-related: this is pr-test-amd.yml (not amd-aiter-scout.yml), aiter version is unchanged across runs, and the failure is a pure torch-semantics / mapping error in sglang multimodal_gen code. No Failure Origin / Baseline A/B section applies.

In-flight Fix Check

• In-flight fix: ❌ none found — search q=repo:sgl-project/sglang is:pr is:open InferenceMode → 0 results; ...is:open update_weights diffusion → only #19152 ("Support offload and wake up of SGLang Diffusion"), a feature PR, not a fix for this crash. Needs a new PR or investigation.

Suggested Triage Steps

• Reproduce locally: run test_update_weights_from_disk.py::TestUpdateWeightsFromDisk::test_update_weights_specific_modules[Qwen-Image] against a 1-GPU diffusion server; confirm POST /update_weights_from_disk → 500.
• In runtime/layers/linear.py weight_loader, guard the inplace assignment for inference tensors (e.g. param.data = param.data.clone() before copy_, or perform the update under a mutable context) — then re-run to see if the shape AssertionError at line 411 still surfaces for specific modules.
• Investigate the ... not found after mapping warnings + shape assertion for the diffusion weight-name mapping in weights_updater.py (load_weights_into_model); these may be a second, independent defect.
• Separately fix the FLUX fixture: _compute_checksum_from_disk (test line 220) assumes per-module weight subdirs that don't exist in the FLUX.2-klein snapshot layout.

Status

• Persistence: Failing every completed schedule run of this job since at least 2026-07-04 (verified Jul 4/6/6/6/7); no green run for this test file in the lookback window.
• Scope: multimodal-gen-test-1-gpu-amd shard 3 (standalone test_update_weights_from_disk.py); 5 failed + 4 errored tests, 1 passed.
• Blocked work: Diffusion update_weights_from_disk (post-training weight reload) CI signal on AMD 1-GPU.
• In-flight fix: ❌ none found in open PRs.

stage-b-test-1-gpu-small-amd (linux-mi325-1gpu-sglang, 4) — failed step(s): Run test

Commit Info

• sglang: e339c83f
• aiter: 9127c94a (v0.1.17.dev110+g9127c94a1, source: dev image preserved — AITER_COMMIT_OVERRIDE empty, matches Dockerfile default)
• CI docker image: rocm/sgl-dev:v0.5.14-rocm700-mi30x-20260707

Failed Tests

Test File	Test Function	Error	Log
test/registered/mla/test_mla_fp8.py	TestMLA.setUpClass	Exception: Server process exited with code -9 (server crashed with KeyError: 'sglang')	Log

The other 8 test files in this shard passed (8/9 passed).

Failure Summary

test_mla_fp8.py (model neuralmagic/DeepSeek-Coder-V2-Lite-Instruct-FP8) failed in setUpClass because the server never came up: the scheduler crashed during model load. The native SGLang DeepseekV2ForCausalLM was unavailable (its import failed with No module named 'cutlass'), so the loader fell back to the Transformers backend, whose remote modeling_deepseek.py raised KeyError: 'sglang' when indexing ATTENTION_CLASSES[config._attn_implementation]. The server then exited with code -9 (SIGQUIT).

Regression Status

New regression — first observed in this run (2026-07-07). Last known passing: 2026-07-06 (run 28814834996, sha 52c6e27e).

Recent history of test/registered/mla/test_mla_fp8.py (TestMLA.setUpClass) in job stage-b-test-1-gpu-small-amd (4) (branch main, event schedule, completed runs only):

Date	Run	Test File Status	Failed Function	Error
Jul 7 12:52	28867568883	❌ Failed	TestMLA.setUpClass	KeyError: 'sglang' → server exit -9
Jul 6 18:40	28814834996	✅ Passed	—	— (loaded native DeepseekV2ForCausalLM)
Jul 6 07:38	28775611300	✅ Passed	—	—
Jul 6 00:44	28760739527	✅ Passed	—	—

(Runs 28848583109 (Jul 7 07:15) and 28833386129 (Jul 7 00:44) are still queued → excluded [IN-FLIGHT]. Run 28794501292 job (4) failed on a different test file — test_mla_fp8.py was sharded elsewhere there — so it is not comparable at the test-file level.)

Failure Cluster

Missing 'cutlass' module → model import fallback to Transformers → KeyError: 'sglang'

Facts (verified observations, no interpretation)

• Scheduler traceback (verbatim, log ~line 4425–4464):

  File ".../transformers.py", line 617, in __init__
    self.model: PreTrainedModel = AutoModel.from_config(
  File ".../modeling_deepseek.py", line 1203, in __init__
    self.self_attn = ATTENTION_CLASSES[config._attn_implementation](
  KeyError: 'sglang'
  

• Preceding log lines: Ignore import error when loading sglang.srt.models.deepseek_v2: No module named 'cutlass', then DeepseekV2ForCausalLM has no SGLang implementation, falling back to Transformers implementation, then Using Transformers backend.
• setUpClass error (log ~line 4479): Exception: Server process exited with code -9.
• No module named 'cutlass' skip counts (verified via grep -c):
  • This run (image 20260707): 120 skips → native deepseek_v2 unavailable.
  • Passing run 28814834996 (image 20260706): 0 skips → Load weight end. type=DeepseekV2ForCausalLM, quant=fp8 (native impl used).
  • Older passing runs (images 20260705/20260706): 0 skips each.
• Image is the cleanly-correlating variable: every run on an image ≤ 20260706 had 0 cutlass skips and passed; the only run on 20260707 has 120 skips and failed.
• The only cutlass/cutedsl-related sglang commit in the window, e85ef548 (#30117), uses lazy/gated imports (if backend.is_cutedsl(): from sglang.jit_kernel.cutedsl_bf16_gemm import ...), so it does not add a module-level import cutlass.
• import cutlass in deepseek_v2's reachable modules (nvfp4_gemm_swiglu_nvfp4_quant.py, cutedsl) is all lazy/function-local in the current tree; these files predate the regression window.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [MEDIUM] The daily ROCm CI image rocm/sgl-dev:v0.5.14-rocm700-mi30x-20260707 dropped (or broke) the cutlass python package that ...-20260706 provided. Without it, 120 model imports — including deepseek_v2 — fail at import time, forcing the broken Transformers fallback.
  • Supporting: cutlass skips jump 0 → 120 exactly at the 20260706 → 20260707 image boundary; all prior completed runs on older images passed; no sglang commit in the window adds an unconditional cutlass import.
  • Disconfirming: I did not directly inspect the two images' pip list to confirm the package delta; the sglang commit SHA also changed between pass and fail (though no commit touches an unconditional cutlass import).
  • How to verify: docker run --rm rocm/sgl-dev:v0.5.14-rocm700-mi30x-20260706 python -c "import cutlass" vs the ...-20260707 tag; compare pip show nvidia-cutlass-dsl.
• [FACT / latent bug] Even when the native impl is unavailable, the Transformers fallback for DeepseekV2ForCausalLM is broken on this model: config._attn_implementation is set to 'sglang', which the remote modeling_deepseek.py ATTENTION_CLASSES dict does not contain → KeyError: 'sglang'. This converts a soft "missing dep" into a hard server crash.
  • Supporting: the verbatim traceback above.
  • Disconfirming: (none — directly observed) but note this only triggers once the native impl is unavailable, so it is a secondary failure gated by the cutlass import.
  • How to verify: with the native impl force-disabled, launch the FP8 model on any backend and confirm the same KeyError.
• [LOW] An sglang code change in 52c6e27e..e339c83f (44 commits) newly forces a hard cutlass import in a shared model path.
  • Supporting: sglang HEAD SHA changed across the pass/fail boundary.
  • Disconfirming: no commit in the window adds a module-level import cutlass; the cutedsl commit's imports are all lazy/gated; cutlass-importing files predate the window. This does not explain 120 simultaneous skips.
  • How to verify: run the failing image with git checkout 52c6e27e — if it still shows 120 cutlass skips, the code is exonerated (image is the cause).

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for No module named cutlass (2 hits, both unrelated: #15755, #22455), attn_implementation sglang (0 hits), and cutlass ROCm (no relevant match). Neither the image regression nor the Transformers-fallback KeyError appears to have an open fix.

Suggested Triage Steps

• Confirm the image regression: diff python -c "import cutlass" / pip show nvidia-cutlass-dsl between rocm/sgl-dev:...-20260706 and ...-20260707. If cutlass is gone from 20260707, the fix belongs in the image build (restore the package) or the model-import guards.
• A/B the sglang code out: run the 20260707 image at sha 52c6e27e; if cutlass skips remain 120, code is exonerated and this is purely an image regression.
• Harden the fallback path (separately from the image fix): when native impl is unavailable, either (a) do not set _attn_implementation='sglang' for models whose remote code lacks that key, or (b) fail fast with a clear message instead of a bare KeyError. This prevents the missing-dep soft failure from becoming a server crash.
• If the image cannot be fixed quickly, consider making the cutlass-dependent import path in deepseek_v2 fully optional on ROCm so the native impl still registers.

Status

• Persistence: First failing completed run 2026-07-07 12:52 UTC; 1 completed failing run so far (two later runs still in-flight).
• Scope: test_mla_fp8.py in this shard; but the underlying No module named 'cutlass' affects 120 model imports on image 20260707, so other AMD tests that load cutlass-gated native models on the new image are likely impacted too.
• Blocked work: MLA FP8 (DeepSeek-Coder-V2-Lite FP8) accuracy signal on AMD.
• In-flight fix: ❌ none found — needs new PR or investigation (image rebuild and/or fallback hardening).

stage-b-test-1-gpu-small-amd (linux-mi325-1gpu-sglang, 5) — failed step(s): Run test

Commit Info

• sglang: e339c83f
• aiter: 9127c94a (v0.1.17.dev110+g9127c94a1; source: dev image preserved — [CI-AITER-CHECK] Dev/patched version detected → skipping rebuild). Not relevant to this failure (pure Python import error).

Failed Tests

Test File	Test Function	Error	Log
test/registered/unit/managers/test_mm_process_config.py	TestOverrideProcessorsConfigInjection.test_ernie45_vl_injects_images_kwargs	ModuleNotFoundError: No module named 'cutlass'	Log

Result: Ran 11 tests in 7.512s / FAILED (errors=1) → file returned exit code 1 → job Process completed with exit code 255. The other 10 tests in the file passed; 9/10 files in the shard passed.

Failure Summary

The unit test test_ernie45_vl_injects_images_kwargs performs a deferred import of sglang.srt.multimodal.processors.ernie45_vl, which transitively imports deepseek_v2 → dsa_indexer → sglang.jit_kernel.dsa → cutedsl_paged_mqa_logits, whose module top-level executes import cutlass. The cutlass (NVIDIA CuTe DSL) package is not installed in the AMD ROCm CI image, so the import chain raises ModuleNotFoundError. This is a Python import-time failure, not a GPU/runtime failure.

Regression Status

New regression — first observed 2026-07-07 (run 28848583109, head 99db3b0f), the first scheduled pr-test-amd run after PR #25220 merged.

Recent history of test/registered/unit/managers/test_mm_process_config.py in job stage-b-test-1-gpu-small-amd (…, 5) (branch main, event schedule, completed runs only):

Date	Run	Job	Test File Status	Failed Function	Error
Jul 07 12:52 UTC	28867568883	shard 5	❌ Failed	test_ernie45_vl_injects_images_kwargs	ModuleNotFoundError: cutlass
Jul 07 07:15 UTC	28848583109	shard 5	❌ Failed	test_ernie45_vl_injects_images_kwargs	ModuleNotFoundError: cutlass (342 occurrences run-wide)
Jul 06 18:40 UTC	28814834996	shard 5	✅ Passed	—	—

First observed failure: 2026-07-07 07:15 UTC. Last known passing: 2026-07-06 18:40 UTC (head 52c6e27e). Regression window 52c6e27e..99db3b0f.

Failure Cluster

ModuleNotFoundError: cutlass on AMD (unguarded import in CuteDSL DSA path)

Facts (verified observations)

• Verbatim traceback (log lines 32706–32724): import chain test_mm_process_config.py:236 → ernie45_vl.py:15 → ernie45_vl.py:42 → ernie45_moe_vl.py:44 → deepseek_v2.py:61 → dsa_indexer.py:11 → jit_kernel/dsa/__init__.py:1 → cutedsl_paged_mqa_logits.py:15 → import cutlass → ModuleNotFoundError: No module named 'cutlass'.
• python/sglang/jit_kernel/dsa/cutedsl_paged_mqa_logits.py:14-17 has unguarded top-level import cutlass, import cutlass.cute as cute, from cutlass.utils import HardwareInfo (verified by reading the file).
• python/sglang/jit_kernel/dsa/__init__.py:1 unconditionally re-exports from cutedsl_paged_mqa_logits, so any import of dsa_indexer on AMD triggers the failure.
• Test file passed in the last pre-merge run and failed in the first post-merge run (verified from both logs: pre-merge {"passed": true}, 0 cutlass errors; post-merge {"passed": false}, 342 cutlass errors).
• Commit fefc1743 (PR #25220, merged 2026-07-06 23:34 PDT / 2026-07-07 06:34 UTC) is the sole commit that created all three files/edits in the import chain (git log shows it is the only/first author of cutedsl_paged_mqa_logits.py, dsa/__init__.py, and the dsa_indexer.py import block). It is an ancestor of 99db3b0f (failing) but NOT of 52c6e27e (passing).

Hypothesised Causes (with confidence, with disconfirming evidence)

• [HIGH] PR #25220 (fefc1743) added an unguarded top-level import cutlass in cutedsl_paged_mqa_logits.py, imported eagerly via jit_kernel/dsa/__init__.py from dsa_indexer.py. On AMD ROCm, cutlass is absent, breaking any import of deepseek_v2/ernie45_vl.
  • Supporting: three independent evidence chains converge — (a) code-level: the exact failing import line is added by this commit; (b) temporal: test passes at 52c6e27e (pre-merge), fails at 99db3b0f (first post-merge run); (c) the stack trace names precisely the files this commit introduced.
  • Disconfirming: (none found). The dev-image aiter is unchanged across pass/fail runs, ruling aiter out.
  • How to verify: on an AMD ROCm image, run python3 -c "import sglang.srt.layers.attention.dsa.dsa_indexer" — expect ModuleNotFoundError: cutlass; then confirm a lazy/guarded import removes it.

In-flight Fix Check

• In-flight fix: ❌ none found — Searched open PRs: cutlass+dsa (5 results, none address the import guard), cutedsl in:title (11 results), ModuleNotFoundError+cutlass (1 result). The closest, PR #30413 "Fix additional kernels being triggered in CuteDSL MQA logits path" (open, a #25220 follow-up), modifies paged_mqa_logits.py, dsa_indexer.py, dsa_backend.py — it does not guard the import cutlass in cutedsl_paged_mqa_logits.py, so it does not fix this AMD breakage.

Suggested Triage Steps

• Guard the CuteDSL imports: make import cutlass / the cutedsl_paged_mqa_logits re-export in python/sglang/jit_kernel/dsa/__init__.py lazy or wrapped in try/except ImportError (or gate behind an SM100/CUDA-only check), so AMD/ROCm imports of deepseek_v2 don't require cutlass.
• Verify with python3 -c "import sglang.srt.models.deepseek_v2" inside the ROCm CI container rocm/sgl-dev:v0.5.14-rocm700-mi30x-20260707 before/after the guard.
• Re-run shard 5 of pr-test-amd.yml (and shards 4, 6, which failed identically in run 28848583109) to confirm resolution across the whole cutlass cluster.

Status (factual, no priority assignment)

• Persistence: Failing since 2026-07-07 07:15 UTC (first post-#25220 scheduled run); every completed scheduled pr-test-amd run since has reproduced it.
• Scope: Broad on AMD — 342 cutlass import errors run-wide in run 28848583109; at least shards 4, 5, 6 of stage-b-test-1-gpu-small-amd affected. Any AMD test importing deepseek_v2/ernie45_vl/dsa_indexer is at risk.
• Blocked work: test_mm_process_config.py and other DSA/DeepSeek-importing unit tests on AMD ROCm.
• In-flight fix: ❌ none found — needs a new PR to guard the import cutlass (PR #30413 does not cover it).

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-07-08 18:54 UTC)

[amd-bot]

pr-test-amd.yml — Per-Job Analysis (continued 3/9)

stage-b-test-1-gpu-small-amd (linux-mi325-1gpu-sglang, 6) — failed step(s): Run test

Commit Info

• sglang: e339c83f
• aiter: 9127c94a (v0.1.17.dev110+g9127c94a1, source: dev image preserved — matches Dockerfile default)

Failed Tests

Test File	Test Function	Error	Log
test/registered/hicache/test_hicache_variants.py	TestHiCacheMLA.setUpClass	Exception: Server process exited with code -9 (underlying KeyError: 'sglang' in scheduler)	Log

1 of 9 test files failed in this shard; the other 8 passed (test_openai_server, test_openai_embedding, test_pooled_hidden_states, test_priority_metrics, test_compressed_tensors_models, test_fp32_lm_head, test_swa_unittest, test_llava, test_common).

Failure Summary

TestHiCacheMLA.setUpClass failed to launch an sglang server for deepseek-ai/DeepSeek-Coder-V2-Lite-Instruct (--enable-hierarchical-cache). The scheduler crashed during model load: DeepseekV2ForCausalLM was not found in the SGLang model registry, so sglang fell back to the Transformers backend, which then hit KeyError: 'sglang' inside the model's trust_remote_code modeling_deepseek.py. The server exited with code -9 and setUpClass raised.

Regression Status

New regression.

Recent history of test/registered/hicache/test_hicache_variants.py (TestHiCacheMLA.setUpClass) in stage-b-test-1-gpu-small-amd (branch main, event schedule, completed runs only; the file is sharded so the shard index varies per run):

Date (UTC)	Run	Job (shard)	Test File Status	Failed Function	Error
Jul 7 12:52	28867568883	shard 6	❌ Failed	TestHiCacheMLA.setUpClass	KeyError: 'sglang' → server exit -9
Jul 7 07:15	28848583109	shard 6	❌ Failed	TestHiCacheMLA.setUpClass	KeyError: 'sglang' → server exit -9
Jul 7 00:44	28833386129	shard 6	✅ Passed	—	—
Jul 6 18:40	28814834996	shard 6	✅ Passed (649s)	—	—
Jul 6 07:38	28775611300	(all small shards ✅)	✅ Passed	—	—

• First observed failure: run 28848583109 (Jul 7 07:15 UTC, head 99db3b0f).
• Last known passing: run 28833386129 (Jul 7 00:44 UTC, head 6c1fb8a9).
• Regression window: 6c1fb8a9..99db3b0f (26 commits).

Failure Cluster

ROCm model-registry import failure (No module named 'cutlass') → DeepseekV2 Transformers fallback → KeyError: 'sglang'

Facts (verified observations)

• Log (this run) shows: Ignore import error when loading sglang.srt.models.deepseek_v2: No module named 'cutlass' (also deepseek_v4, deepseek_v4_nextn, deepseek_nextn, deepseek_ocr, bailing_moe_linear, bailing_moe_nextn). [FACT]
• Consequently: DeepseekV2ForCausalLM has no SGLang implementation, falling back to Transformers implementation. [FACT]
• Scheduler traceback ends at modeling_deepseek.py:1203 → self.self_attn = ATTENTION_CLASSES[config._attn_implementation] → KeyError: 'sglang'; then Received sigquit from a child process, server exits -9. [FACT]
• Registry mechanism: import_model_classes (python/sglang/srt/models/registry.py:104-110) catches any import exception, logs "Ignore import error…", and drops that arch from get_supported_archs(). get_model_architecture (model_loader/utils.py:216-222) then sees is_native_supported=False and routes to the Transformers backend. [FACT]
• Transformers backend sets self.text_config._attn_implementation = "sglang" (python/sglang/srt/models/transformers.py:614) and registers it only in HF's ALL_ATTENTION_FUNCTIONS; the DeepSeek trust_remote_code module uses its own ATTENTION_CLASSES dict (no 'sglang' key). [FACT]
• Import chain to cutlass: deepseek_v2.py:61 from ...dsa.dsa_indexer import Indexer → dsa_indexer.py:11 from sglang.jit_kernel.dsa import (… cutedsl_paged_mqa_logits …) → jit_kernel/dsa/__init__.py:1 → cutedsl_paged_mqa_logits.py:15 unguarded import cutlass. cutlass (NVIDIA CUTLASS Python DSL) is not installed in the ROCm CI image. [FACT]

Hypothesised Causes

• [HIGH] PR #25220 (fefc1743, "Cute-DSL FP8 MQA logits") introduced the unguarded module-level import cutlass reachable from deepseek_v2's import chain, breaking model registration on ROCm.
  • Supporting: the commit's diff adds from sglang.jit_kernel.dsa import (… cutedsl_paged_mqa_logits …) at module scope in dsa_indexer.py and adds jit_kernel/dsa/cutedsl_paged_mqa_logits.py whose line 15 is import cutlass; deepseek_v2.py:61 imports dsa_indexer. The commit lands inside the regression window 6c1fb8a9..99db3b0f, and the log error names exactly No module named 'cutlass' while loading deepseek_v2. Three evidence chains converge (code + log + timing).
  • Disconfirming: I did not A/B-revert the commit in CI, and I did not directly diff the precise merge SHA of #25220 against the last-pass boundary run-by-run (window has 26 commits; other cutlass-importing files exist — nvfp4_gemm_swiglu_nvfp4_quant.py, linear/kernels/*_blackwell/* — but those were already present before the window, and deepseek_v2 imported cleanly at 6c1fb8a9, so they are not the trigger).
  • How to verify: on an MI325 ROCm container, python -c "import sglang.srt.models.deepseek_v2" and confirm ModuleNotFoundError: No module named 'cutlass'; then guard the import cutlass in jit_kernel/dsa/cutedsl_paged_mqa_logits.py (or the dsa_indexer.py:11 import) behind an SM100/CUDA check and re-run test_hicache_variants.py.
• [LOW] CI image lost the cutlass package between runs.
  • Supporting: the symptom is a missing module.
  • Disconfirming: same dev image spans the pass→fail boundary (aiter v0.1.17.dev110+g9127c94a1 unchanged); the direct code change adding the unguarded import fully explains the ROCm-only breakage (NVIDIA installs the CUTLASS DSL, so its deepseek_v2 imports fine).
  • How to verify: check pip show nvidia-cutlass-dsl in the ROCm image at both SHAs.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs with cutlass import dsa, No module named cutlass, dsa_indexer cutlass, cutedsl rocm import; no open PR guards this import for ROCm. (Query: https://api.github.com/search/issues?q=repo:sgl-project/sglang+is:pr+is:open+<keywords>.)

Suggested Triage Steps

• Reproduce on an MI325 ROCm container: python -c "import sglang.srt.models.deepseek_v2" — expect No module named 'cutlass'.
• Bisect / confirm within 6c1fb8a9..99db3b0f; focus on #25220 (fefc1743).
• If confirmed, make the CuteDSL path lazy/guarded: gate import cutlass in python/sglang/jit_kernel/dsa/cutedsl_paged_mqa_logits.py:15 (and the jit_kernel/dsa/__init__.py re-export) behind is_sm100_supported() / non-HIP, or defer the dsa_indexer.py:11 import into the is_cutedsl() branch, so deepseek_v2 registers on ROCm.
• Note the blast radius extends beyond this test: deepseek_v4, deepseek_v4_nextn, deepseek_nextn, deepseek_ocr, bailing_moe_linear, bailing_moe_nextn are all dropped from the ROCm registry by the same import error — verify their AMD coverage after the fix.

Status

• Persistence: failing since Jul 7 07:15 UTC (run 28848583109); both completed schedule runs since have this test red.
• Scope: stage-b-test-1-gpu-small-amd on pr-test-amd.yml (ROCm/MI325). Likely also affects pr-test-amd-rocm720 and nightly AMD DeepSeek/Bailing-MoE tests via the same registry drop (not verified here). NVIDIA unaffected (CUTLASS DSL installed).
• Blocked work: HiCache-MLA CI signal on AMD, plus native DeepSeek-V2/V4 and Bailing-MoE model coverage on ROCm.
• In-flight fix: ❌ none found — needs a new PR to guard the CuteDSL import cutlass on ROCm.

stage-c-test-large-8-gpu-amd (linux-mi325-8gpu-sglang, 1) — failed step(s): Run test

Commit Info

• sglang: e339c83f
• aiter: 9127c94a (v0.1.17.dev110+g9127c94a1, source: Dev image preserved — [CI-AITER-CHECK] reports "Dev/patched version detected → skipping rebuild"; matches Dockerfile default)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/test_deepseek_v32_basic.py	TestDeepseekV32TP.setUpClass	Exception: Server process exited with code -9 — root cause ValueError: Unrecognized configuration class _DeepseekV32ConfigAlias for this kind of AutoModel: AutoModel.	Log

Failure Summary

The deepseek-ai/DeepSeek-V3.2 server (--tp 8, --trust-remote-code) failed to launch during TestDeepseekV32TP.setUpClass. Model loading routed to the transformers fallback backend (python/sglang/srt/models/transformers.py:617 → AutoModel.from_config) instead of sglang's native DeepseekV32ForCausalLM. AutoModel does not recognize sglang's registered _DeepseekV32ConfigAlias config class, raising a ValueError; the child worker then died (sigquit) and the parent was killed with exit code -9. Total elapsed 95s.

Regression Status

New symptom this run for test_deepseek_v32_basic.py. The test file has been intermittently red over the past days, but with a completely different, post-launch symptom (server launched OK, then crashed during inference). This is the first observed server-launch config-resolution failure.

Recent history of test/registered/amd/test_deepseek_v32_basic.py in job stage-c-test-large-8-gpu-amd (branch main, event schedule, completed runs only; the shard index varies because the auto-partitioner distributes files):

Date	Run	Test File Status	Symptom	Elapsed	Image
Jul 7 12:52	28867568883	❌ Failed	NEW: _DeepseekV32ConfigAlias ValueError, server never launched, exit -9	95s	20260707
Jul 6 18:40	28814834996	❌ Failed	server launched, ran 3 tests, KeyError: 'answer' (gsm8k) during inference	1679s	20260707
Jul 6 13:18	28794501292	✅ Passed	—	—	20260706
Jul 6 07:38	28775611300	❌ Failed	server launched, ran 3 tests, c10 crash + KeyError: 'answer'	2108s	20260706
Jul 6 00:44	28760739527	✅ Passed	—	1238s	20260706
Jul 5 18:25	28750542672	✅ Passed	—	1174s	20260705

First observed for the config-ValueError symptom: this run (2026-07-07 12:52 UTC). In every prior completed run the server launched successfully (native DeepseekV32ForCausalLM resolved), so this is a fresh model-resolution regression, not the recurring inference-time failure. (Runs 28833386129, 28848583109, 28890204713 are still queued/in-flight and excluded.)

Failure Cluster

DeepSeek-V3.2 misrouted to transformers fallback → AutoModel config ValueError at server launch

Facts (verified observations)

• Launch command (verbatim): sglang serve --model-path deepseek-ai/DeepSeek-V3.2 --trust-remote-code --tp 8 --model-loader-extra-config '{"enable_multithread_load": true}' --dsa-prefill-backend tilelang --dsa-decode-backend tilelang --device cuda ... (log line ~8184).
• Error (verbatim, log lines 8146 & 8180): ValueError: Unrecognized configuration class <class 'sglang.srt.utils.hf_transformers.common._DeepseekV32ConfigAlias'> for this kind of AutoModel: AutoModel. — HF's own DeepseekV32Config is in the recognized list, but sglang's alias subclass is not.
• Stack trace goes through model_runner.py:1512 load_model → loader.py:761/320 _initialize_model → transformers.py:1143/1102/617 __init__ → AutoModel.from_config (log lines 8150–8180). This is the transformers-backend class, confirming architecture resolution did NOT match a native arch (python/sglang/srt/model_loader/utils.py:221 not is_native_supported → resolve_transformers_arch).
• sglang does natively register DeepseekV32ForCausalLM (python/sglang/srt/models/deepseek_v2.py:2955,2980; python/sglang/srt/configs/model_config.py:108), so under normal resolution the fallback path should never be taken for this model.
• _DeepseekV32ConfigAlias is a subclass of HF DeepseekV3Config with model_type="deepseek_v32", registered into AutoConfig (python/sglang/srt/utils/hf_transformers/common.py:126-132).
• HF_HUB_OFFLINE=0 (log line 8185) — the model config is pulled fresh from the HuggingFace hub at runtime.
• Docker image is identical between the last successful-launch run and this run: both rocm/sgl-dev:v0.5.14-rocm700-mi30x-20260707. The same image launched the DeepSeek-V3.2 server successfully in run 28814834996 ~18h earlier.

Hypothesised Causes (with confidence + disconfirming evidence)

• [LOW] A commit in the window 52c6e27e..e339c83f (44 sglang commits) altered architecture resolution so deepseek_v32 no longer matches a native arch and falls to the transformers backend.
  • Supporting: the symptom is server-launch model-resolution; it appeared exactly at this run; the last launch-OK run predates these 44 commits.
  • Disconfirming: I inspected the window and no commit obviously touches model-arch resolution, the transformers fallback, or the deepseek config registry. The two nearest candidates are not on the path — c8618967 (#30297) only edits server_args.py/arg_groups; 9a6f8e59 (#30333) is DeepSeek V4 MTP, not V3.2.
  • How to verify: git bisect across 52c6e27e..e339c83f running test_deepseek_v32_basic.py on an 8×MI325 container, checking whether the server reaches native model load.
• [LOW] Hub-side change to deepseek-ai/DeepSeek-V3.2 config.json (e.g. architectures field) fetched fresh (HF_HUB_OFFLINE=0), so hf_config.architectures no longer contains a native arch and resolution falls to transformers.
  • Supporting: identical docker image launched fine 18h earlier ⇒ the differentiator is NOT the installed transformers/aiter; a fresh-pulled remote config is the remaining variable that can flip resolution without any sglang/image change.
  • Disconfirming: not directly observed — I cannot inspect the runtime-fetched config from the log. The error listing shows HF transformers itself now knows DeepseekV32Config, which is consistent with a recent hub/transformers-side model definition landing.
  • How to verify: diff the current hub deepseek-ai/DeepSeek-V3.2/config.json architectures/model_type against a cached copy; re-run with a pinned config revision.
• [LOW] Nondeterministic config resolution / AutoConfig.register("deepseek_v32", _DeepseekV32ConfigAlias) conflicting with a now-native transformers DeepseekV32Config, giving inconsistent results across processes.
  • Supporting: same image launched once and failed the next run; registration-order/override conflicts can be nondeterministic.
  • Disconfirming: no direct evidence of nondeterminism in the log; only two data points on the 20260707 image.

No hypothesis reaches MEDIUM: git bisect across 52c6e27e..e339c83f plus a hub-config diff is required to promote any of the above.

In-flight Fix Check

• In-flight fix: ❌ none directly addressing this. Search: DeepseekV32ConfigAlias (0 open PRs), Unrecognized configuration AutoModel → only #28328 "[Fix] Improve Transformers backend compatibility for trust_remote_code models" (open, stale — last updated 2026-06-15). #28328 hardens the transformers fallback path but would not stop DeepSeek-V3.2 from wrongly entering it, so it is not a fix for this root cause.

Suggested Triage Steps

• Reproduce in an 8×MI325 container on image rocm/sgl-dev:v0.5.14-rocm700-mi30x-20260707: sglang serve --model-path deepseek-ai/DeepSeek-V3.2 --trust-remote-code --tp 8 ... and log model_config.hf_config.architectures plus ModelRegistry.get_supported_archs() at python/sglang/srt/model_loader/utils.py:216 to see why is_native_supported is False.
• Diff the runtime-fetched deepseek-ai/DeepSeek-V3.2 config.json (architectures, model_type) against a known-good copy; if the hub changed, pin --revision in the test to isolate hub-vs-code.
• If the config is unchanged, git bisect across 52c6e27e..e339c83f targeting test_deepseek_v32_basic.py server launch.
• Separately track the pre-existing inference-time failures (KeyError: 'answer' / c10 crash at ~1700–2100s) on the same test file — those are a distinct cluster from this launch failure.

Status

• Persistence: First occurrence of this server-launch config-resolution symptom (2026-07-07 12:52 UTC, this run). Distinct from the recurring post-launch inference failures on the same file.
• Scope: 1 test file (test_deepseek_v32_basic.py), 1 job (stage-c-test-large-8-gpu-amd, shard 1) in pr-test-amd.yml. Blocks DeepSeek-V3.2 8-GPU DSA/TP CI signal on AMD.
• In-flight fix: ❌ none directly addressing this; #28328 (stale, unrelated fix scope) does not resolve it.

stage-c-test-large-8-gpu-amd-mi35x (linux-mi35x-gpu-8, 1) — failed step(s): Run test

Commit Info

• sglang: e339c83f
• aiter: 9127c94a — v0.1.17.dev110+g9127c94a1 (source: dev image detected, rebuild skipped). Identical aiter in the last passing run.
• CI image: rocm/sgl-dev:v0.5.14-rocm700-mi35x-20260706 (this run) vs ...-20260705 in the last passing run.

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/test_deepseek_r1_mxfp4_8gpu.py	TestDeepseekR1MXFP4.setUpClass	KeyError: 'sglang' in remote-code modeling_deepseek.py → server exit -9	Log
test/registered/amd/test_kimi_k25_mxfp4.py	TestKimiK25MXFP4.setUpClass	ValueError: KimiK25ForConditionalGeneration has no SGlang implementation and the Transformers implementation is not compatible with SGLang → server exit 1	Log

test/registered/amd/test_qwen3_coder_next_8gpu.py passed (366s) in the same job.

Failure Summary

Two model-launch tests in partition 1 failed at setUpClass (server never came up). For amd/DeepSeek-R1-MXFP4-Preview, sglang reported DeepseekV3ForCausalLM has no SGLang implementation, falling back to Transformers implementation, then the remote-code model crashed with KeyError: 'sglang' when indexing its own ATTENTION_CLASSES[config._attn_implementation]. For amd/Kimi-K2.5-MXFP4, architecture resolution raised ValueError because the Transformers-backend fallback is now judged incompatible with SGLang. Both are in the sglang↔Transformers architecture-resolution / attention-integration path.

Regression Status

New regression. Both test files passed one run earlier and now fail.

Recent history of test_deepseek_r1_mxfp4_8gpu.py + test_kimi_k25_mxfp4.py in job stage-c-test-large-8-gpu-amd-mi35x (linux-mi35x-gpu-8, 1) (branch=main, event=schedule, completed runs only):

Date (UTC)	Run	Job	Test File Status	Failed Function	Error
Jul 7 12:52	28867568883	job	❌ Both Failed	setUpClass (both)	KeyError: 'sglang' / ValueError
Jul 6 18:40	28814834996	job	⚠️ N/A	—	infra: VRAM cleanup unsuccessful pre-check, tests never ran (not comparable)
Jul 6 13:18	28794501292	job	✅ Both Passed	—	deepseek 989s, kimi 578s
Jul 6 07:38	28775611300	job	✅ Passed	—	—
Jul 6 00:44	28760739527	job	✅ Passed	—	—

Last known pass: 80decc78 (Jul 6 13:18). First observed failure: this run, e339c83f (Jul 7 12:52). Runs 28833386129 and 28848583109 sit between but were still queued/incomplete at snapshot [IN-FLIGHT].

Failure Cluster

Transformers-backend architecture-resolution failure for remote-code MXFP4 models (no SGLang implementation fallback → KeyError: 'sglang' / incompatible-backend ValueError).

Facts (verified observations)

• Passing run 28794501292 loaded amd/DeepSeek-R1-MXFP4-Preview with sglang's native DeepseekV3 impl — no fallback message present — and passed in 989s.
• This run emits DeepseekV3ForCausalLM has no SGLang implementation, falling back to Transformers implementation on all 8 TP ranks (log lines 4935–4966), then KeyError: 'sglang' at modeling_deepseek.py:1148 (ATTENTION_CLASSES[config._attn_implementation]), scheduler exception on TP0 (line ~5015), server exit -9.
• Server args are identical across passing and failing runs for DeepSeek: trust_remote_code=False, model_impl='auto', quantization=None.
• Kimi failure raised in python/sglang/srt/model_loader/utils.py:182 (resolve_transformers_arch) → ValueError: ... not compatible with SGLang (log line 5184), server exit 1.
• aiter version is identical in passing and failing runs (v0.1.17.dev110+g9127c94a1, verified via [CI-AITER-CHECK] markers in both logs).
• The architecture-resolution code (model_loader/utils.py, models/transformers.py) is unchanged in the regression window (verified via GitHub compare 80decc78..e339c83f; only model_config.py +8/-1, model_loader/loader.py +1/-0, and several model_executor/* files changed among relevant paths).
• No ImportError/ModuleNotFound is logged before the fallback message.

Hypothesised Causes

• [MEDIUM] CI image change (mi35x-20260705 → 20260706/07) bumped a library (most likely transformers) and broke the sglang↔Transformers attention-interface integration.
  • Supporting: same sglang test command, same aiter, same server args; the two failures are both in the Transformers-backend/attention-resolution path; sglang's resolution source (model_loader/utils.py, models/transformers.py) is unchanged in the window; a transformers attention-interface change would simultaneously (a) make DeepseekV3 fall back and mis-set _attn_implementation='sglang' on a remote-code model whose ATTENTION_CLASSES lacks that key, and (b) make Kimi's fallback fail is_backend_compatible.
  • Disconfirming: the exact transformers version could not be read from either log (baked into the image, not pip-installed in the visible steps), so the "version bumped" claim is inferred from the image tag only, not directly confirmed.
  • How to verify: pip show transformers inside rocm/sgl-dev:...-20260705 vs ...-20260707; if it changed, re-run test_kimi_k25_mxfp4.py + test_deepseek_r1_mxfp4_8gpu.py on the older image.
• [LOW] An sglang commit in the 47-commit window (80decc78..e339c83f) altered config/model resolution — candidate c8618967 ("Resolve config declarations onto server_args at the end of __post_init__", #30295) touches config resolution.
  • Supporting: model_config.py changed (+8/-1) in the window; config resolution feeds architecture/impl selection.
  • Disconfirming: the files that actually emit both errors are unchanged; DeepseekV3 is a core model that other (non-MXFP4) DeepSeek tests exercise without this fallback; no import error logged.
  • How to verify: bisect the window on the fixed passing image; check whether DeepseekV3ForCausalLM still resolves natively at each step.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for Kimi K2.5 MXFP4, transformers backend attention, DeepSeek MXFP4. Closest is #28328 ("Improve Transformers backend compatibility for trust_remote_code mode", 2026-06-15) but it predates the regression and is not a targeted fix. Needs new investigation.

Suggested Triage Steps

• Confirm the two-variable split first: diff pip show transformers (and torch) between CI images rocm/sgl-dev:...-20260705 (passing) and ...-20260706/07 (failing).
• If the library changed: re-run both tests on the passing image tag to A/B-confirm the image is the cause, then decide whether to pin the image or adapt sglang's transformers-backend _attn_implementation handling for remote-code models.
• If the library is unchanged: bisect sglang commits 80decc78..e339c83f on a fixed image, starting with config-resolution commit c8618967, watching for when DeepseekV3ForCausalLM stops resolving to the native impl.
• Reproduce locally: sglang serve --model-path amd/DeepSeek-R1-MXFP4-Preview --tp 8 ... and check whether the native DeepseekV3 impl is selected (no "has no SGLang implementation" line).

Status

• Persistence: First failing completed run is this one (Jul 7 12:52 UTC); both tests passed as recently as Jul 6 13:18 UTC. 1 completed failing run so far (two intermediate runs still queued at snapshot).
• Scope: 2 test files in job stage-c-test-large-8-gpu-amd-mi35x (linux-mi35x-gpu-8, 1), workflow pr-test-amd.yml. test_qwen3_coder_next_8gpu.py in the same partition still passes.
• Blocked work: AMD MI35x 8-GPU MXFP4 large-model CI signal for DeepSeek-R1 and Kimi-K2.5.
• In-flight fix: ❌ none found in open PRs.

stage-c-test-large-8-gpu-amd (linux-mi325-8gpu-sglang, 3) — failed step(s): Run test

Commit Info

• sglang: e339c83f
• aiter: 9127c94a (v0.1.17.dev110+g9127c94a1, source: dev image preserved; AITER_COMMIT_OVERRIDE empty — not a scout run)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/test_deepseek_v3_mtp.py	TestDeepseekV3MTP.setUpClass	Exception: Server process exited with code -9 (root: KeyError: 'sglang')	Log
test/registered/amd/test_deepseek_v3_basic.py	TestDeepseekV3Basic.setUpClass	Exception: Server process exited with code -9 (root: KeyError: 'sglang')	Log

(The other 2 files in this shard passed: test_aiter_allreduce_fusion_amd.py, test_aiter_allgather_amd.py. Shard result: 2/4 passed.)

Failure Summary

Both DeepSeek-V3-0324 server launches (--tp 8, one with EAGLE MTP) crash during model init. sglang's native DeepseekV3ForCausalLM was not in the model registry, so the loader fell back to the Transformers backend, which sets config._attn_implementation = "sglang"; DeepSeek's HF remote modeling_deepseek.py then does ATTENTION_CLASSES['sglang'] → KeyError: 'sglang', killing every TP worker (exit -9).

Regression Status

New regression.

Recent history of test_deepseek_v3_mtp.py / test_deepseek_v3_basic.py in job stage-c-test-large-8-gpu-amd (linux-mi325-8gpu-sglang, 3) (pr-test-amd.yml, event=schedule, branch=main, completed runs only):

Date	Run	Test File Status	Failed Function	Error
Jul 7	28867568883	❌ Failed (both)	setUpClass (both)	KeyError: 'sglang' → server exit -9
Jul 6	28814834996	✅ Passed (4/4)	—	—
Jul 5	28750542672	✅ Passed (4/4)	—	—

First failure: Jul 7 run (sglang e339c83f). Last pass: Jul 6 run (sglang 52c6e27e). Regression window = 52c6e27e..e339c83f (44 commits). The two intervening scheduled runs (28833386129, 28848583109) never executed this shard (still queued), so they cannot narrow the window.

Failure Cluster

ModuleNotFoundError: No module named 'cutlass' unregisters native MoE models on ROCm → Transformers fallback KeyError: 'sglang'

Facts (verified observations)

• Log emits DeepseekV3ForCausalLM has no SGLang implementation, falling back to Transformers implementation for all 8 TP ranks before the crash (log lines 8103–8118).
• 31 model modules fail to import in the failing run, each with Ignore import error when loading sglang.srt.models.<mod>: No module named 'cutlass' — including deepseek_v2, deepseek_nextn, deepseek_v4, glm4_moe, kimi_k25, longcat_flash, ernie4, bailing_moe_linear, pixtral, mistral_large_3, etc. (registry swallows these via import_model_classes try/except, registry.py:106-109).
• deepseek_v2.py:2980 EntryClass = [DeepseekV2ForCausalLM, DeepseekV3ForCausalLM, DeepseekV32ForCausalLM] — the arch is natively defined, but the failed import removes it from ModelRegistry.get_supported_archs(), so get_model_architecture (model_loader/utils.py:217,221) sets is_native_supported=False and routes to resolve_transformers_arch.
• Transformers backend (models/transformers.py:614) sets self.text_config._attn_implementation = "sglang"; stack ends at HF remote modeling_deepseek.py:1147 ATTENTION_CLASSES[config._attn_implementation] → KeyError: 'sglang' (log 8153–8192, 8759–8798).
• A/B vs last-pass log: passing run 28814834996 had zero No module named 'cutlass' errors, zero "no SGLang implementation" warnings, and 4/4 passed (mtp 1304s, basic 308s). The cutlass import failure is genuinely new.

Hypothesised Causes (with confidence)

• [HIGH] Commit fefc1743 ("Cute-DSL FP8 MQA logits", #25220) added an unguarded top-level NVIDIA-CUTLASS import into the DeepSeek import chain, which is absent on ROCm.
  • Supporting: verified import chain — deepseek_v2.py:61 from ...dsa.dsa_indexer import Indexer → dsa_indexer.py:11-16 from sglang.jit_kernel.dsa import (…, cutedsl_paged_mqa_logits, …) → jit_kernel/dsa/__init__.py:1 from .cutedsl_paged_mqa_logits import … → cutedsl_paged_mqa_logits.py:15 import cutlass. fefc1743 created the python/sglang/jit_kernel/dsa/ package and added the top-level from sglang.jit_kernel.dsa import (…) block to dsa_indexer.py (confirmed via git show fefc1743). It lands in the regression window; passing run had no cutlass errors.
  • Disconfirming: (none found) — the chain is fully unguarded top-level and reproduces exactly the logged No module named 'cutlass' on deepseek_v2.
  • How to verify: on an MI325 ROCm container, python -c "import sglang.srt.models.deepseek_v2"; confirm it raises ModuleNotFoundError: No module named 'cutlass' from jit_kernel/dsa/cutedsl_paged_mqa_logits.py. Then confirm reverting fefc1743 (or lazily-importing the cutedsl kernel) restores registration.

Note: the other cutlass-touching window commit e85ef548 (#30117) and the deepseek DSA/nvfp4 imports in deepseek_v2 itself are lazy (inside functions guarded by is_sm100_supported() / backend checks), so they are not the cause.

In-flight Fix Check

• In-flight fix: ⚠️ candidate #30413 "Fix additional kernels being triggered in CuteDSL MQA logits path" (open, 2026-07-07) modifies the exact culprit files python/sglang/srt/layers/attention/dsa/dsa_indexer.py and python/sglang/jit_kernel/dsa/paged_mqa_logits.py. Verify whether it moves/guards the top-level cutlass import (i.e. makes jit_kernel/dsa/__init__.py:1 lazy); if it does not, it will not fix this ROCm registration failure.
• No PR explicitly targeting "No module named cutlass" / ROCm cutedsl import guarding was found in open PRs (searched cutlass import rocm, cutedsl paged mqa, No module named cutlass, dsa_indexer cutlass).

Suggested Triage Steps

• Reproduce the import failure directly on ROCm: python -c "import sglang.srt.models.deepseek_v2" in the CI image rocm/sgl-dev:v0.5.14-rocm700-mi30x-20260707.
• Confirm #30413 resolves the top-level cutlass import in jit_kernel/dsa/__init__.py; if not, the fix direction is to make the CuteDSL kernel import lazy / behind an SM100 (Blackwell) guard so ROCm never imports cutlass at module load (mirroring the pattern already used in unquant.py:96 and deepseek_v2 DSA/nvfp4 call sites).
• Secondary defense to evaluate: the Transformers fallback should fail loudly (or be blocked) when a model's HF ATTENTION_CLASSES has no 'sglang' key, instead of crashing every TP worker with exit -9.
• Broader impact check: the same import break unregisters 30 other native archs on ROCm (GLM4-MoE, Kimi, LongCat, Ernie4, Mistral-Large-3, DeepSeek-VL2/OCR, etc.) — expect correlated failures in any AMD job exercising those models.

Status

• Persistence: New as of the Jul 7 scheduled run (sglang e339c83f); previous two scheduled runs (Jul 5–6) passed this shard.
• Scope: Both DeepSeek-V3 tests in this 8-GPU shard fail identically. Root cause (top-level cutlass import) unregisters 31 native model archs on ROCm, so any AMD test loading DeepSeek/GLM4-MoE/Kimi/LongCat/Ernie4/Mistral-Large-3/DeepSeek-VL/OCR is at risk of the same Transformers-fallback KeyError: 'sglang'.
• Blocked work: DeepSeek-V3 (and other native MoE model) serving/CI signal on AMD MI325.
• In-flight fix: ⚠️ #30413 touches the culprit files — verify it guards the import; otherwise a new fix is needed.

stage-c-test-large-8-gpu-amd-mi35x (linux-mi35x-gpu-8, 0) — failed step(s): Run test

Commit Info

• sglang: e339c83f
• aiter: 9127c94a — v0.1.17.dev110+g9127c94a1 (source: Dev/patched CI image preserved; matches Dockerfile default, no override)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/test_kimi_k25_mxfp4_bcg_mi35x.py	TestKimiK25MXFP4BcgMI35x.test_bcg_gsm8k (variant=bcg)	ValueError: KimiK25ForConditionalGeneration has no SGlang implementation … → server exit code 1	Log

Note: the second file in this job, test/registered/cuda_graph/breakable/test_breakable_cuda_graph.py, passed (a transient Health check failed mid-run self-recovered). Job Test Summary: 1/2 passed.

Failure Summary

The kimi test launches sglang serve --model-path amd/Kimi-K2.5-MXFP4. Server startup aborts because KimiK25ForConditionalGeneration is not registered, so sglang falls back to the (incompatible) Transformers impl and raises ValueError, exiting with code 1; the test then fails in popen_launch_server. The model is unregistered because sglang.srt.models.kimi_k25 failed to import — the log shows Ignore import error when loading sglang.srt.models.kimi_k25: No module named 'cutlass' (and the same for deepseek_v2, glm4_moe, etc.).

Regression Status

New regression.

Recent history of test/registered/amd/test_kimi_k25_mxfp4_bcg_mi35x.py in job stage-c-test-large-8-gpu-amd-mi35x (linux-mi35x-gpu-8, 0) (branch main, event schedule, completed runs only):

Date (UTC)	Run	Job (shard 0)	Test File Status	Failed Function	Error
Jul 7 12:52	28867568883	job	❌ Failed	test_bcg_gsm8k	ValueError (arch not registered)
Jul 6 18:40	28814834996	job	✅ Passed	—	— (2/2 passed, elapsed 2290s)
Jul 6 13:18	28794501292	job	✅ Passed	—	—
Jul 6 07:38	28775611300	job	⚠️ Infra	—	VRAM cleanup unsuccessful — job died before tests ran

• First observed failure: 2026-07-07 12:52 UTC (this run).
• Last known pass: 2026-07-06 18:40 UTC (52c6e27e).
• Regression window: 52c6e27e..e339c83f.

Failure Cluster

ModuleNotFoundError: cutlass at model-registry import → arch unregistered → serve fallback ValueError

Facts (verified observations)

• Error (verbatim, failing run): ValueError: KimiK25ForConditionalGeneration has no SGlang implementation and the Transformers implementation is not compatible with SGLang. (model_loader/utils.py:182, via tokenizer_manager.py → multimodal_processor.py:58).
• Import-error markers (failing run): Ignore import error when loading sglang.srt.models.kimi_k25: No module named 'cutlass'; identical for sglang.srt.models.deepseek_v2, glm4_moe, longcat_flash, etc.
• Passing run had ZERO cutlass import errors — grep -c "Ignore import error.*(kimi_k25|deepseek_v2).*cutlass" on the Jul 6 18:40 log returns 0, and the kimi test ran the full model (2290s).
• Environment is constant: both runs pulled the identical image rocm/sgl-dev:v0.5.14-rocm700-mi35x-20260706 and the identical aiter v0.1.17.dev110+g9127c94a1. The only variable is sglang source.
• Import chain (read from source, HEAD): models/kimi_k25.py:31 → models/deepseek_v2.py:61 (from …dsa.dsa_indexer import Indexer) → dsa/dsa_indexer.py:11 (from sglang.jit_kernel.dsa import …) → jit_kernel/dsa/__init__.py:1 (from .cutedsl_paged_mqa_logits import …) → jit_kernel/dsa/cutedsl_paged_mqa_logits.py:15 import cutlass (unconditional, top-level). cutlass (NVIDIA CUTLASS DSL) is not installed on the ROCm image.

Hypothesised Causes (with confidence)

• [HIGH] Commit fefc1743 "Cute-DSL FP8 MQA logits" (#25220) added an unconditional top-level import cutlass in the new jit_kernel/dsa/ package, which is imported unconditionally by dsa_indexer.py → deepseek_v2.py → kimi_k25.py. On non-NVIDIA (ROCm MI35x) hosts without the cutlass DSL package, this raises ModuleNotFoundError, silently deregistering these models.
  • Supporting: (a) code-level chain verified above ends at cutedsl_paged_mqa_logits.py:15 import cutlass; (b) commit merged 2026-07-06 23:34, inside the 52c6e27e..e339c83f window; (c) passing run (pre-commit) had no cutlass import errors, failing run does; (d) constant image + aiter rule out an environment change; (e) two open PRs target this exact code — #30410 "[NPU] Fix npu cutlass import" edits dsa_indexer.py, and #30413 explicitly cites #25220 as buggy.
  • Disconfirming: (none found) — env is identical across pass/fail and the import chain is deterministic.
  • How to verify: on the MI35x image, python -c "import sglang.srt.models.kimi_k25" should raise ModuleNotFoundError: cutlass; then apply #30410 (guarded import) and re-run test_kimi_k25_mxfp4_bcg_mi35x.py.

In-flight Fix Check

• In-flight fix: ✅ #30410 "[NPU] Fix npu cutlass import" (open since 2026-07-07 15:42 UTC) — edits exactly python/sglang/srt/layers/attention/dsa/dsa_indexer.py to guard the cutlass import for non-NVIDIA platforms; directly addresses this regression. Chase reviewers; do NOT open a duplicate. Related: #30413 (cites #25220 as the source bug).
• Search used: search/issues?q=repo:sgl-project/sglang+is:pr+is:open+cutlass+import and …+cutedsl.

Suggested Triage Steps

• Confirm #30410's guard also covers ROCm/AMD (not only NPU) — the failing path is dsa_indexer.py:11 → jit_kernel/dsa/__init__.py:1; verify the import is lazy or try/except-guarded so import sglang.srt.models.kimi_k25 succeeds without cutlass.
• Alternatively, make jit_kernel/dsa/__init__.py not eagerly import cutedsl_paged_mqa_logits (defer the import cutlass to actual CuteDSL use), so all DeepSeek-derived models import on platforms lacking CUTLASS DSL.
• After the fix merges, re-run stage-c-test-large-8-gpu-amd-mi35x shard 0 and confirm test_kimi_k25_mxfp4_bcg_mi35x.py returns to a ~2290s full run.

Status

• Persistence: First failing run 2026-07-07 12:52 UTC; last green 2026-07-06 18:40 UTC. 1 completed failing run so far.
• Scope: Blocks test_kimi_k25_mxfp4_bcg_mi35x on AMD MI35x. Root cause (unconditional import cutlass in the DeepSeek-V2 import chain) deregisters all DeepSeek-V2-derived models on any host lacking the CUTLASS DSL package, so impact likely extends to other AMD/NPU model tests exercising deepseek_v2/kimi_k25/glm4_moe.
• In-flight fix: ✅ #30410 (open since 2026-07-07) — awaiting merge.

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-07-08 18:54 UTC)

[amd-bot]

pr-test-amd.yml — Per-Job Analysis (continued 4/9)

multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 3) — failed step(s): Run diffusion server tests (1-GPU)

Commit Info

• sglang: 99db3b0f
• aiter: 9127c94a (v0.1.17.dev110+g9127c94a1, source: Dockerfile default — AITER_COMMIT_OVERRIDE empty)

Failed Tests

Job: multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 3) — all in sglang/multimodal_gen/test/single_test_file/test_update_weights_from_disk.py.

Test Function	Param	Error	Log
TestUpdateWeightsFromDisk::test_update_weights_from_disk_default	Qwen-Image	AssertionError: assert 500 == 200 (server returned 500)	Log
TestUpdateWeightsFromDisk::test_update_weights_specific_modules	Qwen-Image	AssertionError: Update failed: … Inplace update to inference tensor outside InferenceMode is not allowed	Log
TestUpdateWeightsFromDisk::test_corrupted_weights_rollback	Qwen-Image	assert (500 == 200)	Log
TestUpdateWeightsFromDiskWithOffload::test_update_weights_with_offload_enabled	FLUX.2-klein-base-4B	AssertionError: Expected 200, got 500	Log
TestUpdateWeightsFromDiskWithOffload::test_update_weights_with_offload_enabled	Qwen-Image	AssertionError: Expected 200, got 500	Log
TestUpdateWeightsFromDisk::test_update_weights_from_disk_default	FLUX.2-klein-base-4B	ERROR at setup: AssertionError: No weights dir for transformer in /tmp/sglang_perturbed_vae_*	Log
TestUpdateWeightsFromDisk::test_update_weights_specific_modules	FLUX.2-klein-base-4B	ERROR at setup: same fixture assertion	Log
TestUpdateWeightsFromDisk::test_update_weights_rejects_invalid_requests	FLUX.2-klein-base-4B	ERROR at setup: same fixture assertion	Log
TestUpdateWeightsFromDisk::test_corrupted_weights_rollback	FLUX.2-klein-base-4B	ERROR at setup: same fixture assertion	Log

Summary line: 5 failed, 1 passed, 11 warnings, 4 errors in 582.83s. The only passing test was test_update_weights_rejects_invalid_requests[Qwen-Image].

Failure Summary

Two distinct symptom groups in the newly-added test_update_weights_from_disk.py (multimodal_gen post-training weight updater), both AMD MI325 shard 3. Group A (Qwen-Image + both offload cases, 5 failed): the POST /update_weights_from_disk endpoint returns HTTP 500; the server-side error is RuntimeError: Inplace update to inference tensor outside InferenceMode is not allowed, surfaced during rollback after an initial AssertionError: param_data.shape == loaded_weight.shape. Group B (all four FLUX.2-klein-base-4B, 4 errors): the diffusion_server_no_offload fixture fails at setup with AssertionError: No weights dir for transformer, so those tests never reach the endpoint.

Regression Status

Never-passed (no green run found in queryable window). This test file was introduced with the multimodal_gen feature merge on 2026-07-04 (b7c3709f, #29903). Every completed scheduled pr-test-amd.yml run since has shown the identical 5 failed / 1 passed / 4 errors in this file on shard 3.

Recent history of test_update_weights_from_disk.py in multimodal-gen-test-1-gpu-amd (…, 3) (branch main, event schedule, completed runs only):

Date (UTC)	Run	Job	Test File Status	Detail
Jul 7 07:15	28848583109	shard 3	❌ Failed	5 failed, 4 errors (this run)
Jul 6 18:40	28814834996	shard 3	❌ Failed	identical 5 failed, 4 errors
Jul 6 13:18	28794501292	shard 3	❌ Failed	identical 5 failed, 4 errors
Jul 6 07:38	28775611300	shard 3	❌ Failed	identical 5 failed, 4 errors
Jul 6 00:44	28760739527	shard 3	❌ Failed	identical 5 failed, 4 errors

(Later runs 28833386129, 28867568883, 28890204713 were still queued/in-flight at snapshot — excluded.)

Failure Cluster

Two symptom-level clusters, both in the multimodal_gen weight-update path:

• A. "InferenceMode inplace-update 500 in update_weights_from_disk" — Qwen-Image + both offload cases.
• B. "diffusion_server_no_offload fixture: missing transformer weights dir (FLUX.2)" — the 4 FLUX.2 setup errors.

Facts (verified observations)

• Group A server stack trace (verbatim, run 28848583109): weights_updater.py:339 _apply_weights → weights_updater.py:139 _load_weights_into_module → weights_updater.py:200 load_weights_into_model → linear.py:411 weight_loader → assert param_data.shape == loaded_weight.shape → AssertionError. This triggers weights_updater.py:350 self._rollback → weights_updater.py:381 → linear.py:533 → linear.py:575 param_data.copy_(loaded_weight) → RuntimeError: Inplace update to inference tensor outside InferenceMode is not allowed. The RuntimeError is the message returned to the client (500).
• Verified source: linear.py:411 is assert param_data.shape == loaded_weight.shape; linear.py:575 is param_data.copy_(loaded_weight) (an in-place op on a parameter that is an inference tensor).
• Group B setup trace (verbatim): test_update_weights_from_disk.py:410 diffusion_server_no_offload → test_update_weights_from_disk.py:220 _compute_checksum_from_disk → assert os.path.exists(...) → AssertionError: No weights dir for transformer in /tmp/sglang_perturbed_vae_*. FLUX.2 tests never reach the endpoint.
• One late FLUX.2 attempt also logged HFValidationError / ValueError: Could not find model at /sgl-data/hf-cache/hub/models--black-forest-labs--FLUX.2-klein-base-4B/snapshots/… (log line ~12689) — a model-path resolution error specific to the FLUX.2 fixture setup.
• Reproducibility: identical 5 failed / 1 passed / 4 errors across all 5 completed runs (Jul 6–7).
• aiter is the Dockerfile default (AITER_COMMIT_OVERRIDE empty) — not a scout run, no aiter delta involved.
• Non-fatal environment noise present but unrelated: MIOpen(HIP): Error [Init] Not found (conv kernel db miss), AITER_USE_SYSTEM_TRITON=1 … triton 3.4.0 < 3.6.0 warning, and a [local-pull] Error response from daemon … context deadline exceeded (registry pull fell back).

Hypothesised Causes (with confidence)

• [MEDIUM] Group A: the weight-update code path modifies model parameters in-place (param_data.copy_ at linear.py:575, linear.py:412) while the model tensors are inference tensors (allocated under torch.inference_mode()), which PyTorch forbids. The rollback path deterministically hits this.
  • Supporting: verified stack trace terminates at param_data.copy_(loaded_weight); error text is PyTorch's inference-tensor guard; fails on every run.
  • Disconfirming: the first exception in the chain is a shape-mismatch AssertionError at linear.py:411, i.e. the weight-shape check fails before the InferenceMode error — so a shape/param-mapping bug in the updater may be the true trigger, and the InferenceMode error is only surfaced because rollback runs. The 500 message therefore masks the primary shape-mismatch fault.
  • How to verify: run test_update_weights_specific_modules[Qwen-Image] locally on MI325 with a breakpoint at weights_updater.py:200; inspect param.data.shape vs loaded_weight.shape and whether param.data is an inference tensor; check whether weight loading is wrapped in torch.inference_mode() / needs a clone.
• [MEDIUM] Group B: the diffusion_server_no_offload fixture builds a "perturbed VAE" temp dir that lacks a transformer/ weights subdir for FLUX.2 (_compute_checksum_from_disk asserts the dir exists), i.e. a test-fixture / model-layout mismatch specific to FLUX.2-klein-base-4B.
  • Supporting: all four FLUX.2 failures are setup ERRORs at the same fixture line 220; the accompanying HFValidationError shows the FLUX.2 snapshot path is being mis-resolved.
  • Disconfirming: Qwen-Image reaches the endpoint fine, so the fixture works for at least one model — the gap is FLUX.2-specific layout, not a wholesale fixture bug.
  • How to verify: inspect the perturbed-dir construction in the fixture and confirm whether FLUX.2's transformer weights are copied/perturbed; check the model snapshot path handling.
• [LOW] Both groups are AMD/ROCm-specific rather than general sglang bugs.
  • Supporting: only observed in the AMD workflow here.
  • Disconfirming: nothing in the stack traces is ROCm-specific (inference-tensor guard and fixture path checks are backend-agnostic); no NVIDIA comparison run was queried. Treat as unproven.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for update_weights_from_disk (12 hits, none matching this InferenceMode/diffusion weight-update failure), InferenceMode diffusion (0), multimodal_gen weight (32 hits, none addressing this test). No open PR targets this specific failure.

Suggested Triage Steps

• Group A: reproduce test_update_weights_specific_modules[Qwen-Image] on MI325; at weights_updater.py:200 inspect the shape mismatch at linear.py:411 and whether param.data is an inference tensor — the fix likely lives in the multimodal_gen weight-updater (clone before in-place copy, or run the update outside inference_mode), plus correct param↔weight shape mapping so rollback isn't triggered.
• Group B: inspect the diffusion_server_no_offload fixture's perturbed-VAE construction for FLUX.2-klein-base-4B (missing transformer/ subdir at test_update_weights_from_disk.py:220) and the snapshot-path resolution that raised HFValidationError.
• Since this test file has never passed on AMD shard 3 since it landed (#29903, 2026-07-04), consider whether it should be gated/quarantined on AMD until the weight-updater path is fixed, to unblock the shard-3 CI signal.
• Optionally A/B against an NVIDIA run of the same test to confirm/deny the [LOW] AMD-specificity hypothesis before scoping the fix.

Status

• Persistence: Failing on every completed scheduled run since the feature merged (2026-07-04); 5 consecutive completed runs verified (Jul 6 00:44 → Jul 7 07:15 UTC), identical 5 failed / 4 errors. Never-passed.
• Scope: multimodal-gen-test-1-gpu-amd shard 3, test_update_weights_from_disk.py. Other multimodal-gen shards also fail in these runs but on different test files (not analyzed here).
• Blocked work: Blocks the multimodal_gen post-training weight-update (update_weights_from_disk) CI signal on AMD.
• In-flight fix: ❌ none found in open PRs — needs new PR or investigation.

multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 1) — failed step(s): Run diffusion server tests (2-GPU)

Commit Info

• sglang: 99db3b0f
• aiter: 9127c94a (v0.1.17.dev110+g9127c94a1, source: Dockerfile default — AITER_COMMIT_OVERRIDE empty)

Failed Tests

Test File	Test Function	Error	Log
python/sglang/multimodal_gen/test/server/test_server_2_gpu.py	TestDiffusionServerTwoGpu.test_diffusion_generation[flux2_modelopt_fp8_tp2_t2i]	openai.InternalServerError: Internal Server Error (server-side: RuntimeError: CUDA error: HIPBLAS_STATUS_NOT_SUPPORTED)	Log

Result: 1 failed, 11 passed, 13 deselected (the other 11 two-GPU diffusion cases passed). Only the FLUX.2-dev modelopt-fp8 case failed.

Failure Summary

The single failing case is flux2_modelopt_fp8_tp2_t2i, which serves black-forest-labs/FLUX.2-dev with the modelopt-fp8 transformer (lmsys/flux2-dev-modelopt-fp8-sglang-transformer) at tp_size=2. The client received an HTTP 500 (openai.InternalServerError) from POST /v1/images/generations. The server-side scheduler error is CUDA error: HIPBLAS_STATUS_NOT_SUPPORTED, raised during the first denoising step while running the fp8 linear projection.

Regression Status

Known recurring failure (not new; an in-flight fix has been open since 2026-06-22).

Recent history of test_server_2_gpu.py (test_diffusion_generation[flux2_modelopt_fp8_tp2_t2i]) in job multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 1) (branch=main, event=schedule, completed runs only):

Date (UTC)	Run	Test Case Status	Error
Jul 7 07:15	28848583109	❌ Failed	HIPBLAS_STATUS_NOT_SUPPORTED
Jul 6 18:40	28814834996	❌ Failed	HIPBLAS_STATUS_NOT_SUPPORTED
Jul 6 07:38	28775611300	❌ Failed	HIPBLAS_STATUS_NOT_SUPPORTED
Jul 6 00:44	28760739527	❌ Failed	HIPBLAS_STATUS_NOT_SUPPORTED

The parent job has failed in every completed scheduled run back to 2026-07-01. No passing run for this case was found in the queryable window (Jul 1–7); combined with in-flight fix #28889 open since 2026-06-22, this case appears to have never passed on AMD.

Failure Cluster

HIPBLAS_STATUS_NOT_SUPPORTED in torch._scaled_mm (fp8 e4m3fn GEMM on ROCm)

Facts (verified observations)

• Client-side traceback ends at test_server_utils.py:1041 → openai.InternalServerError: Internal Server Error (log around the FAILED ...flux2_modelopt_fp8_tp2_t2i line).
• Server-side scheduler traceback (verbatim tail):

  File ".../multimodal_gen/runtime/models/dits/flux_2.py", line 77, in _get_qkv_projections
      query, _ = attn.to_q(hidden_states)
  File ".../multimodal_gen/runtime/layers/linear.py", line 427, in forward
      output_parallel = self.quant_method.apply(self, input_, bias)
  File ".../multimodal_gen/runtime/layers/quantization/modelopt_quant.py", line 425, in apply
      return apply_fp8_linear(
  File ".../sglang/srt/layers/quantization/fp8_utils.py", line 1790, in apply_fp8_linear
      output = torch._scaled_mm(
  RuntimeError: CUDA error: HIPBLAS_STATUS_NOT_SUPPORTED when calling `HIPBLAS_STATUS_NOT_SUPPORTED`
  

• The error surfaces at denoising_step_0 (first denoising step), during the to_q projection of the first transformer block.
• fp8_utils.py:1790 is the ROCm rowwise-scaled branch calling torch._scaled_mm(qinput, weight, out_dtype=..., scale_a=x_scale, scale_b=weight_scale.t(), bias=bias) (verified in current source).
• Reproducibility: identical error (same top-of-stack, same denoising_step_0) in 4/4 most recent completed runs; only this one case fails, the other 11 two-GPU cases pass in the same job.
• Environment: AMD MI325 (gfx942/mi30x), aiter Dockerfile default 9127c94a — no aiter override (this is a regular pr-test-amd.yml schedule run, not AITER Scout).

Hypothesised Causes (with confidence)

• [HIGH] hipBLAS on ROCm does not support the fp8 (e4m3fn) torch._scaled_mm GEMM configuration used by the modelopt-fp8 path for FLUX.2-dev, so the call returns HIPBLAS_STATUS_NOT_SUPPORTED.
  • Supporting: stack trace terminates exactly at torch._scaled_mm in apply_fp8_linear; the error is a hipBLAS status code (a backend-capability error, not a numeric/shape assert); only the fp8-quantized case fails while all non-fp8 cases pass; an AMD-authored fix (#28889) explicitly describes a dequant->bf16 fallback in apply_fp8_linear to work around this exact error.
  • Disconfirming: (none found) — the failure is deterministic and isolated to the fp8 GEMM call.
  • How to verify: run sglang serve --model-path black-forest-labs/FLUX.2-dev --model-type diffusion --num-gpus 2 --tp-size 2 --transformer-path lmsys/flux2-dev-modelopt-fp8-sglang-transformer on MI325 and issue one image request; confirm the exception origin is torch._scaled_mm. Then apply #28889 and re-run the case.

In-flight Fix Check

• In-flight fix: ✅ #28889 "[AMD] Fix FP8 e4m3fn GEMM on ROCm: dequant->bf16 fallback in apply_fp8_linear (fixes flux2 modelopt-fp8 HIPBLAS_STATUS_NOT_SUPPORTED)" — open since 2026-06-22 (author kangwangamd, not a draft, last updated 2026-07-07). Directly targets this failure; chase reviewers rather than opening a duplicate.
• Searched: repo:sgl-project/sglang is:pr is:open flux2 fp8 and ... HIPBLAS.

Suggested Triage Steps

• Verify #28889 still applies cleanly on 99db3b0f and re-run the flux2_modelopt_fp8_tp2_t2i case on MI325 to confirm it resolves the HIPBLAS_STATUS_NOT_SUPPORTED error.
• If merging #28889 is blocked, consider quarantining/skipping flux2_modelopt_fp8_tp2_t2i on AMD until the fp8-GEMM fallback lands, so the 2-GPU diffusion job produces a clean signal for its other 11 cases.
• Confirm whether the same apply_fp8_linear ROCm path affects other modelopt-fp8 diffusion cases (e.g. ideogram4_fp8_tp2_t2i passes here — worth confirming it uses fa/a different code path, which would corroborate the fp8-GEMM-specific hypothesis).

Status

• Persistence: Failing in every completed scheduled run of this job back to at least 2026-07-01; the fix PR has been open since 2026-06-22, indicating a long-standing AMD gap. Same symptom in 4/4 most recently verified completed runs.
• Scope: 1 test case (flux2_modelopt_fp8_tp2_t2i) in the multimodal-gen-test-2-gpu-amd job; the other 11 two-GPU diffusion cases pass. Blocks FLUX.2-dev modelopt-fp8 CI signal on AMD only (regular pr-test-amd.yml, default aiter).
• In-flight fix: ✅ #28889 (open since 2026-06-22) — awaiting merge.

multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 1) — failed step(s): N/A

Commit Info

• sglang: 99db3b0f
• aiter: N/A for this run — the job's log blob is missing (BlobNotFound), so [CI-AITER-CHECK] markers cannot be read for run 28848583109. The adjacent completed part-1 run 85451536113 reports v0.1.17.dev110+g9127c94a1 (dev image preserved; Dockerfile default 9127c94a). aiter is not implicated (see Hypothesised Causes).

Failed Tests

Test File	Test Function	Error	Log
python/sglang/multimodal_gen/test/server/test_server_1_gpu.py	TestDiffusionServerOneGpu::test_diffusion_generation[lingbot_world_realtime_plastic_beach]	TimeoutError: Server not ready within 1200.0s (server stuck downloading model)	Job

⚠️ Test identity for the current run is inferred, not directly observed. The current job's log blob does not exist and no failure artifacts were uploaded (upload step never ran). The test-level identity above is carried over from the immediately adjacent completed part-1 runs, which fail identically and deterministically on this exact test (evidence below). What is directly observable for the current job: it ended completed/failure while step 9 "Run diffusion server tests (1-GPU)" was still in_progress, after running exactly ~60 min.

Failure Summary

Job multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 1) failed. Its log is unavailable (BlobNotFound) and step 9 never reached a conclusion, so the failure could not be read from this run's own log. Every recent completed run of this same partition-1 job fails on test_diffusion_generation[lingbot_world_realtime_plastic_beach] because the robbyant/lingbot-world-fast-diffusers model download stalls and the server never becomes ready within the 1200s deadline.

Regression Status

Known recurring / persistent failure — no passing part-1 run found in the queryable window (Jul 3 – Jul 7). This is NOT a new regression at commit 99db3b0f.

Recent history of test_server_1_gpu.py (test_diffusion_generation[lingbot_world_realtime_plastic_beach]) in job multimodal-gen-test-1-gpu-amd (..., 1) (branch main, event schedule, completed runs only):

Date (UTC)	Run	Test File Status	Failed Function	Error
Jul 7 07:15	28848583109	❌ Failed [log missing]	(inferred) lingbot..plastic_beach	job died mid-step 9, no log/artifacts
Jul 6 18:40	28814834996	❌ Failed	lingbot..plastic_beach	TimeoutError: Server not ready within 1200.0s
Jul 6 13:18	28794501292	❌ Failed	lingbot..plastic_beach	TimeoutError: Server not ready within 1200.0s
Jul 6 07:38	28775611300	❌ Failed	lingbot..plastic_beach	server startup timeout
Jul 6 07:48	28760739527	❌ Failed	lingbot..plastic_beach	server startup timeout
Jul 5 18:25	28750542672	❌ Failed	lingbot..plastic_beach	server startup timeout
Jul 5 12:31	28740865240	❌ Failed	lingbot..plastic_beach	server startup timeout
Jul 3 20:xx	28677620397	❌ Failed	lingbot..plastic_beach	TimeoutError: Server not ready within 1200.0s

First observed failure in window: Jul 3 2026 (earliest completed run I inspected). Last known passing part-1 run: none found in Jul 3 – Jul 7. Runs after the current one (28867568883, 28890204713) were still in progress at snapshot time and are excluded.

Failure Cluster

LingBot-World model download stalls → diffusion server startup TimeoutError (1200s)

Facts (verified observations, no interpretation)

• Current job 85558178107: status=completed, conclusion=failure; step 9 "Run diffusion server tests (1-GPU)" = in_progress (no conclusion); step 10 "Upload diffusion failure artifacts" = pending (never ran). Started 2026-07-07T11:53:07Z, completed 2026-07-07T12:53:09Z → ~60 min exactly.
• Log endpoints for the current job return BlobNotFound; the run-level log archive returns HTTP 404 (run still queued). 0 artifacts on the run.
• Step 9 has timeout-minutes: 90; there is no job-level timeout-minutes for this job (.github/workflows/pr-test-amd.yml:614-753). So ~60 min is not a configured timeout boundary.
• In the adjacent completed run 85451536113, the server stdout tail printed at the timeout shows: Model found in cache but incomplete, will download from HF Hub → Downloading model snapshot ... (attempt 1/5) → Fetching 34 files: 88%|30/34 [00:17<00:02 as the last server output before the 1200s deadline. Server never printed "ready".
• The 1-GPU cache-diagnostics step only pre-checks Wan2.1 / HunyuanVideo / Qwen-Image / FLUX.1 / FLUX.2 — not lingbot-world-fast-diffusers (.github/workflows/pr-test-amd.yml:696-706).
• On retry, a follow-on error appears: RuntimeError: Scheduler port 5555 is unavailable and --strict-ports is enabled — the crashed server's port isn't released before the retry (collateral of the primary timeout, not the root cause).
• The lingbot case is defined at python/sglang/multimodal_gen/test/server/gpu_cases.py:420-435 (model_path="robbyant/lingbot-world-fast-diffusers", LingBotWorldCausalDMDPipeline, --warmup false).

Hypothesised Causes (with confidence, with disconfirming evidence)

• [HIGH] The persistent part-1 test failure is caused by the robbyant/lingbot-world-fast-diffusers model download not completing within the 1200s server-ready deadline. The model is not pre-cached on the AMD runners (diagnostics step omits it) and each run reports the cache as "incomplete" and re-downloads 34 files, stalling at ~30/34.
  • Supporting: identical TimeoutError: Server not ready within 1200.0s on this exact test in every completed part-1 run Jul 3–Jul 6; server tail is frozen inside Fetching 34 files (download phase), never reaching model load / ready; cache-diagnostics step does not warm this model.
  • Disconfirming: I could not read the current run's log to reconfirm the tail; other 1-GPU partitions (0/2/3) sometimes pass, so the runner+HF path is not globally broken.
  • How to verify: pre-seed robbyant/lingbot-world-fast-diffusers into /sgl-data/hf-cache on the mi325 runners (or add it to the diagnostics warm-up loop) and re-run part 1; alternatively raise wait_deadline for this case and confirm it eventually starts.
• [MEDIUM] The current run additionally suffered an abnormal job/runner termination (not a clean test failure): the job ended at exactly ~60 min with step 9 stuck in_progress, the upload step skipped, and the log blob never created — a signature of a lost/killed runner rather than pytest completing.
  • Supporting: missing log blob + in_progress terminal step + no artifacts + ~60 min wall clock with no matching configured timeout.
  • Disconfirming: exact 60 min could be an external/self-hosted enforcement not visible in the workflow YAML; cannot confirm without runner-side logs.
  • How to verify: inspect the self-hosted runner's diagnostic logs for run 28848583109 around 12:53 UTC; check runner agent for a disconnect/kill event.
• [LOW] aiter involvement.
  • Supporting: none specific.
  • Disconfirming: the failure occurs during HF model download, well before any attention/kernel execution; aiter version is unchanged across passing and failing runs.
  • How to verify: not actionable — failure precedes any aiter call.

In-flight Fix Check

• In-flight fix: ❌ none found for the lingbot CI download/timeout. Searched open PRs for lingbot (6 hits — all feature work: #30361 "support LingBot-World 2.0", #28065, #27929, #27755, #27146), lingbot+cache (2 hits, same), and diffusion+server+timeout (19 hits, none addressing this). No open PR pre-caches the model or raises the deadline for this case.

Suggested Triage Steps

• Confirm the download-stall hypothesis: pre-seed robbyant/lingbot-world-fast-diffusers into the persistent /sgl-data/hf-cache on the mi325 runners, or add it to the 1-GPU cache-diagnostics warm-up loop (.github/workflows/pr-test-amd.yml:696-706), then re-run part 1.
• If the stall persists even when warmed, check HF Hub reachability for the last 4 of the 34 files from an mi325 runner (the download freezes at ~30/34), and consider raising wait_deadline for this case in test_server_utils.py as a stop-gap.
• Investigate the current run's abnormal termination separately: pull the self-hosted runner agent logs for run 28848583109 near 12:53 UTC to determine whether the runner was lost/killed (explains the missing log blob + stuck step).
• Independently, consider having the server release scheduler port 5555 on startup failure so retries don't compound into --strict-ports errors.

Status

• Persistence: test_diffusion_generation[lingbot_world_realtime_plastic_beach] in part-1 has failed every completed scheduled run from Jul 3 to Jul 7 2026 (≥8 consecutive completed runs); no green part-1 run in the queryable window.
• Scope: partition 1 of multimodal-gen-test-1-gpu-amd on pr-test-amd.yml (schedule). Other partitions (0/2/3) are intermittently green; the failure is specific to the lingbot case assigned to part 1.
• Blocked work: blocks the AMD 1-GPU diffusion server signal for the LingBot-World realtime path.
• In-flight fix: ❌ none found — needs a new PR (pre-cache the model / warm-up loop entry / deadline bump) plus a separate look at the runner-loss on this specific run.

stage-b-test-1-gpu-small-amd (linux-mi325-1gpu-sglang, 5) — failed step(s): Run test

Commit Info

• sglang: 99db3b0f
• aiter: 9127c94a — v0.1.17.dev110+g9127c94a1 (source: dev image preserved; AITER_COMMIT_OVERRIDE empty). Not relevant to this failure.

Failed Tests

Test File	Test Function	Error	Log
test/registered/unit/managers/test_mm_process_config.py	TestOverrideProcessorsConfigInjection.test_ernie45_vl_injects_images_kwargs	ModuleNotFoundError: No module named 'cutlass'	Log

10 of 11 tests in the file passed; the single error caused FAILED (errors=1) → job exit 255. All 9 other test files in shard 5 passed.

Failure Summary

test_ernie45_vl_injects_images_kwargs errors during collection when it imports sglang.srt.multimodal.processors.ernie45_vl. The import chain reaches deepseek_v2.py → dsa_indexer.py → sglang.jit_kernel.dsa → cutedsl_paged_mqa_logits.py, which does an unconditional top-level import cutlass. The NVIDIA CUTLASS Python package is not installed in the AMD ROCm CI image, so the import raises ModuleNotFoundError.

Regression Status

New regression — first failing on this run.

Recent history of test/registered/unit/managers/test_mm_process_config.py in job stage-b-test-1-gpu-small-amd (…, 5) (pr-test-amd.yml, branch=main, event=schedule, completed runs only):

Date (UTC)	Run	Job	Test File Status	Failed Function	Error
07-07 12:07	28848583109	shard 5	❌ Failed	test_ernie45_vl_injects_images_kwargs	ModuleNotFoundError: No module named 'cutlass'
07-07 01:46	28814834996	shard 5	✅ Passed	—	—

First observed failure: 2026-07-07 (this run). Last known pass: run 28814834996 (head 52c6e27e, started 07-07 01:46 UTC). Intervening scheduled runs (28833386129, 28867568883) are still queued/in-flight and excluded.

Failure Cluster

ModuleNotFoundError: cutlass — unconditional CuteDSL import breaks AMD

Facts (verified observations)

• Verbatim error (log): ModuleNotFoundError: No module named 'cutlass', raised at python/sglang/jit_kernel/dsa/cutedsl_paged_mqa_logits.py line 15 (import cutlass).
• Import chain (from the traceback): test_mm_process_config.py:236 → multimodal/processors/ernie45_vl.py:15 → models/ernie45_vl.py:42 → models/ernie45_moe_vl.py:44 → models/deepseek_v2.py:61 → layers/attention/dsa/dsa_indexer.py:11 → jit_kernel/dsa/__init__.py:1 → cutedsl_paged_mqa_logits.py:15.
• python/sglang/jit_kernel/dsa/cutedsl_paged_mqa_logits.py:14-17 performs unconditional top-level import cutlass / import cutlass.cute / from cutlass.utils import HardwareInfo (verified by reading the file at HEAD).
• python/sglang/jit_kernel/dsa/__init__.py:1 imports cutedsl_paged_mqa_logits unconditionally (no platform guard).
• Both the new file and the top-level from sglang.jit_kernel.dsa import (...) in dsa_indexer.py were introduced by commit fefc1743 (#25220), merged 2026-07-06 23:34 PST / 2026-07-07 06:34 UTC (git show verified).
• Test passed in the last completed scheduled run before that commit (07-07 01:46 UTC, shard 5) and failed in the first scheduled run after it — the run under analysis.

Hypothesised Causes

• [HIGH] Commit fefc1743 (#25220) added an unconditional import cutlass in jit_kernel/dsa/cutedsl_paged_mqa_logits.py that is eagerly pulled in via jit_kernel/dsa/__init__.py → dsa_indexer.py → deepseek_v2.py, so importing any DeepSeek-derived model fails on AMD where cutlass isn't installed.
  • Supporting: the exact added file + line appears at the top of the failing traceback; the commit also converted dsa_indexer.py to a top-level from sglang.jit_kernel.dsa import (...) (git show); timing — passed in last pre-commit run, failed in first post-commit run.
  • Disconfirming: (none found) — the import chain, timing, and code diff all converge.
  • How to verify: on an AMD ROCm container, run python3 -c "import sglang.srt.models.deepseek_v2"; expect ModuleNotFoundError: No module named 'cutlass'. Confirm reverting fefc1743 (or guarding the import) clears it.

In-flight Fix Check

• In-flight fix: ❌ none found addressing the AMD import error. The closest open PR #30413 ("Fix additional kernels being triggered in CuteDSL MQA logits path", open since 2026-07-07, a follow-up to #25220) modifies paged_mqa_logits.py, dsa_indexer.py, dsa_backend.py but does not touch the unconditional import cutlass in cutedsl_paged_mqa_logits.py nor guard dsa/__init__.py, so it will not resolve this ModuleNotFoundError on ROCm. (Searched: cutlass import, cutedsl import, dsa cutlass, ModuleNotFoundError cutlass in open PRs.)

Suggested Triage Steps

• Reproduce on AMD: python3 -c "import sglang.srt.models.deepseek_v2" in the ROCm CI image; confirm the cutlass import error.
• If confirmed, guard the CuteDSL import so AMD/non-CUTLASS platforms don't eagerly load it — e.g. wrap import cutlass (and the from .cutedsl_paged_mqa_logits import ... in jit_kernel/dsa/__init__.py) in try/except ImportError or a platform/SM100 check, since this path targets Blackwell SM100.
• Alternatively, ping the author of #25220 / reviewers of #30413 to fold in the import guard, or open a dedicated fix PR.

Status

• Persistence: New — first failing completed run (2026-07-07 12:07 UTC); prior completed run (07-07 01:46 UTC) passed.
• Scope: 1 test function in test_mm_process_config.py, shard 5. Root cause is broad: any AMD test importing deepseek_v2 (or DeepSeek-derived models: ernie45_vl, etc.) via a top-level import will hit the same error — likely affects additional shards/tests.
• Blocked work: fails the shard-5 job (exit 255); blocks multimodal / DeepSeek-family import paths on AMD ROCm CI.
• In-flight fix: ❌ none found — needs a new PR or an import guard folded into #30413.

stage-b-test-1-gpu-small-amd (linux-mi325-1gpu-sglang, 4) — failed step(s): Run test

Commit Info

• sglang: 99db3b0fa51c6f622ce0c0a0015c46e5b00ca104
• aiter: v0.1.17.dev110+g9127c94a1 (9127c94a) — source: dev/patched image preserved ([CI-AITER-CHECK] Dev/patched version detected → skipping rebuild, log line 4267)

Failed Tests

Test File	Test Function	Error	Log
test/registered/mla/test_mla_fp8.py	TestMLA.setUpClass	Exception: Server process exited with code -9 (root: KeyError: 'sglang' during model load)	Log

Failure Summary

test_mla_fp8.py failed in setUpClass: the server for neuralmagic/DeepSeek-Coder-V2-Lite-Instruct-FP8 crashed (exit code -9) before any test ran (Ran 0 tests, FAILED (errors=1)). The scheduler died with KeyError: 'sglang' because the model could not use the native SGLang DeepseekV2ForCausalLM implementation and fell back to the Transformers backend, whose remote custom modeling code does not recognize the sglang attention implementation. The job reported 8/9 passed — only this file failed.

Regression Status

New regression — first observed in this run (99db3b0fa5, 2026-07-07 07:15 UTC).

Recent history of test/registered/mla/test_mla_fp8.py in job stage-b-test-1-gpu-small-amd (linux-mi325-1gpu-sglang, 4) (branch=main, event=schedule, completed runs only):

Date	Run	Test File Status	Failed Function	Error
Jul 7 07:15	28848583109	❌ Failed	TestMLA.setUpClass	KeyError: 'sglang' → exit -9
Jul 6 18:40	28814834996	✅ Passed	—	— (loaded native type=DeepseekV2ForCausalLM, quant=fp8)
Jul 6 07:38	28775611300	✅ Passed	—	—

(The Jul 6 13:18 run 28794501292 did not run this test file on this shard, so it is excluded. Runs at Jul 7 00:44 and later are still queued → excluded.)

First observed failure: 2026-07-07 07:15 UTC. Last known passing: 2026-07-06 18:40 UTC (52c6e27e7e). Regression window: 52c6e27e7e..99db3b0fa5 (34 commits).

Failure Cluster

Native model import fails on AMD (No module named 'cutlass') → Transformers fallback → KeyError: 'sglang'

Facts (verified observations)

• Failing log line 4461: KeyError: 'sglang', raised at modeling_deepseek.py line 1203 self.self_attn = ATTENTION_CLASSES[config._attn_implementation]( (remote custom code, trust_remote_code=True).
• Failing log line 4414: DeepseekV2ForCausalLM has no SGLang implementation, falling back to Transformers implementation.
• Failing log line 4382: Ignore import error when loading sglang.srt.models.deepseek_v2: No module named 'cutlass'. The failing run has 120 No module named 'cutlass' import errors; the last passing run (job 85451536779) has 0.
• In the passing run the same model loaded natively: Load weight end … type=DeepseekV2ForCausalLM, quant=fp8 (passing-log line 4369). No Transformers fallback occurred there.
• sglang code sets the fallback attention impl at python/sglang/srt/models/transformers.py:614 (self.text_config._attn_implementation = "sglang"), then AutoModel.from_config(...) at line 617; the remote DeepSeek-V2 modeling file does not register a sglang key in its ATTENTION_CLASSES.
• Import chain that breaks on AMD (all top-level, unguarded):
  models/deepseek_v2.py:61 → layers/attention/dsa/dsa_indexer.py:11 (from sglang.jit_kernel.dsa import …) → jit_kernel/dsa/__init__.py:1 (from .cutedsl_paged_mqa_logits import …) → jit_kernel/dsa/cutedsl_paged_mqa_logits.py:15 import cutlass.
• Server exited with code -9; test/registered/mla/test_mla_fp8.py:42 popen_launch_server raised (test_utils.py:1029), Ran 0 tests in 36.247s.

Hypothesised Causes (with confidence)

• [HIGH] Commit fefc1743 ("Cute-DSL FP8 MQA logits", PR #25220) added an unguarded top-level import cutlass in python/sglang/jit_kernel/dsa/cutedsl_paged_mqa_logits.py:15, which is re-exported by jit_kernel/dsa/__init__.py and imported transitively by models/deepseek_v2.py. On AMD/ROCm the NVIDIA cutlass DSL package is not installed, so importing sglang.srt.models.deepseek_v2 raises ModuleNotFoundError; the model registry silently drops DeepseekV2ForCausalLM, forcing the Transformers-backend fallback that ends in KeyError: 'sglang'.
  • Supporting: commit is the only one in the window touching this import chain; it adds import cutlass (git show line 442, 2242); passing run (before commit) had 0 cutlass errors and loaded the model natively, failing run (after commit) has 120 cutlass errors including for deepseek_v2; the full import chain from deepseek_v2.py to the import cutlass line is verified unguarded in the current tree.
  • Disconfirming: (none found) — the A/B between the last passing run and this run is clean; only the cutlass-import delta explains both the new import error and the fallback.
  • How to verify: on an AMD MI325 container, python -c "import sglang.srt.models.deepseek_v2" — expect ModuleNotFoundError: No module named 'cutlass'. Confirm removing/guarding the import cutlass in jit_kernel/dsa/cutedsl_paged_mqa_logits.py restores native load, then re-run test/registered/mla/test_mla_fp8.py.
• [LOW] Secondary contributor: the Transformers-backend fallback path (transformers.py:614) is not compatible with trust_remote_code models whose custom ATTENTION_CLASSES lacks a sglang entry. This only surfaces because of the fallback above; it is not the trigger.
  • Disconfirming: this path is unchanged in the window and worked before because the native impl was used instead.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for cutlass import deepseek, cutedsl import error, No module named cutlass (via GET /search/issues?q=repo:sgl-project/sglang+is:pr+is:open+<kw>); results are unrelated DSv4/SM120/MoE PRs, none addressing the unguarded import cutlass in jit_kernel/dsa.

Suggested Triage Steps

• Reproduce cheaply: python -c "import sglang.srt.models.deepseek_v2" inside the AMD CI image to confirm the cutlass import failure and its propagation.
• Inspect fefc1743 / PR #25220: the import cutlass at jit_kernel/dsa/cutedsl_paged_mqa_logits.py:15 (and its re-export in jit_kernel/dsa/__init__.py) should be lazy/guarded (import inside the cutedsl runner, or wrapped in try/except) so AMD/non-Blackwell environments don't fail the whole deepseek_v2 import.
• If confirmed, the fix belongs on the sglang side (guard the cutlass import), not aiter.

Status

• Persistence: New — first failing completed run 2026-07-07 07:15 UTC; last passing 2026-07-06 18:40 UTC.
• Scope: test/registered/mla/test_mla_fp8.py on AMD. Likely broader — the unguarded cutlass import breaks native import of any DeepSeek-V2/V4-family model (and 120 modules total show the cutlass import error) on ROCm, so other AMD DeepSeek tests that rely on the native path may also regress.
• Blocked work: MLA FP8 accuracy signal (MGSM) on AMD in pr-test-amd stage-b.
• In-flight fix: ❌ none found — needs a new PR to guard the import cutlass in jit_kernel/dsa.

stage-b-test-1-gpu-small-amd (linux-mi325-1gpu-sglang, 6) — failed step(s): Run test

Commit Info

• sglang: 99db3b0f
• aiter: 9127c94a (v0.1.17.dev110+g9127c94a1, source: dev/patched image — AITER_COMMIT_OVERRIDE empty, rebuild skipped)

Failed Tests

Test File	Test Function	Error	Log
test/registered/hicache/test_hicache_variants.py	TestHiCacheMLA.setUpClass	Exception: Server process exited with code -9 (server crash: KeyError: 'sglang')	Log

File-level result: 9/10 passed, only test_hicache_variants.py failed (exit code 1). Within that file, TestHiCacheMLA errored in setUpClass; TestHiCacheStandard passed.

Failure Summary

The MLA HiCache server launch fails during model init. The MLA test model is deepseek-ai/DeepSeek-Coder-V2-Lite-Instruct (DeepseekV2ForCausalLM). Although sglang has a native DeepseekV2ForCausalLM implementation, the log shows it "has no SGLang implementation, falling back to Transformers implementation" — because the deepseek_v2 module failed to import. The Transformers fallback then hits KeyError: 'sglang' in the model's remote code, and the scheduler child crashes (exit -9).

Regression Status

New regression.

Recent history of test_hicache_variants.py (TestHiCacheMLA) in workflow pr-test-amd.yml, stage-b-test-1-gpu-small-amd (branch main, event schedule, completed runs only; shard assignment varies so history is tracked at the test-file level):

Date (UTC)	Run	Head SHA	Test File Status	Detail
2026-07-07 07:15	28848583109	99db3b0f	❌ Failed	KeyError: 'sglang' → server exit -9
2026-07-07 00:44	28833386129	6c1fb8a9	✅ Passed	all stage-b jobs green
2026-07-06 18:40	28814834996	52c6e27e	✅ Passed	10/10 passed (shard 6)

First observed failure: 2026-07-07 07:15 UTC. Last known pass: 2026-07-07 00:44 UTC (head 6c1fb8a9). Regression window: 6c1fb8a9..99db3b0f (26 commits).

Failure Cluster

ModuleNotFoundError: No module named 'cutlass' breaks model registration → Transformers fallback KeyError: 'sglang'

Facts (verified observations)

• Server launch command (log line ~4470): sglang serve --model-path deepseek-ai/DeepSeek-Coder-V2-Lite-Instruct --trust-remote-code --enable-hierarchical-cache --hicache-size 200 ...
• Log line 4387: Ignore import error when loading sglang.srt.models.deepseek_v2: No module named 'cutlass' (FACT). ~30 other model modules fail identically in this run (deepseek_v4, ernie4, glm4_moe, kimi_*, bailing_moe_*, longcat_*, pixtral, etc.).
• The prior passing shard-6 log (job 85451536783) contains zero No module named 'cutlass' errors (FACT) — the import failures are new to this window.
• Stack trace: transformers.py:617 AutoModel.from_config → remote modeling_deepseek.py:1203 → self.self_attn = ATTENTION_CLASSES[config._attn_implementation] → KeyError: 'sglang' (FACT). The Transformers backend sets config._attn_implementation = "sglang" and registers it via ALL_ATTENTION_FUNCTIONS["sglang"] (transformers.py:327), but the DeepSeek remote code uses its own ATTENTION_CLASSES dict that has no 'sglang' key.
• Registry mechanism (model_loader/utils.py, models/registry.py): a failed module import is swallowed (import_model_classes, non-strict), so DeepseekV2ForCausalLM drops out of get_supported_archs(); is_native_supported becomes False → resolve_transformers_arch → the "no SGLang implementation" fallback (FACT, log line 4420–4424).
• Import chain that pulls in cutlass at module-load time (verified by reading source):
  models/deepseek_v2.py:61 → from ...attention.dsa.dsa_indexer import Indexer → dsa_indexer.py:11 → from sglang.jit_kernel.dsa import (...) → jit_kernel/dsa/__init__.py:1 → from .cutedsl_paged_mqa_logits import ... → cutedsl_paged_mqa_logits.py:20 + top-level import cutlass; also cutedsl_fp8_paged_mqa_logits.py top-level import cutlass.

Hypothesised Causes

• [HIGH] #25220 (fefc1743 "Cute-DSL FP8 MQA logits") introduced an unguarded top-level import cutlass into a module chain that deepseek_v2 imports at load time. cutlass (the NVIDIA CUTLASS/Cute-DSL Python package) is not installed in the ROCm CI image, so the import fails and the model is dropped from the registry, forcing the broken Transformers fallback.
  • Supporting: fefc1743 is a git diff new file for jit_kernel/dsa/__init__.py whose line 1 top-level-imports cutedsl_paged_mqa_logits (which does import cutlass); the same commit added from sglang.jit_kernel.dsa import (cutedsl_paged_mqa_logits, ...) to dsa_indexer.py (verified via git show fefc1743). Commit is in the regression window. The prior passing run had zero cutlass import errors. deepseek_v2.py:61 imports Indexer from dsa_indexer at top level.
  • Disconfirming: (none found) — the import chain, the timing, and the diff all converge; the KeyError is a direct downstream consequence of the dropped registration.
  • How to verify: on an AMD MI325 ROCm container, python -c "import sglang.srt.models.deepseek_v2" should raise ModuleNotFoundError: No module named 'cutlass' on 99db3b0f and succeed on 6c1fb8a9. Alternatively guard the import cutlass behind a CUDA/availability check and re-run test_hicache_variants.py.
• [LOW] #30117 (e85ef548 "Support Cutedsl BF16 GEMM JIT kernel") also added a top-level import cutlass (in cutedsl_bf16_gemm.py) and touched deepseek_v2.py.
  • Supporting: in the same window; also references cutlass.
  • Disconfirming: its cutedsl_bf16_gemm import and the new deepseek_v2 import (unquant.get_bf16_gemm_backend) are lazy (inside functions / lazy-loaded), so they do not run at module-load time and cannot cause the registry import failure. The observed error module (dsa) belongs to fefc1743, not this PR.

In-flight Fix Check

• In-flight fix: ⚠️ [#30413](https://github.com/sgl-project/sglang/pull/30413) (open, created 2026-07-07) "Fix additional kernels being triggered in CuteDSL MQA logits path" — explicitly a follow-up to #25220 and touches dsa_indexer.py / dsa/paged_mqa_logits.py, but it is a functional/perf fix and does not add a ROCm/CUDA guard around the top-level import cutlass in jit_kernel/dsa/__init__.py. Unlikely to resolve this AMD import failure — needs confirmation.
• In-flight fix: ❌ none found that specifically guards the cutlass import for ROCm — likely needs a new PR.

Suggested Triage Steps

• Reproduce in isolation on ROCm: python -c "import sglang.srt.models.deepseek_v2" on 99db3b0f — expect No module named 'cutlass'.
• Confirm the culprit: check out fefc1743 vs its parent and diff module-load behavior of sglang.jit_kernel.dsa.
• Fix direction (hypothetical): make the import cutlass in jit_kernel/dsa/__init__.py → cutedsl_paged_mqa_logits.py / cutedsl_fp8_paged_mqa_logits.py lazy or platform-guarded (skip on is_hip()), mirroring how cutedsl_bf16_gemm is imported lazily; then re-run test_hicache_variants.py on AMD MI325.
• Verify scope of the registry-drop: since ~30 models silently fall back to Transformers on AMD, a fix here likely un-breaks many AMD model paths, not just this test.
• Confirm with #30413 author whether that PR (or a companion) should include the ROCm import guard, to avoid a duplicate fix.

Status

• Persistence: First failing completed run 2026-07-07 07:15 UTC (99db3b0f); passed 6h earlier at 6c1fb8a9.
• Scope: Directly fails TestHiCacheMLA in test_hicache_variants.py (MLA/DeepSeek path). Root cause (import cutlass failure) silently degrades ~30 model modules on the ROCm image to the Transformers backend, so blast radius likely extends to other AMD MLA/DeepSeek/MoE tests.
• In-flight fix: ⚠️ #30413 related but not confirmed to add the ROCm import guard; no dedicated guard PR found.

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-07-08 18:54 UTC)