[CI Monitor] Daily Report - 2026-06-19

[amd-bot]

Daily Cross-Workflow Summary — 2026-06-19

Snapshot: 2026-06-19 20:36 UTC · Only completed runs/jobs counted · Auto-updated every 30 min

TL;DR

🔴 RED · ~14 active clusters · 1 🆕 (R193 ROCm 7.2 diffusion Inductor decompose_triton assert) · R191 evolved (image-init SIGSEGV cleared on the new …-20260619 image but a HANG appeared at the same torch.ones init site) · 2 newly-ID'd escalations (R194 Kimi IndexError, R195 Mamba extra_buffer) · ~11 carrying over · 3 mergeable in-flight fixes
👉 Today's ask: Triage 🆕 R193 — chase in-flight #25969 (disable torch.compile for ROCm diffusion layernorm), which targets the exact assert that broke 4/6 ROCm 7.2 diffusion tests today. Secondary: R191 — the Jun-18→Jun-19 image bump removed the native SIGSEGV (supporting yesterday's bad-image hypothesis) but a first-device-op hang now appears at the same init_world_group/torch.ones site — re-run today's SHA on …-20260618 to confirm the bug rode the image. Quick wins: chase 2 review-ready fixes — #27757 (R2 Mistral GSM8K) and #27141 (R195 Mamba extra_buffer). ⚠️ amd-aiter-scout (Thu cron, 27788558636, now completed on the stale …-20260618 image) flags 2 SIGSEGVs as "aiter-caused" — contested: same R191 init-SIGSEGV signature hit the non-override sister nightly on the same image (see R191 disconfirming evidence).

Workflow status

Workflow	Runs (today)	✅	❌	Trend (completed real failures)	Δ vs yesterday
nightly-test-amd	Jun-19 27841462093 (completed)	0	5	6·6·7·6·5	🟢 −1, R191 SIGSEGV gone
nightly-test-amd-rocm720	Jun-19 27841393725 (completed, MI35x 8-GPU jobs not in matrix)	0	5	10·10·~10·5*	🟢 SIGSEGV gone (*partial: fewer MI35x shards)
release-docker-amd-nightly	✅ no failures	✅	0	0·0·0	0
release-docker-amd-rocm720-nightly	✅ no failures	✅	0	0·0·0	0
amd-aiter-scout	1 (Thu cron) 27788558636 completed on …-20260618	0	17 jobs	—	was IN-FLIGHT yesterday; now resolved
pr-test-amd	Jun-18/19 runs 27842786491 + earlier	0	R179/R187/R192/R185 + R191-hang	~6 real	same set
pr-test-amd-rocm720	Jun-19 27841438145 (overall cancelled)	0	🆕R193 + R181 cascade + concurrency cancels	~5 real	🔴 +R193

Notes: (1) Both Jun-19 nightlies are completed, but the rocm720 MI35x 8-GPU shards (DSV3.2/R155, Qwen3-235B-MXFP4/R19, R1-HiCache/R120, MiniMax/R188) are not in this run's matrix — their data carries over from the Jun-18 run 27780200720. (2) pr-test-amd-rocm720 remains dominated by cancel-in-progress concurrency supersessions (overlapping 6-h crons sharing one github.ref group) — orchestration noise. Run 27841438145 cancelled overall (24 success · 22 cancelled · 2 failure), fast-failed by the R181 mori/libabsl build break.

Failure clusters (active, deduplicated across all workflows)

R193 · 🆕 · torch._inductor AssertionError in decompose_triton_kernel_wrapper_functional — ROCm 7.2 diffusion (torch.compile + custom-triton)

• Status: first seen 2026-06-19 in run 27841438145; 4 of 6 diffusion tests across 5 multimodal-gen shards (1-GPU + 2-GPU). One variant (wan2_2_i2v_a14b_2gpu) manifests as a GPU memory-access-fault coredump; an ImportError: cannot import name 'specialize_impl' from 'triton.runtime.jit' (custom triton at /sgl-workspace/triton-custom) is logged just before each.
• Top hypothesis: [MEDIUM] torch/triton version skew in the …-rocm720-mi30x-20260619 image: the post-grad inductor pass decompose_triton_kernel_wrapper_functional asserts, reached straight from a @torch.compile layernorm forward_native. Disconfirming: the missing specialize_impl symbol is logged as a non-fatal warning, so the assert may be a distinct inductor bug, not the import skew; exact internal inductor stack is masked behind the bare AssertionError.
• In-flight fix: ✅ #25969 "[AMD] Disable torch.compile for ROCm diffusion layernorm" (open since 2026-05-21) — targets this exact layernorm.py forward_native compile path; chase reviewers. Suggested triage: re-run one wan2.x case with TORCHDYNAMO_VERBOSE=1; verify #25969 covers the denoise path; inspect triton-custom specialize_impl export vs installed torch.

Workflow	Job (shard)	Test File	Test Function	Error	Log
pr-test-amd-rocm720	multimodal-gen-2-gpu (1)	multimodal_gen/test/server/test_server_2_gpu.py	test_diffusion_generation[wan2_1_t2v_14b_2gpu]	InductorError: AssertionError @ decompose_triton_kernel_wrapper_functional	link
pr-test-amd-rocm720	multimodal-gen-2-gpu (0)	test_server_2_gpu.py	test_diffusion_generation[wan2_2_i2v_a14b_2gpu]	Memory access fault coredump (inductor/triton path)	link
pr-test-amd-rocm720	multimodal-gen-1-gpu (0)	test_server_1_gpu.py	test_diffusion_generation (Qwen-Image)	InductorError: AssertionError (same pass)	link
pr-test-amd-rocm720	multimodal-gen-1-gpu (1)	test_server_1_gpu.py	test_diffusion_generation	InductorError: AssertionError (same pass)	link

R191 · Image-init GPU-runtime instability at torch.ones/init_world_group — EVOLVED: SIGSEGV (…-20260618) → HANG (…-20260619)

• Status: cross-workflow, DAY 2. On the Jun-18 image the symptom was a native SIGSEGV (-11) at parallel_state.py:301 (torch.ones, first device op); on the Jun-19 image the SIGSEGV is gone and a first-device-op hang (stuck in libhsa-runtime64.so) appears at the same call site. aiter byte-identical (v0.1.14rc1.dev240+g7d604afe5) across all.
• Top hypothesis: [MEDIUM→HIGH] the daily ROCm image GPU runtime is unstable on the very first device op. Elevated: the symptom changed with the image tag (SIGSEGV→hang on the 06-18→06-19 bump) while sglang code at the crash site was unmodified — strong evidence the bug rides the image, not application code. Disconfirming: image contents still not diffed; first-device-op faults can be runner-transient; not every shard trips it (flaky draw).
• In-flight fix: ❌ none (init_world_group/torch.ones searches → 0). Suggested triage: re-run today's SHA on …-20260618 (expect SIGSEGV) vs …-20260619 (expect hang); diff torch/ROCm/RCCL versions across the two tags.

Workflow	Job (shard)	Test File	Test Function	Error	Log
nightly-test-amd	nightly-accuracy-2-gpu-vlm	test_vlms_mmmu_eval_amd.py	server-start	HANG in init_world_group→torch.ones (libhsa) — Jun-19 image	link
pr-test-amd	stage-b-1gpu-small (9)	server warmup	setUpClass	HANG in torch.ones world-group init — Jun-19	link
nightly-test-amd	nightly-8-gpu-glm51	test_glm51_eval_amd.py	test_glm51_accuracy	SIGSEGV (-11) — Jun-18 image	link
nightly-test-amd-rocm720	nightly-8-gpu-qwen3-235b-rocm720	test_qwen3_instruct.py	setUpClass	SIGSEGV (-11) fill/ones — Jun-18	link
amd-aiter-scout	v31-rocm720 / v32-rocm720	test_deepseek_v3{1,2}_eval_amd.py	accuracy	SIGSEGV torch.ones init — Jun-18 image (per-job: "aiter-caused"; CONTESTED)	link

R191 / scout contradiction (important): the scout per-job analyses classify the v31/v32-rocm720 SIGSEGVs as aiter-caused because their Jun-17 sister baseline (good image) passed. Disconfirming: the proper same-image sister (nightly-test-amd-rocm720 Jun-18 27780200720) also SIGSEGV'd at the identical torch.ones init site (qwen3-235b) on the …-20260618 image, with default aiter. This is the classic scout two-variable trap — the SIGSEGV is more likely R191 image-init instability than aiter. Treat the "aiter-caused" verdict as [LOW].

R181 · mori build — corrupt libabsl_time.so.20210324 (invalid ELF header) on image …-20260619

• Status: carry-over (was on …-20260618, now …-20260619); today it fast-failed the entire pr-test-amd-rocm720 run 27841438145 (root: stage-b shard 3, job 82401214071), cascading 22 cancellations. In-flight: ❌ none (infra). Triage: readelf -h /lib/x86_64-linux-gnu/libabsl_time.so.20210324 in the image; rebuild/re-pull the layer or pin a known-good tag.

R2 · Mistral/Mixtral GSM8K below threshold (chat-eval) — nightly-amd + rocm720 + scout

• Status: never-passed in window; nightly-amd 82401239597 + rocm720 82401048935 + scout (pre-existing). In-flight fix: ✅ #27757 (open since 06-10, updated 06-19, not draft) — review-ready, chase merge.

R1 · VLM MMMU near-random accuracy — nightly-amd + rocm720 + scout (pre-existing, never-passed)

• Status: 5 models pinned near random; nightly-amd 82401239671 + rocm720 82401049653 + scout. In-flight fix: ❌ none (#26178 stale, thresholds only). Triage: A/B --attention-backend triton vs aiter; compare to NVIDIA nightly.

R195 · 🔼 Mamba radix-cache extra_buffer needs CUDA/MUSA/NPU (FLA) rejected on ROCm (Qwen3.5) — newly ID'd

• Status: new regression since ~Jun-16; now on both nightly-amd qwen35 82401239835 and rocm720 qwen35 82401049106. In-flight fix: ✅ #27141 "[ROCm] Support radix-cache with mamba-extra-buffer for Qwen3.5" (open since 06-03, updated 06-19) — review-ready, chase merge.

R194 · 🔼 Kimi-K2.6 MoE _load_w2 narrow IndexError (TP8 weight load) — newly ID'd, escalating

• Status: new regression Jun-16/17; now on both nightly-amd kimi-k26 82401239882 and rocm720 kimi-k26 82401049069. Alternates IndexError ↔ 3600s timeout depending on enable_multithread_load. In-flight fix: ❌ none. Triage: inspect the narrow bounds in Kimi MoE _load_w2 for asymmetric/compressed shapes on TP8.

R196 · VLM encoder-DP HIP IPC "invalid device pointer" / getIpcDevPtr watchdog hang (Qwen3-VL-32B) — flaky-on-random-model

• Status: deterministic crash, flaky test surface (random model draw); nightly-amd 4-gpu 82401239874 + rocm720 4-gpu 82401048888 + scout. In-flight fix: ⚠️ candidates #27434, #27694 (WIP, no confirmed root-cause fix).

Carrying-over / known-stable clusters (no new action today) · click to expand

ID	Cluster	Where today	In-flight fix
R192	FLUX.2 modelopt-FP8 TP2 HIPBLAS_STATUS_NOT_SUPPORTED (never-passed, new test #28546)	pr-test-amd multimodal-gen-2gpu 82405358804; rocm720 82401213875	❌ none
R187	DeepSeek-R1-MXFP4 EAGLE/MTP accept-length ~1.44 < 2.04 (MI35x 8-GPU, never-passed)	pr-test-amd stage-c-mi35x 82209394954	⚠️ #28378
R179	aiter fused all-reduce+RMSNorm residual 1-bf16-ULP mismatch	pr-test-amd stage-c-8gpu 82209394997	✅ #27815 / ⚠️ #27650
R185	Gated HF repo 403 ideogram-ai/ideogram-4-fp8 (diffusion startup)	pr-test-amd multimodal-gen; rocm720 multimodal-gen	⚠️ #28225 (draft)
R177	HF weight-download stall / 429 / Xet xet_get / filelock → launch timeout	rocm720 Jun-18 (kimi/glm51/qwen35/grok1) + scout mxfp4-tp2	⚠️ #23400 (partial)
R155	DeepSeek-V3.2 MI35x GSM8K below threshold (0.885 Jun-17; Jun-18 hang)	rocm720 MI35x (Jun-18 82202799662; not in Jun-19 matrix)	⚠️ #25559 (WIP)
R120	GPU mem-fault DeepSeek-R1 fp8 HiCache prefill (MI35x, never-passed)	rocm720 Jun-18 + scout 82231478189	❌ none
R19	Qwen3-235B-MXFP4 HIP capturing-stream / IPC abort (MI35x, never-passed)	rocm720 Jun-18 82202799487	✅ #23581 candidate
—	DeepSeek-V4-flash HIP OOM compress-state KV (unified_kv_triton)	rocm720 Jun-18 82202798942	❌ none
—	VLM perf throughput below hardcoded floor (test config too high)	rocm720 stage-b-small (7) 82401214059	❌ none (test config)
—	MoE topk-softmax exact-value mismatch (unseeded input)	pr-test-amd sgl-kernel 82357272099	❌ none (flaky)
—	Triton-backend offline throughput below AMD floor (3179<3500)	rocm720 stage-b-large (0)	❌ none

Infrastructure / orchestration noise (not test failures) · click to expand

• pr-test-amd-rocm720 concurrency cancellations: ~40+ "failed" rows in run 27780231199 and others are cancel-in-progress supersessions (overlapping 6-h crons sharing one github.ref group; long dsv4-* jobs outlive the cron). No test signal.
• MI35x 8-GPU runner starvation: dsv4-* and MI35x nightly jobs cancelled while queued (0 steps). Capacity, not code.
• amd-aiter-scout extra/gate cascade: the call-pr-test-amd-extra subtree gate + sibling jobs cancelled at queue time (concurrency-group collision, scout-specific); jit-kernel-unit-test-amd failed because the amd/aiter-ci scout branch lacks #27947 (fixed on main) — not a real regression.
• Transient HF Hub network drops / runner loss: scout 1-gpu-unit, pr-test-amd multimodal-gen runner-loss shards.

How this report is generated

• Only status == "completed" runs/jobs counted in trends. Jun-19 nightlies are completed; rocm720 MI35x 8-GPU shards are out of this run's matrix, so R155/R19/R120/R188 carry over from Jun-18 27780200720. Both release-docker workflows reported zero failures.
• 🆕 NEW today: R193 (ROCm 7.2 diffusion Inductor assert, [MEDIUM]). Evolved: R191 (SIGSEGV→hang across the 06-18→06-19 image bump). Newly-ID'd escalations: R194 (Kimi IndexError), R195 (Mamba extra_buffer), R196 (encoder-DP IPC).
• amd-aiter-scout (Thu cron) completed on the stale …-20260618 image; its 2 "aiter-caused" SIGSEGV verdicts are downgraded to [LOW] — same R191 init-SIGSEGV signature reproduced in the non-override sister nightly on the same image.
• Confidence: FACT/HIGH/MEDIUM/LOW/SPECULATION. Bot does NOT assign Priority — engineers decide from cluster size + persistence + fix availability.

---

Generated by amd-bot · last updated 2026-06-19 20:36 UTC

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-06-19 20:36 UTC)

---

CI Monitor — 2026-06-19

Repo: sgl-project/sglang

Monitored Workflows:

• nightly-test-amd.yml
• nightly-test-amd-rocm720.yml
• release-docker-amd-nightly.yml
• release-docker-amd-rocm720-nightly.yml
• amd-aiter-scout.yml
• pr-test-amd.yml
• pr-test-amd-rocm720.yml

Per-workflow failure reports are appended as comments below; the Daily Cross-Workflow Summary is rendered above this section.

[amd-bot]

nightly-test-amd.yml — 10 failure(s)

Run started (UTC): 2026-06-18 19:07 UTC
Last scanned (UTC): 2026-06-19 20:32 UTC
Commits: sglang ca88b7f

Summary

Counts: 10 failures · 7 clusters · 3 carrying over (≥3 days) · 2 new today

#	Cluster	Job	Test File	Test Function	Status	Hypothesis (confidence)
1	First-device-op fault at distributed init	nightly-8-gpu-glm51	test/registered/amd/accuracy/mi30x/test_glm51_eval_amd.py	test_glm51_accuracy	Intermittent (Jun 18 + Jun 11; passed Jun 12–17)	[MEDIUM] GPU-runner/HIP fault at first device op; crash site unchanged in window
2	First-device-op fault at distributed init	nightly-8-gpu-kimi-k26	test/registered/amd/accuracy/mi30x/test_kimi_k26_eval_amd.py	TestKimiK26EvalAMD.setUpClass	Flaky (Jun 18 segfault; passed Jun 14)	[LOW] infra flake; no hypothesis ≥ MEDIUM
3	First-device-op fault at distributed init	nightly-accuracy-2-gpu-vlm	test/registered/amd/accuracy/mi30x/test_vlms_mmmu_eval_amd.py	test_mmmu_vlm_models	New symptom today (server hang)	[MEDIUM] wedged GPU/HIP at torch.ones first op
4	GSM8K Mistral/Mixtral accuracy < threshold	nightly-accuracy-2-gpu	test/registered/amd/accuracy/mi30x/test_gsm8k_eval_amd.py	test_gsm8k_eval_amd (Mistral/Mixtral)	~13 days persistent (since Jun 7)	[MEDIUM] chat-eval extraction — fix #27757
5	GSM8K Mistral/Mixtral accuracy < threshold	nightly-accuracy-2-gpu	test/registered/amd/accuracy/mi30x/test_gsm8k_eval_amd.py	test_gsm8k_all_models (Mistral/Mixtral)	~13 days persistent (since Jun 7)	[MEDIUM] chat-eval extraction — fix #27757
6	VLM MMMU accuracy < threshold	nightly-accuracy-2-gpu-vlm	test/registered/amd/accuracy/mi30x/test_vlms_mmmu_eval_amd.py	test_mmmu_vlm_models	Never-passed in window (since ≥Jun 11)	[MEDIUM] thresholds set above achievable AMD scores
7	Kimi MoE w2 weight-load narrow IndexError	nightly-8-gpu-kimi-k26	test/registered/amd/accuracy/mi30x/test_kimi_k26_eval_amd.py	TestKimiK26EvalAMD.setUpClass	Recurring since Jun 17 (last pass Jun 16)	[MEDIUM] #27690 f06e2d3d skips MoE transpose
8	DP-encoder CUDA-IPC getIpcDevPtr hang (exit -9)	nightly-4-gpu	test/registered/vlm/test_encoder_dp.py	test_vlm_mmmu_benchmark	New today (Jun 13 same model passed slow)	[MEDIUM] IPC reconstruct hang/timeout cliff on ROCm
9	HuggingFace xet_get weight-download stall	nightly-8-gpu-mi35x-deepseek-r1-hicache	test/registered/amd/test_deepseek_r1_hicache_mi35x.py	N/A (file timeout)	New symptom today (file Never-passed)	[MEDIUM] infra/network HF xet download stall
10	Qwen3.5 mamba extra_buffer rejected on ROCm	nightly-8-gpu-qwen35	test/registered/amd/accuracy/mi30x/test_qwen35_eval_amd.py	TestQwen35EvalAMD.setUpClass	New regression Jun 19 (last pass Jun 15)	[HIGH] #28151 66a7fd5c flips auto→extra_buffer

Cluster: First-device-op fault at distributed init (parallel_state.py:301)

• Affected: nightly-8-gpu-glm51 (test_glm51_eval_amd.py, test_glm51_accuracy); nightly-8-gpu-kimi-k26 (test_kimi_k26_eval_amd.py, setUpClass); nightly-accuracy-2-gpu-vlm (test_vlms_mmmu_eval_amd.py, test_mmmu_vlm_models).
• Shared facts: all three bottom out at parallel_state.py:301 (torch.ones(..., device=cuda) in init_world_group) → fill_kernel_cuda → libamdhip64/libhsa-runtime64, i.e. the first device tensor op. parallel_state.py:301 is unchanged across the regression windows. Symptom splits: glm51 + kimi = SIGSEGV (-11); vlm = indefinite hang/TimeoutError.
• Top hypothesis: [MEDIUM] MI30x runner GPU/HIP instability at distributed init. Disconfirming: only intermittent; no dmesg/XGMI fault captured to confirm hardware; a distributed-init race cannot be fully excluded.
• In-flight fix: ❌ none found.
• Suggested triage: re-run each job at head; if green, infra confirmed. Capture dmesg -T/rocm-smi on the MI30x node at init time; consider an MI30x runner-pool health check (repeated faults on same node family).

Cluster: GSM8K Mistral/Mixtral accuracy < threshold

• Affected: nightly-accuracy-2-gpu and nightly-accuracy-2-gpu — both test_gsm8k_eval_amd.py, Mistral/Mixtral (base + FP8).
• Shared facts: only the Mistral/Mixtral family fails; all other architectures pass. Scores stable ±0.02 across ~13 days (Mistral-7B ~0.37, Mixtral ~0.29, Mixtral-FP8 ~0.19) — deterministic, not flaky.
• Top hypothesis: [MEDIUM] chat-eval prompt + last-number extraction misfire for Mistral chat models. Disconfirming: Mixtral-FP8 at 0.19 vs 0.69 is a large gap that pure extraction may not fully explain; a real AMD inference component cannot be excluded.
• In-flight fix: ✅ #27757 "Fix Mistral GSM8K chat eval" (open since 2026-06-10) — chase reviewers, do NOT duplicate.
• Suggested triage: re-run with #27757; if scores still fail, inspect /tmp/gsm8k_mistralai_*.html transcripts to separate mis-extraction from genuine inference error.

Cluster: VLM MMMU accuracy < threshold

• Affected: nightly-accuracy-2-gpu-vlm (test_vlms_mmmu_eval_amd.py, test_mmmu_vlm_models). (The Jun 19 vlm job hit the init-hang cluster first, masking this recurring accuracy issue.)
• Shared facts: 4 of 11 VLMs (Qwen2.5-VL, Qwen3-VL-30B, MiniCPM-o, deepseek-vl2) stably ~0.26–0.28 near random baseline; deepseek-vl2 = 0.2802 every run. No green run in ~12 days.
• Top hypothesis: [MEDIUM] thresholds calibrated above achievable AMD scores. Disconfirming: near-random scores on 4 architectures could be a genuine ROCm VLM accuracy bug; lowering thresholds would mask it.
• In-flight fix: ❌ none found.
• Suggested triage: compare AMD vs NVIDIA MMMU baseline for the same models; inspect /tmp/mmmu_*.json for degenerate vs mis-parsed answers.

Cluster: Kimi MoE w2 weight-load narrow IndexError

• Affected: nightly-8-gpu-kimi-k26 (test_kimi_k26_eval_amd.py, setUpClass).
• Shared facts: IndexError: start out of range ... got 448 at fused_moe_triton/layer.py:566 on TP7 (448 = 64×7); shard dim left at 64 instead of 512. First seen Jun 17, last pass Jun 16; layer.py is the only weight-load file changed in that window.
• Top hypothesis: [MEDIUM] #27690 f06e2d3d "asymmetric compressed-tensors MoE" adds and "zero" not in weight_name to the transpose guard, skipping transpose for zero-point weights. Disconfirming: not confirmed Kimi-K2.6 is asymmetric WNA16 with *_zero tensors; no A/B revert done.
• In-flight fix: ❌ none found.
• Suggested triage: git revert f06e2d3d on a branch, re-run on 8×MI325; or print w2/zero tensor shapes at _load_w2.

Cluster: DP-encoder CUDA-IPC getIpcDevPtr hang (exit -9)

• Affected: nightly-4-gpu (test/registered/vlm/test_encoder_dp.py, test_vlm_mmmu_benchmark).
• Shared facts: all TP schedulers active+gil in HIP getIpcDevPtr via schedule_batch.py:365 reconstruct; only heavy Qwen VLMs (Qwen3-VL-32B, Qwen2.5-VL-72B) under --mm-enable-dp-encoder. Same 72B model passed (slow, 1200s) Jun 13.
• Top hypothesis: [MEDIUM] CUDA-IPC multimodal reconstruct hangs/exceeds 300s watchdog on ROCm. Disconfirming: Jun 13 pass of same model → timeout cliff, not a hard deadlock.
• In-flight fix: ⚠️ candidates #27434 (SGLANG_DISABLE_SHM_MM workaround), #27694 (WIP IPC hardening); ❌ none confirmed.
• Suggested triage: re-run pinned to Qwen2.5-VL-72B several times; instrument reconstruct() wall-time vs the 300s watchdog.

Cluster: HuggingFace xet_get weight-download stall

• Affected: nightly-8-gpu-mi35x-deepseek-r1-hicache (test_deepseek_r1_hicache_mi35x.py, file timeout).
• Shared facts: Load weight end never logged; py-spy pins MainThread in xet_get/snapshot_download for ~55 min; 99 soft watchdog timeouts → 3600s timeout. Same runner also failed a local docker-registry pull (context deadline exceeded). Prior days this file failed differently (GPU memory access fault during CUDA graph capture).
• Top hypothesis: [MEDIUM] infra/network — HF xet transfer stall on runner linux-mi35x-gpu-8-zf775-runner-lzmrv. Disconfirming: prior runs fetched the model, so transient; harness "not a network issue" label is exit-code heuristic only.
• In-flight fix: ❌ none found.
• Suggested triage: re-run; pre-stage DeepSeek-R1-0528 in /sgl-data/hf-cache/hub; add a fail-fast download timeout. Separately, the recurring CUDA-graph-capture memory-access fault is the real product blocker once download is healthy.

Cluster: Qwen3.5 mamba extra_buffer rejected on ROCm

• Affected: nightly-8-gpu-qwen35 (test_qwen35_eval_amd.py, setUpClass).
• Shared facts: AssertionError: extra_buffer needs CUDA/MUSA/NPU (FLA). at server_args.py:2828; is_cuda() is False on ROCm (torch.version.cuda is None). New regression: pass Jun 15, fail Jun 19.
• Top hypothesis: [HIGH] #28151 66a7fd5c (merged Jun 18) makes auto select extra_buffer (CUDA/MUSA/NPU-gated) instead of falling back; git log -S confirms the assertion string was introduced by this commit, and the stack lands in the new functions. Disconfirming: device gating predates the refactor, but Jun 14/15 green runs show pre-refactor auto did not pick extra_buffer on ROCm.
• In-flight fix: ✅ #27141 "[ROCm] Support radix-cache with mamba-extra-buffer for Qwen3.5" (open since 2026-06-03) — chase reviewers.
• Suggested triage: A/B exact command on 33719cfb31 vs 66a7fd5c; interim, re-run with --mamba-radix-cache-strategy no_buffer; merge/verify #27141.

Suggested triage order

Rank	Triage step	Targets	Why this first
1	Verify/merge in-flight #27141; A/B 66a7fd5c	row 10 (Qwen3.5 mamba)	[HIGH]-confidence fresh regression with code-level evidence + existing fix
2	Verify/merge in-flight #27757, re-run GSM8K	rows 4–5 (Mistral/Mixtral)	Largest persistent cluster (~13 days), fix already open
3	git revert f06e2d3d on branch, re-run Kimi on 8×MI325	row 7 (Kimi MoE narrow)	Fresh regression w/ MEDIUM code-level hypothesis, no in-flight fix
4	Re-run distributed-init jobs at head; capture dmesg/rocm-smi on MI30x	rows 1–3 (init faults)	Largest cluster; cheap re-run confirms infra vs code
5	Re-run + pre-stage weights (row 9); re-run pinned heavy VLM (row 8)	rows 8–9 (IPC hang, xet stall)	Both likely infra/timeout cliffs — cheap rerun disambiguates

---

Job	Failed Steps	Started
nightly-8-gpu-glm51	Accuracy Test (8-GPU GLM-5.1 DSA)	2026-06-18T19:30
nightly-accuracy-2-gpu-vlm	Nightly Accuracy Test (2-GPU VLM MMMU)	2026-06-18T19:07
nightly-accuracy-2-gpu	Nightly Test (2-GPU)	2026-06-18T21:03
nightly-8-gpu-kimi-k26	Accuracy Test (8-GPU Kimi-K2.6)	2026-06-18T21:56
nightly-8-gpu-mi35x-deepseek-r1-hicache	Accuracy Test MI35x (8-GPU DeepSeek-R1 HiCache)	2026-06-19T16:04
nightly-4-gpu	Nightly Test (4-GPU)	2026-06-19T18:07
nightly-8-gpu-qwen35	Accuracy Test (8-GPU Qwen 3.5)	2026-06-19T18:59
nightly-accuracy-2-gpu-vlm	Nightly Accuracy Test (2-GPU VLM MMMU)	2026-06-19T18:37
nightly-accuracy-2-gpu	Nightly Test (2-GPU)	2026-06-19T18:25
nightly-8-gpu-kimi-k26	Accuracy Test (8-GPU Kimi-K2.6)	2026-06-19T19:16

Per-Job Analysis

nightly-8-gpu-glm51 — failed step(s): Accuracy Test (8-GPU GLM-5.1 DSA)

Commit Info

• sglang: f7632ef8
• aiter: N/A — no [CI-AITER-CHECK] markers in this nightly accuracy log; the container's pre-installed aiter was used (job does not run amd_ci_install_dependency.sh aiter version check).

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi30x/test_glm51_eval_amd.py	test_glm51_accuracy (TestGLM51EvalAMD)	AssertionError: Failed models: ['zai-org/GLM-5.1-FP8 (dsa)']	Log

Failure Summary

The single accuracy test test_glm51_accuracy failed because the zai-org/GLM-5.1-FP8 (dsa) server never came up: the SGLang server launch aborted with RuntimeError: Rank 1 scheduler died during initialization (exit code: -11) (SIGSEGV). Multiple TP ranks (TP1, TP2) hit Fatal Python error: Segmentation fault during distributed init, so the eval harness recorded the model as failed and asserted.

Regression Status

Flaky / intermittent. The nightly-8-gpu-glm51 job's only test file (test_glm51_eval_amd.py) passed in the previous 6 consecutive scheduled runs (Jun 12–17), then failed today. An earlier failure occurred Jun 11.

Recent history of test/registered/amd/accuracy/mi30x/test_glm51_eval_amd.py (test_glm51_accuracy) in job nightly-8-gpu-glm51 (branch main, event schedule, completed jobs only):

Date	Run	Job	Test File Status	Failed Function	Error
Jun 18	27780272530	nightly-8-gpu-glm51	❌ Failed	test_glm51_accuracy	scheduler died exit -11 (SIGSEGV)
Jun 17	27710225107	nightly-8-gpu-glm51	✅ Passed	—	—
Jun 16	27640378823	nightly-8-gpu-glm51	✅ Passed	—	—
Jun 15	27568949589	nightly-8-gpu-glm51	✅ Passed	—	—
Jun 14	27507337214	nightly-8-gpu-glm51	✅ Passed	—	—
Jun 13	27474523511	nightly-8-gpu-glm51	✅ Passed	—	—
Jun 12	27434333913	nightly-8-gpu-glm51	✅ Passed	—	—
Jun 11	27368902882	nightly-8-gpu-glm51	❌ Failed	(not verified)	(not verified)

Last known pass: Jun 17 (27710225107, head f5b041622b). First failure today: Jun 18 (27780272530, head f7632ef8). Lookback window: 8 scheduled runs (Jun 11–18).

Failure Cluster

Segmentation fault during distributed init (first device tensor alloc) on GLM-5.1-FP8 dsa/tp8.

Facts (verified observations)

• Server command (verbatim): sglang serve --model-path zai-org/GLM-5.1-FP8 --reasoning-parser glm45 --tool-call-parser glm47 --dsa-prefill-backend tilelang --dsa-decode-backend tilelang --chunked-prefill-size 131072 --mem-fraction-static 0.80 --tp 8 --device cuda (attention_backend dsa).
• Two ranks crashed identically with Fatal Python error: Segmentation fault right after Init torch distributed begin (TP1 at 19:45:29, TP2 at 19:45:37). Launcher then reported RuntimeError: Rank 1 scheduler died during initialization (exit code: -11).
• Python stack at crash (identical for both ranks): scheduler.py:432 __init__ → tp_worker.py:262 __init__ → model_runner.py:1229 init_torch_distributed → parallel_state.py:1844 init_distributed_environment → parallel_state.py:1509 init_world_group → parallel_state.py:301 __init__ (GroupCoordinator).
• Native stack at crash: torch::autograd::THPVariable_ones → at::native::ones/full → fill_out → fill_kernel_cuda → gpu_kernel_impl_nocast<FillFunctor<int>>. This corresponds exactly to parallel_state.py:301: active_ranks = torch.ones(len(ranks), dtype=torch.int32, device=self.device) — i.e. the first GPU memory op performed by each rank during process-group setup.
• parallel_state.py was not modified in the regression window f5b041622b..f7632ef8 (git log on the file returns no commits); git blame shows line 301 predates the queryable history.
• The same test+model passed 6 consecutive scheduled runs immediately before this failure.

Hypothesised Causes (with confidence)

• [MEDIUM] Infrastructure / GPU-runtime flake on the 8-GPU MI30x runner (transient HIP context / driver fault), not an sglang code regression.
  • Supporting: the crash is a SIGSEGV in native CUDA/HIP code (fill_kernel_cuda) on the very first device tensor allocation during distributed init; the crash-site code (parallel_state.py:301) is unchanged across the regression window; the same test passed 6/6 runs immediately prior and uses the identical command.
  • Disconfirming: Jun 11 (27368902882) also failed for this job — a repeat (not strictly one-off) — though its symptom/stack was not verified here, so it may be a different cluster.
  • How to verify: re-run the nightly-8-gpu-glm51 job at head f7632ef8; if it passes, the flake hypothesis is supported. Check dmesg/GPU health on the runner for HIP/XGMI faults around 19:45 UTC.
• [LOW] A commit in the 57-commit window f5b041622b..f7632ef8 introduced the crash.
  • Supporting: all 57 commits landed after the last green run.
  • Disconfirming: none of the 57 commits touch parallel_state.py, the distributed-init path, or the torch.ones/device-fill call site that appears at the top of the stack; the crash is in a generic torch GPU op.
  • How to verify: if re-run reproduces, bisect f5b041622b..f7632ef8, but prioritize the flake hypothesis first.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for GLM-5.1 segfault (0 results) and init_world_group segfault (0 results). No matching open PR.

Suggested Triage Steps

• Re-run the nightly-8-gpu-glm51 job at head f7632ef8; a clean pass confirms the GPU/runtime-flake hypothesis.
• Inspect runner host logs (dmesg -T, GPU SMI/XGMI status) for HIP faults at ~2026-06-18 19:45 UTC on the MI30x node.
• Verify the Jun 11 failure (80875628432) — if its stack matches parallel_state.py:301 SIGSEGV, treat as a recurring infra flake; if different, it belongs to a separate cluster.
• Only if re-run reproduces deterministically: bisect f5b041622b..f7632ef8.

Status

• Persistence: Intermittent — failed Jun 18 (current) and Jun 11; passed all 6 runs in between (Jun 12–17).
• Scope: 1 job (nightly-8-gpu-glm51), 1 test file, 1 model (zai-org/GLM-5.1-FP8 (dsa)) in nightly-test-amd.yml. Sister jobs nightly-8-gpu-mi35x-glm51 were not assessed here.
• Blocked work: GLM-5.1-FP8 dsa 8-GPU accuracy signal on AMD MI30x for this run.
• In-flight fix: ❌ none found in open PRs.

nightly-accuracy-2-gpu-vlm — failed step(s): Nightly Accuracy Test (2-GPU VLM MMMU)

Commit Info

• sglang: f7632ef8
• aiter: 7d604afe — v0.1.14rc1.dev240+g7d604afe5 (source: dev image, preserved; matches Dockerfile default, no override)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi30x/test_vlms_mmmu_eval_amd.py	TestNightlyVLMMmmuEvalAMD.test_mmmu_vlm_models	AssertionError: The following models failed: (4 VLMs below MMMU accuracy threshold)	Log

The single test function asserts on a suite of 11 VLM models; 7 passed, 4 failed:

Model	TP	Score	Threshold	Status
Qwen/Qwen2.5-VL-7B-Instruct	1	0.280	0.33	❌ FAIL
Qwen/Qwen3-VL-30B-A3B-Instruct	2	0.270	0.29	❌ FAIL
openbmb/MiniCPM-o-2_6	1	0.280	0.32	❌ FAIL
deepseek-ai/deepseek-vl2-small	1	0.280	0.31	❌ FAIL
Qwen/Qwen2-VL-7B-Instruct	1	0.300	0.30	✅ PASS (at threshold)

Failure Summary

The nightly AMD VLM MMMU accuracy test failed because 4 of 11 evaluated VLM models scored below their configured accuracy thresholds on the 100-sample MMMU benchmark. The job ran to completion (no crash/OOM); the failure is a single AssertionError raised in _check_results (test_vlms_mmmu_eval_amd.py:380) after all models were evaluated. The failing models score near the MMMU 4-choice random-guess baseline (~0.25).

Regression Status

Known recurring failure / Never-passed in queryable window. This test file has failed in every completed scheduled run of nightly-accuracy-2-gpu-vlm in the lookback window; no green run exists for it.

Recent history of test_vlms_mmmu_eval_amd.py (test_mmmu_vlm_models) in job nightly-accuracy-2-gpu-vlm (branch main, event schedule, completed runs only):

Date	Run	Test File Status	Failing models (score/threshold)
Jun 18	27780272530	❌ Failed	Qwen2.5-VL 0.280/0.33 · Qwen3-VL-30B 0.270/0.29 · MiniCPM-o 0.280/0.32 · deepseek-vl2 0.280/0.31
Jun 15	27568949589	❌ Failed	Qwen2-VL 0.290/0.30 · Qwen2.5-VL 0.280/0.33 · Qwen3-VL-30B 0.260/0.29 · MiniCPM-o 0.270/0.32 · deepseek-vl2 0.280/0.31
Jun 14	27507337214	❌ Failed	(same family of VLMs below threshold)
Jun 13	27474523511	⏭️ Cancelled	— (excluded)
Jun 12	27434333913	❌ Failed	Qwen2-VL 0.280/0.30 · Qwen2.5-VL 0.290/0.33 · Qwen3-VL-30B 0.260/0.29 · MiniCPM-o 0.270/0.32 · deepseek-vl2 0.280/0.31
Jun 11	27368902882	❌ Failed	Qwen2-VL 0.280/0.30 · Qwen2.5-VL 0.280/0.33 · Qwen3-VL-30B 0.260/0.29 · MiniCPM-o 0.280/0.32 · deepseek-vl2 0.280/0.31

First observed failure in window: Jun 11 (earliest queryable, likely older). Last known passing run for this test file: none found in the window Jun 7–18 (runs on Jun 13/16/17 were cancelled, all other completed runs failed). Lookback range: 12 scheduled runs.

Failure Cluster

VLM MMMU accuracy below threshold (AMD/ROCm) — stable sub-threshold scores, not flaky.

Facts (verified observations)

• Failing scores are stable and reproducible across runs, not noisy: Qwen3-VL-30B = 0.260/0.270 every run; deepseek-vl2-small = 0.2802 every run; Qwen2.5-VL ≈ 0.280–0.290 every run (verified from [METRIC] mmmu_score= lines in Jun 11/12/15/18 logs).
• Three failing models register the identical score 0.2802 repeatedly (deepseek-vl2-small every run; Qwen2.5-VL and MiniCPM-o frequently) — the same value recurs across independent runs, indicating deterministic behavior rather than sampling variance.
• The entire suite scores low (all 11 models between 0.27 and 0.34); passing models clear their thresholds only marginally. Thresholds are tightly calibrated just above observed scores (e.g. Qwen2-VL passes at exactly 0.300 = threshold 0.30 this run, but failed at 0.279/0.289 on Jun 11/12/15 → borderline).
• Job completed normally: 7 passed / 4 failed, total runtime 46.7 min; exit code 1 → harness exit 255 (Exit code 255 indicates test failure (not network issue), not retrying).
• aiter version is unchanged across the failing runs (Dockerfile default 7d604afe, dev image preserved); no aiter override in play.
• Log noise (not causal): repeated KimiVLImageProcessor.preprocess() does not accept {'device'} warnings (Kimi-VL passed), and No module named 'torchcodec' import-skip messages.

Hypothesised Causes (with confidence)

• [MEDIUM] Thresholds in MODEL_CONFIGS are set higher than these AMD-ported VLMs actually achieve on the 100-sample MMMU subset, so the test asserts on an unmet bar every run.
  • Supporting: scores are stable across 4+ runs and consistently just below threshold (Qwen3-VL-30B 0.26–0.27 vs 0.29; deepseek-vl2 0.2802 vs 0.31); no green run exists in the window; thresholds live in the test file (L43-L123).
  • Disconfirming: scores near random-guess (~0.25) on 4 multi-architecture models could instead reflect a genuine accuracy bug (see next hypothesis) rather than merely over-strict thresholds; lowering thresholds would mask that.
  • How to verify: git log -p the threshold block to see whether thresholds were recently raised above achievable scores, or compare these scores to the NVIDIA MMMU baseline for the same models.
• [LOW] A real VLM correctness/accuracy regression on ROCm (image preprocessing, vision-encoder, or MMMU answer extraction) is depressing scores toward random baseline.
  • Supporting: 4 independent model architectures all land near ~0.25–0.28; the recurring identical 0.2802 suggests systematic (deterministic) degraded behavior rather than per-model quality.
  • Disconfirming: 7 models (including Kimi-VL 0.34, DeepSeek-OCR-2 0.34, Janus-Pro 0.29) pass and score above baseline, so any shared-path bug would have to be selective; no crash/error in logs — all models produce parseable outputs.
  • How to verify: manually inspect a few sample predictions for deepseek-vl2-small / Qwen2.5-VL in the generated /tmp/mmmu_*.json report; compare per-model AMD vs NVIDIA accuracy; bisect MMMU score over recent nightly runs to see if scores ever exceeded threshold.
• aiter is not a candidate: version unchanged across passing-history-free window and failure is a stable accuracy gap, not a kernel error.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for VLM MMMU threshold (3 hits: #26639, #13166, #18186) and mmmu amd (5 hits) — none addresses these VLM MMMU accuracy thresholds.

Suggested Triage Steps

• Confirm whether thresholds were recently set above achievable scores: git log -p -- test/registered/amd/accuracy/mi30x/test_vlms_mmmu_eval_amd.py focusing on the accuracy_threshold values for the 4 failing models.
• Compare these AMD MMMU scores against the NVIDIA nightly MMMU baseline for the same models; if NVIDIA scores are similarly low, the thresholds are mis-calibrated (favor adjusting thresholds); if NVIDIA is materially higher, treat as a ROCm accuracy regression.
• Inspect generated per-model reports (/tmp/mmmu_deepseek-ai_deepseek-vl2-small.json, /tmp/mmmu_Qwen_Qwen2.5-VL-7B-Instruct.json) for degenerate/empty answers vs correct-but-misparsed answers to distinguish "model wrong" from "answer-extraction wrong".
• If a genuine accuracy regression is suspected, bisect the nightly MMMU score for one failing model over recent scheduled runs to find when it dropped below threshold (note: no green run exists in the last ~12 days, so the regression — if any — predates the queryable window).

Status

• Persistence: Failing every completed scheduled run in the queryable window (Jun 11, 12, 14, 15, 18; Jun 13/16/17 cancelled). No passing run found for this test file in the last ~12 days.
• Scope: 1 job (nightly-accuracy-2-gpu-vlm) in nightly-test-amd.yml; 4 of 11 VLM models affected. Borderline models (e.g. Qwen2-VL-7B) flip pass/fail run-to-run, so the failing set may grow.
• Blocked work: AMD nightly VLM accuracy signal is permanently red, masking any new VLM accuracy regression on ROCm.
• In-flight fix: ❌ none found — needs threshold recalibration PR or a ROCm VLM accuracy investigation.

nightly-accuracy-2-gpu — failed step(s): Nightly Test (2-GPU)

Commit Info

• sglang: f7632ef8
• aiter: 7d604afe5 (source: dev image preserved — v0.1.14rc1.dev240+g7d604afe5, matches Dockerfile default; rebuild skipped)

Failed Tests

Test file: test/registered/amd/accuracy/mi30x/test_gsm8k_eval_amd.py — a per-model gsm8k accuracy harness. 12 models passed, 5 failed (exit code 1 → job exit 255).

Test File	Model (sub-case)	Error	Log
test_gsm8k_eval_amd.py	mistralai/Mixtral-8x7B-Instruct-v0.1	Score 0.305 < threshold 0.69	Log
test_gsm8k_eval_amd.py	neuralmagic/Mixtral-8x7B-Instruct-v0.1-FP8	Score 0.189 < threshold 0.69	Log
test_gsm8k_eval_amd.py	mistralai/Mistral-7B-Instruct-v0.3	Score 0.381 < threshold 0.47	Log
test_gsm8k_eval_amd.py	neuralmagic/Mistral-7B-Instruct-v0.3-FP8	Score 0.325 < threshold 0.46	Log
test_gsm8k_eval_amd.py	google/gemma-2-27b-it	Score 0.858 < threshold 0.86	Log

Failure Summary

The nightly-accuracy-2-gpu job's gsm8k accuracy suite (test_gsm8k_eval_amd.py) reported 12 passed / 5 failed. Four of the five failures are the entire Mistral/Mixtral family (BF16 and FP8 variants), which score far below threshold; the fifth (gemma-2-27b-it) is a marginal miss (0.858 vs 0.86). The companion test_dumper.py passed (243/243).

Regression Status

Known recurring failure (not new today). The Mistral/Mixtral-family failures reproduce with near-identical scores in every prior completed schedule run of this job — the two most recent runs (Jun 16, Jun 17) were cancelled and excluded.

Recent history of test_gsm8k_eval_amd.py (Mistral/Mixtral sub-cases) in job nightly-accuracy-2-gpu (branch main, event schedule, completed runs only):

Date	Run	Job	Test File Status	Failing models	Mistral-7B / Mixtral score
Jun 18	27780272530	job	❌ Failed	Mistral-7B, Mistral-7B-FP8, Mixtral, Mixtral-FP8, gemma-27b	0.381 / 0.305
Jun 17	27710225107	—	⏭️ Run cancelled	—	—
Jun 16	27640378823	—	⏭️ Run cancelled	—	—
Jun 15	27568949589	job	❌ Failed	Mistral-7B (+others)	0.378 / —
Jun 14	27507337214	job	❌ Failed	Mistral-7B, Mixtral	0.370 / 0.301
Jun 13	27474523511	job	❌ Failed	Mistral-7B, Mistral-7B-FP8, Mixtral, Mixtral-FP8, gemma-27b	0.365 / 0.286

No passing run for the Mistral/Mixtral sub-cases exists in the queryable window (Jun 13–18); effectively Never-passed over this lookback. Mistral-7B is stable at ~0.36–0.38, Mixtral at ~0.29–0.31, Mixtral-FP8 at ~0.18–0.20 — consistent every run, indicating a deterministic problem, not a flake.

Failure Cluster

gsm8k accuracy below threshold — Mistral/Mixtral family (test_gsm8k_eval_amd.py)

Facts (verified observations)

• Suite summary: Passed: 12, Failed: 5; test_gsm8k_eval_amd.py returned exit code 1; First attempt failed with exit code 255 … not retrying (log lines 39777+).
• Today's failing scores (FACT, from [METRIC] gsm8k_score= lines): Mixtral 0.305 (thr 0.69), Mixtral-FP8 0.189 (thr 0.69), Mistral-7B 0.381 (thr 0.47), Mistral-7B-FP8 0.325 (thr 0.46), gemma-2-27b 0.858 (thr 0.86).
• Family pattern (FACT): all non-Mistral models pass — Llama-3.1-8B 0.814, Llama-3.1-70B 0.929, DeepSeek-Coder-V2-Lite 0.820, Qwen family ✅. Only Mistral/Mixtral and the borderline gemma fail.
• Historical scores for Mistral/Mixtral are near-identical across Jun 13/14/15 runs (FACT, from [CI-AITER-CHECK]-independent metric lines), confirming determinism.
• aiter version is a preserved dev image v0.1.14rc1.dev240+g7d604afe5 whose suffix matches the Dockerfile default SHA 7d604afe5… (FACT, [CI-AITER-CHECK] markers, log lines 4028–4030).

Hypothesised Causes (with confidence, with disconfirming evidence)

• [MEDIUM] GSM8K chat-eval prompt + answer-extraction mismatch for Mistral-family chat models (not a pure inference bug). The in-flight fix #27757 "Fix Mistral GSM8K chat eval" identifies that the eval uses a completion-style few-shot prompt with last-number extraction; Mistral/Mixtral responses continue into extra Question:/Answer: text, so the wrong number is extracted.
  • Supporting: family-specific failure pattern (only Mistral/Mixtral affected, all other architectures pass); PR author reports Mistral-7B scored 0.417 even on FlashInfer (i.e. off FA3), implicating the harness rather than the attention backend; scores deterministic across runs.
  • Disconfirming: Mixtral collapses to 0.18–0.31 against a 0.69 threshold — a ~0.4 absolute gap that is larger than a pure last-number-extraction artifact would typically explain, so a real MoE/attention accuracy component on AMD cannot be excluded for the Mixtral sub-cases. The PR body itself notes moving Mistral/Mixtral off FA3 alone did not recover accuracy.
  • How to verify: check out #27757, re-run test_gsm8k_eval_amd.py on AMD MI30x (2-GPU); confirm Mistral-7B/Mixtral scores cross threshold.
• [LOW] gemma-2-27b-it marginal miss (0.858 vs 0.86) is a threshold-boundary flake, distinct from the Mistral cluster.
  • Supporting: it scored 0.857/0.858/0.859 across Jun 13 (three reruns) — straddling 0.86; a tiny variance flips pass/fail.
  • Disconfirming: it has not been observed clearly above 0.86 in the window, so a slightly-too-high threshold (config issue) is equally plausible.
  • How to verify: review the gemma-2-27b threshold in test_gsm8k_eval_amd.py config vs observed score distribution.

In-flight Fix Check

• In-flight fix: ✅ #27757 "Fix Mistral GSM8K chat eval" (open since 2026-06-10, last updated 2026-06-19, not draft) — directly targets the Mistral/Mixtral gsm8k cluster (explicit chat prompt, Answer: extraction, stop sequences, keeps Mistral/Mixtral off FA3 auto-default). Chase reviewers; do NOT open a duplicate.
• In-flight fix (gemma-27b threshold): ❌ none found in open PRs — the marginal gemma miss is not addressed by #27757 and may need a separate threshold review.
• API queries run: search/issues?q=repo:sgl-project/sglang+is:pr+is:open+Mistral+gsm8k and +Mixtral+accuracy.

Suggested Triage Steps

• Land/verify #27757: re-run test_gsm8k_eval_amd.py on AMD MI30x 2-GPU and confirm all four Mistral/Mixtral sub-cases cross threshold.
• If Mixtral (0.18–0.31) still fails after #27757's harness fix, bisect the Mixtral MoE / attention path separately on AMD — the gap is large enough that an inference-side accuracy issue may coexist with the eval-harness issue.
• Separately review the gemma-2-27b-it 0.86 threshold against its observed 0.857–0.859 distribution; treat as flaky/threshold-tuning, independent of the Mistral cluster.

Status

• Persistence: Mistral/Mixtral cluster failing every completed schedule run in the queryable window (Jun 13–18); scores stable (Never-passed in window).
• Scope: 1 job (nightly-accuracy-2-gpu), 1 test file, 4 Mistral/Mixtral sub-cases + 1 marginal gemma sub-case. Other architectures unaffected.
• In-flight fix: ✅ #27757 awaiting merge (covers the 4 Mistral/Mixtral cases; gemma threshold not covered).

nightly-8-gpu-kimi-k26 — failed step(s): Accuracy Test (8-GPU Kimi-K2.6)

Commit Info

• sglang: f7632ef8
• aiter: 7d604afe5 (v0.1.14rc1.dev240+g7d604afe5; source: dev image preserved — [CI-AITER-CHECK] Dev/patched version detected → skipping rebuild)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi30x/test_kimi_k26_eval_amd.py	TestKimiK26EvalAMD.setUpClass	Exception: Server process exited with code 1 (root: RuntimeError: Rank 1 scheduler died during initialization (exit code: -11))	Log

Failure Summary

The Kimi-K2.6 8-GPU accuracy test failed in setUpClass because the server never came up: rank 1's scheduler process died with SIGSEGV (exit code -11) during distributed initialization. The segfault occurred at the first GPU allocation of world-group setup — torch.ones(len(ranks), dtype=torch.int32, device=cuda) at parallel_state.py:301 — before any model weights were loaded. No accuracy numbers were produced (Ran 0 tests).

Regression Status

Flaky test (intermittent) — this job alternates pass/fail across recent scheduled runs.

Recent history of test_kimi_k26_eval_amd.py (TestKimiK26EvalAMD.setUpClass) in job nightly-8-gpu-kimi-k26 (branch main, event schedule, completed runs only):

Date	Run	Job	Test File Status	Failed Function	Error
Jun 18	27780272530	nightly-8-gpu-kimi-k26	❌ Failed	setUpClass	SIGSEGV (-11) in distributed init
Jun 15	27568949589	nightly-8-gpu-kimi-k26	❌ Failed	setUpClass	different symptom: server hang/timeout (py-spy Failed to get stack traces)
Jun 14	27507337214	nightly-8-gpu-kimi-k26	✅ Passed	—	—
Jun 13	27474523511	nightly-8-gpu-kimi-k26	❌ Failed	(log expired — BlobNotFound)	unverifiable
Jun 12	27434333913	nightly-8-gpu-kimi-k26	✅ Passed	—	—
Jun 11	27368902882	nightly-8-gpu-kimi-k26	❌ Failed	(log expired — BlobNotFound)	unverifiable

First observed failure (this exact SIGSEGV symptom): Jun 18. Last known pass: Jun 14. The Jun 16/17 scheduled runs were cancelled (excluded). Jun 13/11 logs have expired so their symptom cannot be confirmed at the file level. Lookback: 12 most recent scheduled runs.

Failure Cluster

GPU segfault (SIGSEGV) during distributed world-group init on AMD

Facts (verified observations)

• Log: Fatal Python error: Segmentation fault then RuntimeError: Rank 1 scheduler died during initialization (exit code: -11) (job log lines 4206, 4427).
• Crashing thread stack (verbatim, lines 4215–4226): parallel_state.py:301 __init__ ← parallel_state.py:1509 init_world_group ← parallel_state.py:1844 init_distributed_environment ← model_runner.py:1229 init_torch_distributed ← tp_worker.py ← scheduler.py:432 __init__.
• The crash is at the first device tensor allocation in the world group (torch.ones(..., device=self.device)), immediately after Init torch distributed begin. (line 4205) — i.e. before weight loading / CUDA-graph capture.
• Exit code is -11 (SIGSEGV), not -9 (SIGKILL/OOM) — so the harness's OOM-killer hint does not apply.
• Command: sglang serve --model-path moonshotai/Kimi-K2.6 --tp 8 --decode-attention-backend triton --prefill-attention-backend aiter ... (line 4429).
• aiter is a preserved dev build v0.1.14rc1.dev240+g7d604afe5; image rocm/sgl-dev:...mi30x.
• This is nightly-test-amd.yml (not an AITER Scout run), so no sister-baseline A/B applies.

Hypothesised Causes (with confidence)

• [LOW] GPU runtime / driver-level fault on the MI30x runner during distributed init (transient infra).
  • Supporting: SIGSEGV occurs at the very first HIP allocation in world-group setup, before any sglang model logic runs; the job has a flaky pass/fail history on the same head code; the Jun 15 failure of the same test was a different symptom (hang), which is characteristic of runner/hardware instability rather than a deterministic code bug.
  • Disconfirming: no dmesg/XGMI/HIP error captured in the available log to directly confirm a hardware fault; cannot fully rule out a process-spawn/distributed-init race.
  • How to verify: re-run the job; if it passes, flaky/infra is confirmed. On repeat failure, capture dmesg -T and a HIP coredump on the runner at init time.
• [LOW] sglang code regression in distributed init between the Jun 14 pass (3cb29f67) and this run (f7632ef8).
  • Supporting: the crash is in parallel_state.py world-group construction.
  • Disconfirming: the crashing line torch.ones(..., device=cuda) is pre-existing (older than the shallow-clone boundary 9c53853e, not newly introduced); the failure is intermittent (Jun 14 passed on nearby code), which argues against a deterministic regression; same code path passes on the mi35x sister job in several runs.
  • How to verify: bisect 3cb29f6747..f7632ef8 only if the failure becomes reproducible.

No hypothesis reaches MEDIUM confidence; the intermittent history and crash location (first GPU op) point to infra/flakiness over a code regression, but neither is confirmed.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for kimi k26 amd (only #28159, a docs/cookbook migration — unrelated) and scheduler died segfault (only #17628, unrelated torch-compile enablement).

Suggested Triage Steps

• Re-run nightly-8-gpu-kimi-k26 once; a green re-run confirms flaky/infra and no code action is needed.
• On any repeat failure, capture dmesg -T | grep -i -E "oom|xgmi|gpu|hsa|hip" and a HIP coredump from the rank-1 scheduler at init time to distinguish driver/hardware fault from a code race.
• If reproducible, bisect 3cb29f6747..f7632ef8 focusing on distributed/parallel_state.py and model_executor/model_runner.py distributed-init changes; revert the implicated commit on a branch and re-run on MI30x.
• Consider whether the MI30x runner pool needs a health check — the Jun 15 hang + Jun 18 segfault on the same test suggest possible node instability.

Status

• Persistence: Intermittent — failed Jun 18 (SIGSEGV) and Jun 15 (hang); passed Jun 14 and Jun 12. Not continuously failing.
• Scope: This job (nightly-8-gpu-kimi-k26, MI30x) only. The mi35x sister job nightly-8-gpu-mi35x-kimi-k26 passed in the same Jun 18 run is not confirmed here, but it passed in several recent runs — this symptom is MI30x-specific in the observed window.
• Blocked work: Kimi-K2.6 8-GPU accuracy signal on AMD MI30x is unreliable.
• In-flight fix: ❌ none found in open PRs.

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-06-19 20:32 UTC)

[amd-bot]

nightly-test-amd-rocm720.yml — 17 failure(s)

Run started (UTC): 2026-06-18 18:17 UTC
Last scanned (UTC): 2026-06-19 20:30 UTC
Commits: sglang ca88b7f

Summary

Counts: 20 failures · 12 clusters · 7 carrying over (≥3 days) · 5 new today

#	Cluster	Job	Test File	Test Function	Status	Hypothesis (confidence)
1	HF download hang → 3600s startup timeout	nightly-accuracy-8-gpu-rocm720	test_grok1_fp8_eval_amd.py	N/A (startup hang)	recurring Jun15+18	[MEDIUM] cold HF weight download stalled; weight path code unchanged
2	HF download hang → 3600s startup timeout	nightly-8-gpu-kimi-k26-rocm720	test_kimi_k26_eval_amd.py	N/A (startup hang)	new Jun18	[HIGH] xet_get weight download stall (3rd-party)
3	HF download hang → 3600s startup timeout	nightly-8-gpu-glm51-rocm720	test_glm51_eval_amd.py	N/A (startup hang)	new Jun18	[HIGH] xet_get weight download stall (3rd-party)
4	HF download hang → 3600s startup timeout	nightly-8-gpu-qwen35-rocm720	test_qwen35_eval_amd.py	N/A (startup hang)	new Jun18	[MEDIUM] tokenizer/processor download filelock stall
5	Mistral/Mixtral GSM8K accuracy below threshold	nightly-accuracy-2-gpu-rocm720	test_gsm8k_eval_amd.py	test_gsm8k_all_models (Mistral/Mixtral)	≥11d persistent	[HIGH] chat-eval extraction — #27757
6	Mistral/Mixtral GSM8K accuracy below threshold	nightly-accuracy-2-gpu-rocm720	test_gsm8k_eval_amd.py	test_gsm8k_all_models (Mistral/Mixtral)	≥11d persistent	[HIGH] chat-eval extraction — #27757
7	VLM MMMU accuracy below threshold (AMD ROCm720)	nightly-accuracy-2-gpu-vlm-rocm720	test_vlms_mmmu_eval_amd.py	test_mmmu_vlm_models	≥8d persistent	[MEDIUM] thresholds too tight vs ROCm VLM accuracy gap
8	VLM MMMU accuracy below threshold (AMD ROCm720)	nightly-accuracy-2-gpu-vlm-rocm720	test_vlms_mmmu_eval_amd.py	test_mmmu_vlm_models	≥8d persistent	[MEDIUM] thresholds too tight vs ROCm VLM accuracy gap
9	HIP "invalid device pointer" in CUDA-IPC MM transport (encoder-DP)	nightly-4-gpu-rocm720	test_encoder_dp.py	test_vlm_mmmu_benchmark	recurring (Jun16/18/19)	[MEDIUM] Qwen3-VL-32B _new_shared_cuda crash — #26304
10	HIP "invalid device pointer" in CUDA-IPC MM transport (encoder-DP)	nightly-4-gpu-rocm720	test_encoder_dp.py	test_vlm_mmmu_benchmark	flaky 3/4 runs	[LOW] pre-existing ROCm IPC race — #27434 workaround
11	SIGSEGV in torch HIP fill kernel during distributed init	nightly-8-gpu-qwen3-235b-rocm720	test_qwen3_instruct_fp8.py	setUpClass	new today	[MEDIUM] daily container image (-20260618) torch/ROCm regression
12	SIGSEGV in torch HIP fill kernel during distributed init	nightly-8-gpu-qwen3-235b-rocm720	test_qwen3_instruct.py	setUpClass	new today	[MEDIUM] daily container image (-20260618) torch/ROCm regression
13	HIP OOM allocating compress-state KV buffer (unified_kv_triton)	nightly-8-gpu-mi35x-deepseek-v4-flash-rocm720	test_deepseek_v4_flash_fp4.py	setUpClass	new today	[LOW] backend mem headroom over mem-fraction-static=0.9
14	HIP OOM allocating compress-state KV buffer (unified_kv_triton)	nightly-8-gpu-mi35x-deepseek-v4-flash-rocm720	test_deepseek_v4_flash_fp8.py	setUpClass	new today	[LOW] backend mem headroom over mem-fraction-static=0.9
15	HIP stream-capture error during decode CUDA-graph capture (EP all-reduce)	nightly-8-gpu-mi35x-qwen3-235b-mxfp4-rocm720	test_qwen3_instruct_mxfp4.py	setUpClass	Never-passed (since Jun7)	[MEDIUM] RCCL EP all-reduce not capture-safe
16	GPU abort in DeepSeek-R1 HiCache forward → SIGABRT/timeout	nightly-8-gpu-mi35x-deepseek-r1-hicache-rocm720	test_deepseek_r1_hicache_mi35x.py	N/A (server crash)	Never-passed (since Jun7)	[MEDIUM] GPU-side fault in forward_extend
17	Hang in EAGLE-MTP verify (DSA cuda-graph metadata)	nightly-accuracy-8-gpu-mi35x-deepseek-v32-mtp-rocm720	test_deepseek_v32_mtp_eval_mi35x.py	test_a_gsm8k	flaky (passed Jun17)	[LOW] per-rank batch divergence → collective deadlock
18	IndexError in MoE _load_w2 narrow (Kimi-K2.6 TP8 load)	nightly-8-gpu-kimi-k26-rocm720	test_kimi_k26_eval_amd.py	setUpClass	recurring since Jun16	[LOW] w2 layout / HF checkpoint — related #22476
19	ROCm launch abort extra_buffer needs CUDA/MUSA/NPU (FLA) (mamba)	nightly-8-gpu-qwen35-rocm720	test_qwen35_eval_amd.py	setUpClass	new Jun19	[HIGH] 66a7fd5c (#28151) dropped is_hip() guard
20	GPU memory access fault during Qwen2-72B-FP8 serving	nightly-accuracy-2-gpu-rocm720	test_gsm8k_eval_amd.py	test_gsm8k_all_models (Qwen2-72B-FP8)	new today (passed Jun13/14)	[LOW] commit in 3cb29f67..f7632ef8; no repro

Cluster: HF download hang → 3600s startup timeout

• Affected: rows 1-4 — grok1, kimi-k26, glm51, qwen35 (weight shards via xet_get/hf_hub_download; qwen35 on tokenizer/processor filelock).
• Shared facts: server never reaches ready; all TP ranks blocked on the HF download filelock while one rank sits in huggingface_hub download; no Load weight end; killed at 3600s → exit 255. Same runner also logged a docker-registry context deadline exceeded (jobs 4 & 8), a second network signal.
• Top hypothesis: [MEDIUM/HIGH] environmental HF/Xet download stall; disconfirming: no explicit network exception logged, byte-progress not printed, so slow-but-progressing cold download not fully excluded.
• In-flight fix: ❌ none.
• Suggested triage: re-run with warm /sgl-data/hf-cache; A/B HF_HUB_DISABLE_XET=1; check runner egress to huggingface.co.

Cluster: HIP "invalid device pointer" in CUDA-IPC MM transport (encoder-DP)

• Affected: rows 9-10 (nightly-4-gpu-rocm720, both runs). Crash at cuda_ipc_transport_utils.py:488 _new_shared_cuda → MMMU accuracy collapses to ~random.
• Top hypothesis: [MEDIUM] Qwen3-VL-32B under --mm-enable-dp-encoder crashes in IPC reconstruct; disconfirming: intermittent (passed Jun17), rank-varying — consistent with an IPC lifetime race.
• In-flight fix: ✅ #26304 (shm re-pickle race), #27434 (SGLANG_DISABLE_SHM_MM workaround), #27694 (IPC hardening) — none confirmed to cover the ROCm path.

Cluster: ROCm launch abort extra_buffer needs CUDA/MUSA/NPU (FLA)

• Affected: row 19. Config-time assertion in _handle_mamba_radix_cache; deterministic.
• Top hypothesis: [HIGH] 66a7fd5c (#28151) rewrote auto strategy and dropped the is_hip() branch; disconfirming: assertion condition pre-existed, only the control flow reaching it changed.
• In-flight fix: ✅ #27141 (ROCm extra_buffer radix-cache support).

Cluster: Mistral/Mixtral GSM8K accuracy below threshold

• Affected: rows 5-6. 4 Mistral-family models ~half threshold, stable across ≥11 days.
• Top hypothesis: [HIGH] chat-eval answer extraction unreliable for Mistral chat models; disconfirming: Mixtral-8x7B-FP8 (0.156) may have a secondary FP8 numeric component.
• In-flight fix: ✅ #27757.

Other clusters (single-job, see per-job detail)

• VLM MMMU accuracy (rows 7-8): [MEDIUM] thresholds vs ROCm accuracy gap; no green baseline in window; ❌ no fix.
• SIGSEGV distributed init (rows 11-12): [MEDIUM] daily image regression — re-run pinned to -20260617; ❌ no fix.
• unified_kv_triton HIP OOM (rows 13-14): [LOW] triton backend passes same launch — bisect 33719cfb31..f7632ef8; ❌ no fix.
• mxfp4 stream-capture (row 15): [MEDIUM] RCCL EP all-reduce in graph capture — A/B --disable-cuda-graph; never-passed; ❌ no fix.
• DeepSeek-R1 HiCache abort (row 16): [MEDIUM] GPU fault in forward_extend — needs writable /coredumps repro; never-passed; ❌ no fix.
• DeepSeek-V32-MTP hang (row 17): [LOW] rank batch divergence; flaky; ❌ no fix.
• Kimi _load_w2 IndexError (row 18): [LOW] w2 layout/checkpoint — related #22476 (GPTQ scope).
• Qwen2-72B-FP8 mem fault (row 20): [LOW] single occurrence; re-run to confirm vs transient.

Suggested triage order

Rank	Triage step	Targets	Why this first
1	Merge/verify #27141 (or set --mamba-radix-cache-strategy no_buffer)	row 19	HIGH-confidence code regression with ready in-flight fix
2	Pre-warm /sgl-data/hf-cache + check runner HF/Xet egress; re-run	rows 1-4	Largest cluster; likely infra, cheap to clear
3	Chase reviewers on #27757	rows 5-6	HIGH-confidence fix, clears 11-day persistent cluster
4	Re-run row 11/12 pinned to image -20260617; diff torch/ROCm pkgs	rows 11-12	Isolates image vs sglang for new segfault
5	A/B SGLANG_DISABLE_SHM_MM (#27434) on encoder-DP	rows 9-10	Confirms CUDA-IPC root cause; in-flight fixes exist

---

Job	Failed Steps	Started
nightly-accuracy-2-gpu-vlm-rocm720	Nightly Accuracy Test ROCm 7.2 (2-GPU VLM MMMU)	2026-06-18T18:39
nightly-4-gpu-rocm720	Nightly Test ROCm 7.2 (4-GPU)	2026-06-18T18:17
nightly-accuracy-2-gpu-rocm720	Nightly Test ROCm 7.2 (2-GPU)	2026-06-18T18:32
nightly-accuracy-8-gpu-rocm720	Accuracy Test ROCm 7.2 (8-GPU Grok1-FP8)	2026-06-18T21:11
nightly-8-gpu-kimi-k26-rocm720	Accuracy Test ROCm 7.2 (8-GPU Kimi-K2.6)	2026-06-18T20:49
nightly-8-gpu-glm51-rocm720	Accuracy Test ROCm 7.2 (8-GPU GLM-5.1 DSA)	2026-06-18T20:49
nightly-8-gpu-qwen3-235b-rocm720	Accuracy Test + Performance Test ROCm 7.2 (8-GPU Qwen3)	2026-06-18T20:48
nightly-8-gpu-qwen35-rocm720	Accuracy Test ROCm 7.2 (8-GPU Qwen 3.5)	2026-06-18T20:58
nightly-8-gpu-mi35x-qwen3-235b-mxfp4-rocm720	Accuracy Test + Performance Test MI35x ROCm 7.2 (8-GPU Qwen3-235B-MXFP4)	2026-06-19T03:03
nightly-8-gpu-mi35x-deepseek-r1-hicache-rocm720	Accuracy Test MI35x ROCm 7.2 (8-GPU DeepSeek-R1 HiCache)	2026-06-19T02:23
nightly-accuracy-8-gpu-mi35x-deepseek-v32-mtp-rocm720	Accuracy Test MI35x ROCm 7.2 (8-GPU DeepSeek-V3.2 TP+MTP)	2026-06-19T03:10
nightly-8-gpu-mi35x-deepseek-v4-flash-rocm720	Accuracy + Performance Test MI35x ROCm 7.2 (8-GPU DeepSeek-V4-Flash FP8 + FP4, unified_kv_triton)	2026-06-19T11:59
nightly-8-gpu-qwen35-rocm720	Accuracy Test ROCm 7.2 (8-GPU Qwen 3.5)	2026-06-19T19:23
nightly-8-gpu-kimi-k26-rocm720	Accuracy Test ROCm 7.2 (8-GPU Kimi-K2.6)	2026-06-19T18:29
nightly-4-gpu-rocm720	Nightly Test ROCm 7.2 (4-GPU)	2026-06-19T18:06
nightly-accuracy-2-gpu-rocm720	Nightly Test ROCm 7.2 (2-GPU)	2026-06-19T18:06
nightly-accuracy-2-gpu-vlm-rocm720	Nightly Accuracy Test ROCm 7.2 (2-GPU VLM MMMU)	2026-06-19T18:06

Per-Job Analysis

nightly-accuracy-2-gpu-vlm-rocm720 — failed step(s): Nightly Accuracy Test ROCm 7.2 (2-GPU VLM MMMU)

Commit Info

• sglang: f7632ef8
• aiter: 7d604afe5 (v0.1.14rc1.dev240+g7d604afe5, source: dev image preserved — Dev/patched version detected → skipping rebuild)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi30x/test_vlms_mmmu_eval_amd.py	test_mmmu_vlm_models	AssertionError: The following models failed: (4 VLMs below MMMU threshold)	Log

Individual model results that tripped the assertion (MMMU score vs threshold):

Model	TP	Score	Threshold	Status
Qwen/Qwen2.5-VL-7B-Instruct	1	0.280	0.33	❌ FAIL
Qwen/Qwen3-VL-30B-A3B-Instruct	2	0.260	0.29	❌ FAIL
openbmb/MiniCPM-o-2_6	1	0.280	0.32	❌ FAIL
deepseek-ai/deepseek-vl2-small	1	0.280	0.31	❌ FAIL

(7 of 11 models passed: Qwen2-VL-7B, InternVL2_5-2B, MiniCPM-v-2_6, Janus-Pro-7B, Kimi-VL-A3B, MiMo-VL-7B-RL, DeepSeek-OCR-2.)

Failure Summary

The accuracy test test_mmmu_vlm_models failed because 4 of 11 vision-language models scored below their configured MMMU accuracy thresholds. The test aggregates all model evals and raises a single AssertionError at test_vlms_mmmu_eval_amd.py:380 when any model is under threshold. This is an accuracy-quality failure on AMD MI325 (ROCm 7.2), not a crash or infra error.

Regression Status

Known recurring failure / Never-passed in the queryable window. The same test file has failed in every completed scheduled run back through at least Jun 11, with the same 4 models consistently under threshold (and Qwen2-VL-7B intermittently failing too in some runs). No green run for this test file exists in the lookback window (Jun 11–18).

Recent history of test_vlms_mmmu_eval_amd.py (test_mmmu_vlm_models) in job nightly-accuracy-2-gpu-vlm-rocm720 (branch main, event schedule, completed runs only):

Date	Run	Test File Status	Failed Models	Note
Jun 18	27780200720	❌ Failed	Qwen2.5-VL, Qwen3-VL-30B, MiniCPM-o-2_6, deepseek-vl2-small (4)	this run
Jun 15	27568866028	❌ Failed	same 4 (Qwen2-VL ~0.280 borderline)	—
Jun 14	27507292665	❌ Failed	same 4	—
Jun 13	27474496879	❌ Failed	same 4 + Qwen2-VL-7B (5)	—
Jun 12	27434236636	❌ Failed	same 4 + Qwen2-VL-7B (5)	—
Jun 11	27368779328	❌ Failed	same 4 + Qwen2-VL-7B (5)	—

(Jun 16/17 scheduled runs were cancelled, so excluded. First observed failure ≥ Jun 11; no passing run found in window — lookback Jun 9–18.)

Failure Cluster

VLM MMMU accuracy below threshold on AMD ROCm720 (scores pinned near random-guess level)

Facts (verified observations, no interpretation)

• Verbatim assertion (log line 14775): AssertionError: The following models failed: listing the 4 models with score=… threshold=….
• Scores are deterministic across reruns: Qwen2.5-VL-7B = 0.2802 and Qwen3-VL-30B-A3B = 0.2601 to 4 decimals in Jun 18, Jun 15, Jun 14, Jun 13, Jun 12 logs ([METRIC] mmmu_score= lines).
• The failing scores (0.26–0.28) cluster near the ~0.25 random-guess baseline for 4-choice MMMU, and the same ~0.28 value recurs across architecturally distinct models (Qwen2.5-VL, MiniCPM-o, deepseek-vl2).
• 7 other VLMs on the same job/runner pass their (lower) thresholds, e.g. Kimi-VL 0.340, MiMo-VL 0.310, DeepSeek-OCR-2 0.320.
• aiter version is unchanged (dev image preserved, no override) — AITER_COMMIT_OVERRIDE empty across all runs.
• Runner: linux-mi325-2gpu-sglang (MI325, gfx942/mi30x), ROCm 7.2 image.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [LOW] Thresholds for these 4 specific models are set higher than achievable on the AMD ROCm720 backend (test-config mismatch rather than a code regression).
  • Supporting: scores are stable and reproducible run-to-run; the failure has existed since the start of the queryable window with no green baseline; 7 sibling models pass against lower thresholds.
  • Disconfirming: scores ~0.26–0.28 sit near the random-guess floor, which is suspiciously low for capable models like Qwen2.5-VL-7B (expected ~0.5+ on MMMU) — this points to a real inference/output-parsing defect, not merely an over-tight threshold.
  • How to verify: read test_vlms_mmmu_eval_amd.py model/threshold config history (git log -p) to see whether thresholds were recently raised, vs. compare to the same models' MMMU scores on the NVIDIA nightly job.
• [LOW] VLM image-understanding pipeline produces near-random outputs for these models on ROCm 7.2 (e.g. broken vision attention / multimodal preprocessing for these specific architectures).
  • Supporting: scores pinned at the random-guess level and identical across reruns; affects multiple distinct model families.
  • Disconfirming: other VLMs on the identical runner/image pass, so any defect would have to be model-architecture-specific, not a blanket ROCm issue.
  • How to verify: manually run one failing model (e.g. Qwen2.5-VL-7B) MMMU eval on the MI325 ROCm720 container and inspect raw generations in /tmp/mmmu_*.html for empty/garbled answers vs. correct-but-misparsed answers.
• No hypothesis reaches MEDIUM confidence. This is a persistent accuracy failure with no green baseline in-window; needs a known-good reference (NVIDIA scores or an earlier green AMD run outside the window) to decide between "threshold too tight" and "real inference defect".

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for MMMU+VLM, Qwen2.5-VL+accuracy; none target the AMD ROCm720 VLM MMMU accuracy regression. (Existing VLM PRs are feature/perf work — piecewise CUDA graph, FP8 vision attn, new models — not this accuracy issue.)

Suggested Triage Steps

• Inspect the rendered eval reports for one failing model (/tmp/mmmu_Qwen_Qwen2.5-VL-7B-Instruct.html/.json) from this run's artifacts to determine whether outputs are empty/garbled (inference defect) or correct-but-misparsed (answer-extraction regex) vs. genuinely wrong (quality regression).
• Cross-check the same 4 models' MMMU scores on the NVIDIA nightly accuracy job; if NVIDIA also sits near ~0.28, the thresholds are mis-set; if NVIDIA scores ~0.5+, the defect is ROCm-backend-specific.
• git log -p test/registered/amd/accuracy/mi30x/test_vlms_mmmu_eval_amd.py to check when these 4 thresholds were introduced/raised relative to the first failing run (≥ Jun 11).
• If confirmed an inference defect, bisect ROCm VLM forward path between the last known-good AMD nightly (predates Jun 11 window) and current; if confirmed threshold mismatch, adjust per-model thresholds for ROCm720.

Status (factual, no priority assignment)

• Persistence: Failing in every completed scheduled run from at least Jun 11 through Jun 18 (8 days); no green run for this test file in the queryable window.
• Scope: 1 job (nightly-accuracy-2-gpu-vlm-rocm720) in nightly-test-amd-rocm720.yml; 4 models consistently fail (5 in some runs incl. Qwen2-VL-7B).
• Blocked work: AMD ROCm720 VLM accuracy signal — these 4 models report no trustworthy nightly accuracy gate.
• In-flight fix: ❌ none found in open PRs — needs new investigation/PR.

nightly-4-gpu-rocm720 — failed step(s): Nightly Test ROCm 7.2 (4-GPU)

Commit Info

• sglang: f7632ef86
• aiter: 7d604afe5 (source: dev/patched image — v0.1.14rc1.dev240+g7d604afe5, preserved, rebuild skipped)

Failed Tests

Test File	Test Function	Error	Log
test/registered/vlm/test_encoder_dp.py	TestVLMEncoderDP.test_vlm_mmmu_benchmark	AssertionError: Model Qwen/Qwen3-VL-32B-Instruct accuracy (0.2778) below expected threshold (0.5500)	Log

Underlying server fault (the real cause of the low accuracy):

Component	Error	Log
Scheduler (TP3, then TP1 on retry)	torch.AcceleratorError: HIP error: invalid device pointer — "Failed to deserialize from CUDA IPC handle" at cuda_ipc_transport_utils.py:488 (_new_shared_cuda)	Log

Failure Summary

The single registered test in this job, test_vlm_mmmu_benchmark, asserts MMMU accuracy ≥ threshold. The CI harness picks one random model per attempt; both the initial attempt and the retry happened to select Qwen/Qwen3-VL-32B-Instruct. For that model the Scheduler crashed during the MMMU eval with HIP error: invalid device pointer while reconstructing multimodal image features from a CUDA IPC handle (the --mm-enable-dp-encoder feature-transport path). The crash (SIGQUIT) killed the server, producing 10,224 "Connection error" log lines, so almost all 900 MMMU requests failed and accuracy collapsed to ~random (0.2522 then 0.2778, vs 0.55 threshold).

Regression Status

Flaky test surface over a persistent, model-specific server crash. The test-file pass/fail is effectively probabilistic — it only goes red when random.choice selects the broken model (Qwen3-VL-32B) on both the initial attempt and the retry. The underlying Qwen3-VL-32B CUDA-IPC crash is deterministic and was already present (but masked) on the Jun-14 passing run.

Recent history of test_encoder_dp.py (test_vlm_mmmu_benchmark) in job nightly-4-gpu-rocm720 (branch=main, event=schedule, completed runs only):

Date	Run	Test File Status	Model picked	Note
Jun 18	27780200720	❌ Failed	Qwen3-VL-32B (both attempts)	IPC crash both attempts → acc 0.25 / 0.28
Jun 17	27710084518	—	—	run cancelled (excluded)
Jun 16	27640284588	—	—	run cancelled (excluded)
Jun 15	27568866028	❌ Failed	none reached	different symptom: timeout at 3605s before any MMMU eval
Jun 14	27507292665	✅ Passed	Qwen3-VL (crash) → retry InternVL2_5-8B (pass)	same IPC crash present on 1st attempt, masked by retry picking a different model
Jun 13	27474496879	✅ Passed	InternVL2_5-8B	no IPC crash

First observed IPC-crash-driven test failure: Jun 18 (this run). Underlying Qwen3-VL crash first observed: Jun 14 (masked). Lookback window: Jun 11–18 scheduled runs.

Failure Cluster

GPU "invalid device pointer" crash in CUDA IPC multimodal feature transport (DP encoder) on Qwen3-VL-32B → MMMU accuracy collapse.

Facts (verified observations)

• Scheduler hit an exception at identical stack on both attempts: scheduler.py:2093 handle_generate_request → _get_multimodal_inputs (scheduler.py:1909) → MultimodalInputs.from_processor_output (schedule_batch.py:516) → mm_item.reconstruct (schedule_batch.py:365) → reconstruct_on_target_device (cuda_ipc_transport_utils.py:488) → torch.UntypedStorage._new_shared_cuda → torch.AcceleratorError: HIP error: invalid device pointer (log lines 4748 and 11300).
• After each crash: SIGQUIT received ... It usually means one child failed, then 10,224 All 5 attempts failed. Last error: Connection error. lines.
• Both eval attempts selected the same model (Testing model: Qwen/Qwen3-VL-32B-Instruct); results 0.25222 and 0.27778 — both ≈ 4-choice random baseline (0.25), consistent with nearly all requests failing.
• Server launched with --mm-enable-dp-encoder --tp=4 --cuda-graph-max-bs 32 (test other_args).
• On Jun-14 passing run, Qwen3-VL-32B produced the same IPC crash + 0.2522; the test passed only because the retry's random.choice returned InternVL2_5-8B (0.5322). InternVL2_5-8B never triggers the IPC crash in observed runs.
• aiter is a preserved dev build (7d604afe5); not a scout/override run.

Hypothesised Causes (with confidence)

• [HIGH] The MMMU threshold assertion is a symptom, not the cause. The Scheduler CUDA-IPC crash kills the server mid-eval → mass connection failures → ~random accuracy.
  • Supporting: 10,224 connection errors temporally follow each SIGQUIT; accuracy ≈ 0.25 (random); identical crash on both attempts.
  • Disconfirming: (none found) — the same model scores ~0.25 every time it crashes; no clean run of Qwen3-VL-32B exists to show genuine accuracy.
• [MEDIUM] The crash is specific to Qwen/Qwen3-VL-32B-Instruct under --mm-enable-dp-encoder (CUDA IPC multimodal feature transport), in the non-pooled _new_shared_cuda redirected-handle path (cuda_ipc_transport_utils.py:486-490).
  • Supporting: Qwen3-VL-32B crashes on Jun 14 and Jun 18; InternVL2_5-8B passes without crash on Jun 13/14; crash is in the IPC reconstruct path that DP-encoder uses.
  • Disconfirming: no data on the other two models in the list (Qwen2.5-VL-72B, GLM-4.1V-9B); cannot confirm it is uniquely Qwen3-VL vs. several VLMs.
  • How to verify: launch Qwen3-VL-32B with --mm-enable-dp-encoder --tp=4 on an MI30x ROCm-7.2 container and replay a few MMMU images; confirm the invalid device pointer crash reproduces deterministically.
• [LOW] A recent change in the CUDA-IPC MM transport may have introduced/exposed the bad handle redirection. Candidate touch-points (NOT verified as the regression point; local clone is shallow so timing unconfirmed): a77abbe00 "Reduce GPU memory footprint of CUDA IPC MM feature transport" (#22662), d2440dcf5 "cache IPC pool handles" (#21418).
  • Supporting: both modify the exact file/path in the stack trace.
  • Disconfirming: both predate the observed runs by weeks and Qwen3-VL passed historically would be needed to bound a window — no such green Qwen3-VL run exists, so this may be a long-standing latent bug rather than a fresh regression.
  • How to verify: bisect Qwen3-VL-32B + DP-encoder across the IPC-transport commit history once a deterministic repro is in hand.

In-flight Fix Check

• In-flight fix: ⚠️ [#18721](https://github.com/sgl-project/sglang/pull/18721) "[BUG] fix mm_enable_dp_encoder hang for Qwen3-VL models" — open but stale (created & last updated 2026-02-12, untouched ~4 months). Same feature + model family, but describes a hang, not the invalid device pointer crash — weak match, verify relevance before relying on it; do not assume it fixes this crash.
• In-flight fix: ❌ none found matching cuda ipc invalid device pointer / encoder_dp in open PRs.
• (Searched: search/issues for mm-enable-dp-encoder, encoder_dp, cuda ipc invalid device pointer.)

Suggested Triage Steps

• Reproduce: run Qwen/Qwen3-VL-32B-Instruct with --mm-enable-dp-encoder --tp=4 --cuda-graph-max-bs 32 on an MI30x ROCm-7.2 container (aiter 7d604afe5) and send MMMU image requests; confirm the cuda_ipc_transport_utils.py:488 invalid device pointer crash is deterministic.
• If reproducible, instrument the non-pooled redirect path (cuda_ipc_transport_utils.py:480-490): log rebuild_device_idx, original_handle[0], and the exporting rank's device to check the handle is being rebuilt on the wrong device.
• Verify whether stale PR #18721 addresses this crash; if not, a new fix/issue is needed.
• Separately, reduce test flakiness/noise: the retry re-randomizes the model, so this crash intermittently hides. Consider making the retry re-test the same model, or quarantining Qwen3-VL-32B in test_encoder_dp.py until the IPC crash is fixed (maintainer decision).
• Note Jun-15's failure was a distinct timeout (3605s, no model reached) — track separately, not part of this cluster.

Status

• Persistence: Underlying Qwen3-VL-32B IPC crash observed Jun 14 and Jun 18 (identical signature); surfaces as a test failure only when both random model picks land on Qwen3-VL-32B. Jun 13/14 test passes were luck-of-the-draw, not a fix.
• Scope: nightly-4-gpu-rocm720 job, single test file test_encoder_dp.py. Crash is in shared CUDA-IPC DP-encoder transport, so it can affect any VLM that exercises that path (NVIDIA also registers this test as nightly-4-gpu).
• Blocked work: Reliable nightly VLM DP-encoder accuracy signal on AMD MI30x ROCm 7.2.
• In-flight fix: ⚠️ stale #18721 (weak match, possibly unrelated); otherwise none.

nightly-accuracy-2-gpu-rocm720 — failed step(s): Nightly Test ROCm 7.2 (2-GPU)

Commit Info

• sglang: f7632ef8
• aiter: 7d604afe5 — v0.1.14rc1.dev240+g7d604afe5 (source: dev image preserved; [CI-AITER-CHECK] reported "Dev/patched version detected → skipping rebuild"; Dockerfile default was the same SHA)

Failed Tests

All failures are model cases inside one parametrized test: TestNightlyGsm8KEval.test_gsm8k_all_models in test/registered/amd/accuracy/mi30x/test_gsm8k_eval_amd.py. 6 of 23 models failed.

Test File	Test Function (model case)	Error	Log
test/registered/amd/accuracy/mi30x/test_gsm8k_eval_amd.py	test_gsm8k_all_models · neuralmagic/Qwen2-72B-Instruct-FP8	GPU memory access fault → Fatal Python error: Aborted; score 0.000 (threshold 0.86)	Log
test/registered/amd/accuracy/mi30x/test_gsm8k_eval_amd.py	test_gsm8k_all_models · neuralmagic/Mixtral-8x7B-Instruct-v0.1-FP8	score 0.170 < threshold 0.69	Log
test/registered/amd/accuracy/mi30x/test_gsm8k_eval_amd.py	test_gsm8k_all_models · mistralai/Mixtral-8x7B-Instruct-v0.1	score 0.306 < threshold 0.69	Log
test/registered/amd/accuracy/mi30x/test_gsm8k_eval_amd.py	test_gsm8k_all_models · neuralmagic/Mistral-7B-Instruct-v0.3-FP8	score 0.322 < threshold 0.46	Log
test/registered/amd/accuracy/mi30x/test_gsm8k_eval_amd.py	test_gsm8k_all_models · mistralai/Mistral-7B-Instruct-v0.3	score 0.367 < threshold 0.47	Log
test/registered/amd/accuracy/mi30x/test_gsm8k_eval_amd.py	test_gsm8k_all_models · google/gemma-2-27b-it	score 0.858 < threshold 0.86 (marginal, 0.002 below)	Log

Failure Summary

The nightly-accuracy-2-gpu-rocm720 job runs GSM8K against 23 models; 6 failed their thresholds. Five are accuracy shortfalls in the Mistral/Mixtral family (and a marginal gemma-2-27b-it). The sixth, neuralmagic/Qwen2-72B-Instruct-FP8, crashed with a GPU memory access fault ("Memory access fault by GPU node-3 ... on address (nil)" → Fatal Python error: Aborted) and scored 0.000. The two failure modes are distinct and tracked separately below.

Regression Status

Two clusters with different status:

• Mistral/Mixtral accuracy cluster — Known recurring (persistent / never-passing). History of test_gsm8k_all_models for these model cases in nightly-accuracy-2-gpu-rocm720 (same branch=main, event=schedule, completed runs):

Date	Run	Mistral-7B-v0.3	Mixtral-8x7B	Mistral-7B-v0.3-FP8	Mixtral-8x7B-FP8	gemma-2-27b-it
Jun 18 (today)	27780200720	❌ 0.367	❌ 0.306	❌ 0.322	❌ 0.170	❌ 0.858
Jun 14	27507292665	❌ 0.377	❌ 0.301	❌ 0.314	❌ 0.143	❌ 0.858
Jun 13	27474496879	❌ 0.367	❌ 0.324	❌ 0.301	❌ 0.164	❌ 0.857

These scores are stable run-to-run and consistently below threshold — a long-standing eval/setup issue, not a new regression.

• Qwen2-72B-FP8 GPU memory access fault — NEW today (regression candidate). Same model case in prior runs:

Date	Run	Qwen2-72B-Instruct-FP8	Status
Jun 18 (today)	27780200720	0.000 (mem fault crash)	❌ Failed
Jun 14	27507292665	0.9346	✅ Passed
Jun 13	27474496879	0.9346	✅ Passed

(Jun 15 run 27568866028 terminated after only 5 models — Qwen2-72B-FP8 did not run, inconclusive; Jun 16–17 runs were cancelled.) Last confirmed pass: Jun 14 (3cb29f6747); first crash: today (f7632ef8). Regression window 3cb29f6747..f7632ef8 (87 commits).

Failure Cluster

• Cluster A: "GSM8K accuracy below threshold — Mistral/Mixtral family" (symptom-level, 5 models incl. marginal gemma).
• Cluster B: "GPU memory access fault during Qwen2-72B-FP8 serving" (crash → score 0.000).

Facts (verified observations)

• Score/threshold pairs verbatim from log: Mistral-7B-v0.3 0.367/0.47; Mixtral-8x7B 0.306/0.69; Mistral-7B-v0.3-FP8 0.322/0.46; Mixtral-8x7B-FP8 0.170/0.69; gemma-2-27b-it 0.858/0.86; Qwen2-72B-FP8 0.000/0.86.
• Crash log (line ~28886): Memory access fault by GPU node-3 (Agent handle: ...) on address (nil). Reason: Unknown. followed by GPU core dump failed / Fatal Python error: Aborted, with the watchdog thread (python/sglang/srt/utils/watchdog.py:147) on the stack. Fault occurred under heavy concurrency (#running-req: ~790, token usage: 0.12, cuda graph: False) during prefill.
• aiter is unchanged vs the prior passing runs' Dockerfile default (dev build 7d604afe5), so aiter is not a differentiator for the Qwen2-72B-FP8 regression.
• Mistral/Mixtral scores are reproducible across ≥3 runs (±0.02), confirming the accuracy cluster is deterministic, not flaky.

Hypothesised Causes (with confidence, with disconfirming evidence)

Cluster A — Mistral/Mixtral accuracy:

• [MEDIUM] Eval-harness issue (chat-template / answer extraction) for instruction-tuned Mistral/Mixtral models, not a runtime/model regression.
  • Supporting: open PR #27757 "Fix Mistral GSM8K chat eval" states the GSM8K chat prompt and answer extraction are wrong for these chat models; scores are stable ~0.3 across many runs (deterministic, characteristic of a prompt/parse issue, not numeric drift).
  • Disconfirming: PR body notes that even after moving Mistral off FA3 the model still scored only 0.417 — so part of the gap may be genuine kernel/accuracy on AMD, not purely eval formatting. gemma-2-27b-it failing marginally (0.858 vs 0.86) is more likely a too-tight threshold than the same chat-eval bug.
  • How to verify: pull #27757, re-run test_gsm8k_all_models for the Mistral/Mixtral cases on MI30x; confirm scores recover above threshold.

Cluster B — Qwen2-72B-FP8 GPU memory access fault:

• [LOW] A commit in window 3cb29f6747..f7632ef8 touching FP8/MoE/quant or AMD kernel paths introduced a memory fault that triggers under high-concurrency prefill for this dense FP8 model. AMD-relevant candidates in the window include 6309fb9ab (HIP topk_ids masking), 9b8c41171 (ROCm layer_first write-back fallback), 2f1390fcb (FP8 block-K configs), 97e3b8998 (quant_config to attention gate projection).
  • Supporting: model was 0.9346 PASS on Jun 13/14 and only began crashing today; fault is at GPU memory level under load, consistent with a kernel/memory-layout change.
  • Disconfirming: 87 commits in the window and no stack frame pointing at a specific sglang source file (only the watchdog thread is captured); the listed candidates are not yet linked to the failing code path. The Qwen2-57B-A14B-FP8 (MoE) case passed in the same run, so a broad FP8/MoE break is unlikely. Could also be a transient hardware/runner memory fault rather than a code regression (single occurrence — Jun 15/16/17 are not usable confirmations).
  • How to verify: re-run the neuralmagic/Qwen2-72B-Instruct-FP8 case alone on MI30x at f7632ef8; if reproducible, bisect 3cb29f6747..f7632ef8. If not reproducible, treat as a one-off runner/GPU memory fault.

No Cluster-B hypothesis reaches MEDIUM; needs a repro + git bisect between 3cb29f6747..f7632ef8.

In-flight Fix Check

• Cluster A (Mistral/Mixtral accuracy): ✅ #27757 "Fix Mistral GSM8K chat eval" (open since 2026-06-10) — directly targets the GSM8K chat prompt + answer extraction for Mistral/Mixtral. Chase reviewers; do NOT open a duplicate. (Searched is:pr is:open gsm8k accuracy.)
• Cluster B (Qwen2-72B-FP8 mem fault): ❌ none clearly matching. (#11154 "Fix memory access fault" exists but is unrelated/old.) Needs new investigation. (Searched is:pr is:open memory access fault amd.)

Suggested Triage Steps

• Cluster A: Verify #27757 applies cleanly and re-run test_gsm8k_all_models (Mistral/Mixtral cases) on MI30x; if scores recover, this clears 4 of the 6 failures. Separately, review the google/gemma-2-27b-it threshold (0.86) — it consistently lands ~0.857–0.860, suggesting the threshold is set fractionally too high.
• Cluster B: Re-run the neuralmagic/Qwen2-72B-Instruct-FP8 case in isolation on MI30x at f7632ef8. If the GPU memory access fault reproduces, bisect 3cb29f6747..f7632ef8, prioritizing AMD/FP8/quant commits. If it does not reproduce, classify as a transient runner GPU fault and monitor the next scheduled run.

Status

• Persistence: Cluster A (Mistral/Mixtral) — persistent for ≥6 days (all completed runs Jun 13–18 show the same below-threshold scores). Cluster B (Qwen2-72B-FP8 crash) — first observed today (Jun 18); was passing Jun 13/14.
• Scope: Single job nightly-accuracy-2-gpu-rocm720 (workflow nightly-test-amd-rocm720.yml); 6 of 23 model cases in test_gsm8k_all_models. Job has reported failure on every completed run Jun 10–15.
• Blocked work: AMD ROCm 7.2 nightly GSM8K accuracy signal is red; the persistent Mistral/Mixtral failures mask new regressions like the Qwen2-72B-FP8 crash in the job's overall pass/fail.
• In-flight fix: Cluster A ✅ #27757 (open since 2026-06-10); Cluster B ❌ none found.

nightly-accuracy-8-gpu-rocm720 — failed step(s): Accuracy Test ROCm 7.2 (8-GPU Grok1-FP8)

Commit Info

• sglang: f7632ef8
• aiter: 7d604afe (v0.1.14rc1.dev240+g7d604afe5, source: dev image preserved — [CI-AITER-CHECK] Dev/patched version detected → skipping rebuild)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi30x/test_grok1_fp8_eval_amd.py	N/A (server never finished startup; whole file timed out)	✗ TIMEOUT after 3600s → exit code 255	Log

Note: the sibling suite in the same job, test_gpt_oss_eval_amd.py, passed (accuracy=0.525/0.860, both ✅). The job failed solely on the Accuracy Test ROCm 7.2 (8-GPU Grok1-FP8) step.

Failure Summary

The nightly-accuracy-8-gpu-rocm720 job timed out on the Grok1-FP8 accuracy step. The lmzheng/grok-1 server never reached readiness: all 8 TP scheduler ranks were stuck loading model weights, and after 3600s the test harness killed it (exit 255). The watchdog began firing a soft timeout at 21:33:11 (300s after weight load began) and repeated every ~5 min for the next hour until the step was force-terminated at 22:27:50.

Regression Status

Known recurring failure (intermittent). Same test file + same hang signature as June 15; passed June 13–14.

Recent history of test_grok1_fp8_eval_amd.py (step Accuracy Test ROCm 7.2 (8-GPU Grok1-FP8)) in job nightly-accuracy-8-gpu-rocm720 (branch main, event schedule, completed runs only):

Date	Run	Test File Status	Error
Jun 18	27780200720	❌ Timeout	stuck in download_weights_from_hf, 3600s timeout
Jun 17	27710084518	⚪ Cancelled	run cancelled (not comparable)
Jun 16	27640284588	⚪ Cancelled	run cancelled (not comparable)
Jun 15	27568866028	❌ Timeout	same: download_weights_from_hf filelock hang, 3600s timeout
Jun 14	27507292665	✅ Passed	—
Jun 13	27474496879	✅ Passed	—

First observed failure: 2026-06-15. Last known pass: 2026-06-14.

Failure Cluster

Grok-1 weight download hang during server startup (HF download_weights_from_hf filelock stall → 3600s test timeout)

Facts (verified observations)

• Grok1-FP8 server launched at 21:28:01 with model_path='lmzheng/grok-1', tp_size=8, quantization='fp8'; Load weight begin followed. The server never printed Load weight end (contrast: the gpt-oss suite earlier in the same job printed Load weight end. elapsed≈60s).
• First soft-watchdog timeout at 21:33:11 — [TP7] Scheduler watchdog timeout (watchdog_timeout=300, soft=True); repeated at 21:38, 21:43, … 22:23 (11 times).
• py-spy dumps at 21:33:12 show all 8 ranks blocked in weight loading:
  • TP3 (pid 11654): _inner_hf_hub_download → hf_hub_download → _hf_hub_download_to_cache_dir → download_weights_from_hf (weight_utils.py:593) → _prepare_presharded_weights (grok.py:960) → load_weights (grok.py:836) — actively downloading from HuggingFace.
  • TP4/TP6 (and others): download_weights_from_hf (weight_utils.py:554) → acquire → _poll_until_acquired (filelock/_api.py:552) — blocked on the download filelock held by the downloader rank.
• Repeated [TP3] Pyspy failed (py-spy dump --native --pid 11654). Error: failed to get os threadid — the downloader rank's native stack couldn't be sampled (consistent with a process blocked in a long network/IO syscall).
• Final outcome: ✗ TIMEOUT … after 3600 seconds (line 15316) → First attempt failed with exit code 255 → ##[error]The action 'Accuracy Test ROCm 7.2 (8-GPU Grok1-FP8)' has timed out after 60 minutes.
• The identical signature (all ranks in download_weights_from_hf + filelock at weight_utils.py:556, 3600s timeout) is present in the Jun 15 failure log (job 81499860198, lines 6235–6341, 15500).
• The grok weight-loading code path is unchanged in the regression window: git log on python/sglang/srt/models/grok.py shows the last touch was 53318911c (a get_parallel() accessor), unrelated to download logic.
• Early in the run, the runner also logged a registry timeout: [local-pull] Error response from daemon: Get "https://10.245.143.50:5000/v2/": context deadline exceeded (line 1250) — independent of model download but a second network-reachability signal on this runner.

Hypothesised Causes (with confidence)

• [MEDIUM] Environment/infra: lmzheng/grok-1 (314B params, ~hundreds of GB) was not cached on the MI30x runner and the HF download did not complete within the 3600s budget (stalled or throttled network).
  • Supporting: downloader rank TP3 is parked in hf_hub_download/_hf_hub_download_to_cache_dir for the full hour; other ranks merely wait on the download filelock; gpt-oss (smaller, faster) loaded fine in the same job; an unrelated registry fetch also timed out on this runner.
  • Disconfirming: the log does not print explicit download byte/percentage progress, so a true network stall vs merely slow-but-progressing cannot be distinguished from the log alone; the est_time for this file is only 1500s, so the 3600s cap may simply be too tight when a cold download is needed.
  • How to verify: pre-warm the lmzheng/grok-1 HF cache on the MI30x runner (or set download_dir to a persistent volume) and re-run; check HF cache state / disk on the runner; inspect runner network egress to huggingface.co at the failure time.
• [LOW] Filelock deadlock (downloader rank died/wedged while holding the lock, leaving the other 7 ranks blocked forever).
  • Supporting: 7 ranks are blocked in _poll_until_acquired; the downloader's native stack repeatedly fails to sample (failed to get os threadid).
  • Disconfirming: the downloader's Python stack is cleanly inside hf_hub_download, i.e. it is alive and inside the network call, not crashed — more consistent with a slow download than a dead lock-holder.
  • How to verify: check whether the download cache dir contained partial .incomplete files after the run; bisect is not warranted (no code change in the weight path).
• No hypothesis reaches HIGH; this is most likely an infra/download-budget issue rather than an sglang code regression (weight-loading code path unchanged across the pass→fail window).

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs is:pr is:open grok1 timeout (0 results) and is:pr is:open grok accuracy (5 results, none addressing grok1-fp8 download/timeout). No existing PR targets this failure.

Suggested Triage Steps

• Confirm cache state: check whether lmzheng/grok-1 is present in the persistent HF cache on the MI30x runner pool; if not, pre-seed it or mount a shared model cache so cold downloads don't run inside the timed step.
• Re-run the Grok1-FP8 step in isolation on the same runner with the cache warm; if it passes quickly, this confirms the cold-download hypothesis.
• Consider raising --timeout-per-file for test_grok1_fp8_eval_amd.py (currently 3600s, est_time only 1500s) to absorb cold-download latency, or set GROK1_MODEL_PATH/download_dir to a persistent local path.
• Inspect runner network egress to huggingface.co around 21:28–22:27 UTC (the registry timeout at job start suggests intermittent network issues on this host).
• Since the grok weight-loading code is unchanged in the regression window, a git bisect is not recommended — focus on the runner environment, not sglang commits.

Status

• Persistence: Recurring/intermittent — failed Jun 15 and Jun 18 with identical signature; passed Jun 13–14; Jun 16–17 runs were cancelled (no signal).
• Scope: One step (Accuracy Test ROCm 7.2 (8-GPU Grok1-FP8)) in one job (nightly-accuracy-8-gpu-rocm720); the gpt-oss suite in the same job passes. Blocks the Grok1-FP8 nightly accuracy signal on AMD ROCm 7.2.
• In-flight fix: ❌ none found — needs runner cache/network investigation (likely infra, not a code fix).

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-06-19 20:30 UTC)

[amd-bot]

nightly-test-amd-rocm720.yml — Per-Job Analysis (continued 2/4)

nightly-8-gpu-kimi-k26-rocm720 — failed step(s): Accuracy Test ROCm 7.2 (8-GPU Kimi-K2.6)

Commit Info

• sglang: f7632ef86
• aiter: 7d604afe5 (v0.1.14rc1.dev240+g7d604afe5, source: dev image preserved — AITER_COMMIT_OVERRIDE empty, dev/patched version detected)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi30x/test_kimi_k26_eval_amd.py	N/A (whole-file timeout; server never reached ready)	✗ TIMEOUT after 3600 seconds → exit 255	Log

Failure Summary

The Kimi-K2.6 accuracy eval job timed out after the full 3600s budget. The sglang server never finished launching: it was stuck downloading the Kimi-K2.6 weights from HuggingFace. The model load began at 20:57:33, TP1 acquired the download lock at 20:57:34, and the HF Xet download thread remained stuck for the entire ~59 minutes until the test was killed at 21:56:34.

Regression Status

New failure (first failing completed run for this test file since it last passed Jun 15).

Recent history of test_kimi_k26_eval_amd.py in job nightly-8-gpu-kimi-k26-rocm720 (branch=main, event=schedule, completed runs only):

Date	Run	Job	Test File Status	Error
Jun 18	27780200720	job	❌ Failed	TIMEOUT 3600s (weight download stall)
Jun 15	27568866028	job	✅ Passed	—
Jun 14	27507292665	job	✅ Passed	—
Jun 13	27474496879	job	✅ Passed	—

(Jun 16 27640284588 and Jun 17 27710084518 workflow runs were cancelled at the workflow level — not comparable, excluded.)

Failure Cluster

HuggingFace Xet weight-download stall during server launch (model never loads → 3600s test timeout)

Facts (verified observations, no interpretation)

• [FACT] Test killed by harness: ✗ TIMEOUT: .../test_kimi_k26_eval_amd.py after 3600 seconds, Fail. Time elapsed: 3605.02s, then ##[error]Process completed with exit code 255 (log:9331). Harness noted: "Exit code 255 indicates test failure (not network issue), not retrying."
• [FACT] Server launch reached weight loading: [20:57:33 TP1] Load weight begin. avail mem=245.26 GB; [20:57:34 TP1] [CI Download] Process 3304 ACQUIRED lock for moonshotai/Kimi-K2.6.
• [FACT] Across every 5-min py-spy watchdog snapshot (21:02, 21:07, 21:12, 21:17, 21:22, 21:27, … through 21:52), the single active download thread ThreadPoolExecutor-3_0 is parked in xet_get (huggingface_hub/file_download.py:566) under ci_download_with_validation_and_retry (model_loader/ci_weight_validation.py:1955) → snapshot_download. The other 7 TP schedulers are all blocked in filelock _poll_until_acquired waiting on TP1's download lock.
• [FACT] Main process never passed _wait_for_scheduler_ready (engine.py:1358); all "Scheduler/TokenizerManager watchdog timeout" lines are soft=True (warnings, not crashes).
• [FACT] hf-xet 1.5.1 and huggingface_hub Xet backend in use; weights served over the Xet protocol.
• [FACT] Tiktoken/config files were cache-hit (/sgl-data/hf-cache/hub/models--moonshotai--Kimi-K2.6/snapshots/7eb5002f.../); the stall is on the safetensors weight shards, not tokenizer files.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [HIGH] HuggingFace Xet weight download stalled/hung (network or HF Xet backend), so the server never became ready within the 3600s budget.
  • Supporting: the same xet_get thread is the only active download worker and is stuck in identical stack frames across the entire ~50-min span of watchdog snapshots; no download-progress advance, no completion, no Load weight end line ever appears.
  • Disconfirming: no explicit network exception/timeout was logged (the harness even classified exit 255 as "not network issue"), so a slow-but-progressing download cannot be 100% excluded from the log alone (no byte-progress counter is printed).
  • How to verify: re-run the job; if it passes with a warm cache, confirms a transient download stall. Inspect /sgl-data/hf-cache/hub/models--moonshotai--Kimi-K2.6 shard sizes on the runner; check HF Xet status for the window.
• [LOW] sglang code regression in the load path.
  • Supporting: today is the first failing completed run.
  • Disconfirming: the stall is entirely inside huggingface_hub/hf_xet download code, not sglang model code; the same job passed Jun 13–15; the hang is at network I/O, not in kimi_k25.py/deepseek_v2.py weight-application logic. This hypothesis is unlikely.
• [LOW] aiter-related — not credible: failure occurs before any kernel executes (weights never load), and aiter is unchanged (dev image preserved, no override).

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for Kimi-K2.6 timeout, xet download timeout, weight download hang; no matching fix PR (results returned only unrelated PRs).

Suggested Triage Steps

• Re-run nightly-8-gpu-kimi-k26-rocm720 first — if it passes with a populated cache, this confirms a transient HF Xet download stall and no code action is needed.
• On the runner, check whether /sgl-data/hf-cache/hub/models--moonshotai--Kimi-K2.6 is fully populated (compare shard count/sizes vs HF repo); a partially-downloaded snapshot can cause repeated lock-held stalls.
• Consider pre-seeding/warming the Kimi-K2.6 weights into the shared /sgl-data/hf-cache before the eval, or setting HF_HUB_DISABLE_XET=1 on this runner as an A/B to isolate the Xet backend, given the stall is specifically inside xet_get.
• If download stalls recur, evaluate adding a wall-clock timeout/abort to ci_download_with_validation_and_retry so a stuck Xet fetch fails fast and retries instead of consuming the entire 3600s test budget.

Status (factual, no priority assignment)

• Persistence: 1 completed run (new on Jun 18; passed Jun 13–15; Jun 16–17 workflow runs cancelled).
• Scope: this job only (nightly-8-gpu-kimi-k26-rocm720); failure is at server-launch weight download, before any test logic or GPU kernel runs.
• Blocked work: Kimi-K2.6 8-GPU accuracy signal on AMD ROCm 7.2.
• In-flight fix: ❌ none found in open PRs.

nightly-8-gpu-glm51-rocm720 — failed step(s): Accuracy Test ROCm 7.2 (8-GPU GLM-5.1 DSA)

Commit Info

• sglang: f7632ef86
• aiter: 7d604afe5 (v0.1.14rc1.dev240+g7d604afe5; source: dev image preserved, matches Dockerfile default — [CI-AITER-CHECK] line 3524)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi30x/test_glm51_eval_amd.py	N/A (whole-file run, no test reached — server stuck loading weights)	TIMEOUT after 3600s → exit code 255	Log

Failure Summary

The single test file test_glm51_eval_amd.py (model zai-org/GLM-5.1-FP8, TP=8) timed out after 3600 s and the job exited with code 255. The server never finished startup: all 8 scheduler ranks were stuck in model-weight loading. No eval/accuracy assertion was ever reached.

Regression Status

New regression (infrastructure-class). This exact job nightly-8-gpu-glm51-rocm720 passed in every recent completed scheduled run; the two intervening days were cancelled (not real failures).

Recent history of test_glm51_eval_amd.py in job nightly-8-gpu-glm51-rocm720 (branch main, event schedule, completed runs only):

Date	Run	Job	Test File Status	Failed Function	Error
Jun 18	27780200720	nightly-8-gpu-glm51-rocm720	❌ Failed	— (timeout)	TIMEOUT 3600s, exit 255
Jun 17	27710084518	—	⏭️ Run cancelled	—	(excluded)
Jun 16	27640284588	—	⏭️ Run cancelled	—	(excluded)
Jun 15	27568866028	nightly-8-gpu-glm51-rocm720	✅ Passed	—	—
Jun 14	27507292665	nightly-8-gpu-glm51-rocm720	✅ Passed	—	—
Jun 13	27474496879	nightly-8-gpu-glm51-rocm720	✅ Passed	—	—
Jun 12	27434236636	nightly-8-gpu-glm51-rocm720	✅ Passed	—	—

First observed failure: 2026-06-18. Last known pass: 2026-06-15. (Jun 16–17 runs cancelled, excluded per completed-runs-only rule.)

Failure Cluster

HuggingFace weight-download hang during model load (Xet backend xet_get stall, all TP ranks blocked on the download filelock)

Facts (verified observations)

• python3 .../test_glm51_eval_amd.py launched 20:56:05; server_args=ServerArgs(model_path='zai-org/GLM-5.1-FP8', ... tp_size=8, watchdog_timeout=1200.0, soft_watchdog_timeout=300, download_dir=None) at 20:56:36 (line 3579).
• Load weight begin. avail mem≈244 GB at 20:57:13 for all 8 ranks (lines 3751–3759). Torch distributed init completed fine (~19 s) — GPUs/RCCL are healthy.
• TP2 (PID 3304) acquired the CI download lock: [CI Download] Process 3304 using lock file: /sgl-data/hf-cache/hub/.sglang_locks/download_0670781a6cf344bd.lock (line ~3760).
• py-spy of TP2 at 21:51 shows it actively downloading: load_model → download_weights_from_hf (weight_utils.py:593) → ci_download_with_validation_and_retry (ci_weight_validation.py:1955) → snapshot_download (_snapshot_download.py:467) → thread_map, with worker thread ThreadPoolExecutor-2_0 stuck in hf_hub_download → _hf_hub_download_to_cache_dir → _download_to_tmp_and_move → xet_get (huggingface_hub/file_download.py:566) (lines 8905–8975).
• The other 7 ranks (TP0/1/3/4/5/6/7) all idle in download_weights_from_hf (weight_utils.py:554) → filelock.acquire → _poll_until_acquired (filelock/_api.py:552) → select — blocked waiting for TP2's lock (e.g. TP5 dump lines 8550–8598).
• Same stuck stack at first watchdog dump 21:01:46 (lines 3826–3844) and last dump 21:51 → process was hung in download for the entire ~55 min window.
• Soft watchdog (300 s) fired repeatedly but self.soft=True, so it only dumped py-spy and did not kill; the 3600 s test-harness timeout finally terminated the run (✗ TIMEOUT ... after 3600 seconds, line 8984).
• No explicit network exception (ConnectionError/HTTPError/timeout) was raised in the log — the Xet transfer simply hung without erroring.

Hypothesised Causes (with confidence)

• [HIGH] HuggingFace weight download stalled in the Xet transfer backend (xet_get). The fault is at the download/transfer layer, not in sglang model code.
  • Supporting: TP2's live stack is inside huggingface_hub Xet download (xet_get) for the entire window; all other ranks are merely blocked on TP2's download filelock; GPU/dist init succeeded; the same job + same test passed on the 4 prior completed runs with no code change to the GLM/deepseek load path implicated.
  • Disconfirming: no explicit HF network error logged (a hard CDN failure would normally raise) — consistent with a silent hang rather than a hard outage, but cannot fully rule out a slow-but-progressing transfer that just exceeded 3600 s on a cold cache.
  • How to verify: re-run the job; check whether /sgl-data/hf-cache was pre-populated with zai-org/GLM-5.1-FP8; inspect runner network/Xet egress during the window; try HF_HUB_DISABLE_XET=1 (force classic transfer) on a re-run.
• [LOW] Cold/empty HF cache on this runner forcing a full fresh download of the large FP8 checkpoint that simply didn't finish within 3600 s.
  • Supporting: download_dir=None, fresh snapshot_download observed; passing runs may have reused a warm cache.
  • Disconfirming: 55 min with apparently zero file completions points to a hang, not merely slow throughput.
  • How to verify: compare cache state / per-file completion logs against a passing run on the same runner.
• [SPECULATION] Stale .sglang_locks/download_0670781a6cf344bd.lock left by a prior cancelled run (Jun 16/17 were cancelled) wedging the download path.
  • Supporting: two immediately-preceding runs were cancelled, which can abort a download mid-lock.
  • Disconfirming: TP2 itself holds and is inside the download (not blocked on a foreign lock), so a stale lock is not what's blocking the active downloader.
  • How to verify: check for orphaned lock files in /sgl-data/hf-cache/hub/.sglang_locks/ on the runner.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for xet+download (0 hits) and weight download timeout (only unrelated keyword matches, e.g. #28260, #16638). No PR targets HF/Xet download-hang handling.

Suggested Triage Steps

• Re-run nightly-8-gpu-glm51-rocm720 and check whether it reproduces — a single Xet stall is often transient infra.
• On the runner, confirm /sgl-data/hf-cache has a complete zai-org/GLM-5.1-FP8 snapshot and clear any orphaned files in .sglang_locks/ left by the Jun 16/17 cancelled runs.
• If it reproduces, re-run with HF_HUB_DISABLE_XET=1 to bypass the Xet transfer path and confirm whether Xet is the culprit.
• Consider (maintainer call) adding a download-level timeout/retry around ci_download_with_validation_and_retry so a hung transfer fails fast instead of consuming the full 3600 s budget — the soft watchdog currently only dumps py-spy and never aborts a stuck download.

Status

• Persistence: First failure 2026-06-18 (1 completed run); job passed Jun 12–15.
• Scope: This job only (nightly-8-gpu-glm51-rocm720); failure is in the HF download path, not GLM/sglang inference code.
• Blocked work: GLM-5.1-FP8 8-GPU ROCm 7.2 nightly accuracy signal for this run.
• In-flight fix: ❌ none found — likely transient infra; needs re-run + runner cache/network check before any code change.

nightly-8-gpu-qwen3-235b-rocm720 — failed step(s): Accuracy Test + Performance Test ROCm 7.2 (8-GPU Qwen3)

Commit Info

• sglang: f7632ef86
• aiter: 7d604afe5 — v0.1.14rc1.dev240+g7d604afe5 (source: dev image preserved; Dockerfile default also 7d604afe5, no override)
• container image: rocm/sgl-dev:v0.5.13.post1-rocm720-mi30x-20260618

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/test_qwen3_instruct_fp8.py	setUpClass	Fatal Python error: Segmentation fault during torch distributed init (TP1) → server exit code 1	Log
test/registered/amd/test_qwen3_instruct.py	setUpClass (TestQwen3Instruct2507)	RuntimeError: Rank 1 scheduler died during initialization (exit code: -11) (SIGSEGV) → server exit code 1	Log

Failure Summary

Both Qwen3-235B test files in job nightly-8-gpu-qwen3-235b-rocm720 failed because the SGLang server crashed during startup. In each case the TP rank-1 scheduler subprocess died with a segmentation fault (signal 11) while initializing the torch distributed world group, before any model forward pass. Ran 0 tests — the crash is in setUpClass/server launch, not in test assertions.

Regression Status

New regression (segfault cluster first observed today). The FP8 test passed as recently as Jun 17; the non-FP8 test failed Jun 17 but with a different symptom (3600 s timeout/hang, not a segfault), so today is the first segfault-during-init occurrence for both.

Recent history of test_qwen3_instruct_fp8.py / test_qwen3_instruct.py in job nightly-8-gpu-qwen3-235b-rocm720 (branch=main, event=schedule, completed runs only):

Date	Run	Job	Test File Status	Failed Function	Error
Jun 18	27780200720	nightly-8-gpu-qwen3-235b-rocm720	❌ both Failed	setUpClass (both files)	SIGSEGV in torch distributed init
Jun 17	27710084518	nightly-8-gpu-qwen3-235b-rocm720	⚠️ 1/2 (fp8 ✅, instruct ❌)	test_qwen3_instruct.py	timeout after 3600s (hang, TP5) — different symptom
Jun 16	27640284588	nightly-8-gpu-qwen3-235b-rocm720	✅ Passed	—	—
Jun 15	27568866028	nightly-8-gpu-qwen3-235b-rocm720	✅ Passed	—	—
Jun 14	27507292665	nightly-8-gpu-qwen3-235b-rocm720	✅ Passed	—	—
Jun 13	27474496879	nightly-8-gpu-qwen3-235b-rocm720	✅ Passed	—	—

First segfault: today (Jun 18, 2026). Last fully-green run: Jun 16 (27640284588). Note Jun 17 run-level was cancelled for the sibling mxfp4 job but this job ran to completion (1/2 passed).

Failure Cluster

Segfault (SIGSEGV) in torch HIP fill kernel during distributed world-group init

Facts (verified observations, no interpretation)

• Crash thread (verbatim, today's log lines ~3584-3618): Fatal Python error: Segmentation fault, current-thread Python frames:
  • parallel_state.py:301 __init__ → parallel_state.py:1509 init_world_group → parallel_state.py:1844 init_distributed_environment → model_runner.py:1229 init_torch_distributed → tp_worker.py:363 _init_model_runner → scheduler.py:778 init_tp_model_worker.
  • Native frames (!!!!!!! Segfault encountered !!!!!!!): at::native::fill_kernel_cuda → at::native::fill_out → wrapper_CUDA_Scalar_fill_.
• parallel_state.py:301 is active_ranks = torch.ones(len(ranks), dtype=torch.int32, device=self.device) — the first GPU tensor allocation in world-group init; wrapper_CUDA_Scalar_fill_ is exactly the kernel that backs torch.ones(device=cuda). The segfault is in libtorch HIP code, not SGLang Python.
• Both failing servers crashed on TP1 / Rank 1 specifically ("Init torch distributed begin" for TP1 immediately precedes the segfault; non-FP8 server reports Rank 1 scheduler died ... exit code: -11).
• aiter version is identical across the last-green (Jun 17) and failing (Jun 18) runs: v0.1.14rc1.dev240+g7d604afe5 (verified via [CI-AITER-CHECK] markers in both logs).
• Container image differs only by daily date tag: ...-20260617 (Jun 17) vs ...-20260618 (today).
• 58 commits in the window 8aaca72c2..f7632ef86. None modifies parallel_state.py, model_runner.py distributed-init path, or the GPU device-setup path (verified via git log/git show --stat). The only device-related commit, 74e2e48c8 (CpuDeviceMixin), touches CPU platform code only.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [MEDIUM] Infrastructure/runtime issue in the daily ROCm 7.2 container image (-20260618): a SIGSEGV in libtorch's HIP fill_kernel_cuda on the very first device tensor allocation is a classic torch/ROCm/driver/runtime-mismatch symptom rather than an application logic bug.
  • Supporting: crash is in libtorch native HIP code, not SGLang Python; it happens before any model/aiter forward op; only this AMD rocm720 nightly job is implicated; image date tag is the one variable that changed alongside the regression.
  • Disconfirming: I did not diff the image contents (torch/ROCm package versions between -20260617 and -20260618), so "image regression" is inferred, not proven; the image also tracks sglang HEAD, so a Python-side change can't be fully excluded by image-date alone.
  • How to verify: re-run the job pinned to image ...-20260617 with today's sglang checkout; compare pip show torch / ROCm runtime versions between the two image tags.
• [LOW] An SGLang commit in 8aaca72c2..f7632ef86 altered TP-1 process/device setup indirectly.
  • Supporting: regression timing aligns with the window; failure is rank-1-specific.
  • Disconfirming: no commit in the window touches parallel_state.py:301 or the distributed-init/device path; the crashing code (torch.ones at line 301) is pre-existing (local git blame attribution to 9c53853ea is unreliable — it is the shallow-clone boundary/graft commit, not the true author).
  • How to verify: bisect the window on an MI30x rocm720 8-GPU runner, re-running test_qwen3_instruct_fp8.py.
• [LOW] aiter dev build incompatibility.
  • Supporting: attention backend is aiter.
  • Disconfirming: aiter version is byte-identical between the last-green Jun 17 run and today (verified via [CI-AITER-CHECK]); the crash precedes any aiter kernel call.
  • How to verify: not actionable — aiter is unchanged across pass/fail.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for segfault+distributed and parallel_state+segfault; only unrelated hits (e.g. #8573). No PR targets this Qwen3-235B rocm720 distributed-init segfault.

Suggested Triage Steps

• Re-run nightly-8-gpu-qwen3-235b-rocm720 pinned to the previous-day image rocm/sgl-dev:v0.5.13.post1-rocm720-mi30x-20260617 with today's sglang checkout — if it passes, the regression is in the new container image (torch/ROCm), not sglang Python.
• Diff package versions (torch, ROCm runtime, hip libraries) between image tags ...-20260617 and ...-20260618 to find what changed in the daily rebuild.
• If image is exonerated, bisect 8aaca72c2..f7632ef86 on an MI30x rocm720 8-GPU runner re-running test_qwen3_instruct_fp8.py.
• Confirm this is not a transient runner/GPU fault by retrying once — a one-shot SIGSEGV in distributed init can also be a flaky GPU/runner state; two consecutive segfault runs would strengthen the regression verdict (today is the first).

Status (factual, no priority assignment)

• Persistence: Segfault cluster first observed today (Jun 18, 2026); 1 completed run so far. The non-FP8 test has been red since Jun 17 but via a different symptom (hang/timeout).
• Scope: 2 test files in 1 job (nightly-8-gpu-qwen3-235b-rocm720) in workflow nightly-test-amd-rocm720.yml. Sibling nightly-8-gpu-mi35x-qwen3-235b-mxfp4-rocm720 has its own separate failures.
• Blocked work: Qwen3-235B-A22B-Instruct-2507 (BF16 + FP8) accuracy/serving signal on AMD MI30x ROCm 7.2.
• In-flight fix: ❌ none found — needs new investigation (start with image A/B).

nightly-8-gpu-qwen35-rocm720 — failed step(s): Accuracy Test ROCm 7.2 (8-GPU Qwen 3.5)

Commit Info

• sglang: f7632ef86
• aiter: N/A — no [CI-AITER-CHECK] markers in this nightly job log; aiter is loaded from the prebuilt image ([aiter] import [module_aiter_core] at 21:05:38) but no version marker is emitted.

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi30x/test_qwen35_eval_amd.py	N/A (server never finished startup — no test body reached)	✗ TIMEOUT after 3600 seconds → Process completed with exit code 255	Log

Failure Summary

The single accuracy test test_qwen35_eval_amd.py (model Qwen/Qwen3.5-397B-A17B, TP8) timed out after the harness's 3600 s budget. The server never finished startup: all 8 TP scheduler processes hung inside init_tokenizer while downloading the tokenizer/processor from the HuggingFace Hub, blocked on a HF cache file lock. No assertion or accuracy failure was reached — the process was killed at the timeout.

Regression Status

New regression / suspected infrastructure hang — first failure of this test file; the previous 5 completed scheduled runs all passed.

Recent history of test_qwen35_eval_amd.py in job nightly-8-gpu-qwen35-rocm720 (branch main, event schedule, completed runs only):

Date	Run	Job	Test File Status	Failed Function	Error
Jun 18	27780200720	nightly-8-gpu-qwen35-rocm720	❌ Failed	N/A (startup hang)	TIMEOUT 3600s → exit 255
Jun 17	27710084518	nightly-8-gpu-qwen35-rocm720	✅ Passed	—	—
Jun 16	27640284588	nightly-8-gpu-qwen35-rocm720	✅ Passed	—	—
Jun 15	27568866028	nightly-8-gpu-qwen35-rocm720	✅ Passed	—	—
Jun 14	27507292665	nightly-8-gpu-qwen35-rocm720	✅ Passed	—	—
Jun 13	27474496879	nightly-8-gpu-qwen35-rocm720	✅ Passed	—	—

First observed failure: 2026-06-18. Last known pass: 2026-06-17 (27710084518).

Failure Cluster

HuggingFace Hub tokenizer/processor download hang on file-lock during server startup (TP8 scheduler init).

Facts (verified observations, no interpretation)

• All 8 TP scheduler processes (TP0–TP7, PIDs 3367–3374) share the identical py-spy stack, idle in select (libc.so.6):
  _poll_until_acquired (filelock/_api.py:552) → acquire (filelock/_api.py:506) → WeakFileLock (huggingface_hub/utils/_fixes.py:99) → _hf_hub_download_to_cache_dir (huggingface_hub/file_download.py:1237) → hf_hub_download → cached_file (transformers/utils/hub.py:278) → from_pretrained (tokenization_auto.py:738) → _load_tokenizer_from_pretrained (processing_utils.py:1490) → from_pretrained (processing_auto.py:440) → get_processor (utils/hf_transformers/processor.py:209) → init_tokenizer (scheduler.py:684) → __init__ (scheduler.py:419).
• Timeline: test Begin 21:05:02; server_args logged 21:05:33; TP processes pinned to CPUs 21:05:42–43; first soft-watchdog py-spy dump (300 s) at 21:10:43 already shows the filelock hang; watchdog repeats every 5 min through 22:00:48; kill_process_tree 22:05:02; ✗ TIMEOUT … after 3600 seconds, Fail. Time elapsed: 3605.03s, exit code 255 (Exit code 255 indicates test failure (not network issue), not retrying).
• Server config: model_path='Qwen/Qwen3.5-397B-A17B', tp_size=8, download_dir=None (default HF cache), watchdog_timeout=1200.0, soft_watchdog_timeout=300.
• Independent network signal earlier in the same run: Local docker registry pull failed; falling back to public registry / Error response from daemon: Get "https://10.245.143.50:5000/v2/": context deadline exceeded (line 1250).
• The hang is entirely inside huggingface_hub/filelock/transformers; no sglang frame is the blocking frame (sglang frames are only callers scheduler.py:419/684, processor.py:209).

Hypothesised Causes (with confidence, with disconfirming evidence)

• [MEDIUM] Infrastructure / network: HF Hub tokenizer/processor download stalled, so the rank holding the HF cache .lock never completed and the other 7 ranks blocked on _poll_until_acquired until the 3600 s timeout.
  • Supporting: blocking frame is 100% in huggingface_hub.file_download / filelock; the same runner showed a network failure earlier (docker registry context deadline exceeded); all 8 ranks share one identical download/lock stack; the test passed in the previous 5 scheduled runs with no code change to the download path.
  • Disconfirming: the exact lock holder's progress is not logged, so an outright HF outage vs. a multi-process lock deadlock cannot be distinguished from this log alone; no explicit ConnectionError/Retrying/429/503 line was emitted (the process simply blocked).
  • How to verify: re-run the job; check HF Hub reachability from the MI30x runner; pre-stage the model into the HF cache / set --download-dir to a shared cache so the 8 ranks don't race the download.
• [LOW] sglang code regression in the Jun 17→18 delta (58 commits, 8aaca72c2..f7632ef86).
  • Supporting: failure window is exactly one nightly cycle.
  • Disconfirming: no commit in the delta touches utils/hf_transformers/processor.py, get_processor, init_tokenizer's download path, or the HF Hub call chain (delta touches tokenizer_manager.py/batch_result_processor.py, which are not in the blocking stack); the workflow change f5b041622 does not alter the qwen35 job's model or args. A pure download/file-lock hang is not a plausible product of these diffs.
  • How to verify: not actionable until the infra hypothesis is ruled out; if reproducible after a clean cache, bisect the delta.

No hypothesis reaches HIGH; the blocking frame is third-party (HF Hub), so this is most consistent with an environmental download stall rather than an sglang regression.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for watchdog tokenizer download (7 hits, all unrelated: NPU tests, kernel ports, server_args refactor) and qwen35 timeout (1 hit, #24651 — an FP8 RMSNorm perf PR, not this hang). No PR addresses the tokenizer/processor download hang.

Suggested Triage Steps

• Re-run nightly-8-gpu-qwen35-rocm720 once; if it passes, this was a transient HF Hub download stall — track as flaky, no code action.
• Pre-stage Qwen/Qwen3.5-397B-A17B into the runner's HF cache (or point --download-dir at a shared, pre-populated cache) so the 8 TP ranks don't all race the same download/file-lock on a cold cache.
• Verify HuggingFace Hub connectivity from the MI30x rocm720 runner around the run window (the same job also hit a docker-registry context deadline exceeded, suggesting node-level network trouble).
• If the hang reproduces on a warm cache, only then bisect 8aaca72c2..f7632ef86 — but no commit there touches the download path, so a code cause is unlikely.

Status

• Persistence: First failure on 2026-06-18; previous 5 completed scheduled runs (Jun 13–17) all passed.
• Scope: 1 job / 1 test file (nightly-8-gpu-qwen35-rocm720 → test_qwen35_eval_amd.py). The sister job nightly-8-gpu-mi35x-qwen35-rocm720 passed in prior runs; the rest of today's run was still in progress at snapshot time.
• Blocked work: Qwen3.5-397B-A17B nightly accuracy signal on AMD MI30x rocm720.
• In-flight fix: ❌ none found — needs a re-run / cache pre-stage / runner network check, not a code PR.

nightly-8-gpu-mi35x-qwen3-235b-mxfp4-rocm720 — failed step(s): Accuracy Test + Performance Test MI35x ROCm 7.2 (8-GPU Qwen3-235B-MXFP4)

Commit Info

• sglang: f7632ef8
• aiter: 7d604afe5 — v0.1.14rc1.dev240+g7d604afe5 (source: dev/patched image preserved; matches Dockerfile gfx950 default, no override)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/test_qwen3_instruct_mxfp4.py	TestQwen3Instruct2507MXFP4.setUpClass	RuntimeError: Rank 0 scheduler died during initialization (exit code: -6) — server never came up	Log

Failure Summary

The server for TestQwen3Instruct2507MXFP4 (Qwen3-235B-A22B-Instruct-2507-mxfp4, --tp 4 --ep 2 --attention-backend aiter) never finished launching: the rank-0 scheduler aborted (SIGABRT, exit -6) during decode CUDA/HIP graph capture, so setUpClass raised Exception: Server process exited with code 1 and the whole test file returned exit code 1. No GSM8K/accuracy assertion was ever reached — this is a startup crash, not a numeric regression.

Regression Status

Known recurring failure / Never-passed in queryable window. This job (nightly-8-gpu-mi35x-qwen3-235b-mxfp4-rocm720) has failed with the identical signature (HIP capturing-stream error → scheduler exit -6) in every completed scheduled run back to 2026-06-07. No green run found in the queryable window.

Recent history of test_qwen3_instruct_mxfp4.py (TestQwen3Instruct2507MXFP4.setUpClass) in job nightly-8-gpu-mi35x-qwen3-235b-mxfp4-rocm720 (branch main, event schedule, completed runs only):

Date	Run	Job	Test File Status	Failed Function	Error
Jun 19	27780200720	job	❌ Failed	setUpClass	scheduler exit -6 (HIP capturing stream)
Jun 17	27710084518	—	⏭️ Run cancelled	—	(excluded)
Jun 16	27640284588	—	⏭️ Run cancelled	—	(excluded)
Jun 15	27568866028	job	❌ Failed	setUpClass	scheduler exit -6 (HIP capturing stream)
Jun 14	27507292665	job	❌ Failed	setUpClass	scheduler exit -6 (HIP capturing stream)
Jun 13	27474496879	job	❌ Failed	setUpClass	scheduler exit -6 (HIP capturing stream)
Jun 12	27434236636	job	❌ Failed	setUpClass	scheduler exit -6
Jun 07–11	27368779328 … 27100187622	(5 runs)	❌ Failed	setUpClass	all failed

First observed failure: at or before 2026-06-07 (start of queryable window). Last known passing date: none found.

Failure Cluster

HIP "operation not permitted when stream is capturing" during decode CUDA graph capture (EP all-reduce inside capturing stream)

Facts (verified observations, no interpretation)

• Fatal error verbatim: HIP error: operation not permitted when stream is capturing (hipErrorStreamCaptureUnsupported) and operation not permitted on an event last recorded in a capturing stream (hipErrorCapturedEvent), raised from the RCCL/NCCL watchdog (ProcessGroupNCCL.cpp:2057), terminating the process with Fatal Python error: Aborted → exit -6 (log L4885, L4904).
• Crash call chain (verified from the Python fault stack, log ~L4835):
  decode_cuda_graph_runner.py:772 capture → decode_cuda_graph_runner.py:887 run_once → qwen3_moe.py:846/298/333 forward / forward_normal → communication_op.py:84 moe_expert_parallel_all_reduce → parallel_state.py:659 all_reduce → parallel_state.py:168 inplace_all_reduce → parallel_state.py:776 _all_reduce_in_place → torch.distributed all_reduce.
• The all-reduce that crashed went through inplace_all_reduce (the plain torch.distributed / RCCL path) — i.e. no custom outplace method (qr / pymscclpp / torch_symm_mem / pynccl) was selected for the MoE EP group (parallel_state.py:629-660). Plain RCCL all-reduce is not HIP-graph-capture-safe.
• Reproducibility: identical signature in 8/8 completed scheduled runs (2026-06-07 → 2026-06-19); 18-22 capturing-stream error lines per failing log.
• Config: --tp 4 --ep 2 --attention-backend aiter, SGLANG_USE_AITER=1 (test_qwen3_instruct_mxfp4.py:30-46), model amd/Qwen3-235B-A22B-Instruct-2507-mxfp4, runner GPU_ARCH=mi35x (gfx950), ROCm 7.2.
• aiter version unchanged across the failing window (dev image v0.1.14rc1.dev240+g7d604afe5 preserved every run).

Hypothesised Causes (with confidence, with disconfirming evidence)

• [MEDIUM] The MoE expert-parallel all-reduce (get_moe_ep_group().all_reduce) falls through to the plain RCCL inplace_all_reduce during decode HIP-graph capture, and RCCL all-reduce is not capture-safe on this MI35x/ROCm 7.2 EP group → hipErrorStreamCaptureUnsupported.
  • Supporting: the fault stack shows inplace_all_reduce/_all_reduce_in_place (not any custom outplace method) being called from inside decode_cuda_graph_runner.capture; the error is exactly the HIP "stream is capturing" guard; the MoE EP group apparently has no capture-safe communicator selected at parallel_state.py:629-660.
  • Disconfirming: the exact regression window is not bounded (test never passed in the queryable window), so this could be a config that was never graph-capture-compatible on MI35x rather than a code change that broke it — cannot point to a specific commit.
  • How to verify: re-run the server with --disable-cuda-graph (or disable decode graph capture) on MI35x/ROCm 7.2; if startup succeeds, the capture path is confirmed. Then check whether the moe-ep group has a custom allreduce communicator enabled on gfx950.
• [LOW] ROCm 7.2 / RCCL or the preserved aiter dev build interacts badly with HIP graph capture for the EP group.
  • Supporting: failure is specific to the rocm720 MI35x config.
  • Disconfirming: aiter version is unchanged across all failing runs (so aiter is not a new trigger); error originates in torch.distributed/RCCL, not aiter code.
  • How to verify: compare against the non-rocm720 (nightly-test-amd.yml) sister config if one runs this model, and check whether plain RCCL allreduce-in-capture fails there too.
• No hypothesis reaches HIGH confidence; needs the --disable-cuda-graph A/B above plus inspection of the moe-ep communicator selection on gfx950 to confirm.

In-flight Fix Check

• Searched open PRs with keywords qwen3 mxfp4 (26 hits, all feature/perf work — none addressing graph-capture allreduce) and cuda graph capture all_reduce (6 hits, none matching this EP capture crash).
• In-flight fix: ❌ none found in open PRs — needs new PR or investigation

Suggested Triage Steps

• Re-run the server launch on MI325/MI35x ROCm 7.2 with --disable-cuda-graph for amd/Qwen3-235B-A22B-Instruct-2507-mxfp4 (--tp 4 --ep 2 --attention-backend aiter); if it starts cleanly, the decode-graph-capture EP all-reduce is confirmed as the trigger.
• Inspect the MoE EP group's communicator selection (parallel_state.py:629-660) on gfx950 — determine why no capture-safe outplace method (qr / pynccl) is chosen so the plain RCCL inplace_all_reduce runs inside graph capture.
• If confirmed, fix direction (hypothetical): route the moe-ep all-reduce through a capture-safe communicator on ROCm during graph capture, or gate decode CUDA-graph capture off for the EP-MoE path on MI35x until supported.
• If graph capture cannot be made capture-safe here near-term, consider disabling decode CUDA graph for this nightly config so the rest of the GSM8K/accuracy signal can run.

Status (factual, no priority assignment)

• Persistence: Failing every completed scheduled run since at least 2026-06-07 (8/8 completed runs; runs on 06-16 and 06-17 were cancelled, not passing).
• Scope: 1 job (nightly-8-gpu-mi35x-qwen3-235b-mxfp4-rocm720) in nightly-test-amd-rocm720.yml. Blocks all Qwen3-235B-mxfp4 nightly signal (GSM8K accuracy + bs-1 speed) on MI35x ROCm 7.2.
• In-flight fix: ❌ none found in open PRs.

nightly-8-gpu-mi35x-deepseek-r1-hicache-rocm720 — failed step(s): Accuracy Test MI35x ROCm 7.2 (8-GPU DeepSeek-R1 HiCache)

Commit Info

• sglang: f7632ef8
• aiter: 7d604afe — v0.1.14rc1.dev240+g7d604afe5 (source: dev image preserved, Dev/patched version detected → skipping rebuild; no AITER_COMMIT_OVERRIDE)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/test_deepseek_r1_hicache_mi35x.py	N/A (server-launch accuracy benchmark, no pytest fn)	TP4 scheduler subprocess (pid=3315) crashed with exit code -6 (SIGABRT) during forward_extend; file timeout after 3600s	Log

Failure Summary

The single test file test_deepseek_r1_hicache_mi35x.py (DeepSeek-R1-0528, TP8, fp8 KV, aiter backend, hierarchical cache file backend) launched its server and reached ~68% (895/1314) of the accuracy benchmark, then one scheduler subprocess (TP4) aborted with SIGABRT. The server then stopped responding, the client flooded Connection error retries, and the run was killed by the 3600s per-file timeout (elapsed=3600, exit code 255).

Regression Status

Known recurring failure / Never-passed (in queryable window).

Recent history of test_deepseek_r1_hicache_mi35x.py in job nightly-8-gpu-mi35x-deepseek-r1-hicache-rocm720 (nightly-test-amd-rocm720.yml, branch main, event=schedule, completed runs only):

Date	Run	Test File Status	Failure Mode
Jun 19	27780200720	❌ Failed	TP4 SIGABRT (exit -6) → 3600s timeout
Jun 18	27710084518	❌ Failed	scheduler_0 crash (exit -6) → 3600s timeout
Jun 17	27640284588	❌ Failed	3600s timeout
Jun 16	27568866028	❌ Failed	exit code 1 (hipError in stack)
Jun 15	27507292665	❌ Failed	3600s timeout
Jun 14	27474496879	❌ Failed	3600s timeout

[FACT] No passing run exists in the queryable window (failed every scheduled run Jun 7 → Jun 19, 12 runs; the 2 newest non-failure runs were cancelled). First-observed-passing date: none found. Lookback range: 2026-06-07 → 2026-06-19.

Failure Cluster

GPU hang/abort during DeepSeek-R1 HiCache forward pass → scheduler SIGABRT → benchmark stall → 3600s file timeout.

Facts (verified observations)

• [FACT] Server launched (02:37) with: --tp 8 --kv-cache-dtype fp8_e4m3 --attention-backend aiter --page-size 64 --enable-hierarchical-cache --hicache-ratio 2 --hicache-io-backend kernel --hicache-write-policy write_through --hicache-storage-backend file --hicache-storage-prefetch-policy best_effort --watchdog-timeout 1200 (log L3852).
• [FACT] Soft watchdog timeouts at 02:37:31–02:37:37 (self.watchdog_timeout=300, self.soft=True) on TokenizerManager + all TP0–TP7 — these occur during the (slow) 671B model load; the server subsequently served to 68% of the benchmark, so they are startup warnings, not the fatal event.
• [FACT] CUDA-graph capture + aiter MLA kernels loaded successfully at 03:05 (Capturing batches (bs=512 …), [aiter] LoadKernel … mla_a8w8 … gfx950).
• [FACT] Fatal event at 03:14: Fatal Python error: Aborted (L20202) followed by Subprocess scheduler_4 (pid=3315) crashed with exit code -6 and SIGQUIT received … one child failed (L21106-area, 03:14:30).
• [FACT] The faulthandler dump was triggered by the watchdog (utils/watchdog.py:147 _watchdog_once, _watchdog_thread); the stuck worker stack is model_runner.py:3436 forward_extend → model_runner.py:3672 _forward_raw → model_runner.py:3545 forward → tp_worker.py:492 forward_batch_generation → scheduler.py:3163 run_batch → scheduler.py:1534 event_loop_overlap.
• [FACT] GPU coredump: Directory "/coredumps not writable or does not exist / GPU core dump failed printed at the abort — the HIP runtime attempted a GPU coredump, indicating a device-side fault/hang.
• [FACT] HiCache controller threads active in the dump: cache_controller.py prefetch_thread_func, prefetch_io_aux_func, backup_thread_func.
• [FACT] Rate limit exception so wait and retry … Connection error begins 03:14:39 (5675 occurrences) and the benchmark progress freezes at 895/1314 until kill_process_tree at 03:31:58; file marked ✗ TIMEOUT … after 3600 seconds.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [MEDIUM] GPU-side fault/hang in the DeepSeek-R1 HiCache forward path (the watchdog caught forward_extend not returning, and HIP attempted a GPU coredump → SIGABRT).
  • Supporting: hard-watchdog dump pins the worker in model_runner.forward_extend; GPU core dump failed printed at the abort; identical exit-code -6 crash recurred Jun 18; Jun 16 shows hipError in the abort stack.
  • Disconfirming: the exact faulting kernel is not in the (garbled, multi-process) dump; the abort could be the watchdog killing a slow-but-not-crashed forward rather than a true illegal-memory fault. No Memory access fault by GPU node line was captured.
  • How to verify: re-run test_deepseek_r1_hicache_mi35x.py on an MI35x (gfx950) container with AMD_LOG_LEVEL=3 and a writable /coredumps, and bisect the watchdog: raise --watchdog-timeout to see whether forward_extend eventually returns (true hang vs. slow).
• [LOW] HiCache file-backend prefetch/backup interaction stalls the scheduler loop (write_through + best_effort prefetch threads active at hang time).
  • Supporting: cache_controller.py prefetch/backup threads present in the dump; failure is specific to the HiCache-configured job.
  • Disconfirming: those threads are expected to be alive every step; their presence in a faulthandler snapshot is not evidence of a stall. The main thread is blocked in forward_extend, not in HiCache code.
  • How to verify: re-run with --disable-hierarchical-cache (or hicache-storage-backend removed) and compare whether the forward-pass abort persists.
• [LOW] aiter v0.1.14rc1.dev240+g7d604afe5 MLA kernel issue on gfx950.
  • Supporting: attention-backend is aiter; the hang is in the forward pass.
  • Disconfirming: aiter version is a preserved dev image (unchanged across these failing runs), yet the failure mode varies (timeout vs exit -6 vs exit 1) and predates this aiter pin in the window — points to an sglang-side or environmental issue, not the aiter delta.
  • How to verify: A/B with a known-good aiter pin on the same sglang SHA.

No hypothesis reaches HIGH; the multi-process faulthandler output is interleaved and the precise faulting kernel/assertion is not legible. Needs a clean single-rank repro.

In-flight Fix Check

• In-flight fix: ❌ none found targeting this specific MI35x DeepSeek-R1 HiCache timeout. Searches run: is:pr is:open hicache deepseek (returns DeepSeek-V4/PP/Mooncake HiCache PRs only — e.g. #27294, #27470, none for R1 MI35x timeout); is:pr is:open mi35x hicache; is:pr is:open deepseek r1 timeout amd. Several open hicache/AMD branches exist (e.g. fix/hicache-dsv4-none-crash, fix_hicache_with_spec) but none map to this failure.

Suggested Triage Steps

• Reproduce on MI35x (gfx950) in the nightly container with a writable /coredumps + AMD_LOG_LEVEL=3 to capture the actual GPU fault that is currently lost (GPU core dump failed).
• A/B the HiCache config: re-run test_deepseek_r1_hicache_mi35x.py with --disable-hierarchical-cache to confirm whether the forward_extend abort is HiCache-dependent or a base DeepSeek-R1/aiter-on-gfx950 issue.
• Raise --watchdog-timeout (or add per-step timing) to distinguish a true GPU illegal-access fault from a slow-forward watchdog kill.
• Because the file routinely exceeds the 3600s --timeout-per-file even when not crashing (est_time=5400.0 > 3600), consider whether the per-file timeout is mis-sized for this model — but treat the SIGABRT crash as the primary signal, not the timeout.
• If a single-rank repro is obtained, bisect sglang model_runner / DeepSeek forward changes against the last date this job was green (none in the 12-run window — may require querying older runs beyond the API page).

Status

• Persistence: Failing every completed scheduled run from 2026-06-07 through 2026-06-19 (12 runs); no green run in the queryable window.
• Scope: This job only — nightly-8-gpu-mi35x-deepseek-r1-hicache-rocm720 in nightly-test-amd-rocm720.yml (single test file).
• Blocked work: DeepSeek-R1 HiCache accuracy signal on AMD MI35x (ROCm 7.2) — no passing baseline.
• In-flight fix: ❌ none found in open PRs — needs new PR or investigation.

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-06-19 20:30 UTC)

[amd-bot]

amd-aiter-scout.yml — 17 failure(s)

Run started (UTC): 2026-06-18 20:52 UTC
Last scanned (UTC): 2026-06-19 09:30 UTC
Commits: sglang f83e4d5

Summary

Counts: 19 failures · 12 clusters · Origin: 3 aiter-caused · 9 pre-existing (sglang) · 7 unclear

#	Cluster	Job	Test File	Test Function	Origin	Status	Hypothesis (confidence)
1	Gate/concurrency cancel cascade	call-gate / pr-gate	N/A	N/A	unclear	new (extra subtree 1st run)	[HIGH] concurrency-group key omits rocm_version → default extra gate cancelled
2	Gate/concurrency cancel cascade	extra-a-test-1-gpu-large-amd	N/A	N/A	unclear	new, 0 steps	[MEDIUM] cancelled by gate cascade
3	Gate/concurrency cancel cascade	extra-a-test-1-gpu-small-amd	N/A	N/A	unclear	new, 0 steps	[LOW] cancelled by gate cascade
4	GPU segfault in distributed init (torch.ones)	nightly-8-gpu-deepseek-v31-rocm720	test/registered/amd/accuracy/mi30x/test_deepseek_v31_eval_amd.py	test_deepseek_v31_accuracy	aiter-caused	new (sister green Jun14/15/17)	[MEDIUM] aiter override a40c487b (105 ahead) HIP-runtime regression
5	GPU segfault in distributed init (torch.ones)	nightly-8-gpu-deepseek-v32-rocm720	test/registered/amd/accuracy/mi30x/test_deepseek_v32_eval_amd.py	DeepSeek-V3.2 (basic) eval	aiter-caused	new (prior scout Jun11 green)	[MEDIUM] newer aiter override segfaults first HIP kernel
6	VLM MMMU accuracy below threshold	nightly-accuracy-2-gpu-vlm-rocm720	test/registered/amd/accuracy/mi30x/test_vlms_mmmu_eval_amd.py	(5 VLM sub-evals)	pre-existing (sglang)	red ≥Jun09 (persistent)	[MEDIUM] degraded VLM baseline / threshold drift
7	VLM MMMU accuracy below threshold	nightly-accuracy-2-gpu-vlm	test/registered/amd/accuracy/mi30x/test_vlms_mmmu_eval_amd.py	test_mmmu_vlm_models	pre-existing (sglang)	red ≥Jun14 (persistent)	[no candidate ≥ MEDIUM]
8	gsm8k accuracy below threshold (Mistral/Mixtral)	nightly-accuracy-2-gpu-rocm720	test/registered/amd/accuracy/mi30x/test_gsm8k_eval_amd.py	test_gsm8k_all_models	pre-existing (sglang)	never-passed (Jun11→18)	[HIGH] sglang-side Mistral/Mixtral accuracy bug, aiter exonerated
9	gsm8k accuracy below threshold (Mistral/gemma)	nightly-accuracy-2-gpu	test/registered/amd/accuracy/mi30x/test_gsm8k_eval_amd.py	Mistral-7B / gemma-2-27b	pre-existing (sglang)	Mistral red Jun15/17/18	[HIGH] Mistral GSM8K pre-existing — fix #27757
10	JIT activation.cuh clang-HIP compile + quant timeout	jit-kernel-unit-test-amd	test/registered/jit/test_activation.py	test_activation_correctness	pre-existing (sglang)	scout branch lags main	[HIGH] amd/aiter-ci missing #27947
11	JIT activation.cuh clang-HIP compile + quant timeout	jit-kernel-unit-test-amd	test/registered/jit/test_per_token_group_quant_8bit.py	test_per_token_group_quant_with_column_major	pre-existing (sglang)	scout branch lags main	[HIGH] missing #27947
12	HF weight-download / Xet stall → server never ready	nightly-2-gpu-mi35x-deepseek-r1-mxfp4-tp2	test/registered/amd/accuracy/mi35x/test_deepseek_r1_mxfp4_tp2_mi35x.py	setUpClass	unclear (infra)	failed Jun11 & Jun18	[HIGH] HF Hub download stall (pre-kernel)
13	HF weight-download / Xet stall → server never ready	nightly-8-gpu-mi35x-deepseek-r1-mxfp4-ar-fusion	test/registered/amd/accuracy/mi35x/test_deepseek_r1_mxfp4_ar_fusion_eval_mi35x.py	test_deepseek_r1_mxfp4_ar_fusion_accuracy	unclear (infra)	single occurrence	[HIGH] HF Xet download hang (pre-kernel)
14	bf16 Qwen3-235B server hang/abnormal termination	nightly-8-gpu-qwen3-235b	test/registered/amd/test_qwen3_instruct.py	bf16 file-level	aiter-caused	new (sister + prior scout green)	[MEDIUM] aiter delta 7d604afe..a40c487b bf16 hang
15	CUDA IPC "invalid device pointer" (encoder-DP)	nightly-4-gpu-rocm720	test/registered/vlm/test_encoder_dp.py	test_vlm_mmmu_benchmark	pre-existing (sglang)	flaky (model-draw) Jun15/16/18	[MEDIUM] encoder-DP IPC handle-redirect fails Qwen3-VL-32B
16	GPU mem fault (write to RO page) VLM warmup encoder-DP	nightly-4-gpu	test/registered/vlm/test_encoder_dp.py	test_vlm_mmmu_benchmark	pre-existing (sglang)	flaky (model-draw) Jun16/18	[MEDIUM] sglang encoder-DP TP=4 bug (GLM-4.1V/Qwen3-VL-32B)
17	DeepSeek-R1 HiCache warmup hang/timeout (MI35x)	nightly-8-gpu-mi35x-deepseek-r1-hicache	test/registered/amd/test_deepseek_r1_hicache_mi35x.py	setUpClass	pre-existing (sglang)	never-passed (Jun09→19)	[no candidate ≥ MEDIUM]
18	HF Hub connection reset during tokenizer startup	nightly-test-1-gpu-unit	test/registered/lora/test_lora_openai_compatible.py	setUpClass	unclear (infra flake)	one-off (sister green)	[HIGH] transient HF Hub connectivity drop
19	TCPStore EADDRINUSE multi-rank grafter	nightly-accuracy-2-gpu	test/registered/debug_utils/test_dumper.py	test_mixed_shape_senders_via_user_transform	unclear (flake)	one-off (sister green)	[no candidate ≥ MEDIUM]

Cluster: GPU segfault in distributed init (torch.ones)

• Affected: rows 4, 5 — both 8-GPU DeepSeek (V3.1 & V3.2) on ROCm 7.2.
• Shared facts: all TP ranks crash at parallel_state.py:300/301 (torch.ones → fill_kernel_cuda), before any model/aiter call; parallel_state.py byte-identical across sglang delta; sisters green with default aiter; override is 105 commits ahead.
• Top hypothesis: [MEDIUM] newer aiter override destabilizes HIP runtime. Disconfirming: faulting frame is core PyTorch, not an aiter symbol; sglang HEAD also moved 65 commits (no same-SHA A/B).
• In-flight fix: ❌ none. Triage: re-run sister at f83e4d59 with default aiter, then bisect aiter 7d604afe..a40c487b.

Cluster: bf16 Qwen3-235B server hang

• Affected: row 14. Shared facts: FP8 sibling passed (0.956); bf16 run cut short ~9 min in (in_progress step on a failure job); sister + prior scout green. Top hypothesis: [MEDIUM] aiter bf16-GEMM regression in the 105-commit delta. Disconfirming: terminal stack truncated (hang vs runner-abort unpinned). In-flight fix: ❌ none. Triage: A/B default vs override aiter on the bf16 test; capture py-spy.

Cluster: Gate/concurrency cancellation cascade

• Affected: rows 1–3. Shared facts: 3 jobs cancelled same instant (0 steps); pr-test-amd-extra.yml:84 concurrency group omits rocm_version, cancel-in-progress=true on schedule → default vs rocm720 collide; rocm720 gate won. Non-signal (no test ran). Top hypothesis: [HIGH] concurrency-group collision. In-flight fix: ❌ none. Triage: add inputs.rocm_version to the concurrency key.

Cluster: HF weight-download / Xet stall

• Affected: rows 12, 13 (MI35x DeepSeek-R1-MXFP4). Shared facts: every py-spy dump stuck in download_weights_from_hf → snapshot_download → xet_get; both runs also failed local docker-registry pull (network). Pre-kernel ⇒ aiter not in path. Top hypothesis: [HIGH] HF Hub/Xet stall. In-flight fix: ❌ none. Triage: pre-cache weights / HF_HUB_DISABLE_XET=1; add hard per-file timeout.

Cluster: VLM encoder-DP memory faults (rows 15, 16)

• Two distinct stack tops (CUDA IPC invalid device pointer at cuda_ipc_transport_utils.py:485 vs warmup write-to-read-only-page), so kept as separate clusters; both pre-existing, reproduce under default aiter, model-draw-flaky (test_encoder_dp.py:32-34 random.choice). Top hypothesis: [MEDIUM] sglang --mm-enable-dp-encoder TP=4 bug for Qwen3-VL-32B/GLM-4.1V. In-flight fix: ⚠️ stale #18721; #25951 topical. Triage: pin model + A/B the DP-encoder flag.

Pre-existing sglang failures (not caused by aiter — already fail in default-aiter runs)

• VLM MMMU (rows 6, 7): same 5 models below threshold with default aiter, ≥Jun14, near-random scores. No in-flight fix.
• gsm8k (rows 8, 9): Mistral/Mixtral far below threshold, never-passed; Mistral fix in-flight #27757; gemma-2-27b is a 0.86 threshold straddle.
• JIT compile/timeout (rows 10, 11): scout branch amd/aiter-ci missing #27947 (merged Jun16, already green on main) — sync branch, no new PR.
• VLM encoder-DP faults (rows 15, 16) and HiCache warmup (row 17, never-passed Jun09→19) — see above / needs new investigation.

Suggested triage order

Rank	Triage step	Targets	Why this first
1	Sync amd/aiter-ci with main (cherry-pick #27947)	rows 10–11	fix already merged; cheap, fully closes cluster
2	Add rocm_version to pr-test-amd-extra.yml concurrency key	rows 1–3	one-line config fix; removes 3 false failures
3	Re-run + A/B (default vs override aiter) the 3 aiter-caused jobs	rows 4, 5, 14	confirms/clears the only genuine aiter signal before bisecting
4	Chase in-flight fix #27757; re-run HF-stall/flake jobs	rows 9, 12, 13, 18, 19	existing PR + likely-transient infra; cheap reruns
5	Deterministic per-model repro of test_encoder_dp.py (bypass random.choice)	rows 15, 16	surfaces the masked sglang encoder-DP crash for bisection

---

Job	Failed Steps	Started
call-nightly-amd-rocm720 / nightly-8-gpu-deepseek-v31-rocm720	Accuracy Test ROCm 7.2 (8-GPU DeepSeek-V3.1)	2026-06-18T23:15
call-nightly-amd-rocm720 / nightly-accuracy-2-gpu-vlm-rocm720	Nightly Accuracy Test ROCm 7.2 (2-GPU VLM MMMU)	2026-06-18T21:19
call-nightly-amd-rocm720 / nightly-accuracy-2-gpu-rocm720	Nightly Test ROCm 7.2 (2-GPU)	2026-06-18T21:13
call-nightly-amd-rocm720 / nightly-8-gpu-deepseek-v32-rocm720	Accuracy Test ROCm 7.2 (8-GPU DeepSeek-V3.2 Basic)	2026-06-18T23:44
call-nightly-amd / nightly-accuracy-2-gpu	N/A	2026-06-18T21:50
call-nightly-amd-rocm720 / nightly-4-gpu-rocm720	Nightly Test ROCm 7.2 (4-GPU)	2026-06-18T20:52
call-nightly-amd / nightly-test-1-gpu-unit	Nightly Unit Test (1-GPU)	2026-06-18T22:39
call-nightly-amd / nightly-accuracy-2-gpu-vlm	Nightly Accuracy Test (2-GPU VLM MMMU)	2026-06-18T21:44
call-nightly-amd / nightly-2-gpu-mi35x-deepseek-r1-mxfp4-tp2	Accuracy Test MI35x (2-GPU DeepSeek-R1-MXFP4 TP2)	2026-06-18T20:52
call-nightly-amd / nightly-4-gpu	Nightly Test (4-GPU)	2026-06-18T20:52
[`call-pr-test-amd / call-pr-test-amd-extra / extra-a-test-1-gpu-large-amd${{ ((inputs.rocm_version && format('-{0}', inputs.rocm_version))		'') }} (linux-${{ (inputs.runner_arch
[`call-pr-test-amd / call-pr-test-amd-extra / extra-a-test-1-gpu-small-amd${{ ((inputs.rocm_version && format('-{0}', inputs.rocm_version))		'') }} (linux-${{ (inputs.runner_arch
call-pr-test-amd / call-pr-test-amd-extra / call-gate / pr-gate	N/A	2026-06-18T20:52
call-pr-test-amd / jit-kernel-unit-test-amd (linux-mi325-1gpu-sglang)	Run JIT kernel unit tests	2026-06-18T22:38
call-nightly-amd / nightly-8-gpu-qwen3-235b	N/A	2026-06-19T00:05
call-nightly-amd / nightly-8-gpu-mi35x-deepseek-r1-mxfp4-ar-fusion	Accuracy Test MI35x (8-GPU DeepSeek-R1-MXFP4 AllReduce Fusion)	2026-06-19T06:48
call-nightly-amd / nightly-8-gpu-mi35x-deepseek-r1-hicache	Accuracy Test MI35x (8-GPU DeepSeek-R1 HiCache)	2026-06-19T08:19

Per-Job Analysis

call-nightly-amd-rocm720 / nightly-8-gpu-deepseek-v31-rocm720 — failed step(s): Accuracy Test ROCm 7.2 (8-GPU DeepSeek-V3.1)

Commit Info

• sglang: f83e4d59
• aiter: a40c487b (source: AITER_COMMIT_OVERRIDE — image shipped default v0.1.14rc1.dev240+g7d604afe5, scout forced a rebuild to the override)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi30x/test_deepseek_v31_eval_amd.py	test_deepseek_v31_accuracy	Exception: Server process exited with code 1 — server died because RuntimeError: Rank 1 scheduler died during initialization (exit code: -11) (SIGSEGV)	Log

Failure Summary

The 8-GPU DeepSeek-V3.1 accuracy server failed to start: multiple TP ranks raised Fatal Python error: Segmentation fault during distributed initialization, and Rank 1 scheduler died during initialization (exit code: -11). The test aborted after 614s with no accuracy produced. The segfault originates in GPU-runtime code, not in the test logic.

Regression Status

New failure surfaced by the aiter override. The same test file + function passes with the Dockerfile-default aiter in the sister nightly-test-amd-rocm720.yml schedule.

Recent history of test_deepseek_v31_eval_amd.py (test_deepseek_v31_accuracy), sister job nightly-8-gpu-deepseek-v31-rocm720 (default aiter, event=schedule, branch=main, completed runs only):

Date	Run	Job	Test File Status	Failed Function	Error
Jun 18	27788558636 (scout, override aiter)	deepseek-v31-rocm720	❌ Failed	test_deepseek_v31_accuracy	SIGSEGV in dist init
Jun 17	27710084518	deepseek-v31-rocm720	✅ Passed	—	accuracy=0.955 threshold=0.9 ✅
Jun 15	27568866028	deepseek-v31-rocm720	✅ Passed	—	—
Jun 14	27507292665	deepseek-v31-rocm720	✅ Passed	—	—

Failure Origin

Test File	Test Function	Origin	Baseline Run	Baseline Status
test/registered/amd/accuracy/mi30x/test_deepseek_v31_eval_amd.py	test_deepseek_v31_accuracy	aiter-caused	27710084518	✅ Passed (sister job, default aiter g7d604afe5, accuracy=0.955)

Failure Cluster

GPU segmentation fault during distributed init (torch.ones on HIP device in world-group setup)

Facts (verified observations)

• Crash thread (verbatim, all TP ranks) terminates in:
  parallel_state.py:300 __init__ → parallel_state.py:1508 init_world_group → parallel_state.py:1843 init_distributed_environment → model_runner.py:1220 init_torch_distributed → tp_worker.py → scheduler.py:432 __init__.
• parallel_state.py:300 is active_ranks = torch.ones(len(ranks), dtype=torch.int32, device=self.device); the C++ frames show at::native::ones → fill_kernel_cuda → FillFunctor segfaulting on the HIP device.
• RuntimeError: Rank 1 scheduler died during initialization (exit code: -11) → SIGSEGV (not OOM/-9).
• aiter: image default 7d604afe; scout forced AITER_COMMIT_OVERRIDE=a40c487b and rebuilt (verified via [CI-AITER-CHECK] markers).
• A/B: sister nightly-8-gpu-deepseek-v31-rocm720 ran the identical test with empty AITER_COMMIT_OVERRIDE (default aiter) on Jun 17 and PASSED (accuracy=0.955 threshold=0.9 ✅); Jun 15 & Jun 14 also passed.
• aiter override a40c487b is 105 commits ahead of default 7d604afe (ahead_by: 105, behind_by: 0).
• sglang delta baseline→scout (8aaca72c..f83e4d59) = 65 commits; parallel_state.py was NOT modified in that window (the exact crashing line is byte-identical between the passing baseline and the failing scout).

Hypothesised Causes (with confidence)

• [MEDIUM] The aiter override (a40c487b, 105 commits ahead of default) introduces a GPU-runtime regression that segfaults a basic torch.ones HIP kernel during world-group init.
  • Supporting: identical test + identical sglang crash site passes with default aiter one day earlier and fails only under the override; the crash is at the HIP kernel level (fill_kernel_cuda), exactly where a rebuilt kernel library would manifest; parallel_state.py is unchanged across the sglang delta, so the Python crash site is not the variable.
  • Disconfirming: the sglang HEAD also advanced 65 commits (8aaca72c→f83e4d59) between the passing baseline and this scout, so a baseline at the exact scout sglang SHA with default aiter was not available (closest sister default-aiter run is one day older); a same-SHA A/B would be needed to fully exonerate sglang.
  • How to verify: re-run nightly-8-gpu-deepseek-v31-rocm720 at sglang f83e4d59 with default aiter (no override) — if it passes, aiter is confirmed; then bisect the 105-commit aiter range.
• [LOW] A specific aiter commit among the 105 is the culprit (cannot be isolated without a bisect).
  • Supporting: several commits in the range touch GEMM/attention/kernel-config paths.
  • Disconfirming: the crash is in generic distributed init (torch.ones), not in any attention/GEMM call path, so log-line proximity to any single aiter commit is absent.
  • How to verify: git bisect the aiter range 7d604afe..a40c487b against this test.

In-flight Fix Check

• In-flight fix: ❌ none found — search of open sglang PRs for deepseek v31 + segfault returned only #8573 (unrelated, lyra w4afp8 moe); aiter scout segfault returned 0. (API: search/issues?q=...is:pr+is:open+deepseek+v31+segfault and ...+aiter+scout+segfault.)

Suggested Triage Steps

• Re-run the sister nightly-8-gpu-deepseek-v31-rocm720 at sglang f83e4d59 with no aiter override — a same-SHA pass confirms aiter-caused and closes the 65-commit sglang confounder.
• If confirmed, git bisect the aiter range 7d604afe..a40c487b (105 commits) against test_deepseek_v31_eval_amd.py on an MI30x 8-GPU runner.
• Capture a HIP coredump / AMD_LOG_LEVEL=4 on the override build to localize the failing kernel in fill_kernel_cuda.

Status

• Persistence: First observed in this scout run (Jun 18); sister default-aiter runs green Jun 14/15/17.
• Scope: 1 job (nightly-8-gpu-deepseek-v31-rocm720) in this scout run; all 8 TP ranks segfault at the same site.
• Blocked work: 8-GPU DeepSeek-V3.1 accuracy signal under the candidate aiter a40c487b on ROCm 7.2.
• In-flight fix: ❌ none found.

call-nightly-amd-rocm720 / nightly-accuracy-2-gpu-vlm-rocm720 — failed step(s): Nightly Accuracy Test ROCm 7.2 (2-GPU VLM MMMU)

Commit Info

• sglang: f83e4d59 (refactor(runner): unify eager-forward DP/MLP-sync padding into one helper #28383)
• aiter: a40c487b (source: override — AITER_COMMIT_OVERRIDE; Dockerfile default is 7d604afe5, image had dev build v0.1.14rc1.dev240+g7d604afe5, rebuilt to the override)

Failed Tests

Single test file test/registered/amd/accuracy/mi30x/test_vlms_mmmu_eval_amd.py loops over VLM models and asserts each MMMU score ≥ a per-model threshold. 5 of 11 model evaluations failed (exit code 1):

Test File	Model (sub-eval)	Error	Log
test_vlms_mmmu_eval_amd.py	Qwen/Qwen2-VL-7B-Instruct	MMMU 0.290 < threshold 0.30	Log
test_vlms_mmmu_eval_amd.py	Qwen/Qwen2.5-VL-7B-Instruct	MMMU 0.290 < threshold 0.33	Log
test_vlms_mmmu_eval_amd.py	Qwen/Qwen3-VL-30B-A3B-Instruct	MMMU 0.260 < threshold 0.29	Log
test_vlms_mmmu_eval_amd.py	openbmb/MiniCPM-o-2_6	MMMU 0.280 < threshold 0.32	Log
test_vlms_mmmu_eval_amd.py	deepseek-ai/deepseek-vl2-small	MMMU 0.280 < threshold 0.31	Log

Passed (6): InternVL2_5-2B (0.290/0.29), MiniCPM-v-2_6, Janus-Pro-7B, Kimi-VL-A3B-Instruct, MiMo-VL-7B-RL, DeepSeek-OCR-2.

Failure Summary

The VLM MMMU accuracy job failed because 5 of 11 model evaluations scored below their per-model MMMU thresholds, returning exit code 1. The misses are marginal (0.01–0.05 below threshold). Notably every model — passing and failing — scores in the 0.26–0.34 band, i.e. essentially MMMU random-chance (~0.25 for 4-option multiple choice), and the thresholds themselves are calibrated down into that band.

Regression Status

Known recurring failure (marginal/boundary). The sister nightly-accuracy-2-gpu-vlm-rocm720 job has concluded failure in every recent scheduled run (06-09 → 06-17). Scores hover at the threshold and flip pass/fail between runs (e.g. Qwen2-VL: 0.300 pass on 06-15, 0.290 fail on 06-17 and in this scout).

Recent history of test_vlms_mmmu_eval_amd.py (Qwen2.5-VL / Qwen3-VL sub-evals) in the sister job (nightly-test-amd-rocm720.yml, event=schedule, branch=main, completed runs only):

Date	Run	Job	Test File Status	Failed sub-eval	Error
Jun 18 (scout)	27788558636	this job	❌ Failed	Qwen2-VL, Qwen2.5-VL, Qwen3-VL-30B, MiniCPM-o, deepseek-vl2	MMMU < threshold
Jun 17	27710084518	sister VLM	❌ Failed	same 5 (identical scores)	MMMU < threshold
Jun 15	27568866028	sister VLM	❌ Failed	Qwen2.5-VL, Qwen3-VL-30B (log truncated)	MMMU < threshold

No green run for this test file found in the queryable window (lookback 06-09 → 06-18); the job has been red continuously. The specific set of failing models varies run-to-run only because several models sit exactly at their threshold.

Failure Origin

pre-existing (sglang) — all 5 failures. The aiter override is exonerated.

Test File	Sub-eval	Origin	Baseline Run	Baseline Status
test_vlms_mmmu_eval_amd.py	Qwen2-VL-7B-Instruct	pre-existing (sglang)	27710084518	❌ Same failure, default aiter (0.290 < 0.30)
test_vlms_mmmu_eval_amd.py	Qwen2.5-VL-7B-Instruct	pre-existing (sglang)	same	❌ Same failure (0.290 < 0.33)
test_vlms_mmmu_eval_amd.py	Qwen3-VL-30B-A3B-Instruct	pre-existing (sglang)	same	❌ Same failure (0.260 < 0.29)
test_vlms_mmmu_eval_amd.py	MiniCPM-o-2_6	pre-existing (sglang)	same	❌ Same failure (0.280 < 0.32)
test_vlms_mmmu_eval_amd.py	deepseek-vl2-small	pre-existing (sglang)	same	❌ Same failure (0.280 < 0.31)

The sister baseline ran with the Dockerfile-default aiter (7d604afe5, no override) and reproduced the identical 5 failures with identical scores. The override (a40c487b) has zero effect on these accuracy results.

Failure Cluster

VLM MMMU accuracy below threshold on AMD ROCm 7.2 (near-random scores across all models)

Facts (verified observations)

• Verbatim from log (step 6): Score: 0.290 (threshold: 0.30) → Status: ❌ FAILED for Qwen2-VL; analogous lines for the other 4 models; final ✗ FAILED: .../test_vlms_mmmu_eval_amd.py returned exit code 1.
• All 11 models score within 0.26–0.34 MMMU; MMMU random chance for 4-option MC ≈ 0.25. Passing models pass only because their thresholds are ≤ their (also low) scores.
• aiter actually executed = override a40c487b (verified via [CI-AITER-CHECK] AITER_COMMIT_OVERRIDE=a40c487b... → forcing rebuild).
• Baseline sister run 27710084518 (06-17, default aiter 7d604afe5, no override) produced identical scores for all 11 models — conclusively exonerating aiter.
• Non-fatal noise in log (not the failure cause): Health check failed... detokenizer (line 4612), failed to execute .../clang++ ... -x hip -c /dev/null (JIT probe), torchcodec import-skip, local docker registry fallback.

Hypothesised Causes (with confidence)

• [FACT] The job fails because 5 VLM MMMU scores are below their thresholds (directly observable in the log).
• [MEDIUM] This is a long-standing degraded VLM accuracy baseline on AMD ROCm 7.2, not a fresh regression in the scout window.
  • Supporting: every model scores near random chance (~0.25–0.34); thresholds are already calibrated down into the random-chance band; the sister job has been red continuously since at least 06-09; failures reproduce without the aiter override.
  • Disconfirming: no green run exists in the queryable window, so the original regression point predates the lookback and cannot be pinned here; this analysis does not isolate the sglang commit that first degraded accuracy.
  • How to verify: bisect the sister job's history further back to find the last run where these models cleared their (pre-lowering) thresholds; inspect the multimodal/image-preprocessing path for these model families on ROCm.
• [LOW] Marginal/boundary flakiness contributes to which specific models fail each run (e.g. Qwen2-VL flips 0.300↔0.290 across runs), but does not explain the systemic near-random band.
• Zero aiter commits cited: failure is reproducible with the default aiter (Origin = pre-existing sglang).

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for VLM+MMMU+accuracy, amd+vlm+mmmu, mmmu+threshold. Matches returned (e.g. #26639, #26606, #18186) are unrelated feature/CI PRs, none targeting this AMD VLM MMMU accuracy regression.

Suggested Triage Steps

• Confirm scope: re-run the sister nightly-accuracy-2-gpu-vlm-rocm720 job; the boundary models (Qwen2-VL, InternVL2_5-2B, Janus-Pro) will likely flip — treat them as quarantine candidates separate from the systemically-low models.
• Investigate the near-random MMMU band: launch Qwen/Qwen2.5-VL-7B-Instruct on AMD MI30x and inspect whether images are actually reaching the model (multimodal preprocessing / torchcodec-skip side effects), since ~0.25 MMMU suggests vision input may be ignored.
• Bisect the sister job's scheduled history before 06-09 to locate the last run where these models scored well above threshold; the regression predates the current lookback window.
• Do not investigate the aiter delta for these 5 tests — the default-aiter baseline reproduces them identically.

Status

• Persistence: VLM MMMU accuracy job red continuously in the sister workflow since at least 2026-06-09; this scout run reproduces it.
• Scope: test_vlms_mmmu_eval_amd.py only, but spanning 5 model families; affects nightly-test-amd-rocm720 and the AITER Scout's call-nightly-amd-rocm720 leg. Not aiter-related.
• Blocked work: VLM accuracy signal on AMD ROCm 7.2 is non-actionable (all models near random; thresholds already lowered to mask it).
• In-flight fix: ❌ none found — needs new investigation into the AMD VLM accuracy degradation.

call-nightly-amd-rocm720 / nightly-accuracy-2-gpu-rocm720 — failed step(s): Nightly Test ROCm 7.2 (2-GPU)

Commit Info

• sglang: f83e4d59
• aiter: a40c487b (source: AITER_COMMIT_OVERRIDE — forced rebuild; Dockerfile default was 7d604afe)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi30x/test_gsm8k_eval_amd.py	test_gsm8k_all_models → check_model_scores	AssertionError: The following models failed (5 models below gsm8k threshold)	Log

Per-model gsm8k scores in this scout run:

Model	Score	Threshold	Result
mistralai/Mistral-7B-Instruct-v0.3	0.3600	0.4700	❌
google/gemma-2-27b-it	0.8592	0.8600	❌ (borderline, −0.0008)
mistralai/Mixtral-8x7B-Instruct-v0.1	0.3097	0.6900	❌
neuralmagic/Mistral-7B-Instruct-v0.3-FP8	0.3364	0.4600	❌
neuralmagic/Mixtral-8x7B-Instruct-v0.1-FP8	0.1484	0.6900	❌

(All other ~16 models in the suite passed, e.g. Llama-3.1-8B 0.816, Llama-3.1-70B 0.927, Qwen3-30B-A3B 0.941.)

Failure Summary

The 2-GPU nightly gsm8k accuracy test test_gsm8k_all_models failed because 5 of ~21 models scored below their accuracy thresholds. The Mistral-7B and Mixtral-8x7B families (both bf16 and FP8 variants) are far below threshold (e.g. Mixtral-8x7B 0.31 vs 0.69; Mixtral-FP8 0.15 vs 0.69), while gemma-2-27b missed its 0.86 threshold by 0.0008. The job otherwise completed; this is an accuracy assertion, not a crash.

Regression Status

Never-passed (no green run found in queryable window). The same Mistral/Mixtral models have been below threshold in every completed run of the sister job (default aiter) Jun 11–14 and in this scout run Jun 18.

Recent history of test_gsm8k_eval_amd.py (test_gsm8k_all_models) — sister job nightly-accuracy-2-gpu-rocm720 on nightly-test-amd-rocm720.yml (branch main, event schedule, completed runs only) plus this scout run:

Date	Run	Aiter	Test File Status	Failing models (Mistral/Mixtral subset)
Jun 18	27788558636 (this scout, override)	a40c487b	❌ Failed	Mistral-7B 0.36, Mixtral 0.31, +FP8 variants, gemma 0.859
Jun 15	27568866028	default	⚠️ Cancelled mid-Mixtral	(partial: Mistral-7B 0.36, gemma 0.859 already seen)
Jun 14	27507292665	default	❌ Failed	Mistral-7B 0.37, Mixtral 0.30, +FP8 variants
Jun 13	27474496879	default	❌ Failed	Mistral-7B 0.36, Mixtral 0.31, gemma 0.859, +FP8
Jun 12	27434236636	default	❌ Failed	Mistral-7B 0.35, Mixtral 0.30
Jun 11	27368779328	default	❌ Failed	Mistral-7B 0.36, Mixtral 0.32

No passing run exists for these models in the lookback window (Jun 11–18). First-observed-failing in the queryable window: Jun 11; the true regression point (if any) predates the window.

Failure Origin

Test File	Test Function	Origin	Baseline Run	Baseline Status
test_gsm8k_eval_amd.py	test_gsm8k_all_models (Mistral-7B-v0.3)	pre-existing (sglang)	Jun 14	❌ Same fail, score 0.37 (default aiter)
test_gsm8k_eval_amd.py	test_gsm8k_all_models (Mixtral-8x7B)	pre-existing (sglang)	Jun 14	❌ Same fail, score 0.30 (default aiter)
test_gsm8k_eval_amd.py	test_gsm8k_all_models (Mistral-7B-FP8)	pre-existing (sglang)	Jun 14	❌ Same fail, score 0.32 (default aiter)
test_gsm8k_eval_amd.py	test_gsm8k_all_models (Mixtral-8x7B-FP8)	pre-existing (sglang)	Jun 14	❌ Same fail, score 0.18 (default aiter)
test_gsm8k_eval_amd.py	test_gsm8k_all_models (gemma-2-27b-it)	pre-existing (sglang)	Jun 13	❌ Same fail, score 0.8592 (default aiter)

All 5 failures reproduce in the sister nightly-test-amd-rocm720.yml runs that use the Dockerfile-default aiter (no override), with scores within ±0.02 of this scout run. The aiter override a40c487b is therefore exonerated.

Failure Cluster

gsm8k accuracy below threshold for Mistral/Mixtral family on AMD MI30x (ROCm 7.2, 2-GPU)

Facts (verified observations)

• [FACT] Scout run uses AITER_COMMIT_OVERRIDE=a40c487b… (log lines 43/3966) — [CI-AITER-CHECK] confirms forced rebuild over Dockerfile default 7d604afe.
• [FACT] 5 models below threshold; verbatim from log line 37347–37351: Mistral-7B 0.3600/0.4700, gemma-2-27b 0.8592/0.8600, Mixtral-8x7B 0.3097/0.6900, Mistral-7B-FP8 0.3364/0.4600, Mixtral-8x7B-FP8 0.1484/0.6900.
• [FACT] Sister default-aiter baseline Jun 14 (81300706193): Mistral-7B 0.3706, Mixtral 0.2983, Mistral-7B-FP8 0.3158, Mixtral-FP8 0.1788 — same 4 models fail without the override.
• [FACT] Sister default-aiter baseline Jun 13 (81211182443): gemma-2-27b 0.8592 — identical to scout, plus same Mistral/Mixtral failures.
• [FACT] Same Mistral/Mixtral failures present in Jun 11 and Jun 12 sister runs too — no green run in window.
• [FACT] The Mixtral models are massively below threshold (≈0.31 and ≈0.15 vs 0.69), not borderline; gemma-2-27b misses by 0.0008.

Hypothesised Causes (with confidence)

• [HIGH] The Mistral/Mixtral-family gsm8k failures are a pre-existing sglang-side accuracy problem on AMD MI30x, independent of the aiter override.
  • Supporting: identical sub-threshold scores reproduce across 5 independent runs Jun 11–18 on the Dockerfile-default aiter (sister job), so the aiter override cannot be the cause. Magnitude (Mixtral 0.31/0.15 vs 0.69) indicates a real correctness issue in the model path/quantization, not noise.
  • Disconfirming: (none found) — no green run exists in the window to contradict "never-passed".
  • How to verify: run test_gsm8k_eval_amd.py for mistralai/Mixtral-8x7B-Instruct-v0.1 on an MI30x ROCm 7.2 container with default aiter and inspect generations for degenerate/garbled output (attention or MoE routing bug).
• [LOW] gemma-2-27b (0.8592 vs 0.8600) is a threshold-edge flake distinct from the Mistral/Mixtral cluster.
  • Supporting: it fails by 0.0008 and is identical (0.8592) across Jun 13 and Jun 18, suggesting a deterministically-slightly-too-high threshold rather than a correctness regression.
  • Disconfirming: it passed in the Jun 14 baseline (not listed as failing there), so it is intermittent around the boundary.
  • How to verify: re-run gemma-2-27b a few times; if it oscillates around 0.86, the threshold needs a small margin adjustment.
• No specific culprit sglang commit can be identified: the failure predates the queryable history (red since the first available run Jun 11), so there is no pass_sha..fail_sha window to bisect within the lookback. Needs investigation against an older known-good baseline if one exists.

In-flight Fix Check

• In-flight fix: ❌ none found directly addressing Mistral/Mixtral gsm8k accuracy on AMD. Related open PRs touching this test exist but are not fixes for the accuracy drop: #26730 "test(amd): run full GSM8K for accuracy checks" and #21622 "Extract shared GSM8K helpers". (Searched is:pr is:open gsm8k accuracy AMD.)

Suggested Triage Steps

• Confirm scope by running test_gsm8k_eval_amd.py for mistralai/Mixtral-8x7B-Instruct-v0.1 and mistralai/Mistral-7B-Instruct-v0.3 on MI30x ROCm 7.2 with default aiter — expect repro, confirming this is not an aiter-override artifact.
• Inspect raw generations for the Mixtral models; a score of 0.15–0.31 (vs 0.69) usually means garbled/empty output → look at MoE routing / MixtralForCausalLM forward path and attention backend (attention_backend='aiter') on gfx942.
• Since this predates the queryable window, locate an older green nightly-amd-rocm720 run (before Jun 11) for these models to obtain a pass_sha..fail_sha bisect range; if none exists, treat as a never-passed AMD port issue and track separately.
• For gemma-2-27b, evaluate whether the 0.86 threshold needs a small downward margin (boundary flake) — separate from the Mistral/Mixtral correctness issue.

Status

• Persistence: Failing in every completed run in the queryable window (Jun 11 → Jun 18); no green run found for these models.
• Scope: 5 models in test_gsm8k_eval_amd.py on nightly-accuracy-2-gpu-rocm720; reproduces in both the scout (override aiter) and the regular nightly (default aiter), so it blocks the 2-GPU AMD ROCm 7.2 accuracy signal regardless of aiter version.
• Blocked work: 2-GPU gsm8k accuracy gate on AMD MI30x ROCm 7.2.
• In-flight fix: ❌ none found addressing the accuracy drop.

call-nightly-amd-rocm720 / nightly-8-gpu-deepseek-v32-rocm720 — failed step(s): Accuracy Test ROCm 7.2 (8-GPU DeepSeek-V3.2 Basic)

Commit Info

• sglang: f83e4d59
• aiter: a40c487b (source: AITER_COMMIT_OVERRIDE, forced rebuild — log line 3522). Dockerfile default was 7d604afe (v0.1.14rc1.dev240+g7d604afe5). Override is 105 commits ahead of the default.

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi30x/test_deepseek_v32_eval_amd.py	eval of deepseek-ai/DeepSeek-V3.2 (basic)	AssertionError: Failed models: ['deepseek-ai/DeepSeek-V3.2 (basic)'] (server never launched — all TP processes hit Fatal Python error: Segmentation fault)	Log

Failure Summary

The 8-GPU DeepSeek-V3.2 basic accuracy test failed because the server never came up: every TP scheduler subprocess (TP0, TP1, TP2, …) died with Fatal Python error: Segmentation fault during distributed init, before any model weights loaded. The Python-level frame is parallel_state.py:301 — torch.ones(len(ranks), dtype=torch.int32, device=self.device), the first GPU kernel launch of init_world_group. The runner then reports AssertionError: Failed models.

Regression Status

New regression (in the scout/override context). The same job passes in the regular nightly (default aiter) and passed in the previous scout (older aiter override).

Recent history of test_deepseek_v32_eval_amd.py in job nightly-8-gpu-deepseek-v32-rocm720 (same branch, completed runs only):

Date	Run	Context (aiter)	Test File Status	Error
Jun 18	27788558636	scout (override a40c487b)	❌ Failed	Segfault in distributed init
Jun 17	27710084518	sister nightly (default aiter)	✅ Passed	—
Jun 15	27568866028	sister nightly (default aiter)	✅ Passed	—
Jun 14	27507292665	sister nightly (default aiter)	✅ Passed	—
Jun 11	27376585476	scout (override 00426ffb)	✅ Passed	—

Failure Origin

Test File	Test Function	Origin	Baseline Run	Baseline Status
test/registered/amd/accuracy/mi30x/test_deepseek_v32_eval_amd.py	DeepSeek-V3.2 (basic) eval	aiter-caused	27710084518	✅ Passed (sister job, default aiter)

A/B check is conclusive: the identical sister job nightly-8-gpu-deepseek-v32-rocm720 passes in the regular nightly-test-amd-rocm720 workflow (Dockerfile-default aiter) on Jun 14/15/17, and the prior scout passed with the older override 00426ffb. The only variable that changed to produce the segfault is the newer aiter override. sglang HEAD is unchanged relative to the passing sister baseline for this code path.

Failure Cluster

Segmentation fault on first GPU kernel during distributed init (torch.ones in init_world_group).

Facts (verified observations)

• [FACT] Job concluded failure; step 6 "Accuracy Test ROCm 7.2 (8-GPU DeepSeek-V3.2 Basic)" failed, step 7 perf skipped. Test runner: 0/1 passed, exit code 255, "not retrying" (log lines 4029–4051).
• [FACT] Python crash stack (log lines 3816–3826, repeated for TP0/TP1/TP2): parallel_state.py:300/301 __init__ → parallel_state.py:1508 init_world_group → parallel_state.py:1843 init_distributed_environment → model_runner.py:1220 init_torch_distributed → model_runner.py:563 __init__ → scheduler.py:777/850/432 → scheduler.py:4112 run_scheduler_process.
• [FACT] Native crash frames (log lines 3826–3829): at::native::ones → at::native::full → fill_out → fill_kernel_cuda → gpu_kernel_impl_nocast<FillFunctor<int>>. This is core PyTorch's HIP fill kernel, not an aiter kernel — the crash is on the very first device tensor allocation/fill, before any model forward.
• [FACT] aiter was rebuilt from override a40c487b (log 3522–3725); the override is 105 commits ahead of the Dockerfile default; the regression window vs the last passing scout override is 00426ffb..a40c487b = 71 aiter commits.
• [FACT] Crash reproduces on multiple independent TP ranks → deterministic, not a single-rank flake.

Hypothesised Causes

• [MEDIUM] The newer aiter override (a40c487b) rebuild destabilizes the GPU runtime for DeepSeek-V3.2 on ROCm 7.2, causing the first device kernel launch (torch.ones) to segfault.
  • Supporting: clean A/B — same test + same sglang HEAD passes with default aiter (sister Jun 17) and passed with the older scout override (Jun 11), and fails only with this newer override. Crash is deterministic across all TP ranks.
  • Disconfirming: the faulting frame is core PyTorch fill_kernel_cuda, not an aiter symbol, so aiter does not appear directly in the stack — the link is environmental (rebuild), inferred from the A/B, not from the trace.
  • How to verify: re-run the deepseek-v32-rocm720 nightly test in the AMD container with AITER_COMMIT_OVERRIDE=a40c487b... vs unset; if only the override segfaults, aiter is confirmed.
• [LOW] A specific aiter commit in the 71-commit window touching TP/allreduce/graph paths, e.g. 6058ed4b1e "Add fused custom AllReduce + MHC post for TP paths (#3791)" or 8113895a38 "route fused_qknorm_allreduce through compile guard (#3735)".
  • Supporting: both land in the regression window and touch collective/TP code exercised on this 8-GPU path.
  • Disconfirming: the segfault occurs before any allreduce/MHC/qknorm call — it's at the very first torch.ones of world-group setup — so neither commit is plausibly on the crashing path. Treat as a confounder, not a cause.
  • How to verify: aiter-side bisect of 00426ffb..a40c487b running this single test.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open sglang PRs for deepseek+v32+aiter (1 hit, unrelated: #27209 gfx950 Dockerfile) and aiter+segfault (2 hits, both unrelated: #28173, #17628). This is a forward-looking scout signal; the fix likely belongs on the aiter side or as a deferred aiter pin.

Suggested Triage Steps

• Confirm aiter causation: re-run test_deepseek_v32_eval_amd.py in the AMD ROCm 7.2 container twice — once with AITER_COMMIT_OVERRIDE=a40c487b3c01dc03fd3872d65b1f7404f669471f, once unset (Dockerfile default). Expect segfault only with the override.
• If confirmed, aiter-side bisect 00426ffb..a40c487b (71 commits) on this single test to find the first bad aiter commit.
• File/await an aiter-side issue before bumping the Dockerfile default aiter past 7d604afe for ROCm 7.2; this scout result indicates a newer aiter is not yet safe for DeepSeek-V3.2 8-GPU on ROCm 7.2.

Status

• Persistence: First scout failure for this job; previous scout (Jun 11) passed. Single occurrence in the current scout run.
• Scope: nightly-8-gpu-deepseek-v32-rocm720 under the scout (newer aiter override). The sister nightly (default aiter) is green, so regular CI signal is unaffected.
• Blocked work: Blocks promotion of aiter a40c487b (and the 71-commit delta) for DeepSeek-V3.2 8-GPU on ROCm 7.2.
• In-flight fix: ❌ none found — needs aiter-side investigation.

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-06-19 09:30 UTC)

[amd-bot]

amd-aiter-scout.yml — Per-Job Analysis (continued 2/3)

call-nightly-amd / nightly-accuracy-2-gpu — failed step(s): N/A

Commit Info

• sglang: f83e4d59
• aiter: a40c487b (source: AITER_COMMIT_OVERRIDE — forced rebuild; Dockerfile default was 7d604afe = v0.1.14rc1.dev240+g7d604afe5)

Failed Tests

Test File	Test Function	Error	Log
test/registered/debug_utils/test_dumper.py	TestGrafterMultiRankCpu::test_mixed_shape_senders_via_user_transform	DistNetworkError: EADDRINUSE port 30058 address already in use (+ cascading rendezvous timeouts)	Log
test/registered/amd/accuracy/mi30x/test_gsm8k_eval_amd.py	mistralai/Mistral-7B-Instruct-v0.3 (gsm8k)	Score 0.361 < threshold 0.47 → ❌ FAILED	Log
test/registered/amd/accuracy/mi30x/test_gsm8k_eval_amd.py	google/gemma-2-27b-it (gsm8k)	Score 0.858 < threshold 0.86 → ❌ FAILED	Log

⚠️ The job log (job 82231477913) is truncated mid-Mixtral-8x7B evaluation (last line 23:08:51), even though the job concluded failure at 23:27:03. Additional accuracy models (Mixtral, FP8 variants) that the sister baseline also evaluated could not be confirmed from the scout log; the three failures above are the verified subset.

Failure Summary

Two distinct test files failed in this 2-GPU accuracy job. test_dumper.py's multi-rank grafter test failed because TCP port 30058 was already bound (a leftover/colliding process on the runner), producing a DistNetworkError and cascading rendezvous timeouts in the other ranks. Separately, the GSM8K accuracy harness flagged Mistral-7B-Instruct-v0.3 (0.361 vs 0.47) and gemma-2-27b-it (0.858 vs 0.86) below their thresholds.

Regression Status

• test_dumper.py (test_mixed_shape_senders_via_user_transform): Flaky / infrastructure — passed in the sister baseline; symptom is a port-bind collision, not a deterministic regression.
• test_gsm8k_eval_amd.py / Mistral-7B: Never-passed (long-standing) — scores ~0.36–0.38 against a 0.47 threshold across every recent run.
• test_gsm8k_eval_amd.py / gemma-2-27b-it: Flaky threshold straddle — score hovers at the 0.86 boundary.

Recent history (sister nightly-test-amd nightly-accuracy-2-gpu, same branch, event=schedule, completed runs only):

Date	Run	Mistral-7B	gemma-2-27b-it	test_dumper
Jun 18 (scout)	run	❌ 0.361	❌ 0.858	❌ EADDRINUSE
Jun 17	run	❌ 0.365	✅ 0.861	✅ passed
Jun 15	run	❌ 0.378	❌ 0.859	— (not confirmed)

(Jun 16 and Jun 17 scout-sister runs were cancelled at the workflow level; the Jun 17 nightly-accuracy-2-gpu job itself still ran to failure and is the primary baseline. gemma straddles the 0.86 line: 0.859 / 0.861 / 0.858 — within gsm8k sampling noise, SE ≈ 0.01 over 1314 samples.)

Failure Origin

Test File	Test Function	Origin	Baseline Run	Baseline Status
test_dumper.py	test_mixed_shape_senders_via_user_transform	unclear	run	✅ Passed in sister — but symptom is a port collision unrelated to aiter (likely flake)
test_gsm8k_eval_amd.py	Mistral-7B-Instruct-v0.3	pre-existing (sglang)	run	❌ Same failure in sister (0.365, default aiter)
test_gsm8k_eval_amd.py	gemma-2-27b-it	unclear	run	✅ Passed at 0.861 (marginal); ❌ 0.859 on Jun 15 — threshold straddle

Failure Cluster

• GSM8K accuracy below threshold (Mistral-7B / gemma-2-27b) — accuracy-eval cluster.
• TCPStore EADDRINUSE during multi-rank grafter test — distributed port-collision cluster (separate symptom).

Facts (verified observations)

• test_dumper.py:3344 raised AssertionError; root error torch.distributed.DistNetworkError: ... port: 30058 ... EADDRINUSE, message: address already in use at dumper.py:1605 (_tcp_rendezvous_handler); other ranks then hit DistStoreError: wait timeout / Failed to recv. ✗ FAILED: test_dumper.py returned exit code 1.
• Scout GSM8K: Mistral-7B Score: 0.361 (threshold: 0.47) ❌; gemma-2-27b Score: 0.858 (threshold: 0.86) ❌; DeepSeek-Coder-V2-Lite 0.816/0.81 ✅; Llama-3.1-70B 0.928/0.89 ✅.
• Sister baseline (Jun 17, default aiter, no override): Mistral-7B 0.365/0.47 ❌ (verified via [METRIC] lines), gemma-2-27b 0.861/0.86 ✅, test_dumper.py "passed": true, elapsed 396.
• aiter override a40c487b was force-rebuilt ([CI-AITER-CHECK] ... → forcing rebuild) over Dockerfile default 7d604afe.

Hypothesised Causes (with confidence)

• [HIGH] Mistral-7B GSM8K failure is pre-existing in sglang, not aiter-related.
  • Supporting: identical sub-threshold score (0.361 scout vs 0.365 sister-baseline with default aiter; 0.378 on Jun 15). Reproduces without the aiter override.
  • Disconfirming: (none found) — A/B baseline confirms.
  • How to verify: already confirmed; track in-flight fix #27757.
• [MEDIUM] gemma-2-27b failure is a threshold straddle / sampling noise, not a true regression.
  • Supporting: 0.858 / 0.859 / 0.861 across three runs around a 0.86 threshold; passes and fails alternately with default aiter present in both outcomes.
  • Disconfirming: the aiter override could perturb numerics slightly; cannot fully exclude a tiny aiter contribution, but the same model also failed (0.859) in a default-aiter baseline on Jun 15.
  • How to verify: re-run gsm8k for gemma-2-27b 3× on both default and override aiter; compare mean vs 0.86.
• [LOW] test_dumper EADDRINUSE is a runner-local port collision / flake, independent of aiter and of the sglang diff.
  • Supporting: error is a TCP bind failure on a hard-coded port 30058; passed in the sister baseline; symptom unrelated to any kernel/numeric path.
  • Disconfirming: passed in baseline, so "fails only with override" pattern technically holds — but EADDRINUSE has no plausible aiter mechanism.
  • How to verify: re-run the job; check for stale processes holding port 30058 on the runner; confirm the test uses a fixed (not ephemeral) port.

In-flight Fix Check

• Mistral-7B GSM8K: ✅ #27757 "Fix Mistral GSM8K chat eval" (open since 2026-06-10) — ping reviewers; do NOT open a duplicate. (search: is:pr is:open gsm8k accuracy threshold)
• gemma-2-27b threshold: ❌ none found specific to the gemma gsm8k threshold. (search: is:pr is:open gemma accuracy)
• test_dumper port collision: ❌ none found; open Grafter PRs (#28024, #27921) are feature work, not port-flake fixes. (search: is:pr is:open dumper grafter)

Suggested Triage Steps

• Chase in-flight fix #27757 for the Mistral-7B GSM8K eval — it directly targets this pre-existing failure.
• For gemma-2-27b, treat as a threshold-tuning issue: re-run gsm8k multiple times on default aiter to characterize variance; if mean ≈ 0.86, lower the threshold slightly or increase sample count to reduce noise.
• For test_dumper, re-run the job; if EADDRINUSE recurs, switch the grafter test from the fixed port 30058 to an ephemeral/free-port allocation, and ensure prior-test process cleanup on the MI30x runner.
• This being an AITER Scout run, no aiter commit should be pursued for these three failures — the A/B baseline (default aiter) reproduces Mistral-7B exactly and exonerates the override for the accuracy cluster.

Status

• Persistence: Mistral-7B GSM8K below threshold across Jun 15 / 17 / 18 (long-standing); gemma-2-27b intermittently straddles 0.86; test_dumper single occurrence (passed Jun 17).
• Scope: job call-nightly-amd / nightly-accuracy-2-gpu only (mirrored in sister nightly-test-amd / nightly-accuracy-2-gpu for the accuracy failures).
• Blocked work: AMD nightly accuracy signal for Mistral-7B and gemma-2-27b is red; obscures genuine accuracy regressions on this job.
• In-flight fix: ✅ #27757 (Mistral-7B); ❌ none for gemma threshold or test_dumper port flake.

call-nightly-amd-rocm720 / nightly-4-gpu-rocm720 — failed step(s): Nightly Test ROCm 7.2 (4-GPU)

Commit Info

• sglang: f83e4d59
• aiter: a40c487b (source: override via AITER_COMMIT_OVERRIDE; Dockerfile default is 7d604afe)

Failed Tests

Test File	Test Function	Error	Log
test/registered/vlm/test_encoder_dp.py	TestVLMEncoderDP.test_vlm_mmmu_benchmark	torch.AcceleratorError: HIP error: invalid device pointer → MMMU accuracy 0.2522 < 0.55 → TIMEOUT 3600s → exit 255	Log

Failure Summary

The job runs a single test, test_vlm_mmmu_benchmark, which in CI picks one random model (test_encoder_dp.py:34) and serves it with --mm-enable-dp-encoder --tp=4. This run selected Qwen/Qwen3-VL-32B-Instruct. A TP2 scheduler crashed at 21:09:30 with HIP error: invalid device pointer while reconstructing multimodal features over CUDA IPC; the server then returned connection errors, the MMMU benchmark scored ~random (0.2522 < 0.55 threshold), and the test ultimately timed out after 3600 s (exit 255).

Regression Status

Flaky / model-selection dependent — fails when Qwen3-VL-32B is the randomly chosen model, passes otherwise.

Recent history of test/registered/vlm/test_encoder_dp.py (test_vlm_mmmu_benchmark) in job nightly-4-gpu-rocm720 (branch=main, event=schedule, completed runs only). The scout run is the aiter-override run; the others are sister nightly-test-amd-rocm720.yml baseline runs (default aiter):

Date	Run	Job	Test File Status	Detail
Jun 18 (scout, override aiter)	run	nightly-4-gpu-rocm720	❌ Failed	Qwen3-VL-32B IPC crash + acc 0.2522 + timeout
Jun 17 (baseline, default aiter)	run	nightly-4-gpu-rocm720	✅ Passed	1/1, 373 s (different model)
Jun 16 (baseline, default aiter)	run	nightly-4-gpu-rocm720	❌ Failed	Identical: Qwen3-VL-32B IPC crash + acc 0.2522 + timeout
Jun 15 (baseline, default aiter)	run	nightly-4-gpu-rocm720	❌ Failed	Timeout 3600 s (no IPC line; likely different/large model)

Failure Origin

pre-existing (sglang) — the identical failure reproduces in the sister baseline with default aiter.

Test File	Test Function	Origin	Baseline Run	Baseline Status
test/registered/vlm/test_encoder_dp.py	test_vlm_mmmu_benchmark	pre-existing (sglang)	Jun 16 run	❌ Same failure (IPC invalid device pointer + Qwen3-VL-32B acc 0.2522 + timeout) in sister job, default aiter

The nearest-by-SHA baseline (Jun 17) passed, but that run randomly selected a different model. The Jun 16 baseline selected Qwen3-VL-32B and reproduced the exact signature without the aiter override — conclusively exonerating aiter.

Failure Cluster

CUDA IPC "invalid device pointer" crash in VLM encoder-DP multimodal feature transport (Qwen3-VL-32B, tp=4).

Facts (verified observations)

• Stack trace (verbatim): crash at cuda_ipc_transport_utils.py:485 → torch.UntypedStorage._new_shared_cuda(...) raises torch.AcceleratorError: HIP error: invalid device pointer, called from schedule_batch.py:362 reconstruct → schedule_batch.py:513 from_processor_output → scheduler.py:1908 _get_multimodal_inputs. (scout log ~line 4701-4728)
• Reading the source: the crash is the non-pooled IPC path at cuda_ipc_transport_utils.py:488 (_new_shared_cuda(*redirected_handle)), which rewrites the handle's device index to rebuild_device_idx before rebuilding. This path is exercised by --mm-enable-dp-encoder (test_encoder_dp.py:24).
• After the TP2 crash, SIGQUIT received ... one child failed; MMMU then logs All 5 attempts failed. Last error: Connection error. repeatedly, and accuracy lands at exactly 0.25222 (≈ random for 4-choice MMMU) — i.e. the accuracy failure and the 3600 s timeout are cascade consequences of the scheduler crash, not independent failures.
• A/B reproducibility: identical signature in Jun 16 sister baseline (default aiter) — [FACT] verified from sister log diff (invalid device pointer + accuracy (0.2522) + TIMEOUT after 3600 seconds).
• aiter override a40c487b was force-rebuilt for this scout run ([CI-AITER-CHECK] markers, scout log 3970-4178).

Hypothesised Causes (with confidence)

• [MEDIUM] The encoder-DP CUDA IPC handle-redirect path (cuda_ipc_transport_utils.py:488, non-pooled _new_shared_cuda) fails to rebuild a multimodal-feature storage on the target device for Qwen3-VL-32B under --mm-enable-dp-encoder --tp=4, raising invalid device pointer.
  • Supporting: the failing line is the literal top of the verified stack trace; failure is model-specific (Qwen3-VL-32B fails, other randomly-chosen models pass on Jun 17); reproduces on both default and override aiter.
  • Disconfirming: cannot pin the introducing commit — the worktree is a shallow clone (94 commits) and git log on cuda_ipc_transport_utils.py does not reach the change; test passed Jun 17, so it is intermittent rather than 100% broken.
  • How to verify: re-run test/registered/vlm/test_encoder_dp.py on AMD MI30x with the model pinned to Qwen/Qwen3-VL-32B-Instruct (bypass the random.choice) and --mm-enable-dp-encoder --tp=4; bisect the history of cuda_ipc_transport_utils.py in a full clone.
• [LOW] aiter override a40c487b involvement.
  • Supporting: this is a scout run with a forced aiter rebuild.
  • Disconfirming: the identical failure occurred in the Jun 16 sister baseline with the default aiter — aiter is ruled out.
  • How to verify: already disconfirmed by A/B baseline.

In-flight Fix Check

• In-flight fix: ⚠️ [#18721](https://github.com/sgl-project/sglang/pull/18721) "[BUG] fix mm_enable_dp_encoder hang for Qwen3-VL models" — open but stale (created and last updated 2026-02-12). Symptom-adjacent (DP encoder + Qwen3-VL) but may predate the current IPC code path; needs verification it addresses the invalid device pointer crash. Searched: mm-enable-dp-encoder, invalid+device+pointer, encoder+dp+IPC.
• In-flight fix: ❌ none found matching the IPC invalid device pointer crash specifically.

Suggested Triage Steps

• Make test_encoder_dp.py deterministic for triage: pin the model to Qwen3-VL-32B (remove/override the random.choice at test_encoder_dp.py:34) so the crash reproduces every run instead of intermittently.
• In a full (non-shallow) clone, bisect git log on python/sglang/srt/utils/cuda_ipc_transport_utils.py and the --mm-enable-dp-encoder reconstruct path; check whether the non-pooled handle-redirect at line 488 mishandles Qwen3-VL-32B feature tensors under tp=4.
• Verify whether #18721 still applies to current cuda_ipc_transport_utils.py; if so, ping reviewers rather than opening a duplicate.
• Do not pursue any aiter-side change — the A/B baseline reproduces the failure without the aiter override.

Status

• Persistence: Intermittent — failed Jun 15, Jun 16, Jun 18; passed Jun 17. Fails specifically when Qwen3-VL-32B is the randomly chosen model.
• Scope: Single test (test_vlm_mmmu_benchmark) in nightly-4-gpu-rocm720; reproduces in both the scout (override aiter) and the regular nightly sister job (default aiter).
• Blocked work: VLM encoder-DP (--mm-enable-dp-encoder) signal on AMD ROCm 7.2 for Qwen3-VL-32B; the accuracy + timeout failures downstream are cascade effects of the CUDA IPC crash.
• In-flight fix: ⚠️ #18721 (stale, needs verification); no current PR targeting the IPC invalid device pointer crash.

call-nightly-amd / nightly-test-1-gpu-unit — failed step(s): Nightly Unit Test (1-GPU)

Commit Info

• sglang: f83e4d59
• aiter: a40c487b (source: AITER_COMMIT_OVERRIDE — forced rebuild; Dockerfile default is 7d604afe)

Failed Tests

Test File	Test Function	Error	Log
test/registered/lora/test_lora_openai_compatible.py	TestLoRAOpenAICompatible.setUpClass	Exception: Server process exited with code 1 (server crashed during startup with httpx.RemoteProtocolError: Server disconnected without sending a response)	Log

Note: the RuntimeError: Test crash after stream_output called 30 times at log line 7929 is not a failure — it is the intentional fault injection of TestCrashDump.test_crash_dump_generated (block 8/9), which passed. The only ✗ FAILED file in the run is test_lora_openai_compatible.py.

Failure Summary

TestLoRAOpenAICompatible.setUpClass failed because popen_launch_server could not bring up the server: meta-llama/Llama-3.2-1B-Instruct was launched in online mode (HF_HUB_OFFLINE=0, required for the LoRA adapter), and the tokenizer initialization aborted while paginating repo templates from huggingface.co with httpx.RemoteProtocolError: Server disconnected without sending a response. The 4 tests in the class all errored on the failed setUpClass.

Regression Status

Flaky / Infrastructure issue (transient HuggingFace Hub network drop).

Recent history of test_lora_openai_compatible.py in the sister job nightly-test-1-gpu-unit (nightly-test-amd.yml, branch main, event=schedule, completed runs only — the scout sister baseline is the comparable signal since scout runs only Mon/Thu):

Date	Run	Job	Test File Status	Failed Function	Error
Jun 17	27710225107	nightly-test-1-gpu-unit	✅ Passed	—	—
Jun 15	27568949589	nightly-test-1-gpu-unit	✅ Passed (job success)	—	—
Jun 14	27507337214	nightly-test-1-gpu-unit	✅ Passed (job success)	—	—

The test passes consistently; this is a one-off transient failure, not a regression.

Failure Origin

Test File	Test Function	Origin	Baseline Run	Baseline Status
test/registered/lora/test_lora_openai_compatible.py	TestLoRAOpenAICompatible.setUpClass	unclear (infrastructure / HF Hub network flake — not aiter)	27710225107	✅ Passed (sister job, default aiter, Jun 17)

Although the mechanical A/B rule ("passes in sister baseline → aiter-caused") would point at aiter, the disconfirming evidence is overwhelming: the crash is entirely in the HuggingFace Hub HTTP layer (huggingface_hub.paginate → httpx → connection reset), with no GPU, kernel, or attention code in the stack trace. The aiter override rebuilds attention/GEMM kernels and cannot cause an HTTP connection to huggingface.co to drop. Origin is therefore reported as unclear (transient infra), and no aiter commits are cited.

Failure Cluster

HF Hub connection reset during server tokenizer startup (transient network flake)

Facts (verified observations)

• [FACT] Server-side root error (log lines 5290–5366): httpcore.RemoteProtocolError: Server disconnected without sending a response, re-raised as httpx.RemoteProtocolError during huggingface_hub/utils/_pagination.py:35 paginate → transformers ... list_repo_templates → tokenizer.from_pretrained.
• [FACT] Launch command (log line 5258 area): sglang serve --model-path meta-llama/Llama-3.2-1B-Instruct ... --enable-lora --lora-paths tool_calling=... with HF_HUB_OFFLINE=0 (online mode, so the run depends on live HF Hub connectivity).
• [FACT] Test harness failure (log line 5369): ERROR: setUpClass (__main__.TestLoRAOpenAICompatible) → Exception: Server process exited with code 1; FAILED (errors=1).
• [FACT] Sister baseline nightly-test-1-gpu-unit (default aiter) passed test_lora_openai_compatible.py on Jun 17 (✓ PASSED, log line 9284), and the job concluded success on Jun 15 and Jun 14.
• [FACT] An unrelated transient network error also appears earlier in this same run at log line 89: RPC failed; curl 56 Recv failure: Connection reset by peer during git fetch — corroborating intermittent network instability on this runner.
• [FACT] The _trigger_crash_for_tests RuntimeError (line 7952) is expected behavior for TestCrashDump; block 8/9 ended normally.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [HIGH] Transient HuggingFace Hub connectivity failure during tokenizer/template fetch.
  • Supporting: stack trace terminates in huggingface_hub pagination over httpx with Server disconnected without sending a response; the server launched in online mode (HF_HUB_OFFLINE=0); a second independent network reset (curl 56) occurred earlier in the same run.
  • Disconfirming: (none found) — no code path in sglang or aiter appears below the HTTP layer in the trace.
  • How to verify: re-run the job; if it passes with no code change, the flake is confirmed.
• [LOW] aiter override a40c487b.
  • Supporting: only that the override forced an aiter rebuild for this run.
  • Disconfirming: identical test passed in the sister baseline with default aiter; failure is purely in the HF Hub HTTP layer, unreachable from any aiter kernel.
  • How to verify: not actionable — no kernel/GPU code in the failing path.

In-flight Fix Check

• In-flight fix: ❌ none found / not applicable — this is a transient network flake, not a code defect. No PR search performed because there is no code path to fix; a maintainer should simply re-run.

Suggested Triage Steps

• Re-run the failed scout job nightly-test-1-gpu-unit (run 27788558636); a clean pass confirms the HF Hub flake.
• If test_lora_openai_compatible.py flakes repeatedly on HF Hub fetches, consider pre-caching meta-llama/Llama-3.2-1B-Instruct (and the tool_calling LoRA) into the runner image so the LoRA path can run with HF_HUB_OFFLINE=1, removing the live-network dependency.

Status

• Persistence: One-off — first and only observed failure of this test on this commit; sister baseline passed Jun 17/15/14.
• Scope: Single test file (test_lora_openai_compatible.py) in one job (call-nightly-amd / nightly-test-1-gpu-unit); a second unrelated network reset (git fetch) in the same run suggests runner-level network instability.
• Blocked work: None of substance — transient infra; LoRA OpenAI-compatible signal is otherwise green on AMD.
• In-flight fix: ❌ none found / not applicable (network flake, not a code defect).

call-nightly-amd / nightly-accuracy-2-gpu-vlm — failed step(s): Nightly Accuracy Test (2-GPU VLM MMMU)

Commit Info

• sglang: f83e4d59
• aiter: a40c487b (source: AITER_COMMIT_OVERRIDE; Dockerfile default was 7d604afe, forced rebuild — verified from [CI-AITER-CHECK] markers at log lines 4029-4031)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi30x/test_vlms_mmmu_eval_amd.py	test_mmmu_vlm_models	AssertionError: The following models failed (5 VLMs below MMMU threshold)	Log

5 models below threshold (scout run):

Model	Score	Threshold
Qwen/Qwen2-VL-7B-Instruct	0.2799	0.3000
Qwen/Qwen2.5-VL-7B-Instruct	0.2901	0.3300
Qwen/Qwen3-VL-30B-A3B-Instruct	0.2601	0.2900
openbmb/MiniCPM-o-2_6	0.2700	0.3200
deepseek-ai/deepseek-vl2-small	0.2802	0.3100

Failure Summary

test_mmmu_vlm_models in test_vlms_mmmu_eval_amd.py failed because 5 of the parameterized VLMs scored below their configured MMMU accuracy thresholds (_check_results, line 380). Other models in the same job passed (e.g. MiMo-VL-7B 0.310/0.27, DeepSeek-OCR-2 0.3099/0.25). The shortfall margins are small and stable (~0.01–0.04 below threshold).

Regression Status

Known recurring failure (persistent in queryable window).

Recent history of test_vlms_mmmu_eval_amd.py (test_mmmu_vlm_models) — comparing the sister nightly-test-amd.yml job nightly-accuracy-2-gpu-vlm (same branch, event=schedule, completed runs only):

Date	Run	Job	Test File Status	Failed Models	Notes
Jun 18	27788558636	scout	❌ Failed	same 5	this run (aiter override)
Jun 17	27710225107	vlm	❌ Failed	same 5	default aiter
Jun 15	27568949589	vlm	❌ Failed	same 5	default aiter
Jun 14	27507337214	vlm	❌ Failed	same 5	default aiter

No passing run found in the Jun 14–18 lookback window (Jun 16/17 top-level nightly runs were cancelled; the vlm job itself failed in every completed instance). First-observed date predates the queryable window.

Failure Origin

Test File	Test Function	Origin	Baseline Run	Baseline Status
test/registered/amd/accuracy/mi30x/test_vlms_mmmu_eval_amd.py	test_mmmu_vlm_models	pre-existing (sglang)	27710225107	❌ Same failure, same 5 models, near-identical scores (sister job, default aiter)

The sister nightly-test-amd.yml baseline (default aiter, no override) fails on the identical test function with the identical 5 models at near-identical scores (e.g. Qwen3-VL-30B = 0.2601 in both; Qwen2-VL-7B = 0.2799 in both). This conclusively exonerates the aiter override — the failure reproduces without it.

Failure Cluster

VLM MMMU accuracy below threshold (5 models, mi30x nightly)

Facts (verified observations)

• Log line 15337-15342: AssertionError: The following models failed listing the 5 models + scores (verbatim from scout log).
• [CI-AITER-CHECK] markers (lines 4029-4031): Dockerfile default aiter 7d604afe, overridden to a40c487b, rebuilt.
• Sister baseline Jun 17 (job 81969145606) log lines 15066-15070: same 5 models fail with default aiter.
• Scores are extremely stable across runs Jun 14/15/17/18 (Qwen3-VL-30B = 0.2601 every run; Qwen2.5-VL spans 0.2802–0.2901 vs 0.33 threshold) — i.e. the shortfall is reproducible, not noise.
• Thresholds in the test file were last modified 2026-03-11 in dc4380e3 (verified via GitHub commits API); unchanged for ~3 months. Current values: Qwen2-VL 0.30, Qwen2.5-VL 0.33, Qwen3-VL-30B 0.29, MiniCPM-o-2_6 0.32, deepseek-vl2-small 0.31 (test_vlms_mmmu_eval_amd.py:50-89).
• Job ran with disable_cuda_graph=True, attention_backend='aiter', mem_fraction_static=0.7 (from server_args dump, line 15183).

Hypothesised Causes (with confidence)

• [LOW] Thresholds miscalibrated vs actual mi30x VLM accuracy (test-config drift, not a code regression).
  • Supporting: shortfalls are small (~0.01–0.04) and stable across 4+ runs and 5 different VLM architectures; thresholds unchanged since March while scores sit consistently just below them; an open WIP PR (#26178) acknowledges mi300 thresholds need relaxing (though for perf tests, not this file).
  • Disconfirming: 5 models all dipping together could equally reflect a real shared-code accuracy regression in the VLM/multimodal path rather than threshold drift; I have no green run in the window to anchor the "scores dropped" claim, so I cannot rule out a genuine quality regression that landed before Jun 14.
  • How to verify: pull mi30x vlm history before Jun 14 to find the last run where these 5 passed; if they never passed at these thresholds, it's config; if they regressed, bisect that window.
• [LOW] Shared multimodal/accuracy regression in sglang affecting multiple VLM families.
  • Supporting: 5 distinct architectures (Qwen2/2.5/3-MoE, MiniCPM-o, deepseek-vl2) all fall just short — consistent with a common code path.
  • Disconfirming: other VLMs in the same job pass (MiMo-VL, DeepSeek-OCR-2); a global multimodal regression would more likely affect them too.
  • How to verify: bisect sglang main on a window ending before Jun 14 using one cheap model (Qwen2-VL-7B) on mi30x.

No hypothesis above LOW confidence; needs a pre-Jun-14 green/red boundary on the sister vlm job to distinguish threshold drift from a real regression.

In-flight Fix Check

• In-flight fix: ❌ none directly applicable. Searched open PRs (mmmu vlm, VLM accuracy amd, Qwen2-VL threshold). The closest is #26178 "[AMD] WIP - relaxing thres for mi300" (open, last updated 2026-05-26), but it only edits perf test files (test_bench_serving_1gpu_part1/2.py, test_vlm_perf_5090.py), not test_vlms_mmmu_eval_amd.py — it does not address this accuracy failure.

Suggested Triage Steps

• Query the sister nightly-accuracy-2-gpu-vlm job further back than Jun 14 to locate the last run where these 5 models passed; this is the single most useful step to separate "thresholds too high" from "accuracy regressed".
• If a regression boundary exists, bisect sglang main across pass_sha..fail_sha using one model (Qwen/Qwen2-VL-7B-Instruct) on mi30x to keep iteration cheap.
• If no passing run is ever found at these thresholds, treat as threshold miscalibration and recalibrate accuracy_threshold values in test_vlms_mmmu_eval_amd.py:50-89 against measured mi30x scores (maintainer decision).
• Do not investigate aiter for this failure — the default-aiter sister baseline reproduces it identically.

Status

• Persistence: Failing in every completed nightly-accuracy-2-gpu-vlm run observed Jun 14–18 (≥5 days); first-observed date predates the queryable window.
• Scope: 1 test function, 5 VLM models; reproduces in both amd-aiter-scout.yml and the regular nightly-test-amd.yml (default aiter).
• Blocked work: AMD mi30x VLM accuracy CI signal for these 5 models.
• In-flight fix: ❌ none directly applicable (#26178 touches perf tests only).

call-nightly-amd / nightly-2-gpu-mi35x-deepseek-r1-mxfp4-tp2 — failed step(s): Accuracy Test MI35x (2-GPU DeepSeek-R1-MXFP4 TP2)

Commit Info

• sglang: f83e4d59
• aiter: a40c487b (source: AITER_COMMIT_OVERRIDE, rebuilt; Dockerfile default was 7d604afe)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi35x/test_deepseek_r1_mxfp4_tp2_mi35x.py	TestDeepSeekR1MXFP4TP2MI35x.setUpClass	TimeoutError: Server failed to start within the timeout period	Log

Failure Summary

The accuracy test for amd/DeepSeek-R1-MXFP4-Preview (TP2, MI35x) failed in setUpClass: popen_launch_server raised TimeoutError after ~60 min because the SGLang server never finished initialization. The schedulers were stuck downloading the model weights from HuggingFace Hub (download_weights_from_hf → snapshot_download → xet_get) the entire time, so Ran 0 tests in 3618s. No test logic or kernel ever executed.

Regression Status

Known recurring failure (intermittent, infra-flavoured) — this job has failed in the last two scout runs but with weight-load symptoms, while the sister nightly passes.

Recent history of test_deepseek_r1_mxfp4_tp2_mi35x.py (TestDeepSeekR1MXFP4TP2MI35x.setUpClass) in job call-nightly-amd / nightly-2-gpu-mi35x-deepseek-r1-mxfp4-tp2 (amd-aiter-scout.yml, event=schedule, branch main, completed runs only):

Date	Run	Test File Status	Failed Function	Error
Jun 18	27788558636	❌ Failed	setUpClass	TimeoutError — stuck in HF weight download (~60 min)
Jun 11	27376585476	❌ Failed	setUpClass	Server process exited with code -9 during weight download/load (~20 min)
Jun 8	27166336072	✅ Passed	—	—

First observed failure (this test, this job): Jun 11 2026. Last known passing: Jun 8 2026. Lookback: last 8 scheduled scout runs.

Failure Origin

Test File	Test Function	Origin	Baseline Run	Baseline Status
test_deepseek_r1_mxfp4_tp2_mi35x.py	TestDeepSeekR1MXFP4TP2MI35x.setUpClass	unclear (infrastructure / HF download)	27568949589	✅ Passed (sister nightly-test-amd, default aiter)

The sister nightly-test-amd.yml job nightly-2-gpu-mi35x-deepseek-r1-mxfp4-tp2 passed on Jun 15 (81500156998) and Jun 14 (81300834061) with the default aiter. Because the failure occurs entirely inside the HuggingFace weight download — before any model forward pass or aiter kernel runs — the aiter override is not in the failing code path. Origin is therefore not aiter-caused; it is an infrastructure/download stall (unclear).

Failure Cluster

Server fails to start — HuggingFace weight download stall during DeepSeek-R1-MXFP4 model load.

Facts (verified observations)

• setUpClass raised TimeoutError: Server failed to start within the timeout period at test_utils.py:1028; Ran 0 tests in 3618.383s, FAILED (errors=1), step exit code 255 (log).
• Server process 4047 launched ~21:01 UTC; Scheduler/TokenizerManager watchdog (watchdog_timeout=300, soft=True) fired every ~5 min from 21:06 through 22:01.
• py-spy dumps throughout show both schedulers blocked in weight download: TP0 in download_weights_from_hf (weight_utils.py:556) → filelock _poll_until_acquired; TP1 in snapshot_download → thread_map → xet_get (file_download.py:566). Stack: model_runner.py:1427 load_model → loader.py:743 → deepseek_v2.py:2742 load_weights.
• Earlier infra hiccup in the same run: Local docker registry pull failed; falling back to public registry (log), indicating degraded network on this runner.
• aiter was rebuilt from override a40c487b (Dockerfile default 7d604afe) — verified from [CI-AITER-CHECK] markers.
• Sister nightly baseline (default aiter) passed Jun 14 & Jun 15.
• Prior scout failure (Jun 11) was also a weight-download failure but ended in Server process exited with code -9 (process killed) after ~20 min, not a clean timeout.

Hypothesised Causes (with confidence)

• [HIGH] HuggingFace Hub weight download stalled/was too slow for amd/DeepSeek-R1-MXFP4-Preview (a very large model) and never completed within the launch timeout.
  • Supporting: every py-spy dump over the full hour shows the schedulers parked in download_weights_from_hf/xet_get/filelock; 0 tests ran; same runner also failed its local docker-registry pull (network degradation).
  • Disconfirming: the same test passed in the sister nightly on Jun 14/15, so the model and code path are healthy when the download succeeds — i.e. this is environment/network-dependent, not deterministic.
  • How to verify: re-run the job; check whether the model is pre-cached on the MI35x runner (branches ci/precache-hf-models, alexnails/model-cache-sync-migration exist) and inspect runner network/xet throughput at launch.
• [LOW] aiter override a40c487b involvement.
  • Supporting: scout forces an aiter rebuild and this job failed twice.
  • Disconfirming: the failure is entirely in HF weight download, before any aiter kernel executes; sister baseline with default aiter passes. aiter is not in the failing path.
  • How to verify: not actionable — no aiter code runs before the stall.

No hypothesis implicates an sglang or aiter code commit; the failing path is the network download, not model/kernel code.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for DeepSeek-R1-MXFP4 / precache / hf download timeout; no matching fix (only unrelated PRs #25647, #13595, #17356). Note: branches ci/precache-hf-models and alexnails/model-cache-sync-migration exist but have no open PR.

Suggested Triage Steps

• Re-run job 82231478040 — if it passes, confirm intermittent download/network flake.
• Pre-cache amd/DeepSeek-R1-MXFP4-Preview on the MI35x runners (or seed the local model cache) so the launch path does not hit HF Hub / xet at test time; track via the ci/precache-hf-models effort.
• If stalls persist, consider raising the server launch timeout for this large-model accuracy test and/or disabling the HF xet backend (HF_HUB_DISABLE_XET=1) to compare download reliability.
• Investigate the runner's degraded network (the local docker-registry pull also failed in this run) before attributing to SGLang/aiter code.

Status

• Persistence: Failed in the last 2 scheduled scout runs of this job (Jun 11 & Jun 18); last pass Jun 8. Symptom is weight-download stall both times, exact error differs (-9 vs TimeoutError).
• Scope: This scout job (nightly-2-gpu-mi35x-deepseek-r1-mxfp4-tp2). Sister nightly-test-amd runs of the same test pass with default aiter, so signal loss is confined to the scout path for this one large model.
• Blocked work: DeepSeek-R1-MXFP4 TP2 accuracy signal on MI35x under the AITER Scout.
• In-flight fix: ❌ none found — needs runner/network investigation or HF pre-cache; not a code revert.

call-nightly-amd / nightly-4-gpu — failed step(s): Nightly Test (4-GPU)

Commit Info

• sglang: f83e4d59
• aiter: a40c487b (source: AITER_COMMIT_OVERRIDE — forced rebuild; Dockerfile default was 7d604afe)

Failed Tests

Test File	Test Function	Error	Log
test/registered/vlm/test_encoder_dp.py	TestVLMEncoderDP.test_vlm_mmmu_benchmark (model zai-org/GLM-4.1V-9B-Thinking)	Exception: Server process exited with code -9 (GPU memory access fault, retries exhausted)	Log
test/registered/vlm/test_encoder_dp.py	TestVLMEncoderDP.test_vlm_mmmu_benchmark (model Qwen/Qwen3-VL-32B-Instruct)	Exception: Server process exited with code -9 (GPU memory access fault)	Log

Failure Summary

The single test file in suite nightly-amd-4-gpu for this job, test/registered/vlm/test_encoder_dp.py, crashed. The VLM server starts and completes CUDA-graph capture successfully, then on the first forward pass (warmup / health_generate) the GPU aborts with Memory access fault by GPU node-2 ... Reason: Write access to a read-only page (log:4722), and the server process is SIGKILLed (exit -9). This occurred for both VLM models the test happened to exercise (GLM-4.1V-9B-Thinking and Qwen3-VL-32B-Instruct), both running attention_backend='aiter', tp_size=4, mm_enable_dp_encoder=True.

Regression Status

Known recurring failure (flaky test design — random model selection). The test picks one model at random per attempt (models_to_test = [random.choice(MODELS)] when is_in_ci(), test_encoder_dp.py:32-33), where MODELS = Qwen2.5-VL-72B, Qwen3-VL-32B, InternVL2_5-8B, GLM-4.1V-9B-Thinking. The underlying memory-access fault hits the GLM-4.1V and Qwen3-VL-32B draws; InternVL2_5-8B passes. File pass/fail is therefore a coin-flip on the random draw.

Recent history of test_encoder_dp.py (test_vlm_mmmu_benchmark) in job nightly-4-gpu (sister workflow nightly-test-amd.yml, event=schedule, branch main, completed runs only):

Date	Run	Job	Test File Status	Model drawn / Failed Function	Error
Jun 18	27788558636 (scout, override aiter)	nightly-4-gpu	❌ Failed	GLM-4.1V + Qwen3-VL-32B	Memory access fault → exit -9
Jun 17	27710225107 (default aiter)	nightly-4-gpu	✅ Passed	GLM-4.1V crashed, retry drew InternVL2_5-8B (passed)	mem fault on GLM only (log:4471)
Jun 16	27640378823 (default aiter)	nightly-4-gpu	❌ Failed	Qwen3-VL-32B + GLM-4.1V	Memory access fault → exit -9
Jun 15	27568949589 (default aiter)	nightly-4-gpu	✅ Passed	(non-crashing model drawn)	—
Jun 14	27507337214 (default aiter)	nightly-4-gpu	✅ Passed	(non-crashing model drawn)	—

The fault reproduces with the default aiter on Jun 16 and Jun 17, so it predates and is independent of the scout's aiter override.

Failure Origin

Test File	Test Function	Origin	Baseline Run	Baseline Status
test/registered/vlm/test_encoder_dp.py	test_vlm_mmmu_benchmark (GLM-4.1V)	pre-existing (sglang)	27710225107 (Jun 17, default aiter)	⚠️ Same GLM-4.1V memory fault present (file passed only because retry drew InternVL2_5-8B)
test/registered/vlm/test_encoder_dp.py	test_vlm_mmmu_benchmark (Qwen3-VL-32B)	pre-existing (sglang)	27640378823 (Jun 16, default aiter)	❌ Same Qwen3-VL-32B memory fault → file failed

A/B verdict: The identical Write access to a read-only page fault on the same test file + the same models occurs in the sister nightly-test-amd.yml runs under the Dockerfile-default aiter (no override). The aiter override a40c487b is therefore exonerated — per project rules, zero aiter commits are cited below.

Failure Cluster

GPU memory access fault (write to read-only page) during VLM warmup — encoder-DP / TP=4

Facts (verified observations)

• Memory access fault by GPU node-2 (...) on address 0x... Reason: Write access to a read-only page — scout log:4722; 6 fault lines total in the scout job.
• Fault fires after Application startup complete + first GET /model_info 200, on the first inference; preceded by aiter module_fmha_v3_varlen_fwd JIT build (log:4742). Server then SIGKILLed (exit code -9).
• Same fault + same models reproduce with default aiter (AITER_COMMIT_OVERRIDE= empty, v0.1.14rc1.dev240+g7d604afe5) in sister run 27640378823 (Jun 16, ❌) and 27710225107 (Jun 17, GLM crashed but file passed via random retry).
• Test selects one random model per attempt (test_encoder_dp.py:32-33); InternVL2_5-8B passes, GLM-4.1V & Qwen3-VL-32B crash.
• Common config across crashing servers: attention_backend='aiter', tp_size=4, mm_enable_dp_encoder=True, enable_multimodal=True, mem_fraction_static=0.68.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [MEDIUM] Pre-existing sglang bug in the VLM encoder data-parallel path (--mm-enable-dp-encoder) at TP=4 for GLM-4.1V / Qwen3-VL-32B — an out-of-bounds / read-only-page write during the first multimodal forward pass.
  • Supporting: reproduces under default aiter on two separate days; fault is a host-visible HSA write-to-read-only-page on first forward; only encoder-DP VLM models at TP=4 are affected; InternVL2_5-8B (same path) survives, suggesting a model/shape-specific buffer issue rather than a generic aiter kernel failure.
  • Disconfirming: no specific commit isolated (test randomness prevents clean pass_sha..fail_sha bisection); cannot rule out that InternVL2_5-8B simply has a smaller footprint that hides the same bug.
  • How to verify: run test_encoder_dp.py on AMD MI30x at TP=4 forcing each model deterministically (bypass random.choice) with default aiter; if GLM-4.1V / Qwen3-VL-32B fault and InternVL2_5-8B passes, the bug is model-specific in the encoder-DP path. Then bisect with --mm-enable-dp-encoder removed to confirm the DP-encoder dependency.
• [LOW] aiter module_fmha_v3_varlen_fwd kernel as the faulting site (it logs immediately before the fault).
  • Supporting: the aiter FMHA build line is the last log before the crash.
  • Disconfirming: log-line proximity is not causation; the fault reproduces with the default aiter, so the override is not responsible; the crash is on the next forward op, not necessarily inside the aiter call.
  • How to verify: not actionable for aiter attribution — baseline already reproduces without the override.

No hypothesis above MEDIUM; deterministic per-model repro + DP-encoder A/B is needed to localize the sglang code path.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for memory access fault (12 open PRs, none match VLM encoder-DP warmup at TP=4) and encoder dp vlm (8 open PRs; #25951 "shard pixel_values per-rank" is the closest topical match but does not reference this crash). No PR addresses this fault.

Suggested Triage Steps

• Reproduce deterministically: patch out random.choice (or set the model explicitly) and run test/registered/vlm/test_encoder_dp.py per-model on AMD MI30x at TP=4 with the default aiter image — confirm GLM-4.1V & Qwen3-VL-32B fault while InternVL2_5-8B passes.
• A/B the DP-encoder flag: rerun the crashing models with and without --mm-enable-dp-encoder to confirm the encoder-DP path is required to trigger the fault.
• If the DP-encoder dependency is confirmed, inspect the multimodal encoder-DP scatter/gather buffers (e.g. per-rank pixel_values sharding, ref. #25951) for an out-of-bounds write for these model shapes.
• Separately, fix the flaky test design: the random single-model draw masks a real crash on 2 of 4 models and produces non-deterministic green/red — consider iterating all models (or pinning) so the crash is always surfaced.
• Do not pursue aiter-side reverts: the baseline reproduces the fault with the Dockerfile-default aiter.

Status

• Persistence: Recurring/flaky — same memory fault on GLM-4.1V / Qwen3-VL-32B observed Jun 16 (❌), Jun 17 (GLM crashed, file passed by random draw), Jun 18 (❌). Latent at least since Jun 16; green runs only reflect lucky model draws.
• Scope: test/registered/vlm/test_encoder_dp.py in nightly-4-gpu across amd-aiter-scout.yml and the sister nightly-test-amd.yml. Blocks reliable VLM encoder-DP (TP=4) CI signal on AMD.
• In-flight fix: ❌ none found — needs new investigation PR.

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-06-19 09:30 UTC)

[amd-bot]

amd-aiter-scout.yml — Per-Job Analysis (continued 3/3)

call-pr-test-amd / call-pr-test-amd-extra / extra-a-test-1-gpu-large-amd${{ ((inputs.rocm_version && format('-{0}', inputs.rocm_version)) || '') }} (linux-${{ (inputs.runner_arch || 'mi325') }}-1gpu-sglang) — failed step(s): N/A

Commit Info

• sglang: f83e4d59 — refactor(runner): unify eager-forward DP/MLP-sync padding into one helper (#28383)
• aiter: N/A — the job never executed, so no [CI-AITER-CHECK] markers were emitted (log blob does not exist).

Failed Tests

Test File	Test Function	Error	Log
N/A	N/A	Job cancelled before execution — no test ran	Job (no log blob)

No tests ran. This is not a test failure.

Failure Summary

Job extra-a-test-1-gpu-large-amd … was cancelled with zero duration (started_at 2026-06-18T20:52:05Z, completed_at 2026-06-18T20:52:04Z — completed before it started), recorded 0 steps, and its log blob returns BlobNotFound. The job never executed a single step. Its sibling extra-a-test-1-gpu-small-amd and the upstream pr-gate job were cancelled in the same instant, and call-gate was skipped — a gate-driven fast-fail cancellation cascade, not a code/test failure.

Regression Status

Infrastructure / orchestration event (not a regression, not a flake). The job was cancelled by the workflow's gate logic before any test executed. There is no per-test history to compare because no test ran in this job.

Run-level context (run 27788558636, event=schedule, AMD AITER Scout) at snapshot time:

Status/Conclusion	Job count
completed / success	32
completed / failure	12
completed / cancelled	3 (incl. this job + extra-a-small + pr-gate)
completed / skipped	1 (call-gate)
queued	48
in_progress	4

The overall run is still in progress [IN-FLIGHT, partial data] — 48 jobs queued, 4 running. No trend or regression conclusion can be drawn for this job.

Failure Origin (required for amd-aiter-scout.yml)

Test File	Test Function	Origin	Baseline Run	Baseline Status
N/A	N/A	unclear	—	No test executed; Baseline A/B check N/A

The Baseline A/B check is not applicable: the job was cancelled before running any test, so there is no failing (test file + function) to compare against the sister pr-test-amd baseline.

Failure Cluster

Gate fast-fail cancellation (job cancelled before execution — no test ran)

Facts (verified observations, no interpretation)

• Job conclusion: cancelled; started_at 2026-06-18T20:52:05Z, completed_at 2026-06-18T20:52:04Z (negative duration ⇒ never executed). [FACT]
• Job recorded 0 steps via the GitHub jobs API. [FACT]
• Log blob returns <Code>BlobNotFound</Code> — no log was ever produced. [FACT]
• Sibling job extra-a-test-1-gpu-small-amd and gate job pr-gate were also cancelled (0 steps each); call-gate was skipped. [FACT]
• Parent run 27788558636 is still in progress (48 queued, 4 in-progress jobs) at snapshot. [FACT]
• sglang head SHA f83e4d59 verified to exist in the local repo. [FACT]

Hypothesised Causes (with confidence, with disconfirming evidence)

• [MEDIUM] The job was cancelled by the workflow's gate/fast-fail logic when an upstream stage failed (12 jobs already failed in this run), causing the extra-a-* downstream jobs and pr-gate to be cancelled before launch.
  • Supporting: pr-gate (the gate job) and both extra-a-* jobs were cancelled in the same instant; call-gate was skipped; 12 sibling jobs had already failed. This matches a fast-fail cascade that skips/cancels not-yet-started downstream jobs.
  • Disconfirming: with no log/annotations on the cancelled jobs, the exact cancellation trigger (gate fast-fail vs. concurrency supersede vs. manual cancel) is not directly observable. No newer scheduled run was confirmed to have superseded this one.
  • How to verify: inspect the pr-gate job's cancellation reason and the workflow's concurrency/fast-fail config in amd-aiter-scout.yml (and the called pr-test-amd.yml gate); check whether the 12 upstream failures triggered the gate to short-circuit the extra stage.
• [LOW] Concurrency cancellation by a newer scheduled run of the same workflow.
  • Supporting: scout runs on a cron; a superseding run can cancel an older one's in-flight/queued jobs.
  • Disconfirming: the rest of run 27788558636 is still actively progressing (48 queued, 4 running), so the whole run was not concurrency-cancelled — only the gate + extra jobs were, which points to gate logic rather than run-level supersede.
  • How to verify: list recent event=schedule runs of amd-aiter-scout.yml and check for a newer run created after 2026-06-18T20:52Z.

In-flight Fix Check

Not applicable — there is no test failure or code defect to fix. A cancelled-before-execution job is an orchestration outcome, not a bug requiring a fix PR. (No PR search performed; no failing test/library keyword exists to search on.)

Suggested Triage Steps

• Treat this job as non-signal: it produced no test result. Do not count it as a failure or a regression for f83e4d59.
• Once run 27788558636 completes, re-evaluate using the 12 executed failures (and any that emerge from the 48 queued jobs) — those carry the real scout signal.
• If extra-stage coverage on the large/small 1-GPU AMD jobs is needed for this scout, re-run the cancelled extra-a-* jobs (or the workflow) after the upstream gate failures are resolved, so the gate does not short-circuit them.
• Inspect the gate behavior in amd-aiter-scout.yml / called pr-test-amd.yml to confirm whether the 12 upstream failures intentionally cancelled the extra stage (expected fast-fail) vs. an unintended cancellation.

Status (factual, no priority assignment)

• Persistence: single occurrence; job cancelled at 2026-06-18T20:52Z before executing. No history to assess (no test ran).
• Scope: 3 cancelled jobs (extra-a-large, extra-a-small, pr-gate) + 1 skipped (call-gate) in this run; separately, 12 jobs failed (executed) in the same run — those are tracked independently.
• Blocked work: the extra-a 1-GPU large/small AMD test stage produced no coverage in this scout run.
• In-flight fix: N/A — orchestration cancellation, not a code/test defect.

call-pr-test-amd / call-pr-test-amd-extra / extra-a-test-1-gpu-small-amd${{ ((inputs.rocm_version && format('-{0}', inputs.rocm_version)) || '') }} (linux-${{ (inputs.runner_arch || 'mi325') }}-1gpu-sglang) — failed step(s): N/A

Commit Info

• sglang: f83e4d59 — refactor(runner): unify eager-forward DP/MLP-sync padding into one helper (#28383)
• aiter: N/A — the job never executed any step, so no [CI-AITER-CHECK] markers were emitted; the log blob does not exist.

Failed Tests

Test File	Test Function	Error	Log
N/A	N/A	Job cancelled before any step ran — not a test failure	Job

Failure Summary

The job extra-a-test-1-gpu-small-amd (ID 82231483725) has conclusion: cancelled, started_at: 2026-06-18T20:52:05Z, completed_at: 2026-06-18T20:52:04Z, and 0 steps. No test ran and no log was produced (GET .../logs → BlobNotFound). This is a job-cancellation/infrastructure event, not a code or test failure.

Regression Status

Infrastructure issue (cancellation at queue time). Not a test regression — no test was ever exercised.

Context from the same run 27788558636 (event=schedule, run still in-flight — status: queued):

• 178 total jobs: 32 success, 12 failure, 48 queued, 4 in_progress, 1 skipped, 3 cancelled.
• The three cancelled jobs all share the same instant (started 20:52:05, completed 20:52:04, 0 steps each) and are siblings under call-pr-test-amd-extra:
  • call-gate / pr-gate (ID 82231478836)
  • extra-a-test-1-gpu-large-amd (ID 82231483308)
  • extra-a-test-1-gpu-small-amd (ID 82231483725) ← this job

The pr-gate gate job was cancelled at the same instant, and the two extra-a-* jobs that depend on it were cancelled together — consistent with a gate/concurrency cancellation cascade at run startup, not a per-test failure.

Failure Origin

unclear — the Baseline A/B check (sister-workflow comparison) is not applicable: no test ran, no log exists, and there is no failing (test_file, test_function) to compare against a sister pr-test-amd.yml baseline. There is nothing to attribute to either sglang or aiter.

Failure Cluster

Job cancelled at startup (gate/concurrency cancellation, no steps run) — symptom-level grouping covering the three sibling jobs cancelled at 20:52:04–05.

Facts (verified observations)

• GET /actions/jobs/82231483725 → status: completed, conclusion: cancelled, steps: [], started_at: 2026-06-18T20:52:05Z, completed_at: 2026-06-18T20:52:04Z (completion timestamp precedes start — cancelled essentially at queue admission).
• GET /actions/jobs/82231483725/logs → HTTP body <Error><Code>BlobNotFound</Code>...</Error> — no log artifact.
• Parent run 27788558636 is still in-flight: status: queued, conclusion: null, run_attempt: 1, 48 jobs queued + 4 in_progress at snapshot.
• Two sibling jobs and the pr-gate gate job were cancelled in the same second with 0 steps.

Hypothesised Causes (with confidence)

• [LOW] Gate/concurrency cancellation cascade. The call-gate / pr-gate job cancelled at the same instant as the two dependent extra-a-* jobs; GitHub cancels dependent matrix jobs when their gate/needs job is cancelled.
  • Supporting: identical completed_at (20:52:04) across all three; 0 steps on each; the gate job is an upstream dependency of the extra-a stage.
  • Disconfirming: no log is available to confirm the exact cancellation reason; the run is still queued overall, so this may also be a transient scheduler/runner-admission event rather than an explicit cancel.
  • How to verify: inspect the workflow amd-aiter-scout.yml needs:/concurrency config for call-pr-test-amd-extra; check whether a later run_attempt of run 27788558636 re-runs these jobs to completion.
• No hypothesis above LOW confidence; this is an infrastructure cancellation, not a code-attributable failure.

In-flight Fix Check

Not applicable — there is no failing test, library, or error signature to search for. No fix PR is warranted for a startup cancellation.

Suggested Triage Steps

• Treat this job as not a real failure: it was cancelled before any step ran. Exclude it from failure tallies for this scout run.
• Wait for run 27788558636 to finish (still queued) — re-evaluate the extra-a-test-1-gpu-small-amd job only once it actually executes. The real signal for this shard is not yet available.
• If these call-pr-test-amd-extra jobs were cancelled because the pr-gate job was cancelled, check the gate job's cancellation reason in the Actions UI; if it's a transient runner-admission issue, a re-run of the run attempt should clear it.
• Do not open an aiter or sglang attribution for this entry — no test was exercised and no A/B baseline is possible.

Status

• Persistence: single occurrence at run startup (2026-06-18 20:52:04 UTC); cannot establish persistence without a real executed run.
• Scope: 3 jobs in this one run cancelled together (pr-gate, extra-a-large, extra-a-small); parent run still in-flight (48 queued / 4 in_progress at snapshot).
• Blocked work: the extra-a 1-GPU small/large AMD test signal for scout run 27788558636 is missing until these jobs re-run.
• In-flight fix: N/A (infrastructure cancellation, no code fix applicable).

call-pr-test-amd / call-pr-test-amd-extra / call-gate / pr-gate — failed step(s): N/A

Commit Info

• sglang: f83e4d59
• aiter: N/A — this is a no-op gate/orchestration job (pr-gate) with zero steps and no downloadable log (blob 404 BlobNotFound); no [CI-AITER-CHECK] markers exist. aiter is not exercised by this job.

Failed Tests

Test File	Test Function	Error	Log
N/A — not a test	N/A	Gate job cancelled after 1s (no steps executed)	job

This job ran no tests. It is the runtime label gate (pr-gate from pr-gate.yml) invoked by call-gate inside pr-test-amd-extra.yml. On a schedule event every step is gated by if: github.event_name == 'pull_request', so the job is a no-op that should pass — its rocm720 sibling did.

Failure Summary

The pr-gate job was cancelled 1 second after starting (20:52:03→20:52:04, no steps). This is not a test or code failure — it is a GitHub Actions concurrency-group collision: within this single scout run, both the default-CUDA extra (call-pr-test-amd / call-pr-test-amd-extra) and the rocm720 extra (call-pr-test-amd-rocm720 / call-pr-test-amd-extra-rocm720) invoke the same pr-test-amd-extra.yml, whose concurrency group omits rocm_version, so the two siblings share one group with cancel-in-progress enabled and one cancels the other. The cancellation then cascaded: extra-a-test-1-gpu-small-amd and extra-a-test-1-gpu-large-amd were cancelled, and the aggregator pr-test-amd-extra-finish reported failure (its logic exit 1s on any cancelled dependency).

Regression Status

Infrastructure / CI-orchestration issue — new this week (not a test regression). The AMD extra subtree did not run at all in the three prior scheduled scout runs; it appears for the first time in this run, and the gate was immediately cancelled.

Recent history of the AMD extra subtree in amd-aiter-scout.yml (event schedule, branch main, completed runs only):

Date	Run	Extra subtree present?	Gate status
Jun 18	27788558636	✅ yes (default + rocm720)	❌ default gate cancelled (collision); rocm720 gate ✅
Jun 11	27376585476	❌ no extra jobs spawned	—
Jun 08	27166336072	❌ no extra jobs spawned	—
Jun 04	26978457660	❌ no extra jobs spawned	—

The chaining of call-pr-test-amd-extra on schedule is recent: 5d194915 (Jun 17, "[AMD] ci: add extra-a 1-gpu-large tier", #28458) is the latest commit touching pr-test-amd-extra.yml and landed between the Jun 11 and Jun 18 scout runs.

Failure Origin

unclear (CI orchestration) — not aiter-caused, not a test failure. Standard Baseline A/B against the sister pr-test-amd.yml is N/A because no test ran; moreover the collision is scout-specific (a standalone pr-test-amd.yml scheduled run invokes only the default extra, never both variants in one run, so it would not collide). Zero aiter commits implicated.

Test File	Test Function	Origin	Baseline	Baseline Status
N/A (gate job)	N/A	unclear (CI orchestration)	rocm720 sibling pr-gate	✅ Succeeded (same scout run, rocm720 variant won the concurrency race)

Failure Cluster

Concurrency-group collision cancels AMD extra gate on scout

Facts (verified observations)

• Job 82231478836: conclusion=cancelled, 0 steps, started_at 2026-06-18T20:52:03Z, completed_at …20:52:04Z (1s). Log blob returns 404 BlobNotFound.
• Cascade timing in the same run: …/call-gate/pr-gate cancelled (20:52:04); extra-a-test-1-gpu-small-amd & extra-a-test-1-gpu-large-amd cancelled; pr-test-amd-extra-finish failure (20:52:07→20:52:10).
• Sister rocm720 gate call-pr-test-amd-rocm720 / call-pr-test-amd-extra-rocm720 / call-gate / pr-gate → success in the same run.
• pr-test-amd-extra.yml:83-85 — concurrency group: pr-test-amd-extra-${{ github.event_name }}-${{ github.head_ref || github.ref_name || 'default' }}-${{ inputs.ref || 'all' }} (does not include inputs.rocm_version); cancel-in-progress: ${{ github.event_name != 'workflow_call' }} → evaluates true on a schedule run.
• Both pr-test-amd.yml:212 and pr-test-amd-rocm720.yml:228 call the same ./.github/workflows/pr-test-amd-extra.yml; rocm720 passes rocm_version: rocm720 (:231), default passes none → for this scout run both resolve to the identical group pr-test-amd-extra-schedule-main-all.
• pr-test-amd-extra.yml:184-207 — pr-test-amd-extra-finish runs if: always() and exit 1s when any dependency is failure or cancelled, explaining its failure.

Hypothesised Causes

• [HIGH] Concurrency-group collision between the default-CUDA and rocm720 invocations of pr-test-amd-extra.yml within one scout run.
  • Supporting: concurrency group key omits rocm_version (:84); cancel-in-progress is true on schedule (:85); both callers target the same file; the two siblings started within the same 2s window and exactly one (default) was cancelled while the other (rocm720) succeeded; the failure surfaced the first time the extra subtree ran under the scout.
  • Disconfirming: (none found) — the rocm720 success vs default cancellation is exactly the asymmetry a shared cancel-in-progress group produces; no test/code evidence points elsewhere.
  • How to verify: re-dispatch amd-aiter-scout.yml and confirm one of the two pr-test-amd-extra gates is cancelled again; inspect run logs for a "Canceling since a higher priority waiting request for 'pr-test-amd-extra-schedule-main-…' exists" message.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs repo:sgl-project/sglang is:pr is:open for concurrency+rocm_version (0 results) and pr-test-amd-extra+concurrency (only unrelated kernel/quant PRs). No PR addresses the concurrency-group collision.

Suggested Triage Steps

• Add the ROCm variant to the concurrency key so the two scout invocations don't collide — e.g. include ${{ inputs.rocm_version || 'default' }} in pr-test-amd-extra.yml:84. Verify by re-running the scout and confirming both extra gates reach success.
• Alternatively, set cancel-in-progress: false for the workflow_call path (the expression already distinguishes events) so chained scout invocations queue instead of cancelling.
• Since the immediate symptom is only an orchestration artifact (no test signal lost yet), confirm whether the extra-a suites are meant to run under the scout at all; if yes, the concurrency fix is required before that signal is trustworthy.

Status

• Persistence: First occurrence — extra subtree ran under the scout for the first time on 2026-06-18; prior 3 scheduled scout runs spawned no extra jobs.
• Scope: One scout run; cancels the default-CUDA call-pr-test-amd-extra gate + its 2 extra-a test jobs and turns pr-test-amd-extra-finish red. rocm720 extra unaffected. No test coverage is actually executed for the default-CUDA extra-a suites in this run (signal lost, not a real test failure).
• In-flight fix: ❌ none found in open PRs — needs a new workflow-config PR.

call-pr-test-amd / jit-kernel-unit-test-amd (linux-mi325-1gpu-sglang) — failed step(s): Run JIT kernel unit tests

Commit Info

• sglang: prompt reports f83e4d59 (main HEAD), but this scout job actually checked out branch amd/aiter-ci (git checkout -B amd/aiter-ci refs/remotes/origin/amd/aiter-ci, log line 107) — current branch head a7fd0920. This divergence is central to the root cause below.
• aiter: a40c487b (source: override — AITER_COMMIT_OVERRIDE=a40c487b3c01dc03fd3872d65b1f7404f669471f, rebuilt over Dockerfile default 7d604afe5; [CI-AITER-CHECK] log lines 4030-4032).

Failed Tests

Test File	Test Function	Error	Log
test/registered/jit/test_activation.py	test_activation_correctness / test_activation_out_param (all params)	RuntimeError: ninja exited with status 1 — clang-HIP compile error in activation.cuh:213	Log
test/registered/jit/test_per_token_group_quant_8bit.py	test_per_token_group_quant_with_column_major (all params)	TIMEOUT after 1200 seconds	Log

Failure Summary

Two of seven JIT-kernel test files failed in jit-kernel-unit-test-amd. test_activation.py fails because the JIT compile of activation.cuh aborts with a clang-HIP error (ninja exited with status 1); test_per_token_group_quant_8bit.py times out after 1200 s running its column-major parametrizations. The other 6/8 test files passed (final tally: 6/8 passed, exit code 255).

Regression Status

Known recurring failure on the scout branch — both files are already fixed on main by PR #27947 (merged 2026-06-16), but the amd/aiter-ci scout branch has not picked it up.

Recent history of test_activation.py / test_per_token_group_quant_8bit.py in the sister job jit-kernel-unit-test-amd (workflow pr-test-amd.yml, branch main, event schedule, completed runs only):

Date	Run	Job	Test File Status	Failed Function	Error
Jun 18	27744699827	jit-kernel-unit-test-amd	✅ Passed	—	—

(On main, the job passes since PR #27947 merged 2026-06-16. The failure is specific to the stale amd/aiter-ci branch.)

Failure Origin

pre-existing (sglang) — both failures. NOT aiter-caused.

Test File	Test Function	Origin	Baseline Run	Baseline Status
test/registered/jit/test_activation.py	test_activation_correctness	pre-existing (sglang)	27744699827	✅ Passed (sister, main + #27947)
test/registered/jit/test_per_token_group_quant_8bit.py	test_per_token_group_quant_with_column_major	pre-existing (sglang)	27744699827	✅ Passed (sister, main + #27947)

⚠️ A/B nuance (do not misread as aiter-caused): The sister baseline passes, which the naive A/B rule would map to "aiter-caused." That mapping is invalid here because the scout branch amd/aiter-ci and main run different sglang source — the scout branch is missing PR #27947. The failing compile is of sglang's own JIT kernel (activation.cuh), not aiter code; the aiter override is irrelevant. The two-variable assumption (identical sglang source, only aiter differs) is broken, so Origin is pre-existing (sglang).

Failure Cluster

"JIT activation.cuh clang-HIP compile failure + per_token_group_quant ROCm timeout (scout branch missing #27947)".

Facts (verified observations)

• [FACT] Job checked out branch amd/aiter-ci, not main (log line 107: git checkout --progress --force -B amd/aiter-ci refs/remotes/origin/amd/aiter-ci).
• [FACT] Compile error, verbatim (log line 4670): activation.cuh:213:14: error: cannot initialize return object of type 'decltype(ActivationKernel<__half, false>::unary_kernel<ActivationKind::kReLU2>)' (aka 'const auto') with an lvalue of type 'void (*const)(const UnaryActivationParams)' → ninja exited with status 1 (line 4653).
• [FACT] amd/aiter-ci branch activation.cuh lines 209-218 use the broken -> decltype(...unary_kernel<...>) return type; line 213 is return ActivationKernel::template unary_kernel<ActivationKind::kReLU2>; — exactly the line the compiler rejects (verified via GitHub contents API on ref amd/aiter-ci).
• [FACT] Main HEAD f83e4d59 activation.cuh uses the fixed unary_kernel_fn_t return type with a comment explicitly documenting the clang-HIP gfx942 fix (verified via git show).
• [FACT] PR #27947 (merged 2026-06-16) modifies exactly activation.cuh (+7/-2), per_token_group_quant_8bit.cuh (+10/0), and test_per_token_group_quant_8bit.py (+1/-2) — addressing both failures.
• [FACT] test_per_token_group_quant_8bit.py → ✗ TIMEOUT ... after 1200 seconds (log line 14665); each column-major param ran ~1 s over hundreds of cases.
• [FACT] Sister baseline 27744699827 (main, 2026-06-18) jit-kernel-unit-test-amd = success.

Hypothesised Causes (with confidence)

• [HIGH] The scout branch amd/aiter-ci is missing PR #27947, so it still has the pre-fix activation.cuh (decltype return type that clang-HIP gfx942 rejects) and the pre-fix per_token_group_quant test (slow ROCm column-major path → 1200 s timeout).
  • Supporting: branch source at line 213 matches the exact compile-error line; main has the fix; the fix PR touches exactly these files; sister baseline on main passes.
  • Disconfirming: (none found) — the aiter override does not touch JIT-compiled sglang source; aiter is exonerated.
  • How to verify: rebase/merge main (or cherry-pick #27947 72d962be) into amd/aiter-ci and re-run the scout jit-kernel-unit-test-amd job.

In-flight Fix Check

• Fix already on main: ✅ #27947 "[AMD] Fix jit-kernel-unit-test-amd: activation.cuh ROCm build + per_token CUDA-only" (merged 2026-06-16, 72d962be). No new PR needed — the scout branch just needs to be synced. (API: search/issues?q=repo:sgl-project/sglang+is:pr+unary_kernel+activation+clang.)

Suggested Triage Steps

• Sync the scout branch: merge main into amd/aiter-ci (or cherry-pick #27947), then re-run jit-kernel-unit-test-amd on the scout — expected to go green.
• Confirm the scout branch maintenance process keeps amd/aiter-ci rebased on main; this failure class will recur for any main-side AMD fix until the branch is regularly synced.
• Do NOT investigate aiter a40c487b for these two tests — the compile/timeout reproduce from sglang source alone and are already fixed on main.

Status

• Persistence: Both files broken on the amd/aiter-ci scout branch as long as it lags main; main itself is green since 2026-06-16.
• Scope: jit-kernel-unit-test-amd (linux-mi325-1gpu-sglang) on the scout run; 2 of 7 JIT test files. Not reproducible on main / sister pr-test-amd.yml.
• In-flight fix: ✅ already merged to main — PR #27947; scout branch needs sync, no duplicate PR.

call-nightly-amd / nightly-8-gpu-qwen3-235b — failed step(s): N/A

Commit Info

• sglang: f83e4d59
• aiter: a40c487b (source: override — AITER_COMMIT_OVERRIDE forced a rebuild over the Dockerfile default 7d604afe; see [CI-AITER-CHECK] markers in log)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/test_qwen3_instruct.py	(file-level run; bf16 Qwen3-235B-A22B-Instruct-2507)	Job terminated mid-test — step 6 left in_progress, job conclusion=failure. No clean assertion/accuracy line emitted (abnormal termination).	Job

Note: test/registered/amd/test_qwen3_instruct_fp8.py passed in the same job — Accuracy 0.956 (log line ~8372, elapsed 898s).

Failure Summary

The 8-GPU Qwen3-235B nightly ran two test files. The FP8 model test (test_qwen3_instruct_fp8.py) passed cleanly (Accuracy 0.956). The job then started the bf16 model test (test_qwen3_instruct.py): the server came up at 00:54:15 and began prefill at 00:54:28, but the job ended at 01:03:54 with conclusion=failure while the test step ("Accuracy Test + Performance Test (8-GPU Qwen3)") was still in_progress. This in_progress step state on a failure job indicates abnormal termination (hang/crash/runner abort) during the bf16 accuracy run, not a clean assertion failure.

Regression Status

New failure (first observed this scout run; same test file passed in the previous scout and in all recent sister baselines).

Recent history of test_qwen3_instruct.py in job nightly-8-gpu-qwen3-235b (completed runs only):

Date	Run	Source	Test File Status	Notes
Jun 18	27788558636	scout (override aiter a40c487b)	❌ Failed	job killed mid-test
Jun 17	27710225107	sister nightly-test-amd (default aiter)	✅ Passed	"Test Summary: 2/2 passed", acc 0.961
Jun 16	27640378823	sister nightly-test-amd (default aiter)	✅ Passed	job success
Jun 15	27568949589	sister nightly-test-amd (default aiter)	✅ Passed	job success
Jun 11	27376585476	prior scout (override aiter)	✅ Passed	job success

Failure Origin

Test File	Test Function	Origin	Baseline Run	Baseline Status
test/registered/amd/test_qwen3_instruct.py	file-level (bf16 Qwen3-235B)	aiter-caused	27710225107 / 81969146006	✅ Passed (sister nightly-8-gpu-qwen3-235b, default aiter)

A/B check: the sister nightly-test-amd.yml job ran the same two test files with the Dockerfile-default aiter and reported "Test Summary: 2/2 passed". The scout differs from that baseline almost entirely in the aiter override (sglang heads are ~1 day apart: scout f83e4d59 Jun 18 vs baseline f5b04162 Jun 17). The prior scout (Jun 11) also passed this job. This isolates the aiter override as the determining variable → aiter-caused.

Failure Cluster

Server hang/abnormal termination during bf16 Qwen3-235B accuracy run (8-GPU, attention_backend=aiter).

Facts (verified observations)

• [CI-AITER-CHECK] (log line 4058-4060): Dockerfile default AITER_COMMIT=7d604afe…; AITER_COMMIT_OVERRIDE=a40c487b… → forced rebuild. Image base aiter was v0.1.14rc1.dev240+g7d604afe5.
• Test 1 test_qwen3_instruct_fp8.py passed: Accuracy 0.956, elapsed=898 (log line ~8372/8489).
• Test 2 test_qwen3_instruct.py (bf16, quantization=None, attention_backend='aiter', tp_size=8) started; server up 00:54:15, prefill 00:54:28; downloaded log ends there. Job completed_at=01:03:54Z, conclusion=failure, step 6 in_progress (verified via jobs API).
• Soft watchdog timeouts (soft=True, 300s) appear during the slow weight load in BOTH the scout and the passing baseline — these are normal for the 235B load and are NOT the failure signature.
• Sister baseline (default aiter): both files passed, "Test Summary: 2/2 passed", test 2 acc 0.961, elapsed=1366 (~23 min). The scout had only ~9 min between test-2 server start and job death — it was cut short well before the test could complete.
• The bf16 path logs continuous [aiter] … not found tuned config in bf16_tuned_gemm.csv, will use default config! using torch solution:0 — bf16 GEMMs fall back to the default/torch solution (no AMD-tuned bf16 config) on this path.

Hypothesised Causes (with confidence)

• [MEDIUM] A regression in the aiter delta 7d604afe..a40c487b (105 commits, Jun 9 → Jun 18) causes a hang/crash on the bf16 Qwen3-235B decode path.
  • Supporting: same test file + same sglang code passes with default aiter in the sister baseline and passed in the prior scout; the only materially-changed variable is the aiter override. The bf16 GEMM path is the dominant operation in the failing run.
  • Disconfirming: the exact terminal error is not visible (log truncated at the still-in_progress step), so "hang vs crash vs runner abort" is not pinned; cannot rule out an infra/runner-level abort independent of aiter.
  • How to verify: re-run the scout, or run test_qwen3_instruct.py (bf16) on MI30x with aiter pinned to default 7d604afe vs override a40c487b and compare.
• [LOW] Specific aiter bf16-GEMM / hang-related commits in the window are candidates, e.g. 1110647c1d ("[OPUS] gfx942 a16w16 bf16 GEMM pipeline family for DSV4", #3594), 064a8f73e7 ("fix: fix hang issue", #3664), 94f4f3da79 ("fix(opus): guard gfx942 bf16ws splitk reduce", #3684).
  • Supporting: these touch gfx942 bf16 GEMM, matching the bf16 fallback path exercised by the failing test.
  • Disconfirming: no stack trace ties the crash to any of these; the 105-commit window has many other attention/GEMM changes — single-commit attribution is speculation without a bisect.
  • How to verify: bisect the aiter window with the bf16 test as the oracle.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open sglang PRs for qwen3 235b and aiter qwen3; matches were unrelated (Qwen3.5/mxfp4 fusion features, e.g. #28244, #28399), none addressing a bf16 Qwen3-235B hang under an aiter bump. This is an aiter-side regression (override only); the fix likely belongs upstream in aiter, not sglang.

Suggested Triage Steps

• Re-run job nightly-8-gpu-qwen3-235b in scout to confirm the failure reproduces (rules out a one-off runner abort, given the in_progress step state).
• A/B on MI30x: run test/registered/amd/test_qwen3_instruct.py with aiter 7d604afe (default) vs a40c487b (override); if it hangs only on the override, confirms aiter-caused.
• If confirmed, bisect aiter 7d604afe..a40c487b using the bf16 test as oracle; start with the gfx942 bf16-GEMM / hang-fix commits listed above.
• Capture a py-spy/coredump on the next failing run to obtain the missing terminal stack trace (current log is truncated at the live step).

Status

• Persistence: New as of this scout run (2026-06-18); previous scout (2026-06-11) and sister baselines (Jun 15/16/17) all passed this job.
• Scope: 1 test file (test_qwen3_instruct.py, bf16) in 1 job (nightly-8-gpu-qwen3-235b); the FP8 sibling test passed.
• Blocked work: bf16 Qwen3-235B accuracy signal on AMD under the new aiter override.
• In-flight fix: ❌ none found.

call-nightly-amd / nightly-8-gpu-mi35x-deepseek-r1-mxfp4-ar-fusion — failed step(s): Accuracy Test MI35x (8-GPU DeepSeek-R1-MXFP4 AllReduce Fusion)

Commit Info

• sglang: f83e4d59
• aiter: a40c487b (source: AITER_COMMIT_OVERRIDE → forced rebuild; Dockerfile default was 7d604afe)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi35x/test_deepseek_r1_mxfp4_ar_fusion_eval_mi35x.py	test_deepseek_r1_mxfp4_ar_fusion_accuracy	AssertionError: Failed models: ['amd/DeepSeek-R1-MXFP4-Preview (ar-fusion)']	Log

Failure Summary

The single accuracy test for amd/DeepSeek-R1-MXFP4-Preview with AllReduce fusion failed. The server never reached ready state: all 8 TP scheduler workers (plus the TokenizerManager) hit soft watchdog timeouts, with py-spy dumps showing every rank blocked inside HuggingFace weight download (xet_get) for the entire ~55-minute window. The test hit its 3600 s time budget (elapsed 3633 s) while weights were still downloading, so the model was marked failed. This is a server-startup/weight-download hang, not a numerical accuracy regression or an aiter compute fault.

Regression Status

Infrastructure issue (likely transient) — not a code regression.

The failure is a HuggingFace Xet download hang during model load; it is not tied to a specific sglang or aiter commit. AMD AITER Scout runs are Mon/Thu only and the prior completed scout runs (27376585476 and earlier) also show failure at the run level, but at the test-file level I did not download each historical log to confirm whether this same DeepSeek-R1-MXFP4 test was the failing one — the current failure mode (download stall) is environmental and would not necessarily recur. No green baseline for this specific test file was established in the queryable window.

Date	Run	Status	Note
Jun 19	27788558636	❌ Failed	This run — HF Xet download hang, server never ready
(run-level only)	27376585476	❌ Failed	run-level failure; per-test status not verified

Failure Origin

unclear (infrastructure) — one line per failing test.

Test File	Test Function	Origin	Baseline Run	Baseline Status
..._deepseek_r1_mxfp4_ar_fusion_eval_mi35x.py	test_deepseek_r1_mxfp4_ar_fusion_accuracy	unclear (infra)	27710225107	⚠️ cancelled (no usable baseline)

The sister workflow nightly-test-amd.yml had no usable baseline near this head SHA: the latest schedule run (27780272530) is still queued and the next-newest (27710225107) was cancelled. A/B check is moot by mechanism: the hang occurs in download_weights_from_hf → snapshot_download → xet_get, before any model weights load or any aiter kernel executes, so the aiter override cannot be the cause.

Failure Cluster

HuggingFace Xet weight-download hang → scheduler watchdog timeout → server never ready

Facts (verified observations)

• Server launch began at 06:54:32 (server_args=ServerArgs(...), log line ~4444). Test command: sglang serve --model-path amd/DeepSeek-R1-MXFP4-Preview --attention-backend aiter ... --enable-aiter-allreduce-fusion --tp 8.
• Repeated py-spy dumps every 5 min from 06:59 through 07:49 show all ranks stuck in the same stack: do_load_weights (deepseek_weight_loader.py:158) → download_weights_from_hf (weight_utils.py:595) → ci_download_with_validation_and_retry (ci_weight_validation.py:1955) → snapshot_download → hf_hub_download → _download_to_tmp_and_move (file_download.py:1861) → xet_get (file_download.py:566) (active thread, log lines 5158, 5631, 6104, … 9890).
• TokenizerManager watchdog timeout at 07:49:40; TP0–TP7 soft watchdog timeouts at 07:49:45–47 (all self.soft=True, so they logged but did not kill).
• Test failed at elapsed 3633 s (est_time 3600 s) with AssertionError: Failed models: ['amd/DeepSeek-R1-MXFP4-Preview (ar-fusion)']; runner exit code 255, "not retrying" (treated as test failure, not network).
• Additional environmental signal earlier: Local docker registry pull failed; falling back to public registry (line 1253) — same run had registry connectivity trouble.
• [CI-AITER-CHECK] confirms aiter was rebuilt from override a40c487b, but no aiter kernel ran before the hang.

Hypothesised Causes (with confidence)

• [HIGH] HuggingFace Hub Xet download stalled, blocking model weight load so the server never became ready before the 3600 s test budget elapsed.
  • Supporting: every 5-min py-spy dump over ~55 min shows the identical active stack ending in xet_get; the test timed out at exactly its est_time; soft watchdogs fired on all ranks; same run also had a docker registry pull failure (corroborating network trouble).
  • Disconfirming: (none found) — no numerical mismatch, no kernel crash, no GPU fault appears anywhere in the log.
  • How to verify: re-run the job; check whether the eval reaches accuracy=… threshold=… output (absent here). If it recurs, inspect runner network egress to HF / Xet CAS endpoints or set HF_HUB_DISABLE_XET=1 to force the classic download path.
• [LOW] aiter override a40c487b involvement.
  • Supporting: this is an AITER Scout run with a forced aiter rebuild.
  • Disconfirming: the hang is in HF download before weights load; no aiter code is on the stuck stack. Mechanism exonerates aiter.
  • How to verify: not actionable — failure precedes any aiter execution.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for xet download (0 results) and weight download timeout (11 results, all unrelated: benchmark/refactor/AMD-kernel PRs). No PR addresses the Xet download hang.

Suggested Triage Steps

• Re-run the job — a download stall is typically transient; a clean re-run will confirm whether this is environmental.
• If it recurs, check runner→HuggingFace/Xet CAS network egress and consider setting HF_HUB_DISABLE_XET=1 (or pre-staging amd/DeepSeek-R1-MXFP4-Preview weights in the CI image/local cache) to bypass the Xet path.
• Consider whether ci_download_with_validation_and_retry should enforce a hard per-file download timeout so a stalled xet_get fails fast and triggers a retry, rather than consuming the whole 3600 s test budget.
• Do NOT pursue aiter or sglang commit bisection for this failure — the hang occurs before model load completes.

Status

• Persistence: Single observed occurrence in this scout run; failure mode is a transient HF Xet download hang.
• Scope: 1 job (call-nightly-amd / nightly-8-gpu-mi35x-deepseek-r1-mxfp4-ar-fusion); the parent scout run 27788558636 is still queued (in-flight) so other jobs' status is not final.
• Blocked work: DeepSeek-R1-MXFP4 AR-fusion accuracy signal on MI35x for this run only.
• In-flight fix: ❌ none found — environmental; re-run first.

call-nightly-amd / nightly-8-gpu-mi35x-deepseek-r1-hicache — failed step(s): Accuracy Test MI35x (8-GPU DeepSeek-R1 HiCache)

Commit Info

• sglang: f83e4d59
• aiter: a40c487b (source: AITER_COMMIT_OVERRIDE → forced rebuild; Dockerfile default was 7d604afe, image had dev build v0.1.14rc1.dev240+g7d604afe5)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/test_deepseek_r1_hicache_mi35x.py	TestDeepSeekR1HiCacheMI35x.setUpClass	Exception: Server process exited with code -9 (preceded by warmup requests.exceptions.ReadTimeout after 600 s)	Log

Failure Summary

The single test file this job runs, test_deepseek_r1_hicache_mi35x.py, failed in setUpClass: the DeepSeek-R1-0528 TP8 + HiCache server never became healthy (/health_generate returned 503 continuously from 09:13 to 09:15), the warmup POST to 127.0.0.1:11000 hit the 600 s read timeout, and the server process was then killed with exit code -9 (SIGKILL). Ran 0 tests in 2837.656s — FAILED (errors=1).

Regression Status

Never-passed (no green run found in queryable window).

This job runs exactly one test file, so job status == test-file status. Recent history of test_deepseek_r1_hicache_mi35x.py in sister job nightly-8-gpu-mi35x-deepseek-r1-hicache (workflow nightly-test-amd.yml, event=schedule, branch=main, completed runs only):

Date	Run	Job	Test File Status	Error
Jun 17	27710225107	hicache-mi35x	❌ Failed	TIMEOUT after 3600 s
Jun 15	27568949589	hicache-mi35x	❌ Failed	—
Jun 14	27507337214	hicache-mi35x	❌ Failed	—
Jun 13	27474523511	hicache-mi35x	❌ Failed	TIMEOUT after 3600 s
Jun 12	27434333913	hicache-mi35x	❌ Failed	—
Jun 11	27368902882	hicache-mi35x	❌ Failed	—
Jun 10	27296882210	hicache-mi35x	❌ Failed	—
Jun 09	27226450921	hicache-mi35x	❌ Failed	—

No passing run found in the lookback window (Jun 09 – Jun 17, test-level verified; the workflow also reports failure on every prior scheduled run back to mid-May, though not test-level confirmed). First observed failure in window: ≥ Jun 09. Last known passing date: none found.

Failure Origin

pre-existing (sglang) — the aiter override is NOT implicated.

Test File	Test Function	Origin	Baseline Run	Baseline Status
test/registered/amd/test_deepseek_r1_hicache_mi35x.py	TestDeepSeekR1HiCacheMI35x.setUpClass	pre-existing (sglang)	27710225107	❌ Same test fails in sister job (default aiter) — TIMEOUT 3600 s

A/B check: The sister job in nightly-test-amd.yml (Dockerfile-default aiter, no override) fails the exact same test file on its latest completed scheduled run (Jun 17) and on every completed run back to Jun 09. The failure reproduces without the aiter override, so it is pre-existing in sglang and MUST NOT be attributed to aiter. Per the rules, zero aiter commits are listed below.

Failure Cluster

DeepSeek-R1 HiCache server warmup hang/timeout on MI35x (gfx950) during startup.

Facts (verified observations, no interpretation)

• Scout: /health_generate returned 503 Service Unavailable continuously 09:13:26 → 09:15:26; warmup POST raised requests.exceptions.ReadTimeout (read timeout=600) from http_server.py:2093 _execute_server_warmup; kill_process_tree then fired and the relaunched server exited with code -9; setUpClass raised Exception: Server process exited with code -9 (test_utils.py:1027 popen_launch_server).
• Sister baseline (Jun 17, default aiter): same test file; CUDA graph capture completed (52/52), server briefly reached health_generate 200 OK at 14:35:43, but the harness killed the run with TIMEOUT after 3600 seconds (est_time 5400 s). Same test, slightly farther progress.
• Server command (both): deepseek-ai/DeepSeek-R1-0528 --tp 8 --mem-fraction-static 0.6 --kv-cache-dtype fp8_e4m3 --attention-backend aiter --page-size 64 --context-length 65536 --chunked-prefill-size 32768 --enable-hierarchical-cache --hicache-ratio 2 --hicache-io-backend kernel --hicache-storage-backend file --hicache-storage-prefetch-policy best_effort --model-loader-extra-config '{"enable_multithread_load": true, "num_threads": 64}'.
• Reproducibility: every completed run of this test, scout and sister, Jun 09 → Jun 19.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [LOW] Heavy HiCache startup config exceeds the warmup/graph-capture time budget on MI35x, and the server is OOM/watchdog-killed (-9) before becoming healthy.
  • Supporting: server stays 503 for the entire warmup window then dies with SIGKILL; config is very heavy (TP8 R1-0528, ctx 65536, file HiCache backend + best_effort prefetch + 64-thread loader); sister baseline needed >3600 s to do the same startup+test.
  • Disconfirming: in the sister run graph capture did finish and the server did return 200 OK once, so a pure "warmup never finishes" story is incomplete — the bottleneck may be downstream (generation/HiCache file IO) rather than warmup itself; the scout's earlier -9 could be a separate resource ceiling on the scout runner.
  • How to verify: launch the exact command on an MI35x (gfx950) container, time graph-capture + warmup, watch host RAM/dmesg for the OOM-killer; try --watchdog-timeout already at 1200 vs the 600 s warmup read timeout.
• [SPECULATION] HiCache file-backend prefetch path is slow/blocking on gfx950, inflating startup beyond limits.
  • Supporting: the differentiator vs other (passing) DeepSeek jobs is the HiCache file storage backend + prefetch.
  • Disconfirming: no code-level evidence captured; not isolated from the general slowness.
  • How to verify: re-run with --hicache-storage-backend disabled / --hicache-storage-prefetch-policy off and compare startup time.

No hypothesis above LOW confidence. Because the test is Never-passed in the queryable window, there is no pass_sha..fail_sha to bisect; root-causing requires determining whether this MI35x HiCache test ever passed and, if not, treating it as a new-port stabilization task.

In-flight Fix Check

• In-flight fix: ❌ none found matching this specific MI35x DeepSeek-R1 HiCache warmup/startup timeout. Searches run: hicache+deepseek, mi35x, gfx950+hicache, deepseek-r1+warmup+amd, hicache+warmup+timeout against open PRs. Nearby-but-unconfirmed candidates (AMD HiCache, not clearly this failure): #28534 "[AMD] Enable JIT staged HiCache write-back and fix CPU-index crash", #28455 "[AMD] Fix DeepSeek-V4 fp8 KV path on gfx942".

Suggested Triage Steps

• Reproduce the server command above on an MI35x (gfx950) container; time CUDA-graph-capture and warmup separately and watch dmesg/host RAM for the OOM-killer (explains exit -9).
• A/B the HiCache file storage path: re-run with --hicache-storage-prefetch-policy off (and/or no --hicache-storage-backend file) to isolate whether prefetch/file IO drives the startup slowdown.
• Determine whether this test has ever passed on MI35x (search beyond the 30-day window). If it never has, treat as a new-port stabilization issue — consider quarantining/marking it on AMD until startup is brought under the timeout, rather than blocking nightly signal.
• This is pre-existing (sglang); do not pursue aiter commits — the sister (default-aiter) baseline reproduces the identical failure.

Status

• Persistence: Failing on every completed run, test-level confirmed Jun 09 → Jun 19 (≥10 days); workflow-level failure visible further back.
• Scope: Both the AITER Scout sub-job and the regular nightly-test-amd sister job; one test file (test_deepseek_r1_hicache_mi35x.py).
• Blocked work: DeepSeek-R1 HiCache nightly signal on MI35x (gfx950).
• In-flight fix: ❌ none found in open PRs — needs new PR or investigation.

---

⏳ 32 job(s) still running (run) — will update when complete

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-06-19 09:30 UTC)

[amd-bot]

pr-test-amd.yml — 40 failure(s)

Run started (UTC): 2026-06-18 00:59 UTC
Last scanned (UTC): 2026-06-20 00:03 UTC
Commits: sglang 9bb9d17

Summary

Counts: 40 failures · 13 clusters · 9 persistent (≥3 days) · ~6 new today

#	Cluster	Job	Test File	Test Function	Status	Hypothesis (confidence)
1	flux2 FP8 _scaled_mm HIPBLAS unsupported	mm-gen-2gpu (1)	python/sglang/multimodal_gen/test/server/test_server_2_gpu.py	test_diffusion_generation[flux2_modelopt_fp8_tp2_t2i]	Never-passed (added Jun 18 #28546)	[MEDIUM] FP8 per-tensor GEMM unsupported on gfx942 ROCm
2	flux2 FP8 _scaled_mm HIPBLAS unsupported	mm-gen-2gpu (1)	test_server_2_gpu.py	[flux2_modelopt_fp8_tp2_t2i]	Never-passed	[MEDIUM] same
3	flux2 FP8 _scaled_mm HIPBLAS unsupported	mm-gen-2gpu (1)	test_server_2_gpu.py	[flux2_modelopt_fp8_tp2_t2i]	Recurring	[MEDIUM] same
4	flux2 FP8 _scaled_mm HIPBLAS unsupported	mm-gen-2gpu (1)	test_server_2_gpu.py	[flux2_modelopt_fp8_tp2_t2i]	New regression (Jun 18 07:46)	[MEDIUM] same
5	flux2 FP8 _scaled_mm HIPBLAS unsupported	mm-gen-1gpu (2)	test_server_1_gpu.py	[flux2_*] + ideogram	Never-passed	[MEDIUM] same
6	flux2 FP8 _scaled_mm HIPBLAS unsupported	mm-gen-2gpu (1)	test_server_2_gpu.py	[flux2_*] + ideogram	Never-passed	[MEDIUM] same
7	MTP/EAGLE spec accept-length < 2.04 (R1-MXFP4 MI35x)	stage-c-8gpu-mi35x (0)	test/registered/amd/test_deepseek_r1_mxfp4_8gpu.py	TestDeepseekR1MXFP4MTP.test_a_gsm8k	Never-passed (since Jun 16)	[LOW] accept length stuck ~1.44
8	MTP/EAGLE spec accept-length < 2.04	stage-c-8gpu-mi35x (0)	same	test_a_gsm8k	New regression (Jun 16 ~01:05)	[LOW] same
9	MTP/EAGLE spec accept-length < 2.04	stage-c-8gpu-mi35x (0)	same	test_a_gsm8k	Never-passed in window	[LOW] same
10	MTP/EAGLE spec accept-length < 2.04	stage-c-8gpu-mi35x (0)	same	test_a_gsm8k	Never-passed in window	[LOW] same
11	MTP/EAGLE spec accept-length < 2.04	stage-c-8gpu-mi35x (0)	same	test_a_gsm8k	Never-passed in window	[LOW] same
12	Runner loss / orchestration cancellation	extra-a-1gpu-large	N/A	N/A	Recurring (fast-fail cascade)	[FACT] job cancelled, no test ran
13	Runner loss / orchestration cancellation	extra-a-1gpu-large	N/A	N/A	Recurring	[MEDIUM] extra-workflow concurrency-group eviction
14	Runner loss / orchestration cancellation	extra-a-1gpu-small	N/A	N/A	Recurring	[MEDIUM] same concurrency eviction
15	Runner loss / orchestration cancellation	mm-gen-1gpu (1)	N/A	N/A	Recurring (runner loss)	[FACT] self-hosted runner terminated
16	Runner loss / orchestration cancellation	mm-gen-1gpu (1)	N/A	N/A	Infra (lost runner)	[FACT] runner lost, no log finalized
17	aiter fused AR+RMSNorm residual mismatch	stage-c-8gpu (3)	test/registered/ops/test_aiter_allreduce_fusion_amd.py	test_fused_ar_rms_residual_accuracy	Never-passed	[MEDIUM] CI aiter predates ROCm/aiter#2586
18	aiter fused AR+RMSNorm residual mismatch	stage-c-8gpu (3)	same	test_fused_ar_rms_residual_accuracy	Recurring/never-passed	[MEDIUM] same
19	aiter fused AR+RMSNorm residual mismatch	stage-c-8gpu (3)	same	test_fused_ar_rms_residual_accuracy	≥3 days persistent	[MEDIUM] same
20	aiter fused AR+RMSNorm residual mismatch	stage-c-8gpu (3)	same	test_fused_ar_rms_residual_accuracy	Persistent (stale tolerance)	[MEDIUM] same
21	Diffusion server warmup hang/timeout	mm-gen-1gpu (2)	test_server_1_gpu.py	test_diffusion_generation[qwen_image_t2i] → port 5555 cascade	New regression (Jun 18)	[LOW] NCCL dist-init hang, orphaned port
22	Diffusion server warmup hang/timeout	mm-gen-2gpu (0)	test_server_2_gpu.py	NCCL init segfault → port 5555 cascade	New regression	[LOW] 2-GPU NCCL init segfault
23	Diffusion server warmup hang/timeout	mm-gen-2gpu (1)	test_disagg_server.py	ZImage disagg _warmup 120s timeout	New regression	[LOW] disagg warmup timeout
24	Diffusion server warmup hang/timeout	mm-gen-2gpu (2)	test_disagg_server.py	TestDisaggZImage1Rank::test_generates_image	New regression	[LOW] disagg warmup timeout
25	GPU/HSA dist-init hang during warmup	stage-b-1gpu-small (5)	N/A (server warmup)	—	Flaky/infra hang	[LOW] dist-init hang, no crash
26	GPU/HSA dist-init hang during warmup	stage-b-1gpu-small (5)	N/A (server warmup)	—	Recurring (partition 5)	[LOW] stuck in libhsa-runtime first HIP op
27	GPU/HSA dist-init hang during warmup	stage-b-1gpu-small (9)	test/registered/scheduler/test_retract_decode.py	world-group init torch.ones hang	New regression (job-level)	[LOW] HSA runtime hang
28	External model/data fetch failure	mm-gen-1gpu (2)	test_server_1_gpu.py	HF Hub 429 + Aliyun OSS download	Infra	[FACT] external rate-limit/network
29	External model/data fetch failure	mm-gen-2gpu (1)	test_server_2_gpu.py	ideogram-ai/ideogram-4-fp8 gated 403	Never-passed in window	[FACT] gated-repo 403
30	External model/data fetch failure	mm-gen-1gpu (2)	test_server_1_gpu.py	Aliyun OSS input-image download fail	New today	[FACT] external network
31	Runner VRAM not clear / zombie KFD ctx	jit-kernel-unit-test	N/A	N/A (pre-test abort)	New today (infra)	[FACT] leaked KFD context
32	Runner VRAM not clear / zombie KFD ctx	stage-b-1gpu-large (2)	N/A	N/A	Infra	[FACT] same
33	Runner VRAM not clear / zombie KFD ctx	stage-b-1gpu-small (3)	N/A	N/A	Infra	[FACT] same
34	mori build failure (corrupt absl, invalid ELF)	stage-c-8gpu (3)	N/A (build)	N/A	Infra	[FACT] libabsl_strings_internal.so invalid ELF
35	mori build failure (corrupt absl, invalid ELF)	stage-c-8gpu (2)	N/A (build)	N/A	Infra	[FACT] same
36	HF cache lock / weight-load hang	stage-c-8gpu (0)	N/A (DeepSeek-V3 startup)	server timeout	New (infra)	[LOW] shared HF cache lock
37	HF cache lock / weight-load hang	stage-b-1gpu-small-mi35x	N/A (tokenizer download)	—	Flaky/infra	[LOW] HF cache file-lock deadlock
38	RCCL 8-GPU sanity SIGSEGV	stage-c-8gpu (2)	scripts/ci/amd/test_rccl_multi_gpu.py	test_rccl_allreduce (SIGSEGV)	New (infra, node-specific)	[LOW] rank-7 init segfault
39	MoE topk-softmax exact-value mismatch	sgl-kernel-unit-test	sgl-kernel unit (topk-softmax)	top-k boundary mismatch	Flaky (new today)	[LOW] unseeded random at top-k boundary
40	Analysis incomplete	stage-b-1gpu-small (13)	N/A	N/A	Per-job agent produced no output	[n/a] manual triage required

Cluster: flux2 FP8 _scaled_mm HIPBLAS unsupported (rows 1–6)

• Affected: 6 diffusion-gen jobs, flux2_modelopt_fp8_tp2_t2i parametrization on test_server_{1,2}_gpu.py.
• Shared facts: HIPBLAS_STATUS_NOT_SUPPORTED from ModelOpt FP8 torch._scaled_mm during DiT warmup on gfx942. Test added Jun 18 by #28546 (3b61dc32); has failed every appearance.
• Top hypothesis: [MEDIUM] per-tensor FP8 scaled GEMM path unsupported on ROCm/gfx942. Disconfirming: no A/B on a CUDA sister run captured here; could be a missing rocBLAS solution rather than code.
• In-flight fix: ❌ none (#27702 is adjacent ROCm-diffusion work, not this path).
• Suggested triage: confirm whether AMD should skip/guard the FP8 _scaled_mm flux2 case until a ROCm fallback exists; verify test_server_2_gpu.py flux2 case was meant to run on AMD CI.

Cluster: MTP/EAGLE spec accept-length < 2.04 (rows 7–11)

• Affected: 5× stage-c-test-large-8-gpu-amd-mi35x (0), test_a_gsm8k on test_deepseek_r1_mxfp4_8gpu.py.
• Shared facts: accuracy OK; avg_spec_accept_length ≈ 1.44–1.46 < 2.04 floor, stable across runs (systematic, not flaky). Test file last modified Jun 7 (10d33bd7).
• Top hypothesis: [LOW] MXFP4 MTP draft-acceptance regression or too-tight threshold on MI35x. Disconfirming: no green run in window to bound a pass→fail commit; could be an MI35x-specific config never tuned.
• In-flight fix: ❌ none (#25715, #26216 touch EAGLE/MXFP4 but not this gate).
• Suggested triage: bisect EAGLE/MXFP4 commits on MI35x; if threshold is the issue, quarantine until tuned.

Cluster: aiter fused AR+RMSNorm residual mismatch (rows 17–20)

• Affected: 4× stage-c-test-large-8-gpu-amd (3), test_fused_ar_rms_residual_accuracy.
• Shared facts: every shape max_diff=6.25e-2 (2⁻⁴), AR_exact=yes (only fused residual diverges). ≥3 days persistent, never-passed on current CI image.
• Top hypothesis: [MEDIUM] CI aiter image predates Fix: Numerical Accuracy in allreduce_fusion_kernel_1stage ROCm/aiter#2586 fix (43b7379b8); test tolerance too strict for current image. Disconfirming: none found.
• In-flight fix: ✅ #28226 (open since 2026-06-15) — relaxes tolerance to 1 bf16 ULP, edits this exact file. Chase reviewers; do NOT duplicate.
• Suggested triage: verify #28226 merges cleanly and re-run shard 3; else quarantine on AMD until aiter image updated.

Cluster: Runner loss / orchestration cancellation (rows 12–16)

• Affected: 2 extra-a jobs (concurrency), 2 diffusion-1gpu (runner loss), 1 fast-fail cascade. No test executed.
• Shared facts: jobs cancelled/terminated before/while tests ran; not code regressions.
• Top hypothesis: [MEDIUM] for extra-a — concurrency-group eviction (pr-test-amd-extra.yml:83-85 group omits run_id, cancel-in-progress true for schedule). Disconfirming: pure runner starvation also fits the lowest-priority extra tier.
• In-flight fix: ❌ none.
• Suggested triage: inspect the cancelling actor in the Actions timeline for 82209375330; test run_id-scoped concurrency group on a branch.

Cluster: Diffusion server warmup hang/timeout (rows 21–24)

• Affected: 4 diffusion-gen jobs — qwen_image NCCL hang + port-5555 cascade, 2-GPU NCCL segfault, ZImage disagg 120s warmup timeout (×2).
• Shared facts: server never reaches ready; --strict-ports cascade poisons remaining params in the shard. All new regressions Jun 18–19.
• Top hypothesis: [LOW] diffusion server distributed-init instability on AMD; symptom varies (hang/segfault/timeout). Disconfirming: differing test files/models suggest possibly multiple root causes lumped by symptom.
• In-flight fix: ❌ none.
• Suggested triage: re-run to confirm flaky vs persistent; separately, harden --strict-ports so one hung server doesn't cascade the shard.

Other clusters (smaller)

• GPU/HSA dist-init hang during warmup (rows 25–27): stuck in libhsa-runtime64.so first HIP op / torch.ones world-group init. In-flight fix ❌. Triage: re-run to confirm flaky; check for node-specific HSA driver issue.
• External model/data fetch (rows 28–30): HF 429, gated-repo 403 (ideogram-4-fp8), Aliyun OSS. CI-infra remediation (caching/token/mirror), not code. Candidate #28225 is NF4-scoped — verify fp8 coverage.
• Runner VRAM not clear / zombie KFD ctx (rows 31–33): pre-test infra abort across 3 jobs in run 27813231755. Triage: runner cleanup/stale-context reaping.
• mori build failure (rows 34–35): libabsl_strings_internal.so invalid ELF header during gtest discovery. Infra/image. Branch origin/bingxche/mori-ci-reinstall-26922 exists but no open PR.
• HF cache lock / weight-load hang (rows 36–37): shared /sgl-data/hf-cache lock stalls weight/tokenizer download. Triage: stale-lock cleanup or per-job cache isolation.
• RCCL 8-GPU sanity SIGSEGV (row 38): rank-7 init segfault before allreduce on test_rccl_multi_gpu.py. Likely node-specific; re-run to confirm.
• MoE topk-softmax mismatch (row 39): exact-value mismatch at top-k boundary with unseeded random input — likely a flaky/seed issue in the sgl-kernel unit test.
• Analysis incomplete (row 40): per-job agent timed out; manual triage of 82265091404.

Suggested triage order

Rank	Triage step	Targets	Why this first
1	Verify #28226 merges + re-run stage-c-8gpu shard 3	Cluster aiter AR+RMSNorm (rows 17–20)	Only cluster with a ready in-flight fix; ≥3 days persistent
2	Decide AMD skip/guard for flux2_modelopt_fp8_tp2_t2i or fix FP8 _scaled_mm fallback	flux2 cluster (rows 1–6)	Largest cluster (6 jobs), never-passed since Jun 18
3	Bisect EAGLE/MXFP4 MI35x commits; check 2.04 threshold validity	spec accept-length (rows 7–11)	5 jobs, systematic & persistent
4	run_id-scope extra-workflow concurrency group + audit runner pool	runner loss/cancellation (rows 12–16)	5 jobs lost with zero signal
5	Re-run diffusion warmup jobs + harden --strict-ports cascade	diffusion warmup (rows 21–24)	4 new regressions, signal-poisoning cascade

Infra clusters (rows 25–38) are runner/image/network issues — batch with a runner-cleanup + image-rebuild pass rather than per-test PRs.

---

Job	Failed Steps	Started
multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 2)	Run diffusion server tests (1-GPU)	2026-06-18T09:57
multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 1)	Run diffusion server tests (2-GPU)	2026-06-18T08:39
call-pr-test-amd-extra / extra-a-test-1-gpu-large-amd (linux-mi325-1gpu-sglang)	Run test	2026-06-18T12:53
stage-b-test-1-gpu-small-amd (linux-mi325-1gpu-sglang, 5)	Run test	2026-06-18T10:34
stage-c-test-large-8-gpu-amd-mi35x (linux-mi35x-gpu-8, 0)	Run test	2026-06-18T14:35
stage-c-test-large-8-gpu-amd (linux-mi325-8gpu-sglang, 3)	Run test	2026-06-18T08:35
call-pr-test-amd-extra / extra-a-test-1-gpu-large-amd (linux-mi325-1gpu-sglang)	N/A	2026-06-18T18:52
call-pr-test-amd-extra / extra-a-test-1-gpu-small-amd (linux-mi325-1gpu-sglang)	N/A	2026-06-18T18:52
multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 1)	Run diffusion server tests (2-GPU)	2026-06-18T19:51
multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 2)	Run diffusion server tests (1-GPU)	2026-06-18T20:54
stage-c-test-large-8-gpu-amd (linux-mi325-8gpu-sglang, 2)	Test RCCL multi-GPU communication	2026-06-18T19:59
stage-c-test-large-8-gpu-amd (linux-mi325-8gpu-sglang, 3)	Install dependencies	2026-06-18T22:05
multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 1)	Post Checkout code, Run diffusion server tests (1-GPU), Upload diffusion failure artifacts	2026-06-18T15:19
stage-c-test-large-8-gpu-amd (linux-mi325-8gpu-sglang, 0)	Run test	2026-06-18T21:25
multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 1)	Run diffusion server tests (2-GPU)	2026-06-18T13:16
stage-b-test-1-gpu-small-amd (linux-mi325-1gpu-sglang, 5)	Run test	2026-06-18T15:56
multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 1)	Run diffusion server tests (2-GPU)	2026-06-18T01:00
stage-c-test-large-8-gpu-amd (linux-mi325-8gpu-sglang, 3)	Run test	2026-06-18T13:16
stage-b-test-1-gpu-small-amd-mi35x (linux-mi35x-gpu-1)	Run test	2026-06-18T00:59
stage-c-test-large-8-gpu-amd (linux-mi325-8gpu-sglang, 3)	Run test	2026-06-18T01:03
stage-c-test-large-8-gpu-amd (linux-mi325-8gpu-sglang, 2)	Install dependencies	2026-06-18T01:22
stage-c-test-large-8-gpu-amd-mi35x (linux-mi35x-gpu-8, 0)	Run test	2026-06-18T18:08
multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 1)	Run diffusion server tests (2-GPU)	2026-06-19T01:46
multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 0)	Run diffusion server tests (2-GPU)	2026-06-19T01:49
stage-c-test-large-8-gpu-amd (linux-mi325-8gpu-sglang, 3)	Run test	2026-06-19T03:19
stage-b-test-1-gpu-small-amd (linux-mi325-1gpu-sglang, 13)	Run test — ⚠️ analysis failed	2026-06-19T02:23
stage-c-test-large-8-gpu-amd-mi35x (linux-mi35x-gpu-8, 0)	Run test	2026-06-19T04:34
stage-c-test-large-8-gpu-amd-mi35x (linux-mi35x-gpu-8, 0)	Run test	2026-06-19T05:47
multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 2)	Run diffusion server tests (2-GPU)	2026-06-19T07:58
multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 1)	Run diffusion server tests (2-GPU)	2026-06-19T07:57
jit-kernel-unit-test-amd (linux-mi325-1gpu-sglang)	Ensure VRAM is clear	2026-06-19T09:19
stage-b-test-1-gpu-large-amd (linux-mi325-1gpu-sglang, 2)	Ensure VRAM is clear	2026-06-19T09:22
stage-b-test-1-gpu-small-amd (linux-mi325-1gpu-sglang, 3)	Ensure VRAM is clear	2026-06-19T09:35
sgl-kernel-unit-test-amd (linux-mi325-1gpu-sglang)	Run test	2026-06-19T13:26
stage-b-test-1-gpu-small-amd (linux-mi325-1gpu-sglang, 9)	Run test	2026-06-19T13:42
multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 1)	Run diffusion server tests (2-GPU)	2026-06-19T13:37
multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 2)	Run diffusion server tests (1-GPU)	2026-06-19T18:48
multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 1)	N/A	2026-06-19T18:45
multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 1)	Run diffusion server tests (2-GPU)	2026-06-19T19:21
stage-c-test-large-8-gpu-amd-mi35x (linux-mi35x-gpu-8, 0)	Run test	2026-06-19T23:13

Per-Job Analysis

multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 2) — failed step(s): Run diffusion server tests (1-GPU)

Commit Info

• sglang: 59001267
• aiter: 7d604afe5 (source: dev image preserved — [CI-AITER-CHECK] Dev/patched version detected: v0.1.14rc1.dev240+g7d604afe5 → skipping rebuild)

Failed Tests

All 10 parametrizations of test_diffusion_generation errored at setup (server never came up). The job is multimodal-gen-test-1-gpu-amd (…, 2), step 9 "Run diffusion server tests (1-GPU)".

Test File	Test Function (param)	Error	Log
python/sglang/multimodal_gen/test/server/test_server_1_gpu.py	TestDiffusionServerOneGpu::test_diffusion_generation[qwen_image_t2i]	TimeoutError: Server not ready within 1200.0s (root)	Log
same	[qwen_image_t2i_cache_dit_scm_config_diffusers_1gpu]	RuntimeError: Scheduler port 5555 is unavailable … --strict-ports (cascade)	Log
same	[flux_image_t2i]	port 5555 unavailable (cascade)	Log
same	[layerwise_offload]	port 5555 unavailable (cascade)	same
same	[zimage_image_t2i_fp8]	port 5555 unavailable (cascade)	same
same	[qwen_image_edit_2509_ti2i]	port 5555 unavailable (cascade)	same
same	[wan2_1_t2v_1.3b]	port 5555 unavailable (cascade)	same
same	[cosmos3_nano_t2v]	port 5555 unavailable (cascade)	same
same	[fastwan2_2_ti2v_5b]	port 5555 unavailable (cascade)	same
same	[sana_wm_ti2v]	port 5555 unavailable (cascade)	same

Failure Summary

The first test in the shard, qwen_image_t2i (model Qwen/Qwen-Image), launched its server, bound the scheduler at tcp://127.0.0.1:5555, then hung at NCCL distributed init (Setting distributed timeout to 3600 seconds) and produced no further output until the test harness gave up after 1200 s (TimeoutError: Server not ready within 1200.0s). The hung/killed server left port 5555 occupied, so the remaining 9 parametrizations (and all retry attempts 2/7) failed immediately at ServerArgs._adjust_network_ports → _require_port with RuntimeError: Scheduler port 5555 is unavailable and --strict-ports is enabled. The entire 10-item shard errored; exit code 1.

Regression Status

New regression (for this shard). Shard 2 passed in the previous 4 completed scheduled runs.

Recent history of test_server_1_gpu.py (test_diffusion_generation) in job multimodal-gen-test-1-gpu-amd (…, 2) (branch main, event schedule, completed runs only):

Date	Run	Job	Test File Status	Failed Function	Error
Jun 18 07:46	27744699827	shard 2	❌ Failed	[qwen_image_t2i] + cascade	Timeout 1200s → port 5555 cascade
Jun 17 18:53	27712340726	shard 2	✅ Passed	—	—
Jun 17 13:21	27692029884	shard 2	✅ Passed	—	—
Jun 17 07:57	27674400949	shard 2	✅ Passed	—	—
Jun 17 01:00	27658537520	shard 2	✅ Passed	—	—

First observed failure: 2026-06-18 07:46 UTC. Last known pass: 2026-06-18 18:53 UTC… correction — last known pass run 27712340726 at 2026-06-17 18:53 UTC. (Run 27729699748, head 74e2e48c, between pass and fail is still queued → excluded.)

Failure Cluster

Diffusion server startup hang on qwen_image_t2i (NCCL distributed-init timeout) → orphaned scheduler port 5555 cascades the rest of the shard.

Facts (verified observations)

• [FACT] qwen_image_t2i server bound Scheduler bind at endpoint: tcp://127.0.0.1:5555, logged Initializing distributed environment with world_size=1, device=cuda:0 and Setting distributed timeout to 3600 seconds at 10:11:33, then emitted no further progress for 1200 s → TimeoutError: Server not ready within 1200.0s (test_server_utils.py:492).
• [FACT] All 9 subsequent parametrizations and the retry attempts failed identically at server_args.py:748 _require_port → RuntimeError: Scheduler port 5555 is unavailable and --strict-ports is enabled.
• [FACT] The hung server was killed (kill_process_tree called: parent_pid=4178, include_parent=False) but port 5555 remained occupied for every following launch in the shard.
• [FACT] Shard 2 passed in the 4 prior completed scheduled runs (table above); the failing test is not fp8 and not ltx2.
• [FACT] aiter version unchanged (dev image preserved, 7d604afe5); a transient infra hiccup also appears earlier (Local docker registry pull failed; falling back to public registry) but is unrelated to the GPU-side hang.

Hypothesised Causes (with confidence + disconfirming evidence)

• [LOW] Infrastructure / GPU-runner hang during NCCL init. The server stalls exactly at single-GPU (world_size=1) distributed init with no output for the full 1200 s, which is an unusual place to hang deterministically; the same test passed on the 4 immediately preceding scheduled runs with near-identical SHAs.
  • Supporting: hang is before any model loading; world_size=1 NCCL init should be near-instant; clean 4/4 green history just hours earlier; a registry/infra hiccup is visible in the same run.
  • Disconfirming: cannot rule out a code path change without a repro; only one observation so far (could equally be a one-off flake vs a persistent regression — next scheduled run will disambiguate).
  • How to verify: re-run shard 2 on a fresh MI325 runner; if it passes, this was an infra/GPU hang, not a code regression.
• [LOW] A commit in window 7cead0fb8f..59001267 regressed diffusion server startup. Window contains two diffusion commits: 3b61dc32 "[diffusion] fix fp8 fused tp scale loading (#28546)" and 05b3fd0f "[diffusion] remove ltx2 snapshot mode (#28533)".
  • Supporting: both land in the regression window; both touch diffusion code.
  • Disconfirming: neither touches NCCL/distributed-init (where the hang occurs), and qwen_image_t2i is neither an fp8 nor an ltx2 path — the hang precedes model weight loading where fp8 scale logic runs. So neither commit plausibly explains a hang at Setting distributed timeout.
  • How to verify: if re-run still hangs, bisect the window; otherwise dismiss.
• No hypothesis above LOW confidence. The hang location (pre-model NCCL init) does not match any code-level change in the window; an infra GPU hang is the leading explanation but needs a re-run to confirm.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for strict-ports, Scheduler+port, diffusion+server+timeout, qwen_image; no PR addresses a diffusion server startup hang or the port-5555 cascade. (Note: the harness's --strict-ports cascade behaviour — one hung server poisoning the whole shard — is a separate robustness gap worth a follow-up, but no such PR exists.)

Suggested Triage Steps

• Re-run the multimodal-gen-test-1-gpu-amd (…, 2) shard on a fresh MI325 runner. A pass confirms an infra/GPU NCCL-init hang (flake); a repeat hang at Setting distributed timeout confirms a real regression.
• If it reproduces, bisect 7cead0fb8f..59001267, prioritizing diffusion server/startup commits, by running only test_diffusion_generation[qwen_image_t2i] on AMD MI325.
• Harden the test harness against the cascade: have test_server_utils.py ensure the scheduler port (5555) is freed on server-start failure/timeout, or pick a free port per test, so one hung server does not error the other 9 items + retries.
• Consider raising or per-model-tuning the 1200 s wait_deadline only if startup is legitimately slow; do not mask a genuine hang.

Status

• Persistence: First and so far only failing completed run (2026-06-18 07:46 UTC); 4 immediately preceding completed scheduled runs passed.
• Scope: 1 job/shard (multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 2)), all 10 test_diffusion_generation parametrizations errored at setup; one root failure (qwen_image_t2i timeout) + 9 cascade failures from the held port 5555.
• Blocked work: AMD 1-GPU diffusion (multimodal_gen) CI signal for this shard.
• In-flight fix: ❌ none found in open PRs.

multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 1) — failed step(s): Run diffusion server tests (2-GPU)

Commit Info

• sglang: 59001267
• aiter: 7d604afe (v0.1.14rc1.dev240+g7d604afe5) — source: dev image preserved (Dev/patched version detected → skipping rebuild)

Failed Tests

Test File	Test Function	Error	Log
python/sglang/multimodal_gen/test/server/test_server_2_gpu.py	TestDiffusionServerTwoGpu::test_diffusion_generation[flux2_modelopt_fp8_tp2_t2i]	openai.InternalServerError: Internal Server Error (server-side RuntimeError: CUDA error: HIPBLAS_STATUS_NOT_SUPPORTED at torch._scaled_mm)	Log
python/sglang/multimodal_gen/test/server/test_server_2_gpu.py	TestDiffusionServerTwoGpu::test_diffusion_generation[ideogram4_fp8_tp2_t2i]	huggingface_hub.errors.GatedRepoError: 403 Client Error (setup ERROR, gated HF model)	Log

Failure Summary

The 2-GPU multimodal-gen job ran 12 items: 9 passed, 1 skipped, 1 failed (flux2_modelopt_fp8_tp2_t2i), 1 error (ideogram4_fp8_tp2_t2i). The flux2 failure surfaces as a client-side openai.InternalServerError (HTTP 500); the server crashed during warmup with RuntimeError: CUDA error: HIPBLAS_STATUS_NOT_SUPPORTED raised from torch._scaled_mm in python/sglang/srt/layers/quantization/fp8_utils.py:1681 (FP8 per-tensor GEMM), reached via the flux2 DiT qkv projection (flux_2.py:179). The ideogram4 error is an unrelated gated-repo 403 (HF access).

Regression Status

Never-passed (newly-added test) — flux2_modelopt_fp8_tp2_t2i did not exist before this window; it was introduced by #28546 (3b61dc32) and fails on its first scheduled appearance on AMD MI325.

Recent history of test_server_2_gpu.py (flux2_modelopt_fp8_tp2_t2i) in job multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 1) (branch=main, event=schedule, completed runs only):

Date	Run	Test Status	Note
Jun 18 07:46	27744699827	❌ Failed	flux2 tp2 test present for the first time; HIPBLAS_STATUS_NOT_SUPPORTED
Jun 17 18:53	27712340726	⬜ Not present	test did not exist (10 passed, 1 error)
Jun 17 13:21	27692029884	⬜ Not present	10 passed, 1 error
Jun 17 07:57	27674400949	⬜ Not present	10 passed, 1 error
Jun 17 01:00	27658537520	⬜ Not present	10 passed, 1 error

The ideogram4_fp8_tp2_t2i ERROR is a persistent infrastructure failure (gated HF repo 403) present in all 5 completed runs above — not new.

Failure Cluster

"HIPBLAS_STATUS_NOT_SUPPORTED in FP8 torch._scaled_mm during diffusion warmup" (flux2). Separately: "GatedRepoError 403 on gated HF model download" (ideogram4).

Facts (verified observations, no interpretation)

• Client-side: openai.InternalServerError: Internal Server Error at test_server_utils.py:1005 → client.images...generate (log line ~10680).
• Server-side root error (verbatim): RuntimeError: CUDA error: HIPBLAS_STATUS_NOT_SUPPORTED when calling \HIPBLAS_STATUS_NOT_SUPPORTED`printed duringWarmup requests 0/1` at 08:49:19 (log ~line 4433).
• Server stack trace: denoising.py:1820 _predict_noise → flux_2.py:1221/862/423 forward → flux_2.py:179 _get_qkv_projections → linear.py:427 forward → modelopt_quant.py:425 apply → fp8_utils.py:1681 apply_fp8_linear (torch._scaled_mm, per-tensor weights & activations path).
• flux2_modelopt_fp8_tp2_t2i is absent from the prior run's shard list and absent from prior logs (0 occurrences) — test is brand new.
• Commit 3b61dc32 (#28546, authored Jun 18 06:38 UTC) added this test to TWO_GPU_CASES in gpu_cases.py AND modified PerTensorScaleParameter fused-slot loading in multimodal_gen/runtime/layers/linear.py (MergedColumnParallelLinear / QKVParallelLinear). It landed in the regression window 7cead0fb8f..59001267c3.
• aiter version unchanged vs prior runs (dev image preserved); NVIDIA equivalents not in scope here.
• Reproducibility: 1 run observed (first appearance); 1 job affected.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [MEDIUM] The FP8 per-tensor scaled GEMM (torch._scaled_mm, fp8_utils.py:1681) is not supported by hipBLASLt for the shape/scale layout produced by the flux2 modelopt FP8 transformer at tp_size=2 on MI325 (gfx942).
  • Supporting: stack trace lands exactly on torch._scaled_mm; the test is the first FP8 modelopt case run at TP2; the 1-GPU flux2_modelopt_fp8_t2i case was simultaneously removed by the same PR, so TP2 is the new variable. The same commit (3b61dc32) changed how fused per-tensor scales are filled (param.data.fill_(...) producing a multi-element scale vector for fused QKV/MergedColumn), which may feed _scaled_mm a scale shape hipBLASLt rejects.
  • Disconfirming: no green baseline ever existed for this test on AMD, so we cannot prove the GEMM itself (vs the scale-loading change) is the trigger; the error is a generic "not supported" with no shape dump in the log. The scale change targets weight loading, while the failure is at GEMM execution — the link between them is inferred, not directly shown.
  • How to verify: on an MI325 container, run sglang serve --model-path black-forest-labs/FLUX.2-dev --num-gpus 2 --tp-size 2 --transformer-path lmsys/flux2-dev-modelopt-fp8-sglang-transformer; print qinput.shape, weight.shape, x_scale.shape, weight_scale.shape at fp8_utils.py:1681; compare TP2 vs TP1 to isolate whether the per-tensor scale became a vector after 3b61dc32.
• [LOW] Pure hipBLASLt/ROCm coverage gap unrelated to the PR (FP8 rowwise/per-tensor _scaled_mm historically ROCm-version-dependent — see fp8_utils.py:1660-1665).
  • Supporting: error is HIPBLAS_STATUS_NOT_SUPPORTED, a backend-capability error.
  • Disconfirming: other FP8 modelopt cases (e.g. 1-GPU flux variants) passed historically, so FP8 _scaled_mm is not globally unsupported on this image.
  • How to verify: A/B the same model at TP1 vs TP2 on the same container.

In-flight Fix Check

• In-flight fix: ❌ none found directly addressing the flux2 FP8 _scaled_mm HIPBLAS error. Search repo:sgl-project/sglang is:pr is:open scaled_mm rocm → 0 results. Search flux2 fp8 → only #27702 ("[diffusion][ROCm]: Enable BF16 attention and MXFP4 compilation on ROCm"), which concerns BF16 attention / MXFP4, not the FP8 per-tensor scaled GEMM — not a direct fix.
• ideogram4 gated-repo 403: not a code issue; needs HF access/token grant for ideogram-ai/ideogram-4-fp8 on the runner.

Suggested Triage Steps

• Reproduce on MI325: serve flux2 modelopt FP8 at --tp-size 2 and dump operand/scale shapes at fp8_utils.py:1681 to confirm whether the scale shape is what hipBLASLt rejects.
• A/B the scale-loading change: temporarily revert the PerTensorScaleParameter fill changes in linear.py from 3b61dc32 on a branch and re-run the flux2 tp2 case; if it passes, the fused-scale fill is implicated.
• If hipBLASLt genuinely lacks support for this shape, consider gating the flux2 modelopt FP8 TP2 case off AMD (or routing to the channelwise/unfused fallback path) until supported — maintainer decision.
• Separately, fix ideogram4_fp8_tp2_t2i by granting runner access to the gated ideogram-ai/ideogram-4-fp8 repo, or mark it skipped on CI until access is provisioned.

Status

• Persistence: flux2 — first observed 2026-06-18 07:46 UTC (1 run; test is brand new). ideogram4 — failing every completed run in the queryable window (gated-repo 403).
• Scope: 1 job, this workflow (pr-test-amd.yml, 2-GPU multimodal-gen shard 1) for flux2. ideogram4 affects the same job across all runs.
• Blocked work: blocks green signal for the new flux2 modelopt FP8 TP2 diffusion path on AMD MI325.
• In-flight fix: ❌ none found for the flux2 FP8 _scaled_mm HIPBLAS error; #27702 is related ROCm-diffusion work but not a direct fix.

call-pr-test-amd-extra / extra-a-test-1-gpu-large-amd (linux-mi325-1gpu-sglang) — failed step(s): Run test

Commit Info

• sglang: 59001267
• aiter: 7d604afe5 (v0.1.14rc1.dev240+g7d604afe5; source: dev/patched image build preserved — [CI-AITER-CHECK] line, log)

Failed Tests

Test File	Test Function	Error	Log
test/registered/sessions/test_streaming_session_extra.py	(none — test interrupted, not failed)	Job externally cancelled mid-run (##[error]The operation was canceled.)	Log

No test assertion failed, no crash, no Python traceback, and no GPU fault occurred. This job's conclusion is cancelled, not failure.

Failure Summary

The job ran the extra-a-test-1-gpu-large-amd suite (3 files enabled: test_streaming_session_extra.py, test_spec_standalone_extra.py, test_fp8kv_triton.py). Only the first file (test_streaming_session_extra.py) began executing (Begin (0/2) at 12:59:29). The SGLang server was healthy and actively serving /generate requests (decode batches at ~930 tok/s, prefill at 30k+ tok/s) right up to the instant of cancellation at 13:07:57. The job was terminated externally before the test file could complete — it was never a test-level failure.

Regression Status

Infrastructure / orchestration cancellation — not a code or test regression.

Recent history of test_streaming_session_extra.py in job extra-a-test-1-gpu-large-amd (pr-test-amd.yml, event=schedule, branch=main, completed runs only):

Date	Run	Job	Test File Status	Note
Jun 18	27744699827	extra-a-test-1-gpu-large-amd	⚠️ Cancelled (interrupted)	test running healthily when cancelled
Jun 17 18:53	27712340726	extra-a group	⚠️ Cancelled	extra-a-test-1-gpu-small-amd cancelled; large job did not appear (cancelled before start)

The extra-a suite jobs are recurrently caught by run-wide fast-fail cancellation in these scheduled runs rather than failing on their own merits. No green-vs-red transition exists for this test file in the queryable window because it keeps getting interrupted, so no pass/fail verdict can be drawn for the test itself.

Failure Cluster

Job cancelled by run-wide fast-fail cascade (in-progress test interrupted)

Facts (verified observations, no interpretation)

• Job conclusion via API = cancelled; started 12:53:12Z, ended 13:08:00Z (jobs API).
• Final log line: ##[error]The operation was canceled. at 13:07:57 (log). Immediately before it, the server was processing decode/prefill batches and returning POST /generate HTTP/1.1 200 OK.
• The sibling job call-pr-test-amd-extra / extra-a-test-1-gpu-small-amd succeeded (ended 12:51:26Z) — the cancellation was not triggered from within the extra group.
• The same run 27744699827 had 7 genuine failure jobs in other stages: multimodal-gen-test-1-gpu-amd (2), multimodal-gen-test-2-gpu-amd (1), stage-b-test-1-gpu-small-amd (5), stage-c-test-large-8-gpu-amd-mi35x (0), stage-c-test-large-8-gpu-amd (3), plus two *-finish aggregation gates.
• pr-test-amd-extra-finish failed (13:08:04Z) only because it aggregates job results and saw extra-a-test-1-gpu-large-amd: cancelled → exit 1 (finish log). It started after our job was already cancelled, so it is downstream, not the cause.
• Two transient Health check failed. Server couldn't get a response from detokenizer for last 20 seconds warnings (13:02:10, 13:07:39) coincide with aiter JIT kernel builds blocking the detokenizer heartbeat — e.g. start build [mha_batch_prefill_...] → finish build ... cost 26.2s. The server recovered immediately after each (subsequent GET /health_generate 200 OK), so these are not the failure.
• AITER_COMMIT_OVERRIDE is empty — this is a regular pr-test-amd.yml scheduled run, not an AITER Scout run (no Baseline A/B / Failure Origin applies).

Hypothesised Causes (with confidence, with disconfirming evidence)

• [HIGH] The job was terminated by the workflow's fast-fail cancellation cascade triggered by genuine failures in other stages of the same run, not by anything in the extra-a-large test itself.
  • Supporting: job conclusion is cancelled (not failure); 7 real failures exist elsewhere in the run; the test server was healthy and serving 200-OK responses at the cancellation instant; the sibling extra-a-small that finished earlier passed; .github/actions/wait-for-jobs implements stage fast-fail (core.setFailed on upstream failures, action.yml:213).
  • Disconfirming: cancel-in-progress is false for schedule events (pr-test-amd.yml:117), so concurrency-group cancel is ruled out; I did not find an explicit gh run cancel step, so the exact cancel emitter (overall run timeout vs. an orchestration controller) is not pinned down at FACT level.
  • How to verify: re-run the extra-a-test-1-gpu-large-amd job in isolation (or with the bypass-fastfail path) and confirm test_streaming_session_extra.py completes; inspect the run's top-level timeout / controller settings for the cancel emitter.
• [LOW] The two detokenizer health-check warnings indicate a real hang.
  • Supporting: warning text mentions a 20 s detokenizer stall.
  • Disconfirming: each stall is explained by a ~26 s aiter JIT kernel compile, and the server resumed normal request handling immediately after both; not present at the cancellation point.
  • How to verify: not actionable — server demonstrably recovered.

In-flight Fix Check

This is a CI orchestration cancellation, not a code defect, so there is no source fix to chase. The actionable signal is the 7 real upstream failures in the run that triggered the cascade (multimodal-gen, stage-b, stage-c) — those should be triaged in their own per-job analyses. No PR-search performed for this job because there is no failing test/library to key on.

Suggested Triage Steps

• Treat this job as a non-failure (cancelled victim) — do not attribute any test defect to test_streaming_session_extra.py from this run.
• Re-run extra-a-test-1-gpu-large-amd alone (or with bypass-fastfail) to obtain a clean pass/fail for test_streaming_session_extra.py, since it has been repeatedly interrupted in scheduled runs.
• Focus root-cause effort on the run's genuine failures: multimodal-gen-test-{1,2}-gpu-amd, stage-b-test-1-gpu-small-amd (5), and the two stage-c-test-large-8-gpu-amd* jobs — one of these is what tripped the fast-fail.
• If extra-a jobs keep getting cancelled before completing, consider decoupling their cancellation from unrelated-stage failures so the suite produces a real signal.

Status (factual, no priority assignment)

• Persistence: This job's extra-a suite has been cancelled (not completed) in at least the last 2 scheduled runs (27744699827, 27712340726).
• Scope: This single job in run 27744699827; cancellation is a side-effect of 7 unrelated failures elsewhere in the same run.
• Blocked work: test_streaming_session_extra.py, test_spec_standalone_extra.py, and test_fp8kv_triton.py get no CI signal when this job is cancelled (the latter two never started).
• In-flight fix: N/A — no code fix applies; resolve the upstream failures that trigger the cascade.

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-06-20 00:03 UTC)

[amd-bot]

pr-test-amd.yml — Per-Job Analysis (continued 2/7)

stage-b-test-1-gpu-small-amd (linux-mi325-1gpu-sglang, 5) — failed step(s): Run test

Commit Info

• sglang: 59001267
• aiter: 7d604afe — v0.1.14rc1.dev240+g7d604afe5 (source: dev image preserved; matches Dockerfile default, no override)

Failed Tests

Test File	Test Function	Error	Log
test/registered/scheduler/test_retract_decode.py	server never reached ready (setUp/launch)	✗ TIMEOUT after 2400 seconds — server hung at "Init torch distributed begin"	Log
test/registered/constrained_decoding/test_constrained_decoding.py	server never reached ready (setUp/launch)	server hung the same way → job-level ##[error] timed out after 60 minutes	Log

Failure Summary

The job hung, not asserted. The first file, test_retract_decode.py (Llama-3.1-8B), launched its server which printed Init torch distributed begin and then never progressed; the scheduler process was stuck for the full 2400 s --timeout-per-file window and was killed as ✗ TIMEOUT. The next file, test_constrained_decoding.py (Llama-3.2-1B), launched and hung in the identical way, consuming the remainder of the 60-minute GitHub step budget until ##[error]The action 'Run test' has timed out after 60 minutes.

Regression Status

Flaky / infrastructure hang — not a code regression. The same partition ran the same two files in the prior scheduled run and passed.

Recent history of test_retract_decode.py in job stage-b-test-1-gpu-small-amd (…, 5) (branch main, event schedule, completed runs only):

Date	Run	Job	Test File Status	Note
Jun 18	27744699827	partition 5	❌ TIMEOUT (hang at dist init)	this run
Jun 17	27712340726	partition 5	✅ Passed	same 2 files; retract_decode elapsed=498s
Jun 17	27692029884	partition 5	✅ Passed	—
Jun 17	27674400949	partition 5	✅ Passed	—

First observed failure: this run (Jun 18). Last passing: Jun 17 18:53 UTC. The partition layout is stable (test_retract_decode.py + test_constrained_decoding.py in partition 5 across all runs above).

Failure Cluster

GPU/distributed-init hang during server warmup (timeout, no crash)

Facts (verified observations)

• Scheduler py-spy dump (PID 3595) at 10:49:24 shows MainThread (active) inside libhsa-runtime64.so.1.18.70000 below __init__ (distributed/parallel_state.py:301) → init_world_group (parallel_state.py:1509) → init_distributed_environment (parallel_state.py:1849) → init_torch_distributed (model_runner.py:1229) → tp_worker.py:262 → scheduler.py:778. The hang is in ProcessGroup creation, blocked in the ROCm HSA runtime.
• The TokenizerManager MainThread (PID 3416) is blocked in _wait_for_scheduler_ready (engine.py:1358) for the whole window — i.e. it is waiting on the wedged scheduler above.
• Last log progress for the first server: [2026-06-18 10:44:25] Init torch distributed begin. Nothing after it except repeated soft-watchdog timeouts (300 s) at 10:49 / 10:54 / 10:59 … until ✗ TIMEOUT … after 2400 seconds at 11:23:14.
• GPU was healthy at job start: ✓ VRAM usage is within acceptable limits / ✓ VRAM is already clear (10:34:27).
• Scope: only 1 of 14 stage-b-small partitions failed in this run (partition 5); 7 failed jobs total across the whole run. The other 13 same-runner-class partitions passed.
• Same code, same partition, same two files passed in the immediately prior scheduled run (27712340726, test_retract_decode.py elapsed=498s).
• aiter unchanged vs passing runs (Dockerfile-default dev build, no override).

Hypothesised Causes

• [MEDIUM] Runner/GPU wedged during the first server's init_world_group (ProcessGroup creation), leaving the GPU in a bad state so the second server also hung.
  • Supporting: scheduler thread is active (spinning), not idle, inside libhsa-runtime64.so at the very first GPU/distributed init; both consecutive servers on this GPU hung identically; isolated to one partition/runner while 13 sibling partitions passed; identical code passed 498 s the prior run; VRAM clean at start.
  • Disconfirming: VRAM pre-check was clean, so no leftover process was detected before the job; no HIP/HSA error string was emitted (a pure hang, not a fault).
  • How to verify: re-run stage-b-test-1-gpu-small-amd (…, 5) on a different MI325 runner; if it passes, confirms runner-specific. Check the runner host (linux-mi325-1gpu-sglang-57dnj-runner-xkmls) for a stuck GPU / leftover process.
• [LOW] A code change in d86a7e7018..59001267 altered distributed-init/warmup timing.
  • Supporting: temporal — failure first appears at HEAD 59001267.
  • Disconfirming: hang is in parallel_state.init_world_group (torch/vendored distributed code), the symptom is a non-deterministic single-partition hang, and the same code passed on partition 5 of the prior run — inconsistent with a deterministic regression.
  • How to verify: re-run at this SHA several times; if green, regression is disproven.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for test_retract_decode / init_torch_distributed hang; the 76 matches are unrelated hang fixes (router, warmup-shutdown, byte-token decode). No PR targets this scheduler/HSA init hang.

Suggested Triage Steps

• Re-run stage-b-test-1-gpu-small-amd (linux-mi325-1gpu-sglang, 5) for run 27744699827; if it passes, classify as a flaky runner-GPU hang and move on.
• If it reproduces, inspect runner host linux-mi325-1gpu-sglang-57dnj-runner-xkmls for a wedged GPU / orphaned scheduler process and consider draining it.
• If reproducible across runners, bisect d86a7e7018..59001267 focusing on changes to model_runner.py init_torch_distributed / parallel_state.py / scheduler init, and run test_retract_decode.py on an MI325 container.

Status

• Persistence: First failure this run (Jun 18 07:46 UTC schedule); previous 3 scheduled runs passed on the same partition.
• Scope: 1 of 14 stage-b-small partitions in this run; the hang blocks 2 files (test_retract_decode.py, test_constrained_decoding.py).
• Blocked work: scheduler retract-decode + constrained-decoding signal on AMD for this run only.
• In-flight fix: ❌ none found — needs a re-run to confirm flaky vs persistent before opening any PR.

stage-c-test-large-8-gpu-amd-mi35x (linux-mi35x-gpu-8, 0) — failed step(s): Run test

Commit Info

• sglang: 59001267
• aiter: 7d604afe5fa7efba63c0dce323b95d9daf2db112 (v0.1.14rc1.dev240+g7d604afe5, source: dev image preserved — equals Dockerfile default, no override)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/test_deepseek_r1_mxfp4_8gpu.py	TestDeepseekR1MXFP4MTP.test_a_gsm8k	AssertionError: 1.452425228575856 not greater than 2.04	Log

Failure Summary

The EAGLE/MTP speculative-decoding variant of the DeepSeek-R1-MXFP4 8-GPU test (TestDeepseekR1MXFP4MTP.test_a_gsm8k) asserts avg_spec_accept_length > 2.04, but the measured average speculative accept length was only ~1.45 (1.446 on the first attempt, 1.452 on the retry). GSM8K accuracy passed (0.98 / 0.97 > 0.94); only the speculative-accept-length assertion at line 163 failed, exhausting the single retry and erroring the test (Ran 3 tests … FAILED (errors=1)).

Regression Status

Never-passed (no green run found). The test file was added on 2026-06-17 02:27 UTC by #25702 (9c53853e, Trevor Morris). The avg_spec_accept_length > 2.04 assertion has failed in every completed scheduled run of this job since the file landed.

Recent history of test/registered/amd/test_deepseek_r1_mxfp4_8gpu.py (TestDeepseekR1MXFP4MTP.test_a_gsm8k) in job stage-c-test-large-8-gpu-amd-mi35x (linux-mi35x-gpu-8, 0) (branch main, event schedule, completed runs only):

Date (UTC)	Run	Job	Test File Status	accept_len	Error
Jun 18 07:46	27744699827	job	❌ Failed	1.452	1.452 not greater than 2.04
Jun 17 18:53	27712340726	job	❌ Failed	1.443	1.443 not greater than 2.04
Jun 17 13:21	27692029884	job	❌ Failed	1.455	1.455 not greater than 2.04
Jun 17 07:57	27674400949	job	❌ Failed	1.459	1.459 not greater than 2.04 (first run with this test)
(before Jun 17 02:27)	—	—	n/a	—	test file did not exist

The measured accept length is remarkably stable across runs (~1.44–1.46), i.e. this is a deterministic threshold mismatch, not a flake.

Failure Cluster

EAGLE/MTP speculative accept-length below threshold (DeepSeek-R1-MXFP4 MI35x TP8)

Facts (verified observations, no interpretation)

• Verbatim error (both attempts): AssertionError: 1.4462281795511223 not greater than 2.04 then 1.452425228575856 not greater than 2.04; then Exception: retry() exceed maximum number of retries. (log lines 13344, 14181, 14202).
• GSM8K accuracy on the MTP run passed: Accuracy: 0.980 / 0.970, well above the 0.94 gate (log lines 13352, 14183).
• Decode-batch telemetry during the run shows low acceptance: accept len: 1.49, accept rate: 0.16 … accept len: 1.38, accept rate: 0.12 (log lines 13304, 14160–14161) — consistent with the final avg_spec_accept_length=1.45.
• Spec config exercised: --speculative-algorithm EAGLE --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 (test file lines 111–118).
• The MTP class sets SGLANG_ENABLE_OVERLAP_PLAN_STREAM=True and SGLANG_USE_AITER_AG=False (test file lines 102–104).
• aiter unchanged: dev image v0.1.14rc1.dev240+g7d604afe5 matches Dockerfile default; AITER_COMMIT_OVERRIDE empty (log lines 4131–4142).
• Reproducibility: identical failure in 4/4 completed runs since the test was introduced; the test has no recorded passing run.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [MEDIUM] The 2.04 threshold was mis-calibrated when the test was authored — the EAGLE draft on this MI35x MXFP4 TP8 configuration deterministically yields ~1.45 accept length, so the assertion never had a green baseline.
  • Supporting: the test has never passed since it was added (#25702); accept length is tightly clustered ~1.44–1.46 across 4 runs (no variance suggesting a transient regression); accuracy passes, so generation itself is correct.
  • Disconfirming: cannot exclude that the EAGLE draft head / MXFP4 path was always under-performing on AMD and the author validated 2.04 on a different (e.g. NVIDIA) backend; no green run exists on any backend in the queryable AMD history to confirm the intended baseline.
  • How to verify: re-run the MTP test on MI35x and inspect avg_spec_accept_length directly; compare against the same model+spec config on a known-good backend to see whether ~2.04 is achievable anywhere.
• [LOW] A speculative-decoding / EAGLE draft-model code path regressed before the test landed, depressing acceptance on AMD MXFP4.
  • Supporting: accept rate (~0.12–0.17) is low for EAGLE with topk=1, num-steps=3.
  • Disconfirming: there is no pre-existing green run to regress from — the test debuted already-red, so a "regression" cannot be localized to a window; this is indistinguishable from hypothesis 1 without an external baseline.
  • How to verify: bisect EAGLE/MXFP4 spec-decode commits using a manual run of this config on MI35x, comparing accept length across SHAs.

In-flight Fix Check

• In-flight fix: ❌ none found targeting this threshold. Searches: is:pr is:open mxfp4 accept (17 hits, none about this accept-length gate) and is:pr is:open DeepSeek MXFP4 (54 hits). Tangentially related open PRs that touch EAGLE/MXFP4 on AMD but do not address this assertion: #25715 (fix(eagle/dsv4): fix EAGLE draft attn init called with unshaped out_cache_loc) and #26216 ([AMD] bug-fix mxfp4 fmoe shuffle and interlave pass).

Suggested Triage Steps

• Confirm intended baseline: manually launch the MTP config on MI35x and read avg_spec_accept_length; if it reliably sits at ~1.45, the 2.04 gate in test_deepseek_r1_mxfp4_8gpu.py:163 (and the matching test_bs_1_speed gate at line 177) is the defect, not production code — ping the #25702 author (Trevor Morris) to recalibrate or to identify the AMD-specific spec-decode gap.
• If a higher accept length is expected on AMD, bisect EAGLE/MXFP4 speculative-decode code paths (draft head, --speculative-eagle-topk 1 path) to find why acceptance is ~1.45; check whether #25715 affects the EAGLE draft init on this model.
• Until resolved, the assertion is a persistent red on this 8-GPU MI35x suite — consider relaxing/quarantining the avg_spec_accept_length gate (keeping the accuracy gate) so the job's other tests are not masked.

Status (factual, no priority assignment)

• Persistence: Failing every completed scheduled run since the test was introduced (~2026-06-17 02:27 UTC); 4/4 completed runs red, deterministic (~1.45 vs 2.04), no green run on record.
• Scope: One test function (TestDeepseekR1MXFP4MTP.test_a_gsm8k) in one job (stage-c-test-large-8-gpu-amd-mi35x (linux-mi35x-gpu-8, 0)) in pr-test-amd.yml. The non-MTP TestDeepseekR1MXFP4 GSM8K and the speed test are unaffected.
• Blocked work: Keeps the MI35x 8-GPU large-test suite red; masks signal for other tests sharded into job index 0.
• In-flight fix: ❌ none found targeting the accept-length threshold.

stage-c-test-large-8-gpu-amd (linux-mi325-8gpu-sglang, 3) — failed step(s): Run test

Commit Info

• sglang: 59001267
• aiter: 7d604afe (v0.1.14rc1.dev240+g7d604afe5, source: dev image — [CI-AITER-CHECK] Dev/patched version detected → skipping rebuild)

Failed Tests

Test File	Test Function	Error	Log
test/registered/ops/test_aiter_allreduce_fusion_amd.py	test_fused_ar_rms_residual_accuracy	AssertionError: Residual accuracy check failed. (every shape max_diff=6.250000e-02, frac_nonzero≈0.28, AR_exact=yes)	Log

Failure Summary

The distributed sub-test test_fused_ar_rms_residual_accuracy (launched via torch.distributed.run --nproc_per_node=4) failed for all 49 tested shapes (hidden dims 2880–8192, batch 1–128). The fused all-reduce + RMSNorm + residual path (tensor_model_parallel_fused_allreduce_rmsnorm) diverges from the unfused reference by max_diff=6.25e-2 while the all-reduce result itself is bit-exact (AR_exact=yes). The other 3 test files in this shard passed (Test Summary: 3/4 passed).

Regression Status

Never-passed (no green run found in the queryable window). This test file was introduced in 9c53853e (2026-06-16) and has failed in every completed schedule-triggered run since.

Recent history of test_aiter_allreduce_fusion_amd.py (test_fused_ar_rms_residual_accuracy) in job stage-c-test-large-8-gpu-amd (…, 3) (branch main, event schedule, completed runs only):

Date	Run	Test File Status	Failed Function	Error
Jun 18	27744699827	❌ Failed	test_fused_ar_rms_residual_accuracy	max_diff=6.25e-2
Jun 17	27712340726	❌ Failed	same	same
Jun 17	27692029884	❌ Failed	same	same
Jun 17	27674400949	❌ Failed	same	same
Jun 17	27658537520	❌ Failed	same	same
Jun 16	27641945084	❌ Failed	same	same
Jun 16	27622613377	❌ Failed	same	same

First observed failure: 2026-06-16 (first run after the test was added). No passing run exists for this test file in the lookback window — it has been red since introduction.

Failure Cluster

"Fused all-reduce+RMSNorm residual bit-exactness mismatch (aiter #2586 not in CI image)"

Facts (verified observations, no interpretation)

• Log evidence: FAIL: test_fused_ar_rms_residual_accuracy, AssertionError: Residual accuracy check failed. at test_aiter_allreduce_fusion_amd.py:317; torchrun child exitcode: 1.
• Symptom is uniform across all 49 shapes: max_diff=6.250000e-02 (= 2⁻⁴), frac_nonzero≈0.28, and crucially AR_exact=yes — the all-reduce is bit-identical; only the residual rounding order diverges.
• The test (_run_residual_accuracy_check, lines 18–138) asserts max_diff == 0.0 (bit-exact); its docstring states this is a regression test for Fix: Numerical Accuracy in allreduce_fusion_kernel_1stage ROCm/aiter#2586 — allreduce_fusion_kernel_1stage accumulates in f32 and adds the residual before rounding to bf16, whereas the unfused path rounds the all-reduce to bf16 first.
• aiter version in image: v0.1.14rc1.dev240+g7d604afe5 (7d604afe), preserved from the dev image (no override; AITER_COMMIT_OVERRIDE empty).
• Fix: Numerical Accuracy in allreduce_fusion_kernel_1stage ROCm/aiter#2586 is closed (title: "Fix: Numerical Accuracy in allreduce_fusion_kernel_1stage"); per PR #28226 the fix is aiter commit 43b7379b8, which the CI image aiter predates.
• Reproducibility: 7/7 completed scheduled runs since 2026-06-16, every shape.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [FACT] The prebuilt aiter kernel in the CI docker image lacks the Fix: Numerical Accuracy in allreduce_fusion_kernel_1stage ROCm/aiter#2586 accuracy fix (43b7379b8). The image ships aiter 7d604afe, which predates that commit, so the fused 1-stage kernel adds the residual before the bf16 round-trip and diverges from the unfused reference by exactly the bf16 rounding magnitude.
  • Supporting: AR_exact=yes everywhere (all-reduce correct) with non-zero residual max_diff=6.25e-2 matches the #2586 description verbatim; PR #28226 body independently identifies the same image/aiter version gap.
  • Disconfirming: (none found) — the symptom, the test's own docstring, and the in-flight PR all converge on the same cause.
  • How to verify: rebuild the CI image with aiter ≥ 43b7379b8 and re-run; or merge PR #28226 which relaxes the assertion to 1 bf16 ULP.

Note: this is pr-test-amd.yml (not a scout run), so no Baseline A/B check / Failure Origin section applies.

In-flight Fix Check

• In-flight fix: ✅ #28226 [AMD] Relax allreduce-fusion residual accuracy tolerance to 1 bf16 ULP (open since 2026-06-15, not draft; edits exactly this test file). Its description pins the same root cause (CI image aiter predates Fix: Numerical Accuracy in allreduce_fusion_kernel_1stage ROCm/aiter#2586 fix 43b7379b8). Chase reviewers — do NOT open a duplicate.
• API used: search/issues?q=repo:sgl-project/sglang+is:pr+is:open+allreduce+fusion+residual.

Suggested Triage Steps

• Land the in-flight fix #28226 (relaxes the bit-exact assertion to 1 bf16 ULP), or alternatively bump the CI docker image's aiter to a build that includes Fix: Numerical Accuracy in allreduce_fusion_kernel_1stage ROCm/aiter#2586 (43b7379b8) so the kernel becomes bit-exact and the original max_diff == 0.0 assertion can stand.
• If neither can land quickly, consider quarantining test_fused_ar_rms_residual_accuracy on AMD until the aiter image is updated, since it has never passed on the current CI image.

Status (factual, no priority assignment)

• Persistence: Failing since 2026-06-16 (first run after the test was added); 7/7 completed scheduled runs red, every shape — never-passed.
• Scope: stage-c-test-large-8-gpu-amd (…, 3) in pr-test-amd.yml; isolated to this one test file (other 3 files in the shard pass).
• Blocked work: blocks a clean green signal for the stage-c-test-large-8-gpu-amd shard on AMD MI325 8-GPU.
• In-flight fix: ✅ #28226 (open since 2026-06-15) — awaiting merge.

call-pr-test-amd-extra / extra-a-test-1-gpu-large-amd (linux-mi325-1gpu-sglang) — failed step(s): N/A

Commit Info

• sglang: 61a8b42c
• aiter: N/A — the job recorded zero steps (never reached amd_ci_install_dependency.sh), so no [CI-AITER-CHECK] markers exist. The GitHub log blob returns BlobNotFound.

Failed Tests

Test File	Test Function	Error	Log
N/A (no test executed)	N/A	Job cancelled while queued for a runner — 0 steps ran	Job

This job ran no tests. Its conclusion is cancelled, not failure. It sat queued for an MI325 1-GPU runner for ~85 min (started 18:52:07Z, cancelled 20:17:23Z) and never executed even "Set up job".

Failure Summary

The job call-pr-test-amd-extra / extra-a-test-1-gpu-large-amd was cancelled, not failed. It is a label-gated, schedule-only "extra" tier job that never acquired a linux-mi325-1gpu-sglang runner. Its sibling extra-a-test-1-gpu-small-amd was cancelled at the identical timestamp (20:17:23Z), and the aggregator pr-test-amd-extra-finish then exited 1 — by design it treats a cancelled dependency as a failure. The surfaced "failure" therefore has no associated test or code regression.

Regression Status

Known recurring failure (infra/orchestration — the extra-finish aggregator trips on a cancelled extra-a in every recent schedule run).

Recent history of extra-a-test-1-gpu-large-amd / pr-test-amd-extra-finish in pr-test-amd.yml (branch main, event=schedule, completed runs only):

Date	Run	extra-a-large Status	Notes
Jun 18 18:52	27782126506	⚠️ Cancelled (0 steps)	this run; small also cancelled (0 steps); finish=failure
Jun 18 07:46	27744699827	⚠️ Cancelled (8 steps, mid-run)	small succeeded; finish=failure
Jun 17 18:53	27712340726	—	small cancelled (0 steps); finish=failure

Every scheduled pr-test-amd.yml run in the last ~2 days concludes failure, driven by this cancel→finish-failure mechanism, not by a red test.

Failure Cluster

"extra-a job cancelled (runner starvation / concurrency) → extra-finish aggregator reports failure"

Facts (verified observations)

• Job conclusion is cancelled; the API steps array is empty — no step (not even "Set up job") executed. [job API]
• Both extra-a jobs cancelled at the exact same instant: 2026-06-18T20:17:23Z. [job API]
• The job log endpoint returns BlobNotFound (no log was ever produced — consistent with a never-started job).
• The finish job log (verbatim): extra-a-test-1-gpu-small-amd: cancelled → The above jobs failed. → ##[error]Process completed with exit code 1. Its if [[ "$result" == "failure" || "$result" == "cancelled" ]] branch fires on cancelled. (finish job)
• call-gate succeeded, so the extra-a if: !cancelled() && needs.call-gate.result == 'success' gate was satisfied — the jobs were eligible and merely queued.
• Three scheduled pr-test-amd.yml runs are simultaneously queued/in-progress (27782126506, 27762119155, 27729699748) — the AMD MI325 pool is heavily backlogged.
• pr-test-amd-extra.yml:83-85 concurrency group is pr-test-amd-extra-schedule-main-all (no run_id in the key) with cancel-in-progress: ${{ github.event_name != 'workflow_call' }}, which evaluates to true for a schedule-triggered call → overlapping schedule runs share one group and cancel each other's extra workflow.

Hypothesised Causes (with confidence)

• [MEDIUM] Concurrency-group eviction. The extra reusable workflow's group key omits run_id and cancel-in-progress is effectively true for schedule runs; with 3+ schedule runs piled up in the backlogged AMD queue, a newer run's extra workflow cancels this run's still-queued extra-a jobs.
  • Supporting: config at pr-test-amd-extra.yml:83-85; multiple concurrent queued schedule runs observed; both extra-a jobs cancelled at one instant (concurrency cancels the whole child workflow atomically).
  • Disconfirming: in my spot-checks I could not pin a sibling schedule run whose gate-pass time lands exactly at each cancellation timestamp (e.g. the 13:08:00Z cancel in run 27744699827 precedes run 27762119155's 13:15 start), so a different trigger may contribute.
  • How to verify: inspect the canceling actor via the Actions UI timeline for job 82209375330 ("canceled because…"); or temporarily add run_id to the extra concurrency group key on a branch and observe whether overlapping schedule runs stop cancelling each other.
• [LOW] Pure runner starvation + an external/manual cancel. The extra-a tier is lowest priority; it never won an MI325 1-GPU runner in 85 min while higher-priority stage jobs consumed the pool.
  • Supporting: 0 steps executed; long queue; stage-c jobs in the same run kept running until 23:01Z.
  • Disconfirming: starvation alone would not cancel a job (it would keep waiting up to the 6h default); something issued an explicit cancel.
• [SPECULATION] Fast-fail cascade from the stage-c failure at 20:11:06Z (6 min before the cancel). Weak: for schedule, fast-fail wait jobs are skipped and stages run with --continue-on-error, so stage-c should not cascade-cancel the extra workflow.

In-flight Fix Check

• In-flight fix: ❌ none found — no open PR matches the extra-workflow concurrency/cancellation issue. Searched repo:sgl-project/sglang is:pr is:open with keywords extra-a-test … concurrency (2 unrelated hits) and amd extra cancel (8 unrelated hits); the pr-test-amd-extra query returned only generic AMD-CI PRs.

Suggested Triage Steps

• Confirm the cancellation actor in the Actions UI timeline for job 82209375330 (it names the run that evicted it) to settle concurrency-vs-other.
• If concurrency is confirmed: consider scoping the extra workflow's concurrency group per-run for schedule events (e.g. include github.run_id for schedule, as the parent pr-test-amd.yml:116 already does) so overlapping scheduled runs don't cancel each other — propose on a branch, do not apply to this read-only workspace.
• Independently, consider whether pr-test-amd-extra-finish should treat cancelled dependencies distinctly from failure so infra cancellations don't masquerade as test failures in the scheduled signal.
• Address the AMD MI325 1-GPU pool backlog (3 schedule runs queued at once) — the extra tier never gets a runner.

Status (factual)

• Persistence: pr-test-amd-extra-finish has reported failure on every scheduled pr-test-amd.yml run for at least the last ~2 days (Jun 17–18), each time because an extra-a job was cancelled.
• Scope: the call-pr-test-amd-extra sub-workflow only (extra-a small + large + finish). Other stages of the parent run ran independently; stage-c-test-large-8-gpu has its own separate, real failures (out of scope for this job).
• Impact: the schedule-only AMD "extra" tier (1-GPU lora/spec/quant e2e) produces no usable signal — green never happens because the jobs are cancelled before/while running.
• In-flight fix: ❌ none found.

call-pr-test-amd-extra / extra-a-test-1-gpu-small-amd (linux-mi325-1gpu-sglang) — failed step(s): N/A

Commit Info

• sglang: 61a8b42c
• aiter: N/A — the job log blob does not exist (BlobNotFound) and the job recorded zero steps, so no [CI-AITER-CHECK] markers are retrievable.

Failed Tests

Test File	Test Function	Error	Log
N/A — no test executed	N/A	Job cancelled before/while running; no test results emitted	Job

No test-level failure exists. The job extra-a-test-1-gpu-small-amd (82209375317) has conclusion = cancelled, 0 steps recorded, and its log blob returns BlobNotFound.

Failure Summary

This was not a test failure — the job was cancelled (conclusion=cancelled) 85 minutes after it started, with no steps logged and no log artifact. Its sister job extra-a-test-1-gpu-large-amd (82209375330) was cancelled at the identical second (2026-06-18T20:17:23Z). The only thing that "failed" in this job group is the downstream fan-in aggregator pr-test-amd-extra-finish (82225267855), which exits 1 by design whenever a dependent is cancelled — a cascade, not an independent fault.

Regression Status

Infrastructure / orchestration cancellation — intermittent, not a code regression.

Recent history of job extra-a-test-1-gpu-small-amd (pr-test-amd.yml, event=schedule, branch=main, completed runs only):

Date (UTC)	Run	Job Status	Duration	Notes
Jun 18 18:52	27782126506	⏹️ Cancelled	85 min	this run; sister large job cancelled same second
Jun 18 13:15	27762119155	✅ Success	~15 min	—
Jun 18 07:46	27744699827	✅ Success	~12 min	—
Jun 18 00:59	27729699748	✅ Success	~13 min	—
Jun 17 18:53	27712340726	⏹️ Cancelled	29 min	evening run also cancelled
Jun 17 13:21	27692029884	✅ Success	~9 min	—

Pattern: the evening (~18:5x UTC) scheduled run was cancelled on both Jun 17 and Jun 18, while the 00:xx / 07:xx / 13:xx slots succeeded. The two cancellations have different durations (29 min vs 85 min) and different wall-clock cancel times (19:22:28 vs 20:17:23), so this is not a fixed timeout.

Failure Cluster

"AMD extra-a reusable-workflow jobs cancelled mid-run (orchestration, not test)"

Facts (verified observations, no interpretation)

• conclusion = cancelled, status = completed, 0 steps in the API job object for 82209375317.
• Log fetch returns <Error><Code>BlobNotFound</Code> — no log was ever persisted.
• Both extra-a jobs cancelled at exactly 2026-06-18T20:17:23Z (started 18:52:07Z).
• call-pr-test-amd-extra is invoked via workflow_call from pr-test-amd.yml:208-217 with continue_on_error: true, so genuine test failures would not fail or cancel the job.
• The extra workflow's Run test steps carry different step timeouts (timeout-minutes: 45 for small at pr-test-amd-extra.yml:145, 60 for large at line 176). A step-timeout produces conclusion=failure, not cancelled, and cannot cancel two jobs with different timeouts at the same instant.
• pr-test-amd-extra-finish log: extra-a-test-1-gpu-small-amd: cancelled → The above jobs failed. → exit code 1 (aggregator at pr-test-amd-extra.yml:184-207).
• The parent run 27782126506 was still in progress at snapshot time (other jobs ran until 23:01:27Z), so the whole run was not cancelled — only the extra-a reusable-workflow jobs were.
• The two overlapping scheduled runs (27762119155, 27729699748) had their extra jobs finish at 17:55 and 01:38 respectively — neither was running at 20:17.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [MEDIUM] Child reusable-workflow concurrency preemption. pr-test-amd-extra.yml:83-85 sets concurrency.group = pr-test-amd-extra-${event_name}-${head_ref||ref_name}-${inputs.ref||'all'} with cancel-in-progress: ${{ github.event_name != 'workflow_call' }}. For a schedule-context call this resolves to the shared, non-run-id-unique group pr-test-amd-extra-schedule-main-all with cancel-in-progress = true. Unlike the parent (pr-test-amd.yml:116 keys schedule runs on full-${{ github.run_id }}, making them unique), the child group is shared across any schedule-context entrant, so a second entrant would cancel these in-progress jobs at one instant — which matches the identical-second cancel of both jobs.
  • Supporting: both jobs cancelled at the exact same second; child group is not run-id-scoped; cancel-in-progress evaluates true for schedule.
  • Disconfirming: I could not identify the concurrent entrant — the two overlapping scheduled runs' extra jobs had already finished (17:55, 01:38), and no new schedule-event pr-test-amd.yml run exists between 18:52 and the next morning. A re-run (run_attempt 2) of an older scheduled run could re-enter the group but was not visible in the queryable run list.
  • How to verify: check the GitHub Actions concurrency timeline / audit log for group pr-test-amd-extra-schedule-main-all around 2026-06-18T20:17:23Z; look for re-runs (run_attempt > 1) of any scheduled pr-test-amd.yml run that day.
• [LOW] Self-hosted AMD runner reclamation / loss. Both jobs ran on the linux-mi325-1gpu-sglang pool; if the underlying runner host(s) were reclaimed, jobs surface as cancelled with no log.
  • Supporting: missing log blob + zero steps is consistent with a runner that never flushed logs.
  • Disconfirming: two independent runners dying at the same second is improbable; an orchestration cancel explains the synchrony better.
  • How to verify: inspect the AMD runner fleet logs for linux-mi325-1gpu-sglang at 20:17Z.
• [LOW] Per-step timeout (45 min) on the small job's Run test. Setup is untimed, so the test step could start late.
  • Disconfirming: step timeouts yield conclusion=failure, not cancelled; and they cannot cancel the small (45 min) and large (60 min) jobs at the identical second. Effectively ruled out.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs with keywords pr-test-amd-extra concurrency and amd extra cancel; all hits were unrelated AMD kernel/Dockerfile PRs (#28173, #28584, etc.). No PR addresses the extra-workflow concurrency cancellation.

Suggested Triage Steps

• Pull the Actions concurrency/audit timeline for group pr-test-amd-extra-schedule-main-all at 2026-06-18T20:17:23Z to find what entered the group and triggered cancel-in-progress.
• Check for re-runs (run_attempt > 1) of any scheduled pr-test-amd.yml run on Jun 17–18; a re-run re-enters the shared child group and would cancel live extra jobs.
• If the concurrency footgun is confirmed, consider making the child group run-id-scoped for schedule context (mirror the parent's full-${{ github.run_id }} keying at pr-test-amd.yml:116) so overlapping scheduled invocations don't cancel each other. (Hypothetical fix direction — maintainer decides.)
• If instead the runner-loss hypothesis holds, no code change is warranted; treat as infra flake and rerun.

Status (factual, no priority assignment)

• Persistence: Intermittent — the evening (~18:5x UTC) scheduled run was cancelled on Jun 17 and Jun 18; all other Jun 17–18 scheduled slots for this job succeeded.
• Scope: Both extra-a jobs (1-gpu-small + 1-gpu-large) of the call-pr-test-amd-extra reusable workflow, cancelled simultaneously; cascades to pr-test-amd-extra-finish. This is the opt-in extra suite — it does not gate the base AMD CI (pr-test-amd-finish does not depend on it, per pr-test-amd.yml:206-207).
• Blocked work: No real test signal from the AMD extra-a suite for this scheduled run; no correctness signal lost (no test actually failed).
• In-flight fix: ❌ none found in open PRs.

multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 1) — failed step(s): Run diffusion server tests (2-GPU)

Commit Info

• sglang: 61a8b42c
• aiter: 7d604afe5 (v0.1.14rc1.dev240+g7d604afe5; source: dev/patched image preserved, no AITER_COMMIT_OVERRIDE)

Failed Tests

Test File	Test Function	Error	Log
python/sglang/multimodal_gen/test/server/test_server_2_gpu.py	TestDiffusionServerTwoGpu::test_diffusion_generation[flux2_modelopt_fp8_tp2_t2i]	openai.InternalServerError: Internal Server Error (server-side RuntimeError: CUDA error: HIPBLAS_STATUS_NOT_SUPPORTED in torch._scaled_mm)	Log
python/sglang/multimodal_gen/test/server/test_server_2_gpu.py	TestDiffusionServerTwoGpu::test_diffusion_generation[ideogram4_fp8_tp2_t2i]	huggingface_hub.errors.GatedRepoError: 403 Client Error (gated repo, setup error)	Log

Failure Summary

The "Run diffusion server tests (2-GPU)" step ended 1 failed, 9 passed, 1 skipped, 12 deselected, 1 error. flux2_modelopt_fp8_tp2_t2i failed because the diffusion server returned HTTP 500 — the server's FP8 GEMM (torch._scaled_mm in python/sglang/srt/layers/quantization/fp8_utils.py:1681) raised HIPBLAS_STATUS_NOT_SUPPORTED during both warmup and the generation request. ideogram4_fp8_tp2_t2i errored at setup because the model repo ideogram-ai/ideogram-4-fp8 is gated and the CI HF account is not authorized (403).

Regression Status

flux2_modelopt_fp8_tp2_t2i: Never-passed (newly added test) — added to TWO_GPU_CASES by #28546 (3b61dc32, merged 2026-06-18 06:38 UTC). It failed on its very first appearance and every completed run since. The "10 passed" on 06-17 reflects the test not yet existing, not a green run for this test.

ideogram4_fp8_tp2_t2i: Known recurring failure — gated-repo 403 present in every queried completed run, including before #28546.

Recent history of test_server_2_gpu.py in job multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 1) (branch main, event schedule, completed runs only):

Date	Run	flux2_modelopt_fp8_tp2_t2i	ideogram4_fp8_tp2_t2i	Summary
06-18 18:52	27782126506	❌ Failed (HIPBLAS)	❌ Error (403)	1 failed, 9 passed, 1 skipped, 1 error
06-18 07:46	27744699827	❌ Failed (HIPBLAS)	❌ Error (403)	1 failed, 9 passed, 1 skipped, 1 error
06-17 18:53	27712340726	— (test not present)	❌ Error (403)	10 passed, 1 error
06-17 13:21	27692029884	— (test not present)	❌ Error (403)	10 passed, 1 error
06-17 07:57	27674400949	— (test not present)	❌ Error (403)	10 passed, 1 error

(Newer runs 27762119155, 27729699748 are still queued → excluded. The Summary column is for context; the verdict is per-test.)

Failure Cluster

• flux2: HIPBLAS_STATUS_NOT_SUPPORTED in FP8 torch._scaled_mm (ModelOpt FP8 diffusion on AMD)
• ideogram4: Gated HF repo 403 at model download (CI access)

Facts (verified observations)

• flux2 server-side stack trace (verbatim): apply_fp8_linear → output = torch._scaled_mm( → RuntimeError: CUDA error: HIPBLAS_STATUS_NOT_SUPPORTED when calling \HIPBLAS_STATUS_NOT_SUPPORTED`atpython/sglang/srt/layers/quantization/fp8_utils.py:1681, reached via flux_2.py:179 _get_qkv_projections→linear.py:427→modelopt_quant.py:425 apply_fp8_linear`.
• The same HIPBLAS error fires first during synthetic warmup (server warmup req (768x768, 2/5 steps) processing failed) and then on the actual request, which surfaces to the client as openai.InternalServerError: Internal Server Error.
• #28546 diff adds the flux2_modelopt_fp8_tp2_t2i DiffusionTestCase to TWO_GPU_CASES (the adjacent ideogram4_fp8_tp2_t2i block is unchanged context); it also edits FP8 per-tensor scale loading in multimodal_gen/runtime/layers/linear.py. Merge time 2026-06-18 06:38 UTC sits inside window [7cead0fb8f..59001267c3].
• ideogram4: GatedRepoError: 403 ... Access to model ideogram-ai/ideogram-4-fp8 is restricted and you are not in the authorized list — both workers die, scheduler exits code 1, server "exited early". Identical 403 in all 06-17/06-18 runs.
• AITER unchanged across all these runs (v0.1.14rc1.dev240+g7d604afe5, dev image), so aiter is not implicated.

Hypothesised Causes (with confidence)

flux2_modelopt_fp8_tp2_t2i

• [HIGH] The FP8 torch._scaled_mm path used by ModelOpt-FP8 flux2 is unsupported by the hipBLASLt/ROCm build on this MI325 runner; the test was newly enabled on AMD (2-GPU TP2) by #28546 where this kernel path does not work.
  • Supporting: error is raised directly inside torch._scaled_mm (fp8_utils.py:1681), a GEMM-level "not supported" status, not a weight/shape error; fails identically at warmup and request; test never passed on AMD; NVIDIA is unaffected by this AMD-only code path.
  • Disconfirming: the same commit also modified FP8 per-tensor scale loading in linear.py — a bad scale could in principle produce odd inputs, but a malformed scale would surface as a shape/dtype/assert error, not HIPBLAS_STATUS_NOT_SUPPORTED from the GEMM itself, so the scale-loading edit is unlikely to be the proximate cause.
  • How to verify: run sglang serve --model-path black-forest-labs/FLUX.2-dev --model-type diffusion --num-gpus 2 --tp-size 2 --transformer-path lmsys/flux2-dev-modelopt-fp8-sglang-transformer on an MI325 container and confirm torch._scaled_mm with the flux2 FP8 shapes raises HIPBLAS_STATUS_NOT_SUPPORTED in isolation; if so, the test needs an AMD guard or a ROCm-supported FP8 GEMM fallback.

ideogram4_fp8_tp2_t2i

• [FACT] Pre-existing CI access/infrastructure failure: ideogram-ai/ideogram-4-fp8 is a gated HF repo and the CI HF token lacks access (403). Unrelated to any code change.
  • Disconfirming: (none — error is a verbatim authorization denial).
  • How to verify: request access / add an authorized HF token to CI secrets, or skip this case on runners without access.

In-flight Fix Check

• flux2 HIPBLAS: In-flight fix: ❌ none found — HIPBLAS_STATUS_NOT_SUPPORTED search returns 0 open PRs; flux2+fp8 returns only #27702 (ROCm BF16 attention + MXFP4 compilation), which does not target the ModelOpt FP8 torch._scaled_mm path. (queried search/issues for flux2 fp8, HIPBLAS_STATUS_NOT_SUPPORTED, diffusion fp8 amd.)
• ideogram4 403: In-flight fix: ❌ none found — access-grant / CI-secret task, not a code PR.

Suggested Triage Steps

• Reproduce the flux2 FP8 GEMM on MI325 in isolation (command above); confirm whether torch._scaled_mm with these shapes is supported by the installed hipBLASLt.
• If unsupported, decide between: gating flux2_modelopt_fp8_tp2_t2i off on AMD (as flux1 modelopt cases may be), or routing the ModelOpt FP8 path to a ROCm-supported GEMM/fallback in fp8_utils.py:apply_fp8_linear — coordinate with #28546 author (Mick).
• For ideogram4: request authorization to ideogram-ai/ideogram-4-fp8 (or provision an authorized HF token), or skip the case on unauthorized runners so the persistent 403 stops masking other signal.

Status

• Persistence: flux2 — failing every completed run since it was introduced 2026-06-18 06:38 UTC (2/2 completed runs that include it). ideogram4 — 403 in every queried completed run (≥06-17 07:57 onward).
• Scope: job multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 1); AMD-only (NVIDIA unaffected by the HIPBLAS path).
• Blocked work: blocks the 2-GPU AMD diffusion CI signal for this shard (FLUX.2 ModelOpt FP8 TP2 coverage); ideogram4 coverage is unobtainable on this runner until access is granted.
• In-flight fix: ❌ none for either; both need a new PR (AMD FP8 guard/fallback) or a CI-access change (ideogram4).

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-06-20 00:03 UTC)

[amd-bot]

pr-test-amd.yml — Per-Job Analysis (continued 3/7)

multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 2) — failed step(s): Run diffusion server tests (1-GPU)

Commit Info

• sglang: 61a8b42c
• aiter: 7d604afe5 — v0.1.14rc1.dev240+g7d604afe5 (source: dev/patched CI image preserved; matches Dockerfile default AITER_COMMIT=7d604afe5...)

Failed Tests

Test File	Test Function	Error	Log
python/sglang/multimodal_gen/test/server/test_server_1_gpu.py	TestDiffusionServerOneGpu::test_diffusion_generation[qwen_image_edit_2509_ti2i]	openai.BadRequestError: 400 - Failed to process image source: Failed to download image from URL https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen-Image/edit2509/edit2509_1.jpg	Log
python/sglang/multimodal_gen/test/server/test_server_1_gpu.py	TestDiffusionServerOneGpu::test_diffusion_generation[qwen_image_t2i_cache_dit_scm_config_diffusers_1gpu] (ERROR at setup)	RuntimeError: Server exited early (code 1) → httpx.HTTPStatusError: 429 Too Many Requests for huggingface.co/api/models/Qwen/Qwen-Image → ValueError: Could not find model at Qwen/Qwen-Image	Log

Failure Summary

The job ran 10 diffusion server cases on shard 2 (1 failed, 7 passed, 1 skipped, 1 error). Both non-passing cases failed at the network/data layer, not in sglang compute: qwen_image_edit_2509_ti2i failed because the test input image at qianwen-res.oss-cn-beijing.aliyuncs.com (Aliyun OSS) could not be downloaded after 3 retries, so the server returned 400 Bad Request; qwen_image_t2i_cache_dit_scm_config_diffusers_1gpu errored at setup because the Qwen/Qwen-Image model snapshot download from HF Hub was rate-limited with HTTP 429 ("you hit the quota of 3000 api requests per 5 minutes period... current: team"), crashing the server during init.

Regression Status

Infrastructure issue (external network / HF rate-limit) — not a code regression.

Recent history of test_server_1_gpu.py (test_diffusion_generation shard 2) in job multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 2) (branch main, event=schedule, completed runs only):

Date	Run	Job	Test File Status	Failed Function	Error
Jun 18 18:52	27782126506	shard 2	❌ Failed	[qwen_image_edit_2509_ti2i] + [qwen_image_t2i_cache_dit_scm...]	image dl 400 / HF 429
Jun 18 07:46	27744699827	shard 2	❌ Failed	most cases (setup)	Scheduler port 5555 unavailable (different infra cause)

Note: the current run 27782126506 is overall queued (other jobs still pending) but this job completed and failed. The Jun 18 07:46 shard-2 failure was a different infra cause (port conflict), so no consistent code-level regression signature exists for this test file; the symptom changes run-to-run, characteristic of environment/network flakiness.

Failure Cluster

External data/model fetch failure during diffusion server test (HF Hub 429 rate-limit + Aliyun OSS image download failure)

Facts (verified observations)

• qwen_image_edit_2509_ti2i: log lines Retrying image download (1/3) and (2/3) for https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen-Image/edit2509/edit2509_1.jpg, then POST /v1/images/edits → 400 Bad Request; pytest raises openai.BadRequestError: Error code: 400 - {'detail': 'Failed to process image source: Failed to download image from URL ...edit2509_1.jpg: '} (raw log lines 5381–5383, 6696). The server itself fired up ("The server is fired up and ready to roll!") — failure is request-time input fetch, not model compute.
• qwen_image_t2i_cache_dit_scm_config_diffusers_1gpu: server worker crashed in maybe_download_model → snapshot_download with httpx.HTTPStatusError: 429 Too Many Requests for huggingface.co/api/models/Qwen/Qwen-Image; HF message: "429 ... you hit the quota of 3000 api requests per 5 minutes period ... current: team" → LocalEntryNotFoundError → ValueError: Could not find model at Qwen/Qwen-Image → parent EOFError (raw log lines 6527–6716). Stack frames are all in huggingface_hub / hf_diffusers_utils.py:808,943,874 download paths — no kernel or model-forward code involved.
• aiter version unchanged (Dockerfile default dev build, no override) — not a variable here.
• Both errors are network/external-service responses (Aliyun OSS, HF Hub), independent of GPU/kernel execution.

Hypothesised Causes (with confidence)

• [FACT] qwen_image_t2i_cache_dit_scm... errored due to HF Hub 429 API rate-limit during model download. Directly observable: HF returns the 429 + quota message verbatim in the log.
  • Supporting: full traceback terminates in huggingface_hub HTTP layer; explicit "3000 api requests per 5 minutes" quota message.
  • Disconfirming: (none found) — the error is unambiguous and external.
  • How to verify: pre-cache Qwen/Qwen-Image in the CI image or set HF_HUB_OFFLINE=1; re-run shard 2.
• [FACT] qwen_image_edit_2509_ti2i failed because the test input image on Aliyun OSS could not be downloaded (3 retries then 400).
  • Supporting: retry lines + 400 Bad Request + openai.BadRequestError naming the exact edit2509_1.jpg URL.
  • Disconfirming: (none found) — server started successfully; only the external image fetch failed.
  • How to verify: curl the OSS URL from a CI runner; mirror the test input asset into the repo / HF dataset instead of fetching from Aliyun at test time.
• [LOW] Any sglang/aiter code involvement.
  • Supporting: none.
  • Disconfirming: stack traces are entirely in HTTP/download libraries; the prior run's same-shard failure had an unrelated infra cause (port conflict). No code path in the diff is implicated.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for huggingface rate limit (6 hits, none relevant to diffusion CI model caching). These are infra/environment problems with no expected source-code fix; the appropriate remediation is CI-side (model pre-caching / asset mirroring / HF token quota), not a PR.

Suggested Triage Steps

• Treat as infrastructure flake: re-run shard 2 of multimodal-gen-test-1-gpu-amd — both errors are transient external-fetch failures.
• For the HF 429: pre-populate Qwen/Qwen-Image (and other diffusion models) in the CI docker image or shared /sgl-data/hf-cache, or run with HF_HUB_OFFLINE=1 so PR CI does not hit the live HF API; consider a higher HF org quota since the "team" plan's 3000 req/5min is being exhausted.
• For the Aliyun OSS image: mirror edit2509_1.jpg (and similar *_ti2i/*_edit inputs) into the repo test assets or an HF dataset, and point test_server_utils.py:generate_image_edit_url at the local/cached copy to remove the runtime dependency on qianwen-res.oss-cn-beijing.aliyuncs.com.
• If failures persist after caching, re-investigate — but current evidence shows no sglang/aiter code involvement.

Status

• Persistence: shard 2 of this job failed in the last 2 completed scheduled runs (Jun 18 07:46 and Jun 18 18:52), but with different infra root causes each time (port conflict, then HF 429 + image dl) — intermittent environment flakiness, not a sustained single failure.
• Scope: 1 job / 1 shard (partition 2). Sister shards 0, 1, 3 passed in the prior completed run, indicating per-shard transient network luck rather than a systemic break.
• Blocked work: diffusion (multimodal-gen) AMD CI signal on shard 2 only.
• In-flight fix: ❌ none found — remediation is CI-infrastructure (model/asset caching), not a code PR.

stage-c-test-large-8-gpu-amd (linux-mi325-8gpu-sglang, 2) — failed step(s): Test RCCL multi-GPU communication

Commit Info

• sglang: 61a8b42c
• aiter: 7d604afe5 (source: dev image preserved — v0.1.14rc1.dev240+g7d604afe5, Dev/patched version detected → skipping rebuild)

Failed Tests

Test File	Test Function	Error	Log
scripts/ci/amd/test_rccl_multi_gpu.py	test_rccl_allreduce (8-GPU torchrun sanity check, not pytest)	ChildFailedError: all ranks exit -11 / Signal 11 (SIGSEGV)	Log

Failure Summary

The job failed at the RCCL multi-GPU communication sanity-check step (torchrun --nproc_per_node=8 scripts/ci/amd/test_rccl_multi_gpu.py), which runs before the actual large-test suite. Ranks 0–6 printed "Initialized successfully" + device/memory, rank 7 never finished init, and then every rank died with a native SIGSEGV (exit -11) — before any rank reached the Before allreduce print. No pytest test in the stage-c suite was ever reached.

Regression Status

New failure for this shard / likely infrastructure (node-specific) crash.

Recent history of scripts/ci/amd/test_rccl_multi_gpu.py step in job stage-c-test-large-8-gpu-amd (linux-mi325-8gpu-sglang, 2) (branch main, event schedule, completed runs only):

Date	Run	Shard-2 Job Status	Notes
Jun 18	27782126506	❌ Failed	RCCL SIGSEGV (this run), runner …ww75q
Jun 18	27744699827	✅ Passed	—
Jun 17	27712340726	✅ Passed	—
Jun 17	27692029884	✅ Passed	—
Jun 17	27674400949	✅ Passed	—
Jun 17	27658537520	✅ Passed	—

Shard 2 passed its RCCL check in all 5 prior completed scheduled runs. First observed RCCL failure for this shard: 2026-06-18 (this run).

Failure Cluster

RCCL multi-GPU sanity-check SIGSEGV during init/first collective (8-GPU torchrun crash)

Facts (verified observations, no interpretation)

• Crash is a native SIGSEGV (signal 11), not a Python exception. torch.distributed.elastic ... ChildFailedError, root-cause rank 0 exitcode -11 (pid 2809); ranks 1–5 also -11; ranks 6–7 received SIGTERM during torchrun cleanup (log lines 4101–4166).
• Ranks 0–6 printed Initialized successfully + device + memory; no rank printed Before allreduce → crash occurred after init_process_group but before/at the first trivial torch.ones(1000, device=device) / all_reduce.
• Despite NCCL_DEBUG=INFO RCCL_DEBUG=INFO, no RCCL INFO lines were emitted before the crash.
• In the same run (same sglang SHA), the other failing shards failed in completely different ways on different runners: shard 0 …9f5wt = test hang (repeated py-spy dump … Failed to get stack traces, DP2 TP2 EP2); shard 3 …9j2kx = cmake build failure (cmake --build . -j 256 exit 1). Shard 1 …nsn44 passed.
• The RCCL sanity script is byte-identical across shards/runs; it is not affected by the shard partition index.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [MEDIUM] Node/runner-level RCCL/GPU fault on runner linux-mi325-8gpu-sglang-…ww75q (transient hardware / driver / RCCL init crash), not a code regression.
  • Supporting: native SIGSEGV during a trivial tensor alloc before any collective math; identical script passed on this shard in the last 5 runs; the three failing shards crashed at three unrelated stages on three different runners (RCCL crash vs hang vs build failure) — inconsistent with a single shared code defect at SHA 61a8b42c.
  • Disconfirming: rank 7 never completing init could hint at a collective/rendezvous bug rather than pure hardware; cannot fully rule out an RCCL/torch image issue without a clean re-run on a different node.
  • How to verify: re-run shard 2 only; if it passes on a different runner, confirms node-specific. If it reproduces across runners at this SHA, escalate to a code/image hypothesis.
• [LOW] sglang code regression at 61a8b42c.
  • Supporting: failure appears on this SHA.
  • Disconfirming: HEAD commit is docs-only (#28668, MMMU-Pro benchmark card); the RCCL sanity check exercises no sglang code (only torch.distributed + RCCL); shard 1 on the same SHA passed its RCCL check. Strong disconfirmation.
• [LOW] aiter dev build v0.1.14rc1.dev240+g7d604afe5.
  • Supporting: AMD-only path.
  • Disconfirming: same aiter image was in the 5 prior passing runs; the RCCL script does not import aiter.

In-flight Fix Check

• In-flight fix: ❌ none found for the RCCL sanity-check SIGSEGV. Search repo:sgl-project/sglang is:pr is:open RCCL multi gpu returned 3 unrelated PRs (#28173 HIP CUDA-graph, #28605 shadow-runner CI test, #17628 torch-compile). None address this crash.

Suggested Triage Steps

• Re-run shard 2 only of run 27782126506; if it passes (likely on a different runner), classify as transient node-specific infra and cordon/inspect runner …ww75q.
• If the RCCL SIGSEGV reproduces across multiple runners at SHA 61a8b42c, inspect the CI image's torch (2.9.0a0+git7bcbafe) / RCCL / ROCm versions vs the last passing run image for a drift, since the script itself is unchanged.
• Track shards 0 and 3 separately — they are distinct failures (test hang and cmake build error), not part of this RCCL cluster, and shard 3 has been failing in every recent run.

Status (factual, no priority assignment)

• Persistence: First RCCL failure on shard 2 in the queryable window (2026-06-18); shard 2 RCCL check passed in the prior 5 completed scheduled runs (Jun 17–18).
• Scope: 1 shard (shard 2) on 1 runner (…ww75q); the RCCL sanity-check step. Other failing shards in the same run failed for unrelated reasons on different runners.
• Blocked work: stage-c large 8-GPU test suite never executed on shard 2 (crashed at the pre-test RCCL gate).
• In-flight fix: ❌ none found — needs re-run to confirm node-specific infra, else image/version investigation.

stage-c-test-large-8-gpu-amd (linux-mi325-8gpu-sglang, 3) — failed step(s): Install dependencies

Commit Info

• sglang: 61a8b42c
• aiter: N/A — job failed during the Install dependencies step before any [CI-AITER-CHECK] marker was emitted (0 markers in log)
• mori: bf99bdf1 (reinstalled with MORI_GPU_ARCHS=gfx942)

Failed Tests

Test File	Test Function	Error	Log
N/A — build/setup failure (no test phase reached)	N/A	error while loading shared libraries: /lib/x86_64-linux-gnu/libabsl_strings_internal.so.20210324: invalid ELF header → ninja build stopped → setup.py build_ext exit 1	Log

Failure Summary

This is not a test failure. The job failed in step 5 (Install dependencies → scripts/ci/amd/amd_ci_install_dependency.sh) while reinstalling the MORI dependency. During the gtest_discover_tests post-link step, CMake tried to run the freshly built test_client_registry_external_kv executable to enumerate its tests, but the executable could not load the system shared library libabsl_strings_internal.so.20210324 ("invalid ELF header"), returning exit 127. That made ninja stop the build (returned non-zero exit status 1), MORI's setup.py raised CalledProcessError, and the job exited 1. The test suite for this shard never ran.

Regression Status

Infrastructure issue (new failure mode for this job) — NOT a code regression.

Recent history of job stage-c-test-large-8-gpu-amd (linux-mi325-8gpu-sglang, 3) (branch main, event schedule, completed runs only):

Date	Run	Job	MORI Build / Install	Failure point
Jun 18 18:52	27782126506	job	❌ Failed (this run) — libabsl…invalid ELF header	Install dependencies (MORI build)
Jun 18 07:46	27744699827	job	✅ Built OK (same MORI SHA)	Later test test_aiter_allreduce_fusion_amd.py (different failure)
Jun 17 18:53	27712340726	job	✅ Built OK (same MORI SHA)	Later test test_aiter_allreduce_fusion_amd.py (different failure)

The MORI commit is identical (bf99bdf1) across all three runs, and the docker image tag is the same (rocm/sgl-dev:v0.5.13.post1-rocm700-mi30x-20260618). MORI built and installed cleanly in the two prior completed runs, so the invalid ELF header is a fresh, transient condition specific to this run/runner — not introduced by the MORI source or by sglang code.

Failure Cluster

MORI build failure — libabsl_strings_internal.so invalid ELF header during gtest discovery

Facts (verified observations)

• [CI-AITER-CHECK] markers in log: 0 (install aborted before the aiter check ran).
• Failing step: step 5, Install dependencies (scripts/ci/amd/amd_ci_install_dependency.sh), MORI reinstall block ([MORI] Reinstalling MORI bf99bdf1… (MORI_GPU_ARCHS=gfx942), log line 2697).
• Exact error (log line 3999, verbatim): /sgl-workspace/mori/build/src/umbp/tests/test_client_registry_external_kv: error while loading shared libraries: /lib/x86_64-linux-gnu/libabsl_strings_internal.so.20210324: invalid ELF header.
• CMake GoogleTestAddTests.cmake:381 reported Result: 127 (executable failed to start), then ninja: build stopped: subcommand failed. (line 4064) and subprocess.CalledProcessError … returned non-zero exit status 1 (line 4097) from mori/setup.py:434.
• The error is on a system library (/lib/x86_64-linux-gnu/libabsl_strings_internal.so.20210324), not a MORI- or sglang-built artifact.
• Same MORI SHA + same docker image built successfully in runs 27744699827 and 27712340726.

Hypothesised Causes (with confidence)

• [MEDIUM] Corrupt / truncated system shared library on this specific runner — libabsl_strings_internal.so.20210324 has an invalid ELF header.
  • Supporting: error is literally invalid ELF header on a /lib/x86_64-linux-gnu system .so; the linked executable was produced fine (link step succeeded) — only loading the absl runtime lib failed. The identical image+MORI SHA loaded this lib successfully in the two preceding completed runs, so the file content is the variable, not the build inputs.
  • Disconfirming: cannot confirm filesystem corruption directly from the log (no md5sum/file output); a parallel-build race (-j 256) overwriting/half-writing a vendored absl is an alternative explanation, though absl here is the system lib, not a MORI 3rdparty.
  • How to verify: on the affected runner/container, run file /lib/x86_64-linux-gnu/libabsl_strings_internal.so.20210324 and readelf -h on it; re-run the job on a fresh container — if it builds, this confirms a transient/host-local corruption.
• [LOW] Bad/partial layer in the pulled docker image (note: local registry pull failed at line 1252 and fell back to the public registry).
  • Supporting: a registry fallback occurred this run (Local docker registry pull failed; falling back to public registry).
  • Disconfirming: the image tag is identical to the prior passing runs and those runs built absl fine; a tag-pinned image is content-addressed, so the absl lib should be byte-identical.
  • How to verify: compare the absl .so checksum across a passing and failing container of the same tag.

No hypothesis reaches HIGH; this is most consistent with a transient infrastructure/runner condition rather than any sglang or MORI code change.

In-flight Fix Check

• In-flight fix: ❌ none found — open-PR search for mori install and mori surfaced only feature/PD/MoE PRs (e.g. #27093, #22646); none addresses an absl/invalid ELF header MORI build break. (API: search/issues?q=repo:sgl-project/sglang+is:pr+is:open+mori.) Note an existing branch origin/bingxche/mori-ci-reinstall-26922 touching MORI CI reinstall, but no open PR matches this symptom.

Suggested Triage Steps

• Re-run the job on a fresh container/runner; if MORI builds and the shard proceeds, treat this as a transient infra flake and close it out.
• On the affected MI325 runner, inspect /lib/x86_64-linux-gnu/libabsl_strings_internal.so.20210324 with file / readelf -h to confirm whether the ELF header is corrupt; if so, rebuild/refresh the container image layer providing absl.
• If the failure recurs across runners, compare the absl .so checksum between a passing image pull and the public-registry fallback image to rule out a bad mirrored layer (the local-registry pull failed and fell back this run).
• Independent of this infra issue: note that when MORI builds successfully, this shard has been failing on test/registered/ops/test_aiter_allreduce_fusion_amd.py ("fused residual output diverges from unfused path") in the two prior completed runs — that is a separate, pre-existing failure to track once the build is unblocked.

Status

• Persistence: First occurrence of this invalid ELF header MORI build failure for this job in the queryable scheduled-run window (Jun 17–18); the two prior completed runs built MORI successfully.
• Scope: Observed in this one job/shard at the dependency-install stage; not a sglang code path.
• Blocked work: This shard's stage-c-test-large-8-gpu-amd test suite did not run, so its CI signal for commit 61a8b42c is missing.
• In-flight fix: ❌ none found — needs infra investigation, not a code PR.

multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 1) — failed step(s): Post Checkout code, Run diffusion server tests (1-GPU), Upload diffusion failure artifacts

Commit Info

• sglang: 97e3b8998d
• aiter: N/A — job log blob is expired/deleted (BlobNotFound from the logs API), so no [CI-AITER-CHECK] markers are recoverable.

Failed Tests

Test File	Test Function	Error	Log
N/A — not a test failure	N/A	Step "Run diffusion server tests (1-GPU)" cancelled mid-run (runner-loss signature)	Job

No specific test file/function can be pinned: the per-job log is no longer available (the API returns BlobNotFound), and the failing step never reached completion to emit pytest summary lines.

Failure Summary

The job's main step 9 (Run diffusion server tests (1-GPU), partition 1 of 4) started at 15:27:51Z and was cancelled at 16:13:07Z (~45 min), well short of its timeout-minutes: 90. The subsequent Upload diffusion failure artifacts step — which is guarded by if: always() — was also cancelled rather than executed. This is not a normal test failure (a test failure would still run the always() upload step); it is the signature of the runner being forcibly terminated mid-test.

Regression Status

Known recurring failure — infrastructure (runner loss), not a code regression.

Recent history of multimodal-gen-test-1-gpu-amd partition 1 (same branch main, event=schedule, completed runs only):

Date (UTC)	Run	Job	Status	Detail
Jun 18 13:15	27762119155	part 1	❌ Cancelled mid-test	step 9 cancelled @45min, always() upload skipped
Jun 18 07:46	27744699827	part 1	✅ Passed	(part 2 failed instead)
Jun 17 18:53	27712340726	part 1	✅ Passed	—
Jun 17 13:21	27692029884	part 1	✅ Passed	—
Jun 17 07:57	27674400949	part 1	✅ Passed	—
Jun 17 01:00	27658537520	part 1	❌ Cancelled mid-test	identical signature: step 9 cancelled, always() upload skipped

Partition 1 passed 4 of the last 5 completed runs. Both failures share the exact same cancellation signature. This is intermittent and runner-specific (in this very run, sibling partition 0 kept running until 16:23:41Z and passed — so the workflow was not cancelled globally).

Failure Cluster

Diffusion 1-GPU test step cancelled mid-run (self-hosted MI325 runner loss / forced termination)

Facts (verified observations, no interpretation)

• Job conclusion failure; step 9 conclusion cancelled with completed_at = null; steps 10 (Upload diffusion failure artifacts, if: always()) and 20 (Post Checkout code) also cancelled/not run (verified via jobs API).
• Step 9 ran 15:27:51Z → job end 16:13:07Z = ~45 min; step timeout-minutes: 90 was not reached (workflow file pr-test-amd.yml:712).
• Per-job log blob is unavailable: logs API returns <Code>BlobNotFound</Code>.
• Sibling partition 0 in the same run started 15:38:26Z and completed success at 16:23:41Z — 10 min after part 1 was cancelled, ruling out a global run/concurrency cancel.
• Prior failure 27658537520 (Jun 17 01:00) shows byte-identical step pattern: step 9 started 02:50:50Z, never completed; steps 10 & 20 not run.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [MEDIUM] Self-hosted MI325 runner was lost / forcibly reclaimed mid-test (infrastructure), not a sglang/aiter code defect.
  • Supporting: step 9 cancelled at 45 min (< 90 min timeout); the if: always() upload step did NOT run, which only happens when the runner itself disappears, not on a test failure or step timeout; identical signature recurred on a different head SHA (27658537520, sha d86a7e7018) — two different commits, same symptom; sibling partitions on other runners passed.
  • Disconfirming: the actual log is unavailable, so an in-test hang/OOM that caused a watchdog kill cannot be fully excluded; however a watchdog/OOM kill inside the container would still let the always() upload step execute, which it did not.
  • How to verify: check the self-hosted runner's host syslog / GitHub runner agent logs around 16:13:07Z 2026-06-18 for an agent disconnect or VRAM-clear/reboot event; re-run partition 1 and confirm it passes (it has on 4/5 recent runs).
• [LOW] A code regression in the diffusion 1-GPU suite for partition-1 models.
  • Supporting: two failures in the window.
  • Disconfirming: partition 1 passed 4 of the last 5 completed runs across multiple SHAs; the failures don't reach a pytest assertion (step never completes); no green-vs-red code diff is implicated.
  • How to verify: re-run on the same head SHA 97e3b8998d; a green re-run confirms it is not a code regression.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for multimodal_gen runner (15 hits, none address runner cancellation) and diffusion amd timeout (7 hits, none address this). No PR targets the runner-loss/cancellation symptom. Closest unrelated diffusion-CI PR is #27587 (enable SGLANG_USE_ROCM_FLYDSL=1 in diffusion e2e), which does not fix this.

Suggested Triage Steps

• Re-run partition 1 of 27762119155 on the same SHA; if it passes, confirms an infra flake (no code action needed).
• Inspect the MI325 self-hosted runner host that served job 82139015759 for an agent disconnect / reboot near 16:13:07Z 2026-06-18 (and the prior occurrence near 02:50–03:xxZ 2026-06-17 on job 81797886738).
• If runner loss is confirmed recurring on partition 1's runner pool, consider draining/recycling that host; this is an infra ops step, not a code fix.

Status (factual, no priority assignment)

• Persistence: Intermittent — 2 failures with identical cancellation signature in the last ~6 completed scheduled runs (Jun 17 01:00 and Jun 18 13:15 UTC); partition 1 passed the 4 runs in between.
• Scope: multimodal-gen-test-1-gpu-amd partition 1 only, in pr-test-amd.yml. Sibling partitions and the rest of the run unaffected (part 0 passed in the same run).
• Blocked work: Intermittently blocks the 1-GPU diffusion CI signal for this shard on AMD; partial impact only.
• In-flight fix: ❌ none found.

stage-c-test-large-8-gpu-amd (linux-mi325-8gpu-sglang, 0) — failed step(s): Run test

Commit Info

• sglang: 61a8b42c
• aiter: 7d604afe (v0.1.14rc1.dev240+g7d604afe5, source: dev image preserved — AITER_COMMIT_OVERRIDE empty, Dockerfile default)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/test_moriep_small.py	N/A (whole-file launcher; server never became ready)	TIMEOUT after 5400s — DeepSeek-V3-0324 weight load hung	Log

Failure Summary

The stage-c-test-large-8-gpu-amd shard-0 job ran test_moriep_small.py, which launches an 8-GPU TP8/EP8/DP8 DeepSeek-V3-0324 server (--moe-a2a-backend mori). All 8 scheduler ranks entered Load weight begin at 21:32:22 but none ever reached Load weight end (0 occurrences in the log). The server never became ready; the test launcher killed it at the 5400s budget with exit code 255.

Regression Status

New failure (infrastructure) — the same test passed in the immediately preceding scheduled run.

Recent history of test/registered/amd/test_moriep_small.py in job stage-c-test-large-8-gpu-amd (linux-mi325-8gpu-sglang, 0) (branch main, event schedule, completed runs only):

Date	Run	Test File Status	Detail
Jun 18 21:25	27782126506	❌ Failed	TIMEOUT 5400s — weight load hung, no Load weight end
Jun 18 07:46	27744699827	✅ Passed	1/1 passed, elapsed 2377s; weight load completed in ~630s

(Same persistent HF cache lock file download_a17b3ca789332b1c.lock is used in both runs — the weights are already cached on this runner; this is not a fresh network download.)

Failure Cluster

Weight-load hang on shared HF cache lock during 8-GPU DeepSeek-V3 startup (server never ready → test timeout)

Facts (verified observations)

• Load weight begin printed for all 8 ranks at 21:32:22; Load weight end never printed (grep -c "Load weight end" = 0).
• DP2 (PID 4120) logged [CI Download] Process 4120 ACQUIRED lock for deepseek-ai/DeepSeek-V3-0324 at 21:32:23 (log line 7654 region) and held it for the remainder of the run.
• The other 7 ranks' pyspy dumps all show download_weights_from_hf (weight_utils.py:554) → acquire (filelock/_api.py:506) → _poll_until_acquired (filelock/_api.py:552) — blocked waiting on DP2's lock (e.g. lines 13380–13450).
• py-spy repeatedly failed to dump the lock holder PID 4120: Pyspy failed (py-spy dump --native --pid 4120). Error: Failed to get stack traces (lines 13728, 14201, 14674, 15147, 15620) — consistent with the holder being stuck in a native / uninterruptible I/O state.
• TokenizerManager + scheduler watchdogs fired every ~300s from 21:36:55 through 23:01 (watchdog_timeout=300, soft=True); main process stuck in _wait_for_scheduler_ready (engine.py:1358).
• Final: ✗ TIMEOUT: …test_moriep_small.py after 5400 seconds → First attempt failed with exit code 255 → Process completed with exit code 255 (lines 15704–15721).
• Contrast (passing run 27744699827): same lock file hash, DP2 acquired the lock at 07:57:35, all ranks reached Load weight end. elapsed≈630s by 08:08, test passed in 2377s.

Hypothesised Causes (with confidence)

• [MEDIUM] Infrastructure hang in the shared /sgl-data/hf-cache weight-load path on this MI325 runner — the lock holder (DP2) stalled mid weight-load and never released the lock, deadlocking the other 7 ranks.
  • Supporting: 0 Load weight end; lock holder held the lock the entire run; py-spy could not read PID 4120's stack (native/IO/D-state stall); the identical code + identical cached weights completed in ~630s 13h earlier.
  • Disconfirming: cannot fully prove an NFS/filesystem stall from the log alone — py-spy failure is also consistent with a GPU/native-library hang during cache read.
  • How to verify: re-run shard 0 on a fresh MI325 runner; if it passes, confirms transient infra. Check runner /sgl-data mount health and dmesg for NFS/disk stalls during 21:32–23:01.
• [LOW] sglang/aiter code regression in the DeepSeek-V3 + mori MoE weight-load path between 59001267c3 (last pass) and 61a8b42c (this run).
  • Supporting: HEAD commit differs from the last passing run.
  • Disconfirming: the hang is in generic download_weights_from_hf/filelock (model-agnostic, not in MoE/aiter code); py-spy shows no sglang model code on the stack of the waiters; the same binary path worked 13h earlier — a code regression would be deterministic, not a one-off hang.
  • How to verify: if a re-run reproduces deterministically, bisect 59001267c3..61a8b42c.

In-flight Fix Check

• In-flight fix: ❌ none found — search repo:sgl-project/sglang is:pr is:open test_moriep OR weight download lock returned only the unrelated #25984 (CUDA PyTorch stack upgrade).

Suggested Triage Steps

• Re-run the stage-c-test-large-8-gpu-amd (linux-mi325-8gpu-sglang, 0) shard; if it passes, mark as transient infra flake (most likely outcome given the prior passing run).
• On the runner, inspect /sgl-data/hf-cache mount health and check for a stale download_a17b3ca789332b1c.lock; verify no orphaned lock from a previously-killed process is left behind.
• If the hang reproduces, capture a kernel-level dump (cat /proc/<pid>/stack, dmesg) of the lock-holder PID since py-spy cannot trace it, to distinguish NFS/IO stall from a native-library hang.
• Only if reproduction is deterministic across runners: bisect 59001267c3..61a8b42c for weight-loader changes.

Status

• Persistence: 1 occurrence — failed this run (21:25 UTC); the immediately preceding scheduled run (07:46 UTC, same job/shard) passed test_moriep_small.py in 2377s.
• Scope: shard-0 of stage-c-test-large-8-gpu-amd (linux-mi325-8gpu-sglang) only; no Load weight end reached, so the entire DeepSeek-V3 mori MoE test was not exercised.
• Blocked work: blocks the 8-GPU DeepSeek-V3 + mori A2A MoE CI signal for this run.
• In-flight fix: ❌ none found.

multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 1) — failed step(s): Run diffusion server tests (2-GPU)

Commit Info

• sglang: 97e3b8998dc0f331423438091067ec0201d35e54
• aiter: N/A (no [CI-AITER-CHECK] markers; AITER_COMMIT_OVERRIDE empty — Dockerfile default, not a scout run)

Failed Tests

Test File	Test Function	Error	Log
python/sglang/multimodal_gen/test/server/test_server_2_gpu.py	TestDiffusionServerTwoGpu::test_diffusion_generation[flux2_modelopt_fp8_tp2_t2i]	openai.InternalServerError: Internal Server Error (500) — server-side RuntimeError: CUDA error: HIPBLAS_STATUS_NOT_SUPPORTED in torch._scaled_mm	Log
python/sglang/multimodal_gen/test/server/test_server_2_gpu.py	TestDiffusionServerTwoGpu::test_diffusion_generation[ideogram4_fp8_tp2_t2i]	huggingface_hub.errors.GatedRepoError: 403 Client Error → Server exited early (code 1)	Log

Failure Summary

Two distinct scenarios in the 2-GPU diffusion suite failed (final tally: 1 failed, 9 passed, 1 skipped, 12 deselected, 1 error). flux2_modelopt_fp8_tp2_t2i returned HTTP 500 because the server crashed mid-denoise with a HIPBLAS error from the FP8 GEMM path. ideogram4_fp8_tp2_t2i errored at setup because the gated HF repo ideogram-ai/ideogram-4-fp8 returns 403 Forbidden, so the server could not load the model and exited early.

Regression Status

Both Known recurring failures (same branch, same event=schedule, completed runs only).

Recent history of test_server_2_gpu.py partition (linux-mi325-2gpu-sglang, 1):

Date	Run	Job	flux2_modelopt_fp8	ideogram4_fp8
Jun 18 13:15	27762119155	job	❌ HIPBLAS 500	❌ 403 GatedRepo
Jun 18 07:46	27744699827	job	❌ HIPBLAS 500	❌ 403 GatedRepo
Jun 17 18:53	27712340726	job	not in partition	❌ 403 GatedRepo
Jun 17 13:21	27692029884	job	not in partition	❌ 403 GatedRepo

• ideogram4_fp8_tp2_t2i: failed with the identical 403 in every completed scheduled run in the window (same pinned revision ee79a72). No green run found — effectively Never-passed in queryable window.
• flux2_modelopt_fp8_tp2_t2i: failed in both runs where it landed in partition 1 (Jun 18). Scenario partitioning is dynamic (the run banner shows Suite 2-gpu Partition 2/3), so its absence from the Jun 17 jobs means it ran in a different shard, not that it passed. No green run found in this partition's window.

Failure Cluster

• flux2_modelopt_fp8_tp2_t2i → HIPBLAS_STATUS_NOT_SUPPORTED in FP8 _scaled_mm during denoise
• ideogram4_fp8_tp2_t2i → HF GatedRepo 403 on ideogram-ai/ideogram-4-fp8 (model access)

Facts (verified observations)

• flux2 server crash (step 9, line 4428) — verbatim stack (server side):

  .../multimodal_gen/runtime/layers/quantization/modelopt_quant.py:425 in apply
    return apply_fp8_linear(...)
  python/sglang/srt/layers/quantization/fp8_utils.py:1681 in apply_fp8_linear
    output = torch._scaled_mm(qinput, weight, out_dtype=input.dtype, scale_a=x_scale, ...)
  RuntimeError: CUDA error: HIPBLAS_STATUS_NOT_SUPPORTED when calling `HIPBLAS_STATUS_NOT_SUPPORTED`
  

  Originates from flux_2.py:179 _get_qkv_projections → attn.to_q(hidden_states) on black-forest-labs/FLUX.2-dev with modelopt FP8 quant. The synthetic warmup also failed the same way (line 4766); the subsequent client request then returned HTTP 500 → openai.InternalServerError (line 10551).
• python/sglang/srt/layers/quantization/fp8_utils.py:1681 confirmed to contain the torch._scaled_mm(...) call (read from current source).
• ideogram4 setup error (line 5296): httpx.HTTPStatusError: 403 Forbidden for https://huggingface.co/ideogram-ai/ideogram-4-fp8/resolve/ee79a72.../​.gitattributes → GatedRepoError → ValueError: Model or revision not found ... ensure you have access → RuntimeError: Server exited early (code 1). Identical Request-ID-bearing 403 reproduced across all 4 runs.
• This is pr-test-amd.yml, event=schedule; AITER_COMMIT_OVERRIDE is empty → regular nightly aiter, not a scout. No Baseline A/B required.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [HIGH] flux2: HIPBLAS on MI325 (gfx942) does not support the FP8 torch._scaled_mm configuration emitted by the modelopt-quantized FLUX.2 to_q linear.
  • Supporting: the crash is HIPBLAS_STATUS_NOT_SUPPORTED raised directly inside torch._scaled_mm (fp8_utils.py:1681); reproduces deterministically on every run where the scenario lands; both warmup and live request fail identically.
  • Disconfirming: HIPBLAS not "supporting" a call can also be a shape/dtype/scale-layout mismatch produced upstream rather than a pure hardware-capability gap — the status code alone does not distinguish them. Cannot localize to a specific commit (the scenario has no green run in the window to bound a regression range).
  • How to verify: launch sglang serve --model-path black-forest-labs/FLUX.2-dev --model-type diffusion --num-gpus 2 --tp-size 2 with modelopt FP8 on an MI325 in container rocm/sgl-dev:v0.5.13.post1-rocm700-mi30x-20260617, single denoise step; if it reproduces, instrument apply_fp8_linear to dump qinput/weight/x_scale/weight_scale shapes+dtypes and check against ROCm _scaled_mm support matrix.
• [FACT] ideogram4: the test cannot access gated repo ideogram-ai/ideogram-4-fp8 (403). This is a model-access/credential or repo-availability issue, not an sglang code defect.
  • Supporting: 403 Forbidden + GatedRepoError on a .gitattributes HEAD request, before any model code runs.
  • Disconfirming: (none — the error is pre-model-load and environmental.)
  • How to verify: confirm the CI HF token has been granted access to ideogram-ai/ideogram-4-fp8, or check whether the gating/license terms changed on the repo.

In-flight Fix Check

• flux2 HIPBLAS: In-flight fix: ❌ none found — searched open PRs for HIPBLAS_STATUS_NOT_SUPPORTED (0 results) and _scaled_mm rocm (0 results). #27702 ("[diffusion][ROCm] Enable BF16 attention and MXFP4 compilation on ROCm") is adjacent (ROCm diffusion quant) but does not target the FP8 _scaled_mm path — treat as candidate-only, not a confirmed fix.
• ideogram4 403: In-flight fix: ❌ none found for the access issue. #27587 ("[AMD][CI] Enable SGLANG_USE_ROCM_FLYDSL=1 in diffusion e2e tests") touches the same suite but does not address repo gating.

Suggested Triage Steps

• flux2: reproduce the single-step denoise locally (command above); if confirmed a ROCm _scaled_mm capability gap, the maintainer can decide whether to route the modelopt-FP8 linear through an aiter/triton GEMM fallback on ROCm or skip/xfail flux2_modelopt_fp8_tp2_t2i on AMD until supported. Check whether #27702's ROCm quant changes incidentally cover this path.
• ideogram4: verify/request HF access for the CI token to ideogram-ai/ideogram-4-fp8; if the repo is permanently gated for CI, skip the scenario on AMD CI rather than letting it error every run.
• Because neither scenario has a green run in the window, bisection is not actionable — treat both as standing AMD-port gaps, not fresh regressions of 97e3b899.

Status

• Persistence: ideogram4_fp8 — 403 in every completed scheduled run Jun 17–18 (4/4), no green run in window. flux2_modelopt_fp8 — HIPBLAS 500 in 2/2 runs where it landed in this partition (both Jun 18).
• Scope: job multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 1) in pr-test-amd.yml; both are AMD/ROCm-specific (FP8 GEMM capability + gated model access).
• Blocked work: blocks green signal for the 2-GPU diffusion FP8 scenarios on AMD MI325.
• In-flight fix: ❌ none found for either failure (see In-flight Fix Check).

stage-b-test-1-gpu-small-amd (linux-mi325-1gpu-sglang, 5) — failed step(s): Run test

Commit Info

• sglang: 97e3b899
• aiter: Dockerfile default (AITER_COMMIT_OVERRIDE empty; no [CI-AITER-CHECK] markers in this test-job log — exact SHA not recoverable here)

Failed Tests

Test File	Test Function	Error	Log
test/registered/scheduler/test_retract_decode.py	(none reached — server never initialized)	Scheduler hang at Init torch distributed begin; soft watchdog timeouts every 5 min; file-level timeout (2400s)	Log
test/registered/constrained_decoding/test_constrained_decoding.py	(none reached — server never initialized)	Same hang at Init torch distributed begin; 60-min job timeout hit mid-file	Log

Failure Summary

The job stage-b-test-1-gpu-small-amd (linux-mi325-1gpu-sglang, 5) timed out after 60 minutes. Both test files it reached (test_retract_decode.py, then test_constrained_decoding.py) launched their SGLang servers but the scheduler hung at Init torch distributed begin and never reached Init torch distributed ends. The py-spy dumps show the scheduler MainThread stuck on the very first GPU operation — torch.ones inside init_world_group — deep in libhsa-runtime64.so. No test function ever executed.

Regression Status

Known recurring failure (partition 5 specifically), began between Jun 17 18:53 and Jun 18 07:46 UTC.

Recent history of partition 5 (stage-b-test-1-gpu-small-amd (…, 5)) on pr-test-amd.yml, event=schedule, branch main, completed runs only:

Date (UTC)	Run	Job	Test File Status	Detail
Jun 18 13:15	27762119155	job	❌ Hang/Timeout	test_retract_decode + test_constrained_decoding hang at torch dist init
Jun 18 07:46	27744699827	job	❌ Hang/Timeout	Identical hang, same two files (head 59001267)
Jun 17 18:53	27712340726	job	✅ Passed	head 7cead0fb

First observed failure: Jun 18 07:46 UTC. Last known pass: Jun 17 18:53 UTC. Regression window: 7cead0fb8f..59001267c3 (46 commits).

Failure Cluster

GPU/torch-distributed init hang during server warmup (stuck in libhsa-runtime64.so on first HIP op)

Facts (verified observations, no interpretation)

• Server log prints […] Init torch distributed begin. (job.log:4116 for test 0/8; job.log:5019 for test 1/8) and never prints Init torch distributed ends.
• py-spy scheduler dump (PID 4323) MainThread is active, stack: libhsa-runtime64.so.1.18.70000 → libamdhip64.so.7.0.70000 → at::native::fill_kernel_cuda → at::native::ones → __init__ (distributed/parallel_state.py:301) → init_world_group (parallel_state.py:1509) → init_distributed_environment (parallel_state.py:1849) → init_torch_distributed (model_runner.py:1229) → __init__ (scheduler.py:432).
• TokenizerManager/Scheduler soft watchdog timeouts fire repeatedly (300s) without progress; ##[error]The action 'Run test' has timed out after 60 minutes. at job.log:5380.
• Test 0/8 (test_retract_decode.py, Llama-3.1-8B --chunked-prefill-size 128) consumed its 2400s file timeout (Begin (0/8) job.log:4067 → End (0/8) job.log:4960); test 1/8 (test_constrained_decoding.py, Llama-3.2-1B --grammar-backend llguidance) was killed by the 60-min job timeout.
• On this exact commit, 13 of 14 partitions of stage-b-test-1-gpu-small-amd on linux-mi325-1gpu-sglang passed; only partition 5 failed.
• AITER_COMMIT_OVERRIDE is empty (job.log:4034) — this is a regular pr-test-amd.yml run, not AITER Scout.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [MEDIUM] Runner/GPU-level wedge on the node hosting partition 5 — the hang is on the first HIP kernel launch (torch.ones), before any model or test logic, and is stuck in the AMD HSA runtime/driver.
  • Supporting: only partition 5 of 14 fails on the identical commit; all sibling partitions on the same linux-mi325-1gpu-sglang runner type pass. The stack is the canonical "GPU hangs on first kernel launch" signature (wedged GPU/driver). Both test files in partition 5 hang at the exact same point regardless of model/grammar config.
  • Disconfirming: the failure reproduces across two distinct scheduled runs 6h apart (a one-off transient wedge would normally clear between runs); cannot rule out a partition-5-specific test config triggering it.
  • How to verify: re-run only partition 5 on a different physical MI325 node; if it passes, confirms node-specific wedge. Check the runner host (linux-mi325-1gpu-sglang-57dnj-runner-fxm2c, job.log:4036) for prior hangs / dmesg GPU resets.
• [LOW] Code regression in window 7cead0fb8f..59001267c3 affecting distributed init.
  • Supporting: the failure window aligns with this 46-commit range (last green Jun 17 18:53 → first red Jun 18 07:46); the window touches parallel-topology code (e.g. #28567 "Add get_parallel()").
  • Disconfirming: a regression in the shared init_torch_distributed/init_world_group path would hang all 14 partitions, but 13 pass on the same SHA — strongly weakens any code-regression hypothesis.
  • How to verify: only pursue if the re-run on a fresh node also hangs; then bisect the window with test_retract_decode.py.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for init torch distributed hang (9 results, none matching) and AMD watchdog timeout (16 results, none matching this first-GPU-op init hang). Cited: search/issues?q=repo:sgl-project/sglang+is:pr+is:open+....

Suggested Triage Steps

• Re-run only partition 5 of stage-b-test-1-gpu-small-amd on a different MI325 node; a pass confirms a node-specific GPU wedge (the leading hypothesis) and clears the regression.
• Inspect runner host linux-mi325-1gpu-sglang-57dnj-runner-fxm2c for GPU reset/dmesg errors and quarantine if wedged.
• Only if the fresh-node re-run also hangs: bisect 7cead0fb8f..59001267c3 using test/registered/scheduler/test_retract_decode.py, starting with parallel-topology commits (#28567).

Status (factual, no priority assignment)

• Persistence: Failing on partition 5 for 2 consecutive completed scheduled runs (since Jun 18 07:46 UTC); last green Jun 17 18:53 UTC.
• Scope: 1 of 14 partitions of stage-b-test-1-gpu-small-amd on linux-mi325-1gpu-sglang; all other partitions green on the same commit. Blocks test_retract_decode.py and test_constrained_decoding.py CI signal.
• In-flight fix: ❌ none found in open PRs.

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-06-20 00:03 UTC)

[amd-bot]

pr-test-amd.yml — Per-Job Analysis (continued 4/7)

multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 1) — failed step(s): Run diffusion server tests (2-GPU)

Commit Info

• sglang: 74e2e48c
• aiter: N/A — this is pr-test-amd.yml (schedule), AITER_COMMIT_OVERRIDE is empty; no [CI-AITER-CHECK] markers in this job's log.

Failed Tests

Test File	Test Function	Error	Log
python/sglang/multimodal_gen/test/server/test_server_2_gpu.py	TestDiffusionServerTwoGpu::test_diffusion_generation[ideogram4_fp8_tp2_t2i]	huggingface_hub.errors.GatedRepoError: 403 Client Error → server worker dies → pytest EOFError at setup	Log

Result: 10 passed, 12 deselected, 1 error — the other 10 cases in this shard passed.

Failure Summary

The single failing item, test_diffusion_generation[ideogram4_fp8_tp2_t2i], errored during server launch (collection/setup), not during assertion. The server subprocess tried to snapshot_download the model ideogram-ai/ideogram-4-fp8 from HuggingFace and got a 403 GatedRepoError ("Access to model ideogram-ai/ideogram-4-fp8 is restricted and you are not in the authorized list"). The diffusion worker (rank 0) then died, and the test harness reported EOFError on the parent pipe (launch_server.py:178 reader.recv()).

Regression Status

Known recurring failure (access/infra) — not a new code regression. The same gated-repo 403 on ideogram-ai/ideogram-4-fp8 appears in every recent completed run of this job. No green run for this test found in the queryable window (lookback: Jun 17 01:00 → Jun 18 08:46 UTC) → effectively Never-passed in this window.

Recent history of test_server_2_gpu.py (test_diffusion_generation[ideogram4_fp8_tp2_t2i]) in job multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 1) (same branch main, event=schedule, completed runs only):

Date (UTC)	Run	Job	Test File Status	Failed Function	Error
Jun 18 08:46	27744699827	job	❌ Failed	ideogram4_fp8_tp2_t2i	Gated repo 403
Jun 18 00:59	27729699748	job	❌ Failed	ideogram4_fp8_tp2_t2i	Gated repo 403
Jun 17 21:25	27712340726	job	❌ Failed	ideogram4_fp8_tp2_t2i	Gated repo 403
Jun 17 13:55	27692029884	job	❌ Failed	ideogram4_fp8_tp2_t2i	Gated repo 403

(The current run 27729699748 is still queued overall, but THIS job completed with failure.)

Failure Cluster

HuggingFace gated-repo 403 on ideogram-ai/ideogram-4-fp8 during diffusion server launch.

Facts (verified observations)

• Server command (step 9): sglang serve --model-path ideogram-ai/ideogram-4-fp8 --port 21000 --model-type diffusion --num-gpus 2 --tp-size 2 --attention-backend fa (log line 4250).
• Verbatim error: huggingface_hub.errors.GatedRepoError: 403 Client Error ... Access to model ideogram-ai/ideogram-4-fp8 is restricted and you are not in the authorized list (log line 4388).
• Stack: diffusers_pipeline.py:402 _load_diffusers_pipeline → hf_diffusers_utils.py:808 maybe_download_model → snapshot_download → GatedRepoError; raised as ValueError: Model or revision not found ... ensure you have access (hf_diffusers_utils.py:853).
• Cascade: Rank 0 scheduler is dead → parent launch_server.py:178 reader.recv() → EOFError (log tail).
• The 403 first appears at registry config read (log line 4217) before model load, confirming an access problem, not a transient download hiccup.
• Reproducibility: identical failure in 4/4 most-recent completed runs of this job (Jun 17–18). All other 10 cases in the shard passed, so this is isolated to the gated model, not a GPU/runner-wide problem.
• Test definition: gpu_cases.py:585-598 defines ideogram4_fp8_tp2_t2i with model_path="ideogram-ai/ideogram-4-fp8".

Hypothesised Causes

• [FACT] The CI HuggingFace token is not on the authorized list for the gated repo ideogram-ai/ideogram-4-fp8, so snapshot_download returns 403.
  • Supporting: explicit GatedRepoError / "you are not in the authorized list" in the log; 403 occurs on the very first metadata fetch; failure is deterministic across all recent runs.
  • Disconfirming: (none found) — error message is unambiguous about gating/authorization.
  • How to verify: check whether $HF_TOKEN used by the AMD MI325 runner has been granted access at https://huggingface.co/ideogram-ai/ideogram-4-fp8; or huggingface-cli download ideogram-ai/ideogram-4-fp8 --token $HF_TOKEN from the runner.

In-flight Fix Check

• In-flight fix: ⚠️ candidate #28225 "Route Ideogram NF4 without gated metadata fetch" (open since 2026-06-15) — targets the NF4 variant's gated-metadata fetch path; may not cover the fp8 repo used by this test. Confirm scope before relying on it.
• In-flight fix: ❌ none found directly addressing CI access to ideogram-ai/ideogram-4-fp8 (fp8). Search: repo:sgl-project/sglang is:pr is:open ideogram-4-fp8 (1 result, unrelated FLYDSL PR #27587).

Suggested Triage Steps

• Verify whether the runner's HF token has access to ideogram-ai/ideogram-4-fp8; if the model is intentionally gated, request access for the CI account, or mirror weights to a non-gated CI-internal location.
• Confirm whether #28225 also removes the gated metadata fetch for the fp8 path (not just NF4); if not, a separate fix or token grant is needed.
• If access cannot be granted promptly, consider skipping/quarantining ideogram4_fp8_tp2_t2i (gpu_cases.py:585) on AMD CI so it stops masking real failures in this shard.

Status

• Persistence: Failing in every completed run of this job from at least Jun 17 13:55 UTC through Jun 18 08:46 UTC (4/4 recent completed runs).
• Scope: One test case (ideogram4_fp8_tp2_t2i) in multimodal-gen-test-2-gpu-amd; the other 10 cases in the shard pass. Access/infra issue, not GPU/kernel.
• Blocked work: Blocks 2-GPU diffusion CI signal for the Ideogram-4 fp8 path on AMD.
• In-flight fix: candidate #28225 (NF4-scoped — verify fp8 coverage); otherwise none.

stage-c-test-large-8-gpu-amd (linux-mi325-8gpu-sglang, 3) — failed step(s): Run test

Commit Info

• sglang: 97e3b8998d
• aiter: 7d604afe5 (source: Dockerfile default; dev/patched image v0.1.14rc1.dev240+g7d604afe5 detected, rebuild skipped — the prebuilt .so kernels in the image ran as-is)

Failed Tests

Test File	Test Function	Error	Log
test/registered/ops/test_aiter_allreduce_fusion_amd.py	test_fused_ar_rms_residual_accuracy	AssertionError: Residual accuracy check failed. (every shape [FAIL] max_diff=6.250000e-02)	Log

Failure Summary

In job stage-c-test-large-8-gpu-amd (linux-mi325-8gpu-sglang, 3), the distributed (torchrun, 4-GPU) sub-check test_fused_ar_rms_residual_accuracy failed. The fused all-reduce+RMSNorm path's residual output diverged from the unfused (allreduce → bf16 → add residual) reference for every tested shape (7 hidden_dims × 7 batch sizes), each with max_diff=6.250000e-02, frac_nonzero≈0.28, and AR_exact=yes. The other 3 files in this shard passed (Test Summary: 3/4 passed); the job exited 255.

Regression Status

Known recurring failure — failing on every completed scheduled run in the queryable window. Never-passed within the lookback range (no green run found, June 16–18). PR #28226 (opened 2026-06-15) corroborates the test "was failing for every shape" on this CI image.

Recent history of test_aiter_allreduce_fusion_amd.py (test_fused_ar_rms_residual_accuracy) in job stage-c-test-large-8-gpu-amd (…, 3) (branch main, event schedule, completed runs only):

Date	Run	Job	Test File Status	Failed Function	Error
Jun 18	27762119155	job	❌ Failed	test_fused_ar_rms_residual_accuracy	exit 1, max_diff=6.25e-02
Jun 18	27744699827	job	❌ Failed	same	exit 1
Jun 17	27712340726	job	❌ Failed	same	exit 1
Jun 17	27692029884	job	❌ Failed	same	exit 1
Jun 17	27658537520	job	❌ Failed	same	exit 1
Jun 16	27641945084	job	❌ Failed	same	exit 1
Jun 16	27603298800	job	❌ Failed	same	exit 1
Jun 16	27587002396	job	❌ Failed	same	exit 1

All 8 completed runs ran the identical aiter version v0.1.14rc1.dev240+g7d604afe5. No green run exists in the queryable window.

Failure Cluster

Fused all-reduce+RMSNorm residual 1-bf16-ULP divergence (test_fused_ar_rms_residual_accuracy bit-exact assertion fails)

Facts (verified observations)

• Every shape fails identically: max_diff=6.250000e-02 (= 1 bf16 ULP at that magnitude), frac_nonzero≈0.27–0.28, AR_exact=yes (verbatim from log, e.g. 1x2880 (1-stage): max_diff=6.250000e-02 frac_nonzero=0.2774 AR_exact=yes [FAIL]).
• AR_exact=yes means the all-reduce output is bit-identical to the unfused path; the divergence is only in the residual-add stage of the fused kernel (test_aiter_allreduce_fusion_amd.py:106-114: expected = fused_ar + residual; passed = max_diff == 0.0).
• The test asserts strict bit-exactness (max_diff == 0.0); a 1-ULP divergence on ~28% of elements is enough to fail.
• aiter version is unchanged (v0.1.14rc1.dev240+g7d604afe5) across all 8 completed runs; the image is a dev/patched build so the rebuild was skipped ([CI-AITER-CHECK] Dev/patched version detected … → skipping rebuild).
• aiter fix commit 43b7379b8 ("Fix numerical accuracy in allreduce_fusion_kernel_1stage", Fix: Numerical Accuracy in allreduce_fusion_kernel_1stage ROCm/aiter#2586) is dated 2026-04-02; the image's nominal aiter SHA 7d604afe5 is dated 2026-06-09.
• This is pr-test-amd.yml (event schedule), not an AITER Scout run — aiter here is the Dockerfile default, not an override.

Hypothesised Causes (with confidence)

• [HIGH] The prebuilt aiter allreduce_fusion_kernel_1stage in the CI docker image does not exhibit the Fix: Numerical Accuracy in allreduce_fusion_kernel_1stage ROCm/aiter#2586 (43b7379b8) accuracy behavior: it keeps extra f32 mantissa bits across the residual add instead of doing a bf16 round-trip first, producing exactly 1 bf16 ULP of divergence. This matches the observed max_diff=6.25e-02 / frac_nonzero≈0.28 / AR_exact=yes signature exactly.
  • Supporting: signature is identical across all 56 shape cases and all 8 runs; the maintainer's PR #28226 describes the same root cause ("prebuilt aiter kernel … predates the accuracy fix … differs by exactly 1 bf16 ULP for ~25-28% of elements").
  • Disconfirming: the image's nominal aiter SHA 7d604afe5 (Jun 9) is a descendant of the fix commit 43b7379b8 (Apr 2), so by source ancestry the fix should be present. The empirical 1-ULP behavior nonetheless shows the compiled kernel in the prebuilt image behaves as pre-fix — i.e. the version string and the actual built .so disagree (rebuild was skipped). Treat the version label, not the runtime behavior, as the unreliable signal.
  • How to verify: in the container, run the kernel and inspect whether allreduce_fusion_kernel_1stage does the bf16 round-trip before residual add; or rebuild aiter from 7d604afe5 (force AITER_COMMIT_OVERRIDE) and rerun the test.
• [FACT] The test's strict max_diff == 0.0 assertion is too tight for this kernel's documented 1-ULP behavior; this is what PR #28226 relaxes (to 1 bf16 ULP). This is a test-policy decision, not a sglang code regression.

In-flight Fix Check

• In-flight fix: ✅ #28226 "[AMD] Relax allreduce-fusion residual accuracy tolerance to 1 bf16 ULP" (open since 2026-06-15, last updated 2026-06-18) — directly targets test_fused_ar_rms_residual_accuracy. Chase reviewers; do NOT open a duplicate.
• Also potentially related (do not duplicate): #27650 "[AMD] Fix PCG capture fail with aiter fused all-reduce+RMSNorm".
• (API query: search/issues?q=repo:sgl-project/sglang+is:pr+is:open+allreduce+fusion+residual.)

Suggested Triage Steps

• Land in-flight PR #28226 (relax assertion to 1 bf16 ULP) — this is the maintainer's chosen path and unblocks stage-c-test-large-8-gpu-amd shard 3.
• If bit-exactness is actually desired instead: rebuild aiter from 7d604afe5 via AITER_COMMIT_OVERRIDE and rerun test_aiter_allreduce_fusion_amd.py --residual-accuracy to confirm whether a fresh build (vs. the prebuilt dev image .so) produces max_diff=0. This isolates "version-string-vs-binary mismatch" from "fix genuinely absent".
• Verify the docker base image (rocm/sgl-dev:v0.5.13-rocm700-mi30x) bakes the post-#2586 allreduce_fusion_kernel_1stage; bump/rebuild the image if not.

Status

• Persistence: Failing every completed scheduled run since at least 2026-06-16 (8/8 completed runs in window); PR #28226 indicates it was already failing on 2026-06-15. No green run in queryable history.
• Scope: 1 test function in test_aiter_allreduce_fusion_amd.py within stage-c-test-large-8-gpu-amd (…, 3); the other 3 files in the shard pass. Fails the shard (exit 255) on pr-test-amd.yml; the same kernel/test would fail on any AMD job running this file with this image.
• In-flight fix: ✅ #28226 (open since 2026-06-15) — awaiting merge.

stage-b-test-1-gpu-small-amd-mi35x (linux-mi35x-gpu-1) — failed step(s): Run test

Commit Info

• sglang: 74e2e48c
• aiter: 7d604afe (v0.1.14rc1.dev240+g7d604afe5, source: dev/patched image — Dev/patched version detected → skipping rebuild)

Failed Tests

Test File	Test Function	Error	Log
test/registered/quant/test_quark_mxfp4.py	Qwen/Qwen3-30B-A3B-Instruct-2507 sub-case (server never reached Ready; harness killed the run)	TIMEOUT after 1200s (est. 106s) — scheduler hung in init_tokenizer → hf_hub_download, spin-waiting on an HF cache file lock	Log

Failure Summary

test/registered/quant/test_quark_mxfp4.py ran two model sub-cases. The first (Qwen/Qwen3-8B) loaded and passed gsm8k. The second sub-case (Qwen/Qwen3-30B-A3B-Instruct-2507) never came up: the scheduler process (PID 9632) blocked in init_tokenizer (scheduler.py:695) → hf_hub_download, spinning indefinitely on a HuggingFace cache file lock at /sgl-data/hf-cache/hub/.locks/models--Qwen--Qwen3-30B-A3B-Instruct-2507/aeb13307…lock. The per-test harness killed the run at its 1200s ceiling, exit code 255.

Regression Status

Flaky / Infrastructure issue — new failure mode, not seen in the immediately-preceding completed runs.

Recent history of test_quark_mxfp4.py in job stage-b-test-1-gpu-small-amd-mi35x (linux-mi35x-gpu-1) (branch main, event schedule, completed runs only):

Date	Run	Job	Test File Status	Failed Function	Error
Jun 18 00:59	27729699748	mi35x-1gpu-small	❌ Failed	Qwen3-30B-A3B sub-case	TIMEOUT 1200s (HF cache lock hang)
Jun 17 18:53	27712340726	mi35x-1gpu-small	✅ Passed	—	—
Jun 17 13:21	27692029884	mi35x-1gpu-small	✅ Passed	—	—
Jun 17 07:57	27674400949	mi35x-1gpu-small	❌ Failed	(different mode)	ValueError: Should have found memory increase in load_weights

The Jun 17 07:57 failure is a distinct symptom (a weight-load memory-check assertion, exit code 1), not the lock hang. The lock-timeout symptom seen today is its first observed occurrence in the queryable window; the test passed in the two most recent completed runs before it.

Failure Cluster

HF cache file-lock deadlock during tokenizer download on shared /sgl-data/hf-cache

Facts (verified observations)

• py-spy dump of the hung scheduler (PID 9632, sglang::scheduler): run_scheduler_process (scheduler.py:4112) → __init__ (scheduler.py:419) → init_tokenizer (scheduler.py:695) → hf_hub_download (huggingface_hub/file_download.py:1015) → _hf_hub_download_to_cache_dir (file_download.py:1236).
• Two lock holders (IDs 127956788691392, 139518657711696) loop Attempting to acquire lock … not acquired … waiting 0.05 seconds continuously from 01:10:42 to 01:27 on the same lock file /sgl-data/hf-cache/hub/.locks/models--Qwen--Qwen3-30B-A3B-Instruct-2507/aeb13307a71acd8fe81861d94ad54ab689df773318809eed3cbe794b4492dae4.lock.
• Timeout on acquiring lock … printed at 01:12:30, 01:19:11, 01:24:42; scheduler soft-watchdog fired at 01:15:40, 01:20:41, 01:25:41 (watchdog_timeout=300, soft=True) but did not kill the process.
• First sub-case Qwen/Qwen3-8B completed gsm8k successfully (01:08–01:10); the hang is isolated to the second model load.
• ✗ TIMEOUT: …/test_quark_mxfp4.py after 1200 seconds → First attempt failed with exit code 255 … Exit code 255 indicates test failure (not network issue), not retrying → Process completed with exit code 255.
• Lock path is on the shared /sgl-data/hf-cache volume, not a container-local cache.

Hypothesised Causes (with confidence)

• [MEDIUM] Stale / contended HuggingFace cache file lock on the shared /sgl-data/hf-cache volume blocks the tokenizer download for Qwen/Qwen3-30B-A3B-Instruct-2507.
  • Supporting: py-spy shows the scheduler parked in hf_hub_download; thousands of not acquired … waiting 0.05s lines on one .lock file spanning the whole 1200s window; the holder never releases. The lock lives on a shared volume that parallel jobs/runs also use, a classic cross-process filelock stall (often a leftover lock from a previously killed download).
  • Disconfirming: the same job passed in the two preceding completed runs on the same shared cache; no second concurrent owner of this exact lock is visible in this job's log (the holder is likely an external process on the shared runner, which this log cannot show).
  • How to verify: on the mi35x runner, check for stale *.lock files under /sgl-data/hf-cache/hub/.locks/models--Qwen--Qwen3-30B-A3B-Instruct-2507/; re-run the job and confirm the tokenizer download completes; inspect whether sibling jobs share /sgl-data/hf-cache concurrently.
• [LOW] sglang code change in window. The regression window for the passing→failing transition (Jun 17 18:53 7cead0fb → Jun 18 00:59 74e2e48c) does not touch HF download / init_tokenizer in a way evidenced by the stack.
  • Supporting: temporal — it began this run.
  • Disconfirming: the stack is entirely in huggingface_hub.file_download waiting on a filesystem lock; no sglang code frame is implicated beyond init_tokenizer which merely calls hf_hub_download. This is the canonical signature of an environment/IO lock stall, not a code defect.
  • How to verify: not actionable until the cache-lock hypothesis is ruled out.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs repo:sgl-project/sglang is:pr is:open hf cache lock → 0 results. No existing PR targets the HF cache-lock stall.

Suggested Triage Steps

• Re-run the job (Re-run); if it passes, this is confirmed transient cache-lock contention.
• On the mi35x runner, clear stale locks: inspect /sgl-data/hf-cache/hub/.locks/models--Qwen--Qwen3-30B-A3B-Instruct-2507/ for orphaned .lock files left by a killed prior download.
• Check whether multiple CI jobs share /sgl-data/hf-cache concurrently on the same node; if so, consider pre-seeding this model into the cache (or per-job HF_HOME) so the test never races on a first-time download.
• Consider whether the per-test 1200s timeout should hard-kill the scheduler when the soft watchdog (300s) trips repeatedly during init_tokenizer, so the failure surfaces faster with a clearer message instead of a 20-min hang.

Status

• Persistence: First occurrence of this lock-hang symptom in the queryable window (this run, Jun 18 00:59 UTC). Test passed in the two preceding completed runs (Jun 17 13:21 and 18:53).
• Scope: This job/test only (stage-b-test-1-gpu-small-amd-mi35x, test_quark_mxfp4.py, Qwen3-30B-A3B sub-case). No other test in the job failed.
• Blocked work: Blocks the quark_mxfp4 quantization signal on mi35x until the cache-lock contention clears.
• In-flight fix: ❌ none found — likely an infra/runner fix (stale-lock cleanup or cache isolation) rather than a code PR.

stage-c-test-large-8-gpu-amd (linux-mi325-8gpu-sglang, 3) — failed step(s): Run test

Commit Info

• sglang: 74e2e48c
• aiter: 7d604afe5 (v0.1.14rc1.dev240+g7d604afe5, source: Dockerfile default / dev image preserved; AITER_COMMIT_OVERRIDE empty → not a scout run)
• Image: rocm/sgl-dev:v0.5.13.post1-rocm700-mi30x-20260617

Failed Tests

Test File	Test Function	Error	Log
test/registered/ops/test_aiter_allreduce_fusion_amd.py	test_fused_ar_rms_residual_accuracy	AssertionError: Residual accuracy check failed — fused residual diverges from unfused path for ALL shapes, max_diff=6.25e-2, AR_exact=yes	Log
test/registered/amd/test_deepseek_v3_mtp.py	(suite timeout — server never became ready)	TIMEOUT after 5400 seconds	Log

Failure Summary

Two independent failures in shard 3 of stage-c-test-large-8-gpu-amd. (1) test_fused_ar_rms_residual_accuracy fails the bit-exact check between the fused aiter all-reduce+RMSNorm path and the unfused reference for every tested shape (max_diff=6.25e-2, AR itself bit-exact). (2) test_deepseek_v3_mtp.py (DeepSeek-V3-0324, EAGLE, tp=8) timed out at the 5400 s cap while the server was still blocked in _wait_for_scheduler_ready waiting on a stalled HuggingFace weight download.

Regression Status

• test_fused_ar_rms_residual_accuracy → Known recurring failure (≥ 3 days persistent).
• test_deepseek_v3_mtp.py → New failure today (infrastructure / weight-download stall; passed in every prior queryable run).

Recent history of test_aiter_allreduce_fusion_amd.py (test_fused_ar_rms_residual_accuracy) in job stage-c-test-large-8-gpu-amd (..., 3) (branch main, event=schedule, completed runs only):

Date (UTC)	Run	Test File Status	Failed Function	Error
Jun 18 00:59	27729699748	❌ Failed	test_fused_ar_rms_residual_accuracy	residual diverges max_diff=6.25e-2
Jun 18 07:46	27744699827	❌ Failed	same	same
Jun 17 18:53	27712340726	❌ Failed	same	same
Jun 17 13:21	27692029884	❌ Failed	same	same
Jun 17 07:57	27674400949	❌ Failed	same	same
Jun 17 01:00	27658537520	❌ Failed	same	same
Jun 16 01:05	27587002396	❌ Failed	same	same

First failure of this test file is at or before Jun 16 01:58 UTC; no green run found in the queryable window (lookback ≈ 12 scheduled runs / ~2.5 days). Broken continuously since.

Recent history of test_deepseek_v3_mtp.py (same job/branch/event, completed runs only):

Date (UTC)	Run	Test File Status	Elapsed
Jun 18 00:59	27729699748	❌ Timeout	5400 s (killed)
Jun 18 07:46	27744699827	✅ Passed	936 s
Jun 17 18:53	27712340726	✅ Passed	956 s
Jun 17 13:21	27692029884	✅ Passed	1027 s
Jun 17 07:57	27674400949	✅ Passed	4020 s
Jun 17 01:00	27658537520	✅ Passed	1006 s

Note the wide elapsed-time swing (936 s → 4020 s) across passing runs is itself evidence of variable weight-download time on this runner.

Failure Cluster

• test_fused_ar_rms_residual_accuracy → "aiter fused all-reduce+RMSNorm residual numeric mismatch (max_diff=2⁻⁴)"
• test_deepseek_v3_mtp.py → "DeepSeek-V3 weight-download stall → server-ready timeout"

Facts (verified observations)

• Every shape fails with an identical max_diff=6.250000e-02 (= 2⁻⁴) and AR_exact=yes — the all-reduce result is bit-exact; only the fused residual/RMSNorm output diverges, uniformly across all 6 hidden dims and all batch sizes (1×2880 … 128×8192). frac_nonzero ≈ 0.28.
• aiter in the CI image: v0.1.14rc1.dev240+g7d604afe5 ([CI-AITER-CHECK] Dev/patched version detected … skipping rebuild), no override applied.
• MTP pyspy main-thread stack (PID 3591) is stuck at engine.py:1358 _wait_for_scheduler_ready; the scheduler proc (PID 3768) repeatedly fails py-spy dumps ("Failed to get stack traces"). Last scheduler log line before the stall: [CI Download] Process 3768 ACQUIRED lock … Local HF snapshot … has no files matching ['*.safetensors']; will attempt download. → download never completed before the 5400 s cap. The model cache had no local safetensors (CI_OFFLINE: Cache validation failed … will use online mode).
• Both failures reproduce in this single shard; the allreduce failure also appears in the parallel ROCm runs (recurring across runs).

Hypothesised Causes

• [HIGH] aiter fused all-reduce+RMSNorm residual mismatch: the prebuilt aiter kernel in the CI docker image predates the accuracy fix ROCm/aiter#2586 (43b7379b8), which inserts a bf16 round-trip after the f32 all-reduce accumulation and before the residual add so the fused path matches the unfused (allreduce → bf16 → add) path bit-for-bit. Without it the fused kernel keeps extra f32 mantissa bits → constant 2⁻⁴ divergence.
  • Supporting: in-flight PR #28226 documents this exact symptom (max_diff=6.250000e-02 for every shape) and names the aiter image/commit gap; symptom (AR exact, fused-residual-only divergence, uniform across shapes) is consistent with an extra-precision-in-fused-path issue, not a logic bug.
  • Disconfirming: I did not independently confirm whether image aiter 7d604afe5 is strictly older than the fix commit 43b7379b8; the PR asserts a different image tag (v0.5.13-rocm700-mi30x) than the one this run used (…post1…-20260617). The conclusion rests on the PR's stated diagnosis matching the byte-identical symptom.
  • How to verify: run test_fused_ar_rms_residual_accuracy in a container whose aiter includes 43b7379b8; or apply PR #28226's relaxed (1 bf16 ULP) tolerance and re-run.
• [MEDIUM] MTP timeout = weight-download/infra stall, not a code regression: stack is blocked in _wait_for_scheduler_ready behind an HF download that found no local safetensors; the test passed in all 5 prior runs (one taking 4020 s, near the cap), indicating download latency — not test logic — governs pass/fail.
  • Supporting: head SHA unchanged in spec-decode/model code vs the immediately preceding passing run; the only differentiator is download progress; CI_OFFLINE cache validation failed.
  • Disconfirming: cannot fully exclude a slow/hanging warmup after a successful download, since the cap hit before any "Capturing"/warmup log appeared (it never got past weight load).
  • How to verify: re-run shard 3; if it passes (or pre-seed the DeepSeek-V3-0324 cache on the runner) the timeout was the cache miss.

In-flight Fix Check

• test_fused_ar_rms_residual_accuracy → In-flight fix: ✅ #28226 "[AMD] Relax allreduce-fusion residual accuracy tolerance to 1 bf16 ULP" (open, created 2026-06-15, updated 2026-06-18) — chase reviewers; do NOT open a duplicate. (Related open PRs touching this path: #27650, #27835.) Query: is:pr is:open aiter allreduce fusion.
• test_deepseek_v3_mtp.py → In-flight fix: ❌ none found — infra/cache issue, no code PR expected. Query: is:pr is:open deepseek_v3_mtp timeout.

Suggested Triage Steps

• For the allreduce failure: verify in-flight PR #28226 and confirm whether the merge fix is "relax tolerance" vs "bump the CI image aiter to include ROCm/aiter#2586"; re-run test_fused_ar_rms_residual_accuracy on MI325 in the PR's branch image.
• Independently confirm the aiter commit ordering: check whether image aiter 7d604afe5 contains 43b7379b8 (git log --oneline 7d604afe5 | grep 43b7379b8 in an aiter checkout) — if it already contains the fix, reopen the root-cause question.
• For the MTP timeout: re-run shard 3 (or pre-seed/warm the DeepSeek-V3-0324 weight cache on linux-mi325-8gpu-sglang); if it passes, mark as a transient cache miss. Consider raising the per-test cap or guaranteeing the cache for this 660 GB model.

Status

• Persistence: test_fused_ar_rms_residual_accuracy failing continuously ≥ 3 days (since ≤ Jun 16 01:58 UTC), every completed scheduled run. test_deepseek_v3_mtp.py timeout is new this run only.
• Scope: shard 3 of stage-c-test-large-8-gpu-amd; the allreduce failure recurs across consecutive scheduled runs (and parallel ROCm jobs).
• Blocked work: red CI signal for the 8-GPU large AMD suite; the allreduce failure masks any genuine new regression in this shard until resolved.
• In-flight fix: ✅ #28226 (allreduce); ❌ none (MTP timeout).

stage-c-test-large-8-gpu-amd (linux-mi325-8gpu-sglang, 2) — failed step(s): Install dependencies

Commit Info

• sglang: 74e2e48c — "Introduce CpuDeviceMixin and CpuSRTPlatform" (#26385)
• aiter: N/A — no [CI-AITER-CHECK] markers; the job died in the Install dependencies step (mori build) before the aiter check ran. AITER_COMMIT_OVERRIDE is empty (this is a regular pr-test-amd.yml schedule run, not a scout).

Failed Tests

Test File	Test Function	Error	Log
N/A — no test ran	N/A	mori dependency build failed during gtest_discover_tests: test_tier_priority_route_get: error while loading shared libraries: /lib/x86_64-linux-gnu/libabsl_strings.so.20210324: invalid ELF header (Result: 127)	Log

This is not a test failure. The job aborted in the bash scripts/ci/amd/amd_ci_install_dependency.sh step while building the ROCm KV-transfer dependency mori from source. No sglang test was ever reached.

Failure Summary

During dependency installation, the mori C++ build (ninja -j 256) stopped because the gtest discovery step for test_tier_priority_route_get could not launch: the system library libabsl_strings.so.20210324 has an invalid ELF header (corrupt/truncated file), giving Result: 127. ninja then reported build stopped: subcommand failed and setup.py raised CalledProcessError, failing the step with exit code 1.

Regression Status

Infrastructure issue (transient, runner-specific) — not a code regression.

Recent history of this job (stage-c-test-large-8-gpu-amd (linux-mi325-8gpu-sglang, 2)) on pr-test-amd.yml, branch main, event schedule, completed runs only:

Date	Run	Job	Build/Install Status	Detail
Jun 18 01:22	27729699748	shard 2	❌ Failed	mori build — corrupt libabsl_strings.so (this run)
Jun 17 18:53	27712340726	shard 2	✅ Passed	dependency build succeeded

The identical shard-2 job passed in the immediately preceding scheduled run, confirming the corrupt library is a fresh, transient condition on this specific runner/container — not introduced by the commit under test.

Failure Cluster

mori dependency build failure — corrupt absl shared library (invalid ELF header)

Facts (verified observations)

• [FACT] Failing step is step 5 "Install dependencies" (amd_ci_install_dependency.sh), not a pytest stage. (log)
• [FACT] Exact error (verbatim): error while loading shared libraries: /lib/x86_64-linux-gnu/libabsl_strings.so.20210324: invalid ELF header → Result: 127. (log line 3998)
• [FACT] ninja: build stopped: subcommand failed. followed by subprocess.CalledProcessError: Command '['cmake', '--build', '.', '-j', '256']' returned non-zero exit status 1. (log line 4063)
• [FACT] The 4 sibling failed jobs in the same run (82033824465, 82033824313, 82033824447, 82033824092) do not contain the absl/ELF error — they got past the build and failed in unrelated tests (test_aiter_allreduce_fusion_amd.py, test_deepseek_r1_mxfp4_8gpu.py, etc.). The build corruption is isolated to this one runner.
• [FACT] Local docker registry pull failed earlier and fell back to the public registry (rocm/sgl-dev:v0.5.13.post1-rocm700-mi30x-20260617). (log line 1247)
• [FACT] The commit under test (74e2e48c) changes only pure-Python files (platforms/__init__.py, platforms/cpu.py, plugin docs, a Python unit test) — no C++ / mori / build-system code.

Hypothesised Causes (with confidence)

• [HIGH] Corrupted system shared library libabsl_strings.so.20210324 on the runner/container (invalid ELF header).
  • Supporting: the executable links and the file exists, but loading it returns invalid ELF header (a truncated/partially-written or filesystem-corrupted .so); occurred right after a local docker-registry pull failure forced a public-registry fallback, suggesting an imperfect image/layer on this host.
  • Disconfirming: (none found) — no other explanation fits an "invalid ELF header" on a base-image library.
  • How to verify: re-run the job on a fresh runner; if it passes, the corruption was host-local. On the failing host, file /lib/x86_64-linux-gnu/libabsl_strings.so.20210324 would confirm truncation.
• [FACT, exonerating] The sglang commit did not cause this — it touches only Python platform code, while the failure is a C++ link-time/load-time corruption in a dependency.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for mori install (results: #22037, #27209, #19777, #28645 — none address corrupt-library / install corruption). This is a transient infra fault, not a code defect, so no fix PR is expected. (A bingxche/mori-ci-reinstall-26922 branch exists but has no matching open PR.)

Suggested Triage Steps

• Re-run the job (rerun-failed) on a fresh runner — a corrupt .so from a bad image pull is expected to clear on retry.
• If it recurs on the same host, quarantine/recycle that runner and verify the docker image integrity (the local registry pull failed and fell back to public — check the fallback layer that ships libabsl_strings.so.20210324).
• No sglang code action required; do not correlate this to 74e2e48c.

Status

• Persistence: One-off. The same shard-2 job passed in the prior completed scheduled run (27712340726, Jun 17 18:53 UTC); first observed this run (Jun 18 01:22 UTC).
• Scope: This single job/runner only. The 4 other failed jobs in the same run failed for unrelated reasons (real test failures), not the mori build.
• Blocked work: Blocks the 8-GPU large-test shard 2 signal for this run only; cleared by a re-run on a healthy runner.
• In-flight fix: ❌ none found — transient infrastructure fault, no PR needed.

stage-c-test-large-8-gpu-amd-mi35x (linux-mi35x-gpu-8, 0) — failed step(s): Run test

Commit Info

• sglang: 74e2e48c
• aiter: 7d604afe5 (v0.1.14rc1.dev240+g7d604afe5; source: dev CI image, matches rocm.Dockerfile default → no rebuild)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/test_deepseek_r1_mxfp4_8gpu.py	TestDeepseekR1MXFP4MTP.test_a_gsm8k	AssertionError: 1.4282435740514077 not greater than 2.04	Log

The sister test file test/registered/amd/test_qwen3_coder_next_8gpu.py passed; only the MXFP4 MTP test failed (job result "1/2 passed").

Failure Summary

On the 8-GPU AMD MI35x shard, test_a_gsm8k in TestDeepseekR1MXFP4MTP failed its speculative-decoding acceptance-length check: avg_spec_accept_length=1.43 is below the required 2.04. GSM8K accuracy itself passed (0.95 > 0.94) — only the EAGLE/MTP draft acceptance rate is degraded, indicating the draft model is producing far fewer accepted tokens than expected (≈1.0 means near-zero speculation accepted).

Regression Status

New regression — first observed 2026-06-16 ~01:05 UTC; last green 2026-06-15 ~01:21 UTC.

Recent history of test_deepseek_r1_mxfp4_8gpu.py (TestDeepseekR1MXFP4MTP.test_a_gsm8k) in job stage-c-test-large-8-gpu-amd-mi35x (linux-mi35x-gpu-8, 0) (branch main, event schedule, completed runs only):

Date (UTC)	Run	Test File Status	avg_spec_accept_length	Note
Jun 18	this run	❌ Failed	1.43	aiter 7d604afe5
Jun 18 07:46	27744699827	❌ Failed	1.45
Jun 17 18:53	27712340726	❌ Failed	1.44
Jun 17 13:21	27692029884	❌ Failed	1.46
Jun 17 01:00	27658537520	❌ Failed	1.44
Jun 16 08:04	27603298800	❌ Failed	1.44
Jun 16 01:05	27587002396	❌ Failed	1.45	first spec-accept fail
Jun 15 19:22	27570634886	❌ Failed (different)	—	accuracy 0.095 + docker-pull timeout (infra, not this cluster)
Jun 15 01:21	27517929362	✅ Passed	2.87	aiter 7a8ff7dd4

Failure Cluster

MTP/EAGLE speculative accept-length below threshold on DeepSeek-R1-MXFP4 (AMD MI35x) — avg_spec_accept_length ≈ 1.44 < 2.04, accuracy unaffected.

Facts (verified observations)

• Verbatim error: AssertionError: 1.4282435740514077 not greater than 2.04 at test_deepseek_r1_mxfp4_8gpu.py:163 (self.assertGreater(avg_spec_accept_length, 2.04)).
• Accuracy passed in the same run: Accuracy: 0.950, metrics={'accuracy': 0.95, ...}.
• Spec config: --speculative-algorithm EAGLE --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 (test_deepseek_r1_mxfp4_8gpu.py:109-118).
• AITER version delta across the boundary (from [CI-AITER-CHECK] markers): last green run = v0.1.14rc1.dev213+g7a8ff7dd4; every failing run incl. this one = v0.1.14rc1.dev240+g7d604afe5.
• Accept length sequence: 2.87 (Jun 15 green) → 1.43–1.46 every completed run since Jun 16 — stable, not flaky.
• Reproducibility: 100% of completed mi35x-gpu-8 shard 0 runs since Jun 16 (≥9 runs).

Hypothesised Causes (with confidence + disconfirming evidence)

• [MEDIUM] AITER bump via sglang PR #28249 (9864059e) changed AITER_COMMIT_DEFAULT from 7a8ff7dd4 to 7d604afe5.
  • Supporting: AMD-only failure; the new aiter version is exactly what runs in every failing run, while the last green run ran the old aiter (verified via [CI-AITER-CHECK]). A draft-only numerical change (accept length halved, target accuracy intact) is consistent with an attention/MoE kernel numeric change affecting the EAGLE draft path.
  • Disconfirming: 49 sglang commits also landed in the 7a8ff7dd4→7d604afe5 window; this is pr-test-amd.yml (no scout A/B baseline) so aiter cannot be isolated without a controlled rerun. The aiter delta itself spans many commits and was not inspected.
  • How to verify: rerun test_deepseek_r1_mxfp4_8gpu.py on MI35x in a container pinned to aiter 7a8ff7dd4 (old) vs 7d604afe5 (new), same sglang HEAD; compare avg_spec_accept_length.
• [LOW] sglang PR #28192 (1a66059c) "Restore index_share_for_mtp_iteration in EAGLE V2 draft worker" modifies eagle_worker_v2.py draft loop.
  • Supporting: directly touches MTP draft iteration; the test runs --speculative-eagle-topk 1, satisfying the self.topk == 1 gate.
  • Disconfirming: the new path is also gated on hf_config.index_share_for_mtp_iteration (NSA/DSA indexer reuse), which is a DeepSeek-sparse-attention feature unlikely set for non-sparse DeepSeek-R1-MXFP4 → likely a no-op for this model.
  • How to verify: log/inspect whether index_share_for_mtp_iteration resolves True for this model; if False, exclude this commit.
• [LOW] Other in-window spec commits — e.g. #28241 (81dcb006 "Use decode kernel for TRT-LLM MHA draft extend"), ce9fad71 "Fix Spec V2 Draft Input ID Dtype". Temporal-only; no code-path evidence tying them to AMD/aiter accept-length.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for mxfp4+mtp, spec+accept+length+amd, deepseek+mxfp4+spec; no PR addressing the post-aiter-bump accept-length regression. (Closest matches are unrelated MXFP4/EAGLE feature PRs.)

Suggested Triage Steps

• A/B the aiter delta: rerun TestDeepseekR1MXFP4MTP.test_a_gsm8k on MI35x with aiter pinned to 7a8ff7dd4 vs 7d604afe5 at sglang HEAD 74e2e48c; if accept length returns to ~2.8 with the old aiter, the aiter bump is confirmed.
• If aiter is exonerated, bisect sglang spec commits in c127ba6483..2dd449ce5e, starting with #28192 and #28241.
• Confirm hf_config.index_share_for_mtp_iteration for DeepSeek-R1-MXFP4 to accept/reject the #28192 hypothesis cheaply.
• If aiter 7d604afe5 is the confirmed cause and a kernel fix is not immediately available, the MI35x diff in the aiter range can be inspected via the ROCm/aiter compare for draft-path kernels.

Status

• Persistence: failing every completed run since 2026-06-16 ~01:05 UTC (~3 days, ≥9 consecutive runs).
• Scope: 1 test on stage-c-test-large-8-gpu-amd-mi35x (linux-mi35x-gpu-8, 0) in pr-test-amd.yml; the gfx950/MI35x DeepSeek-R1-MXFP4 MTP path only.
• Blocked work: blocks green signal for the AMD MXFP4 MTP 8-GPU shard; speculative-decoding acceptance regression (functional, not a crash — accuracy still passes).
• In-flight fix: ❌ none found in open PRs.

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-06-20 00:03 UTC)

[amd-bot]

pr-test-amd.yml — Cross-Run Pattern Summary (8 runs in lookback window)

Last scanned (UTC): 2026-06-20 00:06 UTC

Agent assessment

Persistent clusters (top 3)

1. Diffusion FP8 _scaled_mm HIPBLAS + ideogram4 gated-403 — multimodal-gen-test-2-gpu-amd (…, 1), e.g. run 27799027960. Failed in 8/8 completed runs (the only bucket-confirmed "every run" failure). Two distinct problems in one shard:

• flux2_modelopt_fp8_tp2_t2i → RuntimeError: HIPBLAS_STATUS_NOT_SUPPORTED in torch._scaled_mm (fp8_utils.py:1681). Brand-new test added Jun 18 by #28546 (3b61dc32); never passed on AMD (not a regression of green code). [MEDIUM] hipBLASLt rejects the TP2 per-tensor FP8 scale shape; disconfirming: no green baseline ever existed, so scale-load change vs GEMM gap can't be isolated.
• ideogram4_fp8_tp2_t2i → GatedRepoError: 403 for ideogram-ai/ideogram-4-fp8. [FACT] runner HF token lacks gated-repo access — infra, not code.
• In-flight fix: ❌ none for either.

2. Fused all-reduce+RMSNorm residual bit-exactness mismatch — stage-c-test-large-8-gpu-amd (…, 3), run 27744699827. Failed 5/8 in window (bucket), never-passed since Jun 16 (7/7 by the per-job history). test_fused_ar_rms_residual_accuracy, max_diff=2⁻⁴, AR_exact=yes. [FACT] CI image aiter 7d604afe predates the ROCm/aiter#2586 fix 43b7379b8; disconfirming: (none — symptom, test docstring, and PR all converge).

• In-flight fix: ✅ #28226 (open since 2026-06-15, relaxes assertion to 1 bf16 ULP) — chase reviewers.

3. MTP/EAGLE spec accept-length below floor (DeepSeek-R1-MXFP4, MI35x) — stage-c-test-large-8-gpu-amd-mi35x (…, 0), run 27729699748. Failed 5/8 in window, never-passed since Jun 16 (≥9 consecutive). avg_spec_accept_length ≈ 1.44 < 2.04 (accuracy 0.95 still passes). [MEDIUM] aiter default bump 7a8ff7dd4→7d604afe5 via #28249 (last green ran old aiter, every fail runs new — verified via [CI-AITER-CHECK]); disconfirming: 49 sglang commits also in the window, no scout A/B baseline to isolate aiter.

• In-flight fix: ❌ none found.

Regression candidates

(none) — the bucket shows no latest-run-only failures; the latest completed run 27842786491 introduced nothing new.

Flakes / intermittent

• Diffusion-server infra (multimodal-gen 1-gpu shards 1/2, 2-gpu shards 0/2): NCCL-init hang → port-5555 --strict-ports cascade, lost/terminated runner, HF-Hub 429 + Aliyun-OSS image-download failures, ZImage warmup-timeout. Several shards, ~1–3/8 each — retry; harden harness to free port 5555 on startup failure.
• AMD runner "VRAM not clear / zombie KFD context": jit-kernel-unit-test-amd (1/8), stage-b-test-1-gpu-large-amd (…,2) (1/8), stage-b-test-1-gpu-small-amd (…,3) (1/8) — pre-test infra abort; investigate runner cleanup.
• Orchestration cancellations: call-pr-test-amd-extra / extra-a-test-1-gpu-large/small-amd (2/8, 1/8) — runner-starvation / fast-fail cascade, not test failures; retry.
• 8-GPU intermittent: stage-c (…,2) RCCL SIGSEGV + mori libabsl invalid-ELF build fail; stage-c (…,0) / stage-b-small-mi35x HF-cache file-lock weight-load hang — investigate the absl/ELF build corruption.
• sgl-kernel-unit-test-amd (1/8): MoE topk-softmax exact-value mismatch on unseeded random input — quarantine/seed the test.

Suggested next actions

1. Land in-flight #28226 to clear cluster 2 (stage-c shard 3) cheaply — root cause is a [FACT].
2. A/B aiter 7a8ff7dd4 vs 7d604afe5 on MI35x for cluster 3 (test_deepseek_r1_mxfp4_8gpu.py) at fixed sglang HEAD to confirm/deny the aiter-bump hypothesis; no fix PR exists yet.
3. For cluster 1 flux2, reproduce at TP2 on MI325 and dump operand/scale shapes at fp8_utils.py:1681; decide whether to gate the new modelopt-FP8-TP2 case off AMD until hipBLASLt supports it.
4. Provision gated-HF access for ideogram-ai/ideogram-4-fp8 (or skip on CI) to stop the persistent 403 on multimodal-gen-2gpu shard 1.
5. Investigate the AMD runner "VRAM not clear / zombie KFD" cleanup and the mori/absl invalid-ELF build corruption, since both produce cross-shard infra flakes rather than real test signal.

---

Runs analysed

Run	Started (UTC)	Failed jobs
27729699748	2026-06-18 00:59	5
27744699827	2026-06-18 08:35	6
27762119155	2026-06-18 13:16	5
27782126506	2026-06-18 18:52	8
27799027960	2026-06-19 01:46	4
27813231755	2026-06-19 07:57	5
27828002206	2026-06-19 13:26	4
27842786491	2026-06-19 18:45	3

Persistent failures (in every run)

• multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 1) — in 8/8 runs (9699748, 4699827, 2119155, 2126506, 9027960, 3231755, 8002206, 2786491)

Latest-only failures (potential regressions)

(none)

Flaky / intermittent failures

• call-pr-test-amd-extra / extra-a-test-1-gpu-large-amd (linux-mi325-1gpu-sglang) — in 2/8 runs (4699827, 2126506)
• call-pr-test-amd-extra / extra-a-test-1-gpu-small-amd (linux-mi325-1gpu-sglang) — in 1/8 runs (2126506)
• jit-kernel-unit-test-amd (linux-mi325-1gpu-sglang) — in 1/8 runs (3231755)
• multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 1) — in 2/8 runs (2119155, 2786491)
• multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 2) — in 3/8 runs (4699827, 2126506, 2786491)
• multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 0) — in 1/8 runs (9027960)
• multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 2) — in 1/8 runs (3231755)
• sgl-kernel-unit-test-amd (linux-mi325-1gpu-sglang) — in 1/8 runs (8002206)
• stage-b-test-1-gpu-large-amd (linux-mi325-1gpu-sglang, 2) — in 1/8 runs (3231755)
• stage-b-test-1-gpu-small-amd (linux-mi325-1gpu-sglang, 13) — in 1/8 runs (9027960)
• stage-b-test-1-gpu-small-amd (linux-mi325-1gpu-sglang, 3) — in 1/8 runs (3231755)
• stage-b-test-1-gpu-small-amd (linux-mi325-1gpu-sglang, 5) — in 2/8 runs (4699827, 2119155)
• stage-b-test-1-gpu-small-amd (linux-mi325-1gpu-sglang, 9) — in 1/8 runs (8002206)
• stage-b-test-1-gpu-small-amd-mi35x (linux-mi35x-gpu-1) — in 1/8 runs (9699748)
• stage-c-test-large-8-gpu-amd (linux-mi325-8gpu-sglang, 0) — in 1/8 runs (2126506)
• stage-c-test-large-8-gpu-amd (linux-mi325-8gpu-sglang, 2) — in 2/8 runs (9699748, 2126506)
• stage-c-test-large-8-gpu-amd (linux-mi325-8gpu-sglang, 3) — in 5/8 runs (9699748, 4699827, 2119155, 2126506, 9027960)
• stage-c-test-large-8-gpu-amd-mi35x (linux-mi35x-gpu-8, 0) — in 5/8 runs (9699748, 4699827, 2119155, 2126506, 8002206)

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-06-20 00:06 UTC)

[amd-bot]

pr-test-amd-rocm720.yml — 89 failure(s)

Run started (UTC): 2026-06-18 00:44 UTC
Last scanned (UTC): 2026-06-19 19:28 UTC
Commits: sglang ca88b7f

Summary

Counts: 89 failed jobs · 8 clusters · ~70 jobs are benign concurrency cancellations (persistent ≥2 days) · 19 jobs in the one run that actually executed (27841438145), yielding ~17 genuine test failures across 6 real clusters

Note: For pr-test-amd-rocm720.yml, every recent scheduled run on main is self-cancelled by the next scheduled run via the shared cancel-in-progress concurrency group. ~70 of the 89 "failed" jobs are this orchestration artifact (conclusion=cancelled, 0 steps, BlobNotFound), not test failures. All genuine failures come from the single run that got far enough to execute tests before being cancelled.

#	Cluster	Job	Test File	Test Function	Status	Hypothesis (confidence)
1	Concurrency cancellation (no test ran)	stage-b-1gpu-small (5) +~69 more	N/A	N/A	~70 jobs; every scheduled run cancelled since Jun 17	[FACT] next scheduled run cancels prior via cancel-in-progress
2	Inductor AssertionError in decompose_triton_kernel_wrapper_functional (ROCm diffusion)	multimodal-gen-2gpu (0)	sglang/multimodal_gen/test/server/test_server_2_gpu.py	test_diffusion_generation[wan2_1_t2v_14b_2gpu] (+1.3b_cfg_parallel)	no baseline (job skipped in prior runs)	[MEDIUM] torch-inductor ↔ triton-custom skew (specialize_impl ImportError)
3	Inductor AssertionError (ROCm diffusion)	multimodal-gen-2gpu (1)	test_server_2_gpu.py	wan2_2_t2v_a14b_2gpu, ...teacache_2gpu, ...lora_2gpu, wan2_1_..._cache_dit_sp_only_2gpu	no baseline	[MEDIUM] triton-custom toolchain skew
4	Inductor AssertionError (ROCm diffusion)	multimodal-gen-1gpu (2)	sglang/multimodal_gen/test/server/test_server_1_gpu.py	qwen_image_t2i, qwen_image_edit_2509_ti2i, wan2_1_t2v_1.3b	no baseline	[MEDIUM] triton-custom toolchain skew
5	Inductor AssertionError (ROCm diffusion)	multimodal-gen-1gpu (0)	test_server_1_gpu.py	wan2_1_t2v_1.3b_teacache_enabled	no baseline (logs expired)	[MEDIUM] triton-custom toolchain skew
6	Inductor AssertionError (ROCm diffusion)	multimodal-gen-1gpu (1)	test_server_1_gpu.py	qwen_image_t2i_cache_dit_enabled	no baseline	[MEDIUM] triton-custom toolchain skew
7	MORI build failure — libabsl_time.so invalid ELF header	stage-b-1gpu-small (3) (root) +~8 cascade	N/A (dep-build, not a test)	N/A	infra; cascades fast-fail to stage-c shards	[FACT] corrupt system lib in container during gtest discovery
8	Triton offline throughput below AMD perf floor	stage-b-1gpu-large (0)	test/registered/perf/test_bench_serving_1gpu_part1.py	test_offline_throughput_with_triton_attention_backend	first execution in window; no green baseline	[LOW] ~9% under floor (3179<3500); threshold may be mis-calibrated
9	VLM offline throughput below floor	stage-b-1gpu-small (7)	test/registered/perf/test_vlm_perf_5090.py	TestVLMPerf5090.test_vlm_offline_throughput	deterministic (1196<2000, both retries); no baseline	[LOW] hardcoded floor likely too high for MI3xx
10	Segfault in torch.ones during distributed init (TP1)	stage-b-2gpu-large (0)	test/registered/perf/test_bench_serving_2gpu.py	test_moe_offline_throughput_default	never-passed/undetermined; 5/5 SIGSEGV	[LOW] HIP/ROCm-7.2 runtime fault on first GPU alloc
11	hipBLAS FP8 GEMM NOT_SUPPORTED	multimodal-gen-2gpu (1)	test_server_2_gpu.py	test_diffusion_generation[flux2_modelopt_fp8_tp2_t2i]	no baseline	[LOW] FP8 GEMM unsupported on this hipBLAS
12	HF gated-repo 403 (access)	multimodal-gen-2gpu (1)	test_server_2_gpu.py	test_diffusion_generation[ideogram4_fp8_tp2_t2i]	infra/access, not code	[FACT] GatedRepoError: 403 on ideogram-ai/ideogram-4-fp8

Cluster: Concurrency cancellation (no test ran)

• Affected: ~70 jobs across runs 27780231199, 27761630959 and the dsv4/extra jobs (dsv4-pro-fp4, dsv4-flash-fp4-fp8, extra-a-1gpu-small, …).
• Shared facts: conclusion=cancelled, steps:[], log BlobNotFound. The next scheduled run is created ~1 s before the cancellation cascade; all 39 jobs of a run cancel at one timestamp. The finish gate's exit 1 is just an aggregation of cancelled upstreams.
• Top hypothesis: [FACT] scheduled runs share concurrency group pr-test-amd-rocm720-refs/heads/main with cancel-in-progress: true; the 6-hourly + daily crons overlap and supersede each other. Disconfirming: none — verified via run created_at vs cancellation timestamps.
• In-flight fix: ❌ none found — this is a workflow-config issue (per-run concurrency group needed).
• Suggested triage: give scheduled runs a per-run concurrency key (or stagger crons) so a run completes before the next is created; without it, this workflow produces no usable test signal.

Cluster: Inductor AssertionError in decompose_triton_kernel_wrapper_functional (ROCm diffusion)

• Affected: 5 multimodal-gen jobs (rows 2–6), ~12 test_diffusion_generation params across test_server_1_gpu.py / test_server_2_gpu.py.
• Shared facts: identical traceback ending at torch/_inductor/pattern_matcher.py:314 (node-count assert) via post_grad.py decompose_triton_kernel_wrapper_functional; recurring warning ImportError: cannot import name 'specialize_impl' from 'triton.runtime.jit' (/sgl-workspace/triton-custom/...). enable_torch_compile=false, so the compile is internal to a custom-op (e.g. layernorm) path. Image rocm/sgl-dev:v0.5.13.post1-rocm720-mi30x-20260619.
• Top hypothesis: [MEDIUM] torch-inductor ↔ triton-custom version skew in the rocm720 image. Disconfirming: no green baseline exists (prior runs skipped/cancelled, logs expired), so a recent sglang change to the custom-op path can't be fully excluded.
• In-flight fix: candidate #21839 ("[AMD] ROCm Dockerfile update 720 base image") may refresh the torch/triton stack — unverified; ❌ no PR for the Inductor assert itself.
• Suggested triage: in the image run python -c "from triton.runtime.jit import specialize_impl"; if it fails, A/B triton-custom vs stock triton and evaluate #21839. Reproduce a single param (wan2_1_t2v_14b_2gpu) in isolation.

Cluster: MORI build failure — libabsl_time.so invalid ELF header

• Affected: root in stage-b-1gpu-small (3); fast-fail/cascade-cancelled siblings include stage-c-4gpu, stage-b-2gpu-large (1), stage-c-8gpu (0/2/3), stage-c-mi35x (0/1), stage-b-1gpu-small (6).
• Shared facts: error while loading shared libraries: /lib/x86_64-linux-gnu/libabsl_time.so.20210324: invalid ELF header (Result 127) during MORI gtest_discover_tests running test_external_kv_block_index at build time, inside scripts/ci/amd/amd_ci_install_dependency.sh. No sglang test executed.
• Top hypothesis: [FACT] corrupt/truncated system library in the container/runner image. Disconfirming: none — it's a build-env error, not code.
• In-flight fix: ❌ none found (infra/image issue, not a code PR).
• Suggested triage: rebuild/repull the rocm720 image on the affected runner; verify libabsl_time.so ELF integrity; confirm whether it's a single-runner corruption or image-wide.

Cluster: perf-threshold misses (Triton row 8, VLM row 9)

• Shared facts: both are deterministic threshold misses (all retries identical), both lack any green baseline in the scheduled window (jobs were skipped before).
• Top hypothesis: [LOW] mis-calibrated AMD floors for these partitions, not code regressions. Disconfirming: cannot prove they ever passed >floor on this runner.
• In-flight fix: ❌ none found (searched test_bench_serving throughput, triton throughput threshold).
• Suggested triage: re-run each 2–3× on the same MI325 rocm720 container; if throughput stays in the failing band, re-calibrate/quarantine the threshold rather than bisecting.

Cluster: Segfault in torch.ones during distributed init (row 10)

• Shared facts: SIGSEGV (exit -11) inside at::native::fill_kernel_cuda ← torch.ones at parallel_state.py:301 (first GPU alloc on TP1), 5/5 launch attempts; ROCm 7.2 MI325 2-GPU.
• Top hypothesis: [LOW] HIP/ROCm-7.2 runtime/driver fault on first device kernel launch. Disconfirming: no passing baseline; can't exclude an sglang init-path change.
• In-flight fix: ❌ none cited.
• Suggested triage: run a minimal torch.ones(..., device='cuda') + TP-init repro in the image; if it segfaults standalone, it's a runtime/driver issue.

Suggested triage order

Rank	Triage step	Targets	Why this first
1	Fix the scheduled-run concurrency config (per-run group / stagger crons)	Cluster 1 (~70 jobs)	Largest blast radius; until fixed the workflow yields no end-to-end signal and masks real failures as cancelled
2	Verify libabsl_time.so / rebuild rocm720 image on affected runner	Cluster 7 (root + ~8 cascade)	Single env fix unblocks all stage-c shards via fast-fail chain
3	Check triton-custom vs torch-inductor in image; evaluate #21839	Clusters 2–6 (~12 diffusion params)	One toolchain root likely explains all Inductor asserts
4	Re-run + re-calibrate perf gates; standalone repro for the TP1 segfault	Rows 8, 9, 10	Cheap reruns separate mis-calibration/runtime-fault from code regressions

---

Job	Failed Steps	Started
call-pr-test-amd-extra-rocm720 / extra-a-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang)	N/A	2026-06-18T18:18
dsv4-flash-fp4-fp8-amd-rocm720	N/A	2026-06-18T18:18
dsv4-pro-fp4-amd-rocm720	N/A	2026-06-18T18:18
call-pr-test-amd-extra-rocm720 / extra-a-test-1-gpu-large-amd-rocm720 (linux-mi325-1gpu-sglang)	N/A	2026-06-18T18:18
stage-a-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang)	N/A	2026-06-18T18:18
stage-b-test-1-gpu-large-amd-rocm720 (linux-mi325-1gpu-sglang, 0)	N/A	2026-06-18T18:18
stage-b-test-2-gpu-large-amd-rocm720 (linux-mi325-2gpu-sglang, 1)	N/A	2026-06-18T18:18
stage-b-test-1-gpu-small-amd-nondeterministic-rocm720 (linux-mi325-1gpu-sglang)	N/A	2026-06-18T18:18
stage-b-test-2-gpu-large-amd-rocm720 (linux-mi325-2gpu-sglang, 0)	N/A	2026-06-18T18:18
stage-c-test-large-8-gpu-amd-mi35x-rocm720 (linux-mi35x-gpu-8, 0)	N/A	2026-06-18T18:18
stage-b-test-1-gpu-large-amd-rocm720 (linux-mi325-1gpu-sglang, 1)	N/A	2026-06-18T18:18
stage-c-test-large-8-gpu-amd-rocm720 (linux-mi325-8gpu-sglang, 0)	N/A	2026-06-18T18:18
stage-b-test-large-8-gpu-mi35x-disaggregation-amd-rocm720 (linux-mi35x-gpu-8.fabric)	N/A	2026-06-18T18:18
stage-c-test-large-8-gpu-amd-mi35x-rocm720 (linux-mi35x-gpu-8, 1)	N/A	2026-06-18T18:18
multimodal-gen-test-2-gpu-amd-rocm720 (linux-mi325-2gpu-sglang, 1)	N/A	2026-06-18T18:18
multimodal-gen-test-1-gpu-amd-rocm720 (linux-mi325-1gpu-sglang, 3)	N/A	2026-06-18T18:18
stage-c-test-large-8-gpu-amd-rocm720 (linux-mi325-8gpu-sglang, 2)	N/A	2026-06-18T18:18
multimodal-gen-test-1-gpu-amd-rocm720 (linux-mi325-1gpu-sglang, 1)	N/A	2026-06-18T18:18
multimodal-gen-test-1-gpu-amd-rocm720 (linux-mi325-1gpu-sglang, 2)	N/A	2026-06-18T18:18
stage-c-test-large-8-gpu-amd-rocm720 (linux-mi325-8gpu-sglang, 3)	N/A	2026-06-18T18:18
stage-b-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang, 4)	N/A	2026-06-18T18:18
multimodal-gen-test-2-gpu-amd-rocm720 (linux-mi325-2gpu-sglang, 2)	N/A	2026-06-18T18:18
stage-c-test-large-8-gpu-amd-rocm720 (linux-mi325-8gpu-sglang, 1)	N/A	2026-06-18T18:18
multimodal-gen-test-2-gpu-amd-rocm720 (linux-mi325-2gpu-sglang, 0)	N/A	2026-06-18T18:18
stage-b-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang, 8)	N/A	2026-06-18T18:18
stage-b-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang, 11)	N/A	2026-06-18T18:18
stage-b-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang, 6)	N/A	2026-06-18T18:18
stage-b-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang, 2)	N/A	2026-06-18T18:18
multimodal-gen-test-1-gpu-amd-rocm720 (linux-mi325-1gpu-sglang, 0)	N/A	2026-06-18T18:18
stage-b-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang, 9)	N/A	2026-06-18T18:18
stage-b-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang, 13)	N/A	2026-06-18T18:18
stage-b-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang, 10)	N/A	2026-06-18T18:18
stage-b-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang, 5)	N/A	2026-06-18T18:18
stage-b-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang, 12)	N/A	2026-06-18T18:18
stage-b-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang, 3)	N/A	2026-06-18T18:18
stage-b-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang, 1)	N/A	2026-06-18T18:18
stage-b-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang, 0)	N/A	2026-06-18T18:18
stage-c-test-4-gpu-amd-rocm720	N/A	2026-06-18T18:47
dsv4-pro-fp4-amd-rocm720	N/A	2026-06-18T13:07
dsv4-flash-fp4-fp8-amd-rocm720	N/A	2026-06-18T13:07
call-pr-test-amd-extra-rocm720 / extra-a-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang)	N/A	2026-06-18T13:08
dsv4-flash-fp4-fp8-amd-rocm720	N/A	2026-06-18T07:39
call-pr-test-amd-extra-rocm720 / extra-a-test-1-gpu-large-amd-rocm720 (linux-mi325-1gpu-sglang)	N/A	2026-06-18T13:08
dsv4-pro-fp4-amd-rocm720	N/A	2026-06-18T07:39
call-pr-test-amd-extra-rocm720 / extra-a-test-1-gpu-large-amd-rocm720 (linux-mi325-1gpu-sglang)	N/A	2026-06-18T07:39
dsv4-flash-fp4-fp8-amd-rocm720	N/A	2026-06-18T00:44
call-pr-test-amd-extra-rocm720 / extra-a-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang)	N/A	2026-06-18T07:39
dsv4-pro-fp4-amd-rocm720	N/A	2026-06-18T00:44
call-pr-test-amd-extra-rocm720 / extra-a-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang)	N/A	2026-06-18T00:44
stage-b-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang, 7)	N/A	2026-06-18T18:18
dsv4-pro-fp4-amd-rocm720	N/A	2026-06-18T20:17
dsv4-flash-fp4-fp8-amd-rocm720	N/A	2026-06-18T20:17
call-pr-test-amd-extra-rocm720 / extra-a-test-1-gpu-large-amd-rocm720 (linux-mi325-1gpu-sglang)	N/A	2026-06-19T00:48
call-pr-test-amd-extra-rocm720 / extra-a-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang)	N/A	2026-06-19T00:48
dsv4-pro-fp4-amd-rocm720	N/A	2026-06-19T00:47
dsv4-flash-fp4-fp8-amd-rocm720	N/A	2026-06-19T00:47
call-pr-test-amd-extra-rocm720 / extra-a-test-1-gpu-large-amd-rocm720 (linux-mi325-1gpu-sglang)	N/A	2026-06-19T07:50
call-pr-test-amd-extra-rocm720 / extra-a-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang)	N/A	2026-06-19T07:50
dsv4-pro-fp4-amd-rocm720	N/A	2026-06-19T07:50
dsv4-flash-fp4-fp8-amd-rocm720	N/A	2026-06-19T07:50
call-pr-test-amd-extra-rocm720 / extra-a-test-1-gpu-large-amd-rocm720 (linux-mi325-1gpu-sglang)	N/A	2026-06-19T13:10
call-pr-test-amd-extra-rocm720 / extra-a-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang)	N/A	2026-06-19T13:10
stage-b-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang, 3)	Install dependencies	2026-06-19T18:08
dsv4-pro-fp4-amd-rocm720	N/A	2026-06-19T13:10
dsv4-flash-fp4-fp8-amd-rocm720	N/A	2026-06-19T13:10
dsv4-flash-fp4-fp8-amd-rocm720	N/A	2026-06-19T18:07
dsv4-pro-fp4-amd-rocm720	N/A	2026-06-19T18:07
stage-b-test-1-gpu-large-amd-rocm720 (linux-mi325-1gpu-sglang, 1)	Run test	2026-06-19T18:07
stage-c-test-large-8-gpu-amd-mi35x-rocm720 (linux-mi35x-gpu-8, 0)	N/A	2026-06-19T18:07
stage-c-test-large-8-gpu-amd-mi35x-rocm720 (linux-mi35x-gpu-8, 1)	N/A	2026-06-19T18:07
stage-b-test-1-gpu-large-amd-rocm720 (linux-mi325-1gpu-sglang, 0)	Run test	2026-06-19T18:07
multimodal-gen-test-2-gpu-amd-rocm720 (linux-mi325-2gpu-sglang, 0)	Run diffusion server tests (2-GPU)	2026-06-19T18:08
multimodal-gen-test-2-gpu-amd-rocm720 (linux-mi325-2gpu-sglang, 1)	Run diffusion server tests (2-GPU)	2026-06-19T18:07
multimodal-gen-test-1-gpu-amd-rocm720 (linux-mi325-1gpu-sglang, 0)	Run diffusion server tests (1-GPU)	2026-06-19T18:07
stage-b-test-2-gpu-large-amd-rocm720 (linux-mi325-2gpu-sglang, 0)	Run test	2026-06-19T18:16
multimodal-gen-test-1-gpu-amd-rocm720 (linux-mi325-1gpu-sglang, 2)	Run diffusion server tests (1-GPU)	2026-06-19T18:07
stage-b-test-2-gpu-large-amd-rocm720 (linux-mi325-2gpu-sglang, 1)	Install dependencies	2026-06-19T18:28
stage-c-test-large-8-gpu-amd-rocm720 (linux-mi325-8gpu-sglang, 3)	Start CI container	2026-06-19T18:32
multimodal-gen-test-1-gpu-amd-rocm720 (linux-mi325-1gpu-sglang, 1)	Run diffusion server tests (1-GPU)	2026-06-19T18:07
stage-c-test-large-8-gpu-amd-rocm720 (linux-mi325-8gpu-sglang, 2)	N/A	2026-06-19T18:07
stage-c-test-large-8-gpu-amd-rocm720 (linux-mi325-8gpu-sglang, 1)	N/A	2026-06-19T18:07
stage-c-test-large-8-gpu-amd-rocm720 (linux-mi325-8gpu-sglang, 0)	Start CI container	2026-06-19T18:34
stage-b-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang, 6)	Run test	2026-06-19T18:08
stage-b-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang, 13)	Run test	2026-06-19T18:07
stage-b-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang, 7)	Run test	2026-06-19T18:08
stage-b-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang, 11)	Run test	2026-06-19T18:08
stage-c-test-4-gpu-amd-rocm720	N/A	2026-06-19T18:36
call-pr-test-amd-extra-rocm720 / extra-a-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang)	N/A	2026-06-19T18:36
call-pr-test-amd-extra-rocm720 / extra-a-test-1-gpu-large-amd-rocm720 (linux-mi325-1gpu-sglang)	N/A	2026-06-19T18:36

Per-Job Analysis

call-pr-test-amd-extra-rocm720 / extra-a-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang) — failed step(s): N/A

Commit Info

• sglang: f7632ef8
• aiter: N/A (job was cancelled mid-run; no log was persisted, so no [CI-AITER-CHECK] markers are available)

Failed Tests

Test File	Test Function	Error	Log
N/A	N/A	Job cancelled (not failed) — no test executed to completion	Job

No test failure occurred. The job conclusion is cancelled, it recorded zero steps, and its log blob returns BlobNotFound (logs were never persisted because the job was killed mid-execution).

Failure Summary

This job was not a test failure — it was cancelled ~29 min into execution (started 18:18:17Z, cancelled 18:47:22Z). All 39 in-progress jobs in run 27780231199 were cancelled at the exact same instant (18:47:22Z), one second after the next scheduled run 27781870163 (sha 61a8b42c) was created at 18:47:21Z. This is GitHub Actions concurrency cancellation, not a code or test regression.

Regression Status

Infrastructure issue (CI orchestration / concurrency cancellation) — not a code regression.

Recent pr-test-amd-rocm720.yml scheduled runs on main (event=schedule, completed only):

Date (UTC)	Run	Conclusion	Note
Jun 18 18:47	27781870163	queued/in-progress	The run that cancelled this one
Jun 18 18:18	27780231199	❌ cancelled	This run — cancelled by the 18:47 run
Jun 18 13:07	27761630959	❌ cancelled	cancelled by this run
Jun 18 07:39	27744325971	❌ cancelled	cancelled by the 13:07 run
Jun 18 00:44	27729179568	❌ cancelled	—

Every scheduled run in the queryable window concluded cancelled. Each new scheduled run cancels its still-running predecessor.

Failure Cluster

Concurrency cancellation of scheduled run (next schedule cron supersedes in-progress run; not a test failure)

Facts (verified observations)

• Job 82202881528 conclusion = cancelled, steps: [] (empty), log blob = BlobNotFound. (verified via Jobs API + log fetch)
• Run 27780231199: 7 success, 39 cancelled, 2 failure. The 2 "failure" jobs are the aggregator gates call-pr-test-amd-extra-rocm720 / pr-test-amd-extra-finish (82208456903) and pr-test-amd-rocm720-finish (82225249578) — they fail because their upstream needs were cancelled, not on their own merits. (verified via Jobs API)
• All 39 cancelled jobs share completed_at = 2026-06-18T18:47:22Z (simultaneous, workflow-level cancel). (verified)
• Next scheduled run 27781870163 created_at = 2026-06-18T18:47:21Z — 1 s before the cancellation. (verified via workflow runs API)
• Workflow concurrency config (.github/workflows/pr-test-amd-rocm720.yml:122-123): for schedule events the group resolves to pr-test-amd-rocm720-${{ github.ref }} (= refs/heads/main) and cancel-in-progress: true. So any two schedule runs on main share one group and the newer cancels the older. (verified by reading the workflow)

Hypothesised Causes (with confidence)

• [FACT] The job was terminated by GitHub Actions concurrency cancellation, triggered when the next scheduled run 27781870163 entered the shared pr-test-amd-rocm720-refs/heads/main concurrency group with cancel-in-progress: true.
  • Supporting: new run created 18:47:21Z; all in-progress jobs cancelled 18:47:22Z; workflow config has cancel-in-progress: true for schedule events on a github.ref-keyed group; every prior scheduled run is also cancelled.
  • Disconfirming: (none found) — no test executed to failure, no error in any non-aggregator job.
  • How to verify: not needed; this is directly observable from the run/job metadata and the workflow concurrency block.

In-flight Fix Check

No fix is warranted for an individual cancelled job (this is expected concurrency behavior). If the maintainer wants scheduled runs to stop cancelling each other, that is a workflow-config change, not a bug fix. Searched open PRs for related config work:

• In-flight fix: ❌ none found — curl ".../search/issues?q=repo:sgl-project/sglang+is:pr+is:open+pr-test-amd-rocm720+concurrency" returned no matching open PR.

Suggested Triage Steps

• No code action required for this job — the cancellation is a CI-orchestration artifact, not a regression. Treat it as noise in failure tracking.
• If scheduled-run cancellation is undesirable, consider whether the two crons (30 17 * * * and 0 */6 * * *) plus multi-hour run durations cause chronic overlap; an option is a per-run-id concurrency group (or cancel-in-progress: false) for schedule events — a maintainer decision, not a directive.
• When this job recurs in monitoring with conclusion cancelled + empty steps + missing log, classify it as infrastructure/concurrency, not a test failure, and exclude it from regression trends.

Status

• Persistence: Chronic by design — every pr-test-amd-rocm720.yml scheduled run in the queryable window (Jun 16–18) concluded cancelled.
• Scope: Run-level, not test-level — 39 jobs in run 27780231199 cancelled simultaneously; affects all schedule runs on main.
• Blocked work: This scheduled run produced no AMD ROCm 7.2 CI signal (cancelled before tests completed); the immediately-following run carries the signal instead.
• In-flight fix: ❌ none found — none needed unless maintainers choose to change schedule concurrency behavior.

dsv4-flash-fp4-fp8-amd-rocm720 — failed step(s): N/A

Commit Info

• sglang: f7632ef8 ("[Cookbook] Laguna-M.1: enable FP8 on Blackwell + drop provisional AIME numbers" #28664)
• aiter: N/A (no [CI-AITER-CHECK] markers — the job log was never uploaded; see below)

Failed Tests

Test File	Test Function	Error	Log
N/A — no test failure	N/A	Job was cancelled mid-execution (conclusion cancelled, not failure); log blob does not exist (BlobNotFound)	Job

Failure Summary

The job dsv4-flash-fp4-fp8-amd-rocm720 did not fail a test — its conclusion is cancelled. It started at 18:18:15Z, ran ~29 min, and was cancelled at 18:47:22Z together with ~38 sibling jobs in the same run (run 27780231199 overall conclusion cancelled). The job log was never uploaded (the API returns BlobNotFound), consistent with a mid-run cancellation rather than a test crash.

Regression Status

Infrastructure issue (concurrency cancellation) — not a test regression.

[FACT] The next scheduled run of the same workflow, 27781870163 (sha 61a8b42c, #28668), was created at 18:47:21Z — exactly one second before the mass cancellation of our run's jobs at 18:47:22Z.

[FACT] The workflow concurrency config (.github/workflows/pr-test-amd-rocm720.yml:122-123) puts every schedule-triggered run into the shared group pr-test-amd-rocm720-refs/heads/main with cancel-in-progress: true. The newer scheduled run therefore cancelled the in-progress older run.

[FACT] This is systemic, not a one-off: every scheduled run of this workflow in the last ~2 days carries conclusion cancelled (runs 27761630959, 27744325971, 27729179568, …) because each scheduled run is superseded by the next.

Cancellation timeline (same branch main, event=schedule, completed runs):

Time (UTC)	Event	Run
18:18:02	Our run created (sha f7632ef8)	27780231199
18:18:15	dsv4-flash-fp4-fp8-amd-rocm720 starts	job
18:47:21	Next scheduled run created (sha 61a8b42c) → same concurrency group	27781870163
18:47:22	~38 jobs (incl. our target) cancelled simultaneously	—

Failure Cluster

Concurrency cancellation by superseding scheduled run (no test failure).

Facts (verified observations)

• Target job conclusion = cancelled; steps list empty; html_url job page reachable but log API returns <Error><Code>BlobNotFound</Code> (verified via /actions/jobs/82202873671/logs).
• Run 27780231199 overall: 7 success, 39 cancelled, 2 failure. The 2 failure jobs are aggregator gates only — call-pr-test-amd-extra-rocm720 / pr-test-amd-extra-finish (82208456903) and pr-test-amd-rocm720-finish (82225249578); both failed at step "Check all dependent job statuses" because their dependencies were cancelled — i.e. downstream of the cancellation, not a real test failure.
• 38 jobs share the exact cancellation timestamp 18:47:22Z; the superseding run was created at 18:47:21Z (1-second delta).
• A few kernel-unit-test jobs (different stage, not gated) kept running and passed afterward (e.g. sgl-kernel-unit-test-amd-rocm720 until 20:17:17Z), so this was a selective in-progress cancellation, not a hard run kill.

Hypothesised Causes (with confidence)

• [FACT] Concurrency-group cancellation: the next scheduled run entered the same cancel-in-progress group and superseded this run.
  • Supporting: 1-second timing alignment between new-run creation (18:47:21Z) and mass cancellation (18:47:22Z); concurrency config at pr-test-amd-rocm720.yml:122-123; every historical scheduled run is cancelled.
  • Disconfirming: (none found) — there is no test error, stack trace, or non-zero test exit anywhere; the log was never even uploaded.
  • How to verify: compare created_at of run 27781870163 vs the cancellation time of any cancelled job in run 27780231199 (already done above).

There is no test-level failure attributable to commit f7632ef8 (a docs/cookbook-only change) — the job never completed.

In-flight Fix Check

This is an orchestration artifact, not a code bug, so a "fix" is a CI-config decision rather than a source fix. Searched open PRs for relevant keywords:

curl -s -H "Authorization: token $GH_PAT" "https://api.github.com/search/issues?q=repo:sgl-project/sglang+is:pr+is:open+pr-test-amd-rocm720+concurrency"

• In-flight fix: ❌ none found — no open PR addressing scheduled-run concurrency cancellation for pr-test-amd-rocm720.yml.

Suggested Triage Steps

• No action needed on the code. The target job was collateral of a concurrency cancellation, not a regression. Commit f7632ef8 is cookbook docs only and cannot have caused a dsv4 accuracy failure.
• If a real dsv4 accuracy signal is wanted for sha f7632ef8: re-run the workflow via workflow_dispatch (which gets a unique concurrency group per pr-test-amd-rocm720.yml:122 and is therefore immune to supersession), then inspect dsv4-flash-fp4-fp8-amd-rocm720 to completion.
• If these back-to-back cancellations are unwanted noise: the two crons (30 17 * * * daily full + 0 */6 * * * dsv4-only) can land close together and the dsv4-only run will cancel an in-progress full run sharing the refs/heads/main group. Maintainers may want to give the 6h dsv4-only cron its own concurrency group (or cancel-in-progress: false) so it doesn't kill an in-progress full run. This is a CI-config judgement call, not a directive.

Status

• Persistence: Systemic — every scheduled run of pr-test-amd-rocm720.yml on main over the last ~48h shows conclusion cancelled due to back-to-back scheduled runs sharing one cancel-in-progress concurrency group.
• Scope: Run-orchestration level; affects ~38 in-progress jobs per superseded run. No test correctness impact.
• Blocked work: dsv4 accuracy + full AMD ROCm 7.2 stage signal is not produced on these superseded scheduled runs (they never finish). Kernel-unit-test jobs are unaffected and pass.
• In-flight fix: ❌ none found.

dsv4-pro-fp4-amd-rocm720 — failed step(s): N/A

Commit Info

• sglang: f7632ef8
• aiter: N/A — the job log blob does not exist (BlobNotFound); the job was cancelled before producing any step logs, so no [CI-AITER-CHECK] markers are available. (The run-level finish job shows AITER_COMMIT_OVERRIDE empty, i.e. the Dockerfile default would have been used.)

Failed Tests

This was not a test failure — the job never ran any tests. Job dsv4-pro-fp4-amd-rocm720 (conclusion: cancelled) started 2026-06-18T18:18:15Z and was cancelled at 2026-06-18T18:47:22Z (~29 min). The GitHub API records zero steps for it and the log blob returns BlobNotFound.

Failure Summary

The job did not fail on any test — it was cancelled along with all 39 test jobs in run 27780231199, every one of them killed at the same instant (18:47:22Z). The two finish aggregator jobs (pr-test-amd-rocm720-finish, pr-test-amd-extra-finish) then reported failure, but only because they exit 1 whenever an upstream need is cancelled — a downstream symptom, not an independent failure.

Regression Status

Infrastructure / CI-orchestration issue (not a code regression, not a flaky test).

Recent history of job dsv4-pro-fp4-amd-rocm720 (workflow pr-test-amd-rocm720.yml, branch main, event schedule, completed runs only):

Date (UTC)	Run	Job Conclusion	Duration	Note
Jun 18 18:18	27780231199	⏹️ cancelled	29 min	this run
Jun 18 13:07	27761630959	⏹️ cancelled	~5 h	—
Jun 18 07:39	27744325971	⏹️ cancelled	~5.5 h	—
Jun 18 00:44	27729179568	⏹️ cancelled	~7 h	—
Jun 17 19:22	27712035487	⏹️ cancelled	~5.5 h	—
Jun 17 13:13	27691581201	⏹️ cancelled	~5 h	—

This job is cancelled in every recent scheduled run — it never reaches a natural pass/fail. There is no green baseline and no observed test failure; the job is systematically terminated mid-flight.

Failure Cluster

Scheduled run self-cancelled by concurrency group (no test signal produced)

Facts (verified observations)

• Target job 82202873693: status=completed, conclusion=cancelled, 0 steps, log blob BlobNotFound (verified via API + log fetch). [FACT]
• All 39 test/matrix jobs in run 27780231199 were cancelled at the identical timestamp 2026-06-18T18:47:22Z; 7 jobs succeeded, 2 finish jobs failed. [FACT]
• The next scheduled run 27781870163 was created at 2026-06-18T18:47:21Z — one second before the cancellation cascade. [FACT]
• Workflow concurrency (from f7632ef8 .github/workflows/pr-test-amd-rocm720.yml:117-123): group pr-test-amd-rocm720-${{ … || github.ref }} → pr-test-amd-rocm720-refs/heads/main for schedule events, with cancel-in-progress: true for schedule triggers. [FACT]
• The workflow has two crons: 30 17 * * * (daily, full matrix) and 0 */6 * * * (every 6 h, dsv4 jobs only). Both fired ~48 min late (our run created 18:18:02 for the 17:30 cron; the superseding run created 18:47:21 for the 18:00 cron) and share the same refs/heads/main concurrency group. [FACT]
• The finish log (82225249578) shows its exit 1 was triggered by result == "cancelled" on upstream jobs (e.g. multimodal-gen-test-1-gpu-amd-rocm720: cancelled), confirming the finish "failure" is purely an aggregation of the cancellation. [FACT]

Hypothesised Causes

• [HIGH] The run was cancelled by GitHub Actions concurrency, not by any test or code defect. The daily 30 17 * * * cron run (ours) and the 6-hourly 0 */6 * * * cron run both map to the same concurrency group pr-test-amd-rocm720-refs/heads/main with cancel-in-progress: true; because both crons were delayed and landed ~29 min apart, the newer run 27781870163 (created 18:47:21Z) cancelled ours (18:47:22Z).
  • Supporting: exact 1-second alignment between the new run's creation and the cancellation cascade; concurrency config has cancel-in-progress: true for schedule events; every scheduled run in the queryable window ends in cancelled (steady-state, not specific to this commit); the cancellation hit all 39 jobs simultaneously.
  • Disconfirming: (none found) — no test produced an error; no log evidence of a crash or assertion.
  • How to verify: confirm via the run timeline/UI that run 27781870163 triggered the cancellation of 27780231199; inspect concurrency group membership in the Actions UI.
• [LOW] Any code-level cause tied to commit f7632ef8.
  • Supporting: it is the head SHA of the run.
  • Disconfirming: no test ran; the same cancellation pattern predates this commit across all recent scheduled runs.
  • How to verify: not actionable — there is no failure to attribute to code.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for pr-test-amd-rocm720 + concurrency (0 results) and rocm720 + cancel + schedule (1 result, #23957 "Add CDNA4 Gluon Extend Attention", an unrelated keyword match).

Suggested Triage Steps

• Confirm in the Actions UI that run 27781870163 cancelled run 27780231199 via the shared concurrency group (the "This run was cancelled because…" banner).
• Decouple the two cron schedules: give the 6-hourly dsv4-only cron a distinct concurrency group (e.g. include github.event.schedule in the group key) so it cannot cancel the daily full-matrix run, or set cancel-in-progress: false for schedule events. This would let dsv4-pro-fp4-amd-rocm720 actually run to completion and finally produce a pass/fail signal.
• Because no test signal exists for this job in any recent scheduled run, treat the dsv4 accuracy coverage on ROCm 7.2 as currently unverified until the concurrency collision is resolved.

Status

• Persistence: dsv4-pro-fp4-amd-rocm720 has been cancelled in every completed scheduled run in the queryable window (at least Jun 17 13:13 → Jun 18 18:18 UTC); it has never produced a natural pass/fail in that window.
• Scope: run-wide — all 39 test jobs of run 27780231199 were cancelled together; the pattern recurs on every pr-test-amd-rocm720.yml schedule run.
• Blocked work: dsv4 FP4 accuracy signal (and the full ROCm 7.2 matrix) on AMD is not being collected from scheduled runs due to self-cancellation.
• In-flight fix: ❌ none found — needs a workflow concurrency fix.

call-pr-test-amd-extra-rocm720 / extra-a-test-1-gpu-large-amd-rocm720 (linux-mi325-1gpu-sglang) — failed step(s): N/A

Commit Info

• sglang: f7632ef8
• aiter: N/A — the job log blob does not exist (BlobNotFound); the job was cancelled before producing/uploading a log, so no [CI-AITER-CHECK] markers are available.

Failed Tests

Test File	Test Function	Error	Log
N/A	N/A	Job cancelled (not failed) — no test ever executed to a verdict	Job

This was not a test failure. The target job extra-a-test-1-gpu-large-amd-rocm720 recorded zero steps (steps: []), conclusion = cancelled, started 18:18:17 UTC, cancelled 18:47:22 UTC. No log blob exists (<Error><Code>BlobNotFound</Code>), so no test file or function reached a pass/fail verdict.

Failure Summary

The job, its parent run 27780231199 (conclusion: cancelled), and 38 of its sibling jobs were cancelled, not failed. The run's two failure jobs are only the *-finish aggregators (pr-test-amd-rocm720-finish 82225249578, pr-test-amd-extra-finish 82208456903), which report failure because their upstream jobs were cancelled. No code-level or test-level failure occurred.

Regression Status

Infrastructure / orchestration issue (workflow concurrency cancellation) — not a code regression and not a flaky test.

Recent scheduled runs of pr-test-amd-rocm720.yml (branch=main, event=schedule, completed runs only):

Created (UTC)	Run	Conclusion	Cancelled at	Note
2026-06-18 18:18:02	27780231199	❌ cancelled	18:47:23	this run — cancelled when 18:00 6h-cron run started
2026-06-18 13:07:27	27761630959	❌ cancelled	18:18:10	cancelled when this run was created (18:18:02)
2026-06-18 07:39:23	27744325971	❌ cancelled	13:07:44	cancelled by next 6h-cron run
2026-06-18 00:44:16	27729179568	❌ cancelled	07:39:37	cancelled by next 6h-cron run
2026-06-17 18:48:37	27712035487	❌ cancelled	00:44:31	cancelled by next 6h-cron run

Every recent scheduled run ends in cancelled, each at the moment the next scheduled run is created. There is no green baseline in the queryable window because schedule runs systematically cancel one another.

Facts (verified observations)

• [FACT] Target job 82202881534: status=completed, conclusion=cancelled, steps: [], started 2026-06-18T18:18:17Z, completed 2026-06-18T18:47:22Z.
• [FACT] Job log is unavailable: the logs endpoint returns <Error><Code>BlobNotFound</Code>.
• [FACT] Parent run 27780231199: event=schedule, conclusion=cancelled. Of 48 jobs: 7 success, 39 cancelled, 2 failure (both are *-finish aggregators).
• [FACT] The next scheduled run 27781870163 was created 2026-06-18T18:47:21Z — one second before this job's cancellation timestamp (18:47:22Z).
• [FACT] Workflow pr-test-amd-rocm720.yml defines two schedule crons (.github/workflows/pr-test-amd-rocm720.yml:7-9): 30 17 * * * (daily) and 0 */6 * * * (every 6h).
• [FACT] Concurrency block (.github/workflows/pr-test-amd-rocm720.yml:117-123): for schedule events the group resolves to pr-test-amd-rocm720-refs/heads/main and cancel-in-progress evaluates to true (none of run_all_tests, workflow_call, workflow_dispatch apply).

Hypothesised Causes

• [HIGH] The cancellation is caused by GitHub Actions concurrency cancel-in-progress: the daily 30 17 cron run and the 0 */6 (18:00) cron run share concurrency group pr-test-amd-rocm720-refs/heads/main. The later-created 18:00 run cancelled all in-progress jobs of this (delayed 17:30) run.
  • Supporting: next run 27781870163 created 18:47:21Z; this job cancelled 18:47:22Z; concurrency config has cancel-in-progress: true for schedule; identical pattern across the last 5 scheduled runs (each cancelled exactly when the next is created).
  • Disconfirming: (none found) — the timing chain and config are consistent with every run in the window.
  • How to verify: inspect run 27781870163 — if its jobs ran the same matrix, it superseded this run. Confirm in the run's cancellation reason.

No test-level hypothesis applies — no test executed.

In-flight Fix Check

Searched open PRs for concurrency/cron-collision fixes on this workflow:

curl -s -H "Authorization: token $GH_PAT" "https://api.github.com/search/issues?q=repo:sgl-project/sglang+is:pr+is:open+pr-test-amd-rocm720+concurrency"

• In-flight fix: ❌ none found for the cron-collision cancellation. (Note: this is an orchestration design question, not a clear bug — a maintainer should decide whether the two crons colliding is intended.)

Suggested Triage Steps

• Confirm the cancellation is benign concurrency: check run 27781870163's cancellation/supersede reason and whether it covered the same matrix.
• If the daily 30 17 cron run should not be cancelled by the 18:00 0 */6 cron run, consider differentiating their concurrency groups (e.g. include the cron schedule or github.run_id in the group for the daily cron) — design decision for the maintainer.
• No source-code or aiter investigation is warranted: no test ran and no log exists.

Status

• Persistence: All 5 most-recent completed scheduled runs of pr-test-amd-rocm720.yml end in cancelled (2026-06-17 18:48 → 2026-06-18 18:18), each cancelled when the next scheduled run starts.
• Scope: Run-wide orchestration artifact — 39 of 48 jobs in run 27780231199 cancelled, including this job. Not specific to extra-a-test-1-gpu-large-amd-rocm720.
• Blocked work: Scheduled pr-test-amd-rocm720 runs are not producing usable signal — they are cancelled before completion, so no AMD ROCm 7.2 nightly coverage is being recorded from the daily cron.
• In-flight fix: ❌ none found — needs a maintainer decision on the cron/concurrency collision.

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-06-19 19:28 UTC)

[amd-bot]

pr-test-amd-rocm720.yml — Per-Job Analysis (continued 2/14)

stage-a-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang) — failed step(s): N/A

Commit Info

• sglang: f7632ef8
• aiter: N/A — the job never reached the dependency-install step, so no [CI-AITER-CHECK] markers were emitted; the log blob does not exist.

Failed Tests

Test File	Test Function	Error	Log
— (no test ran)	—	Job cancelled during runner setup; zero steps executed	Job (log blob BlobNotFound)

Failure Summary

The job stage-a-test-1-gpu-small-amd-rocm720 was cancelled, not failed. Its steps[] array is empty (no step ever started) and its log blob returns BlobNotFound, meaning it was terminated during runner provisioning. The cancellation was run-wide: of 48 jobs in run 27780231199, 39 were cancelled, 7 success, and 2 failure (the two *-finish aggregator jobs that fail when upstream jobs are cancelled). The run-level conclusion is cancelled.

Regression Status

Infrastructure issue (concurrency cancellation) — not a code regression.

Recent scheduled (event=schedule, branch=main) runs of pr-test-amd-rocm720.yml, completed runs only:

Date (UTC)	Run	Conclusion	Note
Jun 18 18:18	27780231199	❌ cancelled	this run
Jun 18 13:07	27761630959	❌ cancelled
Jun 18 07:39	27744325971	❌ cancelled
Jun 18 00:44	27729179568	❌ cancelled
Jun 17 18:48	27712035487	❌ cancelled

Over the last 20 completed scheduled runs: 18 cancelled, 1 failure, 1 startup_failure, 0 success. Scheduled runs of this workflow essentially never complete.

Failure Cluster

Run cancelled by concurrency (scheduled-run self-cancellation)

Facts (verified observations)

• [FACT] Job 82202905790 conclusion = cancelled; steps[] is empty; log endpoint returns BlobNotFound.
• [FACT] The job was cancelled at 2026-06-18T18:47:22Z. A newer scheduled run 27781870163 was created at 2026-06-18T18:47:21Z — one second earlier.
• [FACT] .github/workflows/pr-test-amd-rocm720.yml:122-123 sets, for schedule events, concurrency group: pr-test-amd-rocm720-refs/heads/main and cancel-in-progress: true (the run_all_tests / workflow_dispatch / workflow_call exclusions are all false for a schedule event).
• [FACT] Both the daily 30 17 * * * cron and the 6-hourly 0 */6 * * * cron (pr-test-amd-rocm720.yml:7-9) resolve to the same concurrency group on main, so consecutive scheduled runs evict each other.
• [FACT] 18 of the last 20 completed scheduled runs are cancelled.

Hypothesised Causes

• [HIGH] This run was cancelled by the next scheduled run entering the shared concurrency group pr-test-amd-rocm720-refs/heads/main (cancel-in-progress=true).
  • Supporting: superseding run 27781870163 created at 18:47:21Z, one second before this run's jobs were cancelled at 18:47:22Z; concurrency block at pr-test-amd-rocm720.yml:122-123 produces an identical group key for every schedule event on main; the AMD ROCm 7.2 full pipeline takes >30 min, longer than the gap between the delayed daily cron (fired ~18:18) and the 6h cron (~18:47), so the later run always preempts the earlier one.
  • Disconfirming: (none found) — timing, config, and the 18/20-cancelled pattern all converge.
• [LOW] Manual cancellation by a maintainer.
  • Supporting: cancellation is possible by hand.
  • Disconfirming: the one-second alignment with a new scheduled run's creation, and that 18/20 runs share the same fate, make a recurring manual cancel implausible.

In-flight Fix Check

Searched open PRs:

curl -s -H "Authorization: token $GH_PAT" "https://api.github.com/search/issues?q=repo:sgl-project/sglang+is:pr+is:open+pr-test-amd-rocm720+concurrency"

• In-flight fix: ❌ none found in open PRs — no open PR touches the rocm720 concurrency/scheduling config.

Suggested Triage Steps

• Treat this job as no signal: it never ran a test, so it cannot be used to assess f7632ef8. Do not chase a code regression for this job.
• Investigate the workflow-level scheduling/concurrency design in pr-test-amd-rocm720.yml:117-123: the daily 30 17 cron and the 6h 0 */6 cron share one concurrency group on main with cancel-in-progress: true, so a full pipeline that outlives the inter-cron gap is always preempted. Consider keying the concurrency group on github.event.schedule (or github.run_id) so the two cron schedules don't evict each other, mirroring the comment's intent of "unique groups so they never cancel each other."
• Confirm whether the daily full run is expected to complete at all, given the 6h dsv4 cron lands ~30 min later — if not, the daily run will be perpetually cancelled.
• If a genuine test signal is needed for this SHA, re-run the workflow via workflow_dispatch (which gets a unique full-{run_id} group per the same config and won't be cancelled).

Status

• Persistence: scheduled runs of pr-test-amd-rocm720.yml have been cancelled in 18 of the last 20 completed runs (no successful scheduled completion in the queryable window).
• Scope: run-wide — 39 of 48 jobs in run 27780231199 cancelled; structural, affects every scheduled run of this workflow, not this single job.
• Blocked work: the AMD ROCm 7.2 scheduled CI pipeline produces no green/complete signal; the two *-finish aggregator jobs report failure purely as a consequence of the cancellation.
• In-flight fix: ❌ none found — needs a concurrency-config fix or investigation.

stage-b-test-1-gpu-large-amd-rocm720 (linux-mi325-1gpu-sglang, 0) — failed step(s): N/A

Commit Info

• sglang: f7632ef8
• aiter: N/A (job log blob no longer exists — BlobNotFound — and the job was cancelled before/at setup, so no [CI-AITER-CHECK] markers are recoverable)

Failed Tests

Test File	Test Function	Error	Log
N/A — no test ran	N/A	Job cancelled during setup (0 steps recorded, ~29 min in)	Job

This is not a test failure. The job stage-b-test-1-gpu-large-amd-rocm720 (linux-mi325-1gpu-sglang, 0) was cancelled, not failure. It started at 18:18:25 UTC and was cancelled at 18:47:22 UTC with zero step metadata — it never reached the test-execution phase. The log endpoint returns BlobNotFound (logs are not retained for cancelled jobs).

Failure Summary

The entire run 27780231199 was cancelled (39/48 jobs cancelled, 7 success, 2 "finish" aggregator jobs failure — the latter fail only because their upstream was cancelled). The target job was cancelled mid-setup by GitHub Actions concurrency, not by any code or test defect.

Regression Status

Infrastructure issue (recurring, structural) — not a code regression.

Recent history of job stage-b-test-1-gpu-large-amd-rocm720 (linux-mi325-1gpu-sglang, 0) (pr-test-amd-rocm720.yml, event=schedule, branch=main, completed runs only):

Date (UTC)	Run	Job Status	Note
Jun 18 18:18	27780231199	❌ Cancelled	this run; superseded by 27781870163
Jun 18 13:07	27761630959	⏭️ Skipped	6h dsv4-only cron — full job not run
Jun 18 07:39	27744325971	⏭️ Skipped	6h dsv4-only cron
Jun 18 00:44	27729179568	⏭️ Skipped	6h dsv4-only cron
Jun 17 18:15	27710174822	❌ Cancelled	prior daily run, same pattern; superseded by 27712035487

Every one of the last 8 scheduled runs of this workflow concluded cancelled (verified via the workflow runs API). The daily full-test run is cancelled each day before it can execute.

Failure Cluster

Scheduled run cancelled by overlapping schedule trigger (concurrency cancel-in-progress) — no test executed.

Facts (verified observations)

• Job conclusion is cancelled (not failure); steps: 0; ran 18:18:25 → 18:47:22 UTC. (job API)
• Run 27780231199: event=schedule, conclusion=cancelled; 39 cancelled / 7 success / 2 finish-failure jobs. (run + jobs API)
• A newer schedule run 27781870163 was created at 18:47:21 UTC (event=schedule, head 61a8b42c). My run's jobs were cancelled one second later, at 18:47:22 UTC. (timing from both APIs)
• Concurrency config (pr-test-amd-rocm720.yml:117-123):
  • group: pr-test-amd-rocm720-${{ … || github.ref }} → for a schedule event this resolves to pr-test-amd-rocm720-refs/heads/main (the full-{run_id} branch requires run_all_tests/workflow_dispatch, neither true here; inputs.pr_head_sha/inputs.ref are empty for schedule).
  • cancel-in-progress: ${{ !inputs.run_all_tests && github.event_name != 'workflow_call' && github.event_name != 'workflow_dispatch' }} → evaluates true for schedule.
  • ⟹ both schedule runs share one concurrency group with cancel-in-progress: true; the later run cancels the earlier in-progress one.
• The workflow has two crons (pr-test-amd-rocm720.yml:7-9): 30 17 * * * (daily, runs the full job matrix incl. this job) and 0 */6 * * * (every 6h, runs dsv4 jobs only via the call-gate skip at line 133). The full job ran here ⟹ this run is the 17:30 daily cron (delayed to 18:18). The superseding run at 18:47 is the 18:00 6h cron (delayed). They collide in the shared concurrency group. The prior day shows the identical collision (18:15 daily cancelled by 18:48 6h run).

Hypothesised Causes (with confidence)

• [FACT] The job was cancelled by GitHub Actions concurrency, not by a code/test defect. The newer schedule run 27781870163 (created 18:47:21) entered the shared concurrency group pr-test-amd-rocm720-refs/heads/main (cancel-in-progress: true) and cancelled this in-progress run one second later (18:47:22).
  • Supporting: exact 1-second timing alignment; both runs are schedule on refs/heads/main; concurrency block resolves to the same group with cancel-in-progress=true; 0 steps recorded (cancelled before test execution).
  • Disconfirming: (none found) — no test ran, so no test-level failure is even possible here.
• [HIGH] Structural root cause: the daily 30 17 * * * full-test cron and the 0 */6 * * * (18:00) dsv4 cron routinely overlap after GitHub's scheduling delays (~30–50 min observed), and because they share one cancel-in-progress concurrency group, the 6h dsv4 run kills the daily full-test run mid-flight every day.
  • Supporting: all 8 most recent scheduled runs concluded cancelled; identical daily/6h-collision pattern reproduced on Jun 17 (27710174822 cancelled by 27712035487).
  • Disconfirming: did not inspect every one of the 8 runs' superseding run individually beyond Jun 17 and Jun 18, so a minority could in principle have a different cancel cause; the dominant pattern is nonetheless concurrency cancellation.
  • How to verify: re-run pr-test-amd-rocm720.yml via workflow_dispatch (which gets a unique full-{run_id} group and won't be cancelled) and confirm the job reaches test execution; or compare created_at of the superseding schedule run vs the cancel timestamp for the remaining historical runs.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs with keywords rocm720 concurrency cancel (1 hit, unrelated: #23957 attention backend) and pr-test-amd-rocm720 schedule cancel (0 hits). API: GET /search/issues?q=repo:sgl-project/sglang+is:pr+is:open+....

Suggested Triage Steps

• Confirm no real test signal is lost by re-running this stage via workflow_dispatch (gets a unique full-{run_id} concurrency group per the config at pr-test-amd-rocm720.yml:122) and checking the job reaches the test phase.
• To stop the daily full-test run being killed by the 6h dsv4 cron, consider giving the two cron schedules distinct concurrency groups — e.g. fold github.event.schedule into the group: expression so 30 17 * * * and 0 */6 * * * don't share a group. (Investigation direction, not a directive — maintainer decides.)
• Alternatively, set cancel-in-progress: false for schedule events, or stagger the cron times so the daily full run finishes before the next 6h run can start.

Status

• Persistence: All 8 most recent scheduled runs of pr-test-amd-rocm720.yml concluded cancelled; the daily full-test job has been cancelled at least Jun 17 and Jun 18 (same daily/6h collision).
• Scope: Workflow-wide — 39/48 jobs in this run were cancelled, not just this job. Affects the entire daily full-test signal for ROCm 7.2 AMD CI.
• Blocked work: The daily pr-test-amd-rocm720.yml full-matrix run produces no completed test signal; only the 6h dsv4-only crons complete.
• In-flight fix: ❌ none found in open PRs — needs a new PR (concurrency-group separation for the two cron schedules) or maintainer investigation.

stage-b-test-2-gpu-large-amd-rocm720 (linux-mi325-2gpu-sglang, 1) — failed step(s): N/A

Commit Info

• sglang: f7632ef8
• aiter: N/A (job never executed — no [CI-AITER-CHECK] markers; not an aiter-scout workflow)

Failed Tests

Test File	Test Function	Error	Log
N/A	N/A	Job cancelled before any step ran (0 steps executed)	Job

No test failure occurred. The job stage-b-test-2-gpu-large-amd-rocm720 (linux-mi325-2gpu-sglang, 1) was queued (waiting on a linux-mi325-2gpu-sglang runner) from 18:18:25 and cancelled at 18:47:22 UTC without starting any step.

Failure Summary

The job did not fail on any test — it was cancelled by GitHub Actions concurrency control. Run 27780231199 (schedule, sha f7632ef8) was superseded by the next scheduled run 27781870163, which shares the same concurrency group (pr-test-amd-rocm720-refs/heads/main) with cancel-in-progress: true. The two *-finish aggregator jobs then reported failure solely because their dependent jobs were cancelled (the gate's "Check all dependent job statuses" step).

Regression Status

Infrastructure issue (concurrency cancellation) — not a code regression, not a flaky test.

Recent pr-test-amd-rocm720.yml schedule runs on main (completed runs only):

Date (UTC)	Run	Conclusion	Note
Jun 18 18:18	27780231199	❌ cancelled	this run — superseded at 18:47
Jun 18 13:07	27761630959	❌ cancelled	superseded at 18:18 (when this run started)
Jun 18 07:39	27744325971	❌ cancelled	superseded at 13:07
Jun 18 00:44	27729179568	❌ cancelled	superseded at 07:39
Jun 17 18:48	27712035487	❌ cancelled	superseded at 00:44

Every scheduled run is cancelled by its successor — a structural pattern, observed across the entire queryable window. The first-observed date is therefore not meaningful (no "last green" exists for scheduled runs because they never reach completion before the next one supersedes them).

Failure Cluster

Scheduled run cancelled by concurrency supersede (cancel-in-progress)

Facts (verified observations, no interpretation)

• Job conclusion = cancelled, 0 steps executed (verified via jobs API for 82202905794).
• Job timeline: started 2026-06-18T18:18:25Z, completed (cancelled) 2026-06-18T18:47:22Z.
• Run 27780231199 overall conclusion = cancelled; of 48 jobs: 7 success, 2 failure (both *-finish gates), 39 cancelled (all with 0 steps).
• The next scheduled run 27781870163 was created 2026-06-18T18:47:21Z — 1 s before this job's cancellation timestamp.
• Concurrency group (pr-test-amd-rocm720.yml:122) for schedule = pr-test-amd-rocm720-refs/heads/main; cancel-in-progress (:123) = true for schedule events.
• The pr-test-amd-rocm720-finish gate log shows repeated "result": "cancelled" and multimodal-gen-test-1-gpu-amd-rocm720: cancelled → The above jobs failed. — i.e. the gate failed because dependents were cancelled, not because a test failed.
• Job log blob returns BlobNotFound — no test log was ever produced (consistent with 0 steps).

Hypothesised Causes (with confidence, with disconfirming evidence)

• [FACT] The job was cancelled by the concurrency group pr-test-amd-rocm720-refs/heads/main (cancel-in-progress: true) when the next scheduled run 27781870163 started.
  • Supporting: cancellation timestamp (18:47:22) matches the next run's creation (18:47:21) to the second; concurrency config in pr-test-amd-rocm720.yml:122-123 resolves to a shared refs/heads/main group with cancel-in-progress for schedule events; every recent scheduled run is cancelled exactly when its successor starts.
  • Disconfirming: (none found) — the job ran 0 steps, so no test/code failure is even possible.
  • How to verify: re-read the jobs API (0 steps, conclusion cancelled) and compare cancellation time vs successor run creation time (already done).
• [FACT] The *-finish jobs' failure conclusion is a downstream artifact of the cancellation, not an independent failure.
  • Supporting: their only failing step is "Check all dependent job statuses"; the log enumerates cancelled dependents.
  • Disconfirming: (none found).

No code-level (sglang or aiter) hypothesis applies: the changed code was never exercised.

In-flight Fix Check

This is a CI-orchestration property, not a bug needing a code fix, so an in-flight-fix search is not strictly applicable. For completeness:

• In-flight fix: ❌ none expected — the long-running scheduled jobs (≈5-6 h) outlast the 0 */6 cron interval (6 h) plus an extra ~18:47 trigger, so each scheduled run is reliably superseded before finishing. This is a scheduling/runner-capacity characteristic, not a regression with an open PR.

Suggested Triage Steps

• Treat this job as not a real failure for CI-health purposes — exclude cancelled-with-0-steps jobs from failure tracking/clustering (they carry no test signal).
• If full scheduled-run signal on rocm720 is actually wanted: investigate why scheduled runs take ≈5-6 h (longer than the 6 h cron cadence) so they get cancelled before completing — e.g. 2-GPU/8-GPU runner queue contention (this job sat ~29 min unscheduled). Consider reducing scheduled cron frequency, sharding/parallelism, or a per-run unique concurrency group for scheduled runs so successive runs don't cancel each other.
• To obtain a clean signal for this specific stage, re-run via workflow_dispatch (which gets a unique concurrency group full-<run_id> and is not subject to cancel-in-progress), or wait for a pull_request-triggered run that exercises these paths.

Status (factual, no priority assignment)

• Persistence: Every pr-test-amd-rocm720.yml schedule run in the queryable window (≥5 consecutive runs, Jun 17 18:48 → Jun 18 18:18 UTC) ended cancelled; none completed.
• Scope: Structural — affects all 39 test jobs in each scheduled run of this workflow (they are all cancelled together). The target 2-GPU job is one of them.
• Blocked work: Scheduled rocm720 test coverage produces no test results, because runs are superseded before completing. PR-triggered and workflow_dispatch runs are unaffected (separate concurrency groups).
• In-flight fix: ❌ none — this is a scheduling/concurrency characteristic, not a code defect.

stage-b-test-1-gpu-small-amd-nondeterministic-rocm720 (linux-mi325-1gpu-sglang) — failed step(s): N/A

Commit Info

• sglang: f7632ef8
• aiter: N/A — the job log blob does not exist (BlobNotFound), so no [CI-AITER-CHECK] markers are available. The job was cancelled with zero recorded steps, so dependency install never produced inspectable output.

Failed Tests

Test File	Test Function	Error	Log
N/A	N/A	No test failure — job conclusion = cancelled (not failure)	Job

This is not a test failure. The job was cancelled by GitHub Actions concurrency cancellation, recorded no steps, and its log blob does not exist.

Failure Summary

The job stage-b-test-1-gpu-small-amd-nondeterministic-rocm720 did not fail a test — it was cancelled (conclusion: cancelled, no steps recorded) ~29 min after starting. The entire parent run 27780231199 was cancelled (39 cancelled / 7 success / 2 failure jobs); the 2 failure jobs are both *-finish gate jobs that fail by design when their upstream dependencies are cancelled. There is no code-level regression here.

Regression Status

Infrastructure / orchestration — not a real test failure. Every scheduled pr-test-amd-rocm720.yml run on main in the queryable window (back to Jun 17) concludes cancelled — this is the steady-state behavior of this workflow, not a new event.

Recent scheduled runs of pr-test-amd-rocm720.yml (event=schedule, branch=main, completed only):

Created (UTC)	Run	Conclusion
2026-06-18 18:18	27780231199 (this run)	cancelled
2026-06-18 13:07	27761630959	cancelled
2026-06-18 07:39	27744325971	cancelled
2026-06-18 00:44	27729179568	cancelled
2026-06-17 18:48	27712035487	cancelled

Failure Cluster

Concurrency-cancelled scheduled run (overlapping cron, cancel-in-progress)

Facts (verified observations, no interpretation)

• [FACT] Job 82202905814 conclusion = cancelled (not failure); started 2026-06-18T18:18:25Z, completed 2026-06-18T18:47:22Z; no steps recorded (verified via Actions jobs API).
• [FACT] The job log blob returns BlobNotFound — no log was preserved, consistent with an orchestration cancellation rather than a test run that errored.
• [FACT] Parent run 27780231199 conclusion = cancelled; of 48 jobs: 39 cancelled, 7 success, 2 failure. The only 2 failure jobs are call-pr-test-amd-extra-rocm720 / pr-test-amd-extra-finish ([82208456903]) and pr-test-amd-rocm720-finish ([82225249578]) — both are end-of-pipeline gate jobs that fail when upstream jobs are cancelled.
• [FACT] The next scheduled run 27781870163 was created at 2026-06-18T18:47:21Z — 1 second before this job's cancellation timestamp 18:47:22Z.
• [FACT] pr-test-amd-rocm720.yml:122-123 sets group: pr-test-amd-rocm720-${{ ... || github.ref }} (resolves to …-refs/heads/main for schedule events) with cancel-in-progress: ${{ !inputs.run_all_tests && github.event_name != 'workflow_call' && github.event_name != 'workflow_dispatch' }} → true for schedule.
• [FACT] The workflow has two overlapping crons (30 17 * * * and 0 */6 * * *, pr-test-amd-rocm720.yml:8-9), so scheduled runs routinely overlap and supersede each other.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [HIGH] The run was terminated by concurrency-group cancellation: a newer scheduled run entered the shared group pr-test-amd-rocm720-refs/heads/main (which has cancel-in-progress: true for schedule), cancelling this in-progress run.
  • Supporting: next scheduled run 27781870163 created 18:47:21Z, this job cancelled 18:47:22Z (1 s apart); concurrency config at pr-test-amd-rocm720.yml:117-123 confirms shared group + cancel-in-progress for schedule; all recent scheduled runs share the cancelled conclusion.
  • Disconfirming: (none found) — the timing match and config are conclusive.
• [FACT, not a cause] The 2 failure jobs are gate/-finish jobs failing on cancelled dependencies — expected cascade behavior, not an independent failure.

In-flight Fix Check

Not applicable — there is no test/code regression to fix. This is the workflow's own scheduling behavior (overlapping crons + cancel-in-progress).

• In-flight fix: ❌ N/A — no code failure; this is concurrency-driven run cancellation by design

Suggested Triage Steps

• Treat this job as a non-failure; no code investigation is warranted. Do not attribute it to commit f7632ef8 or any sglang/aiter change.
• If repeated scheduled-run cancellations are undesired noise, the maintainer may consider whether the two overlapping crons (30 17 * * * and 0 */6 * * *) plus cancel-in-progress: true are intended to supersede each other, or whether the daily-cron run should use a distinct concurrency group so it isn't cancelled by the 6h-cron run.
• To get a real signal for this stage, look at a run that reached completion (e.g. inspect the most recent run where this specific job concluded success/failure rather than cancelled).

Status (factual, no priority assignment)

• Persistence: All scheduled pr-test-amd-rocm720.yml runs on main in the queryable window (Jun 17–18) conclude cancelled; this is steady-state, not new.
• Scope: This run 27780231199 — 39 of 48 jobs cancelled; the named job recorded no steps and no test failure.
• Blocked work: No code signal lost to a regression; the workflow simply does not produce a clean terminal signal for scheduled runs under the current concurrency config.
• In-flight fix: ❌ N/A — no code failure to fix.

stage-b-test-2-gpu-large-amd-rocm720 (linux-mi325-2gpu-sglang, 0) — failed step(s): N/A

Commit Info

• sglang: f7632ef8
• aiter: N/A — unverifiable. Job log blob has been purged (BlobNotFound from the logs API), so the [CI-AITER-CHECK] markers cannot be read. The run-level env showed AITER_COMMIT_OVERRIDE= (empty), so the Dockerfile default would have applied, but no override was in effect.

Failed Tests

Test File	Test Function	Error	Log
—	—	No test failure — the job was cancelled, not failed	Job

This job did not fail a test. Its conclusion is cancelled (verified via the jobs API): it started 2026-06-18T18:18:25Z and was cancelled 2026-06-18T18:47:22Z, ~29 min in — before reaching (or while still inside) its test phase. The job log blob has already been purged, so no test-level detail is recoverable, but this is moot for a cancellation.

Failure Summary

The job was terminated by GitHub Actions concurrency cancellation, not by any test or code error. [FACT] The next scheduled run of the same workflow, 27781870163, was created at 2026-06-18T18:47:21Z — one second before this run's job was cancelled at 18:47:22Z. The two "failure" jobs in this run (pr-test-amd-rocm720-finish and pr-test-amd-extra-finish) are gate jobs that exit 1 because their upstream needs jobs report cancelled — they are a consequence of the cancellation, not an independent failure.

Regression Status

Infrastructure issue (concurrency cancellation) — persistent, not a code regression.

Recent history of workflow pr-test-amd-rocm720.yml (branch main, event=schedule, completed runs only):

Date (UTC)	Run	Conclusion
Jun 18 18:18	27780231199	❌ cancelled (this run)
Jun 18 13:07	27761630959	❌ cancelled
Jun 18 07:39	27744325971	❌ cancelled
Jun 18 00:44	27729179568	❌ cancelled
Jun 17 18:48	27712035487	❌ cancelled
Jun 17 18:15	27710174822	❌ cancelled
Jun 17 13:13	27691581201	❌ cancelled

Every one of the last 12 scheduled runs is cancelled — this is a stable structural pattern, not a new regression at f7632ef8. No green scheduled run exists in the queryable window.

Failure Cluster

Scheduled run self-cancelled by concurrency group (cancel-in-progress).

Facts (verified observations)

• [FACT] This job's conclusion = cancelled, status = completed; started 18:18:25Z, cancelled 18:47:22Z (jobs API).
• [FACT] Run-level conclusion = cancelled for the whole run 27780231199; of 48 jobs: 7 success, 39 cancelled, 2 failure (the two finish/gate jobs).
• [FACT] The 7 jobs that succeeded are all the short ones (sgl-kernel-unit-test-*, jit-kernel-unit-test, stage-b-test-1-gpu-small-amd-mi35x, gate/check-changes); every long-running 2-GPU / large job was cancelled.
• [FACT] Next scheduled run 27781870163 created 18:47:21Z = 1 s before this job's cancellation.
• [FACT] pr-test-amd-rocm720.yml:122-123 sets the concurrency group to pr-test-amd-rocm720-${{ ... || github.ref }} (→ …-refs/heads/main for schedule events) with cancel-in-progress: true for schedule events. All scheduled runs therefore share one group and cancel each other.
• [FACT] The workflow has two crons (pr-test-amd-rocm720.yml:8-9): 30 17 * * * (daily full gate) and 0 */6 * * * (every 6 h). They share the same concurrency group.
• [FACT] The finish-job log shows the gate logic treats cancelled upstream results as failure (if [[ "$result" == "failure" || "$result" == "cancelled" ]]; then … exit 1), explaining the 2 "failure" jobs.

Hypothesised Causes

• [HIGH] This run was cancelled because a newer scheduled run started in the same cancel-in-progress concurrency group. The full daily gate (cron 30 17) and/or a prior run had not finished its long 2-GPU jobs before the next 0 */6 cron fired, so GitHub cancelled the in-progress run.
  • Supporting: shared concurrency group …-refs/heads/main with cancel-in-progress: true (pr-test-amd-rocm720.yml:122-123); next scheduled run created 1 s before cancellation; every scheduled run in the window is cancelled; only the long jobs were killed while short jobs completed.
  • Disconfirming: (none found) — the log blob is purged so the in-job state at cancellation can't be inspected, but the cancellation mechanism is independently confirmed from the workflow config + run/job timestamps.
  • How to verify: inspect the timeline of any two consecutive scheduled runs; the older's cancellation timestamp matches the younger's creation timestamp.
• [LOW] A real test failure hid behind the cancellation.
  • Supporting: none — job conclusion is cancelled, not failure.
  • Disconfirming: the job was killed ~29 min in across the board; sister short jobs in the same run passed; log unavailable but conclusion is unambiguous.
  • How to verify: not actionable; would require a run that completes without being cancelled.

In-flight Fix Check

• In-flight fix: searched open PRs for concurrency/cancel scheduling fixes on this workflow:

  curl -s -H "Authorization: token $GH_PAT" "https://api.github.com/search/issues?q=repo:sgl-project/sglang+is:pr+is:open+pr-test-amd-rocm720+concurrency"
  

  ❌ none found matching pr-test-amd-rocm720 + concurrency / cancel-in-progress. This appears to be accepted/by-design behavior (the daily full gate is expected to be superseded by the every-6h runs), not a tracked bug — needs a maintainer decision if full scheduled signal is desired.

Suggested Triage Steps

• Confirm the design intent: if the daily 30 17 full gate is meant to complete, give it its own concurrency group (or a unique-per-run group like the workflow_call branch already uses at pr-test-amd-rocm720.yml:122) so the 0 */6 accuracy cron can't cancel it.
• Alternatively, scope cancel-in-progress to PR events only, leaving schedule runs un-cancellable, so each scheduled gate runs to completion.
• If cancellation of overlapping scheduled runs is intentional, treat this job (and the two finish "failures") as expected noise and exclude cancelled scheduled runs from failure tracking — they carry no test signal.
• For any real ROCm 7.2 test signal, look at a pull_request-triggered or workflow_dispatch run (unique concurrency group, not cancelled) instead of scheduled runs.

Status

• Persistence: All 12 most-recent scheduled runs (main, event=schedule) are cancelled, spanning at least Jun 16 → Jun 18 2026. No green scheduled run in the queryable window.
• Scope: Run-wide and workflow-wide — 39/48 jobs cancelled per run; affects every long-running 2-GPU/large job on pr-test-amd-rocm720.yml. Short unit-test jobs still complete.
• Nature: Orchestration/concurrency, not a code or test defect. The cited commit f7632ef8 is not implicated.
• In-flight fix: ❌ none found in open PRs.

stage-c-test-large-8-gpu-amd-mi35x-rocm720 (linux-mi35x-gpu-8, 0) — failed step(s): N/A

Commit Info

• sglang: f7632ef8
• aiter: N/A — the job never executed any steps, so no [CI-AITER-CHECK] markers were emitted (log blob does not exist).

Failed Tests

Test File	Test Function	Error	Log
N/A	N/A	Job cancelled before any step executed — not a test failure	Job

No tests ran. The job stage-c-test-large-8-gpu-amd-mi35x-rocm720 (linux-mi35x-gpu-8, 0) was cancelled with zero steps recorded; its log blob returns BlobNotFound.

Failure Summary

The job did not fail on any test — it was cancelled as part of a whole-run cancellation of run 27780231199 (run conclusion cancelled: 39 jobs cancelled, 7 succeeded, 2 *-finish aggregators failed). The job sat in waiting/queued state from 18:18:25Z and was cancelled at 18:47:22Z before executing any step, so it produced no test results and no log.

Regression Status

Infrastructure issue (concurrency cancellation) — not a test regression.

Recent scheduled runs of pr-test-amd-rocm720.yml (branch main, event=schedule, completed only):

Date (UTC)	Run	Head	Conclusion
Jun 18 18:18	27780231199	f7632ef8	cancelled
Jun 18 13:07	27761630959	97e3b899	cancelled
Jun 18 07:39	27744325971	59001267	cancelled
Jun 18 00:44	27729179568	74e2e48c	cancelled
Jun 17 18:48	27712035487	7cead0fb	cancelled

Every recent scheduled run ends in cancelled — this is the steady-state behavior of this workflow's concurrency setup, not a new regression for this test/job.

Failure Cluster

Whole-run concurrency cancellation of a scheduled workflow (no steps executed, no log produced)

Facts (verified observations, no interpretation)

• Run 27780231199: status=completed, conclusion=cancelled, event=schedule, head f7632ef8.
• Job 82202905837: conclusion=cancelled, steps=[] (empty), started 2026-06-18T18:18:25Z, completed 2026-06-18T18:47:22Z.
• Job log endpoint returns BlobNotFound (no log was ever produced).
• Run-wide job tally: 7 success, 39 cancelled, 2 failure. The 2 failures are pr-test-amd-extra-finish and pr-test-amd-rocm720-finish, both failing only at step 2 "Check all dependent job statuses" (by-design failure when deps are cancelled).
• Next scheduled run 27781870163 was created 2026-06-18T18:47:21Z — within 1s of this run's cancellation.
• Workflow concurrency (pr-test-amd-rocm720.yml:117-123): group pr-test-amd-rocm720-${{ ... || github.ref }}, cancel-in-progress: true for schedule events.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [HIGH] The job was cancelled because a newer scheduled run of the same workflow superseded it via the shared cancel-in-progress concurrency group on refs/heads/main.
  • Supporting: next scheduled run 27781870163 created at 18:47:21Z matches this run's cancel time 18:47:22Z to the second; concurrency block sets cancel-in-progress: true for schedule events on the github.ref group; all recent scheduled runs share the same cancelled conclusion.
  • Disconfirming: (none found) — no test error, stack trace, or non-zero exit exists in any artifact; the job recorded no steps at all.
  • How to verify: inspect run 27781870163's created_at vs this run's job cancel timestamps (already done); confirm cancel-in-progress resolves true for event=schedule.
• [LOW] This 8-GPU MI35x job never acquired a runner and timed out / was reaped.
  • Supporting: zero steps recorded suggests it stayed in waiting/queued for the full 29 min.
  • Disconfirming: conclusion is cancelled (not timed_out), and the cancel timestamp coincides exactly with the next run's creation, pointing to concurrency rather than a runner-acquisition timeout.
  • How to verify: check whether the job ever transitioned to in_progress/got a runner label before cancellation.

In-flight Fix Check

No fix is warranted — this is not a code defect. (Concurrency cancellation of superseded scheduled runs is the intended behavior; no PR search performed because there is no failing test/library to key on.)

Suggested Triage Steps

• No code action needed. Treat this job result as a non-signal: it was cancelled before running, not failed.
• If a real signal is needed for head f7632ef8, re-run stage-c-test-large-8-gpu-amd-mi35x-rocm720 (linux-mi35x-gpu-8, 0) via workflow_dispatch (which gets a unique concurrency group full-<run_id> and will not be cancelled by the cron).
• If frequent cancellation of scheduled runs is undesirable for monitoring, consider whether the two crons (30 17 * * * and 0 */6 * * *) sharing one github.ref concurrency group should instead use distinct groups so they stop cancelling each other.

Status (factual, no priority assignment)

• Persistence: All 8 most recent scheduled runs of this workflow concluded cancelled (steady-state, not new).
• Scope: Run-wide — 39 of 48 jobs in 27780231199 were cancelled; this job is one of them. No isolated failure of this specific test/job.
• Blocked work: This scheduled run produced no AMD ROCm 7.2 test signal for head f7632ef8; coverage relies on a non-superseded run.
• In-flight fix: ❌ none needed — not a code failure.

stage-b-test-1-gpu-large-amd-rocm720 (linux-mi325-1gpu-sglang, 1) — failed step(s): N/A

Commit Info

• sglang: f7632ef8
• aiter: N/A (job was cancelled before any test step ran; no [CI-AITER-CHECK] markers — the log blob has already expired/BlobNotFound)

Failed Tests

Test File	Test Function	Error	Log
—	—	No test failure. Job conclusion is cancelled, not failure.	Job

Failure Summary

Job stage-b-test-1-gpu-large-amd-rocm720 (linux-mi325-1gpu-sglang, 1) did not fail a test — its conclusion is cancelled. The entire run 27780231199 was cancelled (39 cancelled, 7 success, 2 failure out of 48 jobs); the only two failure jobs are the terminal gate jobs (pr-test-amd-rocm720-finish, call-pr-test-amd-extra-rocm720 / pr-test-amd-extra-finish), which fail by design when any dependent job is cancelled. This run was superseded by a newer scheduled run via the workflow's cancel-in-progress concurrency setting.

Regression Status

Infrastructure issue (concurrency cancellation) — not a code/test regression.

Recent schedule runs of pr-test-amd-rocm720.yml on main (completed runs only):

Date (UTC)	Run	Conclusion
2026-06-18 18:18	27780231199	cancelled (this run)
2026-06-18 13:07	27761630959	cancelled
2026-06-18 07:39	27744325971	cancelled
2026-06-18 00:44	27729179568	cancelled
2026-06-17 18:48	27712035487	cancelled

Every recent scheduled run shows the same cancelled outcome, consistent with a recurring concurrency-cancellation pattern rather than a per-test regression.

Failure Cluster

Run cancelled by concurrency (superseded scheduled run) — no test executed to completion.

Facts (verified observations, no interpretation)

• [FACT] Target job 82202905858 conclusion = cancelled, started 18:18:25Z, completed 18:47:22Z, with zero recorded steps (API steps: []).
• [FACT] Job log is unavailable: GitHub returns <Error><Code>BlobNotFound</Code></Error> for the job-logs endpoint.
• [FACT] Run 27780231199 (name "PR Test ROCm 7.2 (AMD)", event schedule) conclusion = cancelled; job conclusions across the run: 7 success, 39 cancelled, 2 failure.
• [FACT] The 2 failure jobs are gate jobs only — their failing step is "Check all dependent job statuses" (82225249578 pr-test-amd-rocm720-finish; 82208456903 pr-test-amd-extra-finish). Steps 1 and 3 succeed; only the dependency-status check fails.
• [FACT] A newer schedule run 27781870163 was created 18:47:21Z — one second before this run's first finish job failed (18:47:25Z) and matching the cancellation timestamp of the target job (18:47:22Z).
• [FACT] Workflow concurrency (.github/workflows/pr-test-amd-rocm720.yml:122-123): group pr-test-amd-rocm720-${{ ... || github.ref }} (i.e. ...-refs/heads/main for schedule runs) with cancel-in-progress: true for schedule events.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [HIGH] The run was cancelled because a newer scheduled run entered the same concurrency group with cancel-in-progress: true.
  • Supporting: schedule runs on main all map to concurrency group pr-test-amd-rocm720-refs/heads/main with cancel-in-progress: true (pr-test-amd-rocm720.yml:122-123); newer run 27781870163 was created 18:47:21Z and the target job was cancelled at 18:47:22Z (1s later); the only failure jobs are gate jobs whose sole failing step is "Check all dependent job statuses".
  • Disconfirming: (none found) — timing and the gate-only failure mode are fully consistent with concurrency cancellation.
• [MEDIUM] The superseding run is the every-6h 0 */6 * * * (dsv4-only) cron arriving late (~18:47) and cancelling the in-progress daily 30 17 * * * full-matrix run, since both share the same github.ref-keyed concurrency group.
  • Supporting: this run executed the full 48-job matrix (stage-a/b/c, multimodal), so it is the daily 30 17 full run (delayed from 17:30 to 18:18); the workflow comment at pr-test-amd-rocm720.yml:127-133 documents that the 0 */6 cron skips the gate and runs dsv4 jobs only, yet it still shares the same concurrency group.
  • Disconfirming: I could not read the github.event.schedule value of the superseding queued run from the API, so the exact cron that triggered 27781870163 is inferred from timing, not directly confirmed.
  • How to verify: inspect the superseding run's event.schedule once it completes, or check whether it ran only the 2 dsv4 jobs (which would confirm it is the 0 */6 cron).

In-flight Fix Check

• Not a code/test failure, so no library/test-name keyword applies. If maintainers want to stop the 6h dsv4 cron from cancelling the daily full run, the relevant area is the concurrency-group key in pr-test-amd-rocm720.yml:122 (e.g. differentiate the group by github.event.schedule). In-flight fix: ❌ none searched — no failing test to key on.

Suggested Triage Steps

• Confirm the superseding run 27781870163 is the 0 */6 dsv4-only cron (check its event.schedule / job list) — if so, the daily full run is being routinely killed by the 6h cron sharing the same concurrency group.
• If confirmed, consider keying the concurrency group on github.event.schedule (or giving the 30 17 daily full run its own group) so the dsv4-only cron no longer cancels the full matrix.
• No source-code investigation of f7632ef8 is warranted — no test executed.

Status (factual, no priority assignment)

• Persistence: All 5 most recent completed schedule runs of pr-test-amd-rocm720.yml on main (2026-06-17 18:48 → 2026-06-18 18:18) concluded cancelled — recurring concurrency cancellation, not a one-off.
• Scope: Run-wide — 39 of 48 jobs in run 27780231199 cancelled; the target job is one of them. No test-level failure in this job.
• Blocked work: The daily full ROCm 7.2 AMD signal is not being produced (full matrix cancelled before completion), so PR CI health for this workflow is effectively unobserved on these scheduled runs.
• In-flight fix: ❌ none found — needs a concurrency-group config change, not a code fix.

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-06-19 19:28 UTC)

[amd-bot]

pr-test-amd-rocm720.yml — Per-Job Analysis (continued 3/14)

stage-c-test-large-8-gpu-amd-rocm720 (linux-mi325-8gpu-sglang, 0) — failed step(s): N/A

Commit Info

• sglang: f7632ef8
• aiter: N/A — the job log blob does not exist (BlobNotFound), so no [CI-AITER-CHECK] markers are available.

Failed Tests

Test File	Test Function	Error	Log
N/A — no test ran	N/A	Job cancelled by CI concurrency control (not a test failure)	Job

This job did not fail a test. Its conclusion is cancelled, it recorded zero steps, and its log blob has expired/was never persisted (BlobNotFound).

Failure Summary

The job stage-c-test-large-8-gpu-amd-rocm720 (linux-mi325-8gpu-sglang, 0) was cancelled, not failed. It started at 2026-06-18T18:18:25Z and was cancelled at 2026-06-18T18:47:22Z — the exact same instant that all 39 other in-progress jobs in run 27780231199 were cancelled. This is a whole-run cancellation triggered by GitHub Actions concurrency control, not a code or test defect.

Regression Status

Infrastructure issue (CI concurrency cancellation) — not a test regression.

Recent pr-test-amd-rocm720.yml scheduled runs on main (event=schedule, completed only):

Date	Run	Conclusion
Jun 18 18:18	27780231199	❌ cancelled (this run)
Jun 18 13:07	27761630959	❌ cancelled
Jun 18 07:39	27744325971	❌ cancelled
Jun 18 00:44	27729179568	❌ cancelled
Jun 17 18:48	27712035487	❌ cancelled
Jun 17 18:15	27710174822	❌ cancelled

Every recent scheduled run is cancelled — a chronic pattern, consistent with scheduled runs repeatedly cancelling one another in a shared concurrency group (see Hypothesised Causes).

Failure Cluster

CI concurrency cancellation of scheduled run (no test executed)

Facts (verified observations, no interpretation)

• [FACT] Job conclusion is cancelled (status completed), with an empty steps array — no step ever started in this job. (Verified via GET /actions/jobs/82202905911.)
• [FACT] The job log fetch returns <Error><Code>BlobNotFound> — the log was never persisted / has expired. No test output exists.
• [FACT] Run 27780231199 overall conclusion is cancelled: of 48 jobs → 7 success, 39 cancelled, 2 failure (the 2 "failures" are the *-finish gate jobs, which fail by design when upstream jobs are cancelled).
• [FACT] All 39 cancelled jobs share the identical completed_at = 2026-06-18T18:47:22Z → a single global cancellation event, not 39 independent failures.
• [FACT] A newer scheduled run 27781870163 (event=schedule, branch main, head 61a8b42c) was created at 2026-06-18T18:47:21Z — exactly 1 second before the cancellation timestamp.
• [FACT] .github/workflows/pr-test-amd-rocm720.yml:117-123 defines concurrency.group: pr-test-amd-rocm720-${{ ... || github.ref }} (resolves to pr-test-amd-rocm720-refs/heads/main for schedule events) with cancel-in-progress: true for event=schedule.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [HIGH] The in-progress run was cancelled by a newer scheduled run entering the same concurrency group.
  • Supporting: both runs are event=schedule on main ⇒ identical concurrency group pr-test-amd-rocm720-refs/heads/main with cancel-in-progress: true (pr-test-amd-rocm720.yml:122-123); newer run 27781870163 created at 18:47:21Z, all 39 jobs cancelled 1 s later at 18:47:22Z; the global single-timestamp cancellation matches a concurrency preemption, not test failures.
  • Disconfirming: (none found) — the timing, shared group, and cancel-in-progress flag all converge.
  • How to verify: in the run's timeline / "cancelled by" annotation, confirm the canceller is run 27781870163. The two schedule crons (30 17 * * * daily full + 0 */6 * * * 6-hourly) firing close together while a delayed run is still in progress reproduces this.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for concurrency/cancel keywords:

  curl -s -H "Authorization: token $GH_PAT" "https://api.github.com/search/issues?q=repo:sgl-project/sglang+is:pr+is:open+pr-test-amd-rocm720+concurrency"
  

  No matching open PR. (This is a CI-orchestration symptom; not a code defect requiring a source fix.)

Suggested Triage Steps

• Open the run page for 27780231199 and confirm the "Canceled" annotation names run 27781870163 as the canceller.
• Treat this job as non-actionable for code: no test ran, no regression. Do not bisect main.
• If full scheduled signal is desired despite overlapping crons, consider giving schedule runs a unique concurrency group (as already done for run_all_tests/workflow_dispatch at pr-test-amd-rocm720.yml:122) or staggering the 30 17 and 0 */6 crons so a delayed daily run is not preempted by the 6-hourly run — a CI-config change for maintainers, not a directive.
• To obtain a real test result on f7632ef8, re-run the workflow via workflow_dispatch (which gets a unique, non-cancellable group).

Status (factual, no priority assignment)

• Persistence: every recent scheduled pr-test-amd-rocm720.yml run (≥6 consecutive, Jun 17–18) concluded cancelled; runs are not completing.
• Scope: whole-run cancellation — 39 of 48 jobs in run 27780231199 cancelled at one instant; this stage-c-test-large-8-gpu job is one of them.
• Blocked work: scheduled ROCm 7.2 AMD CI signal is not being produced (no completed test results on these scheduled runs).
• In-flight fix: ❌ none found in open PRs.

stage-b-test-large-8-gpu-mi35x-disaggregation-amd-rocm720 (linux-mi35x-gpu-8.fabric) — failed step(s): N/A

Commit Info

• sglang: f7632ef8
• aiter: N/A — job never reached a runner, so no [CI-AITER-CHECK] markers were emitted (and this is pr-test-amd-rocm720.yml, not an AITER Scout run).

Failed Tests

Test File	Test Function	Error	Log
N/A — no test ran	N/A	Job cancelled while queued; 0 steps executed, no runner assigned	Job page

There is no test-level failure. The job stage-b-test-large-8-gpu-mi35x-disaggregation-amd-rocm720 (linux-mi35x-gpu-8.fabric) was queued at 18:18:25Z, never picked up by a linux-mi35x-gpu-8.fabric runner, and was cancelled at 18:47:22Z with zero steps and an empty runner name.

Failure Summary

This is not a test failure — it is a CI cancellation event. The job sat in the queue for ~29 minutes waiting for a scarce 8-GPU MI35x fabric runner, then was cancelled (along with 38 other still-queued jobs in the same run) at the exact same instant 18:47:22Z when the next scheduled run of the workflow started and took over the concurrency group. No source code or test executed.

Regression Status

Infrastructure issue (concurrency supersession) — not a code regression.

History of this job stage-b-test-large-8-gpu-mi35x-disaggregation-amd-rocm720 in recent pr-test-amd-rocm720.yml runs (branch main, event schedule, completed runs only):

Date	Run	Job conclusion	Runner	Steps
Jun 18 18:18	27780231199	❌ cancelled (queued, superseded)	none	0
Jun 18 13:07	27761630959	⏭️ skipped	none	0
Jun 18 07:39	27744325971	⏭️ skipped	none	0
Jun 18 00:44	27729179568	⏭️ skipped	none	0

This job has never actually executed in the queryable window (always skipped or cancelled), so there is no green or red test baseline — and nothing to bisect.

Facts (verified observations)

• Job conclusion is cancelled, steps: [], runner_name: "" (verified from actions/jobs/82202905860). It never started any work.
• Run 27780231199 (event schedule) overall conclusion: cancelled. Of 48 jobs: 7 success, 39 cancelled (all at 18:47:22Z), 2 *-finish aggregator jobs failed (the standard consequence of upstream cancellation).
• The job-log blob returns BlobNotFound — consistent with a job that produced no log output.
• All 39 cancelled jobs share runner_name: "" — i.e. they were still queued, never assigned a runner. The 7 successes are exactly the jobs that did secure a runner (GitHub Actions-hosted gate/check-changes jobs, plus a few linux-mi325-*/linux-mi35x-gpu-1 jobs).
• The next scheduled run 27781870163 was created at 18:47:21Z — one second before the mass cancellation at 18:47:22Z.
• Workflow concurrency (.github/workflows/pr-test-amd-rocm720.yml:122-123): for a schedule event the group resolves to pr-test-amd-rocm720-refs/heads/main with cancel-in-progress: true. A new scheduled run therefore cancels the in-progress one sharing that group.
• Recurring pattern: 8 of the last 9 scheduled runs of this workflow have overall conclusion cancelled (verified from the workflow runs list), confirming routine schedule-overlap supersession.

Hypothesised Causes (with confidence)

• [FACT] Concurrency cancel-in-progress supersession. The newer scheduled run 27781870163 (created 18:47:21Z) took over concurrency group pr-test-amd-rocm720-refs/heads/main and cancelled all still-queued jobs of run 27780231199 at 18:47:22Z.
  • Supporting: 1-second alignment between new-run creation and mass cancellation; workflow config cancel-in-progress: true for schedule events (pr-test-amd-rocm720.yml:123); all cancelled jobs were unassigned/queued; recurring cancelled conclusion across prior scheduled runs.
  • Disconfirming: (none found) — evidence is direct and consistent.
• [HIGH] Contributing factor: scarce linux-mi35x-gpu-8.fabric runner capacity. The 8-GPU MI35x job sat queued for ~29 min and never got a runner before supersession. Had a runner been free, it likely would have started before the next schedule tick.
  • Supporting: runner_name empty for the full 29-min queue; other large linux-mi325-8gpu/mi35x jobs in the same run were also stuck queued and cancelled; smaller 1gpu/2gpu jobs did get runners.
  • Disconfirming: cannot prove no runner existed at any instant (no queue-depth telemetry in the API), so this is a strong inference rather than a logged fact.

In-flight Fix Check

Not applicable — there is no code defect to fix. This is expected concurrency behavior plus runner-pool pressure. (No PR search performed because no failing test/library/error keyword exists to search on.)

Suggested Triage Steps

• Treat as noise for code-health purposes — do not attribute to commit f7632ef8 or any PR; no code ran.
• If signal coverage for the 8-GPU MI35x disaggregation suite is desired, investigate why this job persistently never executes (always skipped or cancelled): confirm whether check-changes/path filters gate it, and whether the linux-mi35x-gpu-8.fabric runner pool can satisfy it within the ~5.5h schedule cadence before the next run supersedes it.
• Consider (maintainer decision) whether large 8-GPU MI35x jobs should run on a less aggressive schedule or a concurrency group that does not get cancelled by the 6-hourly dsv4 cron, so they get a chance to drain the queue.

Status (factual)

• Persistence: Run-level cancelled is the norm — 8 of the last 9 scheduled pr-test-amd-rocm720.yml runs concluded cancelled via schedule overlap. This specific job has not executed once in the queryable window.
• Scope: 39 queued jobs in this single run were cancelled together by the supersession; the underlying capacity/scheduling pattern affects all large multi-GPU AMD jobs in this workflow.
• Blocked work: No real CI signal is being produced for the 8-GPU MI35x disaggregation suite under the current schedule + runner-capacity conditions.
• In-flight fix: N/A — no code defect.

stage-c-test-large-8-gpu-amd-mi35x-rocm720 (linux-mi35x-gpu-8, 1) — failed step(s): N/A

Commit Info

• sglang: f7632ef8
• aiter: N/A (job was cancelled before any setup step ran; no [CI-AITER-CHECK] markers — log blob does not exist)

Failed Tests

Test File	Test Function	Error	Log
N/A	N/A	No test ran — job 82202905953 was cancelled (not failed) before executing any step	(log blob 404 — never produced)

Failure Summary

This job did not fail a test — its conclusion is cancelled. The job sat queued from 18:18:25Z and was cancelled at 18:47:22Z having executed zero steps (the GitHub API reports an empty steps array, and the log blob returns BlobNotFound). The two failure jobs in the run (pr-test-amd-rocm720-finish and pr-test-amd-extra-finish) are aggregator gates that exit 1 whenever any upstream job is cancelled — they are downstream symptoms, not independent failures.

Regression Status

Infrastructure issue (concurrency cancellation) — not a code regression.

Recent history of pr-test-amd-rocm720.yml scheduled runs on main (completed runs only):

Date	Run	Conclusion	This job's state
Jun 18 18:18	27780231199	cancelled	cancelled (0 steps, this run)
Jun 18 13:07	27761630959	cancelled	skipped (stage not selected)
Jun 18 07:39	27744325971	cancelled	skipped (stage not selected)
Jun 18 00:44	27729179568	cancelled	—
Jun 17 18:48	27712035487	cancelled	—

Every one of the last 10 scheduled runs concluded cancelled — a chronic concurrency-churn pattern, not a per-test regression.

Failure Cluster

Scheduled run cancelled by concurrency (newer schedule superseded in-progress run)

Facts (verified observations, no interpretation)

• Job 82202905953 conclusion = cancelled, steps: [], started 18:18:25Z, cancelled 18:47:22Z (~29 min, all queued). [FACT]
• Its log blob returns <Code>BlobNotFound</Code> — no log was ever produced because no step executed. [FACT]
• A newer scheduled run 27781870163 was created at 2026-06-18T18:47:21Z — one second before this job's cancel timestamp 18:47:22Z. [FACT]
• Workflow concurrency block (.github/workflows/pr-test-amd-rocm720.yml:117-123): for schedule events group resolves to pr-test-amd-rocm720-refs/heads/main and cancel-in-progress evaluates to true. [FACT]
• Cron schedule (.github/workflows/pr-test-amd-rocm720.yml:7-9): 30 17 * * * plus 0 */6 * * * — overlapping cadences that fire close together. [FACT]
• The 2 failure jobs in the run are *-finish gates whose script does if result == "failure" || "cancelled"; then exit 1 — they failed solely because upstream jobs were cancelled (verified in finish job log). [FACT]

Hypothesised Causes (with confidence, with disconfirming evidence)

• [HIGH] This run was cancelled by GitHub Actions concurrency: a newer scheduled run on main started at 18:47:21Z and, because cancel-in-progress is true for schedule events sharing group pr-test-amd-rocm720-refs/heads/main, it cancelled the in-progress run 27780231199.
  • Supporting: newer run creation time (18:47:21Z) matches this job's cancel time (18:47:22Z) to the second; concurrency config confirms cancel-in-progress for schedule; all 10 recent scheduled runs are cancelled, consistent with repeated supersession.
  • Disconfirming: (none found) — no test step executed, so no code-level failure is possible.
  • How to verify: inspect run 27781870163's start event vs this run's cancel event; both share concurrency group pr-test-amd-rocm720-refs/heads/main.

In-flight Fix Check

• Searched open PRs for concurrency/schedule changes to this workflow:
  • curl -s -H "Authorization: token $GH_PAT" "https://api.github.com/search/issues?q=repo:sgl-project/sglang+is:pr+is:open+pr-test-amd-rocm720+concurrency"
• In-flight fix: ❌ none found — this is a CI-orchestration / scheduling configuration matter, likely not tracked as a test-fix PR.

Suggested Triage Steps

• Treat this as an infra cancellation, not a test failure — do not bisect f7632ef8.. for a code regression; no test ran.
• Review whether the two overlapping cron schedules (30 17 * * * and 0 */6 * * *) plus GitHub's schedule batching cause runs to launch close together and serially cancel each other under group pr-test-amd-rocm720-refs/heads/main. Consider whether scheduled runs should use a per-run-id group (so they don't cancel each other) the way run_all_tests/workflow_dispatch already do at .github/workflows/pr-test-amd-rocm720.yml:122.
• To get real signal for stage-c-test-large-8-gpu-amd-mi35x-rocm720, re-run this stage manually via workflow_dispatch (which uses the full-{run_id} concurrency group and is exempt from cancel-in-progress) and observe whether tests pass.

Status (factual, no priority assignment)

• Persistence: All 10 most recent scheduled runs of pr-test-amd-rocm720.yml concluded cancelled; this specific job has been either skipped or cancelled (never reached test execution) in the queryable window.
• Scope: Run-wide — 39 of 48 jobs in run 27780231199 were cancelled; this is not specific to the 8-GPU stage-c job.
• Blocked work: Scheduled ROCm 7.2 large-GPU test signal is effectively unavailable while runs keep cancelling each other.
• In-flight fix: ❌ none found.

multimodal-gen-test-2-gpu-amd-rocm720 (linux-mi325-2gpu-sglang, 1) — failed step(s): N/A

Commit Info

• sglang: f7632ef8
• aiter: N/A — the job was cancelled before any runner was assigned; no steps executed, log blob does not exist (BlobNotFound), so no [CI-AITER-CHECK] markers are available.

Failed Tests

Test File	Test Function	Error	Log
N/A	N/A	No test failure — job 82202905956 was cancelled (no runner, 0 steps executed, never started any test)	Job

Failure Summary

This is not a test failure. The job multimodal-gen-test-2-gpu-amd-rocm720 (linux-mi325-2gpu-sglang, 1) was cancelled with no runner assigned and zero steps executed — it never ran. The entire parent run 27780231199 was cancelled (39 jobs cancelled, 7 succeeded, 2 finish-gate jobs marked failure as a cancellation side-effect).

Regression Status

Infrastructure / orchestration issue (concurrency cancellation) — recurring.

Recent history of scheduled pr-test-amd-rocm720.yml runs (branch main, event=schedule, completed runs only):

Date	Run	Conclusion
Jun 18 18:18	27780231199	❌ cancelled (this run)
Jun 18 13:07	27761630959	❌ cancelled
Jun 18 07:39	27744325971	❌ cancelled
Jun 18 00:44	27729179568	❌ cancelled
Jun 17 18:48	27712035487	❌ cancelled
Jun 17 18:15	27710174822	❌ cancelled
Jun 17 13:13	27691581201	❌ cancelled

All 8 most-recent scheduled runs are cancelled. The specific test file for this job (multimodal-gen-test-2-gpu) never executed, so no green/red test history can be established from these runs.

Failure Cluster

Scheduled run cancelled by concurrency group (cancel-in-progress) — no tests executed.

Facts (verified observations, no interpretation)

• Job 82202905956: status=completed, conclusion=cancelled, runner_name= (empty), steps=[] (verified via GET /actions/jobs/82202905956).
• Direct log fetch returns <Error><Code>BlobNotFound</Code> — no log was ever produced (job never started a runner).
• Parent run 27780231199: event=schedule, conclusion=cancelled, created_at=2026-06-18T18:18:02Z, head_sha=f7632ef8.
• Run job census (verified via /runs/27780231199/jobs): 7 success, 39 cancelled, 2 failure (the 2 failures are pr-test-amd-rocm720-finish and call-pr-test-amd-extra-rocm720 / pr-test-amd-extra-finish — gate jobs that fail because their dependencies were cancelled).
• This job's completed_at=2026-06-18T18:47:22Z; the next scheduled run 27781870163 has created_at=2026-06-18T18:47:21Z — a 1-second alignment.
• Workflow concurrency config (.github/workflows/pr-test-amd-rocm720.yml:122-123): group pr-test-amd-rocm720-${{ ... || github.ref }} resolves to pr-test-amd-rocm720-refs/heads/main for schedule events; cancel-in-progress evaluates to true for schedule events.
• Workflow has two schedule crons (.github/workflows/pr-test-amd-rocm720.yml:7-8): 30 17 * * * and 0 */6 * * *.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [HIGH] The run was cancelled by a later scheduled run sharing the same concurrency group, because cancel-in-progress=true for schedule events.
  • Supporting: this job's completed_at (18:47:22) is 1 second after the next scheduled run's created_at (18:47:21); both schedule triggers resolve to the same concurrency group pr-test-amd-rocm720-refs/heads/main (line 122) with cancel-in-progress=true (line 123). The two crons (30 17 daily at 17:30 and 0 */6 at 18:00) both fired late (GitHub schedule delay) and landed ~29 min apart, so the 6-hourly run cancelled the still-running daily run.
  • Disconfirming: (none found) — the timing, concurrency group resolution, and cancel-in-progress flag all converge. A manual cancellation cannot be fully excluded from the API alone, but the 1-second alignment with a new run start strongly indicates concurrency eviction rather than a human cancel.
  • How to verify: inspect the GitHub Actions UI for run 27780231199 — a concurrency-cancelled run shows "Canceling since this run was superseded by run 27781870163".
• [LOW] Code regression in multimodal-gen-test-2-gpu.
  • Supporting: none.
  • Disconfirming: the job executed zero steps and produced no log; no test code ran, so a code-level regression cannot be the cause of this particular cancellation.
  • How to verify: not actionable — wait for a scheduled run that completes without being superseded.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for concurrency/schedule-cancellation fixes:
  • curl "https://api.github.com/search/issues?q=repo:sgl-project/sglang+is:pr+is:open+pr-test-amd-rocm720+concurrency" (no matching open PR for this orchestration issue).

Suggested Triage Steps

• Confirm in the Actions UI that run 27780231199 shows "superseded by run 27781870163" to lock in the concurrency-cancellation diagnosis.
• Review whether the two schedule crons (30 17 * * * and 0 */6 * * *) are intended to share one cancel-in-progress concurrency group; with GitHub schedule delays they can overlap and the 6-hourly run will evict the daily run. Consider differentiating the concurrency group by cron/event (e.g. include github.event.schedule in the group key) so the two crons do not cancel each other.
• Since all 8 recent scheduled runs are cancelled, investigate whether scheduled coverage for this workflow is effectively never completing — this is the higher-order signal worth a maintainer's attention, independent of this single job.

Status (factual, no priority assignment)

• Persistence: All 8 most-recent scheduled runs of pr-test-amd-rocm720.yml (Jun 17 13:13 → Jun 18 18:18 UTC) are cancelled.
• Scope: Run-level cancellation — 39 of 48 jobs in run 27780231199 were cancelled, including this job. Not specific to multimodal-gen-test-2-gpu.
• Blocked work: Scheduled ROCm 7.2 AMD test coverage is not completing; no test signal is being produced by these cancelled scheduled runs.
• In-flight fix: ❌ none found — needs investigation of the schedule/concurrency interaction.

multimodal-gen-test-1-gpu-amd-rocm720 (linux-mi325-1gpu-sglang, 3) — failed step(s): N/A

Commit Info

• sglang: f7632ef8
• aiter: N/A (job log blob expired — BlobNotFound — no [CI-AITER-CHECK] markers available)

Failed Tests

Test File	Test Function	Error	Log
(none — job cancelled, not a test failure)	—	Job cancelled by concurrency cancel-in-progress	Job

Failure Summary

The job multimodal-gen-test-1-gpu-amd-rocm720 (linux-mi325-1gpu-sglang, 3) did not fail a test — its conclusion is cancelled. It ran for ~29 min (18:18:25 → 18:47:22 UTC) and was terminated mid-flight, with no per-step records and no retrievable log (the blob returns BlobNotFound). This is an orchestration/infrastructure cancellation, not a code or test regression.

Regression Status

Infrastructure issue (concurrency cancellation) — not a test regression.

The whole run 27780231199 was superseded: of its 48 jobs, 39 were cancelled at the identical instant 2026-06-18T18:47:22Z, 7 succeeded, and the only 2 failures are the gate/finish jobs (pr-test-amd-rocm720-finish, call-pr-test-amd-extra-rocm720 / pr-test-amd-extra-finish) that always go red when upstream jobs are cancelled.

Recent scheduled runs of pr-test-amd-rocm720.yml (event=schedule, branch=main, completed only):

Date (UTC)	Run	Conclusion
Jun 18 13:07	27761630959	cancelled
Jun 18 07:39	27744325971	cancelled
Jun 18 00:44	27729179568	cancelled
Jun 17 18:48	27712035487	cancelled
Jun 17 18:15	27710174822	cancelled

Every recent scheduled run is cancelled — the test file multimodal-gen-test itself has no recoverable pass/fail signal because the runs are killed before completion.

Failure Cluster

Concurrency cancellation of in-progress scheduled run (superseded by next scheduled run)

Facts (verified observations, no interpretation)

• Job conclusion is cancelled, not failure; it has zero recorded steps (verified via /actions/jobs/82202905962).
• Job log is unavailable: GET /actions/jobs/82202905962/logs returns <Code>BlobNotFound</Code> (verified).
• All 39 cancelled jobs in the run share the exact same completed_at = 2026-06-18T18:47:22Z (FACT — min == max across cancelled jobs).
• The next scheduled run 27781870163 (event=schedule, head 61a8b42c) was created at 2026-06-18T18:47:21Z — one second before the cancellation (verified).
• Workflow concurrency config (.github/workflows/pr-test-amd-rocm720.yml:117-123): group pr-test-amd-rocm720-${{ ... || github.ref }} (→ …-refs/heads/main for schedule) with cancel-in-progress: ${{ !inputs.run_all_tests && github.event_name != 'workflow_call' && github.event_name != 'workflow_dispatch' }} (→ true for schedule events).
• The 2 failure jobs in the run are both *-finish gate jobs, which fail-by-design when upstream jobs are cancelled.

Hypothesised Causes (with confidence)

• [FACT] The job was cancelled by GitHub Actions concurrency cancel-in-progress, triggered when the next scheduled run 27781870163 entered the shared concurrency group pr-test-amd-rocm720-refs/heads/main.
  • Supporting: identical cancel timestamp across all 39 jobs; superseding scheduled run created 1s prior; concurrency block resolves to cancel-in-progress: true for schedule events on the same github.ref.
  • Disconfirming: (none found) — no test error exists; the log is absent, so there is no competing failure signal.
  • How to verify: already verified from API metadata and the workflow YAML.
• Why two crons collide: the workflow defines two schedule crons (30 17 * * * and 0 */6 * * *). The 17:30 daily cron (this run, delayed-started 18:18) and the 18:00 6-hourly cron (next run, started 18:47) land within the same window and share the …/main concurrency group, so the later one cancels the earlier. [MEDIUM] — supported by the two-cron config and the run creation times; not independently confirmed that 18:47 run is specifically the 0 */6 cron vs a delayed daily.

In-flight Fix Check

• In-flight fix: ❌ none needed — this is not a defect. Searched not applicable (no failing test/library to key on). If the recurring whole-run cancellation of scheduled pr-test-amd-rocm720 is considered undesirable signal loss, that is a workflow-scheduling concern, not a code bug.

Suggested Triage Steps

• No code triage required — this is not a test or product failure. Treat as an expected concurrency cancellation.
• If reliable nightly signal is wanted: review whether the 30 17 * * * and 0 */6 * * * crons should share the same …/main concurrency group, since back-to-back scheduled runs currently cancel each other (every recent scheduled run is cancelled). Consider giving schedule runs a per-run-id group (like workflow_dispatch already does) so a later cron doesn't kill an in-progress nightly.
• To recover this specific job's signal, look at a later scheduled run that completed without being superseded; the expired log here is unrecoverable.

Status (factual, no priority assignment)

• Persistence: Whole-run cancellation is the recurring norm — all 5 most recent completed scheduled runs (Jun 17–18) concluded cancelled.
• Scope: Run-wide — 39/48 jobs in 27780231199 cancelled simultaneously; this specific job is one of them.
• Blocked work: Scheduled pr-test-amd-rocm720 produces no usable pass/fail signal when runs supersede each other; multimodal-gen-test shard 3 has no recoverable result for this run.
• In-flight fix: none needed (not a defect).

stage-c-test-large-8-gpu-amd-rocm720 (linux-mi325-8gpu-sglang, 2) — failed step(s): N/A

Commit Info

• sglang: f7632ef8
• aiter: N/A (job was cancelled during setup; no [CI-AITER-CHECK] markers were emitted, and the log blob does not exist)

Failed Tests

Test File	Test Function	Error	Log
N/A	N/A	Job cancelled, not failed — no tests ran	Job

This was not a test failure. Job 82202905965 has conclusion = cancelled, an empty steps list, and its log blob returns BlobNotFound — i.e. it was terminated during queue/setup before any test step executed.

Failure Summary

The job stage-c-test-large-8-gpu-amd-rocm720 (linux-mi325-8gpu-sglang, 2) was cancelled at 2026-06-18T18:47:22Z, along with 38 other jobs in run 27780231199 (all 39 cancelled at the same instant). The only two failure jobs in the run are the aggregator *-finish jobs, which fail by design when their upstream matrix jobs are cancelled. No test code ran in the target job.

Regression Status

Infrastructure issue (concurrency cancellation) — not a code regression and not a flaky test.

Root mechanism: this workflow's concurrency block (pr-test-amd-rocm720.yml:117-123) sets
group: pr-test-amd-rocm720-…|| github.ref → pr-test-amd-rocm720-refs/heads/main for schedule events, with cancel-in-progress: true (true whenever the event is not workflow_call/workflow_dispatch and run_all_tests is false). When the next scheduled run is created, it preempts the in-progress one on the shared main group.

Evidence of recurrence (all schedule, branch=main, completed runs — each run is cancelled by its successor):

Created (UTC)	Run	Conclusion	Cancelled when next run created
2026-06-18 18:47:21	27781870163	queued (successor; created 1s before our cancel)	—
2026-06-18 18:18:02	27780231199 (this run)	cancelled @ 18:47:22	← cancelled by 27781870163
2026-06-18 13:07:27	27761630959	cancelled @ 18:18:10	← cancelled by 27780231199
2026-06-18 07:39:23	27744325971	cancelled @ 13:07:44	← cancelled by 27761630959
2026-06-18 00:44:16	27729179568	cancelled @ 07:39:37	← cancelled by 27744325971

Every recent scheduled run carries conclusion = cancelled, confirming a chronic preemption pattern rather than a one-off.

Failure Cluster

Concurrency preemption of scheduled run (job cancelled during setup, no tests executed)

Facts (verified observations, no interpretation)

• [FACT] Job 82202905965: status=completed, conclusion=cancelled, started 18:18:25Z, completed 18:47:22Z, empty steps array.
• [FACT] The job log blob returns <Code>BlobNotFound</Code> — no log was ever persisted, consistent with cancellation before test steps ran.
• [FACT] 48 jobs in run 27780231199: 7 success, 39 cancelled, 2 failure. All 39 cancellations share the identical timestamp 2026-06-18T18:47:22Z.
• [FACT] The 2 failure jobs are aggregators: call-pr-test-amd-extra-rocm720 / pr-test-amd-extra-finish (82208456903) and pr-test-amd-rocm720-finish (82225249578). No test-bearing job reports failure.
• [FACT] Successor scheduled run 27781870163 was created 2026-06-18T18:47:21Z, one second before the mass cancel.
• [FACT] pr-test-amd-rocm720.yml:122-123: schedule-event runs share concurrency group pr-test-amd-rocm720-refs/heads/main with cancel-in-progress: true.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [HIGH] The job was preempted by the next scheduled run via the shared cancel-in-progress concurrency group on refs/heads/main.
  • Supporting: successor run 27781870163 created at 18:47:21Z, exactly 1s before all 39 jobs cancel at 18:47:22Z; concurrency config at pr-test-amd-rocm720.yml:122-123 resolves to group pr-test-amd-rocm720-refs/heads/main + cancel-in-progress: true for schedule events; every recent scheduled run shows conclusion=cancelled.
  • Disconfirming: (none found) — the timing, config, and recurring pattern all converge.
• [LOW] Underlying capacity contention: scheduled runs appear heavily queued/delayed (the daily 30 17 and 6h 0 */6 crons firing far from their nominal times), which lengthens overlap windows so successors preempt predecessors before they finish.
  • Supporting: our run created 18:18:02 (nominal 17:30 cron) and successor 18:47:21 (nominal 18:00 cron) — both delayed.
  • Disconfirming: delay alone doesn't cancel jobs; the cancellation is mechanically the concurrency group, not capacity.

In-flight Fix Check

• Searched open PRs for concurrency/cancel-in-progress changes to the AMD ROCm 7.2 workflow:
  curl -s -H "Authorization: token $GH_PAT" "https://api.github.com/search/issues?q=repo:sgl-project/sglang+is:pr+is:open+concurrency+rocm720" → ❌ none found matching this orchestration issue.
• In-flight fix: ❌ none found — this is a CI-orchestration matter, likely needs a workflow config change or is expected behavior for backed-up scheduled runs.

Suggested Triage Steps

• Treat this job as not a real failure — exclude cancelled jobs (and the *-finish aggregator failures they cause) from failure tracking for this run. No test regression occurred.
• If chronic preemption of the daily 30 17 schedule is undesirable, consider whether the schedule concurrency group should use cancel-in-progress: false (let scheduled runs complete) or a per-run-id group — review pr-test-amd-rocm720.yml:122-123. This is a maintainer config decision, not a directive.
• To obtain real signal for this 8-GPU stage-c shard, re-run the workflow via workflow_dispatch (which gets a unique full-<run_id> concurrency group and cancel-in-progress: false, so it won't be preempted) and re-analyze.
• Confirm runner-capacity/scheduling backlog with infra if delayed scheduled starts (17:30 cron firing at 18:18) are unexpected.

Status (factual, no priority assignment)

• Persistence: All recent schedule runs of pr-test-amd-rocm720.yml on main end as cancelled (at least the last 5+ consecutive runs through 2026-06-18); each is preempted by its successor.
• Scope: 39 of 48 jobs cancelled in run 27780231199; the target 8-GPU stage-c shard never executed any test. No test-level failures anywhere in the run.
• Blocked work: Daily scheduled AMD ROCm 7.2 CI signal is not being produced via the cron path — runs are cancelled before tests finish.
• In-flight fix: ❌ none found in open PRs.

multimodal-gen-test-1-gpu-amd-rocm720 (linux-mi325-1gpu-sglang, 1) — failed step(s): N/A

Commit Info

• sglang: f7632ef8
• aiter: N/A — job log is unavailable (GitHub returned BlobNotFound for the job log); no [CI-AITER-CHECK] markers are recoverable. The job never reached the dependency-install step, so no aiter version was ever resolved.

Failed Tests

Test File	Test Function	Error	Log
N/A	N/A	Job cancelled before any runner was assigned — zero steps executed, no test ran	Job

This is not a test failure. The job multimodal-gen-test-1-gpu-amd-rocm720 (linux-mi325-1gpu-sglang, 1) was cancelled with runner_id: 0 / runner_name: "" and steps: []. It sat queued waiting for an MI325 runner from 18:18:25Z and was cancelled at 18:47:22Z without ever starting.

Failure Summary

The entire run 27780231199 (PR Test ROCm 7.2 (AMD), event=schedule) ended with conclusion=cancelled. Of 48 jobs: 7 succeeded (fast gate/check jobs), 39 were cancelled with no runner assigned (0 steps), and 2 failure jobs are only the pr-test-amd-rocm720-finish / pr-test-amd-extra-finish aggregators reacting to the cancellation. The target multimodal job is one of the 39 that never executed.

Regression Status

Infrastructure issue (concurrency cancellation) — not a code regression and not a flaky test.

Recent history of job multimodal-gen-test-1-gpu-amd-rocm720 (linux-mi325-1gpu-sglang, 1) (workflow pr-test-amd-rocm720.yml, branch=main, event=schedule, completed runs only):

Date	Run	Run Conclusion	This job's status
Jun 18 18:18	27780231199	cancelled	❌ Cancelled (no runner, 0 steps) — target
Jun 18 13:07	27761630959	cancelled	⏭️ Skipped (6h 0 */6 * * * cron runs dsv4 jobs only; multimodal matrix not expanded)
Jun 18 07:39	27744325971	cancelled	⏭️ Skipped (6h cron)

Note: every recent scheduled run of pr-test-amd-rocm720.yml on main carries conclusion=cancelled (12/12 in the queryable window) — this is the steady-state behavior of this scheduled workflow, not a new event. No green completed run of this full matrix exists in the window to serve as a passing baseline for the multimodal test.

Failure Cluster

Run cancelled by concurrency supersede — job never scheduled onto a runner (no test executed).

Facts (verified observations)

• Job 82202905970: status=completed, conclusion=cancelled, runner_id=0, runner_name="", steps=[], queued 18:18:25Z → cancelled 18:47:22Z (~29 min in queue). [FACT]
• Run 27780231199: event=schedule, conclusion=cancelled; 48 jobs → 7 success, 39 cancelled (all runner=NONE, 0 steps), 2 failure (both *-finish aggregators). [FACT]
• The job log is not retrievable: GitHub API returns <Error><Code>BlobNotFound</Code> — consistent with a job that produced no logs because it never ran. [FACT]
• The next scheduled run 27781870163 was created at 18:47:21Z — one second before this run's jobs were cancelled at 18:47:22Z. [FACT]
• Workflow concurrency (.github/workflows/pr-test-amd-rocm720.yml:117-123): group pr-test-amd-rocm720-${{ github.ref }} (= ...-refs/heads/main for schedule) with cancel-in-progress: true when the event is schedule. [FACT]

Hypothesised Causes

• [HIGH] The in-progress run was cancelled by the workflow's own cancel-in-progress concurrency rule when the next scheduled run entered the same group pr-test-amd-rocm720-refs/heads/main.
  • Supporting: next scheduled run created 18:47:21Z ↔ jobs cancelled 18:47:22Z (1 s apart); concurrency config at pr-test-amd-rocm720.yml:122-123 sets cancel-in-progress: true for schedule events on the shared github.ref group; all 39 unstarted jobs cancelled simultaneously with no runner ever assigned.
  • Disconfirming: (none found) — timing, config, and the "no runner / 0 steps" state all converge.
  • How to verify: compare created_at of run 27781870163 vs completed_at of the cancelled jobs (already done); inspect the run's cancellation actor via the GitHub UI timeline.
• [LOW] Runner-capacity starvation: the MI325 (linux-mi325-1gpu-sglang) pool may have been saturated so the job sat queued 29 min and would not have completed even without supersede.
  • Supporting: 29 min in queue with no runner; many MI325 jobs queued in the same run.
  • Disconfirming: the cancellation is fully explained by the concurrency supersede above; queue time alone did not cause the cancel.
  • How to verify: check MI325 runner-pool utilization / queue depth at 18:18–18:47Z.

In-flight Fix Check

Not applicable — there is no code defect to fix. Cancellation is the workflow's intended concurrency behavior. (Searched no PRs, since the "failure" is an orchestration artifact, not a test/code regression.)

Suggested Triage Steps

• Treat this job as "no signal", not "failed". No test ran, so it tells you nothing about f7632ef8. Disregard it for code-health purposes.
• If full multimodal-on-AMD signal is needed for this SHA, re-run the daily-cron matrix on a quiet window, or dispatch pr-test-amd-rocm720.yml via workflow_dispatch (which gets a unique concurrency group and is not auto-cancelled).
• If these back-to-back scheduled cancellations are unwanted, consider whether the daily 30 17 and 6-hourly 0 */6 crons are colliding on the shared refs/heads/main group; the monitoring harness could filter out conclusion=cancelled scheduled runs to avoid false-positive failure alerts.

Status

• Persistence: Single cancelled run; the scheduled workflow has shown conclusion=cancelled on every run in the queryable window (orchestration norm, not a persistent test failure).
• Scope: Run-wide — 39/48 jobs in run 27780231199 cancelled by the same supersede event, not specific to the multimodal job.
• Blocked work: No multimodal-gen AMD ROCm 7.2 CI signal was produced for this SHA; no code is blocked.
• In-flight fix: N/A — not a code defect.

multimodal-gen-test-1-gpu-amd-rocm720 (linux-mi325-1gpu-sglang, 2) — failed step(s): N/A

Commit Info

• sglang: f7632ef8
• aiter: N/A — the job never executed, produced no log (BlobNotFound), so no [CI-AITER-CHECK] markers exist.

Failed Tests

Test File	Test Function	Error	Log
N/A — no test ran	N/A	Job cancelled before any step executed (concurrency cancellation)	Job

This is not a test failure. The job's conclusion is cancelled, it recorded zero steps, and its log blob does not exist (BlobNotFound), meaning it was still queued/waiting for a linux-mi325-1gpu-sglang runner and never started executing any multimodal-gen test.

Failure Summary

The job was cancelled, not failed. The entire run 27780231199 (event=schedule, cancelled) had 39 of 48 jobs cancelled at the identical timestamp 2026-06-18T18:47:22Z, while only 7 lightweight jobs that had already grabbed runners completed successfully. No cancelled job in the run executed a single step. This job exercised no multimodal-gen test.

Regression Status

Infrastructure issue (concurrency cancellation) — not a code regression.

Recent schedule runs of pr-test-amd-rocm720.yml on main (completed only):

Date	Run	Event	Conclusion
Jun 18 18:18	27780231199	schedule	❌ cancelled (this run)
Jun 18 13:07	27761630959	schedule	cancelled
Jun 18 07:39	27744325971	schedule	cancelled
Jun 18 00:44	27729179568	schedule	cancelled
Jun 17 18:48	27712035487	schedule	cancelled

Every recent schedule run ends cancelled — a chronic pattern, because each new schedule run cancels the previous in-progress one on the shared main concurrency group.

Failure Cluster

Job cancelled by concurrency group (run superseded before execution) — orchestration artifact, not a test failure.

Facts (verified observations)

• Job conclusion: cancelled; steps: []; log endpoint returns BlobNotFound → never started executing. [FACT]
• All 7 multimodal-gen shards (1gpu shards 0-3, 2gpu shards 0-2) cancelled at the same 2026-06-18T18:47:22Z. [FACT]
• Run-wide: 7 success, 2 failure (only the *-finish gate jobs, which report failure when dependents are cancelled), 39 cancelled. No cancelled job executed any non-skipped step. [FACT]
• The next schedule run 27781870163 was created 2026-06-18T18:47:21Z — one second before this run's jobs were cancelled. [FACT]
• Workflow concurrency: pr-test-amd-rocm720.yml:122-123 sets group pr-test-amd-rocm720-${{ github.ref }} with cancel-in-progress: true for schedule events. [FACT]
• The 7 jobs that succeeded are lightweight ones (call-gate, check-changes, sgl-kernel-unit-test, jit-kernel-unit-test, a stage-b on mi35x) that obtained runners before the cancellation; multimodal-gen jobs were still waiting for scarce mi325-1gpu runners. [FACT]

Hypothesised Causes

• [FACT] Concurrency-group cancellation. The newer scheduled run 27781870163 (created 18:47:21Z) preempted this run via cancel-in-progress: true on the shared refs/heads/main group (pr-test-amd-rocm720.yml:122-123), cancelling all still-queued jobs at 18:47:22Z.
  • Supporting: one-second alignment between the new run's creation and this run's cancellation; the workflow's own concurrency config; zero executed steps on all cancelled jobs.
  • Disconfirming: (none found) — there is no evidence of any test or build error, since no step ran.
  • How to verify: already verified from job metadata + workflow file; no further action needed.

In-flight Fix Check

Not a code defect, so no fix PR applies. The underlying capacity/orchestration question (schedule runs serially cancelling each other before mi325-1gpu runners free up) is an infra concern, not a sglang source fix.

• In-flight fix: ❌ none found — no source-code defect to fix; this is a runner-capacity / scheduling artifact.

Suggested Triage Steps

• Treat this job as a non-signal: it never ran the multimodal-gen tests, so it neither passes nor fails the PR/commit.
• If genuine multimodal-gen signal on main is needed for f7632ef8, re-run via workflow_dispatch (which uses a unique full-<run_id> concurrency group per pr-test-amd-rocm720.yml:122 and is exempt from cancel-in-progress), so it won't be preempted by the next schedule tick.
• Infra follow-up (optional): the back-to-back schedule cadence (~5-6 h apart, but here a new run appeared ~29 min later) combined with scarce mi325-1gpu runners means scheduled multimodal-gen jobs are routinely cancelled before starting. Consider reducing schedule frequency or increasing mi325-1gpu runner capacity so scheduled runs can complete.

Status

• Persistence: Every recent schedule run of this workflow ends cancelled; multimodal-gen jobs on mi325-1gpu are chronically preempted before execution.
• Scope: Run-wide orchestration artifact — 39/48 jobs cancelled in this run, including all 7 multimodal-gen shards.
• Blocked work: No multimodal-gen schedule signal is being produced on main for this workflow; results come only from on-demand workflow_dispatch runs.
• In-flight fix: ❌ none found — no source-code defect; runner-capacity / scheduling concern.

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-06-19 19:28 UTC)