[CI Monitor] Daily Report - 2026-07-22

[amd-bot]

Daily Cross-Workflow Summary — 2026-07-22

Snapshot: 2026-07-22 22:48 UTC · Only completed runs/jobs counted in trends · Auto-updated every 30 min

TL;DR

🔴 RED · 5 active clusters · 3 🆕 today (R285 + R286 both from PR #31816; R287 reward-model) · R270 + R257 carrying over · in-flight fixes: R270→#31741 (open, needs rebase), R257→#31068/#32003. R285/R286/R287 have NO fix PR.
👉 Today's ask: (1) PR #31816 (745b2ca4, merged 08:18 UTC) get_server_args()→get_parallel() refactor caused two deterministic breaks — R285 (test patches removed communicator.get_server_args) and R286 (Dynamo can't trace ParallelContext.__getattr__ during tc_piecewise DeepSeek prefill capture). Both [HIGH], both cross-workflow, no fix open → needs a follow-up PR or partial revert of the dsa/utils.py+test hunks. (2) R287 reward-model _legacy_load_weights AttributeError from PR #28671 (becf252e6, merged Jul-21 21:32) [HIGH], no fix. (3) Chase/rebase #31741 (R270 mamba NameError, now 7th day).

Caution

Both Jul-22 nightly runs (nightly-amd 29944275726, rocm720 29944173150) are still queued (IN-FLIGHT) at snapshot and run on head 0bdd4730, which contains #31816 — expect the DeepSeek MI35x nightly legs to hit R286 too. Trends below therefore use the latest completed nightly runs (Jul-21 29855050299 / 29854963573, head 9a6c9608 — predates #31816).

Note

Good news: release-docker-amd-nightly run 29920483445 went ✅ green on attempt 2 — yesterday's apt-get libpfm4 archive fetch (LLVM-18/TileLang stage) was a transient network flake, not a code issue. release-docker-amd-rocm720-nightly and nightly-amd-mi355x-disagg remain ✅ green. No AITER Scout run today (Mon/Thu cadence).

Workflow status

Workflow	Latest completed run	❌ jobs	Today's status	Δ vs yesterday
nightly-test-amd	Jul-21 29855050299 (failure)	8	Jul-22 29944275726 (IN-FLIGHT)	R257 + R270 persist
nightly-test-amd-rocm720	Jul-21 29854963573 (failure)	9	Jul-22 29944173150 (IN-FLIGHT)	R257/R262/R270/R273 carry
release-docker-amd-nightly	Jul-22 29920483445 ✅ (attempt 2)	0	green (apt-get flake recovered)	✅ back to green
release-docker-amd-rocm720-nightly	Jul-22 ✅	0	green	✅ still green
nightly-amd-mi355x-disagg	Jul-22 ✅	0	green	✅ still green
amd-aiter-scout	(no run today)	—	n/a	—
pr-test-amd	Jul-22 29920027108 (failure)	completed ❌	🆕 R285/R286/R287	#31816 breaks land
pr-test-amd-rocm720	Jul-22 29944235234 (failure)	completed ❌	🆕 R285/R286; R277 persists	#31816 breaks land

7d run-conclusion trend (older→newer, completed scheduled runs; ✅=green): nightly-test-amd ❌·❌·❌·❌·❌ · nightly-rocm720 ❌·❌·❌·❌·❌ · mi355x ✅·✅·✅·✅·✅ · pr-test-amd ❌·❌·❌·❌·❌ · pr-test-amd-rocm720 ❌·❌·❌·❌·❌ · release-docker-amd ✅·✅·✅·✅·✅ · release-docker-rocm720 ✅·✅·✅·✅·✅.

Failure clusters (deduplicated across all workflows)

R285 · 🆕 · AttributeError: communicator module has no 'get_server_args' — test patches a symbol removed by the get_parallel() refactor — pr-test-amd + pr-test-amd-rocm720 · ❌ none

• Status: NEW today, deterministic (failed on built-in retry). Last passed Jul-22 00:26 UTC (29880220845); first fail this run. Regression window 8ae0eb83...b13abfde contains the merge.
• Top hypothesis: [HIGH] PR #31816 (745b2ca4) deleted the get_server_args import from communicator.py; the test added by #31784 still does mock.patch.object(comm, "get_server_args", …) at test_aiter_allreduce_fusion_amd.py:421 → unittest.mock raises on a non-existent attr. Disconfirming: (none — removed symbol == patched symbol, verified by grep; removal lands exactly in-window).
• In-flight fix: ❌ none (searched get_server_args in title/body → 0 relevant). Suggested triage: repoint the patch to the new accessor (get_parallel), or add a thin get_server_args shim; also check the [LOW] secondary get_parallel mock gap (moe_dense_tp_size).

Workflow	Job (shard)	Test File	Test Function	Error	Log
pr-test-amd	stage-c-large-8gpu (3)	test/registered/ops/test_aiter_allreduce_fusion_amd.py	TestAiterAllreduceFusionGate.test_dense_tp_fuses	AttributeError: …communicator… has no 'get_server_args' → exit 255	Log
pr-test-amd-rocm720	stage-c-large-8gpu-rocm720 (3)	test_aiter_allreduce_fusion_amd.py	test_dense_tp_fuses	same	Log

R286 · 🆕 · Dynamo graph-break capturing DeepSeek prefill under tc_piecewise — __getattribute__ of ParallelContext untraceable — pr-test-amd + pr-test-amd-rocm720 MI35x · ❌ none

• Status: NEW today, deterministic (all 8 TP ranks). Passed 3/3 Jul-21 12:32 (29830494225); first fail this run. Same root PR as R285.
• Top hypothesis: [HIGH] PR #31816 (745b2ca4) changed is_dsa_enable_prefill_cp() (dsa/utils.py:79) from get_server_args().… (traceable field) to get_parallel().…, which resolves via ParallelContext.__getattr__→object.__getattribute__ (runtime_context.py:130) — the exact frame Dynamo reports (gb0156) inside the compiled prefill region. Disconfirming: (none — code diff + timing + verbatim UserDefinedObjectVariable(ParallelContext)/'_config' all converge).
• In-flight fix: ❌ none. Suggested triage: revert only the dsa/utils.py hunk (restore get_server_args() for this leaf) or make ParallelContext config-leaf access Dynamo-traceable / hoist the read out of the compiled region; A/B with --cuda-graph-backend-prefill=breakable.

Workflow	Job (shard)	Test File	Test Function	Error	Log
pr-test-amd	stage-c-large-8gpu-mi35x (1)	test/registered/amd/test_deepseek_r1_mxfp4_8gpu.py	setUpClass (server launch)	RuntimeError: Capture prefill CUDA graph failed / torch._dynamo.exc.Unsupported → -9	Log
pr-test-amd-rocm720	stage-c-large-8gpu-mi35x-rocm720 (2)	test_deepseek_r1_mxfp4_8gpu.py	TestDeepseekR1MXFP4.setUpClass	same Dynamo break → exit -9	Log

R287 · 🆕 · AttributeError: '…SequenceClassification…' object has no attribute '_legacy_load_weights' — reward-model weight load — pr-test-amd nondeterministic shard · ❌ none

• Status: NEW today, deterministic. Two preceding scheduled runs passed; neither head contained the suspect commit. Window e4eea7ce2f..8ae0eb83fc.
• Top hypothesis: [HIGH] PR #28671 (becf252e6, merged Jul-21 21:32) refactored LlamaForCausalLM.load_weights to dispatch to self._legacy_load_weights; reward-model classes call it with a self that doesn't subclass LlamaForCausalLM → unresolvable. Disconfirming: (none — method name in error == method the commit added; modified llama.py:662 is last SGLang frame).
• In-flight fix: ❌ none (searched _legacy_load_weights, SequenceClassification load_weights → 0). Suggested triage: guard the reward-model load path against the new dispatch, or have those classes provide _legacy_load_weights; confirm with #28671 author.

Workflow	Job (shard)	Test File	Test Function	Error	Log
pr-test-amd	stage-b-1gpu-small-nondeterministic	test/registered/prefill_only/test_reward_models.py	TestRewardModels.test_reward_scores	AttributeError: no attribute '_legacy_load_weights' → -9	Log

R270 · NameError: transfer_kv_mamba_lf_pf — mamba JIT transfer kernel CUDA-only import guard on AMD — nightly-amd + nightly-rocm720 kernel · ✅ #31741 (open, needs rebase)

• Status: carry-over, 7th day (first Jul-16), never-passed on AMD. 12 NameError subtests each on completed Jul-21 kernel legs (and again on the IN-FLIGHT Jul-22 runs).
• Top hypothesis: [FACT] #30535/#31501 gated transfer_kv_mamba_* imports behind if _is_cuda: (memory_pool_host.py:41) while the call site (:407) is reachable on HIP. Disconfirming: (none — NameError directly observable).
• In-flight fix: ✅ #31741 (open since Jul-20; mergeable_state: dirty — needs rebase). Suggested triage: rebase/merge #31741 (widens guard to _is_cuda or _is_hip + HIP .cuh compat), re-run test_transfer_mamba.py.

Workflow	Job (shard)	Test File	Test Function	Error	Log
nightly-test-amd	nightly-test-1-gpu-kernel	test/registered/jit/test_transfer_mamba.py	12 subtests (roundtrip/single_item/full_indices)	NameError: transfer_kv_mamba_lf_pf → exit 255	Log
nightly-test-amd-rocm720	nightly-test-1-gpu-kernel-rocm720	test_transfer_mamba.py	same 12 subtests	same	Log

R257 · MI35x (gfx950) GPU hang / HW-exception family — hipErrorCapturedEvent · GPU memory access fault · stream-capture abort · warmup hang — nightly-amd + nightly-rocm720 MI35x · ⚠️ #31068/#32003

• Status: carry-over, 14th day (first Jul-09/10). Model-/workflow-independent; alternates pass/fail on the same physical MI35x nodes → environment over a bisectable regression.
• Top hypothesis: [MEDIUM] non-capturable MoE TP all-reduce inside the decode HIP-graph capture stream → RCCL watchdog stream-capture violation; compounded by MI35x node/driver hang. Disconfirming: alternates on identical SHAs.
• In-flight fix: ⚠️ #31068 (hung-GPU pre-flight recovery), #32003 (route scheduled jobs to MI300). Suggested triage: make MoE TP all-reduce capture-safe on ROCm; chase #31068/#32003; else reboot the MI35x pool.

Workflow	Job (shard)	Test File	Test Function	Error	Log
nightly-test-amd	nightly-8-gpu-mi35x-qwen3-235b-mxfp4	test_qwen3_instruct_mxfp4.py	setUpClass	hipErrorCapturedEvent (aiter allreduce flush_graph_buffers) → -6	Log
nightly-test-amd	nightly-2-gpu-mi35x-glm51-mxfp4	test_glm51_mxfp4_tp2_gsm8k_mi35x.py	setUpClass	GPU hang / server-load hang	Log
nightly-test-amd	nightly-accuracy-8-gpu	test_gsm8k_eval_amd.py (Grok1-FP8)	eval	GPU Hang (HW exception, node-9) → SIGABRT	Log
nightly-test-amd-rocm720	nightly-8-gpu-mi35x-qwen3-235b-mxfp4-rocm720	test_qwen3_instruct_mxfp4.py	setUpClass	hipErrorCapturedEvent (RCCL watchdog) → -6	Log
nightly-test-amd-rocm720	nightly-accuracy-8-gpu-mi35x-rocm720	test_qwen3_moe_eval_mi35x.py	test_qwen3_moe_accuracy (Qwen3-30B-A3B)	GPU memory access fault → abort	Log

Known stable / recurring clusters (carrying over, no new action today) · click to expand

Cluster	Workflow · Rep. job	Test / fn	Status	In-flight fix
R262 InternVL2_5-2B CLIPImageProcessor has no .tokenizer	nightly-rocm720 vlm	test_vlms_mmmu_eval_amd.py (InternVL2_5-2B)	recurring since Jul-14	❌
R273 + accuracy near-misses: DeepSeek-V4-Flash FP8-TBO (0.9098), DeepSeek-V3.2 basic, MiniMax-M2.5, GPT-OSS-20b, MXFP4-requant MoE, DeepSeek-R1-MXFP4 MTP (~0.85)	nightly-rocm720 v4-flash · nightly-amd dsv32 · minimax	test_deepseek_v4_flash_fp8_tbo.py etc.	marginal/boundary	⚠️ threshold recalib precedent #31702
Diffusion (multimodal-gen) family: 1-/2-GPU hang → 60-min step timeout, runner-lost/no-log, HIPBLAS_STATUS_NOT_SUPPORTED (flux2 fp8), update_weights_from_disk shape-mismatch, msgpack ModuleNotFound	pr-test-amd mm-gen 1gpu (0) · rocm720 mm-gen-unit ×many	test_server_1_gpu.py / test_server_2_gpu.py / test_pi05_action_api.py	never-passed/recurring	⚠️ msgpack #31844; skip #31925
Disaggregation family: Gloo P2P deadlock (PP-decode), R277 Mooncake 30s KV-transfer timeout, KeyError:'answer' PD+PP, R282 Kimi-K2.5-MXFP4 load-barrier SIGKILL	rocm720 disagg gloo · mooncake · kimi-k25	test_disaggregation_pp.py / test_kimi_k25_mxfp4_bcg_mi35x.py	recurring/flaky	⚠️ R277→#31800
R284 LoRA concurrent-adapter-update E2E latency 3-4× over 4000 ms gate	pr-test-amd stage-b-1gpu-large (0)	test_bench_serving_1gpu_part1.py test_online_lora_latency_with_concurrent_adapter_updates	flaky/borderline	❌
Triton offline throughput below AMD gate; Mixtral-8x7B gsm8k below threshold; VLM MMMU ~chance (5 models) + GLM-4.1V-9B encoder-DP 3600s timeout; DSA topk_v2 JIT / DeepSeek-V3.2 shortfall gfx950	pr-test-rocm720 triton · nightly-rocm720 mixtral · nightly-amd vlm	various	never-passed/recurring	mostly ❌; DSA #31291
Runner-lost comms / infra (MI325 & MI35x runner death, no log)	pr-test-amd mm-gen 1gpu (1) ×many	infra	recurring	⚠️ #31068, #32003

Workflow drill-down · click to expand

nightly-test-amd · latest completed 29855050299 · 8 failures: R270 (kernel) · R257 (qwen3-235b-mxfp4, glm51-mxfp4, grok1 8-gpu) · accuracy near-miss (dsv32, minimax-m25) · VLM MMMU (2-gpu-vlm) · Mixtral hang (accuracy-2-gpu).

nightly-test-amd-rocm720 · latest completed 29854963573 · 9 failures: R270 (kernel) · R257 (qwen3-235b-mxfp4, qwen3-30b-a3b mem-fault) · R262 (InternVL2_5) · R273 (dsv4-flash fp8-tbo, dsv32 basic) · DeepSeek-R1-MXFP4 MTP accuracy · Mixtral gsm8k.

pr-test-amd · latest completed 29920027108 · 🆕 R285 (stage-c-8gpu-3) · 🆕 R286 (stage-c-8gpu-mi35x-1) · 🆕 R287 (nondeterministic, earlier run) · diffusion family (mm-gen shards) · runner-lost comms.

pr-test-amd-rocm720 · latest completed 29944235234 · 🆕 R285 (stage-c-8gpu-3) · 🆕 R286 (stage-c-8gpu-mi35x-2) · R277 Mooncake (disagg) · Triton throughput · diffusion family + msgpack · DSA piecewise (mi35x-2).

How this report is generated

• Only status == "completed" runs/jobs counted in trends. Both Jul-22 nightly runs (29944275726, 29944173150) were queued at snapshot → (IN-FLIGHT), excluded; latest completed nightlies are the Jul-21 29855050299 / 29854963573 runs.
• 🆕 today: R285 (get_server_args test drift, [HIGH], PR #31816), R286 (Dynamo tc_piecewise prefill graph-break, [HIGH], same PR #31816), R287 (reward-model _legacy_load_weights, [HIGH], PR #28671). All three lack a fix PR. R285+R286 share one root commit 745b2ca4.
• Recovered: release-docker-amd-nightly apt-get flake cleared on rerun (attempt 2 = success).
• Confidence: FACT/HIGH/MEDIUM/LOW/SPECULATION. Bot does NOT assign Priority — engineers decide from cluster size + persistence + fix availability.

---

Generated by amd-bot · last updated 2026-07-22 22:48 UTC

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-07-22 22:48 UTC)

---

CI Monitor — 2026-07-22

Repo: sgl-project/sglang

Monitored Workflows:

• nightly-test-amd.yml
• nightly-test-amd-rocm720.yml
• release-docker-amd-nightly.yml
• release-docker-amd-rocm720-nightly.yml
• nightly-amd-mi355x-disagg.yml
• amd-aiter-scout.yml
• pr-test-amd.yml
• pr-test-amd-rocm720.yml

Per-workflow failure reports are appended as comments below; the Daily Cross-Workflow Summary is rendered above this section.

[amd-bot]

pr-test-amd-rocm720.yml — 19 failure(s)

Run started (UTC): 2026-07-21 17:55 UTC
Last scanned (UTC): 2026-07-22 22:45 UTC
Commits: sglang 0bdd473

Summary

Counts: 19 failures · 10 clusters · 16 carrying over (≥3 days) · 3 new today

#	Cluster	Job	Test File	Test Function	Status	Hypothesis (confidence)
1	Diffusion 1-GPU suite 60-min timeout	mm-gen-1gpu (0)	python/sglang/multimodal_gen/test/run_suite.py	part0 longlive2_t2v	recurring, part0 never-passes	[MEDIUM] 60-min step budget exceeded (heavy video cases)
2	Diffusion 1-GPU suite 60-min timeout	mm-gen-1gpu (1)	.../test/server/test_server_1_gpu.py	test_diffusion_generation[ltx_2_3_hq_pipeline]	recurring since Jul 8	[MEDIUM] 60-min step budget exceeded
3	Diffusion 1-GPU suite 60-min timeout	mm-gen-1gpu (3)	.../test/server/test_server_1_gpu.py	test_diffusion_generation	Never-passed (8-run window)	[LOW] ~18min hard-fail/crash (log unavailable)
4	Diffusion 1-GPU suite 60-min timeout	mm-gen-1gpu (0)	python/sglang/multimodal_gen/test/run_suite.py	part0 (suite killed)	recurring, part0 never-passes	[MEDIUM] 60-min step budget exceeded
5	Diffusion 1-GPU suite 60-min timeout	mm-gen-1gpu (1)	.../test/server/test_server_1_gpu.py	part1 (suite killed)	recurring since Jul 8	[MEDIUM] 60-min step budget exceeded
6	Diffusion runner-lost / no log	mm-gen-2gpu (0)	.../test/server/test_server_2_gpu.py	test_diffusion_generation (case unknown)	Never-passed, 10+ runs	[MEDIUM] runner lost mid-step-9; [LOW] FLUX.2 FP8 HIPBLAS
7	Diffusion runner-lost / no log	mm-gen-2gpu (0)	.../test/server/test_server_2_gpu.py	(log purged, BlobNotFound)	Never-passed, 10+ runs	[MEDIUM] runner lost / no log
8	Diffusion runner-lost / no log	mm-gen-1gpu (3)	.../single_test_file/test_update_weights_from_disk.py	N/A (runner died)	Never-passed since Jul 6	[LOW] runner death; #31925 skips this test
9	Triton throughput below AMD gate	stage-b-1gpu-large (0)	test/registered/perf/test_bench_serving_1gpu_part1.py	test_offline_throughput_with_triton_attention_backend	Never-passed since ~Jul 3	[FACT] 3196 < 3500 gate (perf miss)
10	Triton throughput below AMD gate	stage-b-1gpu-large (0)	test/registered/perf/test_bench_serving_1gpu_part1.py	test_offline_throughput_with_triton_attention_backend	Never-passed since ~Jul 3	[FACT] 3168/3177 < 3500 gate
11	msgpack ModuleNotFoundError	mm-gen-unit-test	test/unit/test_pi05_action_api.py	test_msgpack_roundtrip_preserves_string_keys_and_numpy_payloads	Never-passed since Jul 17	[FACT] msgpack missing in ROCm image
12	msgpack ModuleNotFoundError	mm-gen-unit-test	python/sglang/multimodal_gen/test/unit/test_pi05_action_api.py	test_msgpack_roundtrip_preserves_string_keys_and_numpy_payloads	Never-passed since Jul 17	[FACT] msgpack missing in ROCm image
13	Kimi-MXFP4 unbalanced load → -9	stage-c-8gpu-mi35x (0)	test/registered/amd/test_kimi_k25_mxfp4_bcg_mi35x.py	test_bcg_gsm8k	recurring ~5/6 since Jul 17	[HIGH] infra I/O contention (imbalanced weight load)
14	Kimi-MXFP4 unbalanced load → -9	stage-c-8gpu-mi35x (1)	test/registered/amd/test_kimi_k25_mxfp4_bcg_mi35x.py	test_bcg_gsm8k	last passed Jul 19	[no candidate ≥ MEDIUM] (HIP loader hang)
15	PP-disagg decode Gloo deadlock	stage-b-8gpu-mi35x-disagg	test/registered/amd/disaggregation/test_disaggregation_pp.py	TestDisaggregationDecodePPAccuracy.test_gsm8k (l.293)	4/5 recurring/flaky	[MEDIUM] PP P2P send/recv deadlock
16	Mooncake KV-transfer 30s timeout	stage-b-8gpu-mi35x-disagg	test/registered/amd/disaggregation/test_disaggregation_pp.py	TestDisaggregationPrefillPPDynamicChunkAccuracy.test_gsm8k	recurring flaky	[MEDIUM] Mooncake KV sync 30s timeout
17	GSM8K accuracy below MXFP4 gate	stage-b-1gpu-small-mi35x	test/registered/quant/test_quark_mxfp4.py	TestFP8ToMXFP4MOETP1.test_gsm8k	🆕 new (last passed Jul 21, 0.928)	[FACT] 0.918/0.91 < 0.92 (marginal)
18	communicator.get_server_args removed	stage-c-8gpu-rocm720 (3)	test/registered/ops/test_aiter_allreduce_fusion_amd.py	TestAiterAllreduceFusionGate.test_dense_tp_fuses	🆕 new (Jul 22)	[HIGH] 745b2ca4 (#31816) removed symbol
19	Dynamo graph-break in prefill capture	stage-c-8gpu-mi35x (2)	test/registered/amd/test_deepseek_r1_mxfp4_8gpu.py	TestDeepseekR1MXFP4.setUpClass	🆕 new (Jul 22)	[HIGH] 745b2ca4 (#31816) get_parallel() untraceable

Cluster: Diffusion 1-GPU suite 60-min timeout (rows 1-5)

• Affected: run_suite.py part0 (rows 1, 4) + test_server_1_gpu.py part1/part3 (rows 2, 3, 5), across both runs. Shard 2 consistently passes; shards 0/1/3 fail every run.
• Shared facts: LPT-partitioned heavy T2V/T2I cases; sisters terminate with verbatim The action 'Run diffusion server tests (1-GPU)' has timed out after 60 minutes (last cases longlive2_t2v / ltx_2_3_hq_pipeline). Row 3 is an outlier (~18 min, log purged) — likely a hard case-crash, not the timeout.
• Top hypothesis: [MEDIUM] partition runtime > 60-min step budget on MI325 ROCm720 — a CI-capacity issue, not a product regression. Disconfirming: row 3's 18-min death.
• In-flight fix: ❌ none.
• Suggested triage: pre-seed Lightricks/LTX-2.3 + LongLive-2.0 into the HF cache; rebalance partition estimates or raise timeout-minutes; fix shard-1/3 log-upload gap that blocks diagnosis.

Cluster: Diffusion runner-lost / no log persisted (rows 6-8)

• Affected: 2-GPU shard 0 (rows 6, 7) + 1-GPU shard 3 (row 8). Rows 6 & 8 occurred simultaneously in the same run (steps 9/10/20 = null, no "Complete job"), pointing to a runner/infra event.
• Shared facts: log blobs BlobNotFound, 0 artifacts; jobs ran ~85-110 min (under the 150-min step budget) before dying. Jul 11-16 logs (still intact) show the underlying 2-GPU error: RuntimeError: CUDA error: HIPBLAS_STATUS_NOT_SUPPORTED on flux2_modelopt_fp8_tp2_t2i.
• Top hypothesis: [MEDIUM] runner hard-terminated mid-step; [LOW] underlying FLUX.2 FP8 GEMM unsupported on gfx942. Disconfirming: no surviving log to confirm HIPBLAS occurred on the no-log days.
• In-flight fix: ✅ #28889 (FLUX.2 FP8 GEMM, gfx942) for the underlying 2-GPU failure; ✅ #31925 skips test_update_weights_from_disk on ROCm (row 8); ❌ none for the runner-loss/log-upload aspect.
• Suggested triage: inspect MI325 runner dmesg/GPU health for the death windows; chase #28889/#31925; fix the systematic log-upload gap.

Cluster: Triton throughput below AMD gate (rows 9-10)

• Affected: stage-b-test-1-gpu-large shard 0 in both runs, same test function.
• Shared facts: measured ~3040-3200 tok/s, gate is 3500; tightly clustered for ~3 weeks, no green run in window — a perf-threshold miss, not a crash.
• Top hypothesis: [FACT] throughput below gate. No code-level regression identified. Disconfirming: stable numbers argue against a recent regression → likely mis-set gate.
• In-flight fix: ❌ none found (searched bench_serving+throughput, triton+threshold).
• Suggested triage: confirm whether 3500 gate was ever met on this runner; if not, adjust the threshold or investigate the Triton-backend perf gap.

Cluster: msgpack ModuleNotFoundError (rows 11-12)

• Affected: multimodal-gen-unit-test in both runs, same test.
• Shared facts: ModuleNotFoundError: No module named 'msgpack' — optional dep missing from the ROCm 7.2 diffusion image; env/packaging gap, not a code regression.
• Top hypothesis: [FACT] msgpack not installed. Disconfirming: (none).
• In-flight fix: ✅ #31899 "[AMD] Add msgpack to ROCm diffusion deps" (open since Jul 21, exact fix) — chase reviewers.
• Suggested triage: merge #31899.

Cluster: Kimi-MXFP4 unbalanced weight-load → server -9 (rows 13-14)

• Affected: test_kimi_k25_mxfp4_bcg_mi35x.py::test_bcg_gsm8k across mi35x-8gpu shards 0/1 (matrix re-sharded between runs).
• Shared facts: TP1/TP2 load in ~380-430s but TP0 ~1300s+ / TP3 lags → monitored_barrier (480s) ValueError → SIGKILL (-9); no OOM (~145GB free); local docker-registry pull fallback shows runner I/O variability.
• Top hypothesis: [HIGH] infra I/O contention driving imbalanced load beyond the barrier window. Disconfirming: the row-14 analysis found a HIP-level loader mutex hang with no MEDIUM+ commit — bisect d4801be4..5ab3d90b8 still open.
• In-flight fix: ❌ root-cause none; mitigation candidates #15637/#9305 make the 480s barrier tunable (verify freshness — old PRs).
• Suggested triage: re-run with warm weight cache to test the I/O-skew hypothesis; if runner can't load 133GB/rank in budget, move to nightly.

Cluster: PP-disaggregation decode Gloo deadlock (row 15)

• Affected: test_disaggregation_pp.py::TestDisaggregationDecodePPAccuracy.test_gsm8k (l.293).
• Shared facts: py-spy shows PP0 blocked in SendWork::wait while PP1 blocked in RecvWork::wait inside event_loop_pp_disagg_decode; 300s watchdog kills the ranks; aiter unchanged.
• Top hypothesis: [MEDIUM] PP P2P send/recv count/order mismatch (silent deadlock). Disconfirming: no explicit exception; passed Jul 18, so intermittent not always-on.
• In-flight fix: ❌ none.
• Suggested triage: repro on 8-GPU MI35x, py-spy all scheduler_PP* ranks, instrument point_to_point_pyobj sequence counts.

Cluster: Mooncake KV-transfer 30s timeout (row 16)

• Affected: test_disaggregation_pp.py::TestDisaggregationPrefillPPDynamicChunkAccuracy.test_gsm8k.
• Shared facts: KVTransferError: Failed to get kvcache from prefill instance → KeyError: 'answer' → retries exhausted; the specific failing subtest varies run-to-run, root symptom (Mooncake 30s sync timeout) constant.
• Top hypothesis: [MEDIUM] Mooncake KV receiver stuck in Transferring past the 30s sync timeout.
• In-flight fix: ⚠️ candidates #31800, #29978, #31993 — none confirmed to fix this exact flake.
• Suggested triage: assess whether #31800 resolves the receiver-stuck path; collect flake rate.

Cluster: GSM8K accuracy below MXFP4 gate (row 17)

• Affected: test_quark_mxfp4.py::TestFP8ToMXFP4MOETP1.test_gsm8k — 🆕 relanded Jul 21 (#28291), passed Jul 21 (0.928), failed Jul 22 (0.918→0.91).
• Top hypothesis: [FACT] accuracy marginally below the 0.92 cutoff on MXFP4 online-quant MoE — likely run-to-run variance at a tight threshold. Disconfirming: only 2 data points since reland.
• In-flight fix: ❌ none.
• Suggested triage: gather more runs to measure variance; consider a small threshold margin or seed control.

Cluster: communicator.get_server_args removed (row 18) — 🆕

• Affected: test_aiter_allreduce_fusion_amd.py::test_dense_tp_fuses.
• Shared facts: AttributeError: module 'sglang.srt.layers.communicator' does not have the attribute 'get_server_args' (a mock.patch target); passed Jul 21, broke Jul 22.
• Top hypothesis: [HIGH] 745b2ca4 (#31816) removed the symbol.
• In-flight fix: ✅ already merged #32100 (f5dcbe8f, reverts #31813–#31817, restores get_server_args) — this run predates it by ~10 min. Should self-resolve next run.
• Suggested triage: confirm green on the next scheduled run post-#32100.

Cluster: Dynamo graph-break in prefill CUDA graph capture (row 19) — 🆕

• Affected: test_deepseek_r1_mxfp4_8gpu.py::setUpClass (server crash before any test).
• Shared facts: torch._dynamo.exc.Unsupported: __getattribute__ on ParallelContext._config during piecewise prefill capture → RuntimeError: Capture prefill CUDA graph failed → exit -9. Passed Jul 21, broke Jul 22.
• Top hypothesis: [HIGH] 745b2ca4 (#31816) changed is_dsa_enable_prefill_cp() to read via get_parallel().…, routing through the Dynamo-untraceable ParallelContext.__getattr__. All three evidence chains (diff, timing, error text naming ParallelContext+_config) converge. Disconfirming: (none).
• In-flight fix: ⚠️ possibly covered by merged #32100 (reverts #31813–#31817, which includes #31816) — verify it restores the traceable read; otherwise ❌ no dedicated fix.
• Suggested triage: confirm #32100 reverts the dsa/utils.py read; if not, revert #31816's is_dsa_enable_prefill_cp line or A/B with --cuda-graph-backend-prefill=disabled.

Suggested triage order

Rank	Triage step	Targets	Why this first
1	Confirm merged #32100 restores get_server_args + traceable get_parallel() read; re-run next scheduled	rows 18, 19	Fix already merged; both new-today regressions trace to #31816 — cheapest to verify
2	Merge #31899 (msgpack dep)	rows 11, 12	Exact one-line fix already open; clears a full cluster
3	Chase #28889 + #31925; investigate MI325 runner log-upload gap	rows 6-8	3-job cluster, partial fixes exist; log gap blocks all diffusion diagnosis
4	Rebalance 1-GPU diffusion partition estimates / pre-seed HF cache / raise timeout	rows 1-5	Largest cluster (5 jobs), CI-capacity not a product bug
5	Re-run Kimi bcg with warm cache; decide 480s barrier tunability (#15637/#9305)	rows 13, 14	Confirm I/O-skew vs loader-hang before code changes

---

Job	Failed Steps	Started
stage-c-test-large-8-gpu-amd-mi35x-rocm720 (linux-mi35x-gpu-8, 0)	Run test	2026-07-21T19:34
stage-b-test-large-8-gpu-mi35x-disaggregation-amd-rocm720 (linux-mi35x-gpu-8.fabric)	Run test_disaggregation	2026-07-21T17:55
multimodal-gen-test-1-gpu-amd-rocm720 (linux-mi325-1gpu-sglang, 3)	N/A	2026-07-22T00:29
multimodal-gen-test-2-gpu-amd-rocm720 (linux-mi325-2gpu-sglang, 0)	N/A	2026-07-21T23:56
multimodal-gen-unit-test-amd-rocm720	Run diffusion unit tests	2026-07-22T02:04
multimodal-gen-test-1-gpu-amd-rocm720 (linux-mi325-1gpu-sglang, 1)	N/A	2026-07-22T00:58
stage-b-test-1-gpu-large-amd-rocm720 (linux-mi325-1gpu-sglang, 0)	Run test	2026-07-22T02:20
multimodal-gen-test-1-gpu-amd-rocm720 (linux-mi325-1gpu-sglang, 0)	Run diffusion server tests (1-GPU)	2026-07-22T04:28
stage-b-test-1-gpu-small-amd-mi35x-rocm720 (linux-mi35x-gpu-1)	Run test	2026-07-22T17:53
multimodal-gen-unit-test-amd-rocm720	Run diffusion unit tests	2026-07-22T18:04
stage-b-test-large-8-gpu-mi35x-disaggregation-amd-rocm720 (linux-mi35x-gpu-8.fabric)	Run test_disaggregation	2026-07-22T18:00
stage-b-test-1-gpu-large-amd-rocm720 (linux-mi325-1gpu-sglang, 0)	Run test	2026-07-22T18:04
multimodal-gen-test-1-gpu-amd-rocm720 (linux-mi325-1gpu-sglang, 3)	N/A	2026-07-22T18:05
stage-c-test-large-8-gpu-amd-rocm720 (linux-mi325-8gpu-sglang, 3)	Run test	2026-07-22T18:42
multimodal-gen-test-1-gpu-amd-rocm720 (linux-mi325-1gpu-sglang, 1)	Run diffusion server tests (1-GPU)	2026-07-22T18:04
multimodal-gen-test-1-gpu-amd-rocm720 (linux-mi325-1gpu-sglang, 0)	Run diffusion server tests (1-GPU)	2026-07-22T18:04
stage-c-test-large-8-gpu-amd-mi35x-rocm720 (linux-mi35x-gpu-8, 2)	Run test	2026-07-22T21:30
stage-c-test-large-8-gpu-amd-mi35x-rocm720 (linux-mi35x-gpu-8, 1)	Run test	2026-07-22T21:26
multimodal-gen-test-2-gpu-amd-rocm720 (linux-mi325-2gpu-sglang, 0)	N/A	2026-07-22T20:30

Per-Job Analysis

stage-c-test-large-8-gpu-amd-mi35x-rocm720 (linux-mi35x-gpu-8, 0) — failed step(s): Run test

Commit Info

• sglang: 9a6c9608
• aiter: 9127c94a — v0.1.17.dev110+g9127c94a1 (source: dev image detected, rebuild skipped; matches Dockerfile default gfx950 block)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/test_kimi_k25_mxfp4_bcg_mi35x.py	test_bcg_gsm8k (variant=bcg)	Exception: Server process exited with code -9. Check server logs for errors.	Log

Failure Summary

The test_bcg_gsm8k test launches a amd/Kimi-K2.5-MXFP4 TP4 server and the server process died with exit code -9 (SIGKILL) during model weight loading. The four TP ranks loaded weights at wildly different speeds — TP2 finished in 389s and TP1 in 409s, but TP0 took 1322s and TP3 never reported completion — so the post-load monitored_barrier (480s) fired a ValueError about "other ranks that didn't finish loading", triggering a sigquit → kill_process_tree that the test observed as exit -9.

Regression Status

Known recurring failure (load-speed/infra-driven, intermittent). The identical signature (TP1/TP2 ~380–430s vs TP0 ~1300s+, TP3 lagging/never, barrier ValueError, server exit -9) reproduces on nearly every recent scheduled run; it passes only when the imbalanced load happens to complete within the barrier window (Jul 19, elapsed 2148s).

Recent history of test_kimi_k25_mxfp4_bcg_mi35x.py (test_bcg_gsm8k) in job stage-c-test-large-8-gpu-amd-mi35x-rocm720 (linux-mi35x-gpu-8, 0) (most recent comparable completed scheduled runs on main, event=schedule):

Date	Run	Test File Status	Failed Function	Error
Jul 21	29855003895	❌ Failed	test_bcg_gsm8k	Server exit -9 (unbalanced load barrier; TP0 1322s)
Jul 20	29766743590	❌ Failed	test_bcg_gsm8k	Server exit -9 (TP0 1315s)
Jul 19	29697501385	✅ Passed	—	elapsed 2148s
Jul 18	29654571916	❌ Failed	test_bcg_gsm8k	Server exit -9 (TP0 1388s, TP3 1844s)
Jul 17	29601526478	❌ Failed	test_bcg_gsm8k	Server exit -9 (TP0 1374s)
(Jul 18 run*)	29521557976	❌ Failed	test_bcg_gsm8k	Server exit -9 (TP0 1308s)
Jul 15	29438298954	⏭️ Cancelled	—	excluded (job cancelled)

*Run created 2026-07-16 but its job executed 2026-07-18 05:20 UTC. Window checked: Jul 15–21. Last known pass: Jul 19; ~5/6 completed runs failed with the same signature.

Failure Cluster

Kimi-K2.5-MXFP4 TP4 unbalanced weight-load exceeds monitored_barrier timeout → server SIGKILL (-9)

Facts (verified observations)

• Error verbatim: Exception: Server process exited with code -9. Check server logs for errors. (job log line 17662).
• Per-rank load times (this run): TP2 end elapsed=389.08s (log 8689), TP1 409.25s (log 8691), TP0 1322.38s (log 15160); no Load weight end line for TP3.
• Barrier failure verbatim: ValueError: TP rank 2 could finish the model loading, but there are other ranks that didn't finish loading. It is likely due to unexpected failures (e.g., OOM) or a slow node. raised from load_model_utils.py:270 (log 17644–17645), followed by Received sigquit from a child process and kill_process_tree (log 17647–17648).
• Barrier code: dist_barrier_after_load calls dist.monitored_barrier(..., timeout=UNBALANCED_MODEL_LOADING_TIMEOUT_S, wait_all_ranks=True); UNBALANCED_MODEL_LOADING_TIMEOUT_S = 480 (load_model_utils.py:40).
• No OOM: each rank reports mem usage=133.35 GB, avail mem≈145 GB after load (log 15160 etc.) — ~145GB still free, so this is a slow-load, not an out-of-memory, condition.
• rank2 ... reached monitoredBarrier, but received errors while waiting for send/recv from rank 0 (log 17623) — rank 0 is the lagging faulty rank.
• Same signature in 4 prior failing runs (Jul 17, 18, 20, and the Jul-16/18 run): TP1/TP2 ~380–433s vs TP0 ~1308–1388s (TP3 1844s on Jul 18). The one passing run (Jul 19) completed at elapsed 2148s — barely under budget.
• Runner note: Local docker registry pull failed; falling back to public registry (log 1572–1573) — evidence of I/O/network variability on this runner.

Hypothesised Causes (with confidence)

• [HIGH] Infrastructure / I/O contention causes severely imbalanced per-rank weight loading of a very large MoE model (Kimi-K2.5-MXFP4, 133GB/rank), and the ranks that finish first (TP1/TP2) time out at the 480s monitored_barrier before the slow ranks (TP0 ~1300s, TP3 lagging) finish.
  • Supporting: Identical per-rank timing skew across every failing run (TP1/TP2 ~4×faster than TP0); no OOM (145GB free); code path is identical across all four ranks so the 3× time difference is I/O-bound, not code-bound; the sole passing run (Jul 19) differed only in being fast enough to fit the window; local docker registry pull failure shows runner I/O variability.
  • Disconfirming: (none found) — the failure is not tied to any code path unique to the failing ranks; the barrier's own message lists OOM as an alternative, but memory telemetry rules OOM out.
  • How to verify: re-run with a warm on-disk weight cache (or enable_multithread_load tuned); temporarily raise UNBALANCED_MODEL_LOADING_TIMEOUT_S and confirm the barrier passes; monitor per-rank disk read throughput during load.
• [LOW] The UNBALANCED_MODEL_LOADING_TIMEOUT_S = 480 default is simply too tight for this model on this runner, i.e. a config/threshold issue rather than an infra fault.
  • Supporting: passing run completed at elapsed 2148s vs est 2100s — the whole test is at the edge of its budget; TP0 routinely needs ~1300s while the barrier allows only 480s after the first rank.
  • Disconfirming: even generous timeouts wouldn't fix a rank that never reports completion (TP3 here), pointing back to underlying I/O rather than pure threshold.
  • How to verify: bump the timeout on a branch and re-run; if it then passes consistently, the threshold is the proximate lever (but infra remains the root).

In-flight Fix Check

Search: is:pr is:open with keywords unbalanced loading / kimi mi35x.

• In-flight fix (mitigation candidate): ✅ [#15637](https://github.com/sgl-project/sglang/pull/15637) "Add unbalanced_model_loading_timeout_s Configuration" and ✅ [#9305](https://github.com/sgl-project/sglang/pull/9305) "Support setting UNBALANCED_MODEL_LOADING_TIMEOUT_S through environment variables" — both would make the 480s barrier tunable, mitigating the threshold aspect. Note: these are old, low-numbered PRs and may be stale; verify they still apply before relying on them.
• In-flight fix (root cause): ❌ none found — no open PR addresses the imbalanced load-speed of Kimi-K2.5-MXFP4 on MI35x directly.

Suggested Triage Steps

• Re-run the job and check whether a warm weight cache changes per-rank load times — if the skew disappears, this confirms the [HIGH] I/O-contention hypothesis (infra, not code).
• Temporarily raise UNBALANCED_MODEL_LOADING_TIMEOUT_S (via a branch, or by chasing #15637/#9305 to make it configurable) and re-run test_kimi_k25_mxfp4_bcg_mi35x.py on MI35x; confirm whether the barrier then passes.
• Instrument per-rank disk/HF read throughput during weight load to identify which device/mount is slow on the linux-mi35x-gpu-8 runner.
• If the runner cannot reliably load this 133GB/rank model within budget, consider moving test_kimi_k25_mxfp4_bcg_mi35x.py to nightly (as already done for test_kimi_k25_mxfp4_bcg siblings) or raising est_time and the barrier window together.

Status

• Persistence: Recurring/intermittent since at least 2026-07-17; ~5 of 6 completed scheduled runs in the Jul 15–21 window failed with this signature (one pass on 2026-07-19).
• Scope: Confined to job stage-c-test-large-8-gpu-amd-mi35x-rocm720 (linux-mi35x-gpu-8, 0) on pr-test-amd-rocm720.yml; single test file test_kimi_k25_mxfp4_bcg_mi35x.py.
• Blocked work: Blocks the AMD BCG (breakable CUDA-graph) accuracy signal for Kimi-K2.5-MXFP4 on MI35x; the giant-model load reliability, not the BCG code path, is what fails (the server never reaches capture/eval).
• In-flight fix: Root-cause fix ❌ none found; threshold-mitigation candidates #15637 / #9305 (verify freshness).

stage-b-test-large-8-gpu-mi35x-disaggregation-amd-rocm720 (linux-mi35x-gpu-8.fabric) — failed step(s): Run test_disaggregation

Commit Info

• sglang: 9a6c96083f — [Cookbook] Add Laguna-S-2.1 (#31918)
• aiter: 9127c94a18 (v0.1.17.dev110+g9127c94a1, source: Dockerfile default — AITER_COMMIT_OVERRIDE empty)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/disaggregation/test_disaggregation_pp.py	TestDisaggregationDecodePPAccuracy.test_gsm8k (line 293)	Exception: retry() exceed maximum number of retries. → underlying RuntimeError: Internal Server Error / aiohttp ClientConnectorError: Cannot connect to host 127.0.0.1:11200	Log

Other 3 test files in this job passed: test_disaggregation_basic.py ✅, test_mori_transfer_engine_e2e.py ✅, test_nixl_transfer_engine_e2e.py ✅. Job summary: 3/4 passed.

Failure Summary

The 8-GPU PP-disaggregation decode accuracy test (TestDisaggregationDecodePPAccuracy.test_gsm8k) failed after the decode server became unreachable on port 11200. The four pipeline-parallel decode scheduler processes hung: the scheduler watchdog fired at its 300s timeout, py-spy'd all ranks, and the servers exited (after the 60s CUDA-coredump wait), so the GSM8K client received 500 Internal Server Error / connection-refused and exhausted its retries. This is a server-side hang, not an accuracy mismatch.

Regression Status

Known recurring / flaky failure (test file has no stable green streak; different test_gsm8k methods fail on different days).

Recent history of test_disaggregation_pp.py in job stage-b-test-large-8-gpu-mi35x-disaggregation-amd-rocm720 (comparable completed scheduled runs, event=schedule, branch=main):

Date	Run	Job	Test File Status	Failed Function	Signature
Jul 21	run	job	❌ Failed	test_gsm8k (l.293, DecodePPAccuracy)	300s watchdog timeout, PP Gloo send/recv deadlock
Jul 20	run	job	❌ Failed	test_gsm8k (l.293, DecodePPAccuracy)	coredump-wait + RecvWork::wait (same PP-hang family)
Jul 19	run	job	❌ Failed	test_gsm8k (l.198)	Internal Server Error (different method)
Jul 18	run	job	✅ Passed	—	—
Jul 17	run	job	❌ Failed	test_gsm8k (l.104)	Internal Server Error (different method)
Jul 16	run	job	⏭️ Cancelled	—	job cancelled, excluded
Jul 15	run	job	⏭️ Cancelled	—	job cancelled, excluded

First observed failure of this test file in window: ≥ Jul 17. Last known pass: Jul 18. The specific line-293 PP-decode watchdog-hang variant is observed on Jul 20 and Jul 21.

Failure Cluster

PP-disaggregation decode scheduler hang / watchdog timeout (Gloo P2P send/recv deadlock in event_loop_pp_disagg_decode)

Facts (verified observations)

• Job step 10 "Run test_disaggregation" failed; 3/4 test files passed, only test_disaggregation_pp.py failed ({"file": ".../test_disaggregation_pp.py", "passed": false, "elapsed": 585}).
• Scheduler watchdog fired: [PP0 TP0] Scheduler watchdog timeout (self.watchdog_timeout=300, self.soft=True) at 18:15:00, escalating to self.soft=False at 18:15:26; followed by Waiting 60.0 seconds for CUDA coredumps before exiting.
• py-spy dumps (verbatim stacks):
  • scheduler_PP0_TP0 / PP0_TP1 stuck in _pp_commit_comm_work (scheduler_pp_mixin.py:924) → c10d::ProcessGroupGloo::SendWork::wait → gloo::...::UnboundBuffer::waitSend (blocked sending to next PP stage).
  • scheduler_PP1_TP0 stuck in _pp_recv_pyobj_from_prev_stage (scheduler_pp_mixin.py:973) → point_to_point_pyobj (utils/common.py:2251) → c10d::ProcessGroupGloo::RecvWork::wait → waitRecv (blocked receiving from prev PP stage).
  • scheduler_PP1_TP1 stuck in _pp_recv_pyobj_from_prev_stage (scheduler_pp_mixin.py:984).
• After schedulers exited, the decode server dropped connections: cascade of aiohttp ... ServerDisconnectedError: Server disconnected and 500 Internal Server Error, then client Cannot connect to host 127.0.0.1:11200.
• Client side: few_shot_gsm8k → cache_prefix → _assert_success raised RuntimeError: Internal Server Error, wrapped by test retry() → Exception: retry() exceed maximum number of retries. Ran 1 test in 574.791s / FAILED (errors=1).
• Source confirms deadlock geometry: _pp_commit_comm_work (scheduler_pp_mixin.py:923-925) blocks on p2p_work.work.wait(); the ring send/recv is _pp_send_pyobj_to_next_stage / _pp_recv_pyobj_from_prev_stage.
• aiter unchanged (Dockerfile default, no override); NVIDIA/other backends not implicated.

Hypothesised Causes (with confidence)

• [MEDIUM] Pipeline-parallel P2P communication deadlock in the disagg decode event loop: PP0 is blocked in SendWork::wait while PP1 is blocked in RecvWork::wait, i.e. the two stages are not agreeing on the number/order of point_to_point_pyobj transfers for a microbatch step, so both park until the 300s watchdog kills them.
  • Supporting: py-spy shows a matched send-blocked / recv-blocked pair across PP stages in event_loop_pp_disagg_decode; both hang on Gloo CPU-group P2P (world_group.cpu_group), not on GPU compute; no GPU fault/OOM in the log.
  • Disconfirming: no explicit assertion/exception is thrown by the ring code itself — the hang is silent until the watchdog, so a slow-but-not-deadlocked stall (e.g. one rank falling far behind under load) cannot be fully excluded from the log alone. The test also passed on Jul 18 with a nearby commit, so it is intermittent rather than a hard, always-on deadlock.
  • How to verify: re-run test_disaggregation_pp.py (TestDisaggregationDecodePPAccuracy) on an 8-GPU MI35x rocm720 container; if it hangs, py-spy all four scheduler_PP* ranks and diff how many point_to_point_pyobj calls each stage has issued to confirm a send/recv count mismatch.
• [LOW] Environmental / Gloo TCP transport stall on the fabric runner rather than a logic bug.
  • Supporting: the block is inside gloo::transport::tcp::UnboundBuffer::waitSend/waitRecv; disagg tests are network-heavy (RDMA/NIXL/mori).
  • Disconfirming: RDMA and NIXL/mori transfer-engine e2e tests in the same job passed, and health/model_info calls succeeded earlier — the control-plane network was healthy up to the hang.
  • How to verify: check runner NIC/Gloo counters during a repro; compare against a passing run on the same runner class.
• No hypothesis reaches HIGH. This is not a fresh single-commit regression: the test file was already failing on Jul 17, 19, 20 (with the line-293 PP-hang variant present Jul 20), so a bisect to today's commit is not warranted.

In-flight Fix Check

• In-flight fix: ❌ none found targeting the PP-disagg decode hang. Searched open PRs for disaggregation pp hang, pp disagg decode watchdog, test_disaggregation_pp (queries cited above). Nearest hits are unrelated (e.g. #29584 fixes MI325 pre-existing CI failures — different arch, mentions the file only tangentially; #29193 refactors scheduler event-loop locals but does not target PP P2P). None address the Gloo send/recv deadlock — needs new PR or investigation.

Suggested Triage Steps

• Reproduce on 8-GPU MI35x rocm720 (rocm/sgl-dev:v0.5.15.post1-rocm720-mi35x-20260721) running only TestDisaggregationDecodePPAccuracy.test_gsm8k; on hang, py-spy all scheduler_PP*_TP* ranks and confirm the send-blocked/recv-blocked pairing.
• Instrument point_to_point_pyobj / _pp_commit_comm_work (scheduler_pp_mixin.py:922-984) with per-microbatch send/recv sequence logging to locate the count/order mismatch between adjacent PP stages under decode load.
• Because the failure is intermittent (Jul 18 passed), collect 3-5 reruns to measure flake rate before attributing to any single commit; if a stall (not a true deadlock) is found, consider whether the 300s soft watchdog is too tight for this workload.
• As a stability stopgap while root-causing, consider quarantining/retrying TestDisaggregationDecodePPAccuracy.test_gsm8k on MI35x — maintainer decision, not a directive.

Status

• Persistence: test_disaggregation_pp.py failing 4 of last 5 comparable completed scheduled runs (Jul 17, 19, 20, 21 ❌; Jul 18 ✅). The line-293 PP-decode watchdog-hang variant seen Jul 20 and Jul 21.
• Scope: single job — stage-b-test-large-8-gpu-mi35x-disaggregation-amd-rocm720 in pr-test-amd-rocm720.yml; AMD MI35x only (8-GPU PP disagg). Other 3 disagg test files in the same job pass.
• Blocked work: PP-disaggregation decode accuracy CI signal on MI35x rocm720.
• In-flight fix: ❌ none found in open PRs.

multimodal-gen-test-1-gpu-amd-rocm720 (linux-mi325-1gpu-sglang, 3) — failed step(s): N/A

Commit Info

• sglang: 9a6c9608 ([Cookbook] Add Laguna-S-2.1 (#31918))
• aiter: N/A — the job log was never persisted (see below), so [CI-AITER-CHECK] markers cannot be read.

Failed Tests

Test File	Test Function	Error	Log
python/sglang/multimodal_gen/test/single_test_file/test_update_weights_from_disk.py (suspected — shard-3 standalone partition; unconfirmed for this run)	N/A (runner died mid-step)	No log persisted — runner killed during Run diffusion server tests (1-GPU); job concluded failure with step 9 stuck in_progress	Job

This is not a clean test assertion failure — it is a hard runner death. The GitHub log blob returns BlobNotFound (the redirect target 404s), and steps 9 (Run diffusion server tests), 10 (Upload diffusion failure artifacts, if: always()), and 20 (Post Checkout) all never reached a terminal state. That combination means the runner agent lost contact and never finalized the job — the log was never uploaded.

Failure Summary

The multimodal-gen-test-1-gpu-amd-rocm720 shard 3 job died during the diffusion server test step after ~24 min (00:29:02→00:53:03 UTC, well under the 60-min step timeout), leaving step 9 in_progress and no log persisted. Shard 3 is the standalone partition that runs test_update_weights_from_disk.py (confirmed from the last-available shard-3 log on 2026-07-08 and from issue #31924). The sibling shard 2 on the same rocm720 lane passed in this run, so the crash is specific to shard-3 content, not a blanket infra outage.

Regression Status

Known recurring / long-persistent failure (Never-passed in window). Shard 3 of this job has failed in all 15 comparable completed scheduled runs (event=schedule, branch=main), from 2026-07-06 through 2026-07-21. There are two distinct symptom phases:

Recent history of shard 3 (multimodal-gen-test-1-gpu-amd-rocm720 (…, 3)) — most recent 15 comparable completed scheduled runs:

Date	Run	Job	Test File Status	Signature
Jul 21	29766743590	shard 3	❌ Failed	no-log hard kill (step9 in_progress)
Jul 19	29697501385	shard 3	❌ Failed	no-log hard kill
Jul 18	29654571916	shard 3	❌ Failed	no-log hard kill
Jul 17	29601526478	shard 3	❌ Failed	no-log hard kill
Jul 16	29521557976	shard 3	❌ Failed	no-log hard kill
Jul 15	29438298954	shard 3	❌ Failed	no-log hard kill
Jul 14	29355401408	shard 3	❌ Failed	no-log (step9 unterminated)
Jul 13	29273187026	shard 3	❌ Failed	no-log
Jul 12	29202664092	shard 3	❌ Failed	no-log
Jul 11	29162225443	shard 3	❌ Failed	no-log
Jul 10	29113205616	shard 3	❌ Failed	no-log
Jul 9	29039527644	shard 3	❌ Failed	no-log
Jul 8	28964460204	shard 3	❌ Failed	log present — real test FAILED (InferenceMode-500)
Jul 7	28888292440	shard 3	❌ Failed	log present — real test FAILED
Jul 6	28813230096	shard 3	❌ Failed	log present — real test FAILED

• First observed failure for shard 3: at/before 2026-07-06 (oldest run in the 15-run window; no green shard-3 run found in the window).
• Last known passing: none in the queryable window → Never-passed across 2026-07-06 … 2026-07-21.
• Symptom transition: 2026-07-06→08 were clean test FAILEDs with logs; from 2026-07-09 onward the job dies hard with no log persisted.

Failure Cluster

ROCm 7.2 diffusion shard-3 runner death during update_weights_from_disk test (no log persisted)

Facts (verified observations)

• Job 88717083354 concluded failure; step 9 Run diffusion server tests (1-GPU) = in_progress, step 10 Upload diffusion failure artifacts = pending, step 20 Post Checkout = pending (verified via jobs API).
• Duration ~24 min (started 2026-07-22T00:29:02Z, completed 00:53:03Z), well below the 60-min step timeout (pr-test-amd-rocm720.yml:728).
• Log endpoint returns HTTP 302 → blob URL that returns BlobNotFound; no run artifacts uploaded (artifacts API total_count: 0).
• The identical no-log/step-9-unterminated signature holds for the previous 6 scheduled shard-3 jobs (Jul 15–21) and for the whole Jul 9–14 block (no logs).
• Sibling multimodal-gen-test-1-gpu-amd-rocm720 (…, 2) passed in this same run; shards 0/1 were still queued/in-progress at snapshot.
• Shard 3 is the standalone partition running test_update_weights_from_disk.py: confirmed by the 2026-07-08 shard-3 log (--partition-id 3 --total-partitions 4, running test_update_weights_from_disk.py for FLUX.2-klein-base-4B and Qwen-Image) and by issue #31924 ("shard 3 (standalone partition)").
• On the regular AMD lane, the same shard-3 test produces a clean HTTP 400 failure with a persisted log (e.g. job 88354737021, step9 completed failure, log HTTP 200) — i.e. the rocm720 lane behaves differently (hard crash vs clean fail).
• #31263 (fixes the InferenceMode-500 variant seen in the Jul 8 log) merged 2026-07-20.

Hypothesised Causes (with confidence)

• [MEDIUM] The shard-3 standalone test test_update_weights_from_disk.py triggers a hard ROCm 7.2 runtime/runner crash during the diffusion transformer weight-reload path, rather than a clean assertion.
  • Supporting: shard 3 deterministically runs this test (Jul 8 log + #31924); the same test fails deterministically on ROCm (#31924); sibling shard 2 passed, so the fault is shard-3-content-specific; the failure is 100% reproducible for 2+ weeks.
  • Disconfirming: no rocm720 log exists to prove the crash occurs inside this test; on the regular AMD lane the same test fails cleanly (HTTP 400 with a log), so the "hard runner death" mode is specific to rocm720 and not yet directly attributed to this test.
  • How to verify: after #31925 lands (skips this test on all HIP platforms), check whether shard 3 on the rocm720 lane starts producing a log / passing.
• [LOW] ROCm 7.2-specific GPU/driver fault (memory access fault, hang, or uncorrectable ECC) that severs the runner agent, independent of which test runs.
  • Supporting: no-log + unterminated steps is the classic signature of a runner losing contact; GPU-level faults on ROCm can hang the agent.
  • Disconfirming: the crash concentrates exclusively on shard 3 every single run while other shards on the same runner pool pass — a purely random GPU/driver fault would not be shard-selective.
  • How to verify: cross-reference the shard-3 runner host across runs for hardware specificity; inspect dmesg/GPU logs on the self-hosted MI325 runner if accessible.
• [LOW] A change merged around 2026-07-08→09 turned the shard-3 test failure from a clean FAILED (Jul 6–8, logs present) into a hard crash (Jul 9+, no logs).
  • Supporting: sharp, durable transition in signature exactly at the Jul 8→9 boundary.
  • Disconfirming: cannot enumerate the commit window reliably (shallow clone; git log for the path/date range returned nothing) and no log to correlate; could equally be a runner-image/driver change on the self-hosted fleet rather than an sglang commit.
  • How to verify: bisect scheduled rocm720 runs across the Jul 8→9 head-SHA boundary (85944006119 last-with-log vs 86193013126 first-no-log), and check for a runner-image update on the same dates.

In-flight Fix Check

• In-flight fix: ✅ #31925 (open since 2026-07-21) — "[AMD] Skip test_update_weights_from_disk on ROCm pending reload fix (#31924)". It gates on current_platform.is_hip() in python/sglang/multimodal_gen/test/server/gpu_cases.py, removing test_update_weights_from_disk.py from STANDALONE_FILES["1-gpu"] — this applies to the rocm720 lane too. Chase reviewers; do not open a duplicate.
• Root-cause tracker: #31924 (open) — diffusers checkpoint layout not reconciled with sglang transformer params (shape mismatch → HTTP 400).
• Already-merged related fix: #31263 (merged 2026-07-20) fixed the separate InferenceMode-500 variant seen in the Jul 8 log.

Suggested Triage Steps

• Land / expedite #31925 and confirm shard 3 on pr-test-amd-rocm720.yml starts passing (or at least produces a log) in the next scheduled run — this is the cheapest way to test the MEDIUM hypothesis.
• If shard 3 still dies with no log after #31925, the failure is not this test → investigate the ROCm 7.2 runner/driver (LOW hypothesis 2): pull dmesg/GPU state from the MI325 host and check runner-image versions.
• To attribute the Jul 8→9 signature change: bisect the scheduled rocm720 head-SHA window between 85944006119 (last with log) and 86193013126 (first no-log), and check whether a self-hosted runner image update landed on those dates.
• Because logs are not persisting for this shard, consider adding a step-level timeout-minutes guard + forced log flush / rocm-smi dump before the test so future crashes leave diagnosable evidence.

Status

• Persistence: shard 3 failing in every completed scheduled run for ~16 days (since ≤2026-07-06); the current no-log hard-kill mode persistent since 2026-07-09.
• Scope: multimodal-gen-test-1-gpu-amd-rocm720 shard 3 on pr-test-amd-rocm720.yml. The related test also fails (cleanly) on the regular pr-test-amd shard 3. Sibling shards on the rocm720 lane are unaffected.
• Blocked work: diffusion update_weights_from_disk CI coverage on ROCm 7.2; no log/artifact evidence is being captured for this shard.
• In-flight fix: ✅ #31925 (open since 2026-07-21) — skips the suspect test on all HIP platforms; root cause tracked in #31924.

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-07-22 22:45 UTC)

[amd-bot]

nightly-test-amd-rocm720.yml — 15 failure(s)

Run started (UTC): 2026-07-21 17:55 UTC
Last scanned (UTC): 2026-07-22 22:02 UTC
Commits: sglang 0bdd473

Summary

Counts: 15 failures · 10 clusters (+1 unanalyzed job) · 12 carrying over (≥3 days) · 2 new today · 1 analysis-incomplete

#	Cluster	Job	Test File	Test Function	Status	Hypothesis (confidence)
1	InternVL2_5 processor crash	nightly-accuracy-2-gpu-vlm-rocm720	test/registered/amd/accuracy/mi30x/test_vlms_mmmu_eval_amd.py	test_mmmu_vlm_models[InternVL2_5-2B]	8d (since Jul 14), 7/7	[HIGH] AutoProcessor→bare CLIPImageProcessor; get_tokenizer_from_processor derefs .tokenizer (common.py:529)
2	InternVL2_5 processor crash	nightly-accuracy-2-gpu-vlm-rocm720	test_vlms_mmmu_eval_amd.py	test_mmmu_vlm_models[InternVL2_5-2B]	same, Jul 22	[HIGH] same
3	Qwen3-30B-A3B MoE mem fault	nightly-accuracy-8-gpu-mi35x-rocm720	test/registered/amd/accuracy/mi35x/test_qwen3_moe_eval_mi35x.py	test_qwen3_moe_accuracy	5d (since Jul 17), never-passed	[MEDIUM] OOB in bf16 MoE prefill on gfx950/ROCm7.2
4	Qwen3-30B-A3B MoE mem fault	nightly-accuracy-8-gpu-mi35x-rocm720	test_qwen3_moe_eval_mi35x.py	test_qwen3_moe_accuracy	same, Jul 22	[MEDIUM] same
5	GLM-5.1-MXFP4 runner death	nightly-2-gpu-mi35x-glm51-mxfp4-rocm720	test/registered/amd/accuracy/mi35x/test_glm51_mxfp4_tp2_gsm8k_mi35x.py	TestGLM51MXFP4TP2GSM8KMI35x	runner-death since Jul 17 (6 runs); never-passed	[MEDIUM] AMD MI35x runner/infra death (no log, BlobNotFound)
6	GLM-5.1-MXFP4 runner death	nightly-2-gpu-mi35x-glm51-mxfp4-rocm720	test_glm51_mxfp4_tp2_gsm8k_mi35x.py	TestGLM51MXFP4TP2GSM8KMI35x	same, Jul 22	[MEDIUM] "runner lost communication" (infra)
7	mamba transfer_kv_mamba NameError	nightly-test-1-gpu-kernel-rocm720	test/registered/jit/test_transfer_mamba.py	test_mamba_kernel_* (12)	since Jul 18, never-passed on ROCm	[HIGH] import gated if _is_cuda: (memory_pool_host.py:41)
8	mamba transfer_kv_mamba NameError	nightly-test-1-gpu-kernel-rocm720	test_transfer_mamba.py	test_mamba_kernel_* (12)	same, Jul 22	[HIGH] same
9	EAGLE/MTP accuracy shortfall	nightly-8-gpu-mi35x-deepseek-r1-mxfp4-tp4-rocm720	test_deepseek_r1_mxfp4_tp4_mtp_mi35x.py	test_gsm8k	never-passed since Jul 3 (~0.85 vs 0.944)	[MEDIUM] EAGLE HIP verify degraded / threshold mis-set
10	Mixtral-8x7B accuracy shortfall	nightly-accuracy-2-gpu-rocm720	test/registered/amd/accuracy/mi30x/test_gsm8k_eval_amd.py	test_gsm8k_all_models	predates window (6/6)	[HIGH] Mixtral GSM8K 0.316 << 0.69 (fact); root cause [LOW]
11	DeepSeek-V3.2 basic accuracy	nightly-accuracy-8-gpu-mi35x-deepseek-v32-rocm720	test_deepseek_v32_eval_mi35x.py	test_deepseek_v32_accuracy	never-passed, 5/5 (Jul 17–21)	[no candidate ≥ MEDIUM] (0.88 vs 0.93; threshold?)
12	HIP CUDA-graph capture error	nightly-8-gpu-mi35x-qwen3-235b-mxfp4-rocm720	test_qwen3_instruct_mxfp4.py	TestQwen3Instruct2507MXFP4.setUpClass	5d (since Jul 17), never-passed	[MEDIUM] capture-safety violation (hipErrorCapturedEvent on RCCL watchdog)
13	DeepSeek-V4-Flash TBO accuracy	nightly-8-gpu-mi35x-deepseek-v4-flash-rocm720	test_deepseek_v4_flash_fp8_tbo.py	test_gsm8k_tbo	new Jul 21 (last pass Jul 19); borderline	[MEDIUM] 0.91 threshold inside noise band (0.905–0.924)
14	VLM MMMU 3600s timeout	nightly-4-gpu-rocm720	test/registered/vlm/test_encoder_dp.py	test_vlm_mmmu_benchmark	new Jul 22; flaky (model-draw)	[MEDIUM] slow GLM-4.1V-9B-Thinking draw >3600s
15	(unanalyzed)	nightly-accuracy-2-gpu-rocm720	N/A	N/A	analysis incomplete	likely recurring Mixtral (row 10) — unverified

Cluster: InternVL2_5 processor crash — CLIPImageProcessor has no .tokenizer

• Affected: rows 1–2 (nightly-accuracy-2-gpu-vlm-rocm720 × both runs), test_mmmu_vlm_models.
• Shared facts: identical stack tokenizer_manager.py:3211 → processor.py:271 → common.py:529 return processor.tokenizer; abort is errors=1 → exit 255; popen_launch_server sits outside per-model try/except, so it masks 7 downstream models + 3 borderline Qwen sub-thresholds. 7/7 runs Jul 14–22.
• Top hypothesis: [HIGH] AutoProcessor.from_pretrained returns bare CLIPImageProcessor; InternVL3_5 special-case at processor.py:193 doesn't cover InternVL2_5. Disconfirming: can't confirm from log why AutoProcessor returns CLIP (transformers-support sub-question unverified).
• In-flight fix: ❌ none found.
• Suggested triage: repro AutoProcessor.from_pretrained("OpenGVLab/InternVL2_5-2B") type in the ROCm image; either add to AMD_FAILING_VLM_MODELS or add an InternVL2_5 load path; move popen_launch_server inside the loop try: so one model can't abort the suite.

Cluster: Qwen3-30B-A3B (bf16 MoE) GPU memory access fault

• Affected: rows 3–4 (nightly-accuracy-8-gpu-mi35x-rocm720 × both runs), test_qwen3_moe_accuracy. Other 3 accuracy files pass.
• Shared facts: Memory access fault by GPU ... address ...009000 (identical suffix across nodes/runs) after clean warmup; crashing frame process_batch_result_prefill → torch/cuda/streams.py:231 synchronize (surfaces prior async fault); GPU coredump unwritable, kernel name not captured; 5 consecutive days, different sglang SHAs, same aiter dev image.
• Top hypothesis: [MEDIUM] platform/kernel OOB in unquantized bf16 MoE prefill on gfx950+ROCm7.2. Disconfirming: Reason: Unknown, no kernel named; can't separate sglang vs aiter vs runtime.
• In-flight fix: ⚠️ #31405 same model but TP2/ROCm7.0/FP8 (coverage unconfirmed); ❌ exact bf16/TP8 match.
• Suggested triage: repro with AMD_SERIALIZE_KERNEL=3/writable coredump to name the faulting kernel; A/B SGLANG_USE_AITER=0.

Cluster: GLM-5.1-MXFP4 MI35x runner death (server never reaches GSM8K)

• Affected: rows 5–6 (nightly-2-gpu-mi35x-glm51-mxfp4-rocm720 × both runs), test_glm51_mxfp4_tp2_gsm8k_mi35x.py.
• Shared facts: BlobNotFound log + step stuck in_progress + ~22–25 min duration ≪ 120-min timeout; explicit "self-hosted runner lost communication" annotation (Jul 22). Runner-death signature in 6 runs since Jul 17; on Jul 21 also hit a sibling job on a different host. Multi-cause cluster historically (GPU Hang exit-134 Jul 13/14; gfx950 JIT cooperative_groups.h Jul 16).
• Top hypothesis: [MEDIUM] AMD self-hosted runner instability (host/GPU/agent death). Disconfirming: only 2/46 jobs affected per run → not fleet-wide, so a GPU-side crash with lost logs can't be excluded.
• In-flight fix: ✅ #31068 (MI35x hung-GPU pre-flight), ✅ #32003 (route scheduled jobs to MI300) — infra mitigations; #25575 for the older gfx950-JIT subpattern.
• Suggested triage: inspect MI35x host dmesg/rocm-smi at ~18:14 UTC Jul 22; re-run to capture a real log.

Cluster: mamba transfer_kv_mamba_lf_pf NameError on ROCm

• Affected: rows 7–8 (nightly-test-1-gpu-kernel-rocm720 × both runs), test_transfer_mamba.py (12/16 kernel-backend cases).
• Shared facts: NameError at memory_pool_host.py:407 for io_backend="kernel"; import guarded if _is_cuda: (line 41) while sibling sgl_kernel block uses if _is_cuda or _is_hip:; test skip torch.cuda.is_available() returns True on ROCm. 100% reproducible, AMD-only.
• Top hypothesis: [HIGH] CUDA-only import guard leaves the symbol unbound on HIP. Disconfirming: (none found).
• In-flight fix: ✅ #31741 (open since Jul 20, mergeable_state=blocked) edits exactly these files — chase reviewers, do NOT duplicate.
• Suggested triage: unblock/merge #31741; watch next nightly for the ROCm .cuh JIT build.

Cluster: EAGLE/MTP GSM8K accuracy shortfall (DeepSeek-R1-MXFP4)

• Affected: row 9. test_gsm8k MTP step ~0.85 vs 0.944; plain-decode sister step passes 0.945.
• Top hypothesis: [MEDIUM] EAGLE HIP verify path degraded (stable 9-pt gap, healthy plain decode) or [MEDIUM] threshold copied from plain-decode at test creation. Disconfirming: never a green run → can't prove a green→red regression.
• In-flight fix: ✅ #31214 (EAGLE HIP verify greedy-sampling fix, strongest), ✅ #30105 (secondary).
• Suggested triage: cherry-pick #31214, re-run; confirm intended EAGLE accuracy with #29290 author.

Cluster: Mixtral-8x7B GSM8K accuracy shortfall (2-GPU eval)

• Affected: row 10 (row 15 is the Jul 22 instance, analysis incomplete but same job). test_gsm8k_all_models; Mixtral 0.316 << 0.69.
• Top hypothesis: [HIGH] job red because of Mixtral assertion (fact). Root-cause candidates all [LOW]: GSM8K answer-extraction artifact (#29933 draft) or AMD MoE numerics. Also Qwen3-4B missing from MODEL_SCORE_THRESHOLDS (cosmetic + 3× wasted retries).
• In-flight fix: ⚠️ draft #29933 (unverified); ❌ for Qwen3-4B threshold.
• Suggested triage: pull the Mixtral eval JSON artifact (job log truncates on aiter GEMM spam); re-score with #29933 extractor; A/B SGLANG_USE_AITER.

Cluster: DeepSeek-V3.2 basic GSM8K accuracy (DSA/tilelang)

• Affected: row 11. 0.855–0.895 vs 0.93, 5/5, MTP sister passes.
• Top hypothesis: [LOW] threshold miscalibration or pre-existing accuracy gap — no candidate ≥ MEDIUM (failure predates queryable window; no green baseline).
• In-flight fix: ❌ none. Triage: pull pre-Jul-17 accuracy history to see if 0.93 was ever met; else re-baseline with model owners.

Cluster: Qwen3-235B MXFP4 HIP CUDA-graph capture error

• Affected: row 12. setUpClass crash at Capturing batches 0/52, hipErrorCapturedEvent on RCCL watchdog, exit -6.
• Top hypothesis: [MEDIUM] capture-safety violation (illegal HIP event op on capturing stream). Disconfirming: no green baseline; exact op not identified (async HIP error).
• In-flight fix: 🟡 #27650 (aiter fused allreduce+RMSNorm capture-safety) related but scoped to different flags/model. Triage: repro with AMD_SERIALIZE_KERNEL=3; A/B --disable-cuda-graph.

Cluster: DeepSeek-V4-Flash FP8 DP+TBO accuracy (unified_kv_triton)

• Affected: row 13. test_gsm8k_tbo 0.827 then 0.9098 (both < 0.91); triton step passed.
• Top hypothesis: [MEDIUM] 0.91 threshold sits inside the DP+TBO noise band. Disconfirming: the 0.827 deep drop (with invalid=0.0038) exceeds pure noise. Triage: run N×10 per flashmla backend to characterize distribution; if a real shift, bisect Jul 19→21.
• In-flight fix: ❌ none.

Cluster: VLM MMMU 3600s per-file timeout (random-model)

• Affected: row 14. test_vlm_mmmu_benchmark TIMEOUT on GLM-4.1V-9B-Thinking draw; passed Jul 21/18/17 (Jul 18/17 near ceiling ~2800s).
• Top hypothesis: [MEDIUM] reasoning-model draw emits long CoT past 3600s (server never crashed, still serving 200s at kill). [LOW] aiter torch-solution fallback inflates latency (present in passing runs too).
• In-flight fix: ❌ none. Triage: repro pinned to GLM-4.1V-9B-Thinking; consider raising timeout or per-model budget.

Suggested triage order

Rank	Triage step	Targets	Why this first
1	Unblock/merge #31741, re-run kernel nightly	rows 7–8	HIGH-confidence root cause, fix already exists (blocked)
2	Cherry-pick #31214, re-run MTP test	row 9	✅ in-flight fix targets exact HIP verify path; longest-standing accuracy gap
3	Track #31068/#32003; inspect MI35x host logs	rows 5–6	infra death blocks 2 jobs + threatens fleet; mitigations pending
4	Fix/skip InternVL2_5 processor path; move launch inside try:	rows 1–2	HIGH root cause; unblocks 7 masked VLM models nightly
5	Serialized-kernel repro for Qwen3-30B-A3B mem fault	rows 3–4	5-day platform crash, no fix; needs kernel name to progress

---

Job	Failed Steps	Started
nightly-accuracy-8-gpu-mi35x-deepseek-v32-rocm720	Accuracy Test MI35x ROCm 7.2 (8-GPU DeepSeek-V3.2)	2026-07-21T19:27
nightly-2-gpu-mi35x-glm51-mxfp4-rocm720	N/A	2026-07-21T17:55
nightly-8-gpu-mi35x-qwen3-235b-mxfp4-rocm720	Accuracy Test + Performance Test MI35x ROCm 7.2 (8-GPU Qwen3-235B-MXFP4)	2026-07-21T18:40
nightly-accuracy-8-gpu-mi35x-rocm720	Accuracy Test MI35x ROCm 7.2 (8-GPU GPT-OSS)	2026-07-21T18:55
nightly-8-gpu-mi35x-deepseek-v4-flash-rocm720	Accuracy + Performance Test MI35x ROCm 7.2 (8-GPU DeepSeek-V4-Flash FP8 + FP4, unified_kv_triton)	2026-07-21T18:28
nightly-8-gpu-mi35x-deepseek-r1-mxfp4-tp4-rocm720	Accuracy Test MI35x ROCm 7.2 (DeepSeek-R1-MXFP4 TP4 MTP)	2026-07-21T19:56
nightly-test-1-gpu-kernel-rocm720	Nightly Kernel Benchmark ROCm 7.2 (1-GPU)	2026-07-22T00:22
nightly-accuracy-2-gpu-vlm-rocm720	Nightly Accuracy Test ROCm 7.2 (2-GPU VLM MMMU)	2026-07-21T23:53
nightly-accuracy-2-gpu-rocm720	N/A	2026-07-21T23:54
nightly-test-1-gpu-kernel-rocm720	Nightly Kernel Benchmark ROCm 7.2 (1-GPU)	2026-07-22T18:04
nightly-2-gpu-mi35x-glm51-mxfp4-rocm720	N/A	2026-07-22T17:52
nightly-4-gpu-rocm720	Nightly Test ROCm 7.2 (4-GPU)	2026-07-22T17:51
nightly-accuracy-2-gpu-rocm720	N/A — ⚠️ analysis failed	2026-07-22T18:51
nightly-accuracy-2-gpu-vlm-rocm720	Nightly Accuracy Test ROCm 7.2 (2-GPU VLM MMMU)	2026-07-22T19:59
nightly-accuracy-8-gpu-mi35x-rocm720	Accuracy Test MI35x ROCm 7.2 (8-GPU GPT-OSS)	2026-07-22T21:10

Per-Job Analysis

nightly-accuracy-8-gpu-mi35x-deepseek-v32-rocm720 — failed step(s): Accuracy Test MI35x ROCm 7.2 (8-GPU DeepSeek-V3.2)

Commit Info

• sglang: 9a6c96083
• aiter: 9127c94a1 — v0.1.17.dev110+g9127c94a1 (source: dev image preserved; [CI-AITER-CHECK] reports Dev/patched version detected → skipping rebuild, AITER_COMMIT_OVERRIDE empty)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi35x/test_deepseek_v32_eval_mi35x.py	test_deepseek_v32_accuracy (TestDeepSeekV32EvalMI35x)	AssertionError: Failed models: ['deepseek-ai/DeepSeek-V3.2 (basic)'] — GSM8K accuracy=0.880 < threshold=0.93	Log

Failure Summary

The 8-GPU MI35x DeepSeek-V3.2 (basic, TP=8) GSM8K accuracy benchmark scored 0.880 against the hard-coded pass threshold of 0.93 (test_deepseek_v32_eval_mi35x.py:71), raising AssertionError. The server started and served all 200 GSM8K questions successfully (all POST /generate returned 200 OK); this is a numerical accuracy shortfall, not a crash, hang, or infra error. The run uses attention_backend='dsa' with dsa_prefill_backend='tilelang' / dsa_decode_backend='tilelang' (DeepSeek Sparse Attention path).

Regression Status

Never-passed (no green run found in the queryable window). This job has failed with the same accuracy shortfall on the same test function in every completed scheduled run in the window; the only non-failing runs were cancelled (job never ran to completion). GSM8K accuracy hovers ~0.855–0.895, consistently 3.5–7.5 points below the 0.93 gate.

Recent history of test_deepseek_v32_eval_mi35x.py (test_deepseek_v32_accuracy) in job nightly-accuracy-8-gpu-mi35x-deepseek-v32-rocm720 (most recent comparable completed scheduled runs: main, event=schedule):

Date	Run	Test File Status	Accuracy	Error
Jul 21	29854963573	❌ Failed	0.880	< 0.93 threshold
Jul 20	29766667696	❌ Failed	0.875	< 0.93 threshold
Jul 19	29697452589	❌ Failed	0.895	< 0.93 threshold
Jul 18	29654521083	❌ Failed	0.855	< 0.93 threshold
Jul 17	29601455852	❌ Failed	0.885	< 0.93 threshold
Jul 16	29521474931	⚪ Cancelled	—	job cancelled, did not run

First observed failure in window: Jul 17 (earliest completed run queried). No passing run found — lookback covers the 6 most recent scheduled runs (Jul 16–21); runs before Jul 17 in this set were cancelled.

Failure Cluster

GSM8K accuracy below threshold — DeepSeek-V3.2 basic on MI35x (DSA/tilelang)

Facts (verified observations, no interpretation)

• Log line 15903: accuracy=0.880 threshold=0.93 ❌ FAIL; traceback at test_deepseek_v32_eval_mi35x.py:245 → AssertionError: Failed models: ['deepseek-ai/DeepSeek-V3.2 (basic)'].
• The test ran to completion: Ran 1 test in 1547.304s, all 200 GSM8K requests returned HTTP 200; failure is purely the accuracy assertion.
• Pass threshold is hard-coded at test_deepseek_v32_eval_mi35x.py:71 (accuracy_threshold=0.93) for the basic variant (TP=8, num_questions=200).
• Same test function failed in 5/5 completed comparable runs (Jul 17–21) with accuracy 0.855, 0.875, 0.880, 0.885, 0.895 (verified from each historical job log).
• Sibling jobs in the same runs passed: nightly-accuracy-8-gpu-mi35x-deepseek-v32-mtp-rocm720 and both -perf- jobs were success — only the non-MTP basic accuracy job fails.
• Config: attention_backend='dsa', dsa_prefill_backend='tilelang', dsa_decode_backend='tilelang', kv_cache_dtype='bfloat16', mem_fraction_static=0.85, random_seed=241664357 (fixed seed, so run-to-run variance comes from kernel nondeterminism, not sampling seed).
• aiter version unchanged across the failing window (dev image v0.1.17.dev110+g9127c94a1, Dockerfile default preserved).

Hypothesised Causes (with confidence, with disconfirming evidence)

• [LOW] The DSA/tilelang DeepSeek-V3.2 basic path on MI35x produces genuinely ~0.88 GSM8K accuracy, i.e. the 0.93 gate is set higher than this backend actually achieves (threshold miscalibration or a real accuracy gap vs the reference the threshold was tuned to).
  • Supporting: 5 consecutive completed runs cluster tightly at 0.855–0.895 with a fixed seed and unchanged aiter; the MTP variant (different decode path) passes, isolating the gap to the basic DSA/tilelang path. A ~5-point deficit over 200 questions (~10 questions) is beyond pure noise.
  • Disconfirming: no green baseline exists in the queryable window, so it cannot be confirmed the test ever passed at 0.93 on this hardware — the threshold may have been aspirational from the start, OR a real regression predates the window. Cannot distinguish "threshold too high" from "pre-existing accuracy regression" without an earlier green run or a bisect.
  • How to verify: pull the accuracy history from before Jul 17 (older run artifacts / step summaries) to find any run that scored ≥0.93; if none exists, the gate was never met on MI35x-basic. Cross-check the same model+backend GSM8K score on a non-AMD reference to see whether 0.93 is achievable.
• [SPECULATION] A DSA-path change (indexer/tilelang kernels) degraded DeepSeek-V3.2 basic accuracy on gfx950 at some point before the window.
  • Supporting: several open DSA PRs touch DeepSeek-V3.2 indexer/fusion kernels (e.g. #30342, #31324).
  • Disconfirming: no code-level evidence links any specific commit to the accuracy drop, and the failure predates the queryable window so no pass_sha..fail_sha boundary is available; the MTP variant on the same DSA stack passes.
  • How to verify: bisect scheduled-run accuracy history further back to find the first run below 0.93, then git log the DSA/model code in that window.

No hypothesis above LOW confidence; the failure predates the queryable window, so needs older-run accuracy history to establish whether 0.93 was ever met before a git bisect is meaningful.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for MI35x DeepSeek accuracy threshold, deepseek v32 gsm8k accuracy, DSA accuracy mi35x. Matches are generic DSA/DeepSeek feature PRs (e.g. #30342, #31324, #25559); none targets this accuracy shortfall or the MI35x threshold.

Suggested Triage Steps

• Retrieve GSM8K accuracy for this job from scheduled runs before Jul 17 (older run logs / step-summary artifacts) to determine whether the basic variant ever scored ≥0.93 on MI35x. If it never has, the 0.93 gate at test_deepseek_v32_eval_mi35x.py:71 is likely miscalibrated for this backend — decide with model owners whether to re-baseline the threshold or treat it as a real accuracy bug.
• If an earlier green run exists, bisect the scheduled-run accuracy sequence to find the first run below 0.93, then git log --oneline <pass_sha>..<fail_sha> filtered to DSA/tilelang indexer, DeepSeek-V3.2 model, and MoE/quant kernel paths.
• Re-run the benchmark with a larger GSM8K_NUM_QUESTIONS (env var, default 200) to tighten the accuracy estimate and confirm the ~0.88 figure is stable rather than sampling noise.
• Compare the same DeepSeek-V3.2 basic GSM8K score under a non-DSA / non-tilelang attention backend on the same hardware to isolate whether the deficit is DSA/tilelang-specific.

Status (factual, no priority assignment)

• Persistence: Failing in every completed scheduled run in the queryable window (Jul 17–21, 5/5); no green run found. Accuracy stable at 0.855–0.895 vs 0.93 gate.
• Scope: Single job nightly-accuracy-8-gpu-mi35x-deepseek-v32-rocm720 on nightly-test-amd-rocm720.yml. Sibling MTP-accuracy and both perf jobs pass, so blast radius is the DeepSeek-V3.2 basic accuracy signal on MI35x only.
• Blocked work: Blocks the MI35x DeepSeek-V3.2 (basic/DSA) accuracy gate on the ROCm 7.2 nightly.
• In-flight fix: ❌ none found in open PRs — needs new PR or a threshold re-baseline decision.

nightly-2-gpu-mi35x-glm51-mxfp4-rocm720 — failed step(s): N/A

Commit Info

• sglang: 9a6c96083 — [Cookbook] Add Laguna-S-2.1 (#31918) (a docs/cookbook commit, unrelated to the failing GLM code path)
• aiter: N/A — today's job log is unavailable (BlobNotFound), so [CI-AITER-CHECK] markers cannot be read.

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi35x/test_glm51_mxfp4_tp2_gsm8k_mi35x.py	TestGLM51MXFP4TP2GSM8KMI35x (server launch in setUpClass / test_gsm8k)	Unknown for today — no log; job killed mid-step	Job (log 404 / BlobNotFound)

This job runs exactly one test file (--suite nightly-amd-2-gpu-mi35x-glm51-mxfp4), so the failing test is that file by construction, even though today's per-line failure signature is unreadable.

Failure Summary

The job nightly-2-gpu-mi35x-glm51-mxfp4-rocm720 is marked failure, but its active step 6 (Accuracy Test … GLM-5.1-MXFP4 GSM8K) is still in_progress and step 12 (Post Checkout code) is pending, and the job log returns BlobNotFound (HTTP 404). The job ran only ~25 min (17:55:23 → 18:20:24 UTC), far below the 120-min step timeout (workflow line 293). This step-never-concluded + no-log-upload pattern is the signature of a runner termination / log-upload failure, not a normal test-completion failure. Note: the parent workflow run 29854963573 is still queued/in-flight overall; this individual job has completed.

Regression Status

Never-passed (no green run found in the queryable window) — and today's specific manifestation is an Infrastructure/log-upload failure (log unavailable).

Recent history of test_glm51_mxfp4_tp2_gsm8k_mi35x.py (TestGLM51MXFP4TP2GSM8KMI35x) in job nightly-2-gpu-mi35x-glm51-mxfp4-rocm720 (comparable completed scheduled runs, branch=main, event=schedule):

Date	Run	Test File Status	Failed Function	Error / signature
Jul 21	29854963573	❌ Failed	(unknown)	step stuck in_progress, no log (BlobNotFound); killed ~25 min in
Jul 19	29697452589	❌ Failed	(unknown)	step stuck in_progress, no log; ~25 min
Jul 19	29654521083	❌ Failed	(unknown)	step stuck in_progress, no log; ~23 min
Jul 17	29601455852	❌ Failed	(unknown)	step stuck in_progress, no log; ~26 min
Jul 16	29521474931	❌ Failed	setUpClass (server launch)	JIT build: topk_impl.cuh:27: 'cooperative_groups.h' file not found (gfx950) → Capture cuda graph failed → server exit -9
Jul 14	29355338821	❌ Failed	setUpClass (weight load)	HW Exception … reason :GPU Hang → Fatal Python error: Aborted → exit 134
Jul 13	29273096587	❌ Failed	setUpClass (weight download)	HW Exception … reason :GPU Hang → Fatal Python error: Aborted → exit 134

Lookback range: 2026-07-13 → 2026-07-21. No passing run exists in this window. Scheduled runs before Jul 13 were cancelled at the workflow level. First observed failure ≥ Jul 13; no last-known-passing date found.

Failure Cluster

"GLM-5.1-MXFP4 TP=2 MI35x nightly — server never reaches GSM8K (multi-cause: runner death / GPU Hang / gfx950 JIT build)". Today's occurrence specifically is the "runner death / log-upload failure" sub-pattern.

Facts (verified observations, no interpretation)

• Today's job 88716968459: conclusion=failure; step 6 in_progress, step 12 pending; log endpoint returns BlobNotFound HTTP 404 (verified twice).
• Duration ~25 min vs the 120-min step timeout-minutes — not a step timeout ([FACT]).
• Same no-log/stuck-in_progress signature in 3 prior runs (Jul 17, Jul 19 ×2) ([FACT], verified via job step states).
• Jul 16 logged failure (87843000141): cooperative_groups.h file not found compiling deepseek_v4/topk_impl.cuh for gfx950 → cuda graph capture failed → setUpClass raised, server exit -9 ([FACT], verbatim from log).
• Jul 13 & Jul 14 logged failures: HW Exception by GPU node-* reason :GPU Hang + Fatal Python error: Aborted, exit code 134, during weight download / do_load_weights in setUpClass ([FACT], verbatim from logs).
• HEAD commit 9a6c96083 is a cookbook/docs change; it does not touch the GLM/MXFP4/attention/kernel code path ([FACT]).

Hypothesised Causes (with confidence, with disconfirming evidence)

• [MEDIUM] Today's failure is a runner/infrastructure fault (runner terminated or logs never uploaded), not a code regression.
  • Supporting: BlobNotFound log, step stuck in_progress, Post Checkout code never ran, ~25 min duration ≪ 120-min timeout; identical signature in 3 prior runs; HEAD commit is unrelated docs.
  • Disconfirming: without the log it is possible the server crashed (GPU Hang, like Jul 13/14) and the log upload also failed as a side effect — the two are not mutually exclusive, so a GPU-side crash cannot be ruled out.
  • How to verify: re-run the job and capture the log; check the MI35x runner host for hardware/agent resets around 2026-07-21 18:00–18:20 UTC.
• [LOW] Underlying model-bringup instability on gfx950 (GPU Hang during load) is the real recurring blocker; the runner-death runs are the same crash with lost logs.
  • Supporting: 2 of 3 logged runs (Jul 13, Jul 14) show GPU Hang → abort during setUpClass; the test file has never passed.
  • Disconfirming: Jul 16 failed for a completely different reason (gfx950 JIT cooperative_groups.h build error), so the cluster is multi-cause and today's cause is not necessarily GPU Hang.
  • How to verify: reproduce sglang serve --model-path amd/GLM-5.1-MXFP4 --tp 2 … (args from the test) on an MI35x/gfx950 node and watch for GPU Hang / abort during weight load.
• [LOW] gfx950 JIT build breakage (cooperative_groups.h not found compiling deepseek_v4/topk_impl.cuh) contributes to some runs.
  • Supporting: verbatim Jul 16 log.
  • Disconfirming: only seen in 1 of the logged runs; not present in the GPU-Hang runs.
  • How to verify: attempt the dsv4 topk JIT compile in the ROCm 7.2 container standalone.

No hypothesis reaches HIGH; today's test-level root cause is unknowable without the log.

In-flight Fix Check

• gfx950 JIT cooperative_groups.h / dsv4 topk build error: In-flight fix: ✅ candidate #25575 "fix(dsv4 topk_v2): honor cluster contract in fused kernel SMALL/TRIVIAL branches" — touches the same deepseek_v4 topk kernel; verify it addresses the missing-header build path before assuming it fixes the Jul 16 signature.
• GPU Hang during GLM-5.1-MXFP4 bringup: In-flight fix: ❌ none found (searched glm51 mxfp4, GLM-5.1-MXFP4, GPU Hang open PRs — none target this crash).
• Runner-death / no-log today: In-flight fix: ❌ none found — this is an infra issue, not code.

Suggested Triage Steps

• Re-run job nightly-2-gpu-mi35x-glm51-mxfp4-rocm720 and confirm the log uploads; a captured log is required to classify today's failure at test level.
• Inspect the MI35x/gfx950 runner host health/dmesg for GPU resets or agent aborts around 2026-07-21 17:55–18:20 UTC.
• Locally reproduce the server launch (sglang serve --model-path amd/GLM-5.1-MXFP4 --tp 2 --kv-cache-dtype fp8_e4m3 --dsa-prefill-backend tilelang --dsa-decode-backend tilelang …) on gfx950 to see whether GPU Hang recurs during weight load.
• Check whether #25575 resolves the cooperative_groups.h gfx950 build path; if so, chase reviewers rather than opening a duplicate.
• If the model cannot be stabilized on gfx950, consider quarantining this nightly suite on MI35x until the bringup crash is separately fixed.

Status (factual, no priority assignment)

• Persistence: Failing every completed scheduled run since at least 2026-07-13; never passed in the Jul 13–21 window.
• Scope: This single job / one test file (test_glm51_mxfp4_tp2_gsm8k_mi35x.py) on the linux-mi35x-gpu-2 runner. Multi-cause cluster (runner death, GPU Hang, gfx950 JIT build).
• Today's data quality: Log unavailable (BlobNotFound); parent run 29854963573 still in-flight. Today's test-level root cause is undetermined.
• In-flight fix: gfx950 JIT build → candidate #25575; GPU Hang / runner-death → none found.

nightly-8-gpu-mi35x-qwen3-235b-mxfp4-rocm720 — failed step(s): Accuracy Test + Performance Test MI35x ROCm 7.2 (8-GPU Qwen3-235B-MXFP4)

Commit Info

• sglang: 9a6c96083
• aiter: 9127c94a1 — v0.1.17.dev110+g9127c94a1 (source: dev/patched CI image preserved; matches Dockerfile gfx950 default AITER_COMMIT=9127c94a, per [CI-AITER-CHECK] markers)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/test_qwen3_instruct_mxfp4.py	TestQwen3Instruct2507MXFP4.setUpClass	Server process exited with code 1 → RuntimeError: Rank 0 scheduler died during initialization (exit code: -6); underlying HIP error: operation not permitted on an event last recorded in a capturing stream (hipErrorCapturedEvent)	Log

Ran 0 tests — the failure is in setUpClass (server launch), so no test function executed. Server command: sglang serve --model-path amd/Qwen3-235B-A22B-Instruct-2507-mxfp4 --tp 4 --ep 2 --trust-remote-code --attention-backend aiter --device cuda.

Failure Summary

The Qwen3-235B-A22B-Instruct-2507 MXFP4 server crashed during CUDA/HIP graph capture while launching (Capturing batches (bs=512 avail_mem=24.84 GB): 0%| 0/52). The RCCL/NCCL process-group watchdog thread on rank 0 threw HIP error: operation not permitted on an event last recorded in a capturing stream (and, on other ranks, operation not permitted when stream is capturing), producing Fatal Python error: Aborted (SIGABRT). The engine reported Rank 0 scheduler died during initialization (exit code: -6), so setUpClass failed and 0 tests ran.

Regression Status

Never-passed (no green run found in queryable window). The specific CUDA-graph-capture symptom is recurring since 2026-07-17 (4 consecutive completed runs).

Recent history of test_qwen3_instruct_mxfp4.py (TestQwen3Instruct2507MXFP4.setUpClass) in job nightly-8-gpu-mi35x-qwen3-235b-mxfp4-rocm720 (comparable completed scheduled runs: branch=main, event=schedule):

Date	Run	Job	Test File Status	Failed Function	Error
Jul 21	29854963573	job	❌ Failed	setUpClass	HIP capture error (hipErrorCapturedEvent), exit -6
Jul 19	29697452589	job	❌ Failed	setUpClass	Same HIP capture error, exit -6
Jul 18	29654521083	job	❌ Failed	setUpClass	Same HIP capture error, exit -6
Jul 17	29601455852	job	❌ Failed	setUpClass	Same HIP capture error, exit -6
Jul 16	29521474931	job	⚪ Cancelled	—	Run cancelled (excluded)
Jul 14	29355338821	job	❌ Failed (diff phase)	—	Fatal Python error: Aborted, exit 134 — crashed ~5 min in, before reaching capture
Jul 13	29273096587	job	❌ Failed (diff phase)	—	Fatal Python error: Aborted during HF file_download, exit 134

The Job column is human reference. No passing run exists for this test file in the query window (Jul 13–21); mark Never-passed. Jul 13–14 aborted in an earlier phase (model load), so the CUDA-graph-capture cluster spans Jul 17–21.

Failure Cluster

GPU HIP error during CUDA graph capture (hipErrorCapturedEvent on RCCL watchdog thread)

Facts (verified observations)

• Log (rank 0): [E721 18:55:00 ProcessGroupNCCL.cpp:2057] ... Process group watchdog thread terminated with exception: HIP error: operation not permitted on an event last recorded in a capturing stream; other ranks: operation not permitted when stream is capturing.
• Immediately preceding line: Capturing batches (bs=512 avail_mem=24.84 GB): 0%| 0/52 [00:18<?, ?it/s] — crash occurs at the very start of CUDA graph capture.
• Just before capture, all 4 TP ranks logged: multimem all-gather disabled (module 'torch.distributed._symmetric_memory' has no attribute 'set_signal_pad_size').
• Exit code is -6 (SIGABRT), not -9; avail_mem=24.84 GB reported at capture start — this is not the OOM-killer path (the traceback's own -9 → OOM note does not apply here).
• Engine traceback: engine.py:682 wait_for_ready → engine.py:1385 _wait_for_scheduler_ready → RuntimeError: Rank 0 scheduler died during initialization (exit code: -6); test traceback: test_qwen3_instruct_mxfp4.py:44 setUpClass → popen_launch_server → Exception: Server process exited with code 1.
• Config: --tp 4 --ep 2 --attention-backend aiter, SGLANG_USE_AITER=1, MXFP4 MoE (torch.float4_e2m1fn_x2, QuantType.per_1x32) on gfx950 / ROCm 7.2; aiter falls back to heuristic FlyDSL MoE config (no tuned FlyDSL config ... using heuristic FlyDSL fallback) and bf16 GEMM using torch solution:0.
• Reproducibility: identical capture symptom in 4/4 completed runs Jul 17–21; test file has no green run in the query window.

Hypothesised Causes (with confidence)

• [MEDIUM] A CUDA-graph capture-safety violation: an op inside the captured region performs an illegal HIP event operation on the capturing stream. The error hipErrorCapturedEvent is raised by the RCCL/collective watchdog thread at the first captured batch, consistent with a fused aiter op or collective that records/queries an event during capture.
  • Supporting: crash lands exactly on Capturing batches (bs=512) 0/52; error is an event-vs-capturing-stream conflict; config uses the aiter attention backend + fused MXFP4 MoE + AMD collectives.
  • Disconfirming: no green baseline exists for this test in the window, so a regression window cannot be bounded; the exact op/commit is not identified from the log (async HIP errors "might be asynchronously reported at some other API call"); repo is shallow (89 commits) so local bisect is not possible here.
  • How to verify: relaunch the server with AMD_SERIALIZE_KERNEL=3 and CUDA-graph capture enabled to surface the offending kernel synchronously; then A/B by disabling piecewise/full CUDA graph (e.g. --disable-cuda-graph) to confirm capture is the trigger.
• [LOW] OOM during capture. Disconfirming: exit is -6 not -9, and avail_mem=24.84 GB at capture start — not consistent with OOM-killer.
• [LOW] aiter kernel change (9127c94a1 dev build). Disconfirming: aiter version is the Dockerfile default for gfx950 (not an override); no aiter-vs-baseline delta is available in this non-scout workflow to attribute it.

In-flight Fix Check

• In-flight fix: 🟡 [#27650](https://github.com/sgl-project/sglang/pull/27650) ("[AMD] Fix PCG capture fail with aiter fused all-reduce+RMSNorm", open since 2026-06-09) — related but likely not a direct match. It makes aiter's fused all-reduce+RMSNorm capture-safe, but is scoped to KimiK2.5-MXFP4 with --enable-aiter-allreduce-fusion + --enforce-piecewise-cuda-graph, flags this test does not set. Worth checking whether its capture-safety fix also covers this Qwen3-235B path. (Searched: mxfp4+capture, hipErrorCapturedEvent, capturing+stream+hip, qwen3+235b+mxfp4.)
• In-flight fix: ❌ none found that targets Qwen3-235B MXFP4 capture specifically.

Suggested Triage Steps

• Reproduce on an MI35x/gfx950 ROCm 7.2 container with the launch command above plus AMD_SERIALIZE_KERNEL=3 to synchronously pinpoint the kernel that raises hipErrorCapturedEvent during capture.
• A/B test with --disable-cuda-graph (and/or toggling piecewise CUDA graph) to confirm the crash only occurs during capture and localize to the aiter fused-op / collective path.
• Check whether #27650's capture-safety changes cover this Qwen3-235B --attention-backend aiter path; if so, chase reviewers rather than opening a duplicate.
• If capture is confirmed as the trigger, examine aiter fused MoE / all-reduce ops for event operations issued on the capturing stream (the multimem all-gather disabled fallback path is a candidate to inspect).

Status

• Persistence: CUDA-graph-capture symptom failing for ≥5 days (every completed run since 2026-07-17); test file has no green run in the queried window (Jul 13–21) → Never-passed.
• Scope: 1 job (nightly-8-gpu-mi35x-qwen3-235b-mxfp4-rocm720) in nightly-test-amd-rocm720.yml; single test file (test_qwen3_instruct_mxfp4.py).
• Blocked work: no CI signal for Qwen3-235B MXFP4 accuracy/perf on MI35x ROCm 7.2.
• In-flight fix: 🟡 #27650 related (capture-safety), not confirmed to cover this path.

nightly-accuracy-8-gpu-mi35x-rocm720 — failed step(s): Accuracy Test MI35x ROCm 7.2 (8-GPU GPT-OSS)

Commit Info

• sglang: 9a6c96083
• aiter: 9127c94a1 — v0.1.17.dev110+g9127c94a1 (source: Dockerfile default; dev image detected → rebuild skipped; AITER_COMMIT_OVERRIDE empty)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi35x/test_qwen3_moe_eval_mi35x.py	test_qwen3_moe_accuracy (TestQwen3MoeEvalMI35x)	AssertionError: Failed models: ['Qwen/Qwen3-30B-A3B'] (server crashed: Memory access fault by GPU)	Log

Other 3 files in this suite passed: test_gpt_oss_eval_mi35x.py, test_gpt_oss_w4a8_mxfp4_eval_mi35x.py, test_qwen3_coder_next_eval_mi35x.py (Test Summary: 3/4 passed).

Failure Summary

The nightly-amd-8-gpu-mi35x suite failed on one file. The Qwen/Qwen3-30B-A3B server (bf16, TP8, aiter enabled) launched and reached "server is fired up and ready to roll", then crashed ~2 s later during the GSM8K benchmark with a GPU memory access fault on 3 GPU nodes (node-8, node-3, node-5), triggering Fatal Python error: Aborted. The scheduler crash is surfaced by a CUDA stream synchronize() in the prefill batch-result path. The test caught the resulting server failure and raised AssertionError: Failed models: ['Qwen/Qwen3-30B-A3B'].

Regression Status

Known recurring failure — Never-passed in the queryable window.

Recent history of test_qwen3_moe_eval_mi35x.py (test_qwen3_moe_accuracy) in job nightly-accuracy-8-gpu-mi35x-rocm720 (most recent comparable completed scheduled runs: branch=main, event=schedule):

Date (UTC)	Run	Test File Status	Failed Function	Error
Jul 21	29854963573	❌ Failed	test_qwen3_moe_accuracy	Memory access fault (nodes 8/3/5) → Failed models: ['Qwen/Qwen3-30B-A3B']
Jul 19	29697452589	❌ Failed	test_qwen3_moe_accuracy	Memory access fault (nodes 9/4/7/5/6/3) → same
Jul 18	29654521083	❌ Failed	test_qwen3_moe_accuracy	Memory access fault (nodes 3/8/7/5/4) → same
Jul 17	29601455852	❌ Failed	test_qwen3_moe_accuracy	Memory access fault (nodes 3/6/8/7) → same
Jul 16	29521474931	⏭️ Cancelled	—	run cancelled (excluded)
Jul 14	29355338821	⚠️ No verdict	—	job failed before this file reached a verdict (inconclusive)
Jul 13	29273096587	⚠️ No verdict	—	job failed before this file reached a verdict (inconclusive)

No passing run for this test file exists in the lookback window (Jul 13–21). The 4 most recent completed runs that ran the file to a verdict all failed with the identical GPU memory access fault. This is not a new regression today — it is a stable, reproducible crash for this model/config.

Failure Cluster

GPU memory access fault during Qwen3-30B-A3B (bf16, TP8) prefill on MI35x

Facts (verified observations)

• Server startup succeeded: [TP0] max_total_num_tokens=15575149 ... available_gpu_mem=72.96 GB; The server is fired up and ready to roll! at 19:18:01, i.e. weight load + CUDA graph capture passed.
• Crash within ~2 s of readiness, on the first benchmark request. Verbatim: Memory access fault by GPU node-8 (Agent handle: 0x614b9abefb00) on address 0x7b582e009000. Reason: Unknown. (also node-3, node-5), then GPU coredump: Directory "/coredumps not writable or does not exist → GPU core dump failed → Fatal Python error: Aborted.
• Crashing thread stack (scheduler process): scheduler.py:1611 event_loop_overlap → scheduler.py:1563 pop_and_process → scheduler.py:3596 process_batch_result → batch_result_processor.py:190 process_batch_result_prefill → torch/cuda/streams.py:231 synchronize. The synchronize() merely surfaces a fault from a prior async GPU kernel; the true offending kernel is not identified in the log.
• The fault address suffix ...009000 is identical across all nodes and across all 4 historical runs — consistent with a deterministic out-of-bounds access, not random corruption.
• Config: sglang serve --model-path Qwen/Qwen3-30B-A3B --max-running-requests 128 --mem-fraction-static 0.8 --trust-remote-code --tp 8 (bf16, SGLANG_USE_AITER=1), threshold 0.75.
• Reproducibility: 4/4 completed runs (Jul 17–21) fail with this exact signature; other 3 files in the same job pass on the same nodes/image, so it is model/config-specific, not a whole-runner or whole-image problem.
• Environment context: an active debug branch bingxche/debug-dsv32-mi35x-mtp-mem-access-fault exists upstream, indicating AMD is already tracking mem-access-fault issues on MI35x.

Hypothesised Causes (with confidence)

• [LOW] A kernel in the Qwen3-30B-A3B bf16 MoE forward path (fused MoE / gemm / attention) performs an out-of-bounds GPU access on gfx950 (MI35x) under ROCm 7.2 + aiter v0.1.17.dev110.
  • Supporting: fault is deterministic (same address suffix), model/config-specific (only this file crashes), occurs on the first real prefill after a clean warmup, and the crashing frame is the prefill batch-result synchronize.
  • Disconfirming: the log does not name the offending kernel (Reason: Unknown); synchronize() only reports a prior async fault, so the stack frame is not the culprit. Cannot distinguish sglang model code vs aiter kernel vs ROCm runtime from the log alone.
  • How to verify: re-run test_qwen3_moe_eval_mi35x.py with AMD_SERIALIZE_KERNEL=3 (or HIP_LAUNCH_BLOCKING=1) to pin the exact faulting kernel; try --attention-backend triton and SGLANG_USE_AITER=0 to isolate aiter vs core; bisect aiter dev builds against a fixed sglang commit.
• [SPECULATION] aiter dev build v0.1.17.dev110+g9127c94a1 regressed a bf16 MoE/gemm path for this shape on gfx950.
  • Supporting: aiter is a dev/patched image build (not the pinned release), and aiter is active in the forward pass.
  • Disconfirming: aiter version is not shown to have changed between a passing and failing run (no passing run exists in the window to compare); the failure predates the current aiter check by at least 4 runs. No evidence aiter changed across these runs.
  • How to verify: A/B the same sglang commit with SGLANG_USE_AITER=0; if the fault disappears, narrow within aiter.

No hypothesis reaches MEDIUM confidence — the log does not identify the faulting kernel. Needs a serialized-kernel repro to localize.

In-flight Fix Check

• In-flight fix: ⚠️ partial — #31405 "[AMD] Gate Qwen3-30B-A3B TP2 qkv/o_proj CK block-FP8 GEMM shapes to Triton (ROCm 7.0)" targets the same model but a different config (TP2, ROCm 7.0, FP8 GEMM) than this failure (TP8, ROCm 7.2, bf16). It may or may not cover the bf16/TP8 path — worth checking with the author, but do not assume it fixes this.
• In-flight fix: ❌ none found matching the exact bf16/TP8/ROCm7.2 MI35x mem-fault signature. Searched open PRs for Qwen3-30B-A3B+mi35x, qwen3_moe+mi35x, memory access fault mi35x.

Suggested Triage Steps

• Reproduce with kernel serialization on MI35x (gfx950) container: run test_qwen3_moe_eval_mi35x.py with AMD_SERIALIZE_KERNEL=3 / HIP_LAUNCH_BLOCKING=1 to identify the exact faulting kernel behind the synchronize().
• Isolate aiter: re-run with SGLANG_USE_AITER=0 and/or --attention-backend triton; if the fault clears, the culprit is in the aiter-selected bf16 MoE/gemm path.
• Confirm scope of #31405 with its author — does its Triton-gating cover the TP8/ROCm7.2/bf16 shapes, or only TP2/ROCm7.0/FP8? If not, this bf16 path needs a separate fix or gate.
• If not quickly fixable, consider quarantining Qwen/Qwen3-30B-A3B in test_qwen3_moe_eval_mi35x.py on MI35x/ROCm7.2 to restore green signal for the other 3 files in this suite, tracked against the fix.

Status

• Persistence: Failing on every completed run that ran this file since at least Jul 17 2026 (4 consecutive completed runs; no green run in the Jul 13–21 lookback window).
• Scope: 1 test file (test_qwen3_moe_eval_mi35x.py) in job nightly-accuracy-8-gpu-mi35x-rocm720; model/config-specific (Qwen3-30B-A3B bf16 TP8). The other 3 accuracy files in the same job pass.
• Blocked work: Blocks the unquantized Qwen3-30B-A3B accuracy signal on MI35x / ROCm 7.2; fails the whole nightly-accuracy-8-gpu-mi35x-rocm720 job.
• In-flight fix: ⚠️ #31405 is same-model but different-config (TP2/ROCm7.0/FP8) — coverage unconfirmed; no exact-match PR found.

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-07-22 22:02 UTC)

[amd-bot]

nightly-test-amd-rocm720.yml — Per-Job Analysis (continued 2/3)

nightly-8-gpu-mi35x-deepseek-v4-flash-rocm720 — failed step(s): Accuracy + Performance Test MI35x ROCm 7.2 (8-GPU DeepSeek-V4-Flash FP8 + FP4, unified_kv_triton)

Commit Info

• sglang: 9a6c96083
• aiter: 9127c94a1 (v0.1.17.dev110+g9127c94a1; source: dev/patched image detected → rebuild skipped, AITER_COMMIT_OVERRIDE empty)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/test_deepseek_v4_flash_fp8_tbo.py	TestDeepseekV4FlashFp8Tbo.test_gsm8k_tbo	AssertionError: np.float64(0.8271417740712661) not greater than 0.91 (retry: 0.9097801364670205, still < 0.91) → retry() exceed maximum number of retries → exit 255	Log

Passing in the same job (both flashmla steps): test_deepseek_v4_flash_fp4.py, test_deepseek_v4_flash_fp8.py.

Failure Summary

The job runs the three DeepSeek-V4-Flash tests twice, once per SGLANG_HACK_FLASHMLA_BACKEND value. In the unified_kv_triton step, test_gsm8k_tbo (line 159, assertGreater(accuracy, 0.91)) scored 0.827 on the first attempt (with invalid=0.0038) and 0.9098 on the single retry — both below the 0.91 threshold — so that step exited 255 and the job is marked failed. The subsequent triton step passed 3/3 (test_gsm8k_tbo hit 0.9098 on first try, then 0.916 on retry). test_deepseek_v4_flash_fp4/fp8 passed in both steps (~0.91–0.92).

Regression Status

Borderline / flaky accuracy test at the 0.91 threshold (with one genuine low outlier). The test was added 2026-07-06 in #29362 (81735ecf80). Observed GSM8K accuracy for the DP+TBO path straddles the threshold across runs (0.827, 0.905, 0.909, 0.9098, 0.910, 0.911, 0.912, 0.914, 0.916, 0.917, 0.920, 0.924), so pass/fail depends heavily on retry luck.

Recent history of test_deepseek_v4_flash_fp8_tbo.py (test_gsm8k_tbo) in job nightly-8-gpu-mi35x-deepseek-v4-flash-rocm720 (completed event=schedule, branch=main runs):

Date	Run	Test File Status	Detail
Jul 21	29854963573	❌ Failed	unified_kv_triton step: 0.827 then 0.9098 (both < 0.91). triton step passed (0.916).
Jul 19	29697452589	✅ Passed	unified_kv_triton: 0.920 (first try). triton: sub-threshold first tries (0.905, 0.909) but eventually 3/3 pass.
Jul 18	29654521083	✅ Passed	Job success ⇒ test passed (both steps).
Jul 17	29601455852	✅ Passed	Job success ⇒ test passed (both steps).
Jul 16	29521474931	⚪ Cancelled	Excluded (not completed).
Jul 14	29355338821	⚠️ Job failed	Failing test not verified at file level (log not downloaded).
Jul 13	29273096587	⚠️ Job failed	Failing test not verified at file level (log not downloaded).

First observed failure for this test in this window: 2026-07-21. Last verified pass: 2026-07-19 (unified_kv_triton scored 0.920 first try).

Failure Cluster

GSM8K accuracy below 0.91 threshold on DeepSeek-V4-Flash FP8 DP+TBO (unified_kv_triton flashmla backend)

Facts (verified observations)

• unified_kv_triton step, test_gsm8k_tbo: attempt 1 Accuracy: 0.827, invalid=0.0037907505686125853; retry Accuracy: 0.9098, invalid=0.0. Both fail assertGreater(..., 0.91); Exception: retry() exceed maximum number of retries. → FAILED (errors=1) → step exit 255 (log: "Exit code 255 indicates test failure (not network issue), not retrying").
• triton step (same job): test_gsm8k_tbo = 0.9098 first try (assertion fail), 0.916 on retry → passed; step reports 3/3 passed.
• All healthy GSM8K runs report invalid=0.0; only the 0.827 attempt reported nonzero invalid (0.0038), i.e. a subset of generations were malformed.
• In the last passing run (Jul 19), the triton step also produced sub-threshold first-try scores (0.905, 0.909) and only passed via retry — the threshold sits inside the run-to-run noise band.
• aiter unchanged (dev image v0.1.17.dev110+g9127c94a1 preserved; [CI-AITER-CHECK] markers confirm no rebuild, no override).
• Regression window Jul 19→Jul 21 = 72 sglang commits (compare).

Hypothesised Causes (with confidence)

• [MEDIUM] Threshold too tight for the observed accuracy distribution. The 0.91 assertGreater bound sits in the middle of the DP+TBO accuracy noise band (0.905–0.924).
  • Supporting: exact value 0.9097801364670205 recurred across steps/runs; the Jul-19 passing run itself dipped to 0.905/0.909 on first tries; both flashmla backends produce values on both sides of 0.91.
  • Disconfirming: the 0.827 attempt is far below the noise band, so a pure "threshold noise" story does not explain that single deep drop.
  • How to verify: run test_gsm8k_tbo on MI35x ROCm 7.2 N×10 with both SGLANG_HACK_FLASHMLA_BACKEND values; record the accuracy distribution and compare against 0.91.
• [LOW] Intermittent generation corruption in the DP+TBO unified_kv_triton path. The 0.827 attempt uniquely had invalid=0.0038 (malformed outputs), and only the unified_kv_triton step failed while triton passed.
  • Supporting: nonzero invalid ratio correlates with the deep accuracy drop; backend-specific (only unified_kv_triton).
  • Disconfirming: single occurrence within this job; the very next retry recovered to 0.9098 with invalid=0.0; no crash/stack trace.
  • How to verify: repeat the unified_kv_triton GSM8K run and inspect the invalid completions when accuracy dips; check for nondeterminism in the op_gather/op_moe/op_combine TBO overlap.
• [LOW] A commit in the Jul 19→21 window shifted DP/TBO numerics. Candidates that touch reachable paths: 688a6d23f1 ("[DeepSeek-V4] Fix idle-rank dummy-extend sparse-prefill crash under DP"), 555267ed05 ("Fix ROCm fused KV and KDA paths", #31688).
  • Supporting: both touch DeepSeek-V4/DP or ROCm-KV code the TBO forward exercises; landed in-window.
  • Disconfirming: no code-level link to GSM8K accuracy; the triton step on the same commit passed; the borderline-noise explanation is simpler and the test already dipped below threshold pre-window.
  • How to verify: git revert each on a branch and re-run test_gsm8k_tbo (unified_kv_triton) on MI35x; A/B the accuracy.

No hypothesis reaches HIGH confidence — the noise floor and single-outlier make clean attribution impossible without repeated sampling.

In-flight Fix Check

In-flight fix: ❌ none found — searched open PRs for deepseek+v4+flash+tbo, two-batch-overlap+accuracy, gsm8k+accuracy+tbo; no PR targets this GSM8K DP+TBO accuracy failure (matches were unrelated TBO/est_time PRs).

Suggested Triage Steps

• Characterize the distribution first: run test_gsm8k_tbo on MI35x ROCm 7.2 ~10× per SGLANG_HACK_FLASHMLA_BACKEND (unified_kv_triton, triton); if accuracy routinely straddles 0.91, the threshold/retry policy is the issue, not a code regression.
• If the distribution is centered well above 0.91 except for rare deep drops, investigate the unified_kv_triton DP+TBO path for intermittent invalid generations (correlate the invalid>0 runs).
• If a real shift is confirmed, bisect the Jul 19→21 window, prioritizing 688a6d23f1 and 555267ed05; git revert on a branch and re-run the unified_kv_triton step.

Status

• Persistence: First failure for this test observed 2026-07-21; last verified pass 2026-07-19 (unified_kv_triton scored 0.920). The triton step passed on 2026-07-21, so only the unified_kv_triton flashmla backend is currently red.
• Scope: 1 test (test_gsm8k_tbo), 1 job (nightly-8-gpu-mi35x-deepseek-v4-flash-rocm720), nightly-test-amd-rocm720.yml; only the unified_kv_triton step. Sibling fp4/fp8 tests pass.
• Blocked work: Marks the DeepSeek-V4-Flash MI35x ROCm 7.2 nightly job red; DP+TBO accuracy signal is unreliable due to the borderline threshold.
• In-flight fix: ❌ none found in open PRs — needs new PR or investigation.

nightly-8-gpu-mi35x-deepseek-r1-mxfp4-tp4-rocm720 — failed step(s): Accuracy Test MI35x ROCm 7.2 (DeepSeek-R1-MXFP4 TP4 MTP)

Commit Info

• sglang: 9a6c96083
• aiter: 9127c94a1 (v0.1.17.dev110+g9127c94a1; source: dev/patched CI image — [CI-AITER-CHECK] Dev/patched version detected → skipping rebuild, no override)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi35x/test_deepseek_r1_mxfp4_tp4_mtp_mi35x.py	TestDeepSeekR1MXFP4TP4MTPMI35x.test_gsm8k	AssertionError: 0.8582259287338894 not greater than or equal to 0.944	Log

Failure Summary

The job runs two accuracy steps against amd/DeepSeek-R1-MXFP4-Preview on MI35x TP=4. Step 6 (plain decode) passed at accuracy=0.945. Step 7 — the EAGLE speculative-decoding (MTP) variant — failed: GSM8K accuracy=0.858 (invalid=0.002) versus the required GSM8K_MTP_ACCURACY_THRESHOLD = 0.944. The failure is a numerical accuracy shortfall in the MTP path, not a crash.

Regression Status

Never-passed (no green run of this test in the queryable history since it was added).

The MTP test was added 2026-07-01 by #29290 with the threshold 0.944 (identical to the plain-decode value). It has failed the accuracy gate on every completed run since, always landing ~0.84–0.86.

Recent history of test_deepseek_r1_mxfp4_tp4_mtp_mi35x.py (test_gsm8k) in job nightly-8-gpu-mi35x-deepseek-r1-mxfp4-tp4-rocm720 (most recent comparable completed scheduled runs, event=schedule, branch main):

Date	Run	Test File Status	MTP Accuracy	Non-MTP (step 6)
Jul 21	29854963573	❌ Failed	0.858	✅ 0.945
Jul 19	29697452589	❌ Failed	0.851	✅ 0.952
Jul 18	29654521083	❌ Failed	0.854	✅ 0.946
Jul 17	29601455852	❌ Failed	0.854	✅ 0.945
Jul 16	29521474931	⏭️ Cancelled	—	—
Jul 14	29355338821	❌ Crashed (exit 134)	—	crash
Jul 13	29273096587	❌ Crashed (exit 134)	—	crash
Jul 5	28749501468	❌ Failed	0.840	✅ 0.942
Jul 3	28676295311	❌ Failed	0.849	✅ 0.951

First observed failure: 2026-07-03 (first clean completed run after the Jul 1 add). No passing run has ever been observed. (Jul 13/14 are a separate exit-134 crash cluster; the MTP accuracy gate never produced a clean pass on any run.) Today's overall run 29854963573 is still queued at the workflow level, but this job completed with failure.

Failure Cluster

EAGLE/MTP GSM8K accuracy below threshold on MI35x DeepSeek-R1-MXFP4 (~0.85 vs 0.944)

Facts (verified observations)

• Step 7 log: accuracy=0.858 invalid=0.002 latency=157.7s immediately followed by FAIL: test_gsm8k and AssertionError: 0.8582259287338894 not greater than or equal to 0.944 at test_deepseek_r1_mxfp4_tp4_mtp_mi35x.py:197.
• The sister step 6 (identical model, same TP=4, no speculative decoding) passed at 0.945 in the same job — so the model, weights, quantization, and non-spec decode path are healthy on this run.
• The gap is stable and reproducible: MTP accuracy = 0.840 / 0.849 / 0.851 / 0.854 / 0.854 / 0.858 across 6 completed runs; plain decode = 0.942–0.952 across the same runs. The delta is a consistent ~9 points.
• MTP config (from the test): --speculative-algorithm EAGLE --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4, attention-backend aiter, fp8 KV cache. Runtime logs show accept len: 2.78–2.86, accept rate: 0.59–0.62 — the draft path is active.
• aiter version is unchanged (Dockerfile default, dev image) across passing plain-decode and failing MTP steps within the same run.
• Not a scout run (nightly-test-amd-rocm720.yml), so no aiter override / Baseline A/B applies.

Hypothesised Causes (with confidence)

• [MEDIUM] EAGLE speculative-decode verify path on ROCm/HIP is producing degraded outputs. EAGLE is designed to be (near-)lossless because draft tokens are verified against the target model, yet the same model that scores 0.945 in plain decode stably scores ~0.85 under EAGLE — a discrepancy that points to a correctness bug in the HIP verify/sampling path rather than expected spec-decode variance.
  • Supporting: 9-point gap is stable across 6 runs; plain-decode baseline is healthy; an open PR #31214 "[ROCm] Fix EAGLE spec-decode verify silently sampling greedy on HIP" describes exactly this class of bug (verify path silently forcing greedy sampling on HIP), which would systematically degrade accuracy. #30105 "[AMD][Spec] Fix aiter GQA packing + split-KV routing in NEXTN spec attention (verify & draft_extend)" also targets the spec verify path.
  • Disconfirming: the test has never passed, so there is no green→red transition proving a regression from a specific commit; the ~0.85 level could equally be the true EAGLE accuracy for this checkpoint (see next hypothesis). Neither PR has been confirmed to lift this specific test above 0.944.
  • How to verify: cherry-pick #31214 onto a branch, re-run test_deepseek_r1_mxfp4_tp4_mtp_mi35x.py on MI35x, and check whether accuracy rises toward the ~0.945 plain-decode baseline.
• [MEDIUM] Threshold mis-set at test creation. GSM8K_MTP_ACCURACY_THRESHOLD = 0.944 was copied from the plain-decode value in #29290 and appears never to have been empirically validated against what the EAGLE config actually achieves on this checkpoint (~0.85).
  • Supporting: the very first completed run after the add (Jul 3, 0.849) already failed; accuracy has never approached 0.944 in any run; the threshold exactly equals the non-MTP passing value, suggesting it was reused rather than measured.
  • Disconfirming: a ~9-point drop is large for lossless spec decode — if EAGLE is behaving correctly the two accuracies should be much closer, which argues the low number is a symptom of a real bug (hypothesis above), not an acceptable operating point.
  • How to verify: determine the intended/measured EAGLE accuracy for this checkpoint from #29290 discussion; if ~0.85 is genuinely expected, the threshold is wrong; if ~0.945 is expected, the bug hypothesis holds.

In-flight Fix Check

• In-flight fix: ✅ #31214 "[ROCm] Fix EAGLE spec-decode verify silently sampling greedy on HIP" (open, created 2026-07-14, not draft) — directly targets the EAGLE-on-HIP verify path that this MTP test exercises; strongest candidate. Ping reviewers rather than opening a duplicate.
• In-flight fix: ✅ #30105 "[AMD][Spec] Fix aiter GQA packing + split-KV routing in NEXTN spec attention (verify & draft_extend)" (open, created 2026-07-04, updated 2026-07-22) — secondary candidate touching the spec verify/draft path.
• Searched MXFP4+MTP, DeepSeek+MXFP4+threshold; no PR that only bumps this test's threshold was found.

Suggested Triage Steps

• Cherry-pick #31214 on a branch and re-run test_deepseek_r1_mxfp4_tp4_mtp_mi35x.py on MI35x ROCm 7.2; compare MTP accuracy against the ~0.945 plain-decode baseline to confirm/deny the greedy-sampling-verify hypothesis.
• If #31214 does not close the gap, evaluate #30105 the same way.
• In parallel, confirm with the #29290 author what EAGLE accuracy was expected/measured for this checkpoint. If ~0.85 is the true lossless-EAGLE number, adjust GSM8K_MTP_ACCURACY_THRESHOLD; if ~0.945 is expected, keep the gate and treat this as a correctness bug pending the spec-verify fix.
• Separately track the Jul 13–14 exit-134 crashes as a distinct signature (not the accuracy cluster) if they recur.

Status

• Persistence: MTP accuracy gate has never passed since the test was added 2026-07-01; failing on every completed run (0.840–0.858), first observed 2026-07-03.
• Scope: this one job/test (nightly-8-gpu-mi35x-deepseek-r1-mxfp4-tp4-rocm720, step 7). The plain-decode sister step in the same job is healthy.
• In-flight fix: ✅ #31214 (EAGLE HIP verify fix) awaiting review; ✅ #30105 secondary.

nightly-test-1-gpu-kernel-rocm720 — failed step(s): Nightly Kernel Benchmark ROCm 7.2 (1-GPU)

Commit Info

• sglang: 9a6c9608
• aiter: 9127c94a — v0.1.17.dev110+g9127c94a1 (source: dev image preserved; [CI-AITER-CHECK] reported "Dev/patched version detected → skipping rebuild")

Failed Tests

Test File	Test Function	Error	Log
test/registered/jit/test_transfer_mamba.py	test_mamba_kernel_backup_load_roundtrip[page_first-dtype0/1]	NameError: name 'transfer_kv_mamba_lf_pf' is not defined	Log
test/registered/jit/test_transfer_mamba.py	test_mamba_kernel_backup_load_roundtrip[page_first_direct-dtype0/1]	same	Log
test/registered/jit/test_transfer_mamba.py	test_mamba_kernel_single_item[page_first / page_first_direct-dtype0/1]	same	Log
test/registered/jit/test_transfer_mamba.py	test_mamba_kernel_full_indices[page_first / page_first_direct-dtype0/1]	same	Log

All 12 failing parametrizations share the identical NameError (4 other tests in the file passed → 12 failed, 4 passed). The overall job ran 20/21 test files green; only test_transfer_mamba.py failed.

Failure Summary

The mamba KV-cache transfer test test_transfer_mamba.py fails 12/16 parametrizations on the AMD ROCm 7.2 kernel nightly. Every failure is a NameError: name 'transfer_kv_mamba_lf_pf' is not defined raised at memory_pool_host.py:407 inside _copy_tensor_all_layers_lf_pf, reached from backup_from_device_all_layer when io_backend="kernel". The 4 passing cases are the non-kernel-backend paths that never reference the symbol.

Regression Status

Never-passed (no green run found) on AMD ROCm720. The test file was introduced 2026-07-18 20:33 UTC by #31501 (99f5a6f4, the flashinfer/mamba-kernel sync). It has failed with the identical NameError in every comparable completed scheduled run since it existed.

Recent history of test/registered/jit/test_transfer_mamba.py in job nightly-test-1-gpu-kernel-rocm720 (comparable completed scheduled runs: branch=main, event=schedule; test only exists in runs after 2026-07-18 20:33 UTC):

Date	Run	Job	Test File Status	Failed Function	Error
Jul 22	29854963573	nightly-test-1-gpu-kernel-rocm720	❌ Failed (12)	test_mamba_kernel_*	NameError: transfer_kv_mamba_lf_pf
Jul 19	29697452589	nightly-test-1-gpu-kernel-rocm720	❌ Failed (12)	test_mamba_kernel_*	same
Jul 18*	29654521083	nightly-test-1-gpu-kernel-rocm720	❌ Failed (12)	test_mamba_kernel_*	same

*Run created 2026-07-18T17:45 UTC but the job executed 07-19T08:42 UTC (after the introducing commit merged), so it contains the test. Runs before the introduction do not include the file and are not comparable. No passing run exists for this file on AMD ROCm720 in the lookback window.

Failure Cluster

NameError: transfer_kv_mamba_lf_pf not imported on ROCm/HIP

Facts (verified observations, no interpretation)

• Verbatim error (12×): NameError: name 'transfer_kv_mamba_lf_pf' is not defined at ../python/sglang/srt/mem_cache/memory_pool_host.py:407, entered via backup_from_device_all_layer → _copy_tensor_all_layers_lf_pf with io_backend='kernel' (log lines 4637, 4639).
• In memory_pool_host.py:41-45 the import from sglang.jit_kernel.transfer_mamba import (transfer_kv_mamba_lf_pf, transfer_kv_mamba_pf_lf) is guarded by if _is_cuda: only. The sibling sgl_kernel.kvcacheio import above it (line 31) uses if _is_cuda or _is_hip:.
• transfer_kv_mamba_lf_pf IS defined (in python/sglang/jit_kernel/transfer_mamba.py:64); it is simply never imported on the HIP build, so the module-global name is undefined at call time.
• The test's skip guard is pytest.mark.skipif(not torch.cuda.is_available(), ...) (test_transfer_mamba.py:21). On ROCm, torch.cuda.is_available() returns True, so the test is NOT skipped; it registered for AMD via register_amd_ci(suite="nightly-amd-kernel-1-gpu", nightly=True) (test_transfer_mamba.py:19).
• Runner is GPU_ARCH=mi30x / gfx942 (log line 4186–4188) → HIP build, _is_cuda=False, _is_hip=True.
• Reproducibility: 12/12 kernel-backend parametrizations fail identically in all 3 runs since the test's introduction (this run + 2 prior scheduled runs).

Hypothesised Causes (with confidence, with disconfirming evidence)

• [HIGH] The mamba JIT transfer kernel import is gated behind if _is_cuda: in memory_pool_host.py:41, so on ROCm/HIP transfer_kv_mamba_lf_pf (and _pf_lf) is never bound, while the call site at line 407 runs unconditionally for io_backend="kernel". The test is CUDA-first and its torch.cuda.is_available() skip does not exclude ROCm.
  • Supporting: verified import guard (line 41) vs unconditional call (line 407); the function exists in the jit module; error is a plain module-global NameError; identical across all runs; introduced together with the feature in #31501. The open fix #31741 edits exactly these two files.
  • Disconfirming: (none found) — the symptom, code path, and the in-flight fix's file set are mutually consistent.
  • How to verify: on an AMD MI30x/gfx942 container, confirm _is_hip=True/_is_cuda=False, then apply #31741 (extend the import to if _is_cuda or _is_hip:) and rerun test/registered/jit/test_transfer_mamba.py.

This is not an aiter-related failure (pure Python NameError, no kernel launch reached); the aiter dev version is a non-factor here.

In-flight Fix Check

• In-flight fix: ✅ [#31741](https://github.com/sgl-project/sglang/pull/31741) (open since 2026-07-20, not draft, mergeable_state=blocked) — "[AMD] Enable mamba JIT transfer kernel on ROCm (fix transfer_kv_mamba NameError)", 2 files / +4 −4, touching exactly python/sglang/srt/mem_cache/memory_pool_host.py and python/sglang/jit_kernel/csrc/kvcacheio/transfer_mamba.cuh. Chase reviewers / CI; do NOT open a duplicate. (Search: is:pr is:open transfer_kv_mamba.)

Suggested Triage Steps

• Verify #31741 resolves this: its diff extends the mamba import guard to cover HIP; confirm it flips the import at memory_pool_host.py:41 to if _is_cuda or _is_hip: and rerun the ROCm720 kernel nightly for test_transfer_mamba.py.
• Since mergeable_state=blocked, check why (required reviews or a failing gate) and unblock, rather than opening a new PR.
• If, after the import fix, the ROCm JIT build of transfer_mamba.cuh fails to compile (the PR also edits the .cuh), that would be a separate ROCm-codegen follow-up — watch the next nightly's JIT build step.

Status (factual, no priority assignment)

• Persistence: Failing every scheduled run since the test was introduced (2026-07-18 20:33 UTC); 0 green runs on AMD ROCm720.
• Scope: 1 test file (12 cases) in job nightly-test-1-gpu-kernel-rocm720. Likely also affects the CUDA-first mamba path on any other HIP kernel job that runs this suite; NVIDIA is unaffected (_is_cuda=True).
• Blocked work: Blocks AMD nightly kernel signal for the mamba KV-cache io_backend="kernel" transfer path.
• In-flight fix: ✅ #31741 (open since 2026-07-20, blocked) — awaiting merge.

nightly-accuracy-2-gpu-vlm-rocm720 — failed step(s): Nightly Accuracy Test ROCm 7.2 (2-GPU VLM MMMU)

Commit Info

• sglang: 9a6c9608 (#31918 "[Cookbook] Add Laguna-S-2.1")
• aiter: 9127c94a — v0.1.17.dev110+g9127c94a1 (source: dev image; [CI-AITER-CHECK] reports "Dev/patched version detected → skipping rebuild")

Failed Tests

All failures are in a single parametrized test, test_mmmu_vlm_models in test/registered/amd/accuracy/mi30x/test_vlms_mmmu_eval_amd.py (one subTest per model). Step 6 "Nightly Accuracy Test ROCm 7.2 (2-GPU VLM MMMU)".

Test File	Test Function (subtest)	Error	Log
test/registered/amd/accuracy/mi30x/test_vlms_mmmu_eval_amd.py	test_mmmu_vlm_models[OpenGVLab/InternVL2_5-2B]	AttributeError: 'CLIPImageProcessor' object has no attribute 'tokenizer' → server exits code 1 → Exception: Server process exited with code 1 (unittest errors=1)	Log
test/registered/amd/accuracy/mi30x/test_vlms_mmmu_eval_amd.py	test_mmmu_vlm_models[Qwen/Qwen2-VL-7B-Instruct]	MMMU Score: 0.280 (threshold: 0.30) → ❌ FAILED	Log
test/registered/amd/accuracy/mi30x/test_vlms_mmmu_eval_amd.py	test_mmmu_vlm_models[Qwen/Qwen2.5-VL-7B-Instruct]	MMMU Score: 0.290 (threshold: 0.33) → ❌ FAILED	Log
test/registered/amd/accuracy/mi30x/test_vlms_mmmu_eval_amd.py	test_mmmu_vlm_models[Qwen/Qwen3-VL-30B-A3B-Instruct]	MMMU Score: 0.280 (threshold: 0.29) → ❌ FAILED	Log

Failure Summary

The 2-GPU VLM MMMU accuracy job failed with two distinct symptoms. (1) The OpenGVLab/InternVL2_5-2B server crashed on startup with AttributeError: 'CLIPImageProcessor' object has no attribute 'tokenizer', which popen_launch_server surfaced as Exception: Server process exited with code 1 — this is the errors=1 that terminated the unittest with exit code 255. (2) Three Qwen VLMs scored below their MMMU thresholds (Qwen2-VL 0.280<0.30, Qwen2.5-VL 0.290<0.33, Qwen3-VL-30B 0.280<0.29). Because the InternVL crash halted the run, only 4 of the 11 active models were evaluated; the remaining 7 (MiniCPM ×2, deepseek-vl2-small, Janus-Pro-7B, Kimi-VL, MiMo-VL, DeepSeek-OCR-2) never ran.

Regression Status

Known recurring failure / Never-passed in queryable window. The InternVL2_5-2B CLIPImageProcessor.tokenizer crash (the terminating errors=1) appears in every completed scheduled run of this job in the window, and the Qwen sub-thresholds are borderline-persistent. No green run of this test file exists in the queryable window (lookback 2026-07-13 → 2026-07-21).

Recent history of test_vlms_mmmu_eval_amd.py (test_mmmu_vlm_models) in job nightly-accuracy-2-gpu-vlm-rocm720 (completed event=schedule, branch=main runs; cancelled runs excluded):

Date	Run	Test File Status	Failed Function(s)	Error
Jul 21	29854963573	❌ Failed	InternVL crash (errors=1); Qwen2-VL/2.5/3 below thr	CLIPImageProcessor…tokenizer + accuracy
Jul 20	29766667696	❌ Failed	InternVL crash; Qwen2.5 0.300<0.33, Qwen3 0.260<0.29 (Qwen2-VL passed 0.300)	same
Jul 19	29697452589	❌ Failed	InternVL crash; Qwen2.5 0.280, Qwen3 0.260	same
Jul 18	29654521083	❌ Failed	job failure (same signature family)	—
Jul 17	29601455852	❌ Failed	InternVL crash; Qwen2.5 0.280, Qwen3 0.260 (Qwen2-VL passed)	same
Jul 16	29521474931	❌ Failed	InternVL crash; Qwen2-VL 0.280, Qwen2.5 0.290, Qwen3 0.260	same
Jul 14	29355338821	❌ Failed	InternVL crash; Qwen2.5 0.280, Qwen3 0.260 (Qwen2-VL passed)	same
Jul 13	29273096587	❌ Failed	job failure (log expired/empty, 2 lines)	—

(Jul 15 and several earlier scheduled runs were cancelled at run level and are excluded. First failing date not pinpointed — no passing run exists in the queried window.)

Failure Cluster

• Cluster A — InternVL2_5-2B VLM server crash on processor load (AttributeError: 'CLIPImageProcessor' object has no attribute 'tokenizer'). This is the errors=1 that aborts the test process.
• Cluster B — Qwen VLM MMMU accuracy below threshold (borderline scores ~0.26–0.30 vs thresholds 0.29–0.33).

Facts (verified observations)

• InternVL server-side traceback (verbatim): tokenizer_manager.py:3211 _get_processor_wrapper → processor.py:271 get_processor → common.py:529 get_tokenizer_from_processor → return processor.tokenizer → AttributeError: 'CLIPImageProcessor' object has no attribute 'tokenizer'.
• common.py:523 carries the comment # Some models don't have an available processor, e.g.: InternVL, and get_tokenizer_from_processor returns processor.tokenizer unguarded when the object is neither a PreTrainedTokenizerBase nor a wrapping processor — a bare CLIPImageProcessor has no .tokenizer.
• The InternVL crash is the errors=1; unittest printed Ran 1 test … / FAILED (errors=1), then the runner reported First attempt failed with exit code 255 / … not retrying.
• Only 4 of 11 active models launched (command=sglang serve … count = 4). AMD_FAILING_VLM_MODELS excludes only GLM-4.1V-9B-Thinking, so 11 models were expected; the run stopped after the InternVL error.
• Qwen scores this run: Qwen2-VL 0.280 (thr 0.30), Qwen2.5-VL 0.290 (thr 0.33), Qwen3-VL-30B 0.280 (thr 0.29). Scores fluctuate run-to-run (Qwen2-VL passed at exactly 0.300 on Jul 14/17/20), i.e. genuinely borderline.
• The identical InternVL CLIPImageProcessor.tokenizer crash is present in the Jul 14/16/17/19/20 logs I downloaded — reproducible, not a one-off.
• aiter version identical concern is N/A: this is not an aiter-triggered symptom; the crash is in HF processor loading and the accuracy gap is a model/threshold matter.

Hypothesised Causes (with confidence)

• [HIGH] Cluster A: AutoProcessor.from_pretrained("OpenGVLab/InternVL2_5-2B", trust_remote_code=True) resolves to a bare CLIPImageProcessor (image-only, no .tokenizer), and get_tokenizer_from_processor (common.py:529) dereferences .tokenizer unconditionally.
  • Supporting: full stack trace ends exactly at common.py:529 return processor.tokenizer; the object type in the error is CLIPImageProcessor; the code comment already flags InternVL as processor-less.
  • Disconfirming: cannot confirm from the log alone why AutoProcessor returns CLIPImageProcessor (candidate: installed transformers lacks a registered InternVL processor / the remote-code processor didn't load) — this sub-question is unverified.
  • How to verify: in the CI image run python -c "from transformers import AutoProcessor; print(type(AutoProcessor.from_pretrained('OpenGVLab/InternVL2_5-2B', trust_remote_code=True)))"; if it returns CLIPImageProcessor, the fault is model/transformers-support, not sglang runtime code.
• [LOW] Cluster B: MMMU thresholds (0.29–0.33) are set slightly above the achievable AMD/ROCm score (~0.26–0.30 on 100 samples) rather than a model regression.
  • Supporting: scores are stable and borderline across a week; some models cross exactly at 0.300; small-sample (100-example) MMMU has high variance.
  • Disconfirming: cannot rule out a real numeric regression on ROCm 7.2 without an NVIDIA/older-ROCm baseline comparison, which was not gathered.
  • How to verify: compare the same models' scores on the CUDA nightly VLM job and on an earlier ROCm image; if CUDA passes and ROCm consistently trails, it's a platform accuracy gap, not a threshold-setting issue.
• No hypothesis links this failure to commit 9a6c9608 (#31918 is a cookbook doc change and is unrelated to VLM serving or MMMU eval). The failure predates this commit (identical since ≥ Jul 14).

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for CLIPImageProcessor+tokenizer (0 results), InternVL+processor (10 results, all unrelated feature PRs — e.g. #24469, #24412), and VLM+MMMU+accuracy+threshold (4 results, none addressing this failure).

Suggested Triage Steps

• Cluster A (highest ROI — it's the errors=1 that masks 7 other models): reproduce the AutoProcessor type check above in the ROCm 7.2 CI image. If InternVL2_5-2B genuinely has no HF processor on the pinned transformers, either add it to AMD_FAILING_VLM_MODELS (like GLM-4.1V-9B-Thinking) so the suite reaches the remaining 7 models, or guard get_tokenizer_from_processor for processor-less models — maintainer's call.
• Cluster A robustness: note that a single model's launch crash aborts the whole 11-model suite (popen_launch_server is outside the per-model try/except, so its exception isn't recorded as a soft per-model failure). Consider whether server-launch failures should be caught per-model so one bad model doesn't hide the rest — design decision for maintainers.
• Cluster B: compare these Qwen MMMU scores against the CUDA VLM nightly and an earlier ROCm build to decide whether to retune thresholds (0.29–0.33 → ~0.27) or investigate a ROCm-specific accuracy gap. Do not adjust thresholds until Cluster A is resolved, since the suite currently can't produce a full 11-model summary.

Status (factual)

• Persistence: Failing every completed scheduled run in the queried window (2026-07-13 → 2026-07-21); no passing run found for this test file. InternVL crash signature identical since at least Jul 14.
• Scope: 1 job (nightly-accuracy-2-gpu-vlm-rocm720) in nightly-test-amd-rocm720.yml. The sibling nightly-perf-2-gpu-vlm-rocm720 job passes. Not observed to be aiter-related.
• Blocked work: The InternVL crash aborts the suite after 4/11 models, so 7 AMD VLM models get zero MMMU accuracy signal on ROCm 7.2 each night.
• In-flight fix: ❌ none found in open PRs.

nightly-accuracy-2-gpu-rocm720 — failed step(s): N/A

Commit Info

• sglang: 9a6c9608
• aiter: v0.1.17.dev110+g9127c94a1 (source: dev image preserved; matches Dockerfile default 9127c94a, no AITER_COMMIT_OVERRIDE)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi30x/test_gsm8k_eval_amd.py	TestNightlyGsm8KEval.test_gsm8k_all_models	AssertionError: The following models failed — driven by mistralai/Mixtral-8x7B-Instruct-v0.1 GSM8K score 0.316 < threshold 0.69	Log

The suite (run_suite.py --suite nightly-amd) runs only two files; test_dumper.py passed (line 4237/4257), so test_gsm8k_eval_amd.py is the sole failing file.

Failure Summary

The 2-GPU nightly accuracy job runs test_gsm8k_all_models, which sequentially serves Llama-3.1-8B, Qwen3-8B, Qwen3-4B, Llama-3.1-70B (tp2) and Mixtral-8x7B (tp2) and asserts each GSM8K score clears its threshold. In the current run the job-log API truncates at 00:44:29 UTC during Mixtral decode — the log is dominated by [aiter] shape is M:… using torch solution:0 GEMM-spam and the returned buffer (~18.5k lines) never reaches the final summary/assertion. The most recent completed run with a readable tail (job 88220532358) shows the actual failure: Mixtral-8x7B GSM8K = 0.316, far below its 0.69 threshold, which raises the AssertionError in check_model_scores.

Regression Status

Known recurring failure — no green run in the queried window.

Recent history of test_gsm8k_eval_amd.py (test_gsm8k_all_models) in job nightly-accuracy-2-gpu-rocm720 (most recent completed scheduled runs, branch=main, event=schedule):

Date	Run	Job	Test File Status	Failed Function	Error
Jul 21	29854963573	job	❌ Failed	test_gsm8k_all_models	Log truncated during Mixtral decode (aiter GEMM spam); reached Mixtral (last model)
Jul 19	29697452589	job	❌ Failed	test_gsm8k_all_models	Mixtral 0.316 < 0.69 (readable); Qwen3-4B threshold N/A
Jul 18	29654521083	job	❌ Failed	test_gsm8k_all_models	Mixtral truncated; Qwen3-4B threshold N/A
Jul 17	29601455852	job	❌ Failed	test_gsm8k_all_models	—
Jul 16	29521474931	job	❌ Failed	test_gsm8k_all_models	—
Jul 14	29355338821	job	❌ Failed	test_gsm8k_all_models	—
Jul 13	29273096587	job	❌ Failed	test_gsm8k_all_models	—

First observed failure: at/before Jul 13 (earliest in query window). No passing run found in the last 15 comparable runs — the failure predates the window. Jul 15/20 runs were cancelled at the workflow level (not counted).

Failure Cluster

Mixtral-8x7B GSM8K accuracy below threshold on AMD (nightly-amd 2-GPU eval)

Facts (verified observations)

• check_model_scores raises AssertionError only for models whose score < threshold; missing-threshold models get ⚠️ NO THRESHOLD and are NOT added to failed_models (test_gsm8k_eval_amd.py:165-201).
• Readable completed run 88220532358: Llama-3.1-8B 0.821✅, Qwen3-8B 0.909✅, Qwen3-4B 0.912 (threshold N/A) ❌, Llama-3.1-70B 0.930✅, Mixtral-8x7B 0.316 (score:std=0.465) vs 0.69 → the assertion-triggering failure.
• Qwen/Qwen3-4B is absent from MODEL_SCORE_THRESHOLDS (test_gsm8k_eval_amd.py:37-67 has Qwen/Qwen3-8B but not Qwen3-4B). Because if threshold and score >= threshold: break (line 279) never breaks when threshold is None, Qwen3-4B runs the full 3-attempt retry loop every time (6 Qwen3-4B metric emissions in the current run) — wasted CI time, but does not itself fail the job.
• Current run: Mixtral server launched once as the last model (00:38 UTC), hit a soft watchdog dump at 00:43:37 while still loading weights (_wait_for_scheduler_ready), then decoded healthily (694 req, 14638 tok/s) until the log ends at 00:44:29. soft_watchdog_timeout=300 is non-fatal; --timeout-per-file 7200 was not reached.
• Benign: aiter JIT emits -mllvm -amdgpu-coerce-illegal-types=1 is not supported by hipcc during pa_ragged build under ROCm 7.2, but recovers (finish build) — not the failure.

Hypothesised Causes (with confidence)

• [HIGH] The job is red because test_gsm8k_all_models asserts on Mixtral-8x7B GSM8K 0.316 << 0.69.
  • Supporting: directly observed in the completed sister run; job fails 6/6; current run reached Mixtral before truncation.
  • Disconfirming: current run's exact Mixtral score is not visible (log truncated) — inferred from the persistent pattern, not re-read for Jul 21.
  • How to verify: re-run with SGLANG_… GEMM logging suppressed, or read github_summary.md / /tmp/gsm8k_mistralai_Mixtral-8x7B-Instruct-v0.1.json artifact.
• [LOW] Root cause of the low Mixtral score is a GSM8K answer-extraction artifact rather than a model/kernel regression.
  • Supporting: high score:std=0.465; in-flight draft PR #29933 "extract GSM8K answer from first #### marker, not last number"; a ci/gsm8k-answer-extraction branch exists.
  • Disconfirming: all other models (Llama 8B/70B, Qwen3) score normally with the same harness, so extraction alone would have to be Mixtral-output-specific; 0.316 is a very large gap.
  • How to verify: re-score the saved Mixtral responses with the #### -first extractor from #29933.
• [LOW] Mixtral MoE numerics on AMD/aiter are degraded (the test carries NO_MOE_PADDING_MODELS / SGLANG_USE_AITER toggles and a comment "lower if AMD scores differently").
  • Supporting: 0.316 is consistent with a broken MoE routing/quant path.
  • Disconfirming: no code-level evidence tying a specific commit to Mixtral MoE in the window; other MoE model (Qwen3-8B/4B) score fine.
  • How to verify: run Mixtral GSM8K locally with SGLANG_USE_AITER=0 vs =1 and compare score.

No hypothesis for the root cause of 0.316 exceeds LOW confidence; needs the saved Mixtral eval artifact or a local A/B on the extractor and the aiter MoE toggle.

In-flight Fix Check

• Mixtral low score: In-flight fix: ⚠️ [#29933](https://github.com/sgl-project/sglang/pull/29933) (open, **draft** since 2026-07-02) — GSM8K answer-extraction fix; may raise Mixtral's score if the failure is extraction-driven. Not merge-ready; verify it actually lifts Mixtral ≥ 0.69 before relying on it.
• Qwen3-4B missing threshold: In-flight fix: ❌ none found — needs a one-line add to MODEL_SCORE_THRESHOLDS (or removal of Qwen3-4B from the eval list).

Suggested Triage Steps

• Retrieve the current run's Mixtral score from github_summary.md / /tmp/gsm8k_mistralai_Mixtral-8x7B-Instruct-v0.1.json (job artifact) to confirm Jul 21 matches the 0.316 seen Jul 19 — the job-page log is unusable because aiter GEMM spam truncates it.
• Re-score saved Mixtral responses with PR #29933's "#### -first" extractor; if score jumps to ≥0.69, the failure is a harness bug — chase reviewers to un-draft/merge #29933.
• If score stays low after re-extraction, A/B Mixtral GSM8K with SGLANG_USE_AITER=0 vs 1 (and SGLANG_MOE_PADDING) on MI325/ROCm 7.2 to isolate the MoE path.
• Independently: add Qwen/Qwen3-4B to MODEL_SCORE_THRESHOLDS (≈0.86) to stop the cosmetic ❌ FAILED and the wasteful 3× retry loop, or drop it from the eval groups.
• Consider gating the per-GEMM [aiter] shape is … logging in CI so the job log stops truncating before the summary.

Status

• Persistence: test_gsm8k_eval_amd.py failing in every completed scheduled run in the query window (Jul 13–19, 6/6); predates the window (no green run found).
• Scope: nightly-accuracy-2-gpu-rocm720 job only (this workflow); GSM8K accuracy signal for TP2 instruction models on AMD MI325/ROCm 7.2.
• Blocked work: AMD 2-GPU GSM8K accuracy gate is permanently red, masking any new accuracy regressions in the other four models in this suite.
• In-flight fix: ⚠️ draft #29933 (Mixtral, unverified); ❌ none for Qwen3-4B threshold.

nightly-test-1-gpu-kernel-rocm720 — failed step(s): Nightly Kernel Benchmark ROCm 7.2 (1-GPU)

Commit Info

• sglang: 0bdd4730
• aiter: 9127c94a — v0.1.17.dev110+g9127c94a1 (source: dev image preserved; Dockerfile default AITER_COMMIT=9127c94a... matched, no rebuild). Not relevant to this failure.

Failed Tests

Test File	Test Function	Error	Log
test/registered/jit/test_transfer_mamba.py	test_mamba_kernel_backup_load_roundtrip[page_first-dtype0/1]	NameError: name 'transfer_kv_mamba_lf_pf' is not defined	Log
test/registered/jit/test_transfer_mamba.py	test_mamba_kernel_backup_load_roundtrip[page_first_direct-dtype0/1]	same	Log
test/registered/jit/test_transfer_mamba.py	test_mamba_kernel_single_item[page_first{,_direct}-dtype0/1]	same	Log
test/registered/jit/test_transfer_mamba.py	test_mamba_kernel_full_indices[page_first{,_direct}-dtype0/1]	same	Log

12 failed, 4 passed in test_transfer_mamba.py. All 12 failures share the identical NameError.

Failure Summary

All 12 kernel-backed mamba transfer tests in test/registered/jit/test_transfer_mamba.py fail with NameError: name 'transfer_kv_mamba_lf_pf' is not defined, raised inside python/sglang/srt/mem_cache/memory_pool_host.py:407 when io_backend="kernel". The 4 passing tests are the non-kernel (io_backend != "kernel") variants, which never reference the undefined name.

Regression Status

Never-passed on AMD ROCm — the test file has failed with this exact NameError in every completed scheduled nightly-test-amd-rocm720.yml run since it was first registered to the AMD kernel suite on 2026-07-20 by #31792 (b6be1507).

Recent history of test/registered/jit/test_transfer_mamba.py in job nightly-test-1-gpu-kernel-rocm720 (comparable completed scheduled runs, event=schedule, branch=main):

Date	Run	Test File Status	Failed Function	Error
Jul 22	29944173150	❌ Failed	12× mamba kernel tests	NameError: transfer_kv_mamba_lf_pf
Jul 21	29854963573	❌ Failed	12× mamba kernel tests	NameError: transfer_kv_mamba_lf_pf
Jul 20	29766667696	— Cancelled	—	run cancelled (not comparable)
≤ Jul 19	—	— Not present	—	test not yet registered (added 2026-07-20)

First observed failure: 2026-07-21 (first scheduled run after registration). No passing run exists for this test file on AMD in the queryable window — Never-passed.

Failure Cluster

NameError: transfer_kv_mamba_lf_pf undefined on ROCm (CUDA-only import guard)

Facts (verified observations)

• Verbatim error at memory_pool_host.py:407: NameError: name 'transfer_kv_mamba_lf_pf' is not defined (main log line 4685), stack test_transfer_mamba.py:218 → memory_pool_host.py:481 (backup_from_device_all_layer) → :407 (_copy_tensor_all_layers_lf_pf).
• The mamba kernel import is guarded by if _is_cuda: at python/sglang/srt/mem_cache/memory_pool_host.py:41-45, importing transfer_kv_mamba_lf_pf / transfer_kv_mamba_pf_lf from sglang.kernels.ops.mamba.transfer_mamba. On ROCm/HIP, _is_cuda is False and _is_hip is True, so neither name is ever bound. (Contrast: the sgl_kernel imports directly above at line 31 use if _is_cuda or _is_hip:.)
• The test's own skip guard (test_transfer_mamba.py:21) is pytest.mark.skipif(not torch.cuda.is_available(), ...). torch.cuda.is_available() returns True on ROCm, so the test is NOT skipped on AMD — it runs and then hits the undefined name.
• Reproducibility: identical 12-test failure in both the Jul 22 and Jul 21 completed runs (24 transfer_kv_mamba_lf_pf matches in the Jul 21 log). The 4 non-kernel tests pass in both.
• aiter unchanged across both runs (v0.1.17.dev110+g9127c94a1, dev image preserved per [CI-AITER-CHECK] markers) — not a factor.

Hypothesised Causes

• [HIGH] The if _is_cuda: import guard at memory_pool_host.py:41 excludes ROCm/HIP, so transfer_kv_mamba_lf_pf is never imported, while the test's torch.cuda.is_available() skip check passes on ROCm and lets the kernel path execute.
  • Supporting: the undefined name is exactly the one in the import block; _is_cuda is false on HIP; the sgl_kernel block one line above deliberately uses _is_cuda or _is_hip; failure is 100% reproducible and AMD-only (the CUDA base-b-kernel-unit registration presumably passes since _is_cuda is true there).
  • Disconfirming: (none found) — the failure reproduces deterministically without any aiter/version variance.
  • How to verify: on an AMD MI30x container, check whether sglang.kernels.ops.mamba.transfer_mamba imports cleanly under HIP; if it does, widening the guard to if _is_cuda or _is_hip: (as #31741 proposes) should turn the 12 tests green.

In-flight Fix Check

• In-flight fix: ✅ #31741 — "[AMD] Enable mamba JIT transfer kernel on ROCm (fix transfer_kv_mamba NameError)", open since 2026-07-20, last updated 2026-07-21. This directly targets the transfer_kv_mamba NameError. Chase reviewers; do NOT open a duplicate.
• (Searched: is:pr is:open transfer_kv_mamba and transfer_mamba ROCm — 1 matching PR.)

Suggested Triage Steps

• Verify #31741 resolves the guard (widen if _is_cuda: → if _is_cuda or _is_hip: at memory_pool_host.py:41, or otherwise ensure the mamba transfer kernel imports on HIP), then re-run nightly-amd-kernel-1-gpu suite on an AMD MI30x runner.
• If the mamba JIT kernel is genuinely unsupported on ROCm, the alternative is to tighten the test skip guard in test_transfer_mamba.py:21 to skip the io_backend="kernel" cases on HIP — but confirm intended ROCm support first (the existence of #31741 suggests enabling, not skipping).
• Confirm the CUDA sister registration (base-b-kernel-unit, 1-gpu-large) passes to establish this is AMD-specific.

Status

• Persistence: Failing since 2026-07-21 (first scheduled run after the 2026-07-20 registration); every completed run since has shown the identical 12-test failure. Never-passed on AMD.
• Scope: 1 job (nightly-test-1-gpu-kernel-rocm720) in nightly-test-amd-rocm720.yml. Likely also affects the non-720 AMD nightly kernel suite (same import guard); CUDA is unaffected (_is_cuda true).
• Blocked work: Mamba KV-cache kernel D2H/H2D transfer coverage on ROCm.
• In-flight fix: ✅ #31741 (open since 2026-07-20) — awaiting merge.

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-07-22 22:02 UTC)

[amd-bot]

pr-test-amd.yml — 25 failure(s)

Run started (UTC): 2026-07-21 05:07 UTC
Last scanned (UTC): 2026-07-22 19:32 UTC
Commits: sglang b13abfd

Summary

Counts: 25 failures · 10 clusters · 17 carrying over (≥3 days) · 5 new today · 2 flaky · 1 unknown

#	Cluster	Job	Test File	Test Function	Status	Hypothesis (confidence)
1	1-GPU diffusion hang / runner death (no log)	mm-gen 1gpu (0)	diffusion 1-gpu suite (part 0)	N/A (hang, no log)	never-passed ≥8d	[LOW] server hang, log lost
2	1-GPU diffusion hang / runner death (no log)	mm-gen 1gpu (1)	diffusion 1-gpu suite (part 1)	N/A (runner lost comm)	never-passed ≥8d	[MEDIUM] resource starvation
3	1-GPU diffusion hang / runner death (no log)	mm-gen 1gpu (0)	diffusion 1-gpu suite (part 0)	N/A (runner death)	never-passed ≥8d	[LOW] hang, log lost
4	1-GPU diffusion hang / runner death (no log)	mm-gen 1gpu (1)	diffusion 1-gpu suite (part 1)	N/A (runner lost comm)	never-passed ≥9d	[MEDIUM] CPU/mem starvation
5	1-GPU diffusion hang / runner death (no log)	mm-gen 1gpu (0)	diffusion 1-gpu suite (part 0)	N/A (hang)	never-passed ≥10d	[MEDIUM] partition timeout/hang
6	1-GPU diffusion hang / runner death (no log)	mm-gen 1gpu (1)	diffusion 1-gpu suite (part 1)	N/A (killed mid-step)	recurring ≥4d	[LOW] OOM/hang
7	1-GPU diffusion hang / runner death (no log)	mm-gen 1gpu (0)	diffusion 1-gpu suite (part 0)	N/A (hang)	never-passed	[LOW] hang, log lost
8	1-GPU diffusion hang / runner death (no log)	mm-gen 1gpu (1)	diffusion 1-gpu suite (part 1)	N/A (hang)	never-passed ≥7d	[MEDIUM] partition too heavy
9	update_weights_from_disk reload → HTTP 400	mm-gen 1gpu (3)	test_update_weights_from_disk.py	TestUpdateWeightsFromDisk (Qwen-Image ×4 + FLUX.2 ×4)	never-passed ≥4.5d	[MEDIUM] diffusers layout shape mismatch
10	update_weights_from_disk reload → HTTP 400	mm-gen 1gpu (3)	test_update_weights_from_disk.py	same (4 failed, 4 errors)	never-passed since Jul14	[HIGH] diffusers layout mismatch (#31924)
11	update_weights_from_disk reload → HTTP 400	mm-gen 1gpu (3)	test_update_weights_from_disk.py	same	never-passed	[FACT] diffusers layout mismatch
12	update_weights_from_disk reload → HTTP 400	mm-gen 1gpu (3)	test_update_weights_from_disk.py	same (5 failed, 1 passed)	never-passed	[HIGH] diffusers layout mismatch
13	2-GPU FLUX.2 fp8 HIPBLAS unsupported + hang	mm-gen 2gpu (0)	test_server_2_gpu.py	TestDiffusionServerTwoGpu::test_diffusion_generation	never-passed ≥9d	[LOW] hang (log lost)
14	2-GPU FLUX.2 fp8 HIPBLAS unsupported + hang	mm-gen 2gpu (0)	test_server_2_gpu.py	…[flux2_modelopt_fp8_tp2_t2i]	never-passed ≥8d	[FACT] HIPBLAS_STATUS_NOT_SUPPORTED fp8 GEMM (#28889)
15	2-GPU FLUX.2 fp8 HIPBLAS unsupported + hang	mm-gen 2gpu (0)	test_server_2_gpu.py	…[flux2_modelopt_fp8_tp2_t2i] / hang	never-passed	[MEDIUM] fp8 HIPBLAS
16	MXFP4 MoE GSM8K below threshold	stage-b-1gpu-small-mi35x	test_quark_mxfp4.py	TestFP8ToMXFP4MOETP1.test_gsm8k	new (test <1d old)	[MEDIUM] >0.92 threshold too tight
17	MXFP4 MoE GSM8K below threshold	mm-gen 2gpu (0)	test_quark_mxfp4.py	TestFP8ToMXFP4MOETP1.test_gsm8k	new (first inclusion)	[MEDIUM] threshold too tight
18	KeyError 'answer' on PD+PP disaggregation	stage-b-8gpu-disagg-mi35x	test_disaggregation_pp.py	TestDisaggregationPrefillPPDynamicChunkAccuracy.test_gsm8k (L198)	flaky ~4/9 since Jul18	[MEDIUM] PD+PP consensus race (#31869/#31798)
19	KeyError 'answer' on PD+PP disaggregation	stage-b-8gpu-disagg-mi35x	test_disaggregation_pp.py	TestDisaggregationDecodePPAccuracy.test_gsm8k (L293)	flaky	[MEDIUM] PD+PP consensus race
20	LoRA concurrent-adapter latency > 4000ms	stage-b-1gpu-large (0)	test_bench_serving_1gpu_part1.py	test_online_lora_latency_with_concurrent_adapter_updates	flaky 2/7 since Jul21	[MEDIUM] perf gate too tight / stall
21	LoRA concurrent-adapter latency > 4000ms	stage-b-1gpu-large (0)	test_bench_serving_1gpu_part1.py	test_online_lora_latency_with_concurrent_adapter_updates	flaky	[MEDIUM] perf gate flake
22	reward-model _legacy_load_weights AttributeError	stage-b-1gpu-nondeterm	test_reward_models.py	TestRewardModels.test_reward_scores	🆕 new Jul22	[HIGH] becf252e6 (#28671)
23	Dynamo break: tc_piecewise ParallelContext	stage-c-8gpu-mi35x (1)	test_deepseek_r1_mxfp4_8gpu.py	server launch (setUpClass)	🆕 new Jul22	[HIGH] 745b2ca4 (#31816)
24	communicator missing get_server_args	stage-c-8gpu (3)	test_aiter_allreduce_fusion_amd.py	test_dense_tp_fuses	🆕 new Jul22	[HIGH] 745b2ca4 (#31816) removed accessor
25	analysis incomplete (manual triage)	stage-c-8gpu-mi35x (0)	unknown (step "Run test")	unknown	unknown	[no candidate ≥ MEDIUM]

Cluster: 1-GPU diffusion hang / runner death (no log) (rows 1-8)

• Affected: multimodal-gen-test-1-gpu-amd shards 0 & 1 across all 4 runs. Shard 2 passes; shard 3 fails for a different reason (cluster C).
• Shared facts: step 9 "Run diffusion server tests (1-GPU)" freezes in_progress / runner "lost communication"; job killed at ~64-112 min (under the 90-min step timeout); log blob BlobNotFound, 0 artifacts → zero usable signal, so no per-test granularity possible.
• Top hypothesis: [MEDIUM] host CPU/VRAM starvation or GPU/agent crash on the heaviest LPT partitions (video/large diffusion models). Disconfirming: no log/dmesg to confirm OOM vs network drop; kill time varies.
• In-flight fix: ❌ none (⚠️ #31361 diffusion worker self-kill is [LOW]-confidence, unverified).
• Suggested triage: add flush-before-death diagnostics (rocm-smi/dmesg/free -h to artifact dir) + per-case timeout so a hung shard uploads a log; reproduce shard 0/1 partition locally on MI325 with monitoring.

Cluster: update_weights_from_disk reload → HTTP 400 (rows 9-12)

• Affected: multimodal-gen-test-1-gpu-amd shard 3, every run; test_update_weights_from_disk.py (Qwen-Image ×4 FAILED + FLUX.2 ×4 ERROR at setup).
• Shared facts: server rejects transformer reload with HTTP 400 — server-side assert param_data.shape == loaded_weight.shape at linear.py:410; preceding img_in.bias → x_embedder.bias remap note → diffusers↔sglang layout mismatch. Deterministic since ≥Jul14; AMD-lane only.
• Top hypothesis: [HIGH] diffusers-format checkpoint layout not reconciled in reload path (documented in #31924). Disconfirming: none.
• In-flight fix: ✅ #31925 (skip on ROCm, open since Jul21); real fix tracked in #31924. Do NOT duplicate.

Cluster: 2-GPU FLUX.2 fp8 HIPBLAS unsupported + hang (rows 13-15)

• Affected: multimodal-gen-test-2-gpu-amd part 0; test_server_2_gpu.py. Parts 1 & 2 pass.
• Shared facts: logged runs show FAILED …[flux2_modelopt_fp8_tp2_t2i] with HIPBLAS_STATUS_NOT_SUPPORTED at [denoising_step_0]; other runs escalate to hang/no-log (Mode B).
• Top hypothesis: [FACT] modelopt-FP8 GEMM unsupported by hipBLAS on gfx942. Disconfirming: missing-log runs can't be confirmed as the same cause.
• In-flight fix: ✅ #28889 ("native e4m3fnuz fp8 GEMM for FLUX.2 modelopt-FP8 on gfx942", blocked). Chase reviewers.

Cluster: MXFP4 MoE GSM8K below threshold (rows 16-17)

• Affected: test_quark_mxfp4.py::TestFP8ToMXFP4MOETP1.test_gsm8k (Qwen3-30B-A3B FP8→quark_mxfp4, TP=1). Test introduced Jul21 09:50 by #28291 (dcd9014f).
• Shared facts: assertGreater(acc, 0.92) (strict, zero margin); observed 0.912/0.920 (fail) vs 0.922 (pass). Reference in code is ~0.948.
• Top hypothesis: [MEDIUM] threshold set with no headroom vs GSM8K@500 variance. Disconfirming: only ~2 data points; a genuine numerics regression not fully excluded.
• In-flight fix: ❌ none.

Cluster: KeyError 'answer' on PD+PP disaggregation (rows 18-19)

• Affected: stage-b-test-large-8-gpu-mi35x-disaggregation-amd; test_disaggregation_pp.py PP variants (KeyError: 'answer' at few_shot_gsm8k.py:123). Intermittent ~4/9 runs since Jul18; non-PP disagg tests pass.
• Top hypothesis: [MEDIUM] PD+PP consensus race drops/aborts a request → empty answer. Disconfirming: no captured abort event in this run's logs.
• In-flight fix: ✅ #31869 + #31798 (PD+PP consensus) — validate against this test.

Cluster: LoRA concurrent-adapter latency > 4000ms (rows 20-21)

• Affected: stage-b-test-1-gpu-large-amd; test_online_lora_latency_with_concurrent_adapter_updates asserts median-E2E <4000 — measured 9400-20742 ms. Fail→pass→fail across 3 runs (2/7 fail).
• Top hypothesis: [MEDIUM] flaky/too-tight perf gate; adapter load/unload intermittently stalls decode (Max ITL 18.6 s). Disconfirming: can't exclude a real regression under jitter.
• In-flight fix: ❌ none.

Cluster: reward-model _legacy_load_weights AttributeError (row 22)

• Affected: stage-b-test-1-gpu-small-amd-nondeterministic; test_reward_models.py::TestRewardModels.test_reward_scores → scheduler sigquit (exit -9). 🆕 NEW Jul22 (last green Jul21 12:32).
• Top hypothesis: [HIGH] becf252e6 (#28671) made LlamaForCausalLM.load_weights dispatch to self._legacy_load_weights, which reward classes (not subclasses of LlamaForCausalLM) lack. Disconfirming: none.
• In-flight fix: ❌ none — needs new PR. Blast-radius: any model using unbound LlamaForCausalLM.load_weights(self,…).

Cluster: Dynamo break — tc_piecewise ParallelContext (row 23)

• Affected: stage-c-test-large-8-gpu-amd-mi35x (1); test_deepseek_r1_mxfp4_8gpu.py server launch — torch._dynamo.exc.Unsupported: __getattribute__ of ParallelContext during prefill CUDA-graph capture. 🆕 NEW Jul22 (last green Jul21 12:32).
• Top hypothesis: [HIGH] 745b2ca4 (#31816) switched is_dsa_enable_prefill_cp() to get_parallel().…, routing through ParallelContext.__getattr__ → object.__getattribute__, untraceable by Dynamo. Disconfirming: none.
• In-flight fix: ❌ none.

Cluster: communicator missing get_server_args (row 24)

• Affected: stage-c-test-large-8-gpu-amd (3); test_aiter_allreduce_fusion_amd.py::test_dense_tp_fuses — mock.patch.object(comm, "get_server_args") fails, symbol removed. 🆕 NEW Jul22 (last green Jul22 00:26).
• Top hypothesis: [HIGH] same commit 745b2ca4 (#31816) removed communicator.get_server_args, racing the test added by #31784 that still patches it. Disconfirming: none.
• In-flight fix: ❌ none.

Suggested triage order

Rank	Triage step	Targets	Why this first
1	Investigate #31816 (745b2ca4) — the get_parallel() config-accessor refactor; revert/patch on a branch and re-run	rows 23, 24 (clusters H+I)	One new commit broke two distinct clusters today; cheap high-confidence win
2	Fix reward-model load path from #28671 (becf252e6); grep other unbound LlamaForCausalLM.load_weights callers	row 22 (cluster G)	NEW today, [HIGH] confidence, potential wider blast radius
3	Land skip PR #31925 + drive #31924 real fix	rows 9-12 (cluster C, 4 jobs)	Largest deterministic never-passed cluster with an in-flight fix ready
4	Add flush-before-death diagnostics + per-case timeout to the diffusion suite	rows 1-8, 13-15 (clusters A+B, 11 jobs)	Biggest blast radius but currently zero signal — must fix observability before root-cause
5	Validate #31869/#31798 against test_disaggregation_pp.py; chase #28889 for the fp8 HIPBLAS content failure	rows 18-19, 14-15	In-flight fixes exist; verify/unblock
6	Re-run to characterize distributions; loosen gates if flaky	rows 16-17 (MXFP4), 20-21 (LoRA latency)	Boundary-flake gates; cheap rerun evidence, no code fix likely needed. Row 25 needs manual log re-fetch (analysis incomplete).

Note: clusters H and I share the same root commit 745b2ca4 (#31816) despite different symptoms — addressing it resolves both.

---

Job	Failed Steps	Started
multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 0)	N/A	2026-07-21T06:42
multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 1)	N/A	2026-07-21T05:11
multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 0)	N/A	2026-07-21T05:07
multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 3)	Run diffusion server tests (1-GPU)	2026-07-21T05:11
stage-b-test-1-gpu-large-amd (linux-mi325-1gpu-sglang, 0)	Run test	2026-07-21T05:17
multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 0)	N/A	2026-07-21T18:39
multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 1)	N/A	2026-07-21T18:35
multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 3)	Run diffusion server tests (1-GPU)	2026-07-21T18:20
stage-c-test-large-8-gpu-amd-mi35x (linux-mi35x-gpu-8, 0)	Run test — ⚠️ analysis failed	2026-07-21T12:43
stage-b-test-large-8-gpu-mi35x-disaggregation-amd (linux-mi35x-gpu-8.fabric)	Run test_disaggregation	2026-07-21T12:58
stage-b-test-1-gpu-small-amd-mi35x (linux-mi35x-gpu-1)	Run test	2026-07-22T00:26
multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 0)	N/A	2026-07-21T23:07
stage-b-test-large-8-gpu-mi35x-disaggregation-amd (linux-mi35x-gpu-8.fabric)	Run test_disaggregation	2026-07-22T00:43
multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 3)	Run diffusion server tests (1-GPU)	2026-07-22T02:36
stage-b-test-1-gpu-large-amd (linux-mi325-1gpu-sglang, 0)	Run test	2026-07-22T02:51
multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 1)	N/A	2026-07-22T02:20
multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 0)	N/A	2026-07-22T02:34
stage-b-test-1-gpu-small-amd-nondeterministic (linux-mi325-1gpu-sglang)	Run test	2026-07-22T04:26
multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 0)	N/A	2026-07-22T04:17
multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 3)	Run diffusion server tests (1-GPU)	2026-07-22T13:55
multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 1)	N/A	2026-07-22T14:01
stage-c-test-large-8-gpu-amd-mi35x (linux-mi35x-gpu-8, 1)	Run test	2026-07-22T14:54
multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 0)	N/A	2026-07-22T13:53
multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 0)	N/A	2026-07-22T15:55
stage-c-test-large-8-gpu-amd (linux-mi325-8gpu-sglang, 3)	Run test	2026-07-22T18:10

Per-Job Analysis

multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 0) — failed step(s): N/A

Commit Info

• sglang: 8905cbd4
• aiter: N/A — the current run's job-log blob has expired (BlobNotFound from the logs endpoint), so the [CI-AITER-CHECK] markers cannot be read for this run.

Failed Tests

Test File	Test Function	Error	Log
python/sglang/multimodal_gen/test/server/test_server_2_gpu.py	TestDiffusionServerTwoGpu::test_diffusion_generation[*] (parametrized 2-GPU cases; exact case not recoverable)	Step "Run diffusion server tests (2-GPU)" started but never completed — job killed ~82 min later; log blob unavailable	Job

Partition 0 runs the parametrized file test_server_2_gpu.py (the TWO_GPU_CASES T2V/T2I/I2V/LoRA group). The two sibling shards — part 1 (single_test_file/test_disagg_server.py) and part 2 (single_test_file/test_ar_models.py) — both passed in this run.

Failure Summary

The multimodal-gen-test-2-gpu-amd (…, 0) shard failed at step 9 "Run diffusion server tests (2-GPU)". The step began at 07:06:42Z and never reported completion; the job was terminated at 08:28:49Z (~82 min into the step, well under its 150-min timeout-minutes), with conclusion: failure and the step conclusion left None. No failure artifacts were uploaded (step 10 never ran) and the GitHub log blob returns BlobNotFound, so the specific failing test case and stack trace for this run are not recoverable.

Regression Status

Known recurring failure — this shard (part 0) has failed in all 15 most recent comparable completed scheduled runs (branch main, event=schedule). No green run for test_server_2_gpu.py on this shard exists in the query window (2026-07-13 → 2026-07-21). The failure is not introduced by commit 8905cbd4.

Recent history of test_server_2_gpu.py in job multimodal-gen-test-2-gpu-amd (…, 0) (15 comparable completed event=schedule runs on main):

Date (UTC)	Run	Job	Test File Status	Failure Mode
07-21	run	part 0	❌ Failed	Step never completed → killed (log lost)
07-20	run	part 0	❌ Failed	Step never completed → killed
07-20	run	part 0	❌ Failed	Step never completed → killed
07-19	run	part 0	❌ Failed	Step never completed → killed
07-19	run	part 0	❌ Failed	Step never completed → killed
07-18	run	part 0	❌ Failed	Fast fail (~10 min)
07-18	run	part 0	❌ Failed	1 failed [flux2_modelopt_fp8_tp2_t2i] + error [joy_echo_t2v_2gpu] (HF download)
07-17	run	part 0	❌ Failed	Step never completed → killed
07-17	run	part 0	❌ Failed	HIPBLAS_STATUS_NOT_SUPPORTED → 17 setup errors
07-16	run	part 0	❌ Failed	HIPBLAS_STATUS_NOT_SUPPORTED → 17 setup errors
07-15	run	part 0	❌ Failed	Step never completed → killed
07-15	run	part 0	❌ Failed	Step never completed → killed
07-14	run	part 0	❌ Failed	Step reported failure
07-14	run	part 0	❌ Failed	Step never completed → killed
07-13	run	part 0	❌ Failed	Step never completed → killed

(Run 29461421122 on 07-16 00:25 was cancelled and excluded. First failure predates the query window; no passing run found ⇒ Never-passed within lookback 2026-07-13 → 2026-07-21.)

Failure Cluster

Chronically-red 2-GPU diffusion shard — heterogeneous symptoms: (a) test step hangs and the job is killed (no log flushed), (b) HIPBLAS_STATUS_NOT_SUPPORTED during the denoising stage, (c) individual FLUX.2 fp8 case failure + HF model-download errors.

Facts (verified observations)

• Current run: step 9 "Run diffusion server tests (2-GPU)" started 2026-07-21T07:06:42Z, completed_at: None; job killed at 08:28:49Z; conclusion: failure. Step timeout is 150 min (workflow pr-test-amd.yml:890); only ~82 min elapsed, so the step timeout was not the trigger.
• Current run: logs endpoint returns <Error><Code>BlobNotFound>… and artifacts total_count = 0 — the runner's log/artifacts were never finalized, consistent with runner/agent loss mid-step.
• Part 0 = parametrized test_server_2_gpu.py; parts 1 & 2 (standalone files) passed in this run (job steps all success).
• Historical part-0 failures are not uniform:
  • Runs 29498496614 (07-16) and 29544840981 (07-17): RuntimeError: CUDA error: HIPBLAS_STATUS_NOT_SUPPORTED at [denoising_step_0] / [DenoisingStage] → server crash → 1 failed, 6 passed, 1 skipped, 17 errors (verbatim summary line).
  • Run 29623067212 (07-18): FAILED …[flux2_modelopt_fp8_tp2_t2i] and ERROR …[joy_echo_t2v_2gpu] — the latter is ValueError: Could not find model at jdopensource/JoyAI-Echo … CAS Client Error … error decoding response body → RuntimeError: Server exited early (code 1) (a Hugging Face download/infra error, not an sglang bug).
• 10 of the 15 historical part-0 jobs share the current run's "step never completes → killed" pattern.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [LOW] Current run: runner/GPU hang or runner-agent loss during the 2-GPU test step.
  • Supporting: step never reported completion; log blob and artifacts were never finalized (BlobNotFound, 0 artifacts); killed at ~82 min (< 150-min step timeout); 10/15 recent part-0 jobs show the identical incomplete-step pattern.
  • Disconfirming: without the log, an in-test crash that also crashed the runner cannot be excluded; the exact GPU/host state at kill time is unknown.
  • How to verify: re-run multimodal-gen-test-2-gpu-amd part 0 on an mi325 2-GPU runner; if it hangs again, attach py-spy/rocm-smi monitoring to the server processes during test_server_2_gpu.py.
• [MEDIUM] The HIPBLAS_STATUS_NOT_SUPPORTED cluster (07-16/07-17) stems from an fp8/hipBLAS GEMM path unsupported on gfx942.
  • Supporting: RuntimeError: CUDA error: HIPBLAS_STATUS_NOT_SUPPORTED fires at [denoising_step_0] in the DenoisingStage; the individually-FAILED case in 07-18 is flux2_modelopt_fp8_tp2_t2i (a modelopt-FP8 case), and open PR #28889 explicitly adds a native e4m3fnuz fp8 GEMM for FLUX.2 modelopt-FP8 on gfx942.
  • Disconfirming: this cluster does not appear in every failing run (the current run's symptom is a hang, not HIPBLAS), so it does not explain all 15 failures.
  • How to verify: run test_server_2_gpu.py -k flux2_modelopt_fp8_tp2_t2i on gfx942 with and without PR #28889 applied.
• [FACT] The joy_echo_t2v_2gpu errors are Hugging Face model-download/CAS failures, not sglang defects (verbatim CAS Client Error … error decoding response body).

Overall: no single hypothesis explains all 15 runs; the shard is failing for at least three independent reasons. For the current run specifically, no hypothesis exceeds LOW confidence because the log is unavailable — needs a re-run with the log preserved.

In-flight Fix Check

• fp8/HIPBLAS cluster: ✅ #28889 "[AMD] Use native e4m3fnuz fp8 GEMM for FLUX.2 modelopt-FP8 on gfx942" — directly targets the flux2_modelopt_fp8_tp2_t2i case and the HIPBLAS_STATUS_NOT_SUPPORTED symptom. Chase reviewers rather than opening a duplicate.
• Current-run hang: ❌ none found — no open PR addresses the 2-GPU diffusion shard hang / runner-loss pattern. Searched is:pr is:open with keywords HIPBLAS+diffusion, multimodal_gen+2-gpu, diffusion+amd.

Suggested Triage Steps

• Re-run multimodal-gen-test-2-gpu-amd part 0 at 8905cbd4 and, if it hangs again, capture py-spy dump on the server/worker processes plus rocm-smi to localize the hang inside test_server_2_gpu.py.
• Verify PR #28889 resolves the flux2_modelopt_fp8_tp2_t2i / HIPBLAS_STATUS_NOT_SUPPORTED cluster by running that case on gfx942 with the PR applied.
• Consider splitting/quarantining the heaviest cases in the part-0 parametrized group — the step routinely runs ~1:30–1:45 and repeatedly approaches its window, making hangs likely; separately track the flaky joy_echo_t2v_2gpu HF-download error as infra, not code.
• Because this shard is Never-passed in the lookback window, treat green sibling shards (parts 1 & 2) as the only trustworthy 2-GPU signal until part 0 is stabilized.

Status

• Persistence: Failing on every one of the last 15 completed scheduled pr-test-amd.yml runs (2026-07-13 → 2026-07-21); no green run for test_server_2_gpu.py on this shard in the window.
• Scope: multimodal-gen-test-2-gpu-amd part 0 only; sibling parts 1 & 2 pass. Blocks the 2-GPU parametrized diffusion (T2V/T2I/I2V/LoRA) signal on AMD.
• In-flight fix: ✅ #28889 for the fp8/HIPBLAS cluster; ❌ none for the hang / runner-loss pattern seen in this run.

multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 1) — failed step(s): N/A

Commit Info

• sglang: 8905cbd4
• aiter: N/A — shard-1 job log is unavailable (see below), so no [CI-AITER-CHECK] markers could be read for this job. (The sibling shard-3 log shows the standard Dockerfile-default aiter kernels building normally.)

Failed Tests

Test File	Test Function	Error	Log
N/A — whole 1-gpu partition (--partition-id 1 --total-partitions 4)	N/A (no test result recorded)	Diffusion step hung and was terminated (step conclusion None); job killed at ~90 min with no log or artifacts uploaded	Job

This is not a test-assertion failure — it is a hang/timeout before any per-test result was recorded. Because the runner was force-terminated, the step-9 log blob returns BlobNotFound, the log is absent from the run-level archive (only shards 2 & 3 are present), and the run has 0 artifacts. The specific hanging case therefore cannot be pinned from this job's own evidence.

Failure Summary

The job multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 1) ran step 9 "Run diffusion server tests (1-GPU)" starting 05:25:43 UTC; the step never recorded a conclusion and the job was terminated after ~90 min total wall time (start 05:11:30 → end 06:41:35 UTC, 2026-07-21). Steps 10 (artifact upload) and 20 (post-checkout) also show None, confirming the job was hard-killed rather than failing cleanly. The lighter sibling partition shard 2 passed in 15.3 min in the same run, so the failure is specific to the cases assigned to this LPT partition, not the runner or environment as a whole.

Regression Status

Known recurring failure / persistent hang — Never-passed in the queryable window. Shard 1's diffusion step has failed to complete in every comparable completed scheduled run I could query (lookback 2026-07-14 00:24 → 2026-07-21 00:27 UTC, branch=main, event=schedule, status=completed). No green run for this shard exists in the window, and the failure predates the current commit by ≥7 days.

Recent history of multimodal-gen 1-gpu suite shard 1 in job multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 1) (comparable completed scheduled runs; hung-job logs all BlobNotFound):

Date (UTC)	Run	Diffusion Step (9)	Job dur	Status
07-21	29790347494	None (never completed)	90.1 min	❌ hang→kill
07-20	29743214297	None	84.1 min	❌ hang→kill
07-20	29709549202	None	71.0 min	❌ hang→kill
07-19	29686709871	None	91.1 min	❌ hang→kill
07-19	29666971143	None	75.1 min	❌ hang→kill
07-18	29644059344	None	71.0 min	❌ hang→kill
07-18	29623067212	None	72.1 min	❌ hang→kill
07-17	29580198489	None	75.1 min	❌ hang→kill
07-17	29544840981	None	71.0 min	❌ hang→kill
07-16	29498496614	None	71.1 min	❌ hang→kill
07-15	29415344552	None	69.0 min	❌ hang→kill
07-15	29378873430	None	68.0 min	❌ hang→kill
07-14	29332511138	(step not named)	71.1 min	❌ hang→kill
07-14	29295847781	None	86.0 min	❌ hang→kill

First observed failure in window: 2026-07-14 (earliest queryable run — already failing). Last known passing run for this shard: none found in the 15-run window.

Failure Cluster

multimodal-gen 1-GPU diffusion suite hang → step timeout (no logs/artifacts uploaded)

Facts (verified observations)

• Step 9 "Run diffusion server tests (1-GPU)" has timeout-minutes: 90 (pr-test-amd.yml:751); the job definition has no job-level timeout-minutes.
• For this run, shard 1: step 9 started 2026-07-21T05:25:43Z, conclusion None; job ended 06:41:35Z; steps 10 and 20 also None. → hard kill, not a clean step failure.
• The direct job-log blob returns BlobNotFound; the run-level log archive contains only ...1gpu-sglang, 2 and ...1gpu-sglang, 3 (shard 1 absent). Run artifact count = 0. → no shard-1 log evidence exists.
• Same-run siblings: shard 2 ✅ success in 15.3 min (step 9 completed 05:22:27); shard 0 ❌ hang (99.1 min, step None); shard 3 ❌ (32 min, step 9 failure, log present); 2-GPU shard 0 ❌ hang (106 min). Different end-times across the hung jobs (06:41:35 / 06:47:03 / 08:28:49) ⇒ each was killed individually, not a single global run cancellation.
• The 1-gpu suite is partitioned into 4 shards by LPT estimated runtime (run_suite.py), so the case→shard mapping is dynamic. The suite's 1-GPU cases include very heavy image/video models (gpu_cases.py: FLUX.1-dev, FLUX.2-dev/klein, Wan-AI/Wan2.1-T2V-1.3B, hunyuanvideo-community/HunyuanVideo, Qwen/Qwen-Image, FastVideo/FastWan2.2-TI2V-5B).
• Shard 3's log (same suite, sibling partition) shows each case boots a server and downloads 30–40 GB per model (Qwen-Image 38 GiB) with warmups of ~3 min — i.e. individual cases are inherently long-running.
• The failure is present in all 14 queryable scheduled runs (07-14 → 07-21), i.e. it long predates commit 8905cbd4.

Hypothesised Causes (with confidence)

• [MEDIUM] The 1-gpu LPT partition assigned to shard 1 either contains a case that hangs or its total runtime exceeds the 90-min step budget, so step 9 is terminated before completing.
  • Supporting: step 9 never completes on shard 1 in 14/14 runs; the lighter shard 2 completes in ~15 min; each case downloads tens of GB + slow warmup; consistent ~70–91 min kill window.
  • Disconfirming: with no shard-1 log, a genuine deadlock cannot be distinguished from "partition simply too heavy for 90 min"; the kill wall-time varies (68–91 min) and for this run step 9 ran only ~76 min before the job ended at 90 min total, so the exact termination accounting (step timeout vs. runner limit) is unverified.
  • How to verify: re-run shard 1 with the diffusion step's timeout-minutes raised temporarily (e.g. 150) and --continue-on-error, or run run_suite.py --suite 1-gpu --partition-id 1 --total-partitions 4 -k "not flux_2" -v interactively on an MI325 container to see which case stalls; capture per-case timing.
• [LOW] A specific heavy video model (e.g. HunyuanVideo or FastWan2.2-TI2V-5B) deadlocks/OOMs during warmup on gfx942, stalling the whole partition.
  • Supporting: these are the largest/slowest 1-GPU cases; sibling shard 3 showed setup ERRORs on test_update_weights_from_disk.py[FLUX.2-klein-base-4B] indicating fragility in this suite.
  • Disconfirming: no shard-1 log to confirm the model; the -k "not flux_2" filter does not exclude FLUX.2 node-ids (string mismatch), so the exact excluded set is uncertain — but this is unconfirmed for shard 1.
  • How to verify: same interactive repro as above; watch rocm-smi for a hung/OOM'd worker.
• Commit attribution: not attributable to 8905cbd4 — the hang is present in every queryable run back to 07-14.

In-flight Fix Check

Searched open PRs with keywords multimodal_gen/diffusion/AMD/timeout/hang:

• In-flight fix: ❌ none found — the closest match, #29584 ("ci(amd): fix three pre-existing MI325 CI failures"), targets stage-b FP8/quantization tests, not the multimodal-gen 1-GPU diffusion hang. No open PR addresses this shard-1 timeout. (API: search/issues?q=repo:sgl-project/sglang+is:pr+is:open+<kw>)

Suggested Triage Steps

• Reproduce locally on an MI325 container: docker exec … python3 sglang/multimodal_gen/test/run_suite.py --suite 1-gpu --partition-id 1 --total-partitions 4 -k "not flux_2" -v with a raised timeout, to identify the specific stalling case and get the missing log.
• Add a per-case timeout inside run_suite.py (fail-fast one case instead of hanging the whole 90-min step) so future failures upload logs/artifacts instead of being killed with no evidence.
• Re-balance the LPT partition estimates in gpu_cases.py / STANDALONE_FILE_EST_TIMES if the root cause is cumulative slowness rather than a true hang (shard 2 finishing in 15 min vs shard 1 >90 min suggests estimate skew).
• Verify whether the -k "not flux_2" filter is actually excluding the intended FLUX.2 cases (node-id casing/dot mismatch) — if not, heavy FLUX.2 cases may be running unintentionally on the 1-GPU shards.
• Because shard 0 and 2-GPU shard 0 hang identically, treat this as a suite-wide partition/timeout issue, not a single-shard flake.

Status

• Persistence: Failing continuously for ≥7 days (every completed scheduled run 2026-07-14 → 2026-07-21); no passing shard-1 run in the queryable window.
• Scope: Same-run siblings shard 0 (1-GPU) and shard 0 (2-GPU) hang the same way; shard 2 (1-GPU) passes. Confined to the heavier multimodal-gen diffusion partitions on MI325.
• Evidence gap: No shard-1 log or artifacts exist (killed before upload), so the exact hanging case is unidentified — closing that gap (per-case timeout / local repro) is the prerequisite for root-causing.
• In-flight fix: ❌ none found in open PRs.

multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 0) — failed step(s): N/A

The background task completed and I already incorporated its output into my analysis. The report above is complete. No further action needed — the older-run history (last green partition 0 = run 29093581297 on 2026-07-10, continuous failure since 2026-07-11) was already captured in the Regression Status section.

multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 3) — failed step(s): Run diffusion server tests (1-GPU)

Commit Info

• sglang: 8905cbd4
• aiter: 9127c94a — v0.1.17.dev110+g9127c94a1 (source: dev/patched image detected, rebuild skipped; [CI-AITER-CHECK] markers at job log lines 4285–4288)

Failed Tests

Test file: python/sglang/multimodal_gen/test/single_test_file/test_update_weights_from_disk.py

Test File	Test Function	Error	Log
test_update_weights_from_disk.py	TestUpdateWeightsFromDisk::test_update_weights_from_disk_default[Qwen-Image]	assert 400 == 200 (/update_weights_from_disk rejected)	Log
test_update_weights_from_disk.py	TestUpdateWeightsFromDisk::test_update_weights_specific_modules[Qwen-Image]	AssertionError: Update failed: {'success': False, 'message': "Failed to update module 'transformer': . All modules rolled back..."}	Log
test_update_weights_from_disk.py	TestUpdateWeightsFromDisk::test_corrupted_weights_rollback[Qwen-Image]	assert (400 == 200)	Log
test_update_weights_from_disk.py	TestUpdateWeightsFromDiskWithOffload::test_update_weights_with_offload_enabled[Qwen-Image]	AssertionError: Expected 200, got 400	Log
test_update_weights_from_disk.py	TestUpdateWeightsFromDisk::test_update_weights_from_disk_default[FLUX.2-klein-base-4B] (ERROR at setup)	AssertionError: No weights dir for transformer in /tmp/sglang_perturbed_vae_nwu_0_go (fixture diffusion_server_no_offload, line 410 → _compute_checksum_from_disk line 220)	Log
test_update_weights_from_disk.py	TestUpdateWeightsFromDisk::test_update_weights_specific_modules[FLUX.2-klein-base-4B] (ERROR at setup)	same fixture assertion	Log
test_update_weights_from_disk.py	TestUpdateWeightsFromDisk::test_update_weights_rejects_invalid_requests[FLUX.2-klein-base-4B] (ERROR at setup)	same fixture assertion	Log
test_update_weights_from_disk.py	TestUpdateWeightsFromDisk::test_corrupted_weights_rollback[FLUX.2-klein-base-4B] (ERROR at setup)	same fixture assertion	Log

Summary line: 4 failed, 2 passed, 11 warnings, 4 errors in 1138.02s. Job exited with code 1.

Failure Summary

The standalone test file test_update_weights_from_disk.py (multimodal_gen diffusion weight-reload suite) failed on shard 3. For Qwen-Image, four tests fail because the server's POST /update_weights_from_disk returns HTTP 400 while updating the transformer module (message: Failed to update module 'transformer': . — empty exception text). For FLUX.2-klein-base-4B, four tests error during fixture setup (diffusion_server_no_offload): the perturbed-clone sanity check cannot find a transformer weights dir. Both symptoms are in the same test file and are independent of the commit under test.

Regression Status

Known recurring failure / Never-passed in query window. This test file has failed in every comparable completed scheduled run of pr-test-amd.yml (branch main, event=schedule) that I inspected — all 16 runs from 2026-07-13 through 2026-07-21. No green run exists in the lookback window. The file was introduced on 2026-06-27 by #28762.

Recent history of test_update_weights_from_disk.py in multimodal-gen-test-1-gpu-amd (shard 3), most recent 15 comparable completed scheduled runs (same branch, event=schedule):

Date (UTC)	Run	Test File Status	Failed / Errored	Summary
Jul 21 00:27	29790347494	❌ Failed	4 FAILED (Qwen) + 4 ERROR (FLUX.2)	4 failed, 2 passed, 4 errors
Jul 20 12:42	29743214297	❌ Failed	4 FAILED + 4 ERROR	4 failed, 2 passed, 4 errors
Jul 20 00:28	29709549202	❌ Failed	5 FAILED + 4 ERROR	5 failed, 1 passed, 4 errors
Jul 19 12:17	29686709871	❌ Failed	5 FAILED + 4 ERROR	5 failed, 1 passed, 4 errors
Jul 19 00:27	29666971143	❌ Failed	5 FAILED + 4 ERROR	5 failed, 1 passed, 4 errors
Jul 18 12:17	29644059344	❌ Failed	5 FAILED + 4 ERROR	5 failed, 1 passed, 4 errors
Jul 18 00:25	29623067212	❌ Failed	5 FAILED + 4 ERROR	5 failed, 1 passed, 4 errors
Jul 17 12:25	29580198489	❌ Failed	5 FAILED + 4 ERROR	5 failed, 1 passed, 4 errors
Jul 17 00:27	29544840981	❌ Failed	5 FAILED + 4 ERROR	5 failed, 1 passed, 4 errors
Jul 16 12:31	29498496614	❌ Failed	5 FAILED + 4 ERROR	5 failed, 1 passed, 4 errors
Jul 15 12:28	29415344552	❌ Failed	5 FAILED + 4 ERROR	5 failed, 1 passed, 4 errors
Jul 15 00:22	29378873430	❌ Failed	5 FAILED + 4 ERROR	5 failed, 1 passed, 4 errors
Jul 14 12:26	29332511138	❌ Failed	5 FAILED + 4 ERROR	5 failed, 1 passed, 4 errors
Jul 14 00:24	29295847781	❌ Failed	5 FAILED + 4 ERROR	5 failed, 1 passed, 4 errors
Jul 13 12:48	29251251853	❌ Failed	5 FAILED + 4 ERROR	5 failed, 1 passed, 4 errors
Jul 13 00:28	29215394044	❌ Failed	5 FAILED + 4 ERROR	5 failed, 1 passed, 4 errors

First observed failure: at or before 2026-07-13 (earliest run in window). Last known passing: none in window. (The 2026-07-16 00:25 run 29461421122 was cancelled and excluded.)

Failure Cluster

Two symptom clusters in one test file:

• Cluster A — "diffusion update_weights_from_disk transformer reload → HTTP 400" (Qwen-Image, 4 FAILED).
• Cluster B — "diffusion_server_no_offload fixture: missing transformer/vae weights dir in perturbed clone" (FLUX.2-klein-base-4B, 4 ERROR at setup).

Facts (verified observations)

• Qwen-Image failures all fail on assert status_code == 200 / assert 400 == 200; the server response body is {'success': False, 'message': "Failed to update module 'transformer': . All modules rolled back to original weights."} — the exception text after transformer': is empty (job log lines ~12085–12105).
• FLUX.2 failures are ERROR at setup of ..., raised in fixture diffusion_server_no_offload at test_update_weights_from_disk.py:410, calling _compute_checksum_from_disk(...) at line 220 → AssertionError: No weights dir for transformer in /tmp/sglang_perturbed_vae_nwu_0_go (job log lines ~12060–12105).
• Warnings summary shows PytestUnhandledThreadExceptionWarning from _compute_checksum_from_disk reporting no vae/transformer weights dir in the base FLUX.2 cache models--black-forest-labs--FLUX.2-klein-base-4B/snapshots/... (job log lines ~12134–12205) — the clone threads (_clone_model_with_modified_module) fail silently, so the fixture sanity check then errors.
• This is pr-test-amd.yml (NOT a scout workflow); AITER_COMMIT_OVERRIDE is empty and the image's dev aiter was preserved — aiter is not a variable here.
• Reproducibility: identical failure set in all 16 inspected runs (2026-07-13 → 2026-07-21). Deterministic, not flaky.

Hypothesised Causes (with confidence)

• [FACT] Cluster A (Qwen-Image 400): the diffusion update_weights_from_disk reload path does not reconcile the diffusers-format checkpoint layout with sglang's transformer parameters, so weights_updater.load_weights_into_model hits a bare shape AssertionError (linear.py:410 assert param_data.shape == loaded_weight.shape) and the endpoint returns HTTP 400 with an empty message. Documented by maintainer in tracking issue #31924 with server traceback (Flux2Transformer2DModel: Skipping weight update: parameter 'img_in.bias' not found after mapping to 'x_embedder.bias' → Weight update failed for module 'transformer').
  • Supporting: issue #31924's cited symptom (HTTP 400, empty message, transformer module, both Qwen-Image and FLUX.2) matches this run's log verbatim; skip-PR #31925 targets exactly this test.
  • Disconfirming: (none found) — issue explicitly distinguishes this from the InferenceMode-500 already fixed by #31263.
  • How to verify: re-run test_update_weights_from_disk.py::TestUpdateWeightsFromDisk::test_update_weights_specific_modules[Qwen-Image] on AMD MI325 and capture the server-side weights_updater/linear.py traceback.
• [MEDIUM] Cluster B (FLUX.2 ERROR): the FLUX.2-klein-base-4B checkpoint on disk does not expose top-level transformer/vae module directories in the layout _clone_model_with_modified_module / _compute_checksum_from_disk expect, so the perturbed-clone worker threads fail and the fixture sanity assert errors before the server is even exercised. This may be the same underlying diffusers-layout mismatch as Cluster A, surfacing at test-harness stage instead of server stage.
  • Supporting: thread-exception warnings report missing vae/transformer dirs in the base FLUX.2 cache snapshot, not just the /tmp clone.
  • Disconfirming: issue #31924 lists FLUX.2 cases as HTTP-400 failures, whereas this run shows them as setup errors — the FLUX.2 path may not reach the server here, so it is not certain to be the identical code path as Cluster A.
  • How to verify: inspect the on-disk layout of models--black-forest-labs--FLUX.2-klein-base-4B on the runner and confirm whether transformer/vae subdirs with safetensors exist.

In-flight Fix Check

• In-flight fix: ✅ #31925 "[AMD] Skip test_update_weights_from_disk on ROCm pending reload fix" (open, created 2026-07-21) — skips this test file on the AMD lane (aligning with NVIDIA, which already does not run this standalone file) until the real fix lands. Tracking issue: #31924 (open). Chase reviewers; do NOT open a duplicate.
• Searched is:pr is:open update_weights_from_disk (18 hits) and diffusion+update+weights (35 hits); #31925 is the matching skip PR and #31924 the root-cause tracker.

Suggested Triage Steps

• Merge/approve the existing skip PR #31925 to unblock the multimodal-gen-test-1-gpu-amd shard-3 signal on AMD while the real fix is developed under #31924.
• For the real fix (Cluster A): investigate weights_updater.load_weights_into_model param-name mapping (param_names_mapping) for diffusers→sglang transformer params; the bare assert param_data.shape == loaded_weight.shape at linear.py:410 should surface a descriptive error rather than an empty HTTP 400.
• For Cluster B: verify the FLUX.2-klein-base-4B on-disk module layout on the runner; if transformer/vae subdirs are absent, the fixture's clone/checksum assumptions (_clone_model_with_modified_module, _compute_checksum_from_disk) need updating for that checkpoint format.

Status

• Persistence: Failing in every completed scheduled run inspected from 2026-07-13 through 2026-07-21 (16 runs); deterministic, never-passed in the query window.
• Scope: multimodal-gen-test-1-gpu-amd shard 3 of pr-test-amd.yml on AMD only (standalone file; NVIDIA does not run it). Blocks AMD CI signal for diffusion update_weights_from_disk.
• In-flight fix: ✅ #31925 (skip on ROCm, open since 2026-07-21); root cause tracked in #31924.

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-07-22 19:32 UTC)

[amd-bot]

pr-test-amd.yml — Per-Job Analysis (continued 2/5)

stage-b-test-1-gpu-large-amd (linux-mi325-1gpu-sglang, 0) — failed step(s): Run test

The analysis is complete. The background download task that finished was the historical job logs, which I already used to build the regression history table.

To summarize the finding: the stage-b-test-1-gpu-large-amd job failed on a single flaky/marginal LoRA performance test — test_online_lora_latency_with_concurrent_adapter_updates measured 15641 ms / 11156 ms median E2E latency against a 4000 ms threshold. All 9 other suites in the job passed. The full per-job report was already emitted above.

multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 0) — failed step(s): N/A

Commit Info

• sglang: e4eea7ce
• aiter: N/A — the job log blob is unavailable (BlobNotFound), so no [CI-AITER-CHECK] markers could be read.

Failed Tests

Test File	Test Function	Error	Log
python/sglang/multimodal_gen/test/** (suite 1-gpu, partition 0)	cannot be determined	Job hard-killed while step 9 (Run diffusion server tests (1-GPU)) was still running; no log uploaded (BlobNotFound)	Job

Non-test failure: This is not a clean pytest failure. The Run diffusion server tests (1-GPU) step never completed (stuck in_progress), the always()-Upload diffusion failure artifacts step never ran (pending), and no job log was persisted — the classic signature of a hang / hard-kill (step 90-min timeout, GPU hang, or OOM losing the runner). The specific test/model that hung cannot be pinned because the log was never uploaded.

Failure Summary

The multimodal-gen-test-1-gpu-amd partition-0 job ran for ~122 min (start 18:39:44Z → 20:41:54Z, step-9 timeout-minutes: 90) and was terminated with the diffusion-server test step still in_progress. GitHub marked the job failure but never finalized step-9 or uploaded its log. The suite uses dynamic longest-processing-time partitioning, so partition 0 gets the single heaviest case — consistent with a timeout/hang on the largest model.

Regression Status

Never-passed (no green run found) — partition 0 of this job has failed in every completed scheduled run in the queryable window (2026-07-12 → 2026-07-21, 15+ runs). No passing run exists to bound a regression. Two distinct failure modes are visible in the window: a step-9 hang (no log) from ~2026-07-15 onward, and an earlier-stage failure (no step 9) on 2026-07-12→14.

Recent history of partition 0 of multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 0) (completed scheduled runs, branch=main, event=schedule; the current run 29830494225 is itself still in_progress at snapshot but this job is completed/failure):

Date	Run	Test File Status	Signature
Jul 21 (this)	29830494225	❌ Failed	step9 in_progress, upload pending, no log (hang)
Jul 21	29790347494	❌ Failed	hang, no log
Jul 20	29743214297	❌ Failed	hang, no log
Jul 20	29709549202	❌ Failed	hang, no log
Jul 19	29686709871	❌ Failed	failed before step 9
Jul 19	29666971143	❌ Failed	hang, no log
Jul 18	29644059344	❌ Failed	hang, no log
Jul 18	29623067212	❌ Failed	hang, no log
Jul 17	29580198489	❌ Failed	(log unavailable)
Jul 17	29544840981	❌ Failed	(log unavailable)
Jul 16	29498496614	❌ Failed	hang, no log
Jul 15	29415344552	❌ Failed	step9 pending (never started)
Jul 15	29378873430	❌ Failed	hang, no log
Jul 14	29332511138	❌ Failed	failed before step 9
Jul 13-14	29295847781, 29251251853	❌ Failed	failed before step 9

(No green run for partition 0 anywhere in the 15+ run lookback. The 2026-07-16 cancelled run 29461421122 is excluded.)

Failure Cluster

Diffusion 1-GPU server test hang / hard-kill on AMD (partition 0, no log persisted)

Facts (verified observations)

• The current job (id 88633865544) log endpoint returns BlobNotFound; the run zip contains no partition-0 entry. No partition-0 log exists to analyze — verified across the current run and all sampled historical runs.
• Current job: started 2026-07-21T18:39:44Z, completed 2026-07-21T20:41:54Z (~122 min). Step 9 Run diffusion server tests (1-GPU) = in_progress; step 10 Upload diffusion failure artifacts (an always() step) = pending. Verified via jobs API.
• The same signature (step9 in_progress/pending, upload step never runs, ~74-108 min duration, no log) recurs on partitions 0 and 1 in 29790347494, 29743214297, 29709549202, 29666971143, 29644059344, 29623067212, 29498496614, 29378873430.
• In the same runs, partition 2 passes and partition 3 fails cleanly (log present) on python/sglang/multimodal_gen/test/single_test_file/test_update_weights_from_disk.py — AssertionError: No weights dir for transformer/vae ... (Qwen-Image), 4 failed, 2 passed ... in 1138.02s. This is a different cluster from the partition-0 hang.
• Suite partitioning is dynamic longest-processing-time (partition_items_by_lpt, python/sglang/multimodal_gen/test/run_suite.py:128), so partition 0 receives the single heaviest case.
• Step 9 has timeout-minutes: 90 (.github/workflows/pr-test-amd.yml:751); observed durations cluster around that value.

Hypothesised Causes (with confidence)

• [MEDIUM] The partition-0 diffusion server test hangs (server startup, model download/load, or denoise loop stall) and is killed by the 90-min step timeout or a runner loss, so no log is uploaded.
  • Supporting: step-9 stuck in_progress + always()-upload never runs + zero log blob is exactly the pattern of a force-killed job; durations sit at/around the 90-min step timeout; reproduces on partitions 0 and 1 across ≥8 runs.
  • Disconfirming: without the log, the hang location (download vs load vs inference) is unverified; a clean 90-min pytest timeout would normally still let the always() upload step run, so the runner may be dying harder (GPU hang/OOM) than a simple step timeout.
  • How to verify: re-run partition 0 with a lower per-test timeout + py-spy dump/server-log capture on the runner; check dmesg/rocm-smi for GPU resets or OOM; inspect the CI container for a stuck python3 run_suite.py.
• [LOW] The heaviest model assigned to partition 0 (e.g. HunyuanVideo / FLUX.1-dev) exceeds the AMD MI325 1-GPU time/VRAM budget.
  • Supporting: LPT partitioning puts the longest case in partition 0; the step timeout is the binding constraint.
  • Disconfirming: which model lands in partition 0 is not verifiable without the log or the runtime-estimate plan.
  • How to verify: print the resolved partition plan (case→partition mapping) at suite start and correlate with per-model runtime.
• [LOW] Environment/dependency breakage specific to AMD diffusion (the 2026-07-12→14 "no step 9" runs failed even earlier).
  • Supporting: earlier window runs never reached step 9, suggesting install/setup churn preceded the current hang mode.
  • Disconfirming: those early failures are a distinct mode from the current hang and predate the queryable logs.
  • How to verify: inspect the Install dependencies step logs of the 2026-07-14 runs.

In-flight Fix Check

• In-flight fix (partition-0 hang): ❌ none found — searched open PRs for diffusion+AMD and multimodal_gen; no open PR addresses a partition-0 timeout/hang or the missing 1-GPU logs.
• In-flight fix (partition-3 clean failure, different cluster): ✅ #31925 "[AMD] Skip test_update_weights_from_disk on ROCm pending reload fix (#31924)" — covers the test_update_weights_from_disk.py failures seen in partition 3, NOT the partition-0 hang under analysis.
• Related but non-matching: #31899 (unit-test msgpack ModuleNotFoundError), #31846 (quarantine nested unit failures) — neither targets the 1-GPU server hang.

Suggested Triage Steps

• Reproduce partition 0 in the ci_sglang container: python3 sglang/multimodal_gen/test/run_suite.py --suite 1-gpu --partition-id 0 --total-partitions 4 -k "not flux_2", and log the resolved partition plan to learn which model hangs.
• Add server-side log capture / a shorter per-test timeout so a hung partition uploads a diagnosable log instead of dying at the 90-min step boundary (the always() upload step currently never runs).
• On the runner during a repro, check rocm-smi and dmesg for GPU resets/OOM and py-spy dump the stuck run_suite.py to locate the hang (download vs model load vs denoise).
• Track the partition-3 test_update_weights_from_disk cluster separately under #31925; do not conflate it with the partition-0 hang.

Status (factual)

• Persistence: Partition 0 has failed in every completed scheduled run for ≥10 days (2026-07-12 → 2026-07-21); the step-9 hang mode has been continuous since ~2026-07-15. No green run in the queryable window.
• Scope: Partitions 0 and 1 of multimodal-gen-test-1-gpu-amd share the hang/no-log mode; partition 2 passes and partition 3 fails on a different (weight-update) cluster. Limited to the AMD MI325 1-GPU diffusion suite in pr-test-amd.yml.
• Blocked work: Blocks AMD 1-GPU diffusion (multimodal_gen) CI signal for partitions 0/1; these partitions produce no diagnosable log.
• In-flight fix: ❌ none for the partition-0 hang; #31925 addresses only the separate partition-3 weight-update failures.

multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 1) — failed step(s): N/A

Commit Info

• sglang: e4eea7ce
• aiter: N/A — the job log blob was never uploaded (BlobNotFound), so no [CI-AITER-CHECK] markers are readable. The step env shows AITER_COMMIT_OVERRIDE= (empty), i.e. no override was requested.

Failed Tests

Test File	Test Function	Error	Log
N/A (log unavailable)	N/A	Non-test failure: job terminated during step 9 "Run diffusion server tests (1-GPU)"; step frozen in_progress, steps 10 & 20 never ran, log blob not uploaded	Job

This is a non-test / infrastructure failure, not a pytest assertion. The specific failing test file/function in partition 1 cannot be determined because the job log was never uploaded (see Facts).

Failure Summary

The job 88633865546 (matrix part 1 of 4) was terminated in the middle of step 9 "Run diffusion server tests (1-GPU)". Step 9 started 19:03:48Z and the job ended 20:26:47Z (~1h23m), i.e. below the step's configured timeout-minutes: 90. The step is frozen in_progress, the later steps never executed, and the job's log blob returns BlobNotFound — while sibling shard part 3 in the same run uploaded its 2.2 MB log normally. The runner/container was killed before finalizing.

Regression Status

Known recurring failure (not new today). Partitions 0 and 1 of multimodal-gen-test-1-gpu-amd have failed with this exact "killed mid-step-9, no log" signature in every completed scheduled pr-test-amd.yml run I checked; partition 2 passes; partition 3 fails with real pytest assertions (a different test file). No passing run for part 1 exists in the query window.

History of multimodal-gen-test-1-gpu-amd (part 1) (most recent completed event=schedule, branch main). Test-file-level status is not observable for part 1 because logs are never uploaded — the "Job Status" column reflects the step-9-killed signature:

Date (UTC)	Run	Job	Job Status	Step 9 state	Log
Jul 21 12:32	29830494225	part 1	❌ Failed	frozen in_progress, ~83m (<90m), no log	BlobNotFound
Jul 21 00:27	29790347494	part 1	❌ Failed	frozen in_progress, ~76m, no log	BlobNotFound
Jul 20 12:42	29743214297	part 1	❌ Failed	frozen in_progress, no log	BlobNotFound
Jul 20 00:28	29709549202	part 1	❌ Failed	frozen in_progress, ~64m, no log	BlobNotFound
Jul 19 12:17	29686709871	part 1	❌ Failed	killed (steps n/a), no log	BlobNotFound
Jul 19 00:27	29666971143	part 1	❌ Failed	frozen in_progress, ~66m, no log	BlobNotFound
Jul 18 12:17	29644059344	part 1	❌ Failed	frozen in_progress, ~65m, no log	BlobNotFound

First observed failure (in window): ≥ Jul 18 2026; no passing run found for part 1 in the last 7 completed scheduled runs → Never-passed within the lookback window (Jul 18–21). Likely predates the window.

Failure Cluster

multimodal-gen 1-GPU diffusion test job killed mid-run with logs not uploaded (partitions 0 & 1)

Facts (verified observations)

• Job conclusion is failure; step 9 "Run diffusion server tests (1-GPU)" is frozen status=in_progress, and steps 10 ("Upload diffusion failure artifacts") and 20 ("Post Checkout") never started (verified via job steps API).
• Step 9 started 2026-07-21T19:03:48Z; job completed_at is 2026-07-21T20:26:47Z → step 9 ran ~1h23m, under its timeout-minutes: 90 (workflow pr-test-amd.yml:751).
• The job log endpoint returns BlobNotFound (HTTP 404). Sibling shard part 3 (88633865564) in the same run returned a full 2.2 MB log (HTTP 200) — so the missing log is specific to this job being hard-killed, not a run-wide issue.
• Sibling results this run: part 0 ❌ (same killed-mid-step-9 signature, no log), part 1 ❌ (this job), part 2 ✅ (step 9 completed success, ~5m), part 3 ❌ (step 9 completed failure — real pytest failures in sglang/multimodal_gen/test/single_test_file/test_update_weights_from_disk.py for Qwen-Image / FLUX.2-klein-base-4B).
• Step-9 duration for part 1 across history varies 64–83 min, always < 90 min, always with BlobNotFound — inconsistent with hitting the fixed 90-min step timeout.
• The step runs run_suite.py --suite 1-gpu --partition-id <part> --total-partitions 4 -k "not flux_2" (T2V/T2I/I2V/LoRA models: Wan2.1-T2V-1.3B, HunyuanVideo, Qwen-Image, FLUX.1). Which specific models land in partition 1 is not observable from the API.

Hypothesised Causes (with confidence)

• [LOW] Resource exhaustion (host RAM / GPU VRAM OOM) or a GPU/driver hang during a heavy diffusion model (likely a video model — HunyuanVideo / Wan2.1-T2V) in partitions 0 & 1 causes the container or self-hosted runner to be killed before step 9 finishes, so logs are never flushed.
  • Supporting: failure is confined to partitions 0 & 1 (partition-content-specific), never part 2; termination time varies (64–83 min, not a fixed timeout); logs never upload (consistent with a hard/ungraceful kill rather than a graceful step cancel).
  • Disconfirming: no log, dmesg, or rocm-smi output is available to confirm OOM/hang; the exact partition→model mapping is unverified.
  • How to verify: re-run part 1 with reduced scope / a shorter inner timeout, or add mid-step diagnostics (rocm-smi, dmesg, free -h) that flush to an artifact before the process dies; inspect the runner host's system logs around the kill time.
• [LOW] Diffusion worker self-termination — in-flight PR #31361 "[diffusion] fix: don't self-kill diffusion worker when PID 1 is the real parent" targets a scenario where the diffusion worker kills itself abnormally, which would match "process dies, no clean logs".
  • Supporting: symptom is an abnormal process death mid-run; a self-kill would prevent normal step completion.
  • Disconfirming: no log evidence tying this run to that code path; #31361 may address a different container/PID topology; cannot confirm it exercises partition 1.
  • How to verify: cherry-pick #31361 onto a branch and re-run multimodal-gen-test-1-gpu-amd parts 0 & 1.
• Note: the configured 90-min step timeout is not the cause — every observed termination is under 90 min.

No hypothesis reaches MEDIUM: the authoritative evidence (step-9 log) was never uploaded.

In-flight Fix Check

• In-flight fix: ⚠️ [#31361](https://github.com/sgl-project/sglang/pull/31361) "[diffusion] fix: don't self-kill diffusion worker when PID 1 is the real parent" — plausibly related to abnormal worker death; [LOW] confidence, unverified against this job. (Searched open PRs with keywords diffusion, multimodal-gen, hang.)
• In-flight fix: ❌ none found that directly targets the "part 0/1 1-GPU diffusion job killed with no logs" symptom. Related but different-scope PRs: #31899 (msgpack dep for the unit test, which passed here) and #31846 (AMD CI quarantine of nested unit failures).

Suggested Triage Steps

• Add flush-before-death diagnostics to step 9 (periodic rocm-smi --showmeminfo vram, free -h, dmesg tail written to SGLANG_DIFFUSION_ARTIFACT_DIR) so the next occurrence leaves evidence even when the runner is killed and the step log is lost.
• Determine the partition→test mapping: run python3 sglang/multimodal_gen/test/run_suite.py --suite 1-gpu --partition-id 1 --total-partitions 4 -k "not flux_2" --collect-only to identify exactly which models/tests land in part 1 (the suspected heavy video tests).
• Reproduce part 1 in the ci_sglang container on an MI325 runner with rocm-smi monitoring; check for VRAM/host-RAM OOM or a GPU hang around the 60–80 min mark.
• Evaluate cherry-picking #31361 and re-running parts 0 & 1 to test the diffusion-worker-self-kill hypothesis.
• Investigate part 3's separate, log-bearing failure (test_update_weights_from_disk.py) independently — it is a distinct cluster (real pytest failures), not this one.

Status

• Persistence: Recurring for ≥ 4 days (every completed scheduled run Jul 18 → Jul 21 2026); no passing run for part 1 in the 7-run lookback window.
• Scope: Partitions 0 and 1 of multimodal-gen-test-1-gpu-amd (2 of 4 1-GPU shards), consistently, on pr-test-amd.yml scheduled runs. Part 2 passes; part 3 fails separately with a different (log-bearing) cause.
• Evidence gap: Root-cause diagnosis is blocked because the step-9 log is never uploaded (hard kill). This needs flush-before-death diagnostics or a local repro before a cause can be confirmed.
• In-flight fix: ⚠️ #31361 possibly related ([LOW]); no confirmed fix for this symptom.

multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 3) — failed step(s): Run diffusion server tests (1-GPU)

The analysis is already complete and the report delivered. This is another leftover background historical-log check whose results I already incorporated (Jul 15-16 runs confirmed the test was failing with the same 5-failed/1-passed signature). No further action needed.

stage-c-test-large-8-gpu-amd-mi35x (linux-mi35x-gpu-8, 0) — failed step(s): Run test — ⚠️ analysis failed

Analysis did not complete. The per-job agent produced no usable output (likely a timeout, log download failure, or subprocess crash). Manual triage required.

• Run: https://github.com/sgl-project/sglang/actions/runs/29830494225
• Job log: https://github.com/sgl-project/sglang/actions/runs/29830494225/job/88633866385
• Job ID: 88633866385
• Failed step(s): Run test
• Started (UTC): 2026-07-21 12:43

stage-b-test-large-8-gpu-mi35x-disaggregation-amd (linux-mi35x-gpu-8.fabric) — failed step(s): Run test_disaggregation

That was the download of the last two historical job logs (87217685643 and 86820131755), which I had already accounted for in the report — both are non-KeyError failures (the exit-134 abort cluster and a tiny 28 KB infra failure, respectively), consistent with what's already reflected in the regression history table.

No changes needed. The Job Failure Analysis report is final and complete.

stage-b-test-1-gpu-small-amd-mi35x (linux-mi35x-gpu-1) — failed step(s): Run test

Commit Info

• sglang: 8ae0eb83
• aiter: 9127c94a — v0.1.17.dev110+g9127c94a1 (source: dev image detected via [CI-AITER-CHECK], preserved; matches Dockerfile default)

Failed Tests

Test File	Test Function	Error	Log
test/registered/quant/test_quark_mxfp4.py	TestFP8ToMXFP4MOETP1.test_gsm8k	AssertionError: 0.92 not greater than 0.92	Log

Other 4 test files in the job passed (Test Summary: 4/5 passed). Job exited with code 255.

Failure Summary

The GSM8K accuracy gate for the online-MXFP4-requantized Qwen/Qwen3-30B-A3B-Instruct-2507-FP8 MoE model (TP=1) failed. The eval scored 0.912 on the first attempt and 0.920 on the single retry; the test asserts assertGreater(accuracy, 0.92) (test_quark_mxfp4.py:171, threshold set at test_quark_mxfp4.py:251), so both attempts fall at or below the bar. All other tests in this file (TestFP8ToMXFP4DenseTP1, memory tests) passed.

Regression Status

Flaky test (threshold boundary) — this test was added only ~15 h before this run.

Recent history of test/registered/quant/test_quark_mxfp4.py (TestFP8ToMXFP4MOETP1.test_gsm8k) in job stage-b-test-1-gpu-small-amd-mi35x (comparable completed scheduled runs, branch=main, event=schedule):

Date (UTC)	Run	Job	Test File Status	Failed Function	Error / Accuracy
Jul 22 00:26	29880220845	job	❌ Failed	TestFP8ToMXFP4MOETP1.test_gsm8k	0.912→retry 0.920 (≤ 0.92)
Jul 21 12:32	29830494225	job	✅ Passed	—	accuracy 0.922 (> 0.92)
Jul 21 00:27 and earlier	29790347494	job	⏭️ Not present	—	test added dcd9014f 2026-07-21 09:50 UTC (PR #28291); this run predates it

First observed failure: 2026-07-22 00:26 UTC. Last known passing: 2026-07-21 12:32 UTC. The test has existed for only 2 comparable scheduled runs (1 pass, 1 fail).

Failure Cluster

GSM8K accuracy just below strict threshold (MXFP4 MoE requant)

Facts (verified observations)

• First attempt: Accuracy: 0.912 → AssertionError: 0.912 not greater than 0.92 (main.log:7929). Retry: Accuracy: 0.920 / metrics={'accuracy': 0.92, ...} → AssertionError: 0.92 not greater than 0.92 (main.log:9379); retry() exceed maximum number of retries.
• Prior scheduled run 29830494225 ran the identical test on the same aiter dev image and scored accuracy: 0.922 → passed on first attempt (verified from that job's log; Test Summary: 5/5 passed).
• The threshold 0.92 (assertGreater, i.e. strictly greater, zero margin) and the file were introduced by PR #28291 (dcd9014f, merged 2026-07-21 09:50 UTC). The code comment cites reference accuracy ~0.948, but observed CI accuracy is 0.912–0.922 — right at the gate.
• aiter version is identical (v0.1.17.dev110+g9127c94a1) in the passing and failing runs ([CI-AITER-CHECK] markers), so aiter is not a variable here.
• The Health check failed … detokenizer for last 20 seconds line (main.log:4679) occurred during TestFP8ToMXFP4DenseTP1 warmup while aiter JIT-built mha_batch_prefill… (25.7 s > 20 s heartbeat); the server recovered (health_generate … 200 OK) and that test passed. Unrelated to the MoE accuracy failure.
• 18 commits in the window e4eea7ce..8ae0eb83 (last-pass → this-fail); only one touches MoE/quant code paths (927979e1, PR #31838).

Hypothesised Causes (with confidence)

• [MEDIUM] Threshold-boundary flake: the assertGreater(accuracy, 0.92) gate has effectively zero margin against this model+quant path's run-to-run GSM8K variance (500 questions), which oscillates 0.912–0.922 across the only two runs the test has existed.
  • Supporting: pass run = 0.922, fail run = 0.912/0.920; the strict > comparison makes even 0.920 fail. Reference in the code comment (~0.948) is far above observed CI accuracy, indicating the threshold was tuned to the observed floor with no headroom.
  • Disconfirming: only 2 data points exist (test is <1 day old); a genuine accuracy regression between the two runs cannot be fully excluded from a single delta.
  • How to verify: re-run this job 5–10× at fixed SHA 8ae0eb83; if accuracy centers ~0.92 ± 0.01, it's a threshold flake, not a regression.
• [LOW] Commit 927979e1 (PR #31838, MoE top-k pad-row masking) lands in the pass→fail window and edits layers/moe/topk.py.
  • Supporting: only MoE-touching commit in the delta; the failing test is a MoE model.
  • Disconfirming: the fix is scoped to custom_routing_function under DP attention; this test runs TP=1 with no DP attention and no custom routing function, so the changed branch is not exercised. Accuracy moved only ~0.002 (0.922→0.920), consistent with sampling noise, not a routing bug.
  • How to verify: git revert 927979e1 on a branch and re-run TestFP8ToMXFP4MOETP1.test_gsm8k on MI35x; expect no change.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for test_quark_mxfp4, MXFP4 gsm8k, quark_mxfp4 accuracy (GET /search/issues?q=repo:sgl-project/sglang+is:pr+is:open+<kw>); results are general MXFP4 feature PRs (#29328, #31054, #28734), none addressing this accuracy gate/threshold.

Suggested Triage Steps

• Re-run job 88799241604 several times at 8ae0eb83 to measure the accuracy distribution for TestFP8ToMXFP4MOETP1.test_gsm8k; if it centers at/below 0.92, the gate at test_quark_mxfp4.py:251 needs a margin (loosen threshold or raise num_questions to reduce variance) — ping PR #28291 author.
• If the distribution clearly regressed vs the 0.922 baseline, bisect e4eea7ce..8ae0eb83 (18 commits), starting by reverting 927979e1 on a branch and re-running the MoE test.

Status

• Persistence: 1 failure since introduction (2026-07-22 00:26 UTC); the immediately prior scheduled run passed (0.922). Test is ~15 h old.
• Scope: single test function (TestFP8ToMXFP4MOETP1.test_gsm8k) in job stage-b-test-1-gpu-small-amd-mi35x; the other 3 GSM8K variants in the same file (Dense TP1/TP2, larger-margin thresholds) passed.
• Blocked work: gates online-MXFP4 requant MoE accuracy signal on AMD MI35x in pr-test-amd.yml.
• In-flight fix: ❌ none found in open PRs.

multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 0) — failed step(s): N/A

Commit Info

• sglang: e4eea7ce
• aiter: 9127c94a (v0.1.17.dev110+g9127c94a1, source: dev image preserved — [CI-AITER-CHECK] Dev/patched version detected … → skipping rebuild, no override)

Failed Tests

Test File	Test Function	Error	Log
test/registered/quant/test_quark_mxfp4.py	TestFP8ToMXFP4MOETP1.test_gsm8k	AssertionError: 0.92 not greater than 0.92 → Exception: retry() exceed maximum number of retries → exit 255	Log

The other 4 files in suite stage-b-test-1-gpu-small-amd-mi35x (test_dsa_skip_logits_clean.py, test_verify_splitkv.py, test_mamba_state_scatter_triton.py, test_deepseek_mla_dispatch.py) all passed. Within test_quark_mxfp4.py itself: Ran 5 tests … FAILED (errors=1, skipped=3) — the dense FP8→MXFP4 8B test passed (Accuracy 0.892 vs threshold 0.868); only the MoE case failed.

Failure Summary

A single GSM8K accuracy assertion failed. TestFP8ToMXFP4MOETP1.test_gsm8k serves Qwen/Qwen3-30B-A3B-Instruct-2507-FP8 re-quantized online to quark_mxfp4 and asserts assertGreater(accuracy, 0.92). The MoE model scored 0.912 on the first attempt and 0.920 on the retry — both at or below the strict > 0.92 bar — so the retry helper exhausted its one retry and the suite exited 255. All non-quant work in the job completed successfully; the CUDA-graph capture and 500-question benchmark ran to completion each attempt (this is not a hang/timeout despite the step showing in_progress in the stale job-steps API).

Regression Status

New regression / effectively Never-passed — this test did not exist in scheduled history until yesterday.

The failing test class TestFP8ToMXFP4MOETP1 and its accuracy_threshold=0.92 were introduced by dcd9014f "[AMD][MXFP4] Reland 'Online MXFP4 quantization 2/N …' (#28291)", which merged 2026-07-21 09:50 UTC. Verified via git show 8905cbd42f:…test_quark_mxfp4.py that the class/threshold were absent at the prior scheduled run's head.

Recent history of test/registered/quant/test_quark_mxfp4.py (TestFP8ToMXFP4MOETP1.test_gsm8k) in job multimodal-gen-test-2-gpu-amd (event=schedule, branch=main, completed runs):

Date (UTC)	Run	Head SHA	Test File Status	Note
Jul 21 12:32	29830494225	e4eea7ce	❌ Failed	first scheduled run after reland dcd9014f (09:50 UTC)
Jul 21 00:27	29790347494	8905cbd4	⬜ Not present	TestFP8ToMXFP4MOETP1/0.92 threshold not yet in tree
Jul 20 12:42 → Jul 14	29743214297 … 29295847781	(earlier)	⬜ Not present	test predates reland; no comparable data

First observed failure: 2026-07-21 12:32 UTC (this run). No green run of this test exists in scheduled history — it fails on its first inclusion.

Failure Cluster

GSM8K accuracy below threshold — MXFP4-requantized MoE (Qwen3-30B-A3B FP8→quark_mxfp4)

Facts (verified observations)

• Verbatim: AssertionError: 0.912 not greater than 0.92 (attempt 0, retry() failed once … Will delay 1.59s and retry), then AssertionError: 0.92 not greater than 0.92 (retry) → Exception: retry() exceed maximum number of retries. → ##[error]Process completed with exit code 255.
• Assertion site: test/registered/quant/test_quark_mxfp4.py:171 (self.assertGreater(metrics["accuracy"], accuracy_threshold)), called from test_quark_mxfp4.py:251 (self._test_gsm8k(accuracy_threshold=0.92)).
• Model/config: Qwen/Qwen3-30B-A3B-Instruct-2507-FP8, quantization='quark_mxfp4', tp_size=1, attention_backend='aiter', GSM8K 500 questions / 8-shot. Runtime warned: "Using online MXFP4 quantization … may degrade prediction quality."
• Both attempts produced valid output (Invalid: 0.000); accuracy 0.912 / 0.920 — a ~0.01 spread, straddling the boundary. Code comment states reference accuracy ~0.948.
• During MoE capture, aiter GEMMs logged not found tuned config in /tmp/aiter_configs/bf16_tuned_gemm.csv, will use default config! using torch solution:0 (default fallback path).
• aiter is the standard dev-image build (not an override); the sibling dense 8B MXFP4 test passed (0.892 vs 0.868).
• This is pr-test-amd.yml (not amd-aiter-scout.yml), so no Baseline A/B / Failure Origin section applies.

Hypothesised Causes (with confidence)

• [MEDIUM] The 0.92 threshold set in dcd9014f (#28291) is too tight for the real MXFP4-requant accuracy of this MoE model on the MI35x runner.
  • Supporting: threshold and test authored in the same reland PR (never validated across runs); observed 0.912/0.920 both ≤ 0.92 with assertGreater being strict; GSM8K@500 carries ~±0.01–0.02 sampling variance; online MXFP4 requant is explicitly warned to degrade quality; the sibling BF16 MoE variant uses a looser 0.89.
  • Disconfirming: only one run of this test exists, so a genuine numerics regression cannot be excluded from a single data point.
  • How to verify: re-run the job 2–3× and record the accuracy distribution; if it clusters at 0.91–0.93, it's a threshold-calibration issue.
• [LOW] MXFP4 MoE requant numerics lower accuracy (aiter MoE GEMM default fallback using torch solution:0).
  • Supporting: MoE GEMMs used the untuned default config path; MoE is the only failing case (dense passed).
  • Disconfirming: standard dev-image aiter (no override); no reference A/B showing the MoE path is numerically wrong rather than just marginal; 0.920 is within noise of 0.92.
  • How to verify: compare GSM8K accuracy of this model with quark_mxfp4 vs bf16 baseline offline; inspect the fused MXFP4 MoE quant path.

In-flight Fix Check

In-flight fix: ❌ none found — searched open PRs for quark mxfp4 (33 results — all feature/followups such as #29328 4/N, #31054, none targeting this threshold) and test_quark_mxfp4 (7 results, none addressing the TestFP8ToMXFP4MOETP1 accuracy bar).

Suggested Triage Steps

• Re-run multimodal-gen-test-2-gpu-amd 2–3× to measure the GSM8K accuracy spread of TestFP8ToMXFP4MOETP1.test_gsm8k; if it lands consistently in 0.91–0.93, loosen the threshold (e.g. to 0.89, matching the BF16 MoE variant) on a branch and re-run.
• If accuracy is consistently well below 0.92, treat as an MXFP4 MoE requant numerics issue: A/B the quark_mxfp4 MoE path against a bf16 reference and inspect the aiter fused MXFP4 MoE / default-GEMM fallback.
• Flag the borderline threshold to the reland author of #28291 (dcd9014f), since it was set in the same PR without cross-run validation.

Status

• Persistence: First and only occurrence — the test entered scheduled CI in this run (29830494225, 2026-07-21 12:32 UTC); no prior comparable data (test introduced 2026-07-21 09:50 UTC by the reland).
• Scope: 1 test function in 1 job of pr-test-amd.yml; the 8B dense MXFP4 variant is unaffected.
• Blocked work: Blocks a green stage-b-test-1-gpu-small-amd-mi35x suite signal on MI35x; failure is isolated to MXFP4 MoE accuracy.
• In-flight fix: ❌ none found — needs threshold recalibration or a numerics investigation.

stage-b-test-large-8-gpu-mi35x-disaggregation-amd (linux-mi35x-gpu-8.fabric) — failed step(s): Run test_disaggregation

Commit Info

• sglang: 8ae0eb83
• aiter: N/A (no [CI-AITER-CHECK] markers in this job log; the image ships aiter under /sgl-workspace/aiter, version not stamped in the log)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/disaggregation/test_disaggregation_pp.py	TestDisaggregationPrefillPPDynamicChunkAccuracy.test_gsm8k (line 198)	KeyError: 'answer' → retry() exceed maximum number of retries.	Log

The other 3 disaggregation test files in this job passed (test_disaggregation_basic.py, test_mori_transfer_engine_e2e.py, test_nixl_transfer_engine_e2e.py) → job summary 3/4 passed, process exit code 255.

Failure Summary

The 8-GPU MI35x disaggregation job failed only on test_disaggregation_pp.py. Within that file, the TestDisaggregationPrefillPPDynamicChunkAccuracy.test_gsm8k variant (Prefill TP2/PP2 with --enable-dynamic-chunking + --disable-overlap-schedule, decode TP2, Qwen3-8B) errored. The gsm8k progress bar reached 100% (all 200 requests returned), but at least one returned SGL state was missing the answer variable, raising KeyError: 'answer' in few_shot_gsm8k.run_eval at python/sglang/test/few_shot_gsm8k.py:123; the single retry hit the same error and the test errored out.

Regression Status

Known recurring intermittent (flaky) failure of test_disaggregation_pp.py with KeyError: 'answer', present since at least 2026-07-18. The failing file recurs across runs; the specific failing function rotates between the three test_gsm8k variants (Prefill PP line 104, DynamicChunk PP line 198, Decode PP line 293). For the exact DynamicChunk variant (line 198), this is the first failure in the queried window.

Recent history of test_disaggregation_pp.py in job stage-b-test-large-8-gpu-mi35x-disaggregation-amd (most recent comparable completed scheduled runs: branch=main, event=schedule, completed only):

Date (UTC)	Run	Test File Status	Failed Function	Error
Jul 22 00:27	29880220845	❌ Failed	TestDisaggregationPrefillPPDynamicChunkAccuracy.test_gsm8k (L198)	KeyError: 'answer'
Jul 21 12:32	29830494225	❌ Failed	TestDisaggregationDecodePPAccuracy.test_gsm8k (L293)	KeyError: 'answer'
Jul 21 00:27	29790347494	✅ Passed	—	—
Jul 20 12:42	29743214297	✅ Passed	—	—
Jul 20 00:28	29709549202	✅ Passed	—	—
Jul 19 12:17	29686709871	✅ Passed	—	—
Jul 19 00:27	29666971143	❌ Failed	TestDisaggregationPrefillPPAccuracy (L104) + TestDisaggregationDecodePPAccuracy (L293)	KeyError: 'answer'
Jul 18 12:17	29644059344	✅ Passed	—	—
Jul 18 00:25	29623067212	❌ Failed	TestDisaggregationDecodePPAccuracy.test_gsm8k (L293)	KeyError: 'answer'

Failing ~4/9 recent completed scheduled runs. First observed failing in this window: 2026-07-18 00:25 UTC. Last known passing: 2026-07-21 00:27 UTC.

Failure Cluster

KeyError: 'answer' in gsm8k eval on PD+PP disaggregation (empty/dropped generation)

Facts (verified observations)

• Verbatim error (job log): KeyError: 'answer' at python/sglang/test/few_shot_gsm8k.py:123 (preds.append(get_answer_value(states[i]["answer"]))), then Exception: retry() exceed maximum number of retries. → FAILED (errors=1) → file returned exit code 1 → job exit code 255.
• Test Summary: 3/4 passed; only test_disaggregation_pp.py failed. The three non-PP disaggregation tests in the same job passed.
• The failing test uses pipeline parallelism (--pp-size 2 on prefill, TP2 both sides, --enable-dynamic-chunking, --disable-overlap-schedule, --attention-backend aiter, Qwen3-8B). All three test_gsm8k variants in the file are PP-based.
• The 200-question gsm8k progress bar completed to 100%|██████████| 200/200 before the KeyError, i.e. all requests "returned" but ≥1 state lacked the answer variable → the generation for that request produced no output (not a threshold/accuracy assertion failure — assertGreater(accuracy, 0.70) was never reached).
• Across the queried window the same KeyError: 'answer' recurs on test_disaggregation_pp.py (Jul 18, Jul 19, Jul 21, Jul 22) and passes cleanly (4/4) on other runs — intermittent, and confined to the PP variants of the file.

Hypothesised Causes (with confidence)

• [MEDIUM] PD+PP consensus divergence causes a request to be aborted/dropped on one PP stage, so it returns with empty output and the SGL answer variable is never populated.
  • Supporting: The failure is exclusive to the PP-based disaggregation tests (all use --pp-size 2); the three non-PP disaggregation tests in the same job pass. Two open PRs — #31869 "[PD+PP] Make prefill and decode PP consensus authoritative" and #31798 "[PD+PP] Make decode queue consensus authoritative" — describe exactly this class of race (an abort/failure arriving between PP consensus and queue application causes stages to diverge and requests to be dropped). Symptom is intermittent (1/200 requests) and rotates across PP variants, matching a nondeterministic race rather than a deterministic bug.
  • Disconfirming: No direct stack-trace/log evidence links the missing answer to a specific abort event in this run's server logs (no server-side error/abort message captured near the failing request). A missing answer could also stem from an empty aiter generation unrelated to PP consensus. Confidence capped at MEDIUM without a captured abort/drop event.
  • How to verify: Re-run test_disaggregation_pp.py on MI35x 8-GPU with a branch that includes #31869 + #31798 and check whether the KeyError: 'answer' disappears across repeated runs; independently, instrument few_shot_gsm8k to log the offending request's raw output/finish_reason when answer is missing.
• [LOW] aiter attention backend intermittently returns an empty generation for one request under the dynamic-chunking prefill path.
  • Supporting: All variants use --attention-backend aiter; the log shows aiter falling back to torch solution:0 (no tuned config) for several GEMM shapes.
  • Disconfirming: The non-PP disaggregation tests also use aiter and pass; failure correlates with PP, not aiter usage per se.
  • How to verify: not actionable until the PP-consensus hypothesis is ruled out.

In-flight Fix Check

• In-flight fix: ✅ #31869 ("[PD+PP] Make prefill and decode PP consensus authoritative", open since 2026-07-21) and ✅ #31798 ("[PD+PP] Make decode queue consensus authoritative", open since 2026-07-20) — both target the PD+PP request-divergence race that is the leading hypothesis. Chase reviewers / validate against this test; do NOT open a duplicate. (Search: search/issues?q=repo:sgl-project/sglang+is:pr+is:open+disaggregation+pp.)

Suggested Triage Steps

• Confirm whether #31869 and/or #31798 resolve this: build a branch with both, run test/registered/amd/disaggregation/test_disaggregation_pp.py on MI35x 8-GPU ~5-10× to check the intermittent KeyError: 'answer' clears.
• Add diagnostics in python/sglang/test/few_shot_gsm8k.py around line 123: when states[i]["answer"] is missing, log the request index, finish_reason, and raw text — to distinguish "dropped/aborted request" (supports PP-consensus hypothesis) from "empty generation" (supports aiter hypothesis).
• If confirmed as the known PD+PP race, treat this as tracked by the above PRs rather than a new regression; consider quarantining/marking the PP gsm8k variants flaky on AMD until the consensus fixes merge.

Status

• Persistence: Intermittent since 2026-07-18 00:25 UTC — failing in ~4 of the last 9 completed scheduled runs; last green for this file was 2026-07-21 00:27 UTC.
• Scope: 1 job (stage-b-test-large-8-gpu-mi35x-disaggregation-amd) in pr-test-amd.yml; confined to the PP variants of test_disaggregation_pp.py. Blocks the MI35x 8-GPU disaggregation CI signal whenever it trips.
• In-flight fix: ✅ #31869 + #31798 (PD+PP consensus) — candidate fixes, need validation against this test.

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-07-22 19:32 UTC)

[amd-bot]

nightly-test-amd.yml — 13 failure(s)

Run started (UTC): 2026-07-21 17:57 UTC
Last scanned (UTC): 2026-07-22 20:09 UTC
Commits: sglang 0bdd473

Summary

Counts: 13 failures · 8 clusters · 9 carrying over (≥3 days) · 3 new today · 1 with no per-job analysis

#	Cluster	Job	Test File	Test Function	Status	Hypothesis (confidence)
1	GLM-5.1-MXFP4 TP=2 launch hang (MI35x)	nightly-2-gpu-mi35x-glm51-mxfp4	test/registered/amd/accuracy/mi35x/test_glm51_mxfp4_tp2_gsm8k_mi35x.py	TestGLM51MXFP4TP2GSM8KMI35x.setUpClass	Never-passed (since Jul 13)	[MEDIUM] TP=2 MXFP4 init hangs gfx950 GPU during model load
2	GLM-5.1-MXFP4 TP=2 launch hang (MI35x)	nightly-2-gpu-mi35x-glm51-mxfp4	test/registered/amd/accuracy/mi35x/test_glm51_mxfp4_tp2_gsm8k_mi35x.py	TestGLM51MXFP4TP2GSM8KMI35x.test_gsm8k	Never-passed; log BlobNotFound (runner lost)	[MEDIUM] same init hang; runner dies before log upload
3	VLM MMMU accuracy below threshold (MI30x)	nightly-accuracy-2-gpu-vlm	test/registered/amd/accuracy/mi30x/test_vlms_mmmu_eval_amd.py	test_mmmu_vlm_models	Never-passed (since Jul 13); deterministic	[MEDIUM] per-model thresholds too high for 100-sample MMMU on MI30x
4	VLM MMMU accuracy below threshold (MI30x)	nightly-accuracy-2-gpu-vlm	test/registered/amd/accuracy/mi30x/test_vlms_mmmu_eval_amd.py	test_mmmu_vlm_models	Never-passed; same 5 models	[MEDIUM] same threshold miscalibration
5	NameError transfer_kv_mamba_lf_pf on ROCm	nightly-test-1-gpu-kernel	test/registered/jit/test_transfer_mamba.py	test_mamba_kernel_* (12/16)	Never-passed on AMD (since Jul 16)	[FACT] mamba JIT import gated if _is_cuda: at memory_pool_host.py:41, call site :407 unguarded
6	NameError transfer_kv_mamba_lf_pf on ROCm	nightly-test-1-gpu-kernel	test/registered/jit/test_transfer_mamba.py	test_mamba_kernel_* (12/16)	Never-passed on AMD (≥6 days)	[FACT] same CUDA-only import gate
7	GSM8K accuracy just below hard threshold (MI35x)	nightly-4-gpu-mi35x-minimax-m25	test/registered/amd/accuracy/mi35x/test_minimax_m25_tp4_eval_mi35x.py	test_minimax_m25_tp4_accuracy	New today (0.928 vs 0.93)	[MEDIUM] borderline threshold + fp8-KV numeric variance
8	GSM8K accuracy just below hard threshold (MI35x)	nightly-accuracy-8-gpu-mi35x-deepseek-v32	test/registered/amd/accuracy/mi35x/test_deepseek_v32_eval_mi35x.py	test_deepseek_v32_accuracy (basic)	New today (0.925 vs 0.93, −1 question)	[MEDIUM] zero-margin threshold nondeterminism
9	HIP hipErrorCapturedEvent during decode graph capture	nightly-8-gpu-mi35x-qwen3-235b-mxfp4	test/registered/amd/test_qwen3_instruct_mxfp4.py	TestQwen3Instruct2507MXFP4.setUpClass	Never-passed (~3 weeks, since Jul 02)	[MEDIUM] aiter custom-all-reduce flush_graph_buffers/_gather_ipc_meta collective runs during HIP stream capture
10	Mixtral-8x7B eval hang → external kill	nightly-accuracy-2-gpu	test/registered/amd/accuracy/mi30x/test_gsm8k_eval_amd.py	TestNightlyGsm8KEval.test_gsm8k_all_models	Never-completes (4 runs verified)	[MEDIUM] Mixtral TP=2 gsm8k eval stalls, runner kills job
11	MMMU VLM benchmark timeout on GLM-4.1V-9B-Thinking	nightly-4-gpu	test/registered/vlm/test_encoder_dp.py	TestVLMEncoderDP.test_vlm_mmmu_benchmark	Flaky (3/3 GLM draws fail; model-conditional)	[MEDIUM] thinking model + ~1–8 tok/s decode exceeds 3600s timeout
12	GPU Hang (HW Exception) during 8-GPU eval → SIGABRT	nightly-accuracy-8-gpu	test/registered/amd/accuracy/mi30x/test_grok1_fp8_eval_amd.py	GSM8K eval (Grok1-FP8)	New today (grok1 green Jul 16–19)	[MEDIUM] transient GPU/runner HW hang on MI325 node …4cdsn
13	(no per-job analysis available)	nightly-accuracy-2-gpu	—	—	analysis block truncated in source	—

Row 13 is the later-run instance of nightly-accuracy-2-gpu; its per-job analysis is missing from .ci-context/per-job-analyses.md. By job identity it most likely belongs to the Mixtral-eval-hang cluster (row 10), but this is not verified — re-run the per-job analysis to confirm.

Cluster: NameError transfer_kv_mamba_lf_pf on ROCm

• Affected: nightly-test-1-gpu-kernel & nightly-test-1-gpu-kernel, both test_transfer_mamba.py::test_mamba_kernel_* (12/16).
• Shared facts: identical NameError at memory_pool_host.py:407; import block at line 41 is if _is_cuda: only (sibling kvcacheio block uses _is_cuda or _is_hip); 4 empty-index variants pass via early-return. git show 977ea336 only renamed the module, did not introduce the gap.
• Top hypothesis: [FACT] CUDA-only import gate; unguarded call site on the io_backend="kernel" path. Disconfirming: none. Likely introduced by #30535 [MEDIUM] (test born broken Jul 16).
• In-flight fix: ✅ #31741 (open since 2026-07-20, mergeable_state: dirty — needs rebase). Chase reviewers, do NOT duplicate.
• Suggested triage: rebase #31741, re-run kernel job; verify no CUDA regression on the io_backend="kernel" path.

Cluster: VLM MMMU accuracy below threshold (MI30x)

• Affected: nightly-accuracy-2-gpu-vlm & nightly-accuracy-2-gpu-vlm, test_vlms_mmmu_eval_amd.py::test_mmmu_vlm_models.
• Shared facts: same 5 models fail with near-identical scores across Jul 13–22 (Qwen2-VL 0.28/0.30, Qwen2.5-VL 0.28–0.29/0.33, Qwen3-VL-30B 0.26–0.27/0.29, MiniCPM-o 0.28/0.32, deepseek-vl2 0.28/0.31); deterministic, SHA-independent; aiter unchanged.
• Top hypothesis: [MEDIUM] per-model thresholds miscalibrated for 100-sample MMMU on MI30x. Disconfirming: could mask a real AMD vision-path degradation (all VLMs cluster near ~0.28); no NVIDIA/full-MMMU baseline in this job.
• In-flight fix: ❌ none (#26178 is stale + edits a different file).
• Suggested triage: git log -p the threshold lines to check they were ever met on MI30x; A/B one model on a larger MMMU sample / NVIDIA to separate "thresholds too high" from "real regression" before re-baselining.

Cluster: GLM-5.1-MXFP4 TP=2 launch hang (MI35x)

• Affected: nightly-2-gpu-mi35x-glm51-mxfp4 & nightly-2-gpu-mi35x-glm51-mxfp4.
• Shared facts: never-passed since Jul 13; readable logs (Jul 13/14 HW Exception … GPU Hang in model_runner.load_model exit 134; Jul 16 scheduler-watchdog → TimeoutError); recent runs die ~22 min with BlobNotFound (runner lost). 8-GPU GLM-5.1 peers pass; only TP=2 MXFP4 fails.
• Top hypothesis: [MEDIUM] config-specific init hang (--tp 2 --kv-cache-dtype fp8_e4m3 --dsa-*-backend tilelang) on gfx950. Disconfirming: no bisectable regression window (no green run); multiple failure surfaces (9-min hang / 90-min timeout / 22-min no-log).
• In-flight fix: ⚠️ candidates #30808, #30519 (GLM5/gfx950) — none confirmed for this hang.
• Suggested triage: repro launch cmd on gfx950 with py-spy/rocm-smi/dmesg; A/B toggle tilelang DSA backends and fp8_e4m3 KV; consider quarantine until root-caused.

Cluster: GSM8K accuracy just below hard threshold (MI35x)

• Affected: nightly-4-gpu-mi35x-minimax-m25 (test_minimax_m25_tp4_accuracy, 0.928/0.93) & nightly-accuracy-8-gpu-mi35x-deepseek-v32 (test_deepseek_v32_accuracy basic, 0.925/0.93).
• Shared facts: both new-today near-misses; servers ran clean to completion (no crash); passing runs cleared the bar by only 1–2 questions; aiter unchanged.
• Top hypothesis: [MEDIUM] zero-margin threshold + run-to-run nondeterminism (fp8-KV / sampling). Disconfirming: deepseek trend is monotonically downward over 4 samples (0.935→0.925) — could be a small real drift, not pure noise.
• In-flight fix: ❌ none for either (note #31702/#31748 lowered other zero-margin thresholds).
• Suggested triage: re-run each 3–5× on the same SHA; if scores straddle the bar, treat as threshold-too-tight and widen num_questions/threshold; only bisect if reruns stay below.

Cluster: HIP hipErrorCapturedEvent during decode graph capture

• Affected: nightly-8-gpu-mi35x-qwen3-235b-mxfp4, test_qwen3_instruct_mxfp4.py::setUpClass.
• Shared facts: all 4 TP ranks SIGABRT (exit -6); Python fault stack in aiter custom_all_reduce.py:331 _gather_ipc_meta ← flush_graph_buffers ← capture ← sglang parallel_state.py:580 graph_capture; NCCL watchdog queries a HIP event mid-capture. Signature stable ~3 weeks; aiter SHA fixed.
• Top hypothesis: [MEDIUM] aiter custom-all-reduce runs an IPC-meta collective inside the capture window. Disconfirming: no A/B; aiter source not vendored here; collective could originate elsewhere.
• In-flight fix: ❌ none.
• Suggested triage: repro on gfx950 with AMD_SERIALIZE_KERNEL=3; test --disable-custom-all-reduce; inspect aiter custom_all_reduce.py:303/331 at 9127c94a.

Cluster: Mixtral-8x7B eval hang → external kill

• Affected: nightly-accuracy-2-gpu, test_gsm8k_eval_amd.py::test_gsm8k_all_models (likely also row 13).
• Shared facts: 4/4 log-verified runs stop mid-Mixtral decode (last line healthy at 15028 tok/s), ~19 min silence, then failure with step conclusion None; variable 54–81 min duration argues infra kill, not timeout. Downstream FP8/AMD accuracy groups never run.
• Top hypothesis: [MEDIUM] Mixtral TP=2 eval stalls, runner terminates job. Disconfirming: no server-side crash/OOM in job-page log — hang vs hard kill cannot be disambiguated without server logs.
• In-flight fix: ❌ none.
• Suggested triage: repro Mixtral gsm8k standalone on MI325 2-GPU with server-log + dmesg capture; add explicit timeout-minutes/eval HTTP timeout so it fails diagnosably; consider moving Mixtral to failing_models to unblock the matrix.

Cluster: MMMU VLM benchmark timeout on GLM-4.1V-9B-Thinking

• Affected: nightly-4-gpu, test_encoder_dp.py::test_vlm_mmmu_benchmark.
• Shared facts: model chosen by random.choice; all 3 GLM draws (Jul 13/19/22) fail, all 5 non-GLM draws pass; server made steady 200-OK progress at 1–8 tok/s until the 3600s timeout; 3137 untuned-aiter-GEMM torch solution:0 fallback lines.
• Top hypothesis: [MEDIUM] thinking-model long generations + low decode throughput exceed the per-file timeout. Disconfirming: Jul 13 GLM run failed as a scheduler-watchdog stall at 1340s, not a clean 3600s timeout — suggests a second GLM failure mode.
• In-flight fix: ❌ none.
• Suggested triage: pin the test to GLM (bypass random) and time the sweep on MI325 4-GPU; decide whether to pin the AMD MODELS list, raise the GLM timeout, or cap max_new_tokens.

Cluster: GPU Hang (HW Exception) during 8-GPU eval → SIGABRT

• Affected: nightly-accuracy-8-gpu, test_grok1_fp8_eval_amd.py GSM8K eval.
• Shared facts: HW Exception by GPU node-9 … reason :GPU Hang on all 8 ranks → exit 134; grok1 green the prior 4 nightlies (last 0.830 on Jul 19); GPT-OSS on the same 8 GPUs passed earlier in the job; server had soft watchdog + 90s detokenizer stall before the eval.
• Top hypothesis: [MEDIUM] transient GPU/runner HW hang on MI325 node …4cdsn. Disconfirming: single occurrence — cannot yet separate flake from regression.
• In-flight fix: grok1-specific ❌ none; infra GPU-hang-recovery PRs #31068, #30431, #31229 exist (not root-cause fixes).
• Suggested triage: re-run on a different MI325 runner; if grok1 passes, close as HW flake and check …4cdsn for recurring hangs (quarantine if so).

Suggested triage order

Rank	Triage step	Targets	Why this first
1	Rebase & land #31741, re-run kernel job	Cluster A (rows 5–6)	Clear [FACT] root cause + in-flight fix already exists (just conflicted) — highest ROI
2	Verify VLM MMMU + GSM8K thresholds vs achievable MI35x/MI30x scores; re-run near-misses 3–5×	Clusters B (rows 3–4), D (rows 7–8)	Cheap; 4 accuracy-gate failures likely calibration/noise; confirm not masking a real regression before re-baselining
3	Repro GLM-5.1-MXFP4 TP=2 launch on gfx950; A/B tilelang DSA + fp8 KV	Cluster C (rows 1–2)	Never-passed 9 days, blocks a full accuracy gate; no green anchor for bisect so needs live repro
4	Repro Qwen3-235B capture crash with --disable-custom-all-reduce + inspect aiter custom_all_reduce.py	Cluster F (row 9)	Longest-running (~3 weeks); isolates aiter vs sglang cheaply
5	Re-run Grok1 8-GPU on a non-4cdsn MI325; repro Mixtral eval + GLM-4.1V timeout with server logs	Clusters E (row 12), G (row 10 + row 13), H (row 11)	Likely HW flake / config issues; gather server-side evidence and confirm row 13's cluster

---

Job	Failed Steps	Started
nightly-8-gpu-mi35x-qwen3-235b-mxfp4	Accuracy Test + Performance Test MI35x (8-GPU Qwen3-235B-MXFP4)	2026-07-21T22:41
nightly-accuracy-8-gpu-mi35x-deepseek-v32	Accuracy Test MI35x (8-GPU DeepSeek-V3.2)	2026-07-21T21:20
nightly-2-gpu-mi35x-glm51-mxfp4	N/A	2026-07-21T17:57
nightly-4-gpu-mi35x-minimax-m25	Accuracy Test MI35x (4-GPU MiniMax-M2.5)	2026-07-21T23:03
nightly-accuracy-2-gpu-vlm	Nightly Accuracy Test (2-GPU VLM MMMU)	2026-07-22T00:28
nightly-accuracy-2-gpu	N/A	2026-07-22T00:32
nightly-test-1-gpu-kernel	Nightly Kernel Benchmark (1-GPU)	2026-07-22T02:12
nightly-accuracy-8-gpu	Accuracy Test (8-GPU Grok1-FP8)	2026-07-22T09:11
nightly-test-1-gpu-kernel	Nightly Kernel Benchmark (1-GPU)	2026-07-22T18:05
nightly-2-gpu-mi35x-glm51-mxfp4	N/A	2026-07-22T17:53
nightly-4-gpu	Nightly Test (4-GPU)	2026-07-22T17:53
nightly-accuracy-2-gpu-vlm	Nightly Accuracy Test (2-GPU VLM MMMU)	2026-07-22T18:39
nightly-accuracy-2-gpu	N/A	2026-07-22T18:52

Per-Job Analysis

nightly-8-gpu-mi35x-qwen3-235b-mxfp4 — failed step(s): Accuracy Test + Performance Test MI35x (8-GPU Qwen3-235B-MXFP4)

Commit Info

• sglang: 9a6c96083
• aiter: 9127c94a (v0.1.17.dev110+g9127c94a1, source: dev/patched image — [CI-AITER-CHECK] detected a dev build and skipped rebuild; Dockerfile default is the same SHA)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/test_qwen3_instruct_mxfp4.py	TestQwen3Instruct2507MXFP4.setUpClass	Exception: Server process exited with code 1 → RuntimeError: Rank 0 scheduler died during initialization (exit code: -6)	Log

Failure Summary

The single-suite job launches sglang serve --model-path amd/Qwen3-235B-A22B-Instruct-2507-mxfp4 --tp 4 --ep 2 --attention-backend aiter and the server dies during startup, so setUpClass raises and Ran 0 tests. All 4 TP ranks abort with Fatal Python error: Aborted (SIGABRT / exit code -6) while capturing the decode CUDA graph. The crash is a HIP stream-capture violation: HIP error: operation not permitted on an event last recorded in a capturing stream (hipErrorCapturedEvent), thrown by the ProcessGroupNCCL watchdog thread.

Regression Status

Known recurring failure / Never-passed in the queryable window. This job runs only test_qwen3_instruct_mxfp4.py, so the job's pass/fail equals the test file's status. No green run exists in the last 15 comparable completed scheduled runs; the identical crash signature (hipErrorCapturedEvent in flush_graph_buffers → _gather_ipc_meta, scheduler exit -6) is present as far back as 2026-07-02 (verified from logs on 07-19, 07-18, 07-02).

Recent history of test_qwen3_instruct_mxfp4.py (TestQwen3Instruct2507MXFP4.setUpClass) in job nightly-8-gpu-mi35x-qwen3-235b-mxfp4 (comparable completed scheduled runs, branch=main, event=schedule; cancelled runs where the job did not complete are excluded):

Date	Run	Test File Status	Failed Function	Error
Jul 21	29855050299	❌ Failed	setUpClass	hipErrorCapturedEvent during decode graph capture, exit -6
Jul 19	29697529176	❌ Failed	setUpClass	same signature (verified)
Jul 18	29654601256	❌ Failed	setUpClass	same signature (verified)
Jul 17	29601593052	❌ Failed	setUpClass	job failure (test-level not separately downloaded)
Jul 17	29521595306	⏭️ Cancelled	—	job did not complete (excluded)
Jul 14	29355460805	❌ Failed	setUpClass	different: GPU Hang HW exception, exit 134
Jul 13	29273241962	❌ Failed	setUpClass	job failure (test-level not separately downloaded)
Jul 05	28749559519	❌ Failed	setUpClass	job failure
Jul 04	28714602438	❌ Failed	setUpClass	job failure
Jul 03	28676359018	❌ Failed	setUpClass	job failure
Jul 02	28611376093	❌ Failed	setUpClass	same signature (verified)

Lookback range: 2026-07-02 → 2026-07-21. No passing run for this test file found in the window ⇒ Never-passed here. Note the 07-14 failure was a distinct GPU-hang infra symptom, not the capture crash.

Failure Cluster

HIP stream-capture violation (hipErrorCapturedEvent) during decode CUDA graph capture via aiter custom all-reduce flush_graph_buffers.

Facts (verified observations, no interpretation)

• All 4 TP ranks print [E721 22:54:42 ProcessGroupNCCL.cpp:2055] ... watchdog thread terminated with exception: HIP error: operation not permitted on an event last recorded in a capturing stream (hipErrorCapturedEvent); one rank also reports hipErrorStreamCaptureUnsupported (job log lines ~5407–5524).
• Crashing C++ frame is the NCCL watchdog: ProcessGroupNCCL::WorkNCCL::finishedGPUExecutionInternal() → isCompleted() → Watchdog::runLoop() (querying a collective's HIP event while a stream is capturing).
• Python fault stack (verbatim from log): aiter/dist/device_communicators/custom_all_reduce.py:331 _gather_ipc_meta ← custom_all_reduce.py:303 flush_graph_buffers ← custom_all_reduce.py:503 capture ← sglang/srt/distributed/parallel_state.py:580 graph_capture ← parallel_state.py:1829 graph_capture ← decode_cuda_graph_runner.py:854 capture ← cuda_graph_setup.py:333/106 ← model_runner.py:846 init_cuda_graphs.
• The capture entered right after Capture target decode CUDA graph begin. backend=full ... bs=[1..512] and after multimem all-gather disabled (module 'torch.distributed._symmetric_memory' has no attribute 'set_signal_pad_size') on all ranks (log ~5344, ~5403).
• Result: Fatal Python error: Aborted → Rank 0 scheduler died during initialization (exit code: -6) → Server process exited with code 1 → Ran 0 tests in 440.450s / FAILED (errors=1) → job exit 255.
• aiter version identical across passing-attempt search and this run: v0.1.17.dev110+g9127c94a1 ([CI-AITER-CHECK]), so aiter version is not a changing variable in this window.
• Reproducibility: same signature verified on 07-21, 07-19, 07-18, 07-02; failure affects all 4 TP ranks every run.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [MEDIUM] aiter custom all-reduce capture() context (custom_all_reduce.py:503) runs flush_graph_buffers → _gather_ipc_meta (an IPC-handle all-gather / distributed collective) at the start of graph capture, while the HIP stream is already in capture mode. The NCCL watchdog then calls hipEventQuery on that collective's event → hipErrorCapturedEvent. This is entered from sglang parallel_state.py:580 (maybe_ca_context = ca_comm.capture()), which wraps the whole capture block.
  • Supporting: the Python fault stack points squarely at _gather_ipc_meta inside flush_graph_buffers invoked from the capture context manager; the C++ crash is the NCCL watchdog querying a HIP event during capture; error is operation not permitted on an event last recorded in a capturing stream.
  • Disconfirming: no A/B disconfirmation was run, and the exact aiter code at lines 303/331 was not read in this workspace (aiter is not vendored here); the collective could instead be triggered by a co-scheduled process-group op rather than _gather_ipc_meta itself.
  • How to verify: reproduce on MI35x (gfx950) in the CI image with AMD_SERIALIZE_KERNEL=3; inspect aiter custom_all_reduce.py:303/331 at commit 9127c94a; try disabling custom all-reduce (--disable-custom-all-reduce / equivalent) to see if capture proceeds.
• [LOW] sglang-side interaction: parallel_state.py graph-capture ordering places a collective (or the watchdog's event-query) inside the capture window on the aiter path specifically.
  • Supporting: the capture context is opened by sglang code; the comment block at parallel_state.py:580+ explicitly notes collective-mode fragility during graph capture.
  • Disconfirming: git log shows no recent change to parallel_state.py graph_capture in the failing window that touches this path (most recent edits are unrelated features #29778/#29972/#31501); the failure predates the current HEAD by ≥3 weeks with a fixed aiter SHA, so a fresh sglang regression is unlikely.
  • How to verify: bisect is low-value since the failure is stable across a wide SHA range; prefer the aiter-side repro above.
• [LOW] GPU-hang / infra confound (seen 07-14 only).
  • Supporting: run 29355460805 failed with HW Exception ... GPU Hang (exit 134), a different symptom.
  • Disconfirming: every other queried run shows the deterministic hipErrorCapturedEvent capture crash, not a hang — the 07-14 hang is a separate, intermittent infra event, not the primary cluster.

No hypothesis reaches HIGH: the primary cause is a code-level capture/collective-ordering issue on the aiter custom-all-reduce path but lacks an A/B disconfirmation or in-repo aiter source read.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for hipErrorCapturedEvent (0 results), flush_graph_buffers (0 results), and qwen3-235b mxfp4 (2 results: #29419 sgl-data mount, #27209 gfx950 Dockerfile — both unrelated to the capture crash). Needs a new PR or investigation.

Suggested Triage Steps

• Reproduce on an MI35x (gfx950) runner in image rocm/sgl-dev:v0.5.15.post1-rocm700-mi35x-20260721 with the exact serve command, adding AMD_SERIALIZE_KERNEL=3 to sharpen the async HIP stacktrace.
• Test whether disabling custom all-reduce (so ca_comm is None and maybe_ca_context becomes nullcontext() at parallel_state.py:571-572) lets decode-graph capture complete — this isolates the aiter capture()/flush_graph_buffers path.
• Inspect aiter custom_all_reduce.py:303 flush_graph_buffers / :331 _gather_ipc_meta at 9127c94a for a collective/event op executed inside the capture window; if confirmed, file an aiter issue/PR to move IPC-meta gathering outside graph capture (or suppress the NCCL watchdog event-query during capture).
• Treat the 07-14 GPU Hang (exit 134) as a separate intermittent infra signal; do not fold it into this cluster.

Status

• Persistence: Failing continuously in every completed scheduled run in the queryable window (2026-07-02 → 2026-07-21, ~3 weeks); no green run for this test file found.
• Scope: 1 job / 1 workflow (nightly-test-amd.yml → nightly-8-gpu-mi35x-qwen3-235b-mxfp4), all 4 TP ranks; blocks the MI35x Qwen3-235B-A22B MXFP4 accuracy+perf nightly signal.
• In-flight fix: ❌ none found in open PRs.

nightly-accuracy-8-gpu-mi35x-deepseek-v32 — failed step(s): Accuracy Test MI35x (8-GPU DeepSeek-V3.2)

Commit Info

• sglang: 9a6c96083
• aiter: 9127c94a1 — v0.1.17.dev110+g9127c94a1 (source: dev/patched image, preserved; AITER_COMMIT_OVERRIDE empty)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi35x/test_deepseek_v32_eval_mi35x.py	test_deepseek_v32_accuracy (variant basic)	AssertionError: Failed models: ['deepseek-ai/DeepSeek-V3.2 (basic)'] — accuracy=0.925 threshold=0.93 ❌ FAIL	Log

Failure Summary

The single GSM8K accuracy test for DeepSeek-V3.2 (basic, TP=8, dsa/tilelang backend) scored 0.925 vs a 0.93 threshold on 200 questions. Since the benchmark uses num_questions=200, 0.925 = 185/200 correct and the 0.93 bar requires 186/200 — the run missed the threshold by exactly one question. The server launched, warmed up, and served all 200 requests with HTTP 200; there was no crash.

Regression Status

New regression (marginal near-miss) for this test file — last completed scheduled run passed. However, accuracy has been trending down toward the threshold over the last week, so this reads as a borderline-threshold / near-flaky failure rather than a hard break.

Recent history of test_deepseek_v32_eval_mi35x.py (test_deepseek_v32_accuracy, basic) in job nightly-accuracy-8-gpu-mi35x-deepseek-v32 (most recent comparable event=schedule, branch=main, completed runs):

Date	Run	Test File Status	Accuracy	Notes
Jul 21	29855050299	❌ Failed	0.925 (185/200)	this run — missed by 1 question
Jul 20	29766761021	⏭️ n/a	—	run cancelled (not comparable)
Jul 19	29697529176	✅ Passed	0.935 (187/200)	—
Jul 18	29654601256	✅ Passed	0.935 (187/200)	—
Jul 17	29601593052	✅ Passed	0.930 (186/200)	passed at exactly the threshold
Jul 16	29521595306	⏭️ n/a	—	run cancelled (not comparable)
Jul 14	29355460805	❌ Failed (different)	—	GPU Hang / SIGABRT (exit 134), not an accuracy miss
Jul 13	29273241962	❌ Failed (different)	—	GPU Hang / SIGABRT (exit 134), not an accuracy miss

Last accuracy-pass: Jul 19 (0.935). First accuracy-fail for this metric: Jul 21 (today). The Jul 14/Jul 13 failures are a different cluster (GPU hang crash), not the accuracy near-miss.

Failure Cluster

GSM8K accuracy near-miss at threshold (DeepSeek-V3.2 basic, MI35x)

Facts (verified observations)

• accuracy=0.925 threshold=0.93 ❌ FAIL; AssertionError: Failed models: ['deepseek-ai/DeepSeek-V3.2 (basic)'] at test_deepseek_v32_eval_mi35x.py:245 (verbatim from job log).
• Threshold is 0.93 and num_questions=200 (test_deepseek_v32_eval_mi35x.py:71, :174) ⇒ pass requires ≥186/200; this run got 185/200, a 1-question miss.
• Server ran clean: 200/200 /generate requests returned HTTP 200; Ran 1 test in 1506.173s; no crash, no memory fault, exit code 1 (test assertion), overall step exit 255.
• Trend across passing runs: 0.935 (Jul 19) → 0.935 (Jul 18) → 0.930 (Jul 17, at threshold) → 0.925 (Jul 21, below) — accuracy oscillating within ±1–2 questions of the bar.
• aiter is a preserved dev image (v0.1.17.dev110+g9127c94a1), unchanged vs. the Jul 17–19 passing runs' expectation; AITER_COMMIT_OVERRIDE empty (verified from [CI-AITER-CHECK] markers).
• Commit window since last accuracy-pass: git log d4801be44773..9a6c96083 = 72 commits.

Hypothesised Causes (with confidence)

• [MEDIUM] Borderline threshold / run-to-run nondeterminism — the 0.93 bar has near-zero margin at 200 questions (one question = 0.005). Jul 17 passed at exactly 0.930; today's 0.925 is one question below. GSM8K completion accuracy at TP=8 with dsa/tilelang is not bit-deterministic across runs.
  • Supporting: passing runs cluster at 0.930–0.935 (185–187/200); today is 185/200; no crash or error, just the assertion.
  • Disconfirming: the trend is monotonically downward over 4 samples (0.935→0.935→0.930→0.925), which could indicate a small real numerical drift rather than pure noise — not conclusively random.
  • How to verify: re-run nightly-accuracy-8-gpu-mi35x-deepseek-v32 on the same SHA 3–5×; if accuracy straddles 0.925–0.935, it's threshold nondeterminism.
• [LOW] A numerics-affecting change landed in the 72-commit window d4801be44773..9a6c96083. Candidate touching AMD numerics: dcd9014f1 ("[AMD][MXFP4] Reland Online MXFP4 quantization 2/N", #28291).
  • Supporting: it is an AMD FP8→MXFP4 requantization change and lands in the window.
  • Disconfirming: this test runs with default quantization=None (verified in server_args log line) — the MXFP4 requant path is likely not exercised here; and the 1-question margin is within observed run-to-run noise.
  • How to verify: bisect d4801be44773..9a6c96083 with the accuracy test, or A/B revert dcd9014f1 and re-run.

No hypothesis reaches HIGH confidence; a 1-question miss on a zero-margin threshold cannot be pinned to a specific commit without a rerun/bisect.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for DeepSeek-V3.2+accuracy+threshold, mi35x+deepseek+accuracy, v32+gsm8k+threshold; no open PR addresses this MI35x basic accuracy near-miss or its threshold. (Note #31702/#31748 lowered other tests' thresholds for the same zero-margin reason, but not this one.)

Suggested Triage Steps

• Re-run nightly-accuracy-8-gpu-mi35x-deepseek-v32 on SHA 9a6c96083 3–5× to distinguish threshold noise from a real drift; if it straddles the bar, treat as flaky.
• If confirmed as zero-margin nondeterminism, consider raising num_questions (tighter statistics) or lowering the 0.93 threshold with justified margin — mirroring the approach already taken in #31702 / #31748.
• If reruns stay at ~0.925, bisect d4801be44773..9a6c96083; start with AMD-numerics commit dcd9014f1 (#28291), though the default-quantization=None config makes it a weak candidate.

Status

• Persistence: 1 completed run (Jul 21). Prior three accuracy-producing runs (Jul 17–19) passed; Jul 13/14 failed with a different GPU-hang crash.
• Scope: 1 test file / 1 function, this job only (nightly-accuracy-8-gpu-mi35x-deepseek-v32); the sister -mtp accuracy job is not part of this failure.
• Margin: below threshold by exactly 1 question (185/200 vs 186/200 required).
• In-flight fix: ❌ none found in open PRs.

nightly-2-gpu-mi35x-glm51-mxfp4 — failed step(s): N/A

Commit Info

• sglang: 9a6c96083
• aiter: unknown — this job's log blob is unavailable (BlobNotFound), so [CI-AITER-CHECK] markers cannot be read for this run. Context: adjacent MI35x runs Jul 13–16 all used the Dockerfile gfx950 default dev image v0.1.17.dev110+g9127c94a1 (preserved, no rebuild), unchanged across those runs.

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi35x/test_glm51_mxfp4_tp2_gsm8k_mi35x.py	TestGLM51MXFP4TP2GSM8KMI35x.setUpClass (server launch)	Log blob unavailable for this run; step never completed. See Facts — sister runs fail identically at setUpClass (GPU Hang / server-launch TimeoutError)	Job

The single test file in suite nightly-amd-2-gpu-mi35x-glm51-mxfp4 is test_glm51_mxfp4_tp2_gsm8k_mi35x.py. The failure is in setUpClass (the GLM-5.1-MXFP4 TP=2 server never launches), not in the test_gsm8k accuracy assertion.

Failure Summary

The job nightly-2-gpu-mi35x-glm51-mxfp4 failed (job conclusion: failure) while the accuracy step "Accuracy Test MI35x (2-GPU GLM-5.1-MXFP4 GSM8K)" was still in_progress (step 6 never got a conclusion; step 12 Post Checkout code never ran). Total wall time was ~23 min (17:57:54 → 18:20:55 UTC), far short of the 120-min step timeout. The job's log blob returns BlobNotFound, i.e. logs were never finalized — consistent with an abrupt process/runner termination during server startup, matching the same failure mode captured in this job's loggable sister runs.

Regression Status

Never-passed (no green run found in queryable window). This test file has failed in every comparable completed scheduled run in the lookback window.

Recent history of test_glm51_mxfp4_tp2_gsm8k_mi35x.py (TestGLM51MXFP4TP2GSM8KMI35x.setUpClass) in job nightly-2-gpu-mi35x-glm51-mxfp4 (most recent comparable runs: branch=main, event=schedule, completed only):

Date	Run	Test File Status	Failure signature	Evidence
Jul 21	29855050299	❌ Failed	step6 in_progress, ~23 min, log BlobNotFound (abrupt term)	this run
Jul 20	29766761021	❌ Failed	step6 in_progress, ~23 min, log BlobNotFound	job steps API
Jul 19	29697529176	❌ Failed	step6 in_progress, ~24 min, log BlobNotFound	job steps API
Jul 18	29654601256	❌ Failed	step6 in_progress, ~22 min, log BlobNotFound	job steps API
Jul 17	29601593052	❌ Failed	step6 in_progress, ~27 min, log BlobNotFound	job steps API
Jul 16	29521595306	❌ Failed	setUpClass → TimeoutError: Server failed to start after ~1h37m of scheduler watchdog timeouts	log present
Jul 15	29438351311	⏭️ Cancelled	run-level cancel (~24h) — not comparable	job API
Jul 14	29355460805	❌ Failed	setUpClass → GPU Hang (HW Exception), exit 134 during model_runner.load_model	log present
Jul 13	29273241962	❌ Failed	setUpClass → GPU Hang (HW Exception), exit 134 during model load	log present

First observed failure in window: Jul 13 (earliest loggable). No passing run found; runs before Jul 13 were cancelled at the run level. The test file was registered ~2026-05-28 (#26478) / added ~2026-05-27 (#26396). Lookback range searched: Jul 7 → Jul 21 (15 scheduled runs).

Failure Cluster

GLM-5.1-MXFP4 TP=2 server fails to launch on 2×MI35x (gfx950) — GPU hang / scheduler hang during model load

Facts (verified observations)

• This run: job conclusion=failure; step 6 "Accuracy Test MI35x (2-GPU GLM-5.1-MXFP4 GSM8K)" status=in_progress, conclusion=None; step 12 pending; log endpoint returns BlobNotFound. Duration ~23 min << the 120-min step timeout (verified via jobs API).
• Jul 16 (job): 34× [TP0/TP1] Scheduler watchdog timeout (watchdog_timeout=300, soft=True) from 18:55:26 to 20:15:34; py-spy dump repeatedly "Failed to get stack traces"; then ERROR: setUpClass ... TimeoutError: Server failed to start within the timeout period, FAILED (errors=1), process exit 255. Launch cmd: sglang serve --model-path amd/GLM-5.1-MXFP4 --tp 2 ... --kv-cache-dtype fp8_e4m3 --disable-radix-cache --dsa-prefill-backend tilelang --dsa-decode-backend tilelang --tokenizer-worker-num 4.
• Jul 14 (job) & Jul 13 (job): repeated HW Exception by GPU node-2 ... reason :GPU Hang, first attempt exit 134 (SIGABRT), retried, again GPU Hang, final exit code 134. Stack shows the hang inside model_runner.py:841 load_model / model_runner.py:485 initialize (model load / warmup).
• aiter unchanged across the loggable failing runs: v0.1.17.dev110+g9127c94a1 (Dockerfile gfx950 default, dev image preserved) on Jul 13, 14, 16 — confirmed via [CI-AITER-CHECK] markers.
• Scope: same-day peer GLM-5.1 jobs passed — nightly-8-gpu-mi35x-glm51 ✅ and nightly-8-gpu-mi35x-glm5-mxfp4 ✅ (from run 29855050299 jobs API). Only the 2-GPU TP=2 MXFP4 launch on 2×MI35x fails.
• The test's own docstring states it was added to guard "the GLM-5.1-MXFP4 TP=2 accuracy drop seen on MI355X/gfx950 when aiter selected a bad BF16 GEMM path."

Hypothesised Causes (with confidence)

• [MEDIUM] The GLM-5.1-MXFP4 TP=2 launch on gfx950 hangs during model load / scheduler startup (GPU Hang exit 134, or scheduler-loop hang → server-launch TimeoutError), specific to this config's flags (--tp 2 --kv-cache-dtype fp8_e4m3 --dsa-prefill-backend/--dsa-decode-backend tilelang).
  • Supporting: three independent loggable runs (Jul 13/14/16) all fail at setUpClass/server launch with GPU Hang or watchdog-hang→TimeoutError; stack trace lands in model_runner.load_model; the abrupt-termination runs (Jul 17–21, no log upload) are consistent with the hard GPU-hang variant killing the runner before logs finalize.
  • Disconfirming: no code-level evidence pins a specific commit; the shallow CI clone prevents local bisect, and there is no green run in the window to bound a regression range — this may be a persistent AMD port issue rather than a recent regression. The 8-GPU GLM-5.1 variants pass, so the model/quark checkpoint itself loads fine at other parallelism configs.
  • How to verify: on a 2×MI35x (gfx950) runner in the CI container, run the exact launch cmd above and capture py-spy dump/rocm-smi/dmesg at the hang; A/B toggle --dsa-*-backend tilelang vs default and --kv-cache-dtype fp8_e4m3 vs default to isolate which flag triggers the hang.
• [LOW] aiter-selected kernel path on gfx950 during GLM-5.1-MXFP4 load.
  • Supporting: symptom is a GPU-level HW hang inside model load/warmup; the test docstring references an aiter GEMM-path issue for exactly this model/config.
  • Disconfirming: aiter version is identical (v0.1.17.dev110+g9127c94a1) across all loggable failing runs, so an aiter change cannot explain the persistence; and the same aiter build serves the passing 8-GPU GLM-5.1 jobs.
  • How to verify: not actionable until the launch-flag A/B above narrows the failing path.
• No hypothesis reaches HIGH; needs a live repro on a 2×MI35x runner (no bisectable regression window exists — the test has no green run in the queryable history).

In-flight Fix Check

• In-flight fix: ⚠️ candidates, none confirmed for this launch hang — open PRs touching GLM-5.x on gfx950 that could be relevant: #30808 ("[AMD] [GLM5] Enable dense-MHA short-context prefill fallback on gfx950"), #30519 ("[AMD] [GLM5] fp8 MLA absorbed bmm for GLM-5.2 on gfx950"), and docs-only #29873 ("add --enable-aiter-allreduce-fusion to MXFP4 MI355X command" — hints the recommended launch cmd differs from CI's). None explicitly targets this TP=2 server-launch hang. (Searched is:pr is:open with keywords GLM-5.1-MXFP4, GLM MXFP4 accuracy.)

Suggested Triage Steps

• Reproduce on a 2×MI35x (gfx950) runner: launch sglang serve --model-path amd/GLM-5.1-MXFP4 --tp 2 --kv-cache-dtype fp8_e4m3 --disable-radix-cache --dsa-prefill-backend tilelang --dsa-decode-backend tilelang --tokenizer-worker-num 4 and capture py-spy dump, rocm-smi, and kernel dmesg at the hang point.
• A/B the launch flags to isolate the trigger: drop --dsa-*-backend tilelang (use default), then --kv-cache-dtype fp8_e4m3 → default, then --tp 2→ compare against the passing 8-GPU config.
• Investigate the missing-log pattern separately: 5 consecutive runs (Jul 17–21) leave step 6 in_progress with BlobNotFound — a hard GPU hang appears to kill the runner before log upload; a runner-side watchdog that dumps GPU state on hang before teardown would recover diagnostics.
• Given no green run exists in history for this test file, consider whether it should be quarantined/continue-on-error-only on AMD until the launch hang is fixed, so it stops masking the run-level signal.

Status

• Persistence: Failing in every comparable completed scheduled run in the window (Jul 13, 14, 16, 17, 18, 19, 20, 21) — never-passed in the queryable 15-run lookback; test registered ~2026-05-28.
• Scope: 1 job (nightly-2-gpu-mi35x-glm51-mxfp4) / 1 test file. Peer GLM-5.1 jobs at 8-GPU pass, so blast radius is limited to the 2-GPU TP=2 MXFP4 accuracy gate on MI35x.
• Failure mode split: hard GPU Hang exit 134 during model load (Jul 13/14) vs scheduler-hang → server-launch TimeoutError (Jul 16) vs abrupt runner termination with no log (Jul 17–21, incl. today).
• In-flight fix: ⚠️ candidate GLM5/gfx950 PRs exist (#30808, #30519) but none confirmed to address this launch hang.

nightly-4-gpu-mi35x-minimax-m25 — failed step(s): Accuracy Test MI35x (4-GPU MiniMax-M2.5)

Commit Info

• sglang: 9a6c96083
• aiter: v0.1.17.dev110+g9127c94a1 (source: dev image detected, rebuild skipped — [CI-AITER-CHECK] markers, log lines 3910-3911; Dockerfile default was 9127c94a1, which the dev build is based on)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi35x/test_minimax_m25_tp4_eval_mi35x.py	test_minimax_m25_tp4_accuracy	AssertionError: Failed models: ['MiniMaxAI/MiniMax-M2.5 (TP4+UnifiedAttn+FP8KV)'] — GSM8K accuracy=0.928 threshold=0.93 FAIL	Log

Failure Summary

The single GSM8K accuracy test for MiniMax-M2.5 (TP4, aiter attention backend, fp8_e4m3 KV cache) scored 0.928 against a hard threshold of 0.93 — a miss of 0.002 (~2-3 questions out of 1319). invalid=0.000 and all 1319 questions completed, so this is purely a numeric accuracy dip below threshold, not a crash or serving failure. (Earlier Scheduler watchdog timeout at log 6705 and 503 Service Unavailable / detokenizer health-check warnings at logs 6756-6783 occurred during startup while aiter JIT-built kernels; the server recovered — health returned 200 OK at log 6790 — and the benchmark ran to completion, so those are not the cause.)

Regression Status

New failure (borderline) — first observed failure of this test; the job passed the previous 4 comparable scheduled runs, but always with a thin margin over the 0.93 threshold.

Recent history of test_minimax_m25_tp4_eval_mi35x.py (test_minimax_m25_tp4_accuracy) in job nightly-4-gpu-mi35x-minimax-m25 (most recent comparable completed scheduled runs: branch=main, event=schedule; this job runs only this one test file, so job status == test status):

Date	Run	Job	Test File Status	Accuracy	Error
Jul 21	29855050299	job	❌ Failed	0.928 (thr 0.93)	AssertionError
Jul 20	29766761021	job	✅ Passed	0.938	—
Jul 19	29697529176	job	✅ Passed	0.936	—
Jul 18	29654601256	job	✅ Passed	0.936	—
Jul 17	29601593052	job	✅ Passed	— (success)	—

First observed failure: 2026-07-21. Last known passing: 2026-07-20 (run 29766761021, head 5ab3d90b8).

Failure Cluster

GSM8K accuracy just below hard threshold (MiniMax-M2.5 TP4 fp8-KV on MI35x)

Facts (verified observations)

• accuracy=0.928 threshold=0.93 FAIL, invalid=0.000 latency=102.4s; 1319/1319 questions completed (log 10658-10668).
• Server config: --tp 4 --ep-size 1 --dtype bfloat16 --attention-backend aiter --mem-fraction-static 0.85 --disable-radix-cache --kv-cache-dtype fp8_e4m3 --page-size 16 (log 6796).
• Threshold is a hardcoded constant accuracy_threshold = 0.93 at test_minimax_m25_tp4_eval_mi35x.py:44,66; failure is the acc >= config.accuracy_threshold check at line 216 raising at line 251.
• Historical scores: 0.938 (Jul 20), 0.936 (Jul 19), 0.936 (Jul 18) — margin over threshold was only 0.006-0.008 in every passing run.
• aiter version v0.1.17.dev110+g9127c94a1 preserved from the dev CI image rocm/sgl-dev:v0.5.15.post1-rocm700-mi35x-20260721 (log 3910-3911); no AITER_COMMIT_OVERRIDE.
• Startup watchdog/health warnings were caused by long aiter JIT kernel builds (finish build [...], cost 50.8s etc., logs 6707/6759/6784) and self-resolved before the eval ran.

Hypothesised Causes (with confidence)

• [MEDIUM] Borderline threshold + run-to-run GSM8K sampling/fp8-KV numeric variance, not a code regression.
  • Supporting: passing runs cleared the bar by only 0.006-0.008; a 0.008-0.010 downward wobble is enough to flip pass→fail. invalid=0.000, no errors, full completion — output quality is essentially unchanged. fp8_e4m3 KV cache + aiter attention introduces nondeterminism that easily moves a 1319-question score by a few tenths of a percent.
  • Disconfirming: only one data point below threshold so far — a genuine small regression from a Jul 20→21 commit cannot yet be excluded without a rerun.
  • How to verify: re-run nightly-4-gpu-mi35x-minimax-m25 2-3× at head 9a6c96083; if it oscillates around 0.93, this is a threshold-too-tight flake.
• [LOW] A sglang change in the window 5ab3d90b8..9a6c96083 touching fp8 KV cache / aiter attention numerics.
  • Supporting: the failure lands exactly at the first run after the Jul 20 pass.
  • Disconfirming: the delta is dominated by the marginal-margin pattern above; the workspace is a shallow clone so the full window could not be enumerated here; the HEAD commit (#31918, a Cookbook doc addition) is unrelated to inference.
  • How to verify: enumerate git log 5ab3d90b8..9a6c96083 in a full clone and grep for fp8, kv_cache, aiter, attention; bisect only if reruns confirm a persistent sub-0.93 score.
• [LOW] aiter dev-image kernel change (image dated 20260721).
  • Supporting: aiter kernels JIT-rebuilt this run; MI35x nightly image is re-dated daily.
  • Disconfirming: dev build is still based on g9127c94a1; the accuracy delta is within noise and no numerical/kernel error surfaced.
  • How to verify: compare the aiter dev SHA in the Jul 20 vs Jul 21 image if image provenance is available.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for minimax accuracy (55 hits, none targeting this mi35x threshold) and minimax m25 (2 hits, both unrelated: #27209, #20894).

Suggested Triage Steps

• Re-run nightly-4-gpu-mi35x-minimax-m25 at head 9a6c960832-3× to distinguish a borderline flake from a real regression before any code investigation.
• If scores oscillate around 0.93, consider whether the hardcoded accuracy_threshold=0.93 (test_minimax_m25_tp4_eval_mi35x.py:44,66) has too little headroom for an fp8-KV eval, and widen it or increase num_questions for statistical stability — a human config decision, not a bot directive.
• If reruns stay ≤0.928, enumerate 5ab3d90b8..9a6c96083 in a full clone, grep for fp8/KV/aiter-attention changes, and bisect on MI35x.

Status

• Persistence: 1 run (first failure 2026-07-21 23:32 UTC); passed the prior 4 comparable scheduled runs.
• Scope: 1 job / 1 test (nightly-4-gpu-mi35x-minimax-m25 → test_minimax_m25_tp4_accuracy), MI35x nightly only.
• Blocked work: MiniMax-M2.5 TP4 fp8-KV accuracy signal on MI35x nightly.
• In-flight fix: ❌ none found in open PRs.

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-07-22 20:09 UTC)

[amd-bot]

pr-test-amd.yml — Cross-Run Pattern Summary (4 runs in lookback window)

Last scanned (UTC): 2026-07-22 19:36 UTC

Agent assessment

Persistent clusters

1. Multimodal-gen diffusion suite hang / chronic red — 4 jobs, every run in window (Never-passed).

• Jobs (latest run): 1-gpu part 0, 1-gpu part 1, 1-gpu part 3, 2-gpu part 0.
• 1-GPU shards: diffusion step hangs → job killed at ~70-90 min, no log/artifacts uploaded (BlobNotFound). 2-GPU shard: heterogeneous (hang, HIPBLAS_STATUS_NOT_SUPPORTED, FLUX.2 fp8 case + HF-download errors).
• First observed ≥ 2026-07-14 (earliest queryable); no green run in the 15-run lookback. Not tied to any in-window commit.
• Top hypothesis: [LOW] runner/GPU hang or runner-agent loss during the diffusion step (log never flushed). Disconfirming: without logs an in-test crash cannot be excluded. In-flight fix: ❌ none.

Regression candidates (latest completed run only)

Both fail on 8-GPU shards and both are caused by #31816 (745b2ca4, "read parallel config leaves via get_parallel()").

R1 — stage-c-test-large-8-gpu-amd-mi35x (…, 1) · test/registered/amd/test_deepseek_r1_mxfp4_8gpu.py (server launch / all TestDeepseekR1MXFP4* blocked).

• torch._dynamo.exc.Unsupported: __getattribute__ of ParallelContext during tc_piecewise prefill CUDA-graph capture. Passed 3/3 in prior run 29830494225; commit landed Jul 22 01:18 in window. [HIGH] — modified line dsa/utils.py:79 is the top user frame.

R2 — stage-c-test-large-8-gpu-amd (…, 3) · test/registered/ops/test_aiter_allreduce_fusion_amd.py::test_dense_tp_fuses.

• AttributeError: module 'sglang.srt.layers.communicator' has no attribute 'get_server_args' — the refactor removed the symbol the test still patches. Regression window 8ae0eb83...b13abfde. [HIGH].

Resolution already on main: #32100 (f5dcbe8f, "Revert RuntimeContext config-namespace reads/roles #31813–#31817") reverts #31816 and restores communicator.get_server_args + the dsa/utils.py hunk. Verified: culprit 745b2ca4 is in the failing run's tree b13abfde, and the revert landed after it. No new PR needed — just confirm the next pr-test-amd run is green for both files.

Flakes / intermittent

• stage-b-…-mi35x-disaggregation-amd — test_disaggregation_pp.py KeyError: 'answer' in gsm8k on PD+PP disagg. ~4/9 recent runs (2/4 in window: 29830494225, 29880220845); failing function rotates across the 3 PP variants → investigate (dropped generation on PP path).
• stage-b-test-1-gpu-large-amd — test_online_lora_latency_with_concurrent_adapter_updates ~11-15 s median vs 4000 ms threshold. 2/4 window → retry/quarantine (marginal perf gate).
• stage-b-test-1-gpu-small-amd-mi35x — test_quark_mxfp4.py::TestFP8ToMXFP4MOETP1.test_gsm8k, accuracy 0.912-0.922 vs strict assertGreater(0.92). 1/4 window (new test) → investigate threshold margin ([MEDIUM] zero-margin gate).
• stage-b-test-1-gpu-small-amd-nondeterministic — test_reward_models.py AttributeError: '…' object has no attribute '_legacy_load_weights'. Bucketed flaky (only run 29880220845), but analysis says it's a deterministic [HIGH] regression from becf252e6 (#28671); it did not re-appear in the latest run → confirm whether it still fails on b13abfde.
• stage-c-test-large-8-gpu-amd-mi35x (…, 0) (run 29830494225) — per-job analysis produced no output (infra/timeout). 1/4 window → manual triage.

Suggested next actions

1. Verify the revert cleared R1+R2 — check the next pr-test-amd run after f5dcbe8f is green for test_deepseek_r1_mxfp4_8gpu.py and test_aiter_allreduce_fusion_amd.py; no new fix PR required.
2. Confirm reward-model regression — re-run test_reward_models.py on b13abfde; if it still fails, it needs a fix for #28671 (reverts did not touch llama.py load_weights dispatch).
3. Multimodal diffusion (biggest cluster) — attach py-spy/rocm-smi to a re-run of the 1-GPU/2-GPU diffusion shards to capture the hang before the runner is killed; logs are currently lost.
4. Disagg KeyError: 'answer' — investigate empty/dropped generation on the PP disaggregation path (intermittent, not a threshold assert).
5. Boundary gates — review the zero-margin assertGreater(0.92) MXFP4 gate and the 4000 ms LoRA-latency gate for realistic margins.

---

Runs analysed

Run	Started (UTC)	Failed jobs
29790347494	2026-07-21 05:07	5
29830494225	2026-07-21 12:43	6
29880220845	2026-07-22 00:26	8
29920027108	2026-07-22 13:53	6

Persistent failures (in every run)

• multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 0) — in 4/4 runs (0347494, 0494225, 0220845, 0027108)
• multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 1) — in 4/4 runs (0347494, 0494225, 0220845, 0027108)
• multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 3) — in 4/4 runs (0347494, 0494225, 0220845, 0027108)
• multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 0) — in 4/4 runs (0347494, 0494225, 0220845, 0027108)

Latest-only failures (potential regressions)

• stage-c-test-large-8-gpu-amd (linux-mi325-8gpu-sglang, 3) — in 1/4 runs (0027108)
• stage-c-test-large-8-gpu-amd-mi35x (linux-mi35x-gpu-8, 1) — in 1/4 runs (0027108)

Flaky / intermittent failures

• stage-b-test-1-gpu-large-amd (linux-mi325-1gpu-sglang, 0) — in 2/4 runs (0347494, 0220845)
• stage-b-test-1-gpu-small-amd-mi35x (linux-mi35x-gpu-1) — in 1/4 runs (0220845)
• stage-b-test-1-gpu-small-amd-nondeterministic (linux-mi325-1gpu-sglang) — in 1/4 runs (0220845)
• stage-b-test-large-8-gpu-mi35x-disaggregation-amd (linux-mi35x-gpu-8.fabric) — in 2/4 runs (0494225, 0220845)
• stage-c-test-large-8-gpu-amd-mi35x (linux-mi35x-gpu-8, 0) — in 1/4 runs (0494225)

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-07-22 19:36 UTC)

[amd-bot]

nightly-test-amd.yml — Per-Job Analysis (continued 2/3)

nightly-accuracy-2-gpu-vlm — failed step(s): Nightly Accuracy Test (2-GPU VLM MMMU)

Commit Info

• sglang: 9a6c9608
• aiter: 9127c94a (v0.1.17.dev110+g9127c94a1, source: dev image, matches Dockerfile default — AITER_COMMIT_OVERRIDE empty)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi30x/test_vlms_mmmu_eval_amd.py	test_mmmu_vlm_models (TestNightlyVLMMmmuEvalAMD)	AssertionError: The following models failed: — 5 of 11 VLMs below MMMU threshold	Log

The single test function iterates all 11 models and fails if any is below its per-model threshold. The 5 sub-threshold models in this run:

Model	TP	Score	Threshold	Gap
Qwen/Qwen2-VL-7B-Instruct	1	0.2799	0.30	−0.02
Qwen/Qwen2.5-VL-7B-Instruct	1	0.2802	0.33	−0.05
Qwen/Qwen3-VL-30B-A3B-Instruct	2	0.2601	0.29	−0.03
openbmb/MiniCPM-o-2_6	1	0.2802	0.32	−0.04
deepseek-ai/deepseek-vl2-small	1	0.2802	0.31	−0.03

(6 models passed: InternVL2_5-2B, MiniCPM-v-2_6, Janus-Pro-7B, Kimi-VL-A3B, MiMo-VL-7B-RL, DeepSeek-OCR-2.)

Failure Summary

test_mmmu_vlm_models failed its final _check_results assertion (test_vlms_mmmu_eval_amd.py:380) because 5 of 11 VLMs scored below their configured MMMU thresholds on 100 samples. All 11 model servers started and ran the eval to completion (100/100 questions each) — there was no crash, OOM, or server error. This is purely an accuracy-threshold gate failure.

Regression Status

Never-passed (no green run in queryable window) — persistent, deterministic accuracy gap.

Recent history of test_vlms_mmmu_eval_amd.py (test_mmmu_vlm_models) in job nightly-accuracy-2-gpu-vlm (comparable completed scheduled runs: branch=main, event=schedule; cancelled runs excluded as the job never produced a result):

Date	Run	Test File Status	Failed Models	Note
Jul 22	29855050299	❌ Failed	5 (Qwen2-VL, Qwen2.5-VL, Qwen3-VL-30B, MiniCPM-o, deepseek-vl2)	this run
Jul 19	29697529176	❌ Failed	same 5	scores identical
Jul 18	29654601256	❌ Failed	same 5	scores identical
Jul 17	29601593052	❌ Failed	same 5	scores ~identical
Jul 16	29521595306	❌ Failed	4 (Qwen2-VL passed @0.2799↑)	borderline flip
Jul 15	29438351311	⏭️ cancelled	—	excluded
Jul 14	29355460805	❌ Failed	same 5	scores identical
Jul 13	29273241962	❌ Failed	same 5	scores identical

No passing run for this test file exists in the queryable completed-run window (Jul 13–22). Scores are strikingly deterministic across runs (Qwen2-VL always 0.2799, Qwen2.5-VL 0.2802, Qwen3-VL-30B 0.2601, deepseek-vl2 0.2802), i.e. not flaky — a stable accuracy gap, not noise. Only Qwen2-VL-7B is borderline (0.2799 vs 0.30) and flipped to pass once (Jul 16).

Failure Cluster

VLM MMMU accuracy below threshold (AMD MI30x) — 5 stable sub-threshold models.

Facts (verified observations)

• Assertion raised at test_vlms_mmmu_eval_amd.py:380 via _check_results (:314); job exit code 1 → runner exit 255, not retried ("Exit code 255 indicates test failure (not network issue)").
• All 11 servers launched and evaluated 100/100 MMMU questions; no crash/OOM in the log for the failing models.
• The same 5 models fail with near-identical scores across every completed scheduled run Jul 13–22 (verified from downloaded historical job logs 86895900692, 87161842084, 87843874476, 87954757775, 88106576112, 88220738844).
• Thresholds are hard-coded per-model in MODEL_CONFIG (:44–123), e.g. Qwen2.5-VL=0.33, MiniCPM-o=0.32, deepseek-vl2=0.31. git blame attributes these lines to boundary commit 99f5a6f4 (2026-07-18) but that is the shallow-clone boundary, not necessarily the introduction commit — origin is not reliably datable from this checkout, and the failure predates it (identical failure Jul 13).
• aiter is unchanged (dev image matches Dockerfile default; no override) across the window.
• Passing models also score low in absolute terms (0.29–0.36), suggesting the whole VLM+MMMU-100-sample setup on MI30x yields low scores generally.

Hypothesised Causes (with confidence)

• [MEDIUM] Per-model thresholds are miscalibrated (too high) for the MI30x + 100-sample MMMU setup.
  • Supporting: scores are deterministic and consistently 0.02–0.05 below threshold for the same 5 models across ≥6 runs; passing models also cluster low (0.29–0.36); file comment claims "conservative thresholds" yet 5/11 fail every run; an (unrelated, stale) PR #26178 exists titled "[AMD] WIP - relaxing thres for mi300", indicating threshold miscalibration is a known AMD theme.
  • Disconfirming: cannot see the threshold's introduction commit (shallow clone), so I can't prove these thresholds ever corresponded to a green run; if a real accuracy regression landed before Jul 13 it would look identical.
  • How to verify: run the same 11 models on the MI30x image at an older sglang SHA (pre-Jul 13) and compare scores; if scores match today's, the thresholds — not code — are the gate.
• [LOW] Systematic VLM accuracy/quality degradation on AMD (e.g. image preprocessing, RoPE precision, or attention backend producing worse outputs on ROCm).
  • Supporting: log shows Aiter backend is selected for fused RoPE. This has lower precision. (fused_rope.py:49) and AITER_USE_SYSTEM_TRITON=1 ... aiter gluon kernels require triton>=3.6.0, found 3.4.0 ... unexpected errors may occur; overall low scores across all VLMs.
  • Disconfirming: 6 models pass their thresholds fine; scores are stable not degrading; no numerical-error/NaN evidence in the log. Symptom-level only.
  • How to verify: compare MI30x scores against the NVIDIA VLM MMMU nightly for the same models; a large AMD-vs-NVIDIA gap would support a backend-quality issue rather than threshold calibration.
• No hypothesis reaches HIGH; a code-level root cause cannot be pinned without a green baseline SHA. Needs the A/B baseline run above.

In-flight Fix Check

• In-flight fix: ❌ none found directly targeting this test. Searched open PRs for MMMU+VLM, vlm+accuracy+threshold, Qwen2.5-VL+accuracy. The closest hit, #26178 "[AMD] WIP - relaxing thres for mi300", is a stale WIP (last updated 2026-05-26) that modifies test/registered/perf/test_vlm_perf_5090.py, not test_vlms_mmmu_eval_amd.py — does not fix this failure.

Suggested Triage Steps

• Confirm whether this is calibration or regression: re-run test_vlms_mmmu_eval_amd.py for the 5 failing models on the MI30x image at a pre-Jul-13 sglang SHA; if scores are unchanged (~0.26–0.28), the thresholds are the gate and should be re-baselined (owner decision), not the code.
• Cross-check the same 5 models on the NVIDIA VLM MMMU nightly to quantify any AMD-vs-NVIDIA accuracy gap; a large gap points to a ROCm backend-quality issue (investigate fused RoPE aiter precision / triton 3.4 vs required 3.6).
• If calibration is confirmed, open a PR adjusting the per-model accuracy_threshold values in test_vlms_mmmu_eval_amd.py:44–123 (or mark the borderline Qwen2-VL-7B), referencing observed stable scores — do NOT simply lower to hide a real regression until the baseline check above is done.

Status

• Persistence: Failing every completed scheduled run in the queryable window (Jul 13–22, ≥6 runs); no green run found for this test file. Deterministic, not flaky.
• Scope: 1 job (nightly-accuracy-2-gpu-vlm) in nightly-test-amd.yml; 5 of 11 VLMs affected (Qwen2-VL-7B borderline). Blocks the AMD nightly VLM accuracy signal.
• In-flight fix: ❌ none found targeting this test (see In-flight Fix Check).

nightly-accuracy-2-gpu — failed step(s): N/A

Commit Info

• sglang: 9a6c96083ffa40a998174048f5353e3099db1243
• aiter: 9127c94a18e4398e1eba91f6639e910f0994ad02 (v0.1.17.dev110+g9127c94a1) — source: dev image preserved (Dev/patched version detected → skipping rebuild; Dockerfile default matches; AITER_COMMIT_OVERRIDE empty). Verified via [CI-AITER-CHECK] markers at job log.

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi30x/test_gsm8k_eval_amd.py	TestNightlyGsm8KEval.test_gsm8k_all_models	Job/step terminated abruptly during mistralai/Mixtral-8x7B-Instruct-v0.1 gsm8k eval — no summary emitted, step 6 conclusion None, job failure	Log

This is a non-assertion failure: the process is killed mid-evaluation, so check_model_scores (which raises the only AssertionError) never runs and no summary table is produced.

Failure Summary

The nightly-accuracy-2-gpu job builds and runs successfully, and 4 models evaluate before the failure: Llama-3.1-8B ✅ (0.811), Qwen3-8B ✅ (0.913), Qwen3-4B ❌ (cosmetic — see below), Llama-3.1-70B ✅ (0.928). The suite then launches mistralai/Mixtral-8x7B-Instruct-v0.1 (TP=2), which starts serving and begins gsm8k decode healthily (15028 tok/s, #running-req: 710 at log line 18274). All log output then stops abruptly at 01:12:46, ~2 min into the Mixtral eval. The job remains silent for ~19 min and is marked failure at 01:31:34. Qwen2-57B-A14B (next in the TP2 group) and all FP8/AMD-specific model groups never run.

⚠️ Qwen3-4B "❌ FAILED" is a red herring, not the cause. Qwen/Qwen3-4B scores 0.912 but is absent from MODEL_SCORE_THRESHOLDS, so at test_gsm8k_eval_amd.py:289 passed = threshold and score >= threshold evaluates to None → the per-model loop prints ❌ FAILED. However check_model_scores (test_gsm8k_eval_amd.py:165-177) treats a missing threshold as ⚠️ NO THRESHOLD and does not add it to failed_models, so it would not fail the test. This cosmetic mislabel is present identically in every historical run.

Regression Status

Known recurring failure — Never-completes (no passing/complete run found in the queryable window).

Recent history of test_gsm8k_eval_amd.py (test_gsm8k_all_models) in job nightly-accuracy-2-gpu (most recent comparable completed scheduled runs: branch=main, event=schedule; today's run excluded from "completed" as its overall workflow is still queued):

Date	Run	Job dur	Test File Status	Failure point	Error
Jul 22	29855050299	~59m	❌ Killed	during Mixtral-8x7B eval	abrupt termination, step conclusion None
Jul 19	29697529176	68m	❌ Killed	during Mixtral-8x7B eval	same signature
Jul 18	29654601256	70m	❌ Killed	during Mixtral-8x7B eval	same signature
Jul 17	29601593052	64m	❌ Killed	during Mixtral-8x7B eval	same signature
Jul 16	29521595306	54m	❌ Failed	(step not log-verified)	job failure
Jul 14	29355460805	81m	❌ Failed	(step not log-verified)	job failure
Jul 13	29273241962	58m	❌ Failed	(step not log-verified)	job failure

Log-verified identical Mixtral-hang signature on 4 consecutive runs (Jul 17, 18, 19, 22). No green or fully-completed run of this job exists in the last 15 scheduled runs (all failure/cancelled). The aiter version is unchanged (v0.1.17.dev110+g9127c94a1) across the entire Jul 17–22 window.

Failure Cluster

Job aborts during Mixtral-8x7B-Instruct-v0.1 gsm8k evaluation (silent hang → external kill, no test summary)

Facts (verified observations)

• Job nightly-accuracy-2-gpu: status=completed, conclusion=failure; started 00:32:31Z, completed 01:31:34Z (~59 min). Step 6 "Nightly Test (2-GPU)" and step 12 "Post Checkout code" both have conclusion=None.
• Command: run_suite.py --hw amd --suite nightly-amd --nightly --timeout-per-file 7200 --continue-on-error (log). File timeout = 7200s (120 min).
• The job has no timeout-minutes in nightly-test-amd.yml:239-268 (defaults to 360 min). Neither the 360-min job default nor the 120-min per-file timeout can explain a ~59-min death.
• Last log line: 01:12:46 mid-Mixtral decode, healthy (gen throughput: 15028.85 token/s, #running-req: 710). ~19 min of silence follows before the job is marked failed. No ##[error], no AssertionError, no watchdog/health-check message, no summary table anywhere in the log.
• Runner: linux-mi325-2gpu-sglang (self-hosted AMD MI325, 2-GPU).
• 4 models complete before the failure (verified from log). Reproduced with the same signature and same aiter version on Jul 17/18/19/22.
• Job runtimes vary 54–81 min across the failing runs — inconsistent with a fixed CI timeout.

Hypothesised Causes (with confidence)

• [MEDIUM] The Mixtral-8x7B-Instruct-v0.1 (TP=2) gsm8k evaluation hangs (decode stalls / server becomes unresponsive), and the job is later terminated externally (self-hosted runner / container / GPU-hang watchdog).
  • Supporting: 4/4 log-verified runs stop mid-Mixtral-eval at the same point in the model sequence; output ceases entirely (both client tqdm and server decode logs) while decode was healthy; ~15–19 min silence then a non-clean failure with step conclusion None; variable job duration argues for an infra kill, not a deterministic timeout.
  • Disconfirming: server-side crash/OOM/GPU-fault stack traces are NOT present in the job-page log — cannot confirm a hang vs a hard runner/container death from this log alone; the authoritative server logs are not in the job-page log.
  • How to verify: reproduce Mixtral-8x7B-Instruct-v0.1 gsm8k eval (TP=2, --num-threads 1024, full 1319 examples) standalone on an MI325 2-GPU runner; capture the sglang server log + dmesg/GPU state at the point output stops.
• [LOW] aiter bf16 GEMM path (using torch solution:0, no tuned config) involved in the Mixtral decode contributes to a stall.
  • Supporting: the last log lines before the hang are aiter bf16-gemm fallback messages during Mixtral decode.
  • Disconfirming: log-line proximity is not causation; the aiter version is identical across all failing runs (v0.1.17.dev110+g9127c94a1), and the same fallback path runs fine for Llama-3.1-70B (TP=2) which passes immediately before Mixtral.
  • How to verify: A/B the Mixtral eval with a different attention/gemm backend, or with SGLANG_USE_AITER=0, on the same runner.
• [LOW] Runner/infra instability (self-hosted MI325 pool) independent of the model.
  • Supporting: conclusion=None steps + abrupt total-output cutoff are classic signs of a runner losing the job; the whole nightly-test-amd workflow shows many cancelled runs.
  • Disconfirming: the death is consistently localized to the Mixtral eval across 4 runs, which is too specific for random infra flakes.
  • How to verify: check runner health / bypass-rerun the job pinned to a different MI325 host.

No hypothesis reaches HIGH: the job-page log lacks server-side crash evidence, so hang-vs-kill cannot be disambiguated without the sglang server logs.

In-flight Fix Check

• In-flight fix: ❌ none found for the Mixtral hang / abrupt-termination. Searched open PRs for Mixtral+nightly (2 results: #29933 gsm8k answer-extraction — unrelated; #29677 AMD extend-attention perf — unrelated) and nightly+accuracy+AMD (no hang/termination fix; #26730 "run full GSM8K" touches this test area but does not address the hang).
• Separately, the Qwen3-4B cosmetic ❌ FAILED mislabel (missing threshold) has no open fix PR — worth a one-line fix (add a Qwen/Qwen3-4B threshold or guard the passed print for no-threshold models).

Suggested Triage Steps

• Reproduce Mixtral-8x7B-Instruct-v0.1 gsm8k eval standalone (TP=2, MI325 2-GPU, container v0.1.17.dev110+g9127c94a1 aiter) with sglang server log + dmesg capture to determine hang vs crash vs runner death.
• Add an explicit timeout-minutes (and/or a per-model eval HTTP timeout in test_gsm8k_eval_amd.py) so a stalled Mixtral eval fails fast with a diagnosable error instead of a 19-min silent hang → opaque kill.
• If Mixtral is confirmed to hang: try SGLANG_USE_AITER=0 or an alternate attention backend for Mixtral only (A/B), and consider temporarily moving mistralai/Mixtral-8x7B-Instruct-v0.1 to failing_models in test_gsm8k_eval_amd.py:69 to unblock the rest of the accuracy matrix (FP8 + AMD groups currently never run).
• Fix the Qwen/Qwen3-4B cosmetic FAILED (missing entry in MODEL_SCORE_THRESHOLDS).

Status

• Persistence: nightly-accuracy-2-gpu failing every completed scheduled run in the queryable window; the Mixtral-eval abrupt-termination signature is log-verified on 4 consecutive runs (Jul 17, 18, 19, 22). No green/complete run found in the last 15 scheduled runs.
• Scope: 1 job (nightly-accuracy-2-gpu) on nightly-test-amd.yml, runner linux-mi325-2gpu-sglang. The failure aborts the accuracy matrix at the 2nd TP2 model, so Qwen2-57B-A14B, all FP8 groups, and AMD-specific groups get zero accuracy signal nightly.
• In-flight fix: ❌ none found in open PRs.

nightly-test-1-gpu-kernel — failed step(s): Nightly Kernel Benchmark (1-GPU)

Commit Info

• sglang: 9a6c9608
• aiter: 9127c94a — v0.1.17.dev110+g9127c94a1 (source: dev/patched image build; the CI image carried a dev build matching the Dockerfile default, so no rebuild — [CI-AITER-CHECK] markers, job log)

Failed Tests

Job: nightly-test-1-gpu-kernel · suite nightly-amd-kernel-1-gpu · file result: 12 failed, 4 passed

Test File	Test Function	Error	Log
test/registered/jit/test_transfer_mamba.py	test_mamba_kernel_backup_load_roundtrip (×4: page_first/page_first_direct × float16/bfloat16)	NameError: name 'transfer_kv_mamba_lf_pf' is not defined	Log
test/registered/jit/test_transfer_mamba.py	test_mamba_kernel_single_item (×4)	NameError: name 'transfer_kv_mamba_lf_pf' is not defined	Log
test/registered/jit/test_transfer_mamba.py	test_mamba_kernel_full_indices (×4)	NameError: name 'transfer_kv_mamba_lf_pf' is not defined	Log

(The 4 test_mamba_kernel_empty_indices variants passed — they early-return at if src_indices.numel() == 0: return before reaching the undefined symbol.)

Failure Summary

The nightly-amd-kernel-1-gpu suite ran test/registered/jit/test_transfer_mamba.py and 12 of 16 items failed with NameError: name 'transfer_kv_mamba_lf_pf' is not defined, raised inside MambaPoolHost._copy_tensor_all_layers_lf_pf at python/sglang/srt/mem_cache/memory_pool_host.py:407. All failing variants exercise the io_backend="kernel" mamba KV-cache transfer path; the only passing variants are the empty-index cases that return before the kernel call. This single failing test file caused the job to exit 255.

Regression Status

Never-passed on AMD — the test file has failed in every completed scheduled run since it was introduced.

Recent history of test/registered/jit/test_transfer_mamba.py (test_mamba_kernel_*) in job nightly-test-1-gpu-kernel (most recent comparable completed scheduled runs: branch=main, event=schedule; cancelled runs 2026-07-20 [29766761021] and 2026-07-15 [29438351311] excluded):

Date	Run	Job	Test File Status	Failed Function	Error
Jul 21	29855050299	kernel	❌ Failed (12/16)	test_mamba_kernel_*	NameError transfer_kv_mamba_lf_pf
Jul 19	29697529176	kernel	❌ Failed (12/16)	test_mamba_kernel_*	NameError transfer_kv_mamba_lf_pf
Jul 18	29654601256	kernel	❌ Failed (12/16)	test_mamba_kernel_*	NameError transfer_kv_mamba_lf_pf
Jul 17	29601593052	kernel	❌ Failed (12/16)	test_mamba_kernel_*	NameError transfer_kv_mamba_lf_pf
Jul 16	29521595306	kernel	❌ Failed (12/16)	test_mamba_kernel_*	NameError transfer_kv_mamba_lf_pf (first appearance)
Jul 14	29355460805	kernel	✅ Passed (test absent)	—	file not yet introduced
Jul 13	29273241962	kernel	✅ Passed (test absent)	—	file not yet introduced

First observed failure: 2026-07-16 (run 29521595306, same NameError, 12 occurrences verified in log). The test file did not exist on 2026-07-13/07-14 (0 references in those logs). There is no green run of this test file on AMD — it was born broken on ROCm.

Failure Cluster

NameError: transfer_kv_mamba_lf_pf undefined on ROCm (mamba HiCache kernel transfer)

Facts (verified observations)

• Verbatim error: NameError: name 'transfer_kv_mamba_lf_pf' is not defined at python/sglang/srt/mem_cache/memory_pool_host.py:407, reached via backup_from_device_all_layer → _copy_tensor_all_layers_lf_pf (job log FAILURES block).
• In the current source, the mamba kernel import is gated CUDA-only: memory_pool_host.py:41 if _is_cuda: imports transfer_kv_mamba_lf_pf / transfer_kv_mamba_pf_lf from sglang.jit_kernel.transfer_mamba. The _is_cuda or _is_hip block above it (lines 31–40, from sgl_kernel.kvcacheio) does not include the mamba symbols. So on AMD (_is_hip=True, _is_cuda=False) neither name is ever bound. [FACT]
• The code path is reachable on AMD: _copy_tensor_all_layers_lf_pf calls transfer_kv_mamba_lf_pf(...) unconditionally inside the io_backend == "kernel" branch (memory_pool_host.py:407), with no CUDA guard.
• Blast radius is one test file (12 subtests). All other kernel-suite files in this job passed (e.g. test_fused_verify_triton_gdn.py 12 passed, test_fused_metadata_copy.py 42 passed, plus 90/18/29/26/32/14/5/9/96/118 passed elsewhere). [FACT]
• Reproduced identically across 5 consecutive completed scheduled runs (2026-07-16 → 07-21), same 12 functions, same error string.

Hypothesised Causes (with confidence)

• [FACT] The mamba JIT transfer kernel import is gated behind if _is_cuda: (not _is_cuda or _is_hip) at python/sglang/srt/mem_cache/memory_pool_host.py:41, while the call site at line 407 runs for any io_backend="kernel" including on ROCm. On AMD the name is never bound → NameError. This is directly observable in both the source and the stack trace.
  • Supporting: import block (lines 41–45) is CUDA-only; call site (line 407) is unguarded; the passing empty_indices variants prove the failure is at the kernel call, not setup.
  • Disconfirming: (none found) — the fix PR below independently confirms the same diagnosis.
  • How to verify: already verified by reading source + log; the fix PR #31741 changes exactly this gate.
• [MEDIUM] Introduced by #30535 "[hicache]: add mamba_io_kernel" (merged 2026-07-16, merge commit b296e1a5) — it added both the test file and the CUDA-only kernel path with no ROCm coverage.
  • Supporting: test file first appears in the 2026-07-16 run (absent 07-13/07-14); merge timing aligns exactly; PR title matches the feature.
  • Disconfirming: local repo is a shallow clone (grafted boundary at c68392c5), so git blame could not directly attribute lines 41/407 — attribution rests on run-history timing + the merged-PR search, not a local diff.
  • How to verify: git show b296e1a5 -- python/sglang/srt/mem_cache/memory_pool_host.py on a full clone to confirm it added the if _is_cuda: gate.

In-flight Fix Check

• In-flight fix: ✅ #31741 "[AMD] Enable mamba JIT transfer kernel on ROCm (fix transfer_kv_mamba NameError)" — open since 2026-07-20, not draft, mergeable_state: blocked (awaiting review/checks as of 2026-07-21 03:23 UTC). Touches exactly the two relevant files: python/sglang/srt/mem_cache/memory_pool_host.py (+1/-1, the import gate) and python/sglang/jit_kernel/csrc/kvcacheio/transfer_mamba.cuh (+3/-3, HIP compat). Chase reviewers — do NOT open a duplicate. (Search: is:pr is:open transfer_kv_mamba.)

Suggested Triage Steps

• Verify #31741 merges cleanly and re-run nightly-test-1-gpu-kernel; confirm all 16 test_transfer_mamba.py items pass on AMD MI30x.
• If the HIP JIT kernel (transfer_mamba.cuh) does not build/run on ROCm, an interim option is to broaden the guard at memory_pool_host.py:41 to if _is_cuda or _is_hip: only after confirming the .cuh compiles under HIP — otherwise the import would still fail. #31741 already pairs both changes, so prefer landing it whole.
• Confirm the fix does not regress NVIDIA: the same suite path (io_backend="kernel") must still pass on CUDA.

Status

• Persistence: Failing for 6 days (since 2026-07-16, run 29521595306); every completed scheduled nightly-test-1-gpu-kernel run since has the identical 12-item failure. Never passed on AMD.
• Scope: 1 job (nightly-test-1-gpu-kernel), 1 test file (test_transfer_mamba.py), 12 subtests; AMD/ROCm only (CUDA-gated import means NVIDIA is unaffected).
• Blocked work: Mamba HiCache io_backend="kernel" D2H/H2D transfer is unusable on ROCm; blocks AMD kernel-suite nightly signal for this file.
• In-flight fix: ✅ #31741 (open since 2026-07-20, blocked/awaiting merge) — ping reviewers.

nightly-accuracy-8-gpu — failed step(s): Accuracy Test (8-GPU Grok1-FP8)

Commit Info

• sglang: 9a6c9608
• aiter: 9127c94a — v0.1.17.dev110+g9127c94a1 (source: dev image preserved; AITER_COMMIT_OVERRIDE empty, matches Dockerfile default, rebuild skipped)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi30x/test_grok1_fp8_eval_amd.py	GSM8K eval (whole-file run, no test-fn granularity emitted)	HW Exception by GPU node-9 … reason :GPU Hang → Fatal Python error: Aborted → exit 134	Log

Step 6 Accuracy Test (8-GPU GPT-OSS) ✅ passed (gpt-oss-20b acc=0.855, gpt-oss-120b acc=0.855). Step 7 Accuracy Test (8-GPU Grok1-FP8) ❌ failed. No Grok1 accuracy score was ever produced — the process aborted during the GSM8K eval.

Failure Summary

The 8-GPU Grok1-FP8 accuracy step crashed with a hardware GPU hang: HW Exception by GPU node-9 … reason :GPU Hang on all 8 ranks, producing Fatal Python error: Aborted and terminating the step with exit code 134 (SIGABRT). Before the crash the server showed a long, unstable startup — soft scheduler/tokenizer watchdog timeouts during CUDA-graph capture and repeated detokenizer health-check failures — then briefly recovered, launched the eval, and hung almost immediately at 0/200.

Regression Status

New regression (candidate hardware flake — see Hypothesised Causes).

Recent history of test_grok1_fp8_eval_amd.py in job nightly-accuracy-8-gpu (most recent comparable completed scheduled runs: branch=main, event=schedule; grok1 checked at test-file level, not job level):

Date	Run	Job	Test File Status	Failed Function	Error
Jul 22	29855050299	nightly-accuracy-8-gpu	❌ Failed	GSM8K eval	GPU Hang → Aborted (exit 134)
Jul 20	29766761021	—	⏭️ Cancelled	—	run cancelled, excluded
Jul 19	29697529176	nightly-accuracy-8-gpu	✅ Passed	—	accuracy=0.830 (thr 0.8)
Jul 18	29654601256	nightly-accuracy-8-gpu	✅ Passed	—	job success
Jul 17	29601593052	nightly-accuracy-8-gpu	✅ Passed	—	job success
Jul 16	29521595306	nightly-accuracy-8-gpu	✅ Passed	—	job success
Jul 15	29438351311	—	⏭️ Cancelled	—	run cancelled, excluded
Jul 14	29355460805	nightly-accuracy-8-gpu	✅ Passed	—	job success
Jul 13	29273241962	nightly-accuracy-8-gpu	✅ (grok1)	—	job ❌ but failure was test_gpt_oss_eval_amd.py, not grok1

First observed grok1 failure: 2026-07-22. Last known grok1 pass: 2026-07-19 (accuracy=0.830). GPU-hang signature is absent from the 7/19 and 7/13 logs (checked). The overall run 29855050299 is still queued at the run level, but this job itself is completed/failure.

Failure Cluster

GPU Hang (HW Exception, node-9) during Grok1-FP8 8-GPU eval → SIGABRT

Facts (verified observations)

• Exit: ##[error]Process completed with exit code 134. (job log). Crash message: HW Exception by GPU node-9 (Agent handle: …) reason :GPU Hang on all 8 ranks → Fatal Python error: Aborted.
• Watchdog top-frame at 09:47:50 (soft) was during weight download: ci_download_with_validation_and_retry (ci_weight_validation.py:1955) → snapshot_download — long-load, soft=True, benign.
• 09:50:57 CUDA-graph capture begins; aiter JIT-compiles pa_ragged_* (gqa_ratio=6, head_size=128, bf16, logits_soft_cap_enabled) across ranks.
• 09:52:36–09:52:51 Scheduler watchdog timeout (…=300, self.soft=True) on TP0–TP7 and TokenizerManager watchdog timeout — scheduler stalled during/after capture.
• 09:53:34–09:54:59 /health_generate returns 503; Health check failed. Server couldn't get a response from detokenizer for last 20 seconds.
• 09:55:11 server recovers 200 OK; harness launches the eval server online (mem_fraction_static≈0.7225); weights reload in ~12 s from local snapshot.
• 09:57:20–24 warmup /generate 200 OK, "server is fired up and ready"; 09:57:27 GSM8K eval starts (0/200); 09:57:27–28 GPU Hang → Aborted.
• CI_OFFLINE: Offline launch failed (Server process exited with code 1), retrying with online mode… Online retry succeeded also occurred for the passing gpt-oss-120b earlier — this offline→online retry is a benign recurring pattern, not the failure.
• Runner: linux-mi325-8gpu-sglang-fm4cw-runner-4cdsn (MI325, gfx942).

Hypothesised Causes (with confidence)

• [MEDIUM] Transient GPU/runner hardware hang (infrastructure) on MI325 node 4cdsn.
  • Supporting: symptom is a driver-level HW Exception … reason :GPU Hang on GPU node-9 (not a Python assertion / accuracy miss); grok1 passed the prior 4 nightlies (7/16–7/19) at acc≈0.83 with no code change to the test; the same server instance had earlier soft watchdog timeouts + a 90 s detokenizer stall, indicating the device was already wedged before the eval.
  • Disconfirming: only one occurrence so far — cannot yet distinguish a hardware flake from a genuine regression without a rerun; GPT-OSS on the same 8 GPUs earlier in the job passed cleanly.
  • How to verify: re-run nightly-accuracy-8-gpu (ideally on a different MI325 runner). If grok1 passes → hardware flake on …4cdsn.
• [LOW] sglang/aiter interaction in the Grok1-FP8 path (fp8 fused_moe / pa_ragged aiter kernels) tickling a GPU hang.
  • Supporting: crash follows the CUDA-graph-capture + aiter JIT phase; the hung eval used --quantization fp8 --attention-backend aiter and [fused_moe] using 1stage default … float8_e4m3fnuz.
  • Disconfirming: aiter is unchanged from the 7/19 passing run's Dockerfile default (9127c94a1, dev image both nights); log-line proximity to aiter output is not causation; the hang is a hardware exception, not a kernel launch/compile error.
  • How to verify: after a rerun, if it reproduces, bisect sglang b3a0185c…9a6c9608 for changes touching grok1 / fp8 moe / aiter attention / cuda-graph capture.

In-flight Fix Check

• grok1-specific fix: ❌ none found (is:pr is:open grok1 accuracy → only unrelated Docker/CPU PRs).
• GPU-hang recovery (infra, symptom-matching): ✅ #31068 "[AMD] ci: detect and recover hung GPUs in MI35x pre-flight", ✅ #30431 "ci: self-heal driver-level VRAM leaks + graceful GPU cleanup", ✅ #31229 "ci: bound nvidia-smi in detect_host so a wedged driver fails fast" — these harden CI against exactly this wedged-GPU symptom (though targeted at MI35x pre-flight, not a root-cause fix for this grok1 run). Not duplicates to open.

Suggested Triage Steps

• Re-run nightly-accuracy-8-gpu for commit 9a6c9608; prefer a different MI325 runner than …-4cdsn. If grok1 passes → transient hardware hang, close as flake.
• If it reproduces, capture a py-spy/rocm coredump at the hang and bisect sglang b3a0185c..9a6c9608 for grok1 / fp8-moe / aiter-attention / cuda-graph-capture changes.
• Check whether runner …-4cdsn shows recurring GPU-hang exceptions across other jobs (runner-specific hardware fault); if so, quarantine the node.

Status

• Persistence: First failure 2026-07-22 (this run). Grok1 was green the previous 4 nightlies (last pass 2026-07-19, acc=0.830). Single occurrence so far.
• Scope: 1 job (nightly-accuracy-8-gpu), 1 workflow (nightly-test-amd.yml), 1 runner (MI325 …-4cdsn). GPT-OSS accuracy step in the same job passed.
• Blocked work: Grok1-FP8 8-GPU accuracy signal on AMD nightly.
• In-flight fix: no grok1-specific fix; infra GPU-hang-recovery PRs #31068, #30431, #31229 exist but are not root-cause fixes for this run.

nightly-test-1-gpu-kernel — failed step(s): Nightly Kernel Benchmark (1-GPU)

Commit Info

• sglang: 0bdd4730
• aiter: 9127c94a (v0.1.17.dev110+g9127c94a1, source: dev image — [CI-AITER-CHECK] Dev/patched version detected → skipping rebuild)

Failed Tests

Job nightly-test-1-gpu-kernel — file test/registered/jit/test_transfer_mamba.py returned exit code 1 (12 of 16 parametrized cases failed; the 4 test_mamba_kernel_empty_indices cases passed because they early-return on src_indices.numel()==0).

Test File	Test Function (param)	Error	Log
test/registered/jit/test_transfer_mamba.py	test_mamba_kernel_backup_load_roundtrip [page_first / page_first_direct × dtype0/1]	NameError: name 'transfer_kv_mamba_lf_pf' is not defined	Log
test/registered/jit/test_transfer_mamba.py	test_mamba_kernel_single_item [page_first / page_first_direct × dtype0/1]	NameError: name 'transfer_kv_mamba_lf_pf' is not defined	Log
test/registered/jit/test_transfer_mamba.py	test_mamba_kernel_full_indices [page_first / page_first_direct × dtype0/1]	NameError: name 'transfer_kv_mamba_lf_pf' is not defined	Log

Failure Summary

The AMD kernel nightly's test_transfer_mamba.py fails all 12 non-trivial cases with NameError: name 'transfer_kv_mamba_lf_pf' is not defined, raised inside MambaPoolHost._copy_tensor_all_layers_lf_pf at python/sglang/srt/mem_cache/memory_pool_host.py:407 when invoked via backup_from_device_all_layer(..., io_backend="kernel"). The remaining 20 kernel test files in the shard all passed (Test Summary: 20/21 passed).

Regression Status

Never-passed on AMD (persistent) — NOT a new regression. The same test_transfer_mamba.py NameError appears in every completed scheduled nightly-test-amd kernel job in the query window.

Recent history of test/registered/jit/test_transfer_mamba.py in job nightly-test-1-gpu-kernel (completed event=schedule, branch=main runs):

Date (UTC)	Run	Job	Test File Status	Failed Function	Error
Jul 22	29944275726	kernel	❌ Failed	all 3 mamba fns	NameError transfer_kv_mamba_lf_pf
Jul 22 (early)	29855050299	kernel	❌ Failed	all 3 mamba fns	same
Jul 19	29697529176	kernel	❌ Failed	all 3 mamba fns	same
Jul 18	29654601256	kernel	❌ Failed	all 3 mamba fns	same
Jul 17	29601593052	kernel	❌ Failed	all 3 mamba fns	same
Jul 16	29521595306	kernel	❌ Failed	all 3 mamba fns	same

First observed failure in window: 2026-07-16 (oldest queryable completed run; no passing run found → treat as Never-passed on AMD for the ≥6-day lookback). Persistent for ≥6 days.

Failure Cluster

NameError: transfer_kv_mamba_lf_pf undefined on ROCm (mamba JIT transfer import gated to CUDA)

Facts (verified observations)

• Log (verbatim): E NameError: name 'transfer_kv_mamba_lf_pf' is not defined at ../python/sglang/srt/mem_cache/memory_pool_host.py:407: NameError.
• Source: python/sglang/srt/mem_cache/memory_pool_host.py:40-44 guards the import from sglang.kernels.ops.mamba.transfer_mamba import (transfer_kv_mamba_lf_pf, transfer_kv_mamba_pf_lf) behind if _is_cuda: only — HIP/ROCm (_is_hip) is excluded. The sibling kvcacheio import above (lines 30-39) uses if _is_cuda or _is_hip:.
• Call site memory_pool_host.py:407 (_copy_tensor_all_layers_lf_pf, io_backend=="kernel") calls transfer_kv_mamba_lf_pf unconditionally; transfer_kv_mamba_pf_lf at line 364 has the same exposure.
• Runner platform confirmed ROCm: INFO ... Automatically detected platform rocm. in the historical job logs.
• 12/16 cases fail; the 4 test_mamba_kernel_empty_indices cases pass via the if src_indices.numel()==0: return early-out before the undefined name is referenced (memory_pool_host.py:397-398).
• Reproduced identically in 6 consecutive completed scheduled runs (Jul 16–22), each with 12 NameError occurrences.
• [CI-AITER-CHECK] shows aiter unchanged (dev image preserved) across runs — not an aiter variable.

Hypothesised Causes (with confidence)

• [FACT] The mamba JIT transfer symbols are never imported on ROCm because the import block at memory_pool_host.py:40 is if _is_cuda: (excludes _is_hip), while _copy_tensor_all_layers_lf_pf/backup paths reference them unconditionally on the io_backend="kernel" branch.
  • Supporting: undefined-name error points exactly at line 407; the guarded import at line 40 is CUDA-only; runner is ROCm; sibling kernels one block up use _is_cuda or _is_hip.
  • Disconfirming: (none found) — the error is a deterministic Python NameError, not a numeric/kernel-runtime issue.
  • How to verify: on an AMD ROCm box, python -c "import sglang.srt.mem_cache.memory_pool_host as m; print(hasattr(m,'transfer_kv_mamba_lf_pf'))" → expect False.
• [FACT] Commit 977ea336 (Jul 22, "Kernel Phase 4 batch-2") only renamed the import module (sglang.jit_kernel.transfer_mamba → sglang.kernels.ops.mamba.transfer_mamba) and kept the if _is_cuda: guard — it did NOT introduce the ROCm gap and is NOT the regression trigger.
  • Supporting: git show 977ea336 diff changes only the module path inside the pre-existing if _is_cuda: block; the test already failed identically on Jul 16–19 (before this commit).
  • Disconfirming: (none found).

In-flight Fix Check

• In-flight fix: ✅ #31741 — "[AMD] Enable mamba JIT transfer kernel on ROCm (fix transfer_kv_mamba NameError)", open (non-draft) since 2026-07-20, changes 2 files. ⚠️ mergeable_state: dirty (has merge conflicts) — needs rebase; chase reviewers, do NOT open a duplicate. (Found via search/issues q=repo:sgl-project/sglang+is:pr+is:open+transfer_kv_mamba.)

Suggested Triage Steps

• Rebase/resolve conflicts on #31741 and re-run the nightly-test-1-gpu-kernel job on AMD to confirm it clears all 12 test_transfer_mamba.py cases.
• If #31741 does not land soon, the minimal candidate is to widen the guard at memory_pool_host.py:40 from if _is_cuda: to if _is_cuda or _is_hip: (matching the kvcacheio block above) — provided sglang.kernels.ops.mamba.transfer_mamba builds under HIP; verify the kernel actually exists for ROCm before relying on this.
• If the mamba JIT kernel is genuinely unsupported on ROCm, consider guarding/skip-marking test_transfer_mamba.py::io_backend="kernel" cases on AMD until the kernel is ported, rather than leaving the suite red.

Status

• Persistence: Failing every completed scheduled run for ≥6 days (since 2026-07-16 in the query window; no passing run found).
• Scope: nightly-test-1-gpu-kernel job of nightly-test-amd.yml; 12/16 cases in one test file (rest of the 21-file kernel shard passes).
• Blocked work: Mamba HiCache device↔host kernel-backend transfer path is untested on AMD ROCm.
• In-flight fix: ✅ #31741 (open since 2026-07-20, currently dirty/conflicted — needs rebase).

nightly-2-gpu-mi35x-glm51-mxfp4 — failed step(s): N/A

Commit Info

• sglang: 0bdd4730 — [CI] Fix failures on main (#32091)
• aiter: N/A for this run — the job log blob is unavailable (BlobNotFound, 404), so [CI-AITER-CHECK] markers cannot be read. For reference, the most recent run with a readable log (Jul 16) used aiter v0.1.17.dev110+g9127c94a1 (dev image preserved; Dockerfile default 9127c94a).

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi35x/test_glm51_mxfp4_tp2_gsm8k_mi35x.py	TestGLM51MXFP4TP2GSM8KMI35x.test_gsm8k (server launched in setUpClass)	Not readable for this run — log blob 404 (BlobNotFound). Step never completed.	Job

Failure Summary

Job nightly-2-gpu-mi35x-glm51-mxfp4 failed at ~22 min (started 17:53:19Z, completed failure 18:15:20Z). The only test in its suite is TestGLM51MXFP4TP2GSM8KMI35x.test_gsm8k, which launches an amd/GLM-5.1-MXFP4 TP=2 server on the linux-mi35x-gpu-2 runner. The failing step "Accuracy Test MI35x (2-GPU GLM-5.1-MXFP4 GSM8K)" is stuck at status in_progress while the job conclusion is failure, and the job log is unavailable (BlobNotFound) — the runner was lost mid-step before uploading logs. The step timeout is 120 min and server-launch timeout is 5400 s, so the ~22 min death is neither.

Regression Status

Never-passed (no green run found) — persistent recurring failure of this test file in every comparable completed scheduled run in the queryable window.

Recent history of test_glm51_mxfp4_tp2_gsm8k_mi35x.py (TestGLM51MXFP4TP2GSM8KMI35x.test_gsm8k) in job nightly-2-gpu-mi35x-glm51-mxfp4 (comparable completed scheduled runs: branch=main, event=schedule):

Date	Run	Test File Status	Failure signature	Duration
Jul 22	run	❌ Failed	step stuck in_progress, no log (runner lost)	~22 min
Jul 21	run	❌ Failed	step stuck in_progress, no log (404)	~23 min
Jul 19	run	❌ Failed	step stuck in_progress, no log	~24 min
Jul 18	run	❌ Failed	step stuck in_progress, no log	~22 min
Jul 17	run	❌ Failed	step stuck in_progress, no log	~27 min
Jul 16	run	❌ Failed	TimeoutError: Server failed to start (setUpClass, elapsed 5428s), scheduler watchdog stuck, exit 255	~1h37m
Jul 14	run	❌ Failed	HW Exception … reason :GPU Hang during load_model, Fatal Python error: Aborted, exit 134	~9.5 min
Jul 13	run	❌ Failed	HW Exception … reason :GPU Hang, exit 134	~9 min

No passing run for this test file exists in the queried window (Jul 13–22). First observed failure ≥ Jul 13; no last-known-passing date found.

Failure Cluster

GLM-5.1-MXFP4 TP=2 server fails to initialize on MI35x — GPU Hang / server-start timeout

Facts (verified observations)

• Today's job: step 6 (Accuracy Test MI35x (2-GPU GLM-5.1-MXFP4 GSM8K)) status in_progress, step 12 pending, job conclusion=failure; started 17:53:19Z, ended 18:15:20Z (~22 min). Log blob returns BlobNotFound (404) on repeated fetches.
• Step config (.github/workflows/nightly-test-amd.yml:294-302): timeout-minutes: 120, runs run_suite.py --suite nightly-amd-2-gpu-mi35x-glm51-mxfp4 --nightly --continue-on-error. Server-launch timeout in the test is 5400 s. Neither timeout could fire at 22 min.
• Jul 13 & 14 logs (readable): HW Exception by GPU node-2 (Agent handle: …) reason :GPU Hang followed by First attempt failed with exit code 134 (SIGABRT). Jul 14 shows Fatal Python error: Aborted with the watchdog thread stack inside model_runner.py:841 load_model / model_runner.py:485 initialize — i.e., the GPU hung during model load.
• Jul 16 log (readable): repeated Scheduler watchdog timeout (self.watchdog_timeout=300, self.soft=True) on TP0/TP1, py-spy … failed to get os threadid, then TimeoutError: Server failed to start within the timeout period in setUpClass (elapsed 5428 s), exit 255. Server never became healthy.
• Server args under test: --tp 2 --kv-cache-dtype fp8_e4m3 --disable-radix-cache --dsa-prefill-backend tilelang --dsa-decode-backend tilelang --mem-fraction-static 0.85 on amd/GLM-5.1-MXFP4.
• The test docstring states it exists to catch a "GLM-5.1-MXFP4 TP=2 accuracy drop on MI355X/gfx950 when aiter selected a bad BF16 GEMM path."
• Jul 21 (completed run, 1 day old) also returns BlobNotFound for this job — confirming the missing log is the runner-loss signature, not merely "parent run not finished yet."

Hypothesised Causes (with confidence)

• [MEDIUM] GLM-5.1-MXFP4 TP=2 model initialization hangs the GPU on MI35x/gfx950, crashing the server before/at warmup.
  • Supporting: Jul 13 & 14 explicitly log HW Exception … reason :GPU Hang during load_model → SIGABRT (exit 134); Jul 16 shows the scheduler watchdog wedged and the server never reaching healthy → server-start timeout. The recent ~22 min mid-step deaths with no log upload are consistent with the same hang taking the node down harder (runner agent dies before uploading).
  • Disconfirming: today's and the other in-progress-signature runs have no readable log, so the same mode cannot be confirmed for this exact run; the varying durations (9 min hang vs 90 min timeout vs 22 min no-log) mean more than one failure surface is in play.
  • How to verify: on linux-mi35x-gpu-2 / a gfx950 box, launch sglang serve --model-path amd/GLM-5.1-MXFP4 --tp 2 --kv-cache-dtype fp8_e4m3 --disable-radix-cache --dsa-prefill-backend tilelang --dsa-decode-backend tilelang --mem-fraction-static 0.85 --trust-remote-code and capture dmesg/rocm-smi at the hang; A/B toggle the tilelang DSA backends and fp8_e4m3 KV cache to isolate the hanging path.
• [LOW] linux-mi35x-gpu-2 runner/node instability (pure infra) causing the mid-step runner loss.
  • Supporting: sub-timeout death with no uploaded log is a classic lost-runner signature.
  • Disconfirming: it recurs on this specific job every single run (never a green), which points to a model/config-specific trigger rather than random infra flake.
  • How to verify: check runner-side host logs / VRAM-clear step for node resets around 18:15Z; compare with other MI35x jobs in the same runs that did complete.

In-flight Fix Check

• Searched open PRs with keywords GLM-5.1 MXFP4, GLM51 mxfp4 mi35x. No open PR directly targets this GLM-5.1-MXFP4 MI35x server-init / GPU-hang failure. Loosely related but not a fix for this crash: #30808 [AMD] [GLM5] Enable dense-MHA short-context prefill fallback on gfx950.
• In-flight fix: ❌ none found directly matching — needs new investigation.

Suggested Triage Steps

• Recover a readable log for a fresh occurrence: re-run this job in isolation (job_select=nightly-2-gpu-mi35x-glm51-mxfp4) and, if it dies mid-step again with no artifact, capture the container/server logs directly on the runner (the GitHub job log is empty for the runner-loss cases).
• Reproduce the server launch on a gfx950/MI35x box with the exact args above; watch dmesg and rocm-smi for the GPU Hang / HW Exception during model_runner.load_model.
• A/B the DSA backend and KV-cache dtype: try --dsa-prefill-backend/--dsa-decode-backend alternatives and drop --kv-cache-dtype fp8_e4m3 to see if the hang clears — the docstring already flags an aiter GEMM-path sensitivity on gfx950.
• Since this test has never passed in the queryable window, consider gating/quarantining it on MI35x until the init hang is root-caused, so it stops masking other nightly signal.

Status

• Persistence: Failing every completed scheduled run Jul 13 → Jul 22 (never-passed in the queried window).
• Scope: Single job nightly-2-gpu-mi35x-glm51-mxfp4 on linux-mi35x-gpu-2; test file test_glm51_mxfp4_tp2_gsm8k_mi35x.py.
• Blocked work: No GSM8K accuracy signal for amd/GLM-5.1-MXFP4 TP=2 on MI35x.
• In-flight fix: ❌ none found directly matching.

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-07-22 20:09 UTC)

[amd-bot]

pr-test-amd-rocm720.yml — Per-Job Analysis (continued 2/4)

multimodal-gen-test-2-gpu-amd-rocm720 (linux-mi325-2gpu-sglang, 0) — failed step(s): N/A

Commit Info

• sglang: 9a6c9608
• aiter: N/A — the job log blob was purged (BlobNotFound) and no artifacts were uploaded, so the [CI-AITER-CHECK] markers cannot be read for this run.

Failed Tests

Test File	Test Function	Error	Log
python/sglang/multimodal_gen/test/server/test_server_2_gpu.py	TestDiffusionServerTwoGpu::test_diffusion_generation[...] (specific case undetermined for this run — log purged)	Runner lost mid-execution — step never reported terminal status	Job

Shard 0 (part 0) of the 2-gpu suite runs the full parametrized file. Derivation: run_suite.py sets parametrized_partitions = total_partitions(3) − len(STANDALONE_FILES["2-gpu"])(2) = 1, so part 0 runs all TWO_GPU_CASES in test_server_2_gpu.py; part 1 = test_disagg_server.py, part 2 = test_ar_models.py (both passed this run). [FACT]

Failure Summary

Job multimodal-gen-test-2-gpu-amd-rocm720 (…, 0) failed after ~109 min. Its job-log blob returns BlobNotFound and 0 run artifacts exist. Steps 9 (Run diffusion server tests (2-GPU)), 10 (Upload diffusion failure artifacts, if: always()) and 20 all have null conclusion and there is no "Complete job" step — the signature of a runner lost/terminated mid step 9, not a normal pytest assertion (which would still run the always() upload). The specific failing case cannot be pinned from this run because no log survives.

Regression Status

Never-passed (no green run found) in the queryable window. Shard 0 has failed in all 10 most recent completed scheduled runs (event=schedule, branch=main), i.e. since at least 2026-07-09. Two distinct signatures appear:

Date	Run	Shard-0 Job	Step 9	"Complete job"?	Signature
Jul 22 (this)	run	job	null	❌	❌ Runner lost (~109 min, no log)
Jul 20	run	job	null	❌	❌ Runner lost (~102 min)
Jul 19	run	job	null	❌	❌ Runner lost (~101 min)
Jul 18	run	job	—	❌	❌ Died early (~10 min, before step 9)
Jul 17	run	job	null	❌	❌ Runner lost (~86 min)
Jul 16	run	job	failure	✅	❌ Real step-9 test failure (log exists)
Jul 15	run	job	failure	✅	❌ Real step-9 test failure
Jul 14	run	job	failure	✅	❌ Real step-9 test failure
Jul 13	run	job	failure	✅	❌ Real step-9 test failure
Jul 12	run	job	failure	✅	❌ Real step-9 test failure
Jul 11	run	job	failure	✅	❌ Real step-9 test failure

Test-file-level comparison is only possible for Jul 11–16 (logs survive) — those confirm the failing file is test_server_2_gpu.py. For Jul 17–22 the logs are purged, so the verdict for those relies on the job step signature, not log content.

Failure Cluster

Runner lost / job hard-terminated during 2-GPU diffusion test execution (log + artifacts never uploaded). Underlying suite has been red for 10+ runs.

Facts (verified observations, no interpretation)

• Today's job 88717083212: conclusion=failure, started 2026-07-21T23:56:44Z, completed 2026-07-22T01:45:50Z (~109 min). Steps 9/10/20 conclusion = null; no "Complete job" step. [FACT]
• Job-log blob returns <Code>BlobNotFound</Code>; run artifacts total_count: 0. Step 9 timeout-minutes: 150, so the 109-min run did not hit the step timeout. [FACT]
• Passing sister shard 1 88717083165 shows a complete step chain (9→10→20→21 all success); shards 1 & 2 both passed. [FACT]
• The other failing job in this run, multimodal-gen-test-1-gpu-… (…, 3) 88717083354, shows the identical runner-loss signature (steps 9/10/20 null, no "Complete job"). [FACT]
• Jul 16 baseline 87699749934 (commit 1f34911d) shows the underlying test error: RuntimeError: CUDA error: HIPBLAS_STATUS_NOT_SUPPORTED during [denoising_step_0] / server warmup, first on case test_diffusion_generation[flux2_modelopt_fp8_tp2_t2i], cascading to ~18 ERROR cases on retry attempt 2/7. [FACT]

Hypothesised Causes (with confidence, with disconfirming evidence)

• [MEDIUM] This run failed due to runner loss / GPU hang, not a Python-level test assertion.
  • Supporting: steps 9/10/20 all null, the always() artifact-upload step never ran, no "Complete job" step, and the log blob was never finalized — all consistent with a runner agent that died mid-step-9. The same signature appears simultaneously on a second job in the same run (1-gpu shard 3), suggesting a runner/infra-level event rather than one test.
  • Disconfirming: with no log, an in-container hard crash (e.g. a GPU fault that also killed the runner agent) cannot be distinguished from a pure infrastructure reclaim; "runner loss" is the mechanism, not necessarily the ultimate cause.
  • How to verify: re-run the job and, if it reaches step 9, capture the diffusion-failure artifacts; check the runner host's dmesg/GPU health for the 23:56–01:46 UTC window.
• [LOW] The persistent redness of this shard is the FP8 HIPBLAS_STATUS_NOT_SUPPORTED failure on FLUX.2 modelopt-FP8 (case flux2_modelopt_fp8_tp2_t2i), and today's runner-loss is the same fault manifesting as a hard hang.
  • Supporting: Jul 11–16 completed runs fail on that case with HIPBLAS_STATUS_NOT_SUPPORTED in denoising/warmup; MI325 is gfx942, and open PR #28889 targets exactly this GEMM path.
  • Disconfirming: today's job ran ~109 min before dying, whereas the Jul 16 HIPBLAS failure surfaced at warmup ~16 min into step 9 — the timing does not match, so today's runner-loss may be an unrelated hang later in the long parametrized suite. No log survives to confirm the HIPBLAS error occurred today.
  • How to verify: re-run and inspect step-9 log; if the run reaches flux2_modelopt_fp8_tp2_t2i, confirm whether HIPBLAS or a later case is where it stalls.

In-flight Fix Check

• HIPBLAS / FLUX.2 FP8 (underlying suite failure): ✅ #28889 "[AMD] Use native e4m3fnuz fp8 GEMM for FLUX.2 modelopt-FP8 on gfx942" (open since 2026-06-22, last updated 2026-07-21) — directly targets the failing flux2_modelopt_fp8_tp2_t2i GEMM path on gfx942/MI325. Chase reviewers; do NOT open a duplicate. (Searched HIPBLAS, flux2 fp8.)
• Runner-loss / no-log 2-GPU job: ❌ none found matching multimodal-gen 2-gpu in open PRs — the infra/runner-loss aspect needs separate investigation.

Suggested Triage Steps

• Re-run this shard and confirm whether it reproduces as a runner loss or reaches a Python-level failure; if it reaches step 9, collect the diffusion-failures-amd-rocm720-2gpu-0-* artifact.
• Because the job log is purged, use the Jul 11–16 completed runs (logs intact) to characterize the underlying test failure; verify PR #28889 resolves the flux2_modelopt_fp8_tp2_t2i HIPBLAS_STATUS_NOT_SUPPORTED case on MI325/gfx942.
• Investigate the runner/host health for MI325 2-GPU runners over 2026-07-17 → 2026-07-22 (the window where shard 0 shifted from clean step-9 failures to runner-loss on jobs running ~85–110 min); a GPU hang late in the long parametrized suite is a candidate.
• If the FP8 case is confirmed unsupported on ROCm and no fix is imminent, consider gating flux2_modelopt_fp8_tp2_t2i on AMD until #28889 lands, to restore signal for the rest of test_server_2_gpu.py.

Status (factual, no priority assignment)

• Persistence: shard 0 has been red in every one of the last 10 completed scheduled runs (since ≥ 2026-07-09); runner-loss signature specifically on Jul 17, 19, 20, 22 (+ early die Jul 18).
• Scope: this run also lost multimodal-gen-test-1-gpu-… (…, 3) with the identical no-log signature; sister 2-GPU shards 1 & 2 passed.
• Blocked work: no CI signal for the full 2-GPU diffusion server suite (test_server_2_gpu.py) on ROCm 7.2 / MI325.
• In-flight fix: ✅ #28889 for the underlying FLUX.2 FP8 HIPBLAS failure; ❌ none for the runner-loss aspect.

multimodal-gen-unit-test-amd-rocm720 — failed step(s): Run diffusion unit tests

Commit Info

• sglang: 9a6c96083
• aiter: 9127c94a1 (v0.1.17.dev110+g9127c94a1, source: dev image detected → rebuild skipped, per [CI-AITER-CHECK] markers)

Failed Tests

Test File	Test Function	Error	Log
test/unit/test_pi05_action_api.py	test_msgpack_roundtrip_preserves_string_keys_and_numpy_payloads	ModuleNotFoundError: No module named 'msgpack'	Log

(Path is python/sglang/multimodal_gen/test/unit/test_pi05_action_api.py. All other items passed: 1 failed, 793 passed, 1 skipped, 50 subtests passed.)

Failure Summary

The multimodal_gen unit suite ran 794 tests; exactly one failed. test_msgpack_roundtrip_preserves_string_keys_and_numpy_payloads calls unpack_msgpack(pack_msgpack(payload)), and pack_msgpack in runtime/entrypoints/vla/protocol.py:59 does an unconditional import msgpack, which raises ModuleNotFoundError because the msgpack package is not present in the ROCm 7.2 CI image. The sibling test test_action_raw_response_can_preserve_numpy_for_msgpack passed because it never invokes the pack/unpack path.

Regression Status

Never-passed on AMD ROCm 7.2 — the test fails in every queryable completed scheduled run since it first appeared. This is an environment/packaging gap, not a code regression from the current commit.

Recent history of test/unit/test_pi05_action_api.py (test_msgpack_roundtrip_preserves_string_keys_and_numpy_payloads) in job multimodal-gen-unit-test-amd-rocm720 (comparable completed scheduled runs, branch=main, event=schedule):

Date	Run	Test File Status	Failed Function	Error
Jul 22	29855003895	❌ Failed	test_msgpack_roundtrip...	ModuleNotFoundError: No module named 'msgpack'
Jul 20	29766743590	❌ Failed	test_msgpack_roundtrip...	same
Jul 19	29697501385	❌ Failed	test_msgpack_roundtrip...	same
Jul 18	29654571916	❌ Failed	test_msgpack_roundtrip...	same
Jul 17	29601526478	❌ Failed	test_msgpack_roundtrip...	same

No passing run found in the queryable window; earlier scheduled runs (Jul 9–16) predate the test's introduction. First observed failure: 2026-07-17 (earliest run where the test exists).

Failure Cluster

ModuleNotFoundError: No module named 'msgpack' in multimodal_gen unit tests (missing optional dep in ROCm image).

Facts (verified observations)

• Verbatim log: E ModuleNotFoundError: No module named 'msgpack' at protocol.py:59 (import msgpack), triggered from test_pi05_action_api.py:237 (job log line 5810).
• msgpack is declared only in the diffusion optional-dependency group in python/pyproject.toml:110, not in the core deps — so it is not guaranteed to be present in every image.
• The test imports pack_msgpack / unpack_msgpack at module top (line 20-21) but the actual import msgpack is lazy, inside the functions; collection therefore succeeds and only this one test fails at call time. There is no pytest.importorskip("msgpack") guard.
• Same exact failure reproduces in all 5 verified completed scheduled runs (Jul 17–22); build/compile stages succeeded (only compiler -Wpass-failed warnings, no build errors).
• aiter version is a preserved dev build, identical across runs — unrelated to this failure.

Hypothesised Causes (with confidence)

• [FACT] The msgpack Python package is not installed in the ROCm 7.2 CI image used by this job.
  • Supporting: direct ModuleNotFoundError at the import msgpack line; msgpack sits in the diffusion extras group (pyproject.toml:110), which is evidently not fully installed in this image.
  • Disconfirming: (none found) — the error is unambiguous and reproducible.
  • How to verify: run python -c "import msgpack" / pip show msgpack inside the ROCm 7.2 container.
• [MEDIUM] The test lacks an availability guard, so a missing optional dep hard-fails instead of skipping.
  • Supporting: no importorskip; the sibling test that avoids the pack path passes.
  • Disconfirming: this is a test-robustness issue, not the root packaging cause; installing msgpack also resolves it.
  • How to verify: adding pytest.importorskip("msgpack") would convert the failure to a skip.

In-flight Fix Check

• In-flight fix: ✅ #31899 "[AMD] Add msgpack to ROCm diffusion deps (fix multimodal-gen unit test ModuleNotFoundError)" (open since 2026-07-21) — exact fix for this failure; chase reviewers, do NOT open a duplicate.
• Also related: ✅ #31844 "[AMD] ci: enable nested unit tests behind LAPACK/msgpack env skips" (open since 2026-07-20) — alternative approach that guards the test behind an msgpack skip.
• Search: search/issues?q=repo:sgl-project/sglang+is:pr+is:open+msgpack.

Suggested Triage Steps

• Confirm #31899 adds msgpack to the ROCm image/diffusion install path, merge it, and re-run multimodal-gen-unit-test-amd-rocm720 to confirm the test turns green.
• If the maintainers prefer keeping the ROCm image lean, land #31844 instead (skip the test when msgpack is absent), or add pytest.importorskip("msgpack") to test_pi05_action_api.py.
• Decide whether msgpack should be a core dependency rather than a diffusion-only extra, since runtime/entrypoints/vla/protocol.py is runtime code, not test-only.

Status

• Persistence: Failing in every completed scheduled run since 2026-07-17 (5/5 verified runs); Never-passed on AMD ROCm 7.2.
• Scope: 1 test in the multimodal-gen-unit-test-amd-rocm720 job of pr-test-amd-rocm720.yml. Likely also affects the CUDA/other-image variants only if those images also omit msgpack (not verified here).
• Blocked work: Blocks a green multimodal_gen unit signal on AMD ROCm 7.2 (single test, rest of suite passes).
• In-flight fix: ✅ #31899 (open since 2026-07-21) — exact fix awaiting merge.

multimodal-gen-test-1-gpu-amd-rocm720 (linux-mi325-1gpu-sglang, 1) — failed step(s): N/A

Commit Info

• sglang: 9a6c96083
• aiter: 9127c94a1 — v0.1.17.dev110+g9127c94a1, dev/patched image build, Dockerfile default, no override (source: [CI-AITER-CHECK] markers in the most recent completed run 88434814040; this run's own log blob is unavailable because the workflow run is still in progress).

Failed Tests

Test File	Test Function	Error	Log
python/sglang/multimodal_gen/test/server/test_server_1_gpu.py (suite 1-gpu, partition 1/4, via run_suite.py)	N/A — step-level wall-clock timeout, not a single test assertion	The action 'Run diffusion server tests (1-GPU)' has timed out after 60 minutes	Job

This is not a test-assertion failure. The 1-gpu diffusion server suite (partition 1 of 4) runs a sequence of model servers; multiple testcases pass, then the 60-minute step budget expires mid-suite.

Failure Summary

The step "Run diffusion server tests (1-GPU)" (partition 1) exceeded its timeout-minutes: 60 limit and was killed. In the current run, step 9 started 2026-07-22T01:10:12Z and the job was terminated at 02:15:33Z (65+ min of step runtime → hard timeout at 02:10:12Z + teardown). In the most recent completed run (88434814040, 07-20), the log explicitly ends with ##[error]The action 'Run diffusion server tests (1-GPU)' has timed out after 60 minutes after several testcases passed (longlive2_t2v, wan2_1_t2v_1.3b_teacache_enabled, wan2_1_t2v_1.3b_frame_interp_2x, wan2_1_t2v_1.3b_frame_interp_2x_upscaling_4x) with Lightricks/LTX-2.3 (LTX2TwoStageHQPipeline) still running at the cutoff.

Regression Status

Known recurring failure (step timeout), persistent since 2026-07-08.

Recent history of partition 1 (multimodal-gen-test-1-gpu-amd-rocm720 (linux-mi325-1gpu-sglang, 1)) — 15 most recent comparable completed scheduled runs (branch main, event=schedule, completed only). "Status" reflects the suite step; ✅ = step completed under budget, ❌ = step timed out / killed:

Date (run_started)	Run	Step 9 dur	Suite Step Status	Signal
2026-07-20	29766743590	71m	❌ Failed	explicit "timed out after 60 minutes"
2026-07-19	29697501385	75m	❌ Failed	step9 killed (in_progress)
2026-07-18	29654571916	75m	❌ Failed	step9 killed
2026-07-17	29601526478	72m	❌ Failed	step9 killed
2026-07-16	29521557976	71m	❌ Failed	step9 killed
2026-07-15	29438298954	72m	❌ Failed	step9 killed
2026-07-14	29355401408	74m	❌ Failed	over budget
2026-07-13	29273187026	68m	❌ Failed	over budget
2026-07-12	29202664092	64m	❌ Failed	over budget
2026-07-11	29162225443	72m	❌ Failed	over budget
2026-07-10	29113205616	70m	❌ Failed	over budget
2026-07-09	29039527644	70m	❌ Failed	step9 completed/failure
2026-07-08	28964460204	71m	❌ Failed	first fail (log expired)
2026-07-07	28888292440	54m	✅ Passed	last pass (under budget)
2026-07-06	28813230096	56m	❌ Failed	under budget → different failure

• First observed timeout: 2026-07-08. Last known pass: 2026-07-07 (54m, comfortably under the 60-min budget).
• Duration steps 54m → 71m at the 07-07→07-08 boundary and has stayed 64–75m every completed run since (13/13). This is a stable regression, not a flake.
• The 07-06 failure (56m, under budget) was a different pre-existing failure, not this timeout.

Failure Cluster

Diffusion 1-GPU server suite (partition 1) exceeds 60-minute step timeout

Facts (verified observations)

• ##[error]The action 'Run diffusion server tests (1-GPU)' has timed out after 60 minutes — verbatim, prev run log line 8408.
• Current job step timing: step 9 Run diffusion server tests (1-GPU) started 01:10:12Z, no completion recorded; job killed 02:15:33Z (timeout fires at 02:10:12Z). Setup steps (checkout→diagnose) all succeeded; Start CI container alone took 8m23s.
• Step config: timeout-minutes: 60 (.github/workflows/pr-test-amd-rocm720.yml:728).
• Scope within the current run: partition 2 ✅ 15m; partition 1 ❌ 77m (timeout); partition 3 ❌ 24m (step9 killed at 24m — a different, non-timeout failure); partition 0 still queued. Partitioning is very uneven (15m vs >60m).
• Suite partitions cases by estimated runtime (run_suite.py docstring lines 1-5); partition 1's cumulative runtime now exceeds the budget.
• In the 07-20 partition-1 run, non-fatal warnings appeared but did not cause the failure: AITer backend does not support Grouped Query Attention yet (Cosmos3-Nano → fell back to native) and a LongLive warmup ValueError: num_frames must be divisible by num_frames_per_block (longlive2_t2v still ultimately PASSED).
• aiter version unchanged across passing (07-07) and failing runs (Dockerfile default dev build); irrelevant to a wall-clock timeout.

Hypothesised Causes (with confidence)

• [MEDIUM] Partition-1's cumulative suite runtime exceeds the 60-min step budget (wall-clock exhaustion, not a hang on one model).
  • Supporting: 07-20 completed run explicitly timed out after 60 min with several cases PASSED and LTX-2.3 still running at cutoff; duration jumped 54m→71m exactly at the last-pass/first-fail boundary and has stayed 64–75m for 13 consecutive runs; sibling partition 2 finishes in 15m (proves the runner/infra is healthy and the issue is per-partition load).
  • Disconfirming: I could not read the current in-flight run's log or the 07-08 first-fail log (blobs expired), so I confirmed the explicit "60 minutes" message on the 07-20 run and inferred it for the current run from step timing + identical duration pattern.
  • How to verify: re-run partition 1 with -e SGLANG_TEST_NUM_INFERENCE_STEPS unchanged and time each [server-test] PASSED testcase; sum per-case wall-clock and compare against 60 min.
• [LOW] New heavy diffusion models added 07-09→07-14 sustain/worsen the overflow: LongLive 2.0 (cfe4eefabb #31200-era), LTX-2.3 / Z-Image-Turbo (e9493a015c #30584), Pi0.5 (649ce5dd3d #30633), cosmos3 action output (edd91cbdd5 #27168).
  • Supporting: these models appear in the 07-21 partition-1 run and increase per-partition load; runtime-based partition plan may not be rebalanced for ROCm timing.
  • Disconfirming: the first timeout (07-08) predates all of these commits, so they cannot explain the initial regression — they only prolong it. No commit touched testcase_configs.py/gpu_cases.py/run_suite.py in the exact 07-07→07-08 window (checked via GitHub commits API).
  • How to verify: bisect the 46-commit window 11cea29c90..8d2b66fd90 for a shared-path slowdown (model loading / offload / attention / warmup) rather than a new-model add.
• No hypothesis above MEDIUM. The initial 07-08 trigger is genuinely unclear (logs expired, no config-file commit in the window). It is a real slowdown of the partition-1 workload, but whether it stems from a shared code path change or slower ROCm inference/warmup + AITER JIT rebuilds is not yet code-level confirmed.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for diffusion+timeout, multimodal-gen+timeout, diffusion+partition, diffusion+CI+ROCm. Matches (e.g. #31899 msgpack deps, #31846 unit-test quarantine, #29584 MI325 fixes) address other diffusion CI issues, not the partition-1 60-min server-test timeout. Needs a new PR (rebalance partitions / raise timeout-minutes / trim ROCm suite).

Suggested Triage Steps

• Re-run partition 1 and capture per-testcase wall-clock (sum the [server-test] PASSED timestamps) to confirm the suite genuinely needs >60 min on MI325/ROCm 7.2 vs a single slow/hanging model (LTX-2.3 LTX2TwoStageHQPipeline is the prime suspect — it was the case running at every observed cutoff).
• Compare the runtime-based partition plan against measured ROCm times; partition 1 vs partition 2 (15m) is badly imbalanced — rebalance total-partitions/estimates or move LTX-2.3 to a lighter partition.
• If confirmed as pure budget exhaustion, options for maintainers: raise timeout-minutes for this step, increase --total-partitions, or -k-exclude the heaviest ROCm-slow model (as FLUX.2 already is).
• Separately investigate partition 3 (24m, non-timeout kill) — it is a different failure and out of scope for this partition-1 report.
• For the initial 07-08 regression, bisect 11cea29c90..8d2b66fd90 (46 commits) for a shared-path slowdown if partitioning alone doesn't restore green.

Status

• Persistence: Failing every completed scheduled run for ~14 days (since 2026-07-08; 13/13 completed runs, last pass 2026-07-07).
• Scope: Partition 1 of the 1-GPU diffusion suite on pr-test-amd-rocm720.yml. Partition 2 passes; partition 3 fails for a different reason; partition 0 pending. The parent workflow has been red every scheduled run for weeks (multiple independent job failures).
• Blocked work: 1-GPU diffusion (T2V/T2I/I2V/LoRA) server-test signal on AMD ROCm 7.2 for this partition.
• In-flight fix: ❌ none found — needs a new PR or investigation.

stage-b-test-1-gpu-large-amd-rocm720 (linux-mi325-1gpu-sglang, 0) — failed step(s): Run test

Commit Info

• sglang: 9a6c96083
• aiter: 9127c94a1 (source: dev image v0.1.17.dev110+g9127c94a1 preserved; matches Dockerfile default 9127c94a — no override, no rebuild)

Failed Tests

Test File	Test Function	Error	Log
test/registered/perf/test_bench_serving_1gpu_part1.py	TestBenchServing1GPUPart1.test_offline_throughput_with_triton_attention_backend	AssertionError: 3196.09 not greater than 3500 (line 120)	Log

The other two tests in this file (test_offline_throughput_default, test_online_*) passed — Ran 3 tests ... FAILED (errors=1). The remaining 9 test files in the job (test_zaya, test_awq, test_kda_kernels, etc.) all passed; job overall = 9/10 passed, exit code 255.

Failure Summary

The single failing test drives 500 offline requests against meta-llama/Llama-3.1-8B-Instruct with --attention-backend triton --context-length 8192 and asserts output throughput > 3500 tok/s on AMD CI (line 120). Measured throughput was 3177.95 tok/s (first attempt) and 3196.09 tok/s (retry) — both below the gate. The test's built-in single retry also failed, so the job exited with code 255.

Regression Status

Known long-persistent failure / Never-passed in queryable window. This exact test function has failed in every comparable completed scheduled run examined, back to at least 2026-07-03. Measured throughput is tightly clustered at ~3040–3200 tok/s across 3 weeks and never approaches 3500 — no green run exists in the lookback window.

Recent history of test_bench_serving_1gpu_part1.py (test_offline_throughput_with_triton_attention_backend) in job stage-b-test-1-gpu-large-amd-rocm720 (linux-mi325-1gpu-sglang, 0) (comparable completed event=schedule, branch=main runs):

Date	Run	Test File Status	Failed Function	Measured / Threshold
Jul 21	29855003895	❌ Failed	..._with_triton_attention_backend	3196.09 / 3500
Jul 20	29766743590	❌ Failed	same	3159.69 / 3500
Jul 19	29697501385	❌ Failed	same	3174.22 / 3500
Jul 18	29654571916	❌ Failed	same	3155.41 / 3500
Jul 17	29601526478	❌ Failed	same	3104.10 / 3500
Jul 16	29521557976	❌ Failed	same	3157.62 / 3500
Jul 15	29438298954	❌ Failed	same	3162.05 / 3500
Jul 14	29355401408	❌ Failed	same	3167.52 / 3500
Jul 13	29273187026	❌ Failed	same	3174.25 / 3500
Jul 05	28749537427	❌ Failed	same	3044.11 / 3500
Jul 03	28676345359	❌ Failed	same	3126.33 / 3500

First observed failure: before the 15-run window (still failing at the earliest run checked, 2026-07-03). No last-passing run found in the queryable window.

Failure Cluster

Triton-backend offline throughput below AMD perf gate (perf-threshold miss, not a crash)

Facts (verified observations)

• Assertion: self.assertGreater(res["output_throughput"], 3500) at test_bench_serving_1gpu_part1.py:120, inside the if is_in_amd_ci() branch (non-AMD gate is 3700, line 122).
• This run's serving benchmark: 500 successful requests, Output token throughput 3196.09 tok/s, benchmark duration 166.61 s, Mean TTFT 15876 ms, Mean TPOT 143.52 ms (verbatim from the "Serving Benchmark Result" block).
• Both the first attempt (3177.95) and the retry (3196.09) fell below 3500; retry() exceed maximum number of retries → exit code 1 → job exit 255.
• Historical measured throughput is stable at ~3040–3200 tok/s across 11 runs / 3 weeks (see table); it does not trend downward and never nears 3500.
• aiter was the Dockerfile-default dev build (v0.1.17.dev110+g9127c94a1); no AITER_COMMIT_OVERRIDE was set. This is not a scout run.

Hypothesised Causes (with confidence)

• [HIGH] The AMD threshold of 3500 tok/s is set above what this config actually achieves on MI325 / ROCm 7.2 (--attention-backend triton, Llama-3.1-8B, ctx 8192). The gate has been red deterministically for ≥19 days with a tight measured band of 3040–3200 tok/s.
  • Supporting: 11 consecutive scheduled runs all fail with the same test, values clustered ~3150; no green run in the window; symptom is a threshold assertion, not a crash or numeric error.
  • Disconfirming: no green baseline was found to bound the start, so a sustained real perf regression that predates 2026-07-03 (with the threshold set for pre-regression hardware) cannot be fully excluded from log evidence alone.
  • How to verify: extend the run history before 2026-07-03 to find the last passing run; git log the threshold/config to see whether 3500 was ever satisfied on this runner, or whether a perf-affecting change to the triton decode path landed just before the first failure.
• [LOW] Genuine triton-attention-backend throughput regression on ROCm predating the query window.
  • Supporting: only the triton-backend variant fails; the default-backend throughput test in the same file passes.
  • Disconfirming: throughput is flat, not degrading; a fresh regression would typically show a step-down from a higher green value, but no green value exists here.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for test_bench_serving+throughput and triton+throughput+threshold; no PR targets this AMD perf gate (matches were unrelated feature/kernel PRs).

Suggested Triage Steps

• Pull scheduled-run history for this job before 2026-07-03 to locate the last green run (if any) and bound the regression window; if none exists, treat this as a mis-calibrated AMD threshold rather than a regression.
• If a green run is found, bisect pass_sha..fail_sha for changes to the triton attention decode path or run_bench_serving config; revert the candidate on a branch and re-run this one test on MI325/ROCm 7.2.
• If no green run exists, the maintainer may consider re-calibrating the AMD gate (currently 3500) to the observed stable ~3150 band, or splitting the AMD threshold per GPU arch — decision left to a human, since it trades off regression sensitivity.

Status

• Persistence: Failing every completed scheduled run for ≥19 days (since at least 2026-07-03), always the same test function, measured ~3040–3200 tok/s vs 3500 gate.
• Scope: test_offline_throughput_with_triton_attention_backend only; the other 2 tests in the file and the other 9 files in this job pass. Confined to AMD ROCm 7.2 1-GPU perf job (stage-b-test-1-gpu-large-amd-rocm720).
• Blocked work: keeps this AMD perf shard red, masking any new perf regression that would push throughput even lower.
• In-flight fix: ❌ none found in open PRs.

multimodal-gen-test-1-gpu-amd-rocm720 (linux-mi325-1gpu-sglang, 0) — failed step(s): Run diffusion server tests (1-GPU)

Commit Info

• sglang: 9a6c9608 — [Cookbook] Add Laguna-S-2.1 (#31918)
• aiter: N/A — the diffusion install path emits no [CI-AITER-CHECK] markers; AITER_COMMIT_OVERRIDE is empty, so aiter is the Dockerfile default baked into image rocm/sgl-dev:v0.5.15.post1-rocm720-mi30x-20260721.

Failed Tests

Test File	Test Function	Error	Log
python/sglang/multimodal_gen/test/run_suite.py (--suite 1-gpu --partition-id 0 --total-partitions 4 -k "not flux_2")	N/A — whole suite step killed by wall-clock timeout	##[error]The action 'Run diffusion server tests (1-GPU)' has timed out after 60 minutes.	Job

This is not a single test assertion failure — the GitHub step was hard-killed at the 60-minute action timeout while the suite was still actively running.

Failure Summary

The Run diffusion server tests (1-GPU) step ran run_suite.py for partition 0 of the 1-GPU diffusion suite (T2V / T2I / I2V / LoRA across many models). The step started ~04:40:03 UTC and was killed by the hard 60-minute GitHub-action timeout at 05:40:11 UTC. At the moment of the kill the suite was mid-work: the last model (a LongLive2 I2V generation, 960×928×61) had finished LongLive2CausalDenoisingStage at 05:38:40 and its DecodingStage had just started; after that only GET /v1/videos client polling appears until the timeout.

Regression Status

Known recurring failure (partition 0 never passes in the queryable window).

Recent history of run_suite.py (1-gpu, partition 0) in job multimodal-gen-test-1-gpu-amd-rocm720 (linux-mi325-1gpu-sglang, 0) (comparable completed scheduled runs, event=schedule, branch=main):

Date	Run	Job	Test File Status	Failure	Error
Jul 22	29855003895	partition 0	❌ Failed	step timeout	timed out after 60 minutes
Jul 20/21	29766743590	partition 0	❌ Failed	step timeout	timed out after 60 minutes (verified in log)
Jul 19	29697501385	partition 0	❌ Failed	(partition 0 failed)	—
Jul 18	29654571916	partition 0	❌ Failed	(partition 0 failed)	—
Jul 17	29601526478	partition 0	❌ Failed	step timeout	timed out after 60 minutes (verified in log)
Jul 16	29521557976	partition 0	❌ Failed	(partition 0 failed)	—
Jul 15	29438298954	partition 0	❌ Failed	(partition 0 failed)	—
Jul 14	29355401408	partition 0	❌ Failed	step timeout	timed out after 60 minutes (verified in log)

Partition 0 failed in every run examined (Jul 14 → Jul 22); no passing run for partition 0 in the window. By contrast, partition 2 passes consistently in the same runs, and partitions 1 & 3 also fail — so the timeout is partition-content-specific, not a whole-workflow outage. First observed failure in window: at least 2026-07-14; no last-known-passing date for partition 0 in the 15-run window.

Failure Cluster

1-GPU diffusion suite (partition 0) exceeds the 60-minute step timeout

Facts (verified observations)

• Timeout marker (verbatim): ##[error]The action 'Run diffusion server tests (1-GPU)' has timed out after 60 minutes. at 05:40:11 UTC.
• Test command: python3 sglang/multimodal_gen/test/run_suite.py --suite 1-gpu --partition-id 0 --total-partitions 4 -k "not flux_2".
• All 5 primary models were NOT cached at run start (✗ NOT CACHED for Wan2.1-T2V-1.3B-Diffusers, HunyuanVideo, Qwen-Image, FLUX.1-dev, FLUX.2-dev) → downloaded during the timed step, consuming budget.
• Per-model overhead is large: aiter JIT build finish build [module_rope_2c_cached_fwd], cost 67.0s; Wan server warmup alone 188.23s/req; each model spins up a fresh per-model server (multiple server_args blocks + The server is fired up events).
• Last productive work before the kill: [LongLive2CausalDenoisingStage] finished in 5.9154 seconds (05:38:40) then [DecodingStage] started... (216.88 GB left); no completion line after that, only GET /v1/videos polling (594 total polls in the step).
• Secondary (non-fatal) error: NotImplementedError: AITer backend does not support Grouped Query Attention yet at python/sglang/multimodal_gen/runtime/layers/attention/backends/aiter.py:128 while loading nvidia/Cosmos3-Nano transformer (cosmos3video.py:604 → layer.py:585 → aiter.py:128). The worker shut down for that model and the suite proceeded to the next model, so this did not by itself cause the wall-clock timeout.
• AITER_COMMIT_OVERRIDE is empty (default aiter); not an AITER Scout run.

Hypothesised Causes (with confidence)

• [HIGH] The partition-0 workload legitimately does not fit in the 60-minute step budget on this ROCm 7.2 / MI325 configuration.
  • Supporting: the step hits the exact 60-min action timeout on ≥4 verified runs (Jul 14/17/21/22) with the same signature; the suite is still actively processing (LongLive2 decoding) when killed; large contributors visible in-log — uncached model downloads, 60–70s aiter JIT builds, ~190s per-model warmups, and per-model server restarts across many models.
  • Disconfirming: partition 2 completes within budget in the same runs, so it is not a global infra/runner outage; partitions are uneven and partition 0 happens to carry the heaviest set.
  • How to verify: re-run partition 0 with model weights pre-warmed into /sgl-data/hf-cache and compare wall-clock; measure cumulative per-stage time from the perf-dump lines to see how much of the 60 min is download vs. compute.
• [LOW] A specific model stalls (e.g. LongLive2 DecodingStage, or a hung HF download of one of the 5 uncached models).
  • Supporting: the last log line is DecodingStage started with no completion; suite spent the whole hour and only reached LongLive2 near the very end.
  • Disconfirming: only ~1.5 min elapsed in that decode before the 60-min wall hit — far too short to establish a genuine hang; the binding constraint is the accumulated 58 min spent on prior models, not this one stage.
  • How to verify: bump the step timeout (or reduce models per partition) on a branch and see whether LongLive2 decode completes; check HF download timing lines per model.
• Note on commit 9a6c9608: it is a cookbook doc addition (#31918) and touches no diffusion/runtime code; it does not explain this failure, consistent with the failure predating it by ≥8 days.

In-flight Fix Check

• In-flight fix: ❌ none found targeting the partition-0 60-min timeout. Searched open PRs for diffusion+timeout and multimodal_gen+1-gpu. Related-but-not-a-fix candidates: #31483 [AMD] ci: run vetted nested multimodal_gen unit tests on AMD, #19114 perf(diffusion): parallelize pipeline module loading (could reduce per-model load time), and #30105 [AMD][Spec] Fix aiter GQA packing... (relates to the Cosmos3 GQA NotImplementedError, not the timeout). None directly addresses the partition-0 wall-clock overrun.

Suggested Triage Steps

• Rebalance partitions: inspect run_suite.py --suite 1-gpu partitioning and move some models off partition 0 (partition 2 finishes comfortably), or raise the 60-min step timeout for this job in pr-test-amd-rocm720.yml.
• Pre-cache the 5 uncached models (Wan2.1-T2V-1.3B, HunyuanVideo, Qwen-Image, FLUX.1-dev, FLUX.2-dev) into /sgl-data/hf-cache before the timed step so downloads don't consume the budget; note HF_HUB_ENABLE_HF_TRANSFER is now deprecated (warning in log) — consider HF_XET_HIGH_PERFORMANCE.
• Reduce per-model startup cost: persist/prebuild aiter JIT kernels (AITER_JIT_DIR=/sgl-data/aiter-kernels) and evaluate #19114 to parallelize module loading.
• Separately track the Cosmos3-Nano GQA NotImplementedError (aiter backend) — it is a distinct correctness/coverage gap, not the timeout cause; cross-reference #30105.

Status

• Persistence: Partition 0 has hit the 60-minute step timeout on every scheduled run examined from 2026-07-14 through 2026-07-22 (identical signature verified in the Jul 14/17/21/22 logs). Recurring for ≥8 days.
• Scope: This one job/shard (partition 0) of multimodal-gen-test-1-gpu-amd-rocm720. Partitions 1 & 3 of the same job also fail; partition 2 passes. No evidence it blocks other workflows.
• Blocked work: 1-GPU AMD ROCm 7.2 diffusion CI signal for partition-0 models is effectively absent (never green in the window).
• In-flight fix: ❌ none found directly addressing the timeout.

stage-b-test-1-gpu-small-amd-mi35x-rocm720 (linux-mi35x-gpu-1) — failed step(s): Run test

Commit Info

• sglang: 0bdd4730
• aiter: 9127c94a (v0.1.17.dev110+g9127c94a1, source: Dockerfile default — dev image detected, rebuild skipped)

Failed Tests

Test File	Test Function	Error	Log
test/registered/quant/test_quark_mxfp4.py	TestFP8ToMXFP4MOETP1.test_gsm8k	AssertionError: 0.918 not greater than 0.92 (retry: 0.91 not greater than 0.92)	Log

The other 4 tests in the suite passed (test_dsa_skip_logits_clean.py, test_verify_splitkv.py, test_mamba_state_scatter_triton.py, test_deepseek_mla_dispatch.py). Job exited with code 255 solely due to this one test.

Failure Summary

TestFP8ToMXFP4MOETP1.test_gsm8k online-requantizes Qwen/Qwen3-30B-A3B-Instruct-2507-FP8 to MXFP4 and asserts GSM8K accuracy > 0.92 over 500 questions. The first attempt scored 0.918 and the single retry scored 0.910 — both just below the 0.92 threshold, so the test errored out. This is a marginal (sub-1%) accuracy miss, not a crash or kernel failure.

Regression Status

New / marginal-flaky — the test is only one day old on this branch. TestFP8ToMXFP4MOETP1 was relanded 2026-07-21 02:50 UTC by dcd9014f (#28291), which introduced the test_gsm8k assertion at test_quark_mxfp4.py:251. Only two comparable completed scheduled runs exist since.

Recent history of test/registered/quant/test_quark_mxfp4.py (TestFP8ToMXFP4MOETP1.test_gsm8k) in job stage-b-test-1-gpu-small-amd-mi35x-rocm720 (linux-mi35x-gpu-1) (all comparable completed event=schedule, branch=main runs since the test was relanded):

Date	Run	Job	Test File Status	Failed Function	Accuracy / Error
Jul 22	29944235234	...small-amd-mi35x-rocm720	❌ Failed	TestFP8ToMXFP4MOETP1.test_gsm8k	0.918, then 0.91 (< 0.92)
Jul 21	29855003895	...small-amd-mi35x-rocm720	✅ Passed	—	0.928 (> 0.92)
≤ Jul 20	—	—	(test did not exist)	—	relanded 2026-07-21, no prior data

First observed failure: 2026-07-22. Last known passing: 2026-07-21 (accuracy 0.928). The Job column is human reference; the verdict is based on Test File Status.

Failure Cluster

GSM8K accuracy just below threshold (MXFP4 online-quant MoE)

Facts (verified observations)

• Verbatim: AssertionError: 0.918 not greater than 0.92 (attempt 1), AssertionError: 0.91 not greater than 0.92 (retry), then Exception: retry() exceed maximum number of retries. → FAILED (errors=1, skipped=3).
• The assertion is self.assertGreater(metrics["accuracy"], accuracy_threshold) at test_quark_mxfp4.py:171, called with accuracy_threshold=0.92 at test_quark_mxfp4.py:251 for model Qwen/Qwen3-30B-A3B-Instruct-2507-FP8 (comment: "reference accuracy: ~0.948").
• Same job + same test on the immediately-prior scheduled run (29855003895, head 9a6c9608) scored 0.928 and passed.
• Accuracy spread across the three measured attempts: 0.928 / 0.918 / 0.910 — a band of 0.018 (≈9 of 500 questions), straddling the 0.92 cutoff.
• aiter version is identical between the passing (07-21) and failing (07-22) runs (Dockerfile default dev image v0.1.17.dev110+g9127c94a1), so aiter did not change.
• 58 sglang commits landed in the window 9a6c9608..0bdd4730; quant/MoE-touching ones include 8bb0d8d0 (per_token_group_quant JIT unification, #30924) and 4f882063 (#32047), but both are FP8 blockwise-quant paths, not the AMD Quark MXFP4 online-requant path this test exercises.

Hypothesised Causes (with confidence)

• [MEDIUM] Marginal/borderline threshold — the 0.92 cutoff sits inside the run-to-run accuracy noise band of this MXFP4 online-quant MoE eval.
  • Supporting: measured accuracy 0.928 (pass) vs 0.918/0.910 (fail) differs by only ~9 questions out of 500; the reference accuracy comment says ~0.948 but observed values cluster much lower and cross 0.92 between consecutive days; GPU MoE + online quantization reductions are not bit-deterministic across runs.
  • Disconfirming: only two data points exist (1 pass, 1 fail), so "flaky" is not yet statistically established vs a real ~1% regression; the 0.928→0.91 drop is monotone in this small sample.
  • How to verify: re-run test_quark_mxfp4.py (TestFP8ToMXFP4MOETP1.test_gsm8k) on linux-mi35x-gpu-1 rocm720 several times at the same head SHA; if accuracy oscillates around 0.92, it is threshold-flakiness, not a regression.
• [LOW] An sglang quant-path change in 9a6c9608..0bdd4730 shifted MXFP4 numerics down ~1%.
  • Supporting: the only variable that changed between the 07-21 pass and 07-22 fail is the sglang commit range (aiter is unchanged).
  • Disconfirming: no commit in the window touches the AMD Quark MXFP4 online-requant kernel/path; the candidate quant commits (#30924, #32047) are FP8 blockwise callers, not this code path; the delta is within eval noise.
  • How to verify: bisect 9a6c9608..0bdd4730 running only this test on rocm720 mi35x; if no single commit flips it reliably, the LOW hypothesis is refuted in favor of flakiness.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for quark_mxfp4 (2 results: #29328 feature follow-up, #28734 MLA Quark PTPC — neither addresses this threshold) and mxfp4 accuracy (no PR targeting this test's threshold). No open PR adjusts the 0.92 cutoff or fixes MXFP4 MoE accuracy on AMD.

Suggested Triage Steps

• Re-run TestFP8ToMXFP4MOETP1.test_gsm8k 5–10× at head 0bdd4730 on linux-mi35x-gpu-1 (rocm720) to measure the accuracy distribution around 0.92 and decide flaky-vs-regression.
• If the distribution centers below 0.92, the reland's threshold (test_quark_mxfp4.py:251) may be too tight for this hardware — consider whether a small margin (as sibling classes use, e.g. 0.868/0.85/0.89) is warranted; raise with #28291 author.
• If accuracy centers ≥ 0.92 but occasionally dips, treat as flaky and quarantine/retry rather than block CI.
• Only if re-runs are consistently ~0.91 at this SHA but ~0.928 at 9a6c9608: bisect 9a6c9608..0bdd4730, focusing on quant commits 8bb0d8d0 and 4f882063.

Status

• Persistence: 1 failing run (2026-07-22). Passed on the only prior run since the test relanded (2026-07-21, accuracy 0.928).
• Scope: 1 test in 1 job (stage-b-test-1-gpu-small-amd-mi35x-rocm720) in pr-test-amd-rocm720.yml. Not a crash — other 4 tests in the suite passed.
• Blocked work: blocks the rocm720 mi35x stage-b small-1GPU signal; MXFP4 online-quant accuracy gate is red.
• In-flight fix: ❌ none found — needs re-run to classify flaky-vs-regression, then either a threshold review on #28291 or a bisect.

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-07-22 22:45 UTC)

[amd-bot]

pr-test-amd.yml — Per-Job Analysis (continued 3/5)

multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 3) — failed step(s): Run diffusion server tests (1-GPU)

Commit Info

• sglang: 8ae0eb83f
• aiter: 9127c94a1 (v0.1.17.dev110+g9127c94a1, source: dev image detected — AITER_COMMIT_OVERRIDE empty, rebuild skipped)

Failed Tests

Test File	Test Function	Error	Log
test/single_test_file/test_update_weights_from_disk.py	TestUpdateWeightsFromDisk::test_update_weights_from_disk_default[Qwen-Image]	assert 400 == 200 (server weight update returned HTTP 400)	Log
test/single_test_file/test_update_weights_from_disk.py	TestUpdateWeightsFromDisk::test_update_weights_specific_modules[Qwen-Image]	AssertionError: Update failed: {...} "Failed to update module 'transformer': . All modules rolled back..." → 400 == 200	Log
test/single_test_file/test_update_weights_from_disk.py	TestUpdateWeightsFromDisk::test_corrupted_weights_rollback[Qwen-Image]	assert (400 == 200)	Log
test/single_test_file/test_update_weights_from_disk.py	TestUpdateWeightsFromDiskWithOffload::test_update_weights_with_offload_enabled[Qwen-Image]	AssertionError: Expected 200, got 400	Log
test/single_test_file/test_update_weights_from_disk.py	TestUpdateWeightsFromDisk::test_update_weights_from_disk_default[FLUX.2-klein-base-4B]	ERROR at setup: AssertionError: No weights dir for transformer in /tmp/sglang_perturbed_vae_... (fixture diffusion_server_no_offload)	Log
test/single_test_file/test_update_weights_from_disk.py	TestUpdateWeightsFromDisk::test_update_weights_specific_modules[FLUX.2-klein-base-4B]	ERROR at setup (same)	Log
test/single_test_file/test_update_weights_from_disk.py	TestUpdateWeightsFromDisk::test_update_weights_rejects_invalid_requests[FLUX.2-klein-base-4B]	ERROR at setup (same)	Log
test/single_test_file/test_update_weights_from_disk.py	TestUpdateWeightsFromDisk::test_corrupted_weights_rollback[FLUX.2-klein-base-4B]	ERROR at setup (same)	Log

Result line: 4 failed, 2 passed, 11 warnings, 4 errors in 939.66s, then ##[error]Process completed with exit code 1.

Failure Summary

The standalone test file test_update_weights_from_disk.py (diffusion update_weights_from_disk endpoint) failed on AMD MI325. All 4 Qwen-Image update-weights tests failed because the server rejected the weight update with HTTP 400; the server-side cause (visible in captured server logs) is a bare AssertionError from assert param_data.shape == loaded_weight.shape at python/sglang/multimodal_gen/runtime/layers/linear.py:410, reached via weights_updater.py while loading the transformer module (Weight update failed for module 'transformer': . Rolling back...). Separately, all 4 FLUX.2-klein-base-4B tests errored at fixture setup when the diffusion_server_no_offload sanity check computed a checksum on the perturbed clone dir /tmp/sglang_perturbed_vae_* that lacked a transformer subdir.

Regression Status

Known recurring / never-passed on AMD — this test file has failed in every comparable completed scheduled run in the entire 15-run queryable window with the identical server-side shape-mismatch signature. No green run found for this test file in the window.

Recent history of test_update_weights_from_disk.py in job multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 3) (pr-test-amd.yml, event=schedule, branch=main, completed runs; partition assignment may shift so history verified at the test-file level):

Date (UTC)	Run	Job	Test File Status	Failed Functions	Signature
Jul 22 02:37	29880220845	1gpu (3)	❌ Failed	4 Qwen-Image failed + 4 FLUX.2 errors	param_data.shape mismatch → 400 (×7)
Jul 21 12:32	29830494225	1gpu (3)	❌ Failed	4 failed, 4 errors	same
Jul 21 00:27	29790347494	1gpu (3)	❌ Failed	4 failed, 4 errors	same
Jul 20 12:42	29743214297	1gpu (3)	❌ Failed	4 failed, 4 errors	same
Jul 20 00:28	29709549202	1gpu (3)	❌ Failed	5 failed, 4 errors	same
Jul 19 12:17	29686709871	1gpu (3)	❌ Failed	5 failed, 4 errors	same
Jul 14 12:26	29332511138	1gpu (3)	❌ Failed	5 failed, 4 errors	same

First observed failure: at or before the oldest queryable run (Jul 14 12:26 UTC). Last known passing: none in window. Failing-test count wobbles between 4 and 5 (a secondary flake in which FLUX.2 offload test lands as fail vs error), but the file has never passed.

Failure Cluster

Diffusion update_weights_from_disk transformer reload shape mismatch → HTTP 400

Facts (verified observations)

• Server-side stack (captured log): weights_updater.py:434 _apply_weights → :203 _load_weights_into_module → :295 load_weights_into_model → linear.py:410 weight_loader → assert param_data.shape == loaded_weight.shape → bare AssertionError; server prints Weight update failed for module 'transformer': . Rolling back 2 module(s) ... ['text_encoder', 'transformer'].
• Immediately preceding log line: Skipping weight update for Flux2Transformer2DModel: parameter 'img_in.bias' not found after mapping to 'x_embedder.bias' — indicates a diffusers→sglang parameter-name/layout mismatch on the transformer.
• The shape-mismatch assertion fires 7 times in this run; identical signature (7 or 14 hits) in every historical run checked back to Jul 14.
• FLUX.2 ERRORs originate in the test's own fixture: test_update_weights_from_disk.py:410 sanity check → _compute_checksum_from_disk (:220) asserts os.path.exists(/tmp/sglang_perturbed_vae_*/transformer), which is False. The perturbed clone (built by a background _clone_model_with_modified_module thread) did not materialize a transformer symlink.
• Environment: AITER_COMMIT_OVERRIDE empty → this is a regular pr-test-amd.yml scheduled run, not an AITER Scout run (no Baseline A/B / Failure Origin required). aiter unchanged across passing-vs-failing comparison is moot since the test never passes.
• NVIDIA does not run this file (standalone; excluded from the NVIDIA diffusion partition plan), so this is AMD-lane-only signal.

Hypothesised Causes (with confidence)

• [HIGH] The diffusion update_weights_from_disk reload path does not reconcile the diffusers-format checkpoint layout with the sglang transformer parameters (naming/shape), so a transformer weight fails assert param_data.shape == loaded_weight.shape (linear.py:410) and the update returns HTTP 400. This is the documented root cause in issue #31924.
  • Supporting: exact stack trace + assertion in the server log; img_in.bias→x_embedder.bias remap note preceding the failure; identical deterministic signature across every run since Jul 14; affects both Qwen-Image and FLUX.2 per #31924.
  • Disconfirming: (none found) — reproduced deterministically, not flaky in root cause.
  • How to verify: run test_update_weights_from_disk.py against a diffusers-checkpoint model on AMD MI325 and inspect the transformer param shapes at the weight_loader call in linear.py:410.
• [LOW] The FLUX.2 fixture-setup ERRORs are a distinct secondary defect: the background clone-thread race / helper fails to symlink the transformer dir into the perturbed clone, so the fixture sanity check errors before the server is even exercised.
  • Supporting: ERROR traceback is entirely inside test code (test_update_weights_from_disk.py:410/:220), path is /tmp/sglang_perturbed_vae_*, not the model cache.
  • Disconfirming: _clone_model_with_modified_module (:249-252) is written to symlink non-target module dirs, so under normal execution transformer should exist — points to a race/partial-clone rather than logic; also both defects live in the same file and are covered by the same skip PR, so they may share the transformer-reload root once the server path is fixed.
  • How to verify: re-run FLUX.2 subset in isolation and inspect the contents of the perturbed clone dir after _clone_model_with_modified_module joins.

In-flight Fix Check

• In-flight fix: ✅ #31925 "[AMD] Skip test_update_weights_from_disk on ROCm pending reload fix (#31924)" — open since 2026-07-21, skips this file on the AMD lane (aligning with NVIDIA, which already doesn't run it) while the real fix is developed. Root cause tracked in issue #31924. Do NOT open a duplicate; chase reviewers on #31925.
• Note: #31924 explicitly distinguishes this shape-mismatch/400 failure from the InferenceMode-500 issue already fixed by #31263.
• API queried: search/issues?q=repo:sgl-project/sglang+is:pr+is:open+update_weights_from_disk and +diffusion+weight+update.

Suggested Triage Steps

• Merge/land the skip PR #31925 to unblock the AMD multimodal-gen-test-1-gpu (3) partition signal in the short term.
• For the real fix (tracked in #31924): reconcile the diffusers checkpoint layout with sglang transformer params in the reload path (weights_updater.py load path feeding linear.py:410); ensure the transformer weight name/shape mapping (e.g. img_in→x_embedder) is applied during update_weights_from_disk, not only at initial load.
• Replace the bare assert param_data.shape == loaded_weight.shape at linear.py:410 with a descriptive error so the 400 response carries a diagnostic message instead of an empty one (aids future triage).
• Separately investigate the FLUX.2 fixture-setup ERROR (missing transformer in /tmp/sglang_perturbed_vae_*) as a possible clone-thread race in _clone_model_with_modified_module.

Status

• Persistence: Failing deterministically in every completed scheduled run in the queryable window (since ≥ Jul 14 12:26 UTC, 7/7 checked runs).
• Scope: multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 3) on pr-test-amd.yml; AMD-lane only (NVIDIA does not run this standalone file). Both Qwen-Image (FAILED) and FLUX.2-klein-base-4B (ERROR at setup) affected.
• Blocked work: Blocks green signal for the diffusion update_weights_from_disk (RL/post-training weight reload) coverage on AMD.
• In-flight fix: ✅ #31925 (skip on ROCm, open since 2026-07-21); real fix tracked in #31924.

stage-b-test-1-gpu-large-amd (linux-mi325-1gpu-sglang, 0) — failed step(s): Run test

Commit Info

• sglang: 8ae0eb83f
• aiter: 9127c94a1 (v0.1.17.dev110+g9127c94a1, source: dev image preserved — AITER_COMMIT_OVERRIDE empty)

Failed Tests

Test File	Test Function	Error	Log
test/registered/perf/test_bench_serving_1gpu_part1.py	TestBenchServing1GPUPart1.test_online_lora_latency_with_concurrent_adapter_updates	AssertionError: 9400.92 not less than 4000 (median E2E latency ms)	Log

Other 9 test files in this job passed (Test Summary: 9/10 passed).

Failure Summary

The LoRA concurrent-adapter-update latency benchmark exceeded its median E2E latency threshold: test_online_lora_latency_with_concurrent_adapter_updates asserts median_e2e_latency_ms < 4000 (test_bench_serving_1gpu_part1.py:169) but measured 9400.92 ms (first attempt 20742 ms, single retry 9400 ms — both fail). The benchmark itself completed all 400 requests successfully; only the latency gate tripped. Tail latencies were very high (P90 30737 ms, P99 51064 ms, Max ITL 18617 ms).

Regression Status

Flaky / intermittent — the same test file passes in adjacent scheduled runs with near-identical SHAs.

Recent history of test_bench_serving_1gpu_part1.py (test_online_lora_latency_with_concurrent_adapter_updates) in job stage-b-test-1-gpu-large-amd (linux-mi325-1gpu-sglang, 0) (most recent comparable completed scheduled runs: branch=main, event=schedule, completed only):

Date (UTC)	Run	Test File Status	Failed Function	Error
07-22 00:26	29880220845	❌ Failed	test_online_lora_latency_with_concurrent_adapter_updates	9400.92 not less than 4000
07-21 12:32	29830494225	✅ Passed	—	—
07-21 00:27	29790347494	❌ Failed	test_online_lora_latency_with_concurrent_adapter_updates	11156.09 not less than 4000
07-20 12:42	29743214297	✅ Passed	—	—
07-20 00:28	29709549202	✅ Passed	—	—
07-19 12:17	29686709871	✅ Passed	—	—
07-19 00:27	29666971143	✅ Passed	—	—

First observed failure for this test: 07-21 00:27 (29790347494); last known pass: 07-21 12:32 (29830494225) — i.e. it failed → passed → failed across three consecutive runs, confirming intermittency rather than a clean regression. (The Job column's overall pass/fail matches Test File Status here because this test file is the only failing file in each failing run.)

Failure Cluster

LoRA concurrent-adapter-update median E2E latency exceeds 4000 ms threshold (perf gate flake)

Facts (verified observations, no interpretation)

• Assertion at test_bench_serving_1gpu_part1.py:169: self.assertLess(res["median_e2e_latency_ms"], 4000).
• Current run: attempt 1 median E2E = 20742.33 ms, retry median E2E = 9400.92 ms; both > 4000 ⇒ retry() exceed maximum number of retries (1 retry configured). log
• Benchmark completed fully: 400/400 successful requests, req throughput 3.97 req/s, but Mean E2E 12639 ms, P99 E2E 51064 ms, Max ITL 18617 ms — heavy tail under concurrent adapter load/unload.
• Prior failing run 29790347494: same test, 15641 ms → 11156 ms (both > 4000). log
• Threshold 4000 is not new: the current file is a move (all + lines in 7a03d3014, a file relocation into test/registered/perf/, not the threshold's origin).
• aiter is a preserved dev image (v0.1.17.dev110+g9127c94a1), unchanged by this run; no AITER_COMMIT_OVERRIDE.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [MEDIUM] Flaky/too-tight performance gate: the concurrent LoRA adapter load/unload background task intermittently stalls decode on the MI325 runner, and the 4000 ms median-E2E threshold is below the observed run-to-run variance.
  • Supporting: 2/7 recent completed runs fail; passing and failing runs share near-identical code; failing values span 9400–20742 ms while passing runs are < 4000 ms; extreme tails (Max ITL 18617 ms, P99 E2E 51 s) indicate transient serving stalls, not a steady slowdown.
  • Disconfirming: cannot fully exclude a genuine latency regression that only manifests under scheduling jitter; no A/B repro was run.
  • How to verify: run test/registered/perf/test_bench_serving_1gpu_part1.py::TestBenchServing1GPUPart1.test_online_lora_latency_with_concurrent_adapter_updates ~10× on a linux-mi325-1gpu-sglang runner at this SHA and record the median-E2E distribution vs the 4000 ms gate.
• [LOW] A recent LoRA/serving change regressed adapter-swap latency.
  • Supporting: LoRA runtime area sees active churn.
  • Disconfirming: no LoRA-runtime commit in the pass→fail window touches the adapter load/unload hot path; the test passed on 29830494225 at a SHA only hours older than the failing SHA, so a persistent regression is unlikely.
  • How to verify: bisect e4eea7ce2f..8ae0eb83f only if the perf-flake hypothesis is disproven by a stable repro.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for lora latency and concurrent adapter; no PR targets this test's threshold or the concurrent-adapter latency path (query: search/issues?q=repo:sgl-project/sglang+is:pr+is:open+lora+latency and ...+concurrent+adapter).

Suggested Triage Steps

• Confirm flakiness: rerun the single test ~10× on linux-mi325-1gpu-sglang at 8ae0eb83f and capture the median-E2E spread; if it straddles 4000 ms, the gate is too tight for this workload.
• If flaky, consider raising the retry count or the threshold for the concurrent-adapter variant (its sister test_online_lora_latency uses 2400 ms without background churn), or quarantining on AMD until the tail is understood.
• If the distribution is bimodal (occasional multi-second stalls), profile the adapter load/unload background task for decode-loop contention (Max ITL 18617 ms suggests a serving stall during a swap).
• Only if a stable repro shows a monotonic slowdown, bisect e4eea7ce2f..8ae0eb83f.

Status

• Persistence: Intermittent — failed 07-21 00:27 and 07-22 00:26, passed 07-21 12:32 in between (2/7 recent completed runs failed).
• Scope: Single test file (test_bench_serving_1gpu_part1.py) in one job (stage-b-test-1-gpu-large-amd, MI325 1-GPU) of pr-test-amd.yml; other 9 files in the job pass.
• Blocked work: Intermittently reddens the AMD 1-GPU perf stage; not a hard block since it passes on reruns.
• In-flight fix: ❌ none found in open PRs.

multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 1) — failed step(s): N/A

Commit Info

• sglang: 8ae0eb83f (#31985)
• aiter: N/A — the job log is unavailable (BlobNotFound, see Facts), so [CI-AITER-CHECK] markers cannot be read.

Failed Tests

Test File	Test Function	Error	Log
N/A (non-test failure)	N/A	The self-hosted runner lost communication with the server	Job

This is not a pytest failure. The job died mid-execution during step 9 "Run diffusion server tests (1-GPU)" and the log was never finalized, so the specific test/model that was running when the runner died cannot be recovered from the log. The failure is recorded only as a GitHub check-run annotation (verbatim): "The self-hosted runner lost communication with the server. Verify the machine is running and has a healthy network connection. Anything in your workflow that terminates the runner process, starves it for CPU/Memory, or blocks its network access can cause this error."

Failure Summary

Job multimodal-gen-test-1-gpu-amd (part 1) was killed by a self-hosted runner loss of communication while running the 1-GPU diffusion server tests (step 9). Step 9 started at 02:32:08Z and the job was terminated at 03:45:55Z (~73.8 min into the step, under its 90-min timeout-minutes), leaving step 9 stuck in in_progress, no failure artifacts uploaded, and the job log unretrievable. This is a partition-specific runner-death, distinct from a normal test assertion failure.

Regression Status

Known recurring / persistent infrastructure failure — NOT a new regression from the head commit.

Because part-1 logs are systematically lost (BlobNotFound), test-file-level tracking is impossible for this partition; status is tracked at the job level via the check-run annotation. The identical "runner lost communication" annotation appears in every comparable completed scheduled run checked (17 consecutive, Jul 13 → Jul 22). No passing part-1 run exists in the query window ⇒ Never-passed in queryable history at the job level.

Recent history of job multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 1) (branch main, event=schedule, completed runs only):

Date	Run	Job Status	Annotation	Log available?	Step-9 runtime
Jul 22	29880220845	❌ Failed	runner lost communication	❌ BlobNotFound	~73.8 min (killed)
Jul 21	29830494225	❌ Failed	runner lost communication	❌	job ~111 min
Jul 21	29790347494	❌ Failed	runner lost communication	❌	job ~90 min
Jul 20	29743214297	❌ Failed	runner lost communication	❌	job ~84 min
Jul 20	29709549202	❌ Failed	runner lost communication	❌	job ~71 min
Jul 19	29666971143	❌ Failed	runner lost communication	❌	job ~75 min
Jul 15	29378873430	❌ Failed	runner lost communication	❌	—
Jul 13	29215394044	❌ Failed	runner lost communication	❌	—

(All 12 runs Jul 15–22 verified failure + annotation; 5 older runs Jul 13–15 spot-checked identical. First observed failure: ≥ Jul 13 06:41 UTC; no known passing run in the query window.)

Failure Cluster

Self-hosted runner lost communication during 1-GPU diffusion server tests (part 1)

Facts (verified observations)

• Annotation (FACT): check-run 88799241355 carries a single failure annotation: "The self-hosted runner lost communication with the server… Anything in your workflow that terminates the runner process, starves it for CPU/Memory, or blocks its network access can cause this error."
• Log unavailable (FACT): GET /actions/jobs/88799241355/logs → BlobNotFound; run-level /logs → Not Found. Same BlobNotFound for all 6 historical part-1 job IDs checked — logs are never finalized when the runner dies.
• Step state (FACT): step 9 "Run diffusion server tests (1-GPU)" = in_progress/no completed_at; steps 10 (Upload diffusion failure artifacts) and 20 = pending. Job started 02:20:51Z, completed 03:45:55Z; step 9 started 02:32:08Z → killed ~73.8 min later, below its timeout-minutes: 90. No step timeout was hit.
• No artifacts (FACT): run artifact count = 0; the diffusion-failures-amd-1gpu-1 artifact (step 10, if: always()) never uploaded because the runner was gone.
• Partition isolation (FACT): in the same run, sibling part 2 succeeded in ~22 min, and part 3 failed cleanly with normal pytest assertions and a finalized log (4 failed, 2 passed … in 939.66s — test_update_weights_from_disk.py[Qwen-Image], weight shape mismatch). Only part 1 exhibits the runner-death/no-log signature. Part 1 is also the longest-running partition (71–111 min historically).
• Persistence (FACT): identical annotation in 17 consecutive completed scheduled runs (Jul 13 → Jul 22).
• Note: part 3's test_update_weights_from_disk.py[Qwen-Image] failures are a separate, distinct recurring failure in a different partition — NOT what killed part 1, and out of scope for this job.

Hypothesised Causes (with confidence + disconfirming evidence)

• [MEDIUM] Host CPU/memory starvation (or a GPU-side crash) during the heavy 1-GPU diffusion server workloads assigned to part 1 terminates the runner agent process.
  • Supporting: annotation explicitly lists CPU/Memory starvation as a cause; part 1 is consistently the longest/heaviest partition (up to 111 min, loading large models — FLUX.1, HunyuanVideo, Wan2.1, Qwen-Image); the lighter part 2 passes in ~22 min; failure is deterministic per-partition, not random.
  • Disconfirming: logs are lost, so there is no OOM-killer / GPU-fault evidence to confirm starvation vs. network vs. hardware; the annotation is generic and lists three possible causes.
  • How to verify: reproduce part-1's partition (run_suite.py --suite 1-gpu --partition-id 1 --total-partitions 4 -k "not flux_2") on an MI325 runner while monitoring free -h / dmesg (OOM-killer) and rocm-smi; check the runner host's _diag agent logs around 03:45:55Z on the run date.
• [LOW] Transient network/hardware flakiness of the specific self-hosted MI325 runner.
  • Supporting: annotation lists network access as a possible cause.
  • Disconfirming: the failure recurs on every run for this partition specifically (17/17), which is inconsistent with random transient network/hardware faults.
  • How to verify: correlate the physical runner assigned to part-1 across runs; if it varies but the failure persists, network/hardware is unlikely.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for runner lost communication (0 results), diffusion OOM memory (10 results, all unrelated feature PRs), and multimodal-gen partition 1gpu (2 results, unrelated). No PR targets the part-1 runner-death.

Suggested Triage Steps

• Pull the runner host diagnostics (agent _diag logs, dmesg/OOM-killer, rocm-smi history) for the MI325 runner that ran job 88799241355 around 02:32–03:46 UTC on 2026-07-22 — the GitHub-side log is permanently gone, so host-side telemetry is the only way to see the death cause.
• Reproduce part-1's exact partition locally on an MI325 container (--suite 1-gpu --partition-id 1 --total-partitions 4 -k "not flux_2") with host memory + GPU monitoring to confirm/deny the CPU/Memory-starvation hypothesis and identify the offending model/test.
• If starvation is confirmed, consider re-balancing the LPT partition estimates (partitioning.py) or lowering per-model memory pressure for the 1-GPU suite so no single partition exhausts the host.
• Because failure artifacts (step 10) never upload on a runner kill, consider streaming/periodic diagnostics during step 9 so future runner-deaths leave forensic evidence.

Status

• Persistence: Failing continuously for ≥ 9 days (since at least 2026-07-13 06:41 UTC), identical "runner lost communication" annotation, every completed scheduled run.
• Scope: Isolated to multimodal-gen-test-1-gpu-amd part 1 on pr-test-amd.yml (MI325 1-GPU). Sibling part 2 passes; part 3 fails for an unrelated pytest reason.
• Signal quality: This partition produces no usable CI signal — logs and failure artifacts are lost on every run, so real test regressions in part-1's model set are currently invisible.
• In-flight fix: ❌ none found — needs a new infra/CI investigation (runner host telemetry), not a code revert.

multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 0) — failed step(s): N/A

Commit Info

• sglang: 8ae0eb83
• aiter: N/A — the job log blob returns BlobNotFound (never uploaded), so no [CI-AITER-CHECK] markers are available for this run.

Failed Tests

Test File	Test Function	Error	Log
python/sglang/multimodal_gen/test/server/test_server_2_gpu.py (part 0 shard)	TestDiffusionServerTwoGpu::test_diffusion_generation	Unknown for this run — no log; job terminated before step 9 concluded	Job

The target job produced no readable log (BlobNotFound) and no artifacts (upload step still pending). The specific failing testcase in this run cannot be confirmed. The row above reflects the shard that ran; the persistent content failure for this shard (confirmed from historical runs that did log) is documented under Facts.

Failure Summary

Job multimodal-gen-test-2-gpu-amd (…, 0) is marked failure, but step 9 "Run diffusion server tests (2-GPU)" never reached a conclusion (in_progress) and the job was terminated at ~85 min (02:34:53 → 03:59:57 UTC), well under the 150-min step timeout. The log blob does not exist and failure artifacts were never uploaded, so this run gives no direct evidence of which test failed. The parent run 29880220845 is still in progress at snapshot time.

Regression Status

Never-passed (no green run found in window) — this shard has failed in all 15 comparable completed scheduled runs (branch=main, event=schedule), spanning at least 2026-07-14 → 2026-07-21. Two distinct failure modes recur:

• Mode A — step 9 completes with failure, log present: testcase flux2_modelopt_fp8_tp2_t2i fails with HIPBLAS_STATUS_NOT_SUPPORTED.
• Mode B — step 9 stays in_progress, job killed ~85-109 min, log blob BlobNotFound (this run's signature).

Recent history of test_server_2_gpu.py shard (…-2gpu-sglang, 0) in job multimodal-gen-test-2-gpu-amd (most recent 15 comparable completed scheduled runs; the 2026-07-16 00:25 run was cancelled and excluded):

Date	Run	Test File Status	Mode / Evidence
Jul 22 00:26	29880220845	❌ Failed	B — killed ~85m, no log (this run)
Jul 21 12:32	29830494225	❌ Failed	B — no log (BlobNotFound verified)
Jul 21 00:27	29790347494	❌ Failed	B — no log
Jul 20 12:42	29743214297	❌ Failed	B — no log
Jul 20 00:28	29709549202	❌ Failed	B — no log
Jul 19 12:17	29686709871	❌ Failed	killed ~91m
Jul 19 00:27	29666971143	❌ Failed	B — no log
Jul 18 12:17	29644059344	❌ Failed	early infra failure (~10m)
Jul 18 00:25	29623067212	❌ Failed	A — flux2_modelopt_fp8_tp2_t2i (1 failed, 22 passed)
Jul 17 12:25	29580198489	❌ Failed	B — no log
Jul 17 00:27	29544840981	❌ Failed	A — flux2_modelopt_fp8_tp2_t2i (1 failed, 17 errors)
Jul 16 12:31	29498496614	❌ Failed	A (step9 completed/failure)
Jul 15 12:28	29415344552	❌ Failed	B — no log
Jul 15 00:22	29378873430	❌ Failed	B — no log
Jul 14 12:26	29332511138	❌ Failed	A (step9 completed/failure)

No passing run for this shard exists anywhere in the 15-run window (2026-07-14 → 2026-07-22).

Failure Cluster

"flux2 modelopt-FP8 GEMM unsupported on ROCm hipBLAS (HIPBLAS_STATUS_NOT_SUPPORTED) — 2-GPU diffusion shard; some runs escalate to job termination / missing log"

Facts (verified observations, no interpretation)

• This run: job 88799241370 conclusion=failure; step 9 "Run diffusion server tests (2-GPU)" status=in_progress/conclusion=None; step 10 "Upload diffusion failure artifacts" pending. Started 2026-07-22T02:34:53Z, completed 03:59:57Z (~85 min); step-9 timeout is 150 min. Log endpoint returns BlobNotFound; run artifacts list is empty.
• Missing-log signature is recurring, not unique to this run: most-recent prior run 29830494225 (Jul 21) has the identical step9=in_progress + BlobNotFound signature (verified by re-fetching its log).
• Content failure (Mode A, from runs that logged): part-0 shard runs 25 items from test_server_2_gpu.py::TestDiffusionServerTwoGpu::test_diffusion_generation. In 29623067212: FAILED …[flux2_modelopt_fp8_tp2_t2i], == 1 failed, 22 passed, 1 skipped, 1 error in 6344.39s (1:45:44) ===. Server warmup log: CUDA error: HIPBLAS_STATUS_NOT_SUPPORTED when calling HIPBLAS_STATUS_NOT_SUPPORTED.
• Same testcase + same error reproduced in 29544840981: [server-test] FAILED diffusion testcase: flux2_modelopt_fp8_tp2_t2i, error raised in [DenoisingStage]/[denoising_step_0] GEMM path.
• Testcase flux2_modelopt_fp8_tp2_t2i is defined at python/sglang/multimodal_gen/test/server/gpu_cases.py:597. Perf baselines exist only for NVIDIA (h100.json, b200.json) — no AMD baseline for this case.
• Parts 1 and 2 of the 2-GPU matrix passed in this run; only part 0 (which carries flux2_modelopt_fp8_tp2_t2i) failed.
• Note: full part-0 suite takes ~105-126 min when it runs to completion; Mode-B jobs die at 85-109 min (before completion, before the 150-min step timeout).

Hypothesised Causes (with confidence, with disconfirming evidence)

• [FACT] The flux2_modelopt_fp8_tp2_t2i testcase fails on this ROCm runner with HIPBLAS_STATUS_NOT_SUPPORTED — the modelopt-FP8 GEMM configuration is not supported by the hipBLAS path used for FLUX.2 on gfx942.
  • Supporting: identical error + identical failing testcase in two independent logged runs (Jul 17, Jul 18); an open PR (#28889) exists specifically to add a native e4m3fnuz FP8 GEMM for exactly this case on gfx942.
  • Disconfirming: (none for the content failure) — but this does not directly explain this run's missing log (Mode B).
• [LOW] This run's termination + missing log (Mode B) is caused by the same FP8 GEMM path hard-hanging/crashing the GPU (or the runner being reclaimed) rather than raising cleanly, so the runner is lost before logs/artifacts upload.
  • Supporting: Mode-B jobs consistently die at ~85-109 min (mid-suite), and the shard's only content failure is the FP8 GEMM error; the missing-log signature repeats across many runs of the same shard.
  • Disconfirming: no log means no direct evidence the FP8 case is what ran when the job died; earlier docker-registry timeouts (Client.Timeout exceeded) were also seen in logs, and one run died at ~10 min from an unrelated early infra failure — so pure infra flakiness (spot reclaim / runner disconnect) cannot be ruled out.
  • How to verify: re-run this shard with step-level log streaming / shorter per-testcase timeout; check runner-side syslog for GPU reset / OOM-kill around the 85-min mark.

In-flight Fix Check

• In-flight fix: ✅ #28889 "[AMD] Use native e4m3fnuz fp8 GEMM for FLUX.2 modelopt-FP8 on gfx942" — open since 2026-06-22, mergeable_state: blocked (last updated 2026-07-21). Directly targets the flux2_modelopt_fp8 / HIPBLAS_STATUS_NOT_SUPPORTED content failure. Chase reviewers / unblock; do NOT open a duplicate.
• (Searched open PRs with keywords flux2 fp8, hipblas, diffusion amd.)

Suggested Triage Steps

• Verify #28889 resolves the content failure: rebase it, re-run multimodal-gen-test-2-gpu-amd part 0, and confirm flux2_modelopt_fp8_tp2_t2i passes.
• Separately investigate the missing-log / job-termination (Mode B) signature: it may mask the real result. Inspect runner host logs for GPU reset/OOM around the ~85-min mark, or add per-testcase timeouts so a hung FP8 GEMM aborts the case (with a log) instead of killing the runner.
• If the FP8 GEMM cannot be supported near-term, consider @skipif-gating flux2_modelopt_fp8_tp2_t2i on ROCm/gfx942 until #28889 lands, so the rest of the part-0 shard yields a usable signal.
• Because this run has no log, treat its root cause as unconfirmed — do not attribute it to commit 8ae0eb83; the failure predates it by ≥8 days.

Status

• Persistence: Failing every completed scheduled run for ≥8 days (since at least 2026-07-14); never-passed in the 15-run window.
• Scope: Part-0 shard of multimodal-gen-test-2-gpu-amd only; parts 1 and 2 passed this run. Content failure isolated to flux2_modelopt_fp8_tp2_t2i.
• Data gap: This specific run (2026-07-22) has no log and no artifacts; the failing testcase is inferred from historical logged runs, not confirmed for this run.
• In-flight fix: ✅ #28889 (open since 2026-06-22, blocked) — addresses the content failure.

stage-b-test-1-gpu-small-amd-nondeterministic (linux-mi325-1gpu-sglang) — failed step(s): Run test

Commit Info

• sglang: 8ae0eb83fc
• aiter: 9127c94a18 (v0.1.17.dev110+g9127c94a1; source: dev image detected & preserved, no override — AITER_COMMIT_OVERRIDE empty)

Failed Tests

Test File	Test Function	Error	Log
test/registered/prefill_only/test_reward_models.py	TestRewardModels.test_reward_scores	AttributeError: 'LlamaForSequenceClassificationWithNormal_Weights' object has no attribute '_legacy_load_weights' → scheduler sigquit → exit code -9	Log

The other file in this suite, test/registered/models/test_vlm_models.py, passed (1/2 passed). The job's overall failure (exit 255) is driven solely by the reward-model test.

Failure Summary

TestRewardModels.test_reward_scores fails during model weight loading: the SGLang scheduler crashes while loading URM-LLaMa-3.1-8B. The reward model class LlamaForSequenceClassificationWithNormal_Weights delegates weight loading to LlamaForCausalLM.load_weights(self, weights), which now dispatches to self._legacy_load_weights(...) — a method that does not exist on the reward class, raising AttributeError and killing the scheduler child (exit code -9).

Regression Status

New regression. First failing run is today's; the two immediately preceding scheduled runs passed this job, and neither of their head SHAs contains the suspected commit.

Recent history of test/registered/prefill_only/test_reward_models.py (TestRewardModels.test_reward_scores) in job stage-b-test-1-gpu-small-amd-nondeterministic (most recent comparable completed scheduled runs: branch=main, event=schedule):

Date	Run	Job	Test File Status	Failed Function	Error
Jul 22	29880220845	…nondeterministic	❌ Failed	test_reward_scores	AttributeError: no attribute '_legacy_load_weights'
Jul 21 12:32	29830494225	…nondeterministic	✅ Passed	—	—
Jul 21 00:27	29790347494	…nondeterministic	✅ Passed	—	—

• First observed failing: 2026-07-22 (run 29880220845).
• Last known passing: 2026-07-21 12:32 UTC (run 29830494225, head e4eea7ce2f).
• Regression window: e4eea7ce2f..8ae0eb83fc.
• Note: the parent run 29880220845 is still queued at run level, but this job completed with conclusion=failure — the result is final.

Failure Cluster

AttributeError _legacy_load_weights missing on reward-model weight loading

Facts (verified observations)

• Verbatim traceback (job log lines ~4350–4382): model.load_weights(weights) → llama_reward.py:140 super().load_weights(weights) → llama_reward.py:77 LlamaForCausalLM.load_weights(self, weights) → llama.py:662 return self._legacy_load_weights(weights) → torch/nn/modules/module.py:1964 __getattr__ → AttributeError: 'LlamaForSequenceClassificationWithNormal_Weights' object has no attribute '_legacy_load_weights'.
• Scheduler then logs Received sigquit from a child process; runner reports ✗ FAILED: …test_reward_models.py returned exit code -9, and the step exits 255.
• Class hierarchy (verified in python/sglang/srt/models/llama_reward.py): LlamaForSequenceClassification(nn.Module) (line 33) and LlamaForSequenceClassificationWithNormal_Weights(LlamaForSequenceClassification) (line 80) — neither inherits from LlamaForCausalLM. Both reach LlamaForCausalLM.load_weights only via the explicit unbound call at llama_reward.py:77.
• Current LlamaForCausalLM.load_weights (python/sglang/srt/models/llama.py:658-662) dispatches to self._load_weights_v2 / self._legacy_load_weights; the concrete loader body was renamed to _legacy_load_weights (line 664).
• git log -L 658,665:llama.py shows this dispatch + rename was introduced by becf252e6 ("AutoWeightLoader support Sglang native models 1: demo", #28671), committed 2026-07-21 21:32 UTC.
• git merge-base --is-ancestor becf252e6 <sha>: the two passing runs' heads (e4eea7ce2f, 8905cbd42f) do NOT contain the culprit; today's failing head 8ae0eb83fc does.
• aiter version is identical to prior runs (Dockerfile default dev image, no override) — not an aiter-driven change.

Hypothesised Causes (with confidence)

• [HIGH] Commit becf252e6 (#28671) refactored LlamaForCausalLM.load_weights to dispatch to self._legacy_load_weights / self._load_weights_v2. The reward-model classes call LlamaForCausalLM.load_weights(self, ...) with a self that does not subclass LlamaForCausalLM, so self._legacy_load_weights is unresolvable → AttributeError.
  • Supporting: exact method name in the AttributeError (_legacy_load_weights) matches the method the commit introduced; the modified line llama.py:662 is the last SGLang frame in the traceback; test passed in the run immediately before the commit landed and failed in the first run after; version window e4eea7ce2f..8ae0eb83fc contains only this as a load_weights-dispatch change.
  • Disconfirming: (none found) — the aiter version, runner, and all other environment facts are unchanged between the last passing and first failing run; the VLM test in the same job still passes, isolating the fault to the reward-model load path.
  • How to verify: on a branch, git revert becf252e6 (or set SGLANG_ENABLE_WEIGHT_LOADER_V2 / restore an unbound _legacy_load_weights-independent path) and re-run test/registered/prefill_only/test_reward_models.py on AMD MI325. Alternatively confirm locally that LlamaForSequenceClassificationWithNormal_Weights lacks _legacy_load_weights in its MRO.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for _legacy_load_weights (0), llama_reward (0), and SequenceClassification load_weights (0) via GET /search/issues?q=repo:sgl-project/sglang+is:pr+is:open+<kw>.

Suggested Triage Steps

• Bisect / confirm within e4eea7ce2f..8ae0eb83fc — becf252e6 is the only load_weights-dispatch change in the window.
• If confirmed, fix direction (hypothetical): either make the reward classes route through the new dispatch (define/inherit _legacy_load_weights), or have llama_reward.py:77 call LlamaForCausalLM._legacy_load_weights(self, weights) directly, or make the dispatch tolerate classes without the v2/legacy split. Maintainer/PR author of #28671 should decide, since other native models reusing LlamaForCausalLM.load_weights via unbound calls may share this breakage.
• Grep other models that call LlamaForCausalLM.load_weights(self, …) (or similar unbound cross-class loader calls) to scope blast radius before merging a fix.

Status

• Persistence: First failure this run (2026-07-22); passed in the two prior scheduled runs (last green 2026-07-21 12:32 UTC).
• Scope: Confirmed in job stage-b-test-1-gpu-small-amd-nondeterministic (test_reward_models.py). Likely affects any suite exercising LlamaForSequenceClassification* reward models; other Llama-derived native models using unbound LlamaForCausalLM.load_weights calls may be at risk.
• Blocked work: Reward-model (URM/sequence-classification) CI signal on AMD.
• In-flight fix: ❌ none found — needs new PR or investigation.

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-07-22 19:32 UTC)

[amd-bot]

pr-test-amd.yml — Per-Job Analysis (continued 4/5)

multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 0) — failed step(s): N/A

Commit Info

• sglang: 8ae0eb83 — "[Perf] Fold dspark dense draft embedding into the draft graph via forward_embed (#31985)"; touches only python/sglang/srt/models/dflash.py and dspark.py (speculative-decode models), not the diffusion/multimodal_gen path.
• aiter: N/A — the job log blob returns BlobNotFound (never uploaded, see below), so [CI-AITER-CHECK] markers cannot be read for this run.

Failed Tests

Test File	Test Function	Error	Log
N/A (non-test failure)	N/A	Abnormal job/runner termination during step "Run diffusion server tests (1-GPU)" — step never concluded, no log/artifact uploaded	Job

The specific hanging test cannot be pinned for this shard. The run_suite.py --suite 1-gpu --partition-id 0 --total-partitions 4 split is done at runtime by longest-processing-time bin-packing (python/sglang/multimodal_gen/test/partitioning.py), so partition 0's contents are not statically knowable, and neither the job log nor the diffusion-failures artifact was uploaded (both the test step and the if: always() upload step have conclusion None).

Failure Summary

The multimodal-gen-test-1-gpu-amd (shard 0) job entered step 9 ("Run diffusion server tests (1-GPU)") at 04:32:11 UTC and the job ended at 05:45:55 UTC (~74 min) without the step ever reaching a conclusion. Step 10 ("Upload diffusion failure artifacts", if: always()), "Post Checkout code", and "Complete job" never ran, and GitHub has no log blob for the job. This is the signature of an abnormal runner/container termination (hang or kill) mid-test, not a clean test assertion failure. Note it did not hit the 90-min timeout-minutes limit (74 min < 90).

Regression Status

Known recurring / persistent failure (Never-passed within the 15-run query window). This shard has failed in all 15 most-recent comparable scheduled runs (same branch main, event=schedule, status=completed), the oldest being 2026-07-14. No green run exists in the window, so a pass_sha..fail_sha regression bisect cannot be anchored.

Recent history of shard 0 multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 0) (most recent 15 comparable completed scheduled runs):

Date	Run	Test/Step Status	Failure mode
Jul 21 12:32	29830494225	❌ Failed	step 9 None (runner died mid-test)
Jul 21 00:27	29790347494	❌ Failed	step 9 None
Jul 20 12:42	29743214297	❌ Failed	step 9 None
Jul 20 00:28	29709549202	❌ Failed	step 9 None
Jul 19 12:17	29686709871	❌ Failed	early failure (no steps recorded)
Jul 19 00:27	29666971143	❌ Failed	step 9 None
Jul 18 12:17	29644059344	❌ Failed	step 9 None
Jul 18 00:25	29623067212	❌ Failed	step 9 None
Jul 17 12:25	29580198489	❌ Failed	step 9 None
Jul 17 00:27	29544840981	❌ Failed	early failure (no steps recorded)
Jul 16 12:31	29498496614	❌ Failed	step 9 None
Jul 15 12:28	29415344552	❌ Failed	"Start CI container" failed (infra)
Jul 15 00:22	29378873430	❌ Failed	early failure (no steps recorded)
Jul 14 12:26	29332511138	❌ Failed	early failure (no steps recorded)
Jul 14 00:24	29295847781	❌ Failed	early failure (no steps recorded)

(One 2026-07-16 00:25 scheduled run was cancelled and excluded. The whole current run 29880220845 is still in-progress at snapshot; the shard-0 job itself is completed.)

Failure Cluster

"Runner/container death during 1-GPU diffusion server test (no log uploaded)"

Facts (verified observations)

• Step 9 "Run diffusion server tests (1-GPU)" has conclusion: null; steps 10 (if: always() artifact upload), 20 (Post Checkout), and Complete job never ran. (verified from job steps API for job 88799241374)
• Job log blob returns <Error><Code>BlobNotFound> and run has total artifacts: 0 — no test-level evidence was captured.
• Step-9 wall time ≈ 74 min (04:32:11 → 05:45:55 UTC), under the configured timeout-minutes: 90, so this is not a GitHub step timeout.
• Same run, other shards behave the same: shard 1 88799241355 and 2-GPU shard 0 88799241370 also show step 9 None after ~85 min. Shard 3 88799241359 instead produced a clean failure with an uploaded log.
• Sibling shard 3 log evidence (partition 3 = test_update_weights_from_disk.py): 4 failed, 2 passed, 4 errors in 939.66s, all AssertionError: No weights dir for vae/transformer ... for FLUX.2-klein-base-4B and Qwen-Image snapshots under /sgl-data/hf-cache/hub/... — i.e. missing/incomplete diffusion model weights in the shared HF cache. (log line ~12148)
• HEAD commit 8ae0eb83 modifies only spec-decode model files; the failure predates it by 8+ days.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [LOW] Runner/container abnormally terminated (hang or OOM/GPU fault) during a long-running diffusion server test in partition 0.
  • Supporting: step 9 concluded None and the if: always() upload step never ran across the majority of the 15 historical runs; ~74 min duration; no log ever uploaded (consistent with the self-hosted runner losing the process/host rather than pytest exiting).
  • Disconfirming: with no log or artifact, the hang location is unobservable; could equally be a networking/download stall rather than a GPU/OOM fault.
  • How to verify: re-run shard 0 with a shorter internal per-test timeout and server-side stdout tee'd to the mounted diffusion-failures/ dir; watch rocm-smi/dmesg on the runner during the hang.
• [LOW] Diffusion model weights missing/incomplete in the shared /sgl-data/hf-cache, causing a download/load stall on the large model(s) that LPT bin-packing places in partition 0 (FLUX.1 / HunyuanVideo / Wan2.1).
  • Supporting: sibling partition 3 fails today with explicit No weights dir for vae/transformer for FLUX.2-klein-base-4B and Qwen-Image, proving the shared cache is at least partially broken; a stalled re-download of a multi-GB model would explain a 74-min no-output hang.
  • Disconfirming: partition 3 produced a fast, clean assertion (26 min) rather than a hang — a missing-weights condition asserted quickly there, so it does not obviously explain a 74-min hang in partition 0.
  • How to verify: inspect the "Diagnose HF cache" step output for shard 0 (✓/✗ CACHED lines) and confirm which partition-0 model shows ✗ NOT CACHED.
• [FACT] HEAD commit 8ae0eb83 is not the cause: it changes only dspark.py/dflash.py (speculative decoding), and the same shard-0 failure exists in every scheduled run back to 2026-07-14.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for multimodal_gen+amd (25 results, none address a shard hang / runner death), diffusion+amd+hang (4, unrelated kernel PRs), diffusion+test+timeout (23, none matching). No open PR targets the shard-0 abnormal-termination / missing-diffusion-weights problem.

Suggested Triage Steps

• Treat this as an infra/hang investigation, not a code regression — there is no green baseline to bisect against (Never-passed in the 15-run window).
• Reproduce shard 0 on an MI325 1-GPU runner with the exact command (python3 sglang/multimodal_gen/test/run_suite.py --suite 1-gpu --partition-id 0 --total-partitions 4 -k "not flux_2"), capturing server stdout to a mounted path so evidence survives a runner death.
• Check the shard-0 "Diagnose HF cache and system resources" step output to see which partition-0 diffusion model is ✗ NOT CACHED; pre-seed /sgl-data/hf-cache/hub with complete snapshots (the sibling No weights dir failures show the shared cache is currently incomplete).
• Add an internal per-test watchdog/timeout shorter than 90 min inside run_suite.py so a stalled model download/inference fails fast with a log instead of killing the runner and losing all evidence.
• During a repro hang, capture rocm-smi, dmesg, and a py-spy dump of the server process to distinguish a GPU fault from a download/network stall.

Status

• Persistence: shard 0 has failed in every completed scheduled run since at least 2026-07-14 (8+ days); dominant mode is step-9 None (runner death), with occasional container-start infra failures.
• Scope: not isolated to shard 0 — shard 1 and 2-GPU shard 0 show the identical None termination in the same run; the broader multimodal-gen-test suite is red (shards 0/1/3 + 2-GPU shard 0 all failing today). Shared root appears to be the diffusion test environment (HF weight cache and/or runner stability), not one test.
• Blocked work: blocks all AMD MI325 1-GPU diffusion/multimodal_gen CI signal.
• In-flight fix: ❌ none found in open PRs — needs new investigation.

multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 3) — failed step(s): Run diffusion server tests (1-GPU)

Commit Info

• sglang: b13abfde
• aiter: 9127c94a (v0.1.17.dev110+g9127c94a1, source: Dockerfile default — AITER_COMMIT_OVERRIDE empty, no dev/patched build)

Failed Tests

Test file: python/sglang/multimodal_gen/test/single_test_file/test_update_weights_from_disk.py (4 failed, 2 passed, 4 errors in 1010.86s).

Test File	Test Function	Error	Log
test_update_weights_from_disk.py	TestUpdateWeightsFromDisk::test_update_weights_from_disk_default[Qwen-Image]	assert 400 == 200 (line 432)	Log
test_update_weights_from_disk.py	TestUpdateWeightsFromDisk::test_update_weights_specific_modules[Qwen-Image]	AssertionError: Update failed: {...} All modules rolled back → assert 400 == 200 (line 454)	Log
test_update_weights_from_disk.py	TestUpdateWeightsFromDisk::test_corrupted_weights_rollback[Qwen-Image]	assert (400 == 200) (line 535)	Log
test_update_weights_from_disk.py	TestUpdateWeightsFromDiskWithOffload::test_update_weights_with_offload_enabled[Qwen-Image]	AssertionError: Expected 200, got 400 (line 638)	Log
test_update_weights_from_disk.py	TestUpdateWeightsFromDisk::test_update_weights_from_disk_default[FLUX.2-klein-base-4B]	ERROR at setup: AssertionError: No weights dir for transformer in /tmp/sglang_perturbed_vae_* (line 410→220)	Log
test_update_weights_from_disk.py	TestUpdateWeightsFromDisk::test_update_weights_specific_modules[FLUX.2-klein-base-4B]	ERROR at setup: same as above	Log
test_update_weights_from_disk.py	TestUpdateWeightsFromDisk::test_update_weights_rejects_invalid_requests[FLUX.2-klein-base-4B]	ERROR at setup: same as above	Log
test_update_weights_from_disk.py	TestUpdateWeightsFromDisk::test_corrupted_weights_rollback[FLUX.2-klein-base-4B]	ERROR at setup: same as above	Log

Failure Summary

The multimodal-gen-test-1-gpu-amd shard-3 job runs one test file, test_update_weights_from_disk.py, parametrized over two models. All Qwen-Image cases fail: the server rejects the on-disk weight update with HTTP 400. All FLUX.2-klein-base-4B cases error during fixture setup because the perturbed model directory has no transformer weights subdir. Two passing cases are the /nonexistent/path rejection test and one Qwen-Image invalid-request test.

Regression Status

Known recurring failure (persistent) — not a new regression at b13abfde. The same test file fails with the identical two symptoms in every scheduled run examined; no green run found in the query window.

Recent history of test_update_weights_from_disk.py in job multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 3) (scheduled, event=schedule, branch main, completed runs; ✅/❌ marks test-file status where log-verified, else job conclusion):

Date (UTC)	Run	Job	Test File Status	Failed Functions	Error
Jul 22 12:32	29920027108	job	❌ Failed (verified)	Qwen-Image ×4 + FLUX.2 ×4	shape mismatch / no weights dir
Jul 21 12:32	29830494225	job	❌ Failed (verified)	same (4 failed, 4 errors)	same
Jul 21 00:27	29790347494	job	❌ Failed (job)	—	—
Jul 20 12:42	29743214297	job	❌ Failed (job)	—	—
Jul 20 00:28	29709549202	job	❌ Failed (job)	—	—
Jul 19 12:17	29686709871	job	❌ Failed (job)	—	—
Jul 19 00:27	29666971143	job	❌ Failed (verified)	same (5 failed, 4 errors)	same
Jul 18 12:17	29644059344	job	❌ Failed (job)	—	—
Jul 18 00:25	29623067212	job	❌ Failed (verified)	same (5 failed, 4 errors)	same

Lookback range: Jul 18 00:25 → Jul 22 12:32 UTC (9 completed scheduled runs). Log-verified same test-file + same symptoms on Jul 18, 19, 21, 22; the intervening runs are ❌ at job level. First observed failure: at/before the earliest verified run (Jul 18); no passing run found for this test file in the window → treat as Never-passed on the AMD lane within the query window.

Failure Cluster

Two distinct symptom clusters in the same job:

• Cluster A — "Qwen-Image on-disk weight update rejected (HTTP 400) via linear.py weight_loader shape assert"
• Cluster B — "FLUX.2 update-weights fixture setup error: perturbed model dir missing transformer weights subdir"

Facts (verified observations)

• Final result: 4 failed, 2 passed, 11 warnings, 4 errors in 1010.86s.
• Cluster A (Qwen-Image) server-side traceback (log ~14:17:23): weights_updater.py:434 _apply_weights → :203 _load_weights_into_module → :295 load_weights_into_model → weight_loader(param, loaded_weight.to(param.dtype)) → linear.py:410 assert param_data.shape == loaded_weight.shape → AssertionError. Server logs: Weight update failed for module 'transformer': . Rolling back … ['text_encoder','transformer'], so client receives HTTP 400 and every Qwen-Image test that expects 200 fails.
• Cluster B (FLUX.2) all four cases error at setup of fixture diffusion_server_no_offload: test_update_weights_from_disk.py:410 (sanity check that base vs perturbed checksums differ) → :220 _compute_checksum_from_disk → AssertionError: No weights dir for transformer in /tmp/sglang_perturbed_vae_*. The perturbed-VAE temp dir does not contain a transformer/ subdir when the checksum sanity check runs.
• The two implicated source files, python/sglang/multimodal_gen/runtime/layers/linear.py:410 and .../runtime/post_training/weights_updater.py, are byte-identical between failing HEAD b13abfde and the earlier failing runs (local shallow clone shows only boundary commit 97e0647b for these files) — no recent code change to the asserting path.
• aiter is unchanged (Dockerfile default 9127c94a, no override) — not an aiter-variable failure; NVIDIA is not the affected lane.
• Non-fatal noise in log: MIOpen(HIP): Error [Init] Not found (conv kernel db miss), ROCm GPU memory not fully cleared after 60s warnings, Log pipe thread error: I/O operation on closed file at teardown — none are the test-failure cause.

Hypothesised Causes (with confidence)

• [MEDIUM] Cluster A: Qwen-Image transformer weight tensors loaded from disk do not match the instantiated parameter shapes on the ROCm lane (assert param_data.shape == loaded_weight.shape at linear.py:410, after the TP narrow at :403). Reproduces every run with unchanged code + unchanged aiter, so it is a stable model/loader-vs-AMD interaction, not a flake.
  • Supporting: identical traceback and module (transformer) across Jul 18/19/21/22; only the AMD lane is exercised here.
  • Disconfirming: no green baseline exists for this test file in the window, so it may be a Never-passed AMD port rather than a true regression; without an NVIDIA/x-lane comparison the shape-mismatch origin (TP config vs checkpoint layout) is unconfirmed.
  • How to verify: run test_update_weights_from_disk.py::TestUpdateWeightsFromDisk::test_update_weights_from_disk_default[Qwen-Image] on AMD MI325 (rocm700 image) with extra logging of param_data.shape vs loaded_weight.shape at linear.py:410; compare the same test on an NVIDIA multimodal-gen lane.
• [MEDIUM] Cluster B: the perturbed-VAE fixture builds a temp model dir that lacks a transformer/ subdir (or maybe_download_model returns a path without it) for FLUX.2-klein-base-4B, so the _DIFFERING_MODULES sanity assertion at :408-410 fails before the server starts.
  • Supporting: error is in fixture setup (ERROR at setup), identical string across runs; only FLUX.2 parametrization hits it (Qwen-Image reaches the server).
  • Disconfirming: could be an incomplete/partial FLUX.2 download on the runner rather than a test-logic bug — the _perturb/_truncate_safetensor background threads may not have populated transformer/.
  • How to verify: inspect _resolve_active_model_pairs / the perturbed-dir construction for FLUX.2 and list /tmp/sglang_perturbed_vae_*/ contents on the runner during setup.
• [LOW] Both clusters are AMD-only environment issues (kernel/db or download) rather than code — weakened by the deterministic, repeated, identical signatures.

In-flight Fix Check

• In-flight fix: ❌ none found for the shape-mismatch / setup-error itself. Closest related open PR: #31483 "[AMD] ci: run vetted nested multimodal_gen unit tests on AMD" (open since 2026-07-16) — it governs which multimodal_gen tests run on AMD but targets the unit/ suite, not this single_test_file/ test, so it is not a fix. Searched is:pr is:open update_weights_from_disk (17 hits, none address AMD shape mismatch) and multimodal_gen weights (23 hits).

Suggested Triage Steps

• Confirm scope: run the failing test file on AMD MI325 (rocm/sgl-dev:*-rocm700-mi30x) and capture the two shapes at linear.py:410 for Qwen-Image transformer to determine whether the mismatch is a TP-sharding assumption or a checkpoint-layout mismatch.
• For Cluster B, dump the perturbed-VAE temp dir tree during fixture setup for FLUX.2 to confirm whether transformer/ is missing due to download vs perturbation-thread logic.
• Since no green run exists on this lane in the window, treat these as Never-passed AMD ports: consider gating/xfail on AMD for the affected [Qwen-Image] and [FLUX.2-klein-base-4B] params until root-caused, and coordinate with the owner of the multimodal_gen AMD enablement (#31483).
• Compare against an NVIDIA multimodal-gen run of the same test file to establish whether these are AMD-specific.

Status

• Persistence: Failing on every completed scheduled pr-test-amd.yml run since at least 2026-07-18 00:25 UTC (log-verified Jul 18/19/21/22; job-level ❌ on all intervening runs) — ~4.5 days, no green run in the 9-run window.
• Scope: multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 3) on the AMD lane; two symptom clusters (Qwen-Image weight-update 400; FLUX.2 fixture setup error) in one test file.
• Blocked work: on-disk weight-update (RL post-training) CI signal for diffusion models on AMD.
• In-flight fix: ❌ none found — needs new investigation; #31483 is related CI-enablement, not a fix.

multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 1) — failed step(s): N/A

Commit Info

• sglang: b13abfde — "Fix LongCat n-gram token-table crashes on padded batches" (#31312)
• aiter: N/A — the job log blob is BlobNotFound (runner died before uploading logs), so no [CI-AITER-CHECK] markers are recoverable.

Failed Tests

Test File	Test Function	Error	Log
N/A (non-test failure)	N/A	The self-hosted runner lost communication with the server (runner death during step 9 "Run diffusion server tests (1-GPU)")	Job

This is not a test-level failure — no pytest results were produced. The job (matrix shard linux-mi325-1gpu-sglang, 1) died mid-execution.

Failure Summary

The multimodal-gen-test-1-gpu-amd (…, 1) job ran for ~77 min (14:01:25 → 15:18:33 UTC) and then the self-hosted MI325 runner linux-mi325-1gpu-sglang-ng8rq-runner-vrhhq lost communication with GitHub while step 9 "Run diffusion server tests (1-GPU)" was still in_progress. Because the runner died, step 9 never concluded (conclusion: None), step 10 (artifact upload) never ran, and the log blob was never uploaded (BlobNotFound). The GitHub check annotation records the failure verbatim as a lost-runner-communication event.

Regression Status

Known recurring / Infrastructure — Never-passed in the queryable window. Shard 1 of this matrix job has failed in all 15 most-recent completed scheduled runs on main, every time with the identical "runner lost communication" annotation. Shard 0 shows the same signature; shard 2 passes; shard 3 has genuine test failures — indicating shards 0 & 1 reliably starve/kill the runner during the diffusion server tests. No green run exists in the lookback window (2026-07-14 → 2026-07-22).

Recent history of multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 1) (most recent 15 comparable completed scheduled runs: branch=main, event=schedule, completed only):

Date	Run	Job	Test File Status	Failed Function	Error
Jul 22	run	shard 1	❌ Runner died	— (step 9 in_progress)	runner lost communication
Jul 21	run	shard 1	❌ Runner died	—	runner lost communication
Jul 21	run	shard 1	❌ Runner died	—	runner lost communication
Jul 20	run	shard 1	❌ Runner died	—	runner lost communication
Jul 20	run	shard 1	❌ Runner died	—	runner lost communication
Jul 19	run	shard 1	❌ Runner died	—	runner lost communication
Jul 19	run	shard 1	❌ Runner died	—	runner lost communication
Jul 18	run	shard 1	❌ Runner died	—	runner lost communication
Jul 18	run	shard 1	❌ Runner died	—	runner lost communication
Jul 17	run	shard 1	❌ Runner died	—	runner lost communication
Jul 17	run	shard 1	❌ Runner died	—	runner lost communication
Jul 16	run	shard 1	❌ Runner died	—	runner lost communication
Jul 15	run	shard 1	❌ Runner died	—	runner lost communication
Jul 15	run	shard 1	❌ Runner died	—	runner lost communication
Jul 14	run	shard 1	❌ Runner died	—	runner lost communication

First observed failure in window: 2026-07-14 (earliest queried); no passing run found for shard 1 in the entire 15-run window → Never-passed. (Annotation verified verbatim on 6 of these runs; identical step9=in_progress/NA + failure signature on the remaining ones.)

Failure Cluster

Self-hosted MI325 runner lost communication during diffusion server tests (multimodal-gen shards 0 & 1)

Facts (verified observations, no interpretation)

• Check annotation (verbatim): "The self-hosted runner lost communication with the server. Verify the machine is running and has a healthy network connection. Anything in your workflow that terminates the runner process, starves it for CPU/Memory, or blocks its network access can cause this error."
• Job conclusion: failure; step 9 "Run diffusion server tests (1-GPU)" conclusion: None (never finished); step 10 "Upload diffusion failure artifacts" pending; log blob = BlobNotFound; 0 run artifacts.
• Runner: linux-mi325-1gpu-sglang-ng8rq-runner-vrhhq (label linux-mi325-1gpu-sglang). Duration 14:01:25 → 15:18:33 UTC (~77 min).
• Matrix pattern across the last 15 scheduled runs: shard 1 = fail (runner lost) in 15/15; shard 0 = same signature; shard 2 = pass; shard 3 = genuine test failure (step9: failure).
• Identical annotation confirmed on historical shard-1 jobs 88633865546, 88510605277, 88354737052, 88251525271, 88138987994.
• Head commit b13abfde touches only n_gram_embedding.py / ngram_embedding_manager.py (LongCat speculative n-gram) — unrelated to diffusion/multimodal code paths.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [MEDIUM] Resource starvation (VRAM/host-RAM/OOM) or process crash during the diffusion server tests on shards 0 & 1 kills the runner agent.
  • Supporting: annotation explicitly lists "terminates the runner process, starves it for CPU/Memory" as causes; the death occurs specifically inside the heaviest step (step 9, diffusion server startup); shards 0 & 1 consistently die while shard 2 (a different model subset) always survives → workload-partition dependent, consistent with an OOM/crash triggered by the specific models assigned to those shards.
  • Disconfirming: no server/OOM log is recoverable (BlobNotFound), so the crash cannot be directly confirmed vs. a pure network/host drop; a transient network fault would produce the same annotation.
  • How to verify: rerun shard 1 with host dmesg/nvidia-smi/rocm-smi memory logging enabled, or reduce the shard's model set; check GHA runner host metrics at the death timestamp.
• [LOW] Genuine self-hosted infra/network instability unrelated to the workload.
  • Supporting: annotation names network access as a possible cause.
  • Disconfirming: the failure is perfectly deterministic (15/15 runs, always shards 0 & 1, never shard 2) — pure network flakiness would not be so precisely partition-correlated.
  • How to verify: correlate against other jobs on the same runner pool at the same times; if only shards 0/1 die, infra-network is unlikely.
• Not the head commit: b13abfde is a LongCat n-gram speculative-decoding fix; it does not touch any diffusion/multimodal path and the failure predates it (Never-passed for ≥8 days). No regression window to bisect.

In-flight Fix Check

• In-flight fix: ❌ none found for the runner-lost-communication crash on the diffusion server test shards. Searched repo:sgl-project/sglang is:pr is:open with keywords runner lost communication (0 hits), multimodal-gen diffusion amd, diffusion server test runner.
• Related but non-matching open PRs (different job/symptom, do NOT rely on these to fix shard 1): #31899 fixes multimodal-gen-unit-test-amd-rocm720 ModuleNotFoundError: msgpack (a unit test, ROCm720, ImportError — not a runner death); #31846 quarantines pre-existing nested unit failures. Neither addresses the diffusion server-test runner crash.

Suggested Triage Steps

• Reproduce shard 1 in the linux-mi325-1gpu-sglang container with host resource logging (rocm-smi, free -m, dmesg -w) tailing during step 9 to confirm/deny an OOM or process kill at the death timestamp.
• Compare the model/test partition assigned to shards 0 & 1 vs shard 2 (which always survives) in the diffusion server test matrix — a heavier model set on 0/1 would support the resource-starvation hypothesis.
• If OOM confirmed, consider lowering per-shard concurrency or memory footprint for the diffusion server on MI325; if network/infra, escalate to the runner pool owner with the death timestamps.
• Head commit b13abfde needs no action for this job — the failure is pre-existing and unrelated.

Status (factual, no priority assignment)

• Persistence: Failing continuously for ≥8 days (every completed scheduled run 2026-07-14 → 2026-07-22, 15/15); Never-passed in the queryable window.
• Scope: Deterministically affects shards 0 & 1 of multimodal-gen-test-1-gpu-amd on pr-test-amd.yml (MI325 runner). Shard 2 passes; shard 3 fails for a separate (real test) reason.
• Blocked work: Removes diffusion/multimodal-gen server-test signal for shards 0 & 1 on AMD; no artifacts/logs produced, so those shards give zero coverage.
• In-flight fix: ❌ none found for the runner-lost crash; related unit-test PRs #31899 / #31846 do not address it.

stage-c-test-large-8-gpu-amd-mi35x (linux-mi35x-gpu-8, 1) — failed step(s): Run test

Commit Info

• sglang: b13abfde
• aiter: 9127c94a — v0.1.17.dev110+g9127c94a1 (source: dev/patched image preserved; Dockerfile default AITER_COMMIT=9127c94a..., rebuild skipped). AITER_COMMIT_OVERRIDE empty — this is not a scout run, so no Baseline A/B check applies.

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/test_deepseek_r1_mxfp4_8gpu.py	server launch (setUpClass-level; all TestDeepseekR1MXFP4* methods blocked)	RuntimeError: Capture prefill CUDA graph failed: Unsupported method call → torch._dynamo.exc.Unsupported	Log

The other two files in this shard passed: test_kimi_k25_mxfp4.py ✅, test_qwen3_coder_next_8gpu.py ✅. Shard result: 2/3 files, exit code 255.

Failure Summary

The DeepSeek-R1-MXFP4 server (TP=8, --cuda-graph-backend-prefill=tc_piecewise) crashes during startup while capturing the prefill CUDA graph. All 8 TP scheduler ranks hit torch._dynamo.exc.Unsupported: Unsupported method call — Dynamo does not know how to trace method __getattribute__ of class type (gb0156) while torch.compile traces the DeepSeek-V2 forward pass. The server never becomes ready, so every test function in the file is blocked and the file exits with code 1.

Regression Status

New regression.

Recent history of test_deepseek_r1_mxfp4_8gpu.py in job stage-c-test-large-8-gpu-amd-mi35x (linux-mi35x-gpu-8, 1) (comparable scheduled main runs; partition 1 is where this file runs):

Date	Run	Job	Test File Status	Failed Function	Error
Jul 22 12:32	29920027108	partition 1	❌ Failed	server launch	Unsupported method call (Dynamo)
Jul 21 12:32	29830494225	partition 1	✅ Passed (3/3)	—	—
Jul 22 00:26	29880220845	(partition 1)	[IN-FLIGHT]	—	run still in_progress at snapshot; head 8ae0eb83 predates the suspect commit

Runs on Jul 20 and earlier all predate the suspect commit and cannot exhibit this failure. First observed failure: 2026-07-22 (this run). Last known pass: 2026-07-21 12:32 UTC — verified at the test-file level (baseline log shows Test Summary: 3/3 passed for this file). The current run b13abfde is the first completed scheduled run whose tree contains commit 745b2ca4 (merged Jul 22 01:18 UTC, between the pass and the fail).

Failure Cluster

Dynamo graph break during tc_piecewise prefill CUDA graph capture (__getattribute__ of ParallelContext)

Facts (verified observations)

• Error (verbatim): torch._dynamo.exc.Unsupported: Unsupported method call / Explanation: Dynamo does not know how to trace method __getattribute__ of class type / Developer debug context: call_method BuiltinVariable(object) __getattribute__ [UserDefinedObjectVariable(ParallelContext), ConstantVariable(str: '_config')] (log ~line 4977–4998).
• Stack trace (user code): deepseek_v2.py:2597 forward → deepseek_v2.py:2231 forward → communicator_dsa_cp.py:63 maybe_prefetch_next_full_attention_kv → dsa/utils.py:277 dsa_use_prefill_cp → dsa/utils.py:79 is_dsa_enable_prefill_cp → runtime_context.py:130 ParallelContext.__getattr__ → config = object.__getattribute__(self, "_config").
• Outer stack: PrefillCudaGraphRunner.__init__ → resolve_prefill_backend → TcPiecewiseCudaGraphBackend.__init__ → _run_compile_pass → _run_dummy_forward → torch.compile trampoline (compile.py:224).
• Reproducibility: all 8 TP ranks (TP0–TP7) hit the identical error; deterministic, not flaky.
• Config: cuda_graph_backend_prefill='tc_piecewise', enable_dsa_prefill_context_parallel=False, attention_backend='aiter', TP=8.
• Code-level: python/sglang/srt/runtime_context.py:126-140 — ParallelContext.__getattr__ serves config leaves via object.__getattribute__(self, "_config"). enable_dsa_prefill_context_parallel is a config leaf (not a @property), so access routes through __getattr__.
• Environment: baseline (pass) and current (fail) both use the same aiter dev image v0.1.17.dev110+g9127c94a1; aiter is unchanged across the pass→fail boundary.

Hypothesised Causes (with confidence)

• [HIGH] Commit 745b2ca4 ("config: read parallel config leaves via get_parallel()", #31816) changed is_dsa_enable_prefill_cp() in dsa/utils.py:79 from get_server_args().enable_dsa_prefill_context_parallel to get_parallel().enable_dsa_prefill_context_parallel.
  • Supporting: (a) the exact modified line dsa/utils.py:79 is the top user frame in the failing stack; (b) the crash is inside ParallelContext.__getattr__ (runtime_context.py:130), reached only because enable_dsa_prefill_context_parallel is a config-bag leaf served via __getattr__→object.__getattribute__, which Dynamo cannot trace (gb0156). The pre-commit get_server_args() returned the frozen ServerArgs where this is a plain traceable field; (c) temporal correlation is tight — merged Jul 22 01:18, between the last pass (Jul 21 12:32) and this fail; (d) test passed 3/3 in the immediately preceding run and fails deterministically now, aiter unchanged.
  • Disconfirming: (none found) — the failing frame is precisely the line this commit changed, and no other commit in the window touches this call path.
  • How to verify: on a branch, revert the dsa/utils.py hunk of 745b2ca4 (restore get_server_args() for is_dsa_enable_prefill_cp), or run the file with --cuda-graph-backend-prefill=breakable / =disabled, and re-run test_deepseek_r1_mxfp4_8gpu.py on AMD MI35x.
• [LOW] Pre-existing Dynamo-incompatibility in ParallelContext.__getattr__ (object.__getattribute__) that only became reachable now.
  • Supporting: the untraceable construct lives in runtime_context.py, not in 745b2ca4.
  • Disconfirming: runtime_context.py was not on any Dynamo-traced path for this test before the call-site switch, so it is a latent hazard exposed by 745b2ca4, not an independent cause.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for get_parallel+tc_piecewise, ParallelContext+dynamo, enable_dsa_prefill_context_parallel, and prefill cuda graph get_parallel (all 0 results). The three PRs referencing #31816 (#31817, #31818) are all closed. Needs a new PR/investigation.

Suggested Triage Steps

• Revert only the dsa/utils.py hunk of 745b2ca4 on a branch and re-run test_deepseek_r1_mxfp4_8gpu.py on MI35x to confirm attribution.
• If confirmed, likely fix directions (maintainer decides): make ParallelContext config-leaf access Dynamo-traceable (avoid object.__getattribute__ on the compiled path), or guard is_dsa_enable_prefill_cp() behind is_in_tc_piecewise_cuda_graph() / read the flag outside the compiled region, or keep get_server_args() for leaves accessed inside torch.compile.
• Interim unblock for CI signal: this file/model runs prefill CUDA graph with tc_piecewise; the server's own error message suggests --cuda-graph-backend-prefill=breakable or =disabled as a workaround to verify the rest of the pipeline.

Status

• Persistence: First failing run — new as of 2026-07-22 (last pass 2026-07-21 12:32 UTC).
• Scope: test_deepseek_r1_mxfp4_8gpu.py on job stage-c-test-large-8-gpu-amd-mi35x (…, 1) in pr-test-amd.yml. Any DeepSeek-V2/R1 model launched with tc_piecewise prefill CUDA graph on a path reaching is_dsa_enable_prefill_cp() is at risk; other files in the shard (Kimi, Qwen3-Coder) are unaffected.
• Blocked work: DeepSeek-R1-MXFP4 8-GPU CI signal on AMD MI35x.
• In-flight fix: ❌ none found — needs new PR or investigation.

multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 0) — failed step(s): N/A

Commit Info

• sglang: b13abfde — "Fix LongCat n-gram token-table crashes on padded batches (#31312)"
• aiter: N/A — the part-0 job log blob does not exist (see Facts), so no [CI-AITER-CHECK] markers are readable; this is not a scout run.

Failed Tests

Test File	Test Function	Error	Log
(cannot pin — no log)	(cannot pin — no log)	Step 9 Run diffusion server tests (1-GPU) hung ~78 min, never concluded; job killed with conclusion=failure; step left in_progress	Job

This is not a clean test failure — it is a job that was terminated mid-step. The multimodal_gen 1-GPU server suite (python/sglang/multimodal_gen/test/run_suite.py --suite 1-gpu --partition-id 0) runs a deterministic LPT-partitioned subset of heavy diffusion e2e cases (ONE_GPU_CASES in python/sglang/multimodal_gen/test/server/gpu_cases.py:71 — Wan2.1/2.2 T2V/I2V/TI2V, FastHunyuan, FLUX, Qwen-Image). The exact hanging case for partition 0 cannot be identified because no log or artifact survived.

Failure Summary

The part-0 shard of multimodal-gen-test-1-gpu-amd failed because step 9 Run diffusion server tests (1-GPU) ran for ~78 minutes and never returned — the job was killed (conclusion=failure) with the step still in_progress. The GitHub job-log blob returns BlobNotFound, zero run artifacts were produced, and the if: always() "Upload diffusion failure artifacts" step never executed, indicating the runner/step was terminated abruptly rather than failing an assertion.

Regression Status

Never-passed (persistent) — the part-0 shard has failed in every completed scheduled run in the queryable 15-run window (2026-07-14 → 2026-07-22). No green run found. Lookback range: runs 29295847781 (2026-07-14) → 29920027108 (2026-07-22, current).

Recent history of multimodal-gen-test-1-gpu-amd (…, 0) (job-level; test-function granularity unavailable because part-0 logs do not survive the hang), most recent comparable completed scheduled runs (branch=main, event=schedule, completed only):

Date	Run	Job	Shard Status	Signature
Jul 22	run	job	❌ Failed	step 9 in_progress, ~78m hang, no log
Jul 21	run	job	❌ Failed	step 9 in_progress, ~112m hang
Jul 21	run	job	❌ Failed	step 9 in_progress hang
Jul 20	run	job	❌ Failed	step 9 in_progress hang
Jul 20	run	job	❌ Failed	step 9 in_progress hang
Jul 19	run	job	❌ Failed	failed (pre-step-9 / non-hang)
Jul 19	run	job	❌ Failed	step 9 in_progress hang
Jul 18	run	job	❌ Failed	step 9 in_progress hang
Jul 18	run	job	❌ Failed	step 9 in_progress hang
Jul 17	run	job	❌ Failed	step 9 in_progress hang
Jul 16	run	job	❌ Failed	step 9 in_progress hang
Jul 15	run	job	❌ Failed	failed (pre-step-9 / non-hang)
Jul 14	run	job	❌ Failed	failed (pre-step-9 / non-hang)
Jul 14	run	job	❌ Failed	failed (pre-step-9 / non-hang)

No passing run exists in this window → first observed failure predates the lookback (≥ 2026-07-14); no last-known-passing date found.

Failure Cluster

Diffusion 1-GPU server test hangs → job killed mid-step, no logs/artifacts

Facts (verified observations)

• Step 9 Run diffusion server tests (1-GPU) started 2026-07-22T14:01:07Z; job completed conclusion=failure at 2026-07-22T15:19:16Z (~78 min later) with the step still status=in_progress, conclusion=None (verified via jobs API).
• Step 9 has timeout-minutes: 90 (.github/workflows/pr-test-amd.yml:751); the job died at ~78 min, before the step's own 90-min timeout — so a clean step-level timeout did not fire.
• The job-log blob returns BlobNotFound (both the API .../jobs/88922971025/logs and its signed Azure redirect). No part-0 log is retrievable.
• Run artifacts total = 0; step 10 Upload diffusion failure artifacts (if: always()) is pending and never ran — the runner was terminated before graceful teardown.
• Same-run siblings (rules out a runner-wide outage): part 1 = ❌ hang (~68m, step 9 in_progress); part 3 = ❌ clean fail (~17m, step 9 concluded — test_update_weights_from_disk.py::TestUpdateWeightsFromDisk::test_update_weights_from_disk_default[Qwen-Image] FAILED, [FLUX.2-klein-base-4B] ERROR "No weights dir for transformer/vae"); part 2 = ✅ pass (~5m). The multimodal-gen-unit-test-amd job passed.
• Runner: linux-mi325-1gpu-sglang (AMD MI325, 1-GPU), runner version 2.336.0 (from part-3 log).

Hypothesised Causes (with confidence)

• [LOW] A heavy video/image diffusion e2e case assigned to partition 0 (Wan T2V/I2V/TI2V, FastHunyuan, or similar in ONE_GPU_CASES) hangs on MI325 during server startup or the denoise loop, so step 9 spins until an external watchdog kills the runner.
  • Supporting: parts 0 and 1 both hang for 68–78 min while part 2 (light cases) passes in 5 min on the same runner type; the hang recurs identically across the entire 15-run window (deterministic LPT partition ⇒ part 0 always gets the same heavy cases).
  • Disconfirming: no part-0 log or artifact survives, so the specific hanging model/stack is unverified; this is symptom-level clustering, not a code-level root cause. The kill at ~78 min (< the 90-min step timeout) is not fully explained and could also be a runner-host/GPU watchdog rather than the test itself.
  • How to verify: re-run part 0 with a shorter per-case timeout and --continue-on-error, or run each partition-0 case individually on an MI325 in the CI container to find which model hangs; capture py-spy dump / rocm-smi while hung.
• [LOW] Infra/runner-loss (GPU reset, host OOM, or watchdog) terminates the runner before logs upload.
  • Supporting: BlobNotFound, zero artifacts, always() upload never ran — classic abrupt-termination signature.
  • Disconfirming: it recurs on this specific shard every run for 8+ days, which is too consistent for random infra flake; part 2 succeeds on the same runner pool.
  • How to verify: check the self-hosted runner host's dmesg/system logs and any GPU-reset events for linux-mi325-1gpu-sglang at the affected times.

No hypothesis reaches MEDIUM: the definitive evidence (part-0 log/artifact) does not exist. Needs a live re-run with instrumentation to identify the hanging case.

In-flight Fix Check

• In-flight fix: ❌ none found for the part-0 diffusion-server hang specifically (searched open PRs for diffusion + amd, multimodal-gen, diffusion hang/timeout — no PR targets this shard's hang). Query: search/issues?q=repo:sgl-project/sglang+is:pr+is:open+diffusion+hang+OR+timeout.
• Note: #31925 "[AMD] Skip test_update_weights_from_disk on ROCm pending reload fix (#31924)" (open since 2026-07-21) addresses the part-3 test_update_weights_from_disk failure, not the part-0 hang — do not conflate them.

Suggested Triage Steps

• Reproduce on an MI325 in the CI container: run each partition-0 case from the 1-gpu suite individually (run_suite.py --suite 1-gpu --partition-id 0 --total-partitions 4 then narrow with -k) to identify which model hangs; attach py-spy dump and rocm-smi while it is stuck.
• Add a per-case/server-startup timeout so a hung case fails fast and uploads the diffusion-failure artifact instead of taking the whole runner down (currently step 9's 90-min timeout is never reached because the runner dies first).
• Inspect the linux-mi325-1gpu-sglang runner host dmesg / GPU-reset logs around the affected timestamps to distinguish a test-level deadlock from a GPU/host watchdog kill.
• If a single model is confirmed to hang on ROCm, quarantine that case on AMD (as done for test_update_weights_from_disk in #31925) until separately fixed.

Status

• Persistence: Failing on the part-0 shard in every completed scheduled run for ≥ 8 days (since ≥ 2026-07-14; no green run in the 15-run window).
• Scope: Affects multimodal-gen-test-1-gpu-amd shards 0, 1, and 3 in the current run (2 hang, 1 clean-fail); shard 2 and the unit-test job pass. Confined to pr-test-amd.yml diffusion 1-GPU tests on MI325.
• Blocked work: Blocks AMD diffusion/multimodal_gen 1-GPU CI signal; the whole pr-test-amd.yml scheduled workflow is red partly because of this.
• In-flight fix: ❌ none for the part-0 hang; #31925 covers only the separate part-3 test_update_weights_from_disk failure.

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-07-22 19:32 UTC)

[amd-bot]

pr-test-amd.yml — Per-Job Analysis (continued 5/5)

multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 0) — failed step(s): N/A

Commit Info

• sglang: b13abfde ("Fix LongCat n-gram token-table crashes on padded batches (#31312)")
• aiter: N/A — the failing part-0 job's log was never persisted (see below), so [CI-AITER-CHECK] markers cannot be read. This is pr-test-amd.yml (no AITER_COMMIT_OVERRIDE); a sibling part-1 log from the prior scheduled run shows the override empty, i.e. Dockerfile-default aiter.

Failed Tests

Test File	Test Function	Error	Log
N/A (part-0 log never uploaded)	N/A	Abnormal job termination during "Run diffusion server tests (2-GPU)" — step ran ~90 min then the job ended in failure with the step never finalizing (conclusion: None); no log blob and no failure artifact were produced	Job

This is NOT a clean test failure — it is an abnormal termination. Test-file/function granularity is impossible to report for this job: GitHub returns BlobNotFound for its log (jobs/88922971070/logs), and the same absence is seen in every completed historical run (part-0's log is missing even from the run-level logs zip while parts 1 & 2 are present). The step is killed before it can flush logs.

Failure Summary

The multimodal-gen-test-2-gpu-amd partition 0 job ran the 2-GPU diffusion server tests for ~90 minutes (16:11:34 → job end 17:42:07 UTC), then terminated in failure without the test step (Run diffusion server tests (2-GPU), timeout-minutes: 150) ever finalizing — its step conclusion is None and no log/artifact was uploaded. Sibling partitions 2 (✅ 22 min) passed and 1 is still queued (the overall run is still in-flight). Partition 0 runs the largest LPT-weighted slice of heavy parametrized 2-GPU cases (Wan2.2-T2V-A14B, Wan2.1-T2V-14B, Qwen-Image, FLUX.1).

Regression Status

Known recurring failure — never-passed in the queryable window. Partition 0 of this job has failed in all 15 most recent comparable completed scheduled runs (branch=main, event=schedule), with the identical signature (long ~85-110 min run, step never finalizes, no log uploaded), while parts 1 & 2 pass. First failure is at/before the window edge (≥ 2026-07-13), so the true start predates this window.

Recent history of partition-0 job multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 0) (most recent 15 comparable completed scheduled runs; part-0 logs are never persisted, so status is at the partition/job level — per-test status is unknowable):

Date (UTC)	Run	Part-0 Job	Status	Notes
Jul 22 00:26	run	job	❌ Failed	85 min, step9 None, no log; parts 1&2 ✅
Jul 21 12:32	run	job	❌ Failed	~100 min
Jul 21 00:27	run	job	❌ Failed	~106 min
Jul 20 12:42	run	job	❌ Failed	~109 min
Jul 20 00:28	run	job	❌ Failed	~103 min
Jul 19 00:27	run	job	❌ Failed	~98 min
Jul 18 12:17	run	job	❌ Failed	~10 min (faster/different failure)
Jul 18 00:25	run	job	❌ Failed	~116 min
Jul 17 12:25	run	job	❌ Failed	~90 min
Jul 16 12:31	run	job	❌ Failed	~117 min
Jul 15 12:28	run	job	❌ Failed	~88 min
Jul 15 00:22	run	job	❌ Failed	~85 min
Jul 14 12:26	run	job	❌ Failed	~127 min
Jul 13 12:48	run	job	❌ Failed	~99 min
Jul 13 00:28	run	job	❌ Failed	~101 min

(No passing part-0 run exists in the 15-run window → Never-passed for this window. Current run 29920027108 is [IN-FLIGHT] overall: part 2 ✅, part 0 ❌, part 1 still queued.)

Failure Cluster

2-GPU diffusion partition-0 hang / abnormal job termination (step never finalizes, log never uploaded)

Facts (verified observations, no interpretation)

• Current job 88922971070: conclusion=failure. Step Run diffusion server tests (2-GPU) started 2026-07-22T16:11:34Z, completed_at=None, conclusion=None; job completed_at=2026-07-22T17:42:07Z (~90.5 min of step runtime, under the 150-min step timeout).
• Job log endpoint returns BlobNotFound (verified on repeated attempts). Run-level logs zip returns HTTP 404 (run still in-flight).
• No diffusion-failures-amd-2gpu-0-* artifact was produced (run artifacts total_count=0; the upload step never ran, conclusion=None).
• Prior completed run 29880220845: part-0 job failure 85 min, step9 conclusion=None; part-0 log is absent from the run-level logs zip, while part-1 and part-2 logs are present. Parts 1 (✅ 29 min) and 2 (✅ 22 min) passed.
• Same three-way pattern (part 0 fails long, parts 1/2 pass short) holds across all 15 comparable completed scheduled runs.
• Runner memory in the sibling part-1 log was healthy post-test (Mem: 3.0Ti total, 2.5Ti free), i.e. host RAM is not the bottleneck for the partitions that finish.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [LOW] A specific heavy 2-GPU case assigned to partition 0 by LPT (e.g. Wan2.2-T2V-A14B or Wan2.1-T2V-14B on 2×MI325) hangs or hard-crashes the process/runner, so the step is killed before the 150-min timeout and before logs flush.
  • Supporting: part 0 always gets the largest LPT slice and always runs 85-127 min then dies; parts 1/2 get lighter slices and finish in 22-29 min and pass. The abnormal-termination signature (step None, no log, no artifact) is consistent with a runner-level kill / lost connection rather than a pytest assertion.
  • Disconfirming: no log-level evidence exists to name the case or confirm a hang vs OOM vs GPU fault; one run (Jul 18 12:17) failed in only ~10 min, which does NOT fit the "long hang" pattern and may be a separate infra failure.
  • How to verify: re-run part 0 with --continue-on-error and per-case timeouts, or run partition 0's case list serially in the ci_sglang container on a 2×MI325 host and watch rocm-smi + dmesg for GPU fault / OOM-kill; capture output to a file so it survives a runner-loss.
• [LOW] GitHub log-upload/runner-infrastructure failure independent of the test (runner disconnects, so failure is reported with no log).
  • Supporting: BlobNotFound and missing artifact are hallmarks of a job that never finalized its steps.
  • Disconfirming: it is always the same partition (0) that fails, and always after a long runtime — a pure infra flake would not be partition-0-selective across 15 consecutive runs.
  • How to verify: check the self-hosted runner's system logs / GitHub runner diagnostics for the part-0 job around the ~90-min mark for an OOM-kill or agent disconnect.
• No hypothesis reaches MEDIUM: needs a captured part-0 log (redirect test output to a file inside the container) before root cause can be established.

In-flight Fix Check

• In-flight fix: ❌ none found targeting the part-0 hang/termination specifically. Searched open PRs for multimodal-gen 2-gpu AMD, diffusion 2-gpu timeout, diffusion AMD hang — matches are generic AMD/diffusion work (e.g. #31483, #27587, #25969), none addressing this partition-0 abnormal-termination signature.

Suggested Triage Steps

• Reproduce with log capture: run python3 sglang/multimodal_gen/test/run_suite.py --suite 2-gpu --partition-id 0 --total-partitions 3 inside ci_sglang on a 2×MI325 host, redirecting stdout/stderr to a file (so output survives a runner-loss), and monitor rocm-smi + dmesg/OOM-killer during the run.
• Bisect the case list: add --continue-on-error and per-case timeouts to identify which partition-0 case hangs; once identified, run only that case to confirm.
• Fix the observability gap first: have the CI step tee the test output into SGLANG_DIFFUSION_ARTIFACT_DIR and upload it even on hard failure, so future part-0 failures produce a diagnosable log/artifact.
• If a single heavy T2V case is implicated, consider re-balancing the LPT partition weights or splitting that case onto a standalone partition; treat as a triage step, not a directive.

Status (factual, no priority assignment)

• Persistence: Partition 0 has failed in every comparable completed scheduled run in the window (15/15), spanning ≥ 9 days (since at least 2026-07-13 00:28 UTC), true start likely earlier.
• Scope: Isolated to multimodal-gen-test-2-gpu-amd partition 0 on linux-mi325-2gpu-sglang; sibling parts 1 & 2 pass. Not observed to cascade to other job families.
• Observability gap: Part-0 logs and failure artifacts are never persisted (BlobNotFound, empty artifact set), which blocks test-file/function-level root-causing.
• In-flight fix: ❌ none found in open PRs.

stage-c-test-large-8-gpu-amd (linux-mi325-8gpu-sglang, 3) — failed step(s): Run test

Commit Info

• sglang: b13abfde
• aiter: 9127c94a — v0.1.17.dev110+g9127c94a1 (source: dev image preserved; [CI-AITER-CHECK] reports "Dev/patched version detected → skipping rebuild"). Not implicated — the failure is a pure-Python AttributeError, no kernel involvement.

Failed Tests

Test File	Test Function	Error	Log
test/registered/ops/test_aiter_allreduce_fusion_amd.py	TestAiterAllreduceFusionGate.test_dense_tp_fuses	AttributeError: <module 'sglang.srt.layers.communicator'> does not have the attribute 'get_server_args'	Log

Other files in this shard passed: test_deepseek_v3_mtp.py ✅, test_deepseek_v3_basic.py ✅, test_aiter_allgather_amd.py ✅ (Test Summary: 3/4 passed). The remaining 3 tests in TestAiterAllreduceFusionGate did not error — only test_dense_tp_fuses is shown failing before the suite aborted with FAILED (errors=1).

Failure Summary

test_dense_tp_fuses calls _evaluate_gate, which does mock.patch.object(comm, "get_server_args", ...) at test_aiter_allreduce_fusion_amd.py:421. The sglang.srt.layers.communicator module no longer exposes get_server_args, so unittest.mock raises AttributeError on context-manager entry. The failure is deterministic (it recurred on the built-in retry) and is a test/source drift, not a runtime numerical or kernel issue.

Regression Status

New regression — first failing in today's run (2026-07-22 12:32 UTC). The test file was introduced 2026-07-20 (PR #31784) and passed until the get_server_args accessor was removed from communicator.py.

Recent history of test/registered/ops/test_aiter_allreduce_fusion_amd.py (test_dense_tp_fuses) in job stage-c-test-large-8-gpu-amd (linux-mi325-8gpu-sglang, 3) (most recent comparable completed scheduled runs: branch=main, event=schedule, completed only):

Date (UTC)	Run	Job	Test File Status	Failed Function	Error
Jul 22 12:32	29920027108	shard 3	❌ Failed	test_dense_tp_fuses	AttributeError: get_server_args
Jul 22 00:26	29880220845	shard 3	✅ Passed	—	—
Jul 21 12:32	29830494225	shard 3	✅ Passed	—	—
Jul 21 00:27	29790347494	shard 3	✅ Passed	—	—
Jul 20 12:42	29743214297	shard 3	⚪ N/A (test not yet added)	—	—

The job's overall conclusion was failure in every run (this shard bundles multiple test files), but the shard job succeeded in the three prior in-window runs where this test file existed — so at the test-file level the last passing point is run 29880220845 (head 8ae0eb83, Jul 22 00:26 UTC). First failing point is this run (head b13abfde). Regression window: 8ae0eb83...b13abfde.

Failure Cluster

AttributeError: communicator module missing get_server_args (test/source drift after config-accessor refactor)

Facts (verified observations)

• Log (verbatim): AttributeError: <module 'sglang.srt.layers.communicator' from '/sglang-checkout/python/sglang/srt/layers/communicator.py'> does not have the attribute 'get_server_args', raised via unittest/mock.py:1420 get_original → contextlib enter_context → test_aiter_allreduce_fusion_amd.py:420 (_evaluate_gate) → :439 (test_dense_tp_fuses). Suite ends Ran 4 tests in 56.661s / FAILED (errors=1), then exit code 255.
• The test patch target is present at test_aiter_allreduce_fusion_amd.py:421: mock.patch.object(comm, "get_server_args", lambda: server_args). git blame attributes this line to 7fc545b6 (Ke Bao, 2026-07-20, PR #31784).
• grep of python/sglang/srt/layers/communicator.py at HEAD finds zero occurrences of get_server_args; the module imports only get_exec, get_forward, get_parallel, get_spec from runtime_context.
• Commit 745b2ca4 (PR #31816, "config: read parallel config leaves via get_parallel()", 2026-07-22 08:18 UTC) removed the get_server_args import from communicator.py (line 79 deletion) and replaced every get_server_args().X with get_parallel().X (verified in git show).
• Ancestry (verified): 745b2ca4 is an ancestor of HEAD b13abfde but not of the last-passing head 8ae0eb83 — i.e. the removal landed exactly inside the regression window.
• aiter unchanged and irrelevant: [CI-AITER-CHECK] shows the same dev image (v0.1.17.dev110+g9127c94a1) preserved; the error is a Python attribute lookup with no kernel in the stack.

Hypothesised Causes

• [HIGH] PR #31816 (745b2ca4) removed communicator.get_server_args, while the test added by PR #31784 (7fc545b6) still patches that name. The two PRs raced: the test still referenced the old accessor when the refactor deleted it, so neither PR's CI caught the pairing.
  • Supporting: three independent chains converge — (a) the removed symbol is exactly the one the test patches (code-level match), (b) removal commit lands inside the verified 8ae0eb83...b13abfde window (timing), (c) the error message names communicator and get_server_args verbatim.
  • Disconfirming: (none found) — the last passing shard predates the removal and the first failing shard contains it.
  • How to verify: on a branch, either add get_server_args back as a thin shim in communicator.py, or update the test to patch get_parallel / the new accessor, then re-run python3 test/registered/ops/test_aiter_allreduce_fusion_amd.py on an MI325 8-GPU runner.
• [LOW] Secondary latent breakage even after the patch line is fixed: should_fuse_mlp_allreduce_with_next_layer now reads get_parallel().moe_dense_tp_size (via enable_moe_dense_fully_dp), but the test's get_parallel mock (:414) returns SimpleNamespace(tp_size=...) only.
  • Supporting: the refactor moved several fields the test used to feed via server_args onto the get_parallel() namespace.
  • Disconfirming: not yet observed — execution aborts at the get_server_args patch before reaching this path.
  • How to verify: after fixing the primary patch, run the test locally and check whether the get_parallel mock needs moe_dense_tp_size and other leaves.

In-flight Fix Check

• In-flight fix: ❌ none found — search/issues?q=repo:sgl-project/sglang+is:pr+is:open+get_server_args returns 2 unrelated PRs (#30905, #31710); +aiter_allreduce_fusion returns 0. Needs a new PR to realign the test with the get_parallel() accessor refactor.

Suggested Triage Steps

• Confirm the fix direction with PR #31816's author (Cheng Wan): update test_aiter_allreduce_fusion_amd.py:421 to patch the new accessor (get_parallel, or whichever namespace now carries enable_attn_tp_input_scattered / moe_dense_tp_size) instead of the removed get_server_args.
• Re-run test/registered/ops/test_aiter_allreduce_fusion_amd.py on an MI325 8-GPU runner after the edit; watch for the [LOW] secondary get_parallel mock gap.
• Broader: grep the test tree for other stale get_server_args patch targets that PR #31816 may have orphaned — git grep -n "get_server_args" test/ on HEAD.

Status

• Persistence: New — first failing run 2026-07-22 12:32 UTC; last passing 2026-07-22 00:26 UTC (~12 h). Deterministic (failed on both retry attempts).
• Scope: test_dense_tp_fuses in TestAiterAllreduceFusionGate, shard stage-c-test-large-8-gpu-amd (linux-mi325-8gpu-sglang, 3) of pr-test-amd.yml. Any other shard/workflow that runs test_aiter_allreduce_fusion_amd.py on a checkout containing 745b2ca4 will hit the identical error. aiter/backend unaffected.
• In-flight fix: ❌ none found — needs new PR.

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-07-22 19:32 UTC)

[amd-bot]

release-docker-amd-nightly.yml — 1 failure(s)

Run started (UTC): 2026-07-22 12:39 UTC
Last scanned (UTC): 2026-07-22 16:30 UTC
Commits: sglang b13abfd

Job	Failed Steps	Started
publish (gfx950, all)	Build and Push to rocm/sgl-dev	2026-07-22T12:39

Per-Job Analysis

publish (gfx950, all) — failed step(s): Build and Push to rocm/sgl-dev

Commit Info

• sglang: b13abfdedfc6eb5556e0a04e9714ebd91b323c1d
• aiter: N/A (docker-build workflow — no [CI-AITER-CHECK] markers in log)

Failed Tests

Test File	Test Function	Error	Log
N/A (docker image build)	N/A	apt-get failed to fetch libpfm4_*.deb from archive.ubuntu.com → exit code: 100	Log

This is not a test failure. The publish (gfx950, all) job builds and pushes the ROCm docker image; the failure is in the image build (step 7, Build and Push to rocm/sgl-dev), not in any test.

Failure Summary

The docker build failed in the [stage-5 21/29] TileLang build layer (buildkit step #28). After ROCm's bundled llvm-config was not found, the layer fell back to installing LLVM 18 from apt.llvm.org/Ubuntu archives; apt-get install llvm-18 then failed to download a transitive dependency (libpfm4) from archive.ubuntu.com, aborting the build with exit code: 100. No image was produced, so all downstream push/artifact steps were skipped.

Regression Status

Infrastructure issue (transient network flake) — new failure for this job (previous run passed), but the cause is a package-mirror fetch failure, not a code change.

Recent history of publish (gfx950, all) image build in workflow release-docker-amd-nightly.yml (most recent completed scheduled runs, branch=main, event=schedule):

Date	Run	Job Status	Failure Signature
Jul 22	29920483445	❌ Failed	TileLang stage — apt-get fetch libpfm4 from archive.ubuntu.com failed
Jul 21	29830904222	✅ Passed	—
Jul 20	29743696137	✅ Passed	—
Jul 19	29686913858	✅ Passed	—
Jul 18	29644235932	✅ Passed	—
Jul 17	29580608704	✅ Passed	—
Jul 16	29499051508	✅ Passed	—
Jul 15	29415796071	✅ Passed	(run failed in push_local_registry, not this job)
Jul 14	29332975533	✅ Passed	(run failed in push_local_registry, not this job)
Jul 12	29192571824	✅ Passed	—
Jul 11	29152576569	❌ Failed	Different — MORI stage (git clone/checkout), exit code: 1

First observed failure for this signature (TileLang apt-fetch): 2026-07-22 12:39 UTC (this run). Last known passing: 2026-07-21. The Jul 11 failure is a distinct signature (MORI build layer) and does not share this cluster.

Failure Cluster

apt-get archive fetch failure during LLVM-18 install in TileLang docker stage (transient network flake)

Facts (verified observations, no interpretation)

• Failing buildkit step: #28 [stage-5 21/29] RUN ... [TileLang] Building TileLang for gfx950 (log line 20769).
• The layer logged [TileLang] ROCm llvm-config not found; installing LLVM 18... (log line 21368), confirming it entered the apt-based LLVM-18 fallback path.
• Verbatim error (log lines 21430–21431):
  • E: Failed to fetch https://archive.ubuntu.com/ubuntu/pool/main/libp/libpfm4/libpfm4_4.11.1%2bgit32-gd0b85fb-1ubuntu0.1_amd64.deb Could not wait for server fd - select (11: Resource temporarily unavailable) [IP: 185.125.190.83 443]
  • E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?
• buildkit reported #28 ERROR: process "..." did not complete successfully: exit code: 100 (log line 21433) — 100 is apt-get's package-fetch failure code.
• Overall step 7 (Build and Push to rocm/sgl-dev) failed; steps 8–11 (save tag / upload artifact / lmsys login / push) were skipped (from job step metadata).
• libpfm4 is a transitive dependency of llvm-18; the download stalled on a socket read from archive.ubuntu.com (185.125.190.83), i.e. a network/mirror-side transient error, not a missing/renamed package.
• Prior run (Jul 21) built the identical TileLang stage successfully → the Dockerfile / TileLang pin itself is buildable.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [HIGH] Transient network / Ubuntu-mirror unavailability while fetching libpfm4_*.deb during the LLVM-18 apt install.
  • Supporting: error is Could not wait for server fd - select (11: Resource temporarily unavailable) against archive.ubuntu.com — a socket-level transient failure, and exit code: 100 is the apt fetch code. Same stage built fine in the Jul 21 run with no Dockerfile change to this layer.
  • Disconfirming: (none found) — the message is unambiguously a fetch/network error, not a package-resolution or compile error.
  • How to verify: re-run the failed job (gh run rerun 29920483445 --failed); a green rerun with no code change confirms the flake.
• [LOW] The apt-based LLVM-18 fallback path itself is fragile (no retry / --fix-missing), so any mirror hiccup fails the whole 30-min TileLang layer.
  • Supporting: the RUN chain uses set -euo pipefail and a plain apt-get install -y llvm-18 with no retry loop; the ROCm llvm-config was not found so this fallback ran.
  • Disconfirming: this is a robustness gap, not the trigger; the underlying event was still a transient fetch failure.
  • How to verify: check whether /opt/rocm/llvm/bin/llvm-config normally exists in the base image (if so, the fallback should rarely run and the flake surface is small).

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for TileLang+apt (1 hit, #22037 "Multi-stage build for ROCm image to reduce size" — unrelated to apt-fetch retry) and apt-get+retry (0 hits). No PR targets apt-fetch resilience for the TileLang/LLVM-18 layer.

Suggested Triage Steps

• Re-run the failed job first: gh run rerun 29920483445 --failed. A clean pass with the same commit confirms a transient mirror flake and closes this out.
• If this apt-fetch failure recurs across runs, consider hardening the TileLang layer's apt fallback (retry loop / apt-get -o Acquire::Retries=3 install / alternate mirror) — track as a CI-robustness item, not a code regression.
• No sglang source bisect is warranted: the last scheduled run (29830904222) built the identical layer successfully and the error is network-side, not code-side.

Status (factual, no priority assignment)

• Persistence: 1 run (first seen 2026-07-22 12:39 UTC); the immediately preceding 6+ scheduled runs of this job passed.
• Scope: publish (gfx950, all) image build only, in release-docker-amd-nightly.yml. Downstream push/tag/artifact steps skipped as a consequence.
• Blocked work: no nightly ROCm gfx950 dev image published for 2026-07-22.
• In-flight fix: ❌ none found — likely a transient infra flake; re-run before opening any PR.

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-07-22 16:30 UTC)