Skip to content

[ROCm][Bugfix] Drop sparse MLA metadata host sync; use persistent AITER for gfx950 long BF16 prefills - #66400

Merged
AndreasKaratzas merged 4 commits into
vllm-project:mainfrom
AndreasKaratzas:akaratza_rocm_sparse_mla_metadata_sync
Nov 9, 2026
Merged

AndreasKaratzas merged 4 commits into
vllm-project:mainfrom
AndreasKaratzas:akaratza_rocm_sparse_mla_metadata_sync

Conversation

@AndreasKaratzas

@AndreasKaratzas AndreasKaratzas commented Nov 8, 2026 •

Copy link
Copy Markdown
Member

Overview

This PR removes the host synchronize() that ROCMAiterMLASparseMetadataBuilder.build() runs after each get_mla_metadata_v1 rebuild. That sync fails VLLM_GPU_SYNC_CHECK=error inside execute_model. The PR also stops gfx950 from sending long pure BF16 prefills to AITER's nonpersistent ASM decode kernels, which give wrong results that change from run to run at 32, 64 and 128 heads. That kernel is what made large_q_pure_prefill-auto-ROCMAiterMLA flaky, on main and once the sync was gone.

The persistent-metadata block keeps its pre-existing gpu_sync_allowed() seq_lens[:num_reqs].cpu() read. Because of the routing change, gfx950 long pure BF16 prefill steps now enter that block and so now hit that allowed read; before this PR they skipped it. Removing that read is follow-up work, not part of this PR.

This is part of the work to re-enable VLLM_GPU_SYNC_CHECK=error on AMD CI, which #61939 dropped from docker/Dockerfile.rocm. It replaces the gpu_sync_allowed() wrap that #64176 used for this site, following njhill's review there (fix the syncs instead of allowing them). Related PRs: #66397, #66398, #66399; #64176 turns the check back on and depends on all of them.

Claims

  • Post-metadata synchronize() removed, with no allow-wrap in its place. The pre-existing allowed seq_lens read in the same block remains (see Overview). Before this PR, the removed sync (rocm_aiter_mla_sparse.py:726 on main) failed these AMD jobs with the check on: in build 97758, mi300 v1-attention shard 1/3, mi355 dpx v1-attention shard 2/4, mi355 lm-eval deepseek-v3-2-tp and -dp; in amd-ci 15532, mi300 v1-attention shard 1 and shard 2, mi355 dpx v1-attention shard 1 and shard 2, mi355 lm-eval deepseek-v3-2-tp and -dp.
  • Flaky test fixed. test_sparse_backend_decode_correctness[deepseek-1.0-1.0-64-1-auto-large_q_pure_prefill-ROCMAiterMLA] now gives the same result on every run on gfx950. Pristine main: 3/60 in-process failures. This PR: 0/60.
  • Correct at every head count on gfx950. For long pure BF16 prefills, AITER's persistent kernel was deterministic and within 0.006 of an fp32 reference at 16-128 heads. The nonpersistent kernels main uses are nondeterministic with errors up to 0.036 (32 heads), 0.092 (64) and 0.173 (128), and at 48/80/96/112 heads AITER has no nonpersistent kernel at all (table in Details).
  • Faster only at 128 heads. At 128 heads the persistent kernel is about 1.2-1.5x faster per layer (kernel table below). At fewer heads it is slower: at TP4 DeepSeek-V3.2 (32 heads per rank) TTFT on an 8192-token prompt rises by 17.0 ms (+2.6%) (see Model eval). Each such step also pays one get_mla_metadata_v1 launch (1.3 ms at 8192 tokens) and the allowed seq_lens read.
  • No metadata that AITER would ignore. The builder skips persistent metadata and its seq_lens read when AITER would not use the persistent kernel for the batch (gfx950, BF16, 16 heads, at least 32 tokens). With AITER 0.1.28.post1 on MI350X (gfx950), 16 heads and BF16, output with and without metadata was bit-identical at 32, 64, 512 and 4096 tokens over 3 reruns each. At 31 tokens, below the threshold, the two outputs differ (max 2.4e-4) because a different kernel runs; that is the positive control. At TP8 DeepSeek-V3.2 the kernel trace now matches main exactly. This covers batches of 32 or more tokens; a 16-head long-context prefill with fewer than 32 new tokens (prefix hit or short chunked tail) still builds metadata and runs the persistent kernel, which was deterministic in the study.
  • gfx942 unchanged. Long BF16 prefills there still skip persistent metadata and take the Triton fallback.

Validation

All local runs: MI350X (gfx950), ROCm torch 2.13, amd-aiter 0.1.28.post1, VLLM_ROCM_USE_AITER=1. Commands 1 and 2 were re-run on 2deae9c3dd and gave the same results: 155 passed / 108 skipped, and 30/30. The other runs used 36442e9f. Both earlier commits differ from 14f9dc4 only in docstrings. The command 1, unit-test and 25-run fresh-process counts were rerun at the head with the metadata-skip change and the head-count test parametrization.

Check main this PR
Sparse MLA CI set with VLLM_GPU_SYNC_CHECK=error (command 1) 2 failed (test_rocm_fused_v32_dense_prefill_matches_mqa[auto], [fp8]: GPU<->CPU sync detected at the removed line), 141 passed, 108 skipped 181 passed, 108 skipped, 0 failed (the extra passes come from the gfx942 and head-count parametrizations and the 2 metadata-skip tests)
kernels/attention/test_rocm_aiter_mla_sparse_metadata_sync.py (this PR's test file; main column runs it against main's source) 2 failed (test_sparse_persistent_metadata_recomputes_without_host_sync, test_sparse_metadata_workload_length_dtype_matrix[prefill-long-auto-False]), 40 passed 68 passed
large_q_pure_prefill-auto-ROCMAiterMLA, repeated in one process 3/60 failed (mismatched elements: 11464, 11640, 217); separately 5/30 0/60; also 0/30 with VLLM_GPU_SYNC_CHECK=error (command 2)
Same test, one fresh process per run 3/12 failed 0/40, and 0/20 with VLLM_GPU_SYNC_CHECK=error; 0/25 with VLLM_GPU_SYNC_CHECK=error after the metadata-skip change
Integration stack (sync-fix/rocm-enable-gpu-sync-check: this fix at 36442e9f plus #66397/#66398/#66399 and the Dockerfile change), VLLM_GPU_SYNC_CHECK=error, command 1 - 155 passed, 108 skipped, no syncs

The unit matrix is the deterministic guard for the routing change. The e2e large_q_pure_prefill test only catches the bad kernel about 5-25% of the time on main (3/60, 5/30, 3/12), so it is a weak regression signal on its own.

Extra stress runs and commands

Independent reviewer stress of the target test: 0/50 fresh-process failures and 100/100 in-process passes (one reviewer), and another 100/100 in-process (a second reviewer). A full run of the ROCMAiterMLA cases of test_sparse_backend_decode_correctness, repeated 10 times, gave 108 passed / 108 skipped every time.

# 1. CI set (from tests/)
VLLM_ROCM_USE_AITER=1 VLLM_GPU_SYNC_CHECK=error pytest -q \
  v1/attention/test_sparse_mla_backends.py \
  kernels/attention/test_rocm_aiter_mla_sparse_metadata_sync.py \
  -k "rocm or sparse_persistent or sink"

# 2. Target test, repeated in one process (REPEAT_N from a small local repeat plugin)
VLLM_ROCM_USE_AITER=1 VLLM_GPU_SYNC_CHECK=error REPEAT_N=30 pytest -q -p repeat_plugin \
  "v1/attention/test_sparse_mla_backends.py::test_sparse_backend_decode_correctness[deepseek-1.0-1.0-64-1-auto-large_q_pure_prefill-ROCMAiterMLA]"

Kernel timings (MI350X, BF16, CUDA events, mean over 20 iterations, us). The AITER-only repro and timing script uses random inputs, a fresh zeroed output and a device sync around every call. "Differing reruns" counts reruns whose output differed from the first run.

shape (reqs x tokens, topk, heads) nonpersistent (old gfx950 path) persistent (this PR) Triton differing reruns: nonpersistent / persistent
2x256, 128, 128 83.5 71.1 385 20-49/49 (two runs) / 0/49
1x4096, 2048, 128 2958 2005 36272 5/19 / 0/19
4x2048, 2048, 128 4236 3017 49728 7/19 / 0/19
1x4096, 2048, 64 1214 1163 18264 0/19 / 0/19

The extra get_mla_metadata_v1 launch runs once per step and all layers share it: 113 us at 512 tokens, 1.3 ms at 8192, 2.6 ms at 16384.

Model eval and end-to-end latency. DeepSeek-V3.2 on 8x MI350X (gfx950), amd-aiter 0.1.28.post1, main (fd0c471) vs this PR with the same arguments. The changed path is reached only by pure-prefill steps whose longest prefill exceeds topk=2048, with BF16 KV. So the runs use num_concurrent=1, 25-shot GSM8K (all 1319 prompts are 3151-5087 tokens) and --max-num-batched-tokens 32768, so each prompt prefills in one step. MTP is off and CUDA graphs are on (FULL_DECODE_ONLY). Dispatch was checked with rocprofv3 on an 8192-token request.

  • TP4 (32 heads per rank): the PR runs the persistent mla_a16w16_qh64_qseqlen1_gqaratio64_v3_ps plus kn_mla_reduce_v1_ps instead of main's nonpersistent mla_a16w16_qh16_m16x4_n32x1. That is 155.0 vs 144.0 ms over 61 layers, plus 2.7 ms of reduce and a 1.44 ms metadata launch.
  • TP8 (16 heads per rank): AITER falls back to nonpersistent split-KV (mla_dec_stage1_bf16_a16w16_subQ16_mqa16) on both sides. With the metadata skip, kernel names and counts match main on all 8 ranks. Without it, the only extra work was one ignored metadata launch (1.44 ms) and one seq_lens copy.

TP4 (32 heads per rank):

TP4 GSM8K flexible GSM8K strict TTFT med (ms) TPOT med (ms) E2E med (ms)
main 0.9545 0.9553 667.2 13.45 2372.8
this PR 0.9560 0.9568 684.2 13.46 2393.8

TP8 (16 heads per rank):

TP8 GSM8K flexible GSM8K strict TTFT (ms) TPOT (ms) E2E (ms)
main 0.9545 0.9545 493.4 ± 1.8 12.87 2127.7 ± 2.0
this PR without the metadata skip - - 493.8 ± 2.1 12.87 2127.6 ± 1.5
this PR 0.9522 0.9530 493.0 ± 2.8 12.86 2126.4 ± 0.9
  • Latency is from vllm bench serve with random 8192 in / 128 out, concurrency 1, 50 prompts plus 5 warmups, --ignore-eos.
    • TP4: 3 repeats per side on each of two GPU sets, alternating, then swapped (300 requests per side). The result is +17.0 ms TTFT (+2.6%) and +21.1 ms E2E (+0.9%), and it holds with the GPU sets swapped.
    • TP8: 4 fresh-server rounds in rotated order, 3 repeats each. Values are the mean ± sd of the per-round medians. The 1.44 ms metadata launch is below restart-to-restart noise.
  • Accuracy: GSM8K standard error is about ±0.006. Paired McNemar tests give p = 0.79 (TP4) and p = 0.61 / 0.79 (TP8, flexible/strict). At TP8 main and this PR run identical kernels and still share only 579/1319 identical generations, so differences of this size come from run-to-run nondeterminism.
  • The CI-parity eval (configs/models-mi355-deepseek-v32-tp.txt, TP4 + MTP + eager, 5-shot) has not been rerun; its 0.6k-1.6k token prompts and high concurrency probably never reach the changed path.
VLLM_ROCM_USE_AITER=1 VLLM_ROCM_USE_AITER_MLA=1 VLLM_ROCM_USE_AITER_MOE=1 \
vllm serve deepseek-ai/DeepSeek-V3.2 --tensor-parallel-size {4,8} --enable-expert-parallel \
  --attention-backend ROCM_AITER_MLA_SPARSE --moe-backend aiter \
  --kv-cache-dtype bfloat16 --max-model-len 16384 --max-num-batched-tokens 32768 \
  --no-enable-prefix-caching
lm_eval --model local-completions --tasks gsm8k --num_fewshot 25 \
  --model_args model=deepseek-ai/DeepSeek-V3.2,base_url=http://localhost:8000/v1/completions,num_concurrent=1,tokenized_requests=False
vllm bench serve --model deepseek-ai/DeepSeek-V3.2 --dataset-name random \
  --random-input-len 8192 --random-output-len 128 --max-concurrency 1 \
  --num-prompts 50 --num-warmups 5 --ignore-eos

Not validated locally:

  • gfx942/MI300. gfx942 routing is unchanged, but its FP8-KV persistent path also loses the sync. CI's mi300 v1-attention shards cover that path, but not MTP + CUDA graphs at 200k context ([ROCm] Synchronize sparse MLA metadata before graph replay #47404's setup, see below).
  • MTP with DeepSeek-V3.2 at model level. TP4/TP8 runs above use CUDA graphs (FULL_DECODE_ONLY) without MTP. Kernel-level graph replay after an unsynced metadata rebuild was checked (see "Why the sync can go").

Details

Why the sync can go. With fast_mode=True, get_mla_metadata_v1 dispatches to AITER's get_mla_metadata_v1_2_device. That function launches one device kernel on aiter::getCurrentHIPStream(), and AITER's compile_ops sets that stream to torch's current stream before each call. Nothing is staged through host memory, there is no hipMemcpy, and the function owns no buffers. The decode kernel, whether eager or captured in a graph and replayed, runs later on the same stream and reads the metadata on the device, so stream order already puts the write before the read. The block also already starts with a blocking seq_lens.cpu(), so the removed sync only waited for the single metadata kernel launched after it. The dense rocm_aiter_mla builder already rebuilds the same metadata every step without a sync. Two investigation experiments back this up:

  • Outputs stayed bit-identical with no sync even with a 5M-cycle torch.cuda._sleep queued before the metadata kernel, which lets the host run far ahead of the GPU.
  • Direct graph-replay race test (AITER persistent FP8 MLA decode, 128 heads, 256 tokens, same metadata arguments as vLLM): capture a CUDA graph, then per iteration rewrite kv_indptr to a different config, queue a 20M-cycle sleep, rebuild metadata with no host sync and replay. Same stream: 0/50 mismatches against per-config eager references. Positive control replaying on an unordered side stream: 49/50 mismatches.

Concern about #47404. The sync came from #47404, which reported that it removed a fault on 8x MI300X (gfx942), TP8, ROCM_AITER_MLA_SPARSE + MTP + CUDA graphs, FP8 KV, max_model_len=200000. That PR gave no root cause beyond "graph replay could observe stale metadata". The kernel-level graph-replay test above shows that same-stream ordering covers that case, but it ran on gfx950 with one GPU and no model. #47404's exact configuration has not been rerun with this PR; all local hardware is gfx950. Plan before leaving draft: rerun that configuration (100+ requests) on MI300X with this PR, through AMD CI or a reviewer with MI300X access, or get confirmation from the #47404 author; if none is possible, the regression risk stays open and is called out to reviewers.

Why the flaky test failed. large_q_pure_prefill (2x256 tokens, topk 128, BF16 KV) never reached the sync. On main, _is_long_bf16_prefill() skipped persistent metadata for long pure BF16 prefills on every architecture, but only gfx942 sent those batches to Triton. On gfx950 they fell through to AITER's nonpersistent mla_dec_stage1_bf16_a16w16_subQ128_mqa128. With bit-identical inputs and device syncs around every call, that kernel returns different rows from run to run. Against an fp32 reference its max error is 0.12-0.21, compared with 0.0053 for the persistent kernel. So the kernel computes wrong values; this is not an ordering race. Syncs placed before or after stage 1, AMD_SERIALIZE_KERNEL=3 and HIP_LAUNCH_BLOCKING=1 all still reproduce the failure. It reproduces with AITER alone and the newest AITER nightly ships the same binary; reported as ROCm/aiter#7631. The 32- and 64-head nonpersistent kernels are also nondeterministic, mostly on multi-request shapes (the single-request 1x4096 64-head row above showed 0/19). The sink/LSE path still calls the nonpersistent kernel. This PR leaves it alone, and #66454 handles it separately (see Tradeoffs).

Determinism by head count (AITER only, sparse layout: one entry per query token, ragged top-k indices, page_size 1; same inputs every run, NaN-poisoned output, with and without an L2 flush; 50-400 runs per config at 2x256 to 1x8192 tokens; error is the max against an fp32 reference):

padded heads nonpersistent kernel nondeterministic configs max err persistent nondeterministic configs max err
16 subQ16_mqa16 0/18 0.0059 0/12 (AITER runs the nonpersistent kernel) 0.0051
32 qh16_m16x4_n32x1 4/22 0.036 0/22 0.0052
64 qh64_qseqlen1_gqaratio64_v3 5/18 0.092 0/18 0.0057
128 subQ128_mqa128 12/18 0.173 0/18 0.0060
48/80/96/112 none (cannot get heuristic kernel) - - 0/4 each ≤0.0059

The fix. _is_long_bf16_prefill is renamed to _use_triton_long_bf16_prefill and now checks on_gfx942() first. The builder and _forward_mla share this helper, so they cannot disagree about which path a batch takes. The import of on_gfx942 stays lazy so that the tests' monkeypatch still applies. The gfx942-only Triton routing for these prefills was in place before this PR. The all-architecture skip of persistent metadata came from #49263 (cc tjtanaa), which gave no reason for it. Routing gfx950 to Triton instead would be deterministic, but 5-18x slower (table above).

Tradeoffs and limitations.

  • On gfx950, long pure BF16 prefill steps now enter the persistent block. So they now also pay the existing gpu_sync_allowed() seq_lens.cpu() read and one metadata launch per step. At TP4 DeepSeek-V3.2 (32 heads per rank) the persistent kernel is also slower than the nonpersistent one, so 8192-token TTFT rises by 17 ms (+2.6%). That is the cost of not running the 32-head nonpersistent kernel, which gives wrong values (see the determinism table).
  • The one exception is 16-head BF16 batches of at least AITER_MLA_DECODE_PERSISTENT_MAX_BATCH tokens (32 by default). There, AITER's mla_decode_fwd ignores work_meta_data and runs the nonpersistent split-KV kernel. The builder asks AITER's own _use_persistent_mla_decode and skips both the metadata and the read in that case, so these steps do what main did. The check also covers decode and mixed batches, where main built metadata that AITER then dropped. If the installed AITER lacks that function, the builder keeps the old behavior.
  • Layers with sinks or use_aiter_sparse_mla set _use_persistent_metadata=False. This PR does not change sink layers. On gfx950, BF16 sink layers still run AITER's nonpersistent mla_decode_fwd_lse, which [gfx950] mla_dec_stage1_bf16_a16w16_subQ128_mqa128.co gives nondeterministic, wrong output when a workgroup handles more than 48 KV tokens ROCm/aiter#7631 also covers (return_lse=True). [ROCm][Bugfix] Route gfx950 BF16 sparse MLA sink layers to Triton #66454 routes those layers to Triton until [Encoder decoder] Add cuda graph support during decoding for encoder-decoder models #7631 is fixed. The two PRs are independent, and either can land first.
  • metadata_key does not include max_split_per_batch. That was already true before this PR; it is only flagged for follow-up.

Tests.

  • test_sparse_persistent_metadata_syncs_only_after_recompute is renamed to test_sparse_persistent_metadata_recomputes_without_host_sync and now asserts that no sync happens.
  • test_sparse_metadata_workload_length_dtype_matrix is parametrized over (on_gfx942, num_heads) in (True, 128), (False, 32), (False, 64), (False, 128): only gfx942 may skip persistent metadata for long BF16 prefills. Against main's source [prefill-long-auto-False] failed (before the head parametrization), and a 128-only rule fails the 32- and 64-head cases, so this is the deterministic guard for the routing change.
  • test_build_skips_metadata_aiter_ignores[-1|0] uses AITER's real predicate with gfx950 and its default threshold patched in. At the threshold, it checks that the builder launches no get_mla_metadata_v1, enters no gpu_sync_allowed() read and returns work_meta_data=None. One token below the threshold, it checks that metadata is still built. Without the source change, [0] fails.
  • The previously flaky test_sparse_backend_decode_correctness[...large_q_pure_prefill-ROCMAiterMLA] runs in the AMD v1-attention CI shards, but it is a low-rate catcher (see Validation).

Duplicate check. Searched open PRs for VLLM_GPU_SYNC_CHECK, gpu_sync_allowed, "sync rocm" and sparse MLA metadata keywords. #64176 is the submitter's own predecessor, and this PR replaces its approach for this site. #66003 removes a different allowed sync, in the ROCM_AITER_FA builder, so the two do not overlap. #62946 targets the INT4 RHT-signs site, which is handled in #66399. No open PR touches this site.

AI assistance. AI assistance (Claude Code) was used to investigate, implement and validate this change. The human submitter is responsible for the change.


Pull Request Checklist
  • I used vLLM's /pr-checklist skill. (Mandatory for agents, optional for humans).

  • AI assistance was used during the creation of this PR.

  • Design Fit: Minimizes impact on core components, reuses existing functionality, and justifies added complexity.

  • Testing and Validation: Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI. (The [ROCm] Synchronize sparse MLA metadata before graph replay #47404 gfx942 rerun and the CI-parity DeepSeek-V3.2 eval are pending.)

  • Code Quality and Style: Keeps code and comments clear and concise, and updates relevant documentation and examples.

  • Pull Request Contents: Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.

BEFORE SUBMITTING, PLEASE READ https://docs.vllm.ai/en/latest/contributing (anything written below this line will be removed by GitHub Actions)

…or long BF16 prefills

ROCMAiterMLASparseMetadataBuilder.build() synchronized the current stream
after every get_mla_metadata_v1 rebuild (vllm-project#47404), which trips
VLLM_GPU_SYNC_CHECK=error inside execute_model. The sync is not needed:
get_mla_metadata_v1 launches a device kernel on aiter's current HIP
stream, which compile_ops sets to torch's current stream before each
call, with no host staging; the decode kernel (eager or graph replay)
runs later on the same stream. The dense rocm_aiter_mla builder already
rebuilds the same metadata every step without a sync.

Dropping the sync alone looked like a regression because
test_sparse_backend_decode_correctness[...-auto-large_q_pure_prefill-
ROCMAiterMLA] flaked afterwards, but that case never reached the sync.
Long pure BF16 prefills (prefill_max_seq_len > topk) skipped persistent
metadata on every arch, while only gfx942 routes them to Triton. On
gfx950 they fell through to AITER's nonpersistent ASM decode
(mla_dec_stage1_bf16_a16w16_subQ128_mqa128), which returns run-to-run
varying rows on bit-identical inputs even with device syncs around each
call (AITER-only repro: 20/49 to 49/49 reruns differ for 2x256 tokens,
topk 128, 128 heads). Pristine main fails the test intermittently too.

Skip persistent metadata only where the Triton fallback is taken
(gfx942). On gfx950 the persistent kernel is deterministic and faster
for these shapes (128 heads, bf16): 2x256/topk128 71 us vs 84 us;
1x4096/topk2048 2.0 ms vs 3.0 ms; 4x2048/topk2048 3.0 ms vs 4.2 ms;
the metadata kernel runs once per step (1.3 ms at 8192 tokens).

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andreas Karatzas <akaratza@amd.com>

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

@AndreasKaratzas

Copy link
Copy Markdown
Member Author

/ci run

@github-actions

github-actions Bot commented Nov 8, 2026

Copy link
Copy Markdown

✅ Triggered Buildkite CI #100933 for commit 14f9dc450e5a.

ℹ️ This PR is 60 commits behind upstream main. Merged upstream main into this branch as 14f9dc450e5a before starting CI. Pull the branch before pushing more changes.

@mawong-amd mawong-amd left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The standalone AITER repro and repeated gfx950 results make the non-sink BF16 workaround well motivated. The historical FULL-replay source also appears to use current-stream ordering, but that does not explain the original empirical fault. Rerunning #47404’s MI300X TP8/MTP/FP8/200k configuration at this head would close that validation gap.

For model validation, TP4 DeepSeek—or a dispatch-verified GLM/equivalent configuration—with BF16 KV, concurrency 1, and prompts above top-k would exercise the intended persistent path. Main-vs-PR accuracy and TTFT/E2E latency would establish its behavior and net cost. If the redundant metadata work remains on the 16-head fallback, that configuration also needs a latency comparison; a TP4 persistent-path run would not measure the overhead introduced there.

…gnore

AITER falls back to its non-persistent kernel for gfx950 BF16 16-head
batches of 32 or more tokens and drops the work metadata, so building it
only cost a host read and a metadata launch. Ask AITER's own gate first.

Co-authored-by: Claude <noreply@anthropic.com>
Signed-off-by: Andreas Karatzas <akaratza@amd.com>
@AndreasKaratzas

AndreasKaratzas commented Nov 9, 2026 •

Copy link
Copy Markdown
Member Author

Thanks! Ran your setup on DeepSeek-V3.2 (8x MI350X, AITER 0.1.28.post1, BF16 KV): full gsm8k 25-shot at concurrency 1 with every prompt above top-k, plus 8k-in/128-out latency.

gsm8k strict TTFT (ms) E2E (ms)
TP4 main 0.9553 667 2373
TP4 PR 0.9568 684 2394
TP8 main 0.9545 493.4 2128
TP8 PR 0.9530 493.0 2126

At TP4 the persistent path costs about 17 ms TTFT (+2.6%). I tried limiting it to 128 heads, but on gfx950 the non-persistent BF16 kernel is also nondeterministic at 32 and 64 heads (max error 0.036 and 0.092, vs ~0.005 for persistent), and there's no non-persistent kernel at all for 48-112 heads. So I kept persistent everywhere except where AITER drops the metadata itself, and narrowed the speedup claim in the body to 128 heads.

At TP8 the skip from your other comment makes the trace match main kernel for kernel; the metadata it avoids was 1.44 ms of GPU time, well below e2e noise.

@AndreasKaratzas

Copy link
Copy Markdown
Member Author

/ci run

@AndreasKaratzas

Copy link
Copy Markdown
Member Author

/ci run

@github-actions

github-actions Bot commented Nov 9, 2026

Copy link
Copy Markdown

✅ Triggered Buildkite CI #101208 for commit 377d5cd8f66a.

@mawong-amd mawong-amd left a comment •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the thorough validation. I reran the original GLM-5.1-FP8 TP8/MTP/FP8/200k setup on 8×MI300X at 25a2b312, whose PR files are unchanged by the latest main merge: both that revision and a merge-base control with only #47404’s sync removed completed 110/110 requests. The control passing adds support for the same-stream ordering argument; the original fault may have been timing-sensitive behavior fixed elsewhere since #47404, or simply not triggered by this workload. Its root cause remains unknown.

The BF16 routing change is a separate correctness workaround. The small TP4 TTFT regression is the cost of avoiding the nonpersistent kernels’ numerical corruption, not of removing the sync, while the 16-head fallback retains main’s no-metadata behavior. Once ROCm/aiter#7631 is fixed, we should revisit the persistent/nonpersistent defaults so the faster nonpersistent path can be restored where it is numerically safe. #66454 covers sink/LSE separately, and I have no remaining blockers.

@mawong-amd mawong-amd added the ready ONLY add when PR is ready to merge/full CI is needed label Nov 9, 2026
@mawong-amd

Copy link
Copy Markdown
Member

Also as a future process note: GSM8K is good as a quick sanity check but is long-saturated as a benchmark and not sensitive to subtle/rare numerical corruption as in this issue. More robust evals should be used for these cases.

@AndreasKaratzas
AndreasKaratzas merged commit f9fe2f1 into vllm-project:main Nov 9, 2026
156 checks passed
@github-project-automation github-project-automation Bot moved this from Todo to Done in AMD Nov 9, 2026
@AndreasKaratzas
AndreasKaratzas deleted the akaratza_rocm_sparse_mla_metadata_sync branch November 9, 2026 20:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working ready ONLY add when PR is ready to merge/full CI is needed rocm Related to AMD ROCm

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

3 participants