Repository navigation
[ROCm][Bugfix] Drop sparse MLA metadata host sync; use persistent AITER for gfx950 long BF16 prefills - #66400
Conversation
…or long BF16 prefills ROCMAiterMLASparseMetadataBuilder.build() synchronized the current stream after every get_mla_metadata_v1 rebuild (vllm-project#47404), which trips VLLM_GPU_SYNC_CHECK=error inside execute_model. The sync is not needed: get_mla_metadata_v1 launches a device kernel on aiter's current HIP stream, which compile_ops sets to torch's current stream before each call, with no host staging; the decode kernel (eager or graph replay) runs later on the same stream. The dense rocm_aiter_mla builder already rebuilds the same metadata every step without a sync. Dropping the sync alone looked like a regression because test_sparse_backend_decode_correctness[...-auto-large_q_pure_prefill- ROCMAiterMLA] flaked afterwards, but that case never reached the sync. Long pure BF16 prefills (prefill_max_seq_len > topk) skipped persistent metadata on every arch, while only gfx942 routes them to Triton. On gfx950 they fell through to AITER's nonpersistent ASM decode (mla_dec_stage1_bf16_a16w16_subQ128_mqa128), which returns run-to-run varying rows on bit-identical inputs even with device syncs around each call (AITER-only repro: 20/49 to 49/49 reruns differ for 2x256 tokens, topk 128, 128 heads). Pristine main fails the test intermittently too. Skip persistent metadata only where the Triton fallback is taken (gfx942). On gfx950 the persistent kernel is deterministic and faster for these shapes (128 heads, bf16): 2x256/topk128 71 us vs 84 us; 1x4096/topk2048 2.0 ms vs 3.0 ms; 4x2048/topk2048 3.0 ms vs 4.2 ms; the metadata kernel runs once per step (1.3 ms at 8192 tokens). Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Andreas Karatzas <akaratza@amd.com>
|
/ci run |
|
✅ Triggered Buildkite CI #100933 for commit ℹ️ This PR is 60 commits behind upstream |
mawong-amd
left a comment
There was a problem hiding this comment.
The standalone AITER repro and repeated gfx950 results make the non-sink BF16 workaround well motivated. The historical FULL-replay source also appears to use current-stream ordering, but that does not explain the original empirical fault. Rerunning #47404’s MI300X TP8/MTP/FP8/200k configuration at this head would close that validation gap.
For model validation, TP4 DeepSeek—or a dispatch-verified GLM/equivalent configuration—with BF16 KV, concurrency 1, and prompts above top-k would exercise the intended persistent path. Main-vs-PR accuracy and TTFT/E2E latency would establish its behavior and net cost. If the redundant metadata work remains on the 16-head fallback, that configuration also needs a latency comparison; a TP4 persistent-path run would not measure the overhead introduced there.
…gnore AITER falls back to its non-persistent kernel for gfx950 BF16 16-head batches of 32 or more tokens and drops the work metadata, so building it only cost a host read and a metadata launch. Ask AITER's own gate first. Co-authored-by: Claude <noreply@anthropic.com> Signed-off-by: Andreas Karatzas <akaratza@amd.com>
|
Thanks! Ran your setup on DeepSeek-V3.2 (8x MI350X, AITER 0.1.28.post1, BF16 KV): full gsm8k 25-shot at concurrency 1 with every prompt above top-k, plus 8k-in/128-out latency.
At TP4 the persistent path costs about 17 ms TTFT (+2.6%). I tried limiting it to 128 heads, but on gfx950 the non-persistent BF16 kernel is also nondeterministic at 32 and 64 heads (max error 0.036 and 0.092, vs ~0.005 for persistent), and there's no non-persistent kernel at all for 48-112 heads. So I kept persistent everywhere except where AITER drops the metadata itself, and narrowed the speedup claim in the body to 128 heads. At TP8 the skip from your other comment makes the trace match main kernel for kernel; the metadata it avoids was 1.44 ms of GPU time, well below e2e noise. |
|
/ci run |
|
/ci run |
|
✅ Triggered Buildkite CI #101208 for commit |
There was a problem hiding this comment.
Thanks for the thorough validation. I reran the original GLM-5.1-FP8 TP8/MTP/FP8/200k setup on 8×MI300X at 25a2b312, whose PR files are unchanged by the latest main merge: both that revision and a merge-base control with only #47404’s sync removed completed 110/110 requests. The control passing adds support for the same-stream ordering argument; the original fault may have been timing-sensitive behavior fixed elsewhere since #47404, or simply not triggered by this workload. Its root cause remains unknown.
The BF16 routing change is a separate correctness workaround. The small TP4 TTFT regression is the cost of avoiding the nonpersistent kernels’ numerical corruption, not of removing the sync, while the 16-head fallback retains main’s no-metadata behavior. Once ROCm/aiter#7631 is fixed, we should revisit the persistent/nonpersistent defaults so the faster nonpersistent path can be restored where it is numerically safe. #66454 covers sink/LSE separately, and I have no remaining blockers.
|
Also as a future process note: GSM8K is good as a quick sanity check but is long-saturated as a benchmark and not sensitive to subtle/rare numerical corruption as in this issue. More robust evals should be used for these cases. |
Overview
This PR removes the host
synchronize()thatROCMAiterMLASparseMetadataBuilder.build()runs after eachget_mla_metadata_v1rebuild. That sync failsVLLM_GPU_SYNC_CHECK=errorinsideexecute_model. The PR also stops gfx950 from sending long pure BF16 prefills to AITER's nonpersistent ASM decode kernels, which give wrong results that change from run to run at 32, 64 and 128 heads. That kernel is what madelarge_q_pure_prefill-auto-ROCMAiterMLAflaky, on main and once the sync was gone.The persistent-metadata block keeps its pre-existing
gpu_sync_allowed()seq_lens[:num_reqs].cpu()read. Because of the routing change, gfx950 long pure BF16 prefill steps now enter that block and so now hit that allowed read; before this PR they skipped it. Removing that read is follow-up work, not part of this PR.This is part of the work to re-enable
VLLM_GPU_SYNC_CHECK=erroron AMD CI, which #61939 dropped fromdocker/Dockerfile.rocm. It replaces thegpu_sync_allowed()wrap that #64176 used for this site, followingnjhill's review there (fix the syncs instead of allowing them). Related PRs: #66397, #66398, #66399; #64176 turns the check back on and depends on all of them.Claims
synchronize()removed, with no allow-wrap in its place. The pre-existing allowedseq_lensread in the same block remains (see Overview). Before this PR, the removed sync (rocm_aiter_mla_sparse.py:726on main) failed these AMD jobs with the check on: in build 97758,mi300 v1-attention shard 1/3,mi355 dpx v1-attention shard 2/4,mi355 lm-eval deepseek-v3-2-tpand-dp; in amd-ci 15532,mi300 v1-attention shard 1andshard 2,mi355 dpx v1-attention shard 1andshard 2,mi355 lm-eval deepseek-v3-2-tpand-dp.test_sparse_backend_decode_correctness[deepseek-1.0-1.0-64-1-auto-large_q_pure_prefill-ROCMAiterMLA]now gives the same result on every run on gfx950. Pristine main: 3/60 in-process failures. This PR: 0/60.get_mla_metadata_v1launch (1.3 ms at 8192 tokens) and the allowedseq_lensread.seq_lensread when AITER would not use the persistent kernel for the batch (gfx950, BF16, 16 heads, at least 32 tokens). With AITER 0.1.28.post1 on MI350X (gfx950), 16 heads and BF16, output with and without metadata was bit-identical at 32, 64, 512 and 4096 tokens over 3 reruns each. At 31 tokens, below the threshold, the two outputs differ (max 2.4e-4) because a different kernel runs; that is the positive control. At TP8 DeepSeek-V3.2 the kernel trace now matches main exactly. This covers batches of 32 or more tokens; a 16-head long-context prefill with fewer than 32 new tokens (prefix hit or short chunked tail) still builds metadata and runs the persistent kernel, which was deterministic in the study.Validation
All local runs: MI350X (gfx950), ROCm torch 2.13, amd-aiter 0.1.28.post1,
VLLM_ROCM_USE_AITER=1. Commands 1 and 2 were re-run on 2deae9c3dd and gave the same results: 155 passed / 108 skipped, and 30/30. The other runs used 36442e9f. Both earlier commits differ from 14f9dc4 only in docstrings. The command 1, unit-test and 25-run fresh-process counts were rerun at the head with the metadata-skip change and the head-count test parametrization.VLLM_GPU_SYNC_CHECK=error(command 1)test_rocm_fused_v32_dense_prefill_matches_mqa[auto],[fp8]:GPU<->CPU sync detectedat the removed line), 141 passed, 108 skippedkernels/attention/test_rocm_aiter_mla_sparse_metadata_sync.py(this PR's test file; main column runs it against main's source)test_sparse_persistent_metadata_recomputes_without_host_sync,test_sparse_metadata_workload_length_dtype_matrix[prefill-long-auto-False]), 40 passedlarge_q_pure_prefill-auto-ROCMAiterMLA, repeated in one processVLLM_GPU_SYNC_CHECK=error(command 2)VLLM_GPU_SYNC_CHECK=error; 0/25 withVLLM_GPU_SYNC_CHECK=errorafter the metadata-skip changesync-fix/rocm-enable-gpu-sync-check: this fix at 36442e9f plus #66397/#66398/#66399 and the Dockerfile change),VLLM_GPU_SYNC_CHECK=error, command 1The unit matrix is the deterministic guard for the routing change. The e2e
large_q_pure_prefilltest only catches the bad kernel about 5-25% of the time on main (3/60, 5/30, 3/12), so it is a weak regression signal on its own.Extra stress runs and commands
Independent reviewer stress of the target test: 0/50 fresh-process failures and 100/100 in-process passes (one reviewer), and another 100/100 in-process (a second reviewer). A full run of the
ROCMAiterMLAcases oftest_sparse_backend_decode_correctness, repeated 10 times, gave 108 passed / 108 skipped every time.Kernel timings (MI350X, BF16, CUDA events, mean over 20 iterations, us). The AITER-only repro and timing script uses random inputs, a fresh zeroed output and a device sync around every call. "Differing reruns" counts reruns whose output differed from the first run.
The extra
get_mla_metadata_v1launch runs once per step and all layers share it: 113 us at 512 tokens, 1.3 ms at 8192, 2.6 ms at 16384.Model eval and end-to-end latency. DeepSeek-V3.2 on 8x MI350X (gfx950), amd-aiter 0.1.28.post1, main (fd0c471) vs this PR with the same arguments. The changed path is reached only by pure-prefill steps whose longest prefill exceeds topk=2048, with BF16 KV. So the runs use
num_concurrent=1, 25-shot GSM8K (all 1319 prompts are 3151-5087 tokens) and--max-num-batched-tokens 32768, so each prompt prefills in one step. MTP is off and CUDA graphs are on (FULL_DECODE_ONLY). Dispatch was checked with rocprofv3 on an 8192-token request.mla_a16w16_qh64_qseqlen1_gqaratio64_v3_pspluskn_mla_reduce_v1_psinstead of main's nonpersistentmla_a16w16_qh16_m16x4_n32x1. That is 155.0 vs 144.0 ms over 61 layers, plus 2.7 ms of reduce and a 1.44 ms metadata launch.mla_dec_stage1_bf16_a16w16_subQ16_mqa16) on both sides. With the metadata skip, kernel names and counts match main on all 8 ranks. Without it, the only extra work was one ignored metadata launch (1.44 ms) and oneseq_lenscopy.TP4 (32 heads per rank):
TP8 (16 heads per rank):
vllm bench servewith random 8192 in / 128 out, concurrency 1, 50 prompts plus 5 warmups,--ignore-eos.configs/models-mi355-deepseek-v32-tp.txt, TP4 + MTP + eager, 5-shot) has not been rerun; its 0.6k-1.6k token prompts and high concurrency probably never reach the changed path.VLLM_ROCM_USE_AITER=1 VLLM_ROCM_USE_AITER_MLA=1 VLLM_ROCM_USE_AITER_MOE=1 \ vllm serve deepseek-ai/DeepSeek-V3.2 --tensor-parallel-size {4,8} --enable-expert-parallel \ --attention-backend ROCM_AITER_MLA_SPARSE --moe-backend aiter \ --kv-cache-dtype bfloat16 --max-model-len 16384 --max-num-batched-tokens 32768 \ --no-enable-prefix-caching lm_eval --model local-completions --tasks gsm8k --num_fewshot 25 \ --model_args model=deepseek-ai/DeepSeek-V3.2,base_url=http://localhost:8000/v1/completions,num_concurrent=1,tokenized_requests=False vllm bench serve --model deepseek-ai/DeepSeek-V3.2 --dataset-name random \ --random-input-len 8192 --random-output-len 128 --max-concurrency 1 \ --num-prompts 50 --num-warmups 5 --ignore-eosNot validated locally:
Details
Why the sync can go. With
fast_mode=True,get_mla_metadata_v1dispatches to AITER'sget_mla_metadata_v1_2_device. That function launches one device kernel onaiter::getCurrentHIPStream(), and AITER'scompile_opssets that stream to torch's current stream before each call. Nothing is staged through host memory, there is nohipMemcpy, and the function owns no buffers. The decode kernel, whether eager or captured in a graph and replayed, runs later on the same stream and reads the metadata on the device, so stream order already puts the write before the read. The block also already starts with a blockingseq_lens.cpu(), so the removed sync only waited for the single metadata kernel launched after it. The denserocm_aiter_mlabuilder already rebuilds the same metadata every step without a sync. Two investigation experiments back this up:torch.cuda._sleepqueued before the metadata kernel, which lets the host run far ahead of the GPU.kv_indptrto a different config, queue a 20M-cycle sleep, rebuild metadata with no host sync and replay. Same stream: 0/50 mismatches against per-config eager references. Positive control replaying on an unordered side stream: 49/50 mismatches.Concern about #47404. The sync came from #47404, which reported that it removed a fault on 8x MI300X (gfx942), TP8,
ROCM_AITER_MLA_SPARSE+ MTP + CUDA graphs, FP8 KV,max_model_len=200000. That PR gave no root cause beyond "graph replay could observe stale metadata". The kernel-level graph-replay test above shows that same-stream ordering covers that case, but it ran on gfx950 with one GPU and no model. #47404's exact configuration has not been rerun with this PR; all local hardware is gfx950. Plan before leaving draft: rerun that configuration (100+ requests) on MI300X with this PR, through AMD CI or a reviewer with MI300X access, or get confirmation from the #47404 author; if none is possible, the regression risk stays open and is called out to reviewers.Why the flaky test failed.
large_q_pure_prefill(2x256 tokens, topk 128, BF16 KV) never reached the sync. On main,_is_long_bf16_prefill()skipped persistent metadata for long pure BF16 prefills on every architecture, but only gfx942 sent those batches to Triton. On gfx950 they fell through to AITER's nonpersistentmla_dec_stage1_bf16_a16w16_subQ128_mqa128. With bit-identical inputs and device syncs around every call, that kernel returns different rows from run to run. Against an fp32 reference its max error is 0.12-0.21, compared with 0.0053 for the persistent kernel. So the kernel computes wrong values; this is not an ordering race. Syncs placed before or after stage 1,AMD_SERIALIZE_KERNEL=3andHIP_LAUNCH_BLOCKING=1all still reproduce the failure. It reproduces with AITER alone and the newest AITER nightly ships the same binary; reported as ROCm/aiter#7631. The 32- and 64-head nonpersistent kernels are also nondeterministic, mostly on multi-request shapes (the single-request 1x4096 64-head row above showed 0/19). The sink/LSE path still calls the nonpersistent kernel. This PR leaves it alone, and #66454 handles it separately (see Tradeoffs).Determinism by head count (AITER only, sparse layout: one entry per query token, ragged top-k indices, page_size 1; same inputs every run, NaN-poisoned output, with and without an L2 flush; 50-400 runs per config at 2x256 to 1x8192 tokens; error is the max against an fp32 reference):
subQ16_mqa16qh16_m16x4_n32x1qh64_qseqlen1_gqaratio64_v3subQ128_mqa128cannot get heuristic kernel)The fix.
_is_long_bf16_prefillis renamed to_use_triton_long_bf16_prefilland now checkson_gfx942()first. The builder and_forward_mlashare this helper, so they cannot disagree about which path a batch takes. The import ofon_gfx942stays lazy so that the tests' monkeypatch still applies. The gfx942-only Triton routing for these prefills was in place before this PR. The all-architecture skip of persistent metadata came from #49263 (cctjtanaa), which gave no reason for it. Routing gfx950 to Triton instead would be deterministic, but 5-18x slower (table above).Tradeoffs and limitations.
gpu_sync_allowed()seq_lens.cpu()read and one metadata launch per step. At TP4 DeepSeek-V3.2 (32 heads per rank) the persistent kernel is also slower than the nonpersistent one, so 8192-token TTFT rises by 17 ms (+2.6%). That is the cost of not running the 32-head nonpersistent kernel, which gives wrong values (see the determinism table).AITER_MLA_DECODE_PERSISTENT_MAX_BATCHtokens (32 by default). There, AITER'smla_decode_fwdignoreswork_meta_dataand runs the nonpersistent split-KV kernel. The builder asks AITER's own_use_persistent_mla_decodeand skips both the metadata and the read in that case, so these steps do what main did. The check also covers decode and mixed batches, where main built metadata that AITER then dropped. If the installed AITER lacks that function, the builder keeps the old behavior.use_aiter_sparse_mlaset_use_persistent_metadata=False. This PR does not change sink layers. On gfx950, BF16 sink layers still run AITER's nonpersistentmla_decode_fwd_lse, which [gfx950] mla_dec_stage1_bf16_a16w16_subQ128_mqa128.co gives nondeterministic, wrong output when a workgroup handles more than 48 KV tokens ROCm/aiter#7631 also covers (return_lse=True). [ROCm][Bugfix] Route gfx950 BF16 sparse MLA sink layers to Triton #66454 routes those layers to Triton until [Encoder decoder] Add cuda graph support during decoding for encoder-decoder models #7631 is fixed. The two PRs are independent, and either can land first.metadata_keydoes not includemax_split_per_batch. That was already true before this PR; it is only flagged for follow-up.Tests.
test_sparse_persistent_metadata_syncs_only_after_recomputeis renamed totest_sparse_persistent_metadata_recomputes_without_host_syncand now asserts that no sync happens.test_sparse_metadata_workload_length_dtype_matrixis parametrized over(on_gfx942, num_heads)in(True, 128), (False, 32), (False, 64), (False, 128): only gfx942 may skip persistent metadata for long BF16 prefills. Against main's source[prefill-long-auto-False]failed (before the head parametrization), and a 128-only rule fails the 32- and 64-head cases, so this is the deterministic guard for the routing change.test_build_skips_metadata_aiter_ignores[-1|0]uses AITER's real predicate with gfx950 and its default threshold patched in. At the threshold, it checks that the builder launches noget_mla_metadata_v1, enters nogpu_sync_allowed()read and returnswork_meta_data=None. One token below the threshold, it checks that metadata is still built. Without the source change,[0]fails.test_sparse_backend_decode_correctness[...large_q_pure_prefill-ROCMAiterMLA]runs in the AMD v1-attention CI shards, but it is a low-rate catcher (see Validation).Duplicate check. Searched open PRs for
VLLM_GPU_SYNC_CHECK,gpu_sync_allowed, "sync rocm" and sparse MLA metadata keywords. #64176 is the submitter's own predecessor, and this PR replaces its approach for this site. #66003 removes a different allowed sync, in theROCM_AITER_FAbuilder, so the two do not overlap. #62946 targets the INT4 RHT-signs site, which is handled in #66399. No open PR touches this site.AI assistance. AI assistance (Claude Code) was used to investigate, implement and validate this change. The human submitter is responsible for the change.
Pull Request Checklist
I used vLLM's
/pr-checklistskill. (Mandatory for agents, optional for humans).AI assistance was used during the creation of this PR.
Design Fit: Minimizes impact on core components, reuses existing functionality, and justifies added complexity.
Testing and Validation: Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI. (The [ROCm] Synchronize sparse MLA metadata before graph replay #47404 gfx942 rerun and the CI-parity DeepSeek-V3.2 eval are pending.)
Code Quality and Style: Keeps code and comments clear and concise, and updates relevant documentation and examples.
Pull Request Contents: Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.
BEFORE SUBMITTING, PLEASE READ https://docs.vllm.ai/en/latest/contributing (anything written below this line will be removed by GitHub Actions)