Repository navigation
[ROCm][Bugfix] Remove step-path GPU syncs in EPLB skip-rearrange, AITER GDN prefill metadata, Moondream3 BOS and INT4 RHT signs - #66399
Conversation
The ROCm-only skip-rearrange check in EplbState.rearrange evaluated bool((physical_to_logical_map >= 0).all()), `mean_load == 0` and `.item()` on GPU tensors inside the checked step, which trips VLLM_GPU_SYNC_CHECK=error in sync EPLB runs. The host-side rebalance policy already needs CPU copies of the global load window and the current physical-to-logical map, fetched under gpu_sync_allowed(). Keep those copies and run the heuristic on them instead, so it adds no further device round-trips. The policy call itself only touches CPU tensors and no longer needs the allowance. The heuristic's results are unchanged (float32 on CPU vs GPU). Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Andreas Karatzas <akaratza@amd.com>
…sync With VLLM_GPU_SYNC_CHECK=error, Qwen3.5 on gfx950 failed in the first prefill step: GDNAttentionMetadataBuilder.build() calls rocm_aiter_ops.build_gdn_flydsl_prefill_metadata(), and AITER's build_gated_delta_rule_prefill_metadata() materializes its whole schedule with torch.tensor(<python list>, device=cu_seqlens.device), a blocking pageable H2D copy inside execute_model. The schedule only depends on the host-side sequence lengths, so pass AITER a host cu_seqlens built from seq_lens_cpu, then upload its five int32 tensors with one pinned non-blocking copy (async_tensor_h2d) and rebind the frozen metadata to the device cu_seqlens via dataclasses.replace. This mirrors how the FLA path builds chunk indices on CPU and async-copies them. The resulting metadata is identical to what AITER built before. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Andreas Karatzas <akaratza@amd.com>
…odal embed_multimodal built the BOS token id with torch.tensor([bos_token_id], device=device) on every multimodal step. That is a blocking pageable H2D copy, which VLLM_GPU_SYNC_CHECK=error flags inside execute_model (seen in the AMD "Distributed Models" and "Multimodal Models (Extended Generation)" jobs). The id is a model constant, so register it once at init as a non-persistent buffer and embed it from there. No other syncs remain on the Moondream3 vision/embedding path (tilings are already keep_on_cpu). Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Andreas Karatzas <akaratza@amd.com>
_get_rht_signs() draws the deterministic RHT sign vector on CPU and copies it to the device on the first call, which happened lazily inside the first forward that wrote the INT4 per-token-head KV cache. That blocking pageable H2D copy is a GPU<->CPU sync, so with VLLM_GPU_SYNC_CHECK=error the first real step failed whenever no warmup dummy run had happened to exercise the INT4 KV write first (AMD "Quantized Models" CI, test_per_token_head_kv_cache_accuracy). Whether it passed depended only on which warmup runs a given worker performs. Build the signs in TritonAttentionImpl.__init__ (the only backend with int4_per_token_head) for the head size's power-of-two block on the current device, so the per-step lookup is always a cache hit. The signs are drawn as float32 explicitly, so building them under the model loader's default dtype yields the same values and dtype as the old lazy build; numerics are unchanged. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Andreas Karatzas <akaratza@amd.com>
1c7acbc to
83807e5
Compare
|
/ci run |
|
✅ Triggered Buildkite CI #100932 for commit |
|
/ci retry |
|
✅ Triggered Buildkite CI #101005 for commit |
mawong-amd
left a comment
There was a problem hiding this comment.
Thanks for the fixes. The four-site scope is appropriate, and current-head CI is green. I left one small robustness suggestion inline; no additional tests seem necessary.
| ) | ||
|
|
||
| return build_gated_delta_rule_prefill_metadata( | ||
| host_cu_seqlens = torch.tensor( |
There was a problem hiding this comment.
Nit: since this helper is explicitly building host metadata, could we pass device="cpu" here? Under torch.set_default_device("cuda"), AITER would otherwise still build on the GPU and async_tensor_h2d would receive a device tensor. Current serving callers use the normal CPU default, so this is just making the host-only intent robust.
There was a problem hiding this comment.
good call, done. checked it under torch.set_default_device("cuda"): AITER now gets a CPU cu_seqlens and builds the schedule on the host.
Co-authored-by: Claude <noreply@anthropic.com> Signed-off-by: Andreas Karatzas <akaratza@amd.com>
|
/ci run |
|
✅ Triggered Buildkite CI #101244 for commit ℹ️ This PR is 68 commits behind upstream |
|
/ci retry |
|
✅ The previous CI build is still running: https://buildkite.com/vllm/ci/builds/101244 |
|
/ci run |
|
✅ Triggered Buildkite CI #101347 for commit ℹ️ This PR is 22 commits behind upstream |
|
/ci retry |
|
✅ Triggered Buildkite CI #101372 for commit |
Overview
Removes the GPU<->CPU syncs that
VLLM_GPU_SYNC_CHECK=errorflags on ROCm at four step-path sites: the EPLB ROCm skip-rearrange heuristic, the AITER FlyDSL GDN prefill metadata build, the Moondream3 BOS embedding, and the lazy INT4 KV-cache RHT sign upload. No fix usesgpu_sync_allowed(). This is part of the series that replaces the allow-wrap approach of #64176 (asnjhillsuggested in review) so the ROCm test image can turn the check back on (#61939 dropped it).Claims
mainit runs on device tensors:bool((map >= 0).all())once, thenrank_load_imbalance()twice, each doingmean_load == 0and.item(), and the second call also copies the CPU policy map to the GPU (mapping.to(device=...), pageable). That is up to five D2H syncs and one pageable H2D copy per rearrange step. Skip decisions are unchanged.torch.tensor(..., device=cuda). The metadata is bit-identical and gsm8k does not change.torch.tensor(..., device=...)on every multimodal step.TritonAttentionImpl.__init__instead of on a cache miss in the first forward that writes the INT4 cache. On currentmainthis is latent (see Details); it applies to CUDA as well.Validation
MI355X (gfx950), ROCm torch 2.13, amd-aiter 0.1.28.post1. Each before/after pair is an end-to-end
LLMrepro withVLLM_GPU_SYNC_CHECK=error, run against the base file and the fixed file. The per-site repros ran on this PR's merge base,mainat 5629b26.GPU<->CPU sync detectedatbool((physical_to_logical_map >= 0).all())(same stack as CIqwen3-30b-a3b-fp8-block-sync-eplb-accuracy-tp2)max_rel_diff=1.192e-07 skip_decision_mismatches=0aiter/ops/prefill_batch_metadata.pypacked = torch.tensor((same stack as CIlm-eval-qwen3-5-models-shard-1-2)max_num_batched_tokens=256) and prefix cachingmoondream3.pyembed_multimodalint4_per_token_head, TRITON_ATTN, eager, sign cache reset to its post-load_modelcontents when the check is enabled_get_rht_signs->signs.to(device)(same stack as the CItest_per_token_head_kv_cache_accuracyfailure)RHT cache at check enable: [(64, 0, 'cuda:0')]The per-site repros ran before the final INT4 fixup that draws the signs explicitly as float32; the INT4 rows below were re-run on this branch.
Tests run on this branch
From
tests/withPYTHONPATH=<worktree>, one MI355X unless noted.Series integration pass (all fixes of the series applied together,
VLLM_GPU_SYNC_CHECK=error VLLM_ROCM_USE_AITER=1, before the INT4 float32 fixup), 0sync detectedlines each: the EPLB e2e repro (5 skipped, 2 moved), the Qwen3.5 GDN e2e repro,models/multimodal/generation/test_moondream3.py(6 passed, 1 skipped), andmodels/quantization/test_per_token_kv_cache.py -k int4(1 passed; a first attempt failed on the shared-GPU "Memory of devices not free" wait, not on a sync).Model eval: gsm8k on Qwen/Qwen3.5-0.8B (GDN metadata path)
VLLM_ROCM_USE_AITER=1, 1x MI355X; both logs show "Dispatching AITER FlyDSL GDN prefill (prepare=flydsl, chunk=flydsl)". The baseline is this branch with onlyvllm/_aiter_ops.pytaken frommain(the only file of this PR on that model's path)._aiter_ops.pyfrommain)Details
Not duplicating existing work. I searched open PRs for
VLLM_GPU_SYNC_CHECK,gpu_sync_allowed, "sync rocm" and each site's keywords:aarushjain29) reported the INT4 RHT-sign sync first and wraps it ingpu_sync_allowed(); credit to it for the report. Here the signs are built at attention-impl init, so no allowance is needed and no forward takes the miss.CI with the check re-enabled on ROCm: vllm/ci #97758, vllm/amd-ci #15532.
EPLB ROCm skip-rearrange
In sync EPLB mode,
EplbState.rearrange()runs inside the checkedsample_tokensstep. The generic path already copies the global load window and the currentphysical_to_logical_mapto CPU undergpu_sync_allowed()because the default policy runs in numpy; the ROCm heuristic then ignored those copies and re-read the device tensors. The copies are now kept as locals and the heuristic uses them, andpolicy.rebalance_expertsmoves out of the allowance since it only touches CPU tensors.DefaultEplbPolicy, the only registered policy, does not mutate its inputs, so reusing the CPU map is safe. The float32 reduction order on CPU can differ by about 1e-7 relative; every rank computes on the same all-reduced data, so all ranks still agree. CUDA is unaffected (the heuristic isis_rocm()-guarded).AITER GDN prefill metadata
GDNAttentionMetadataBuilder.build()callsrocm_aiter_ops.build_gdn_flydsl_prefill_metadata()on every prefill step. AITER computes the schedule as Python lists and materializes it oncu_seqlens.devicewith a blocking pageable H2D copy, the only sync on this path. The wrapper keeps its signature: it passes AITER a hostcu_seqlensbuilt fromseq_lens_cpu, uploads the five int32 tensors with oneasync_tensor_h2d, and usesdataclasses.replaceon AITER's frozen dataclasses to point at the device tensors and the caller's devicecu_seqlens(so AITER'slayout.cu_seqlens is cu_seqlenscheck holds). This mirrors the FLA path's_build_chunk_metadata.Tradeoffs and limitations:
replace; a new tensor field would silently stay on the CPU.cu_seqlensmatchesseq_lensnow compares the host cumsum with itself. Both callers derive the two from the samequery_start_loc, so it could not fail in practice.Moondream3 BOS
embed_multimodalcalledtorch.tensor([bos_token_id], device=device)on every multimodal step. The value is a model constant, so it is registered once as a non-persistent buffer, the same pattern the file uses for_expert_map. It is not in the state dict, so weight loading is unaffected. With the base code,VLLM_GPU_SYNC_CHECK=warnreported only this line on the path.INT4 RHT signs
_get_rht_signs()draws the signs on CPU and calls.to(device)on a cache miss, and nothing filled the cache before the first forward that writes the INT4 KV cache. In the failing CI run no warmup forward reached that write before the check was enabled, sotest_per_token_head_kv_cache_accuracyfailed. On currentmainthat test passes, only because the MRV2 post-capture_dummy_runfrom #62560 happens to fill the cache first; nothing guarantees this, on ROCm or CUDA. The INT4 e2e "before" above therefore resets the cache to its post-load_modelcontents to reproduce the CI ordering, and the existing CI test is not a regression signal for this site.TritonAttentionImpl.__init__(the only backend supportingint4_per_token_head) now builds the signs for the head size's power-of-two block on the current device. The signs are drawn explicitly as float32: init runs under the model loader's default dtype, so otherwise they would be cached as bf16 rather than the float32 the lazy path produced (values are ±1 either way; callers pass float32 activations). Eager init is used instead of a pinned lazy upload so a first miss during CUDA-graph capture cannot leave a capture-only tensor in the cache.Coverage gaps
test_eplb_execute.pyandtest_eplb_fused_moe_layer.py(hit a 1500 s local timeout; neither exercisesEplbState.rearrange).VLLM_GPU_SYNC_CHECK=erroragain ([ROCm][CI] Re-enable VLLM_GPU_SYNC_CHECK=error in the ROCm test image #64176), the EPLB accuracy jobs, Qwen3.5 lm-eval and the Moondream3 generation tests fail without these fixes. INT4 has no CI regression signal on currentmain(see above). CUDA CI already runs with the check on.Pull Request Checklist
I used vLLM's
/pr-checklistskill. (Mandatory for agents, optional for humans).AI assistance was used during the creation of this PR.
Design Fit: Minimizes impact on core components, reuses existing functionality, and justifies added complexity.
Testing and Validation: Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI.
Code Quality and Style: Keeps code and comments clear and concise, and updates relevant documentation and examples.
Pull Request Contents: Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.
AI assistance (Claude Code) was used to investigate, implement and validate these changes.
BEFORE SUBMITTING, PLEASE READ https://docs.vllm.ai/en/latest/contributing (anything written below this line will be removed by GitHub Actions)