Skip to content

[ROCm][Bugfix] Remove step-path GPU syncs in EPLB skip-rearrange, AITER GDN prefill metadata, Moondream3 BOS and INT4 RHT signs - #66399

Merged
AndreasKaratzas merged 10 commits into
vllm-project:mainfrom
AndreasKaratzas:akaratza_rocm_step_host_syncs
Nov 10, 2026
Merged

AndreasKaratzas merged 10 commits into
vllm-project:mainfrom
AndreasKaratzas:akaratza_rocm_step_host_syncs

Conversation

@AndreasKaratzas

@AndreasKaratzas AndreasKaratzas commented Nov 8, 2026 •

Copy link
Copy Markdown
Member

Overview

Removes the GPU<->CPU syncs that VLLM_GPU_SYNC_CHECK=error flags on ROCm at four step-path sites: the EPLB ROCm skip-rearrange heuristic, the AITER FlyDSL GDN prefill metadata build, the Moondream3 BOS embedding, and the lazy INT4 KV-cache RHT sign upload. No fix uses gpu_sync_allowed(). This is part of the series that replaces the allow-wrap approach of #64176 (as njhill suggested in review) so the ROCm test image can turn the check back on (#61939 dropped it).

Claims

  • EPLB (sync mode, ROCm): the skip-rearrange heuristic from [AMD][Bugfix][EPLB] Fix elastic EP scaling accuracy on ROCm #47206 now runs on the CPU copies of the load window and current map that the host policy already fetched. On main it runs on device tensors: bool((map >= 0).all()) once, then rank_load_imbalance() twice, each doing mean_load == 0 and .item(), and the second call also copies the CPU policy map to the GPU (mapping.to(device=...), pageable). That is up to five D2H syncs and one pageable H2D copy per rearrange step. Skip decisions are unchanged.
  • AITER FlyDSL GDN prefill (Qwen3.5 on ROCm): the per-step metadata is built on the host and uploaded with one pinned non-blocking copy, instead of AITER's blocking pageable torch.tensor(..., device=cuda). The metadata is bit-identical and gsm8k does not change.
  • Moondream3: the BOS id is a non-persistent buffer registered at init, instead of a blocking torch.tensor(..., device=...) on every multimodal step.
  • INT4 per-token-head KV cache: the RHT signs are built in TritonAttentionImpl.__init__ instead of on a cache miss in the first forward that writes the INT4 cache. On current main this is latent (see Details); it applies to CUDA as well.

Validation

MI355X (gfx950), ROCm torch 2.13, amd-aiter 0.1.28.post1. Each before/after pair is an end-to-end LLM repro with VLLM_GPU_SYNC_CHECK=error, run against the base file and the fixed file. The per-site repros ran on this PR's merge base, main at 5629b26.

Site Before (base) After (this PR) Output equivalence
EPLB: DeepSeek-V2-Lite-Chat, TP2+EP, sync EPLB (window 5, interval 20), eager rc=1, GPU<->CPU sync detected at bool((physical_to_logical_map >= 0).all()) (same stack as CI qwen3-30b-a3b-fp8-block-sync-eplb-accuracy-tp2) rc=0, 0 sync errors. Both heuristic outcomes ran under the checker: after the profile pass, 7 rearrange steps, 6 skipped and 1 that moved weights. The split depends on scheduling (the series integration run gave 5 skipped, 2 moved) Old (GPU) vs new (CPU) heuristic on 150 random load/map cases (EP2/4/8, with and without redundant experts): max_rel_diff=1.192e-07 skip_decision_mismatches=0
GDN: Qwen/Qwen3.5-0.8B, MRV2, CUDA graphs, FlyDSL prefill rc=1, sync at aiter/ops/prefill_batch_metadata.py packed = torch.tensor( (same stack as CI lm-eval-qwen3-5-models-shard-1-2) rc=0, 0 sync errors; also with chunked prefill (max_num_batched_tokens=256) and prefix caching Pickled metadata before vs after: equal. 10 greedy generations identical
Moondream3: moondream/moondream3-preview, bf16, eager, single and batched image requests rc=1, sync at moondream3.py embed_multimodal rc=0, 0 sync errors Greedy outputs identical to base
INT4: Llama-3.2-1B-Instruct, int4_per_token_head, TRITON_ATTN, eager, sign cache reset to its post-load_model contents when the check is enabled rc=1, sync at _get_rht_signs -> signs.to(device) (same stack as the CI test_per_token_head_kv_cache_accuracy failure) rc=0, RHT cache at check enable: [(64, 0, 'cuda:0')] Token ids identical to an unmodified base run

The per-site repros ran before the final INT4 fixup that draws the signs explicitly as float32; the INT4 rows below were re-run on this branch.

Tests run on this branch

From tests/ with PYTHONPATH=<worktree>, one MI355X unless noted.

# EPLB
pytest -q distributed/test_eplb_algo.py distributed/test_eplb_utils.py \
  distributed/test_eplb_events.py distributed/test_eplb_migration_scheduler.py
  => 53 passed, 2 skipped
# GDN
VLLM_ROCM_USE_AITER=1 pytest -q v1/attention/test_gdn_aiter_flydsl.py v1/attention/test_gdn_metadata_builder.py
  => 79 passed
pytest -q v1/attention/test_gdn_aiter_flydsl.py v1/attention/test_gdn_metadata_builder.py \
  kernels/mamba/test_gdn_rocm_layout_dispatch.py kernels/mamba/test_gdn_forward_core_split.py
  => 106 passed, 6 skipped
# Moondream3
VLLM_GPU_SYNC_CHECK=error pytest models/multimodal/generation/test_moondream3.py \
  models/multimodal/generation/test_moondream3_moe.py
  => 11 passed, 1 skipped (test_tensor_parallel needs 2 GPUs)
VLLM_GPU_SYNC_CHECK=error pytest models/multimodal/generation/test_common.py -k moondream3
  => 1 passed (vLLM vs HF)
# INT4
pytest -q quantization/test_per_token_kv_cache.py => 85 passed
VLLM_GPU_SYNC_CHECK=error pytest -q models/quantization/test_per_token_kv_cache.py -k "int4 and TRITON_ATTN"
  => 1 passed (int4_per_token_head), 0 `sync detected` lines (GPU 7, HEAD 1c7acbc722)
VLLM_GPU_SYNC_CHECK=error pytest -q kernels/attention/test_attention_selector.py -k int4 => 6 passed

Series integration pass (all fixes of the series applied together, VLLM_GPU_SYNC_CHECK=error VLLM_ROCM_USE_AITER=1, before the INT4 float32 fixup), 0 sync detected lines each: the EPLB e2e repro (5 skipped, 2 moved), the Qwen3.5 GDN e2e repro, models/multimodal/generation/test_moondream3.py (6 passed, 1 skipped), and models/quantization/test_per_token_kv_cache.py -k int4 (1 passed; a first attempt failed on the shared-GPU "Memory of devices not free" wait, not on a sync).

Model eval: gsm8k on Qwen/Qwen3.5-0.8B (GDN metadata path)

VLLM_ROCM_USE_AITER=1, 1x MI355X; both logs show "Dispatching AITER FlyDSL GDN prefill (prepare=flydsl, chunk=flydsl)". The baseline is this branch with only vllm/_aiter_ops.py taken from main (the only file of this PR on that model's path).

lm_eval --model vllm \
  --model_args pretrained=Qwen/Qwen3.5-0.8B,max_model_len=4096,gpu_memory_utilization=0.6,seed=0 \
  --tasks gsm8k --num_fewshot 5 --limit 250 --batch_size auto
flexible-extract strict-match
baseline (_aiter_ops.py from main) 0.360 ± 0.0304 0.356 ± 0.0303
this PR 0.360 ± 0.0304 0.356 ± 0.0303

Details

Not duplicating existing work. I searched open PRs for VLLM_GPU_SYNC_CHECK, gpu_sync_allowed, "sync rocm" and each site's keywords:

CI with the check re-enabled on ROCm: vllm/ci #97758, vllm/amd-ci #15532.

EPLB ROCm skip-rearrange

In sync EPLB mode, EplbState.rearrange() runs inside the checked sample_tokens step. The generic path already copies the global load window and the current physical_to_logical_map to CPU under gpu_sync_allowed() because the default policy runs in numpy; the ROCm heuristic then ignored those copies and re-read the device tensors. The copies are now kept as locals and the heuristic uses them, and policy.rebalance_experts moves out of the allowance since it only touches CPU tensors. DefaultEplbPolicy, the only registered policy, does not mutate its inputs, so reusing the CPU map is safe. The float32 reduction order on CPU can differ by about 1e-7 relative; every rank computes on the same all-reduced data, so all ranks still agree. CUDA is unaffected (the heuristic is is_rocm()-guarded).

AITER GDN prefill metadata

GDNAttentionMetadataBuilder.build() calls rocm_aiter_ops.build_gdn_flydsl_prefill_metadata() on every prefill step. AITER computes the schedule as Python lists and materializes it on cu_seqlens.device with a blocking pageable H2D copy, the only sync on this path. The wrapper keeps its signature: it passes AITER a host cu_seqlens built from seq_lens_cpu, uploads the five int32 tensors with one async_tensor_h2d, and uses dataclasses.replace on AITER's frozen dataclasses to point at the device tensors and the caller's device cu_seqlens (so AITER's layout.cu_seqlens is cu_seqlens check holds). This mirrors the FLA path's _build_chunk_metadata.

Tradeoffs and limitations:

  • The wrapper depends on AITER's dataclass field names. A rename fails loudly in replace; a new tensor field would silently stay on the CPU.
  • AITER's async device assert that cu_seqlens matches seq_lens now compares the host cumsum with itself. Both callers derive the two from the same query_start_loc, so it could not fail in practice.
  • AITER's "building during stream capture" error no longer triggers because the build runs on a CPU tensor. The metadata builders run outside capture today.
  • The long-term fix is a pinned, non-blocking build path in AITER, after which this wrapper can go back to a plain call.

Moondream3 BOS

embed_multimodal called torch.tensor([bos_token_id], device=device) on every multimodal step. The value is a model constant, so it is registered once as a non-persistent buffer, the same pattern the file uses for _expert_map. It is not in the state dict, so weight loading is unaffected. With the base code, VLLM_GPU_SYNC_CHECK=warn reported only this line on the path.

INT4 RHT signs

_get_rht_signs() draws the signs on CPU and calls .to(device) on a cache miss, and nothing filled the cache before the first forward that writes the INT4 KV cache. In the failing CI run no warmup forward reached that write before the check was enabled, so test_per_token_head_kv_cache_accuracy failed. On current main that test passes, only because the MRV2 post-capture _dummy_run from #62560 happens to fill the cache first; nothing guarantees this, on ROCm or CUDA. The INT4 e2e "before" above therefore resets the cache to its post-load_model contents to reproduce the CI ordering, and the existing CI test is not a regression signal for this site.

TritonAttentionImpl.__init__ (the only backend supporting int4_per_token_head) now builds the signs for the head size's power-of-two block on the current device. The signs are drawn explicitly as float32: init runs under the model loader's default dtype, so otherwise they would be cached as bf16 rather than the float32 the lazy path produced (values are ±1 either way; callers pass float32 activations). Eager init is used instead of a pinned lazy upload so a first miss during CUDA-graph capture cannot leave a capture-only tensor in the cache.

Coverage gaps

  • Not run locally: gfx942/MI300; CUDA (all four paths are shared with CUDA except the ROCm-only EPLB heuristic); the CI models Qwen3-30B-A3B-FP8 (EPLB, DP2) and Qwen3.5-35B-A3B (DEP2/TP2/MXFP4), whose code paths were exercised with DeepSeek-V2-Lite and Qwen3.5-0.8B; Moondream3 TP=2; EPLB test_eplb_execute.py and test_eplb_fused_moe_layer.py (hit a 1500 s local timeout; neither exercises EplbState.rearrange).
  • No new tests. Once the ROCm test image sets VLLM_GPU_SYNC_CHECK=error again ([ROCm][CI] Re-enable VLLM_GPU_SYNC_CHECK=error in the ROCm test image #64176), the EPLB accuracy jobs, Qwen3.5 lm-eval and the Moondream3 generation tests fail without these fixes. INT4 has no CI regression signal on current main (see above). CUDA CI already runs with the check on.

Pull Request Checklist
  • I used vLLM's /pr-checklist skill. (Mandatory for agents, optional for humans).

  • AI assistance was used during the creation of this PR.

  • Design Fit: Minimizes impact on core components, reuses existing functionality, and justifies added complexity.

  • Testing and Validation: Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI.

  • Code Quality and Style: Keeps code and comments clear and concise, and updates relevant documentation and examples.

  • Pull Request Contents: Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.

AI assistance (Claude Code) was used to investigate, implement and validate these changes.

BEFORE SUBMITTING, PLEASE READ https://docs.vllm.ai/en/latest/contributing (anything written below this line will be removed by GitHub Actions)

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

AndreasKaratzas and others added 4 commits November 8, 2026 21:07
The ROCm-only skip-rearrange check in EplbState.rearrange evaluated
bool((physical_to_logical_map >= 0).all()), `mean_load == 0` and
`.item()` on GPU tensors inside the checked step, which trips
VLLM_GPU_SYNC_CHECK=error in sync EPLB runs.

The host-side rebalance policy already needs CPU copies of the global
load window and the current physical-to-logical map, fetched under
gpu_sync_allowed(). Keep those copies and run the heuristic on them
instead, so it adds no further device round-trips. The policy call
itself only touches CPU tensors and no longer needs the allowance.
The heuristic's results are unchanged (float32 on CPU vs GPU).

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andreas Karatzas <akaratza@amd.com>
…sync

With VLLM_GPU_SYNC_CHECK=error, Qwen3.5 on gfx950 failed in the first
prefill step: GDNAttentionMetadataBuilder.build() calls
rocm_aiter_ops.build_gdn_flydsl_prefill_metadata(), and AITER's
build_gated_delta_rule_prefill_metadata() materializes its whole schedule
with torch.tensor(<python list>, device=cu_seqlens.device), a blocking
pageable H2D copy inside execute_model.

The schedule only depends on the host-side sequence lengths, so pass AITER
a host cu_seqlens built from seq_lens_cpu, then upload its five int32
tensors with one pinned non-blocking copy (async_tensor_h2d) and rebind
the frozen metadata to the device cu_seqlens via dataclasses.replace. This
mirrors how the FLA path builds chunk indices on CPU and async-copies
them. The resulting metadata is identical to what AITER built before.

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andreas Karatzas <akaratza@amd.com>
…odal

embed_multimodal built the BOS token id with
torch.tensor([bos_token_id], device=device) on every multimodal step. That
is a blocking pageable H2D copy, which VLLM_GPU_SYNC_CHECK=error flags inside
execute_model (seen in the AMD "Distributed Models" and "Multimodal Models
(Extended Generation)" jobs).

The id is a model constant, so register it once at init as a non-persistent
buffer and embed it from there. No other syncs remain on the Moondream3
vision/embedding path (tilings are already keep_on_cpu).

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andreas Karatzas <akaratza@amd.com>
_get_rht_signs() draws the deterministic RHT sign vector on CPU and
copies it to the device on the first call, which happened lazily inside
the first forward that wrote the INT4 per-token-head KV cache. That
blocking pageable H2D copy is a GPU<->CPU sync, so with
VLLM_GPU_SYNC_CHECK=error the first real step failed whenever no warmup
dummy run had happened to exercise the INT4 KV write first (AMD
"Quantized Models" CI, test_per_token_head_kv_cache_accuracy). Whether it
passed depended only on which warmup runs a given worker performs.

Build the signs in TritonAttentionImpl.__init__ (the only backend with
int4_per_token_head) for the head size's power-of-two block on the
current device, so the per-step lookup is always a cache hit. The signs
are drawn as float32 explicitly, so building them under the model
loader's default dtype yields the same values and dtype as the old lazy
build; numerics are unchanged.

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andreas Karatzas <akaratza@amd.com>
@AndreasKaratzas
AndreasKaratzas force-pushed the akaratza_rocm_step_host_syncs branch from 1c7acbc to 83807e5 Compare November 8, 2026 21:19
@AndreasKaratzas

Copy link
Copy Markdown
Member Author

/ci run

@github-actions

github-actions Bot commented Nov 8, 2026

Copy link
Copy Markdown

✅ Triggered Buildkite CI #100932 for commit 83807e5ceb01.

@AndreasKaratzas

Copy link
Copy Markdown
Member Author

/ci retry

@github-actions

github-actions Bot commented Nov 9, 2026

Copy link
Copy Markdown

✅ Triggered Buildkite CI #101005 for commit 6bb95b4a9998, running 1 failed step(s) from Buildkite CI #100932.

@mawong-amd mawong-amd left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the fixes. The four-site scope is appropriate, and current-head CI is green. I left one small robustness suggestion inline; no additional tests seem necessary.

Comment thread vllm/_aiter_ops.py
)

return build_gated_delta_rule_prefill_metadata(
host_cu_seqlens = torch.tensor(

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: since this helper is explicitly building host metadata, could we pass device="cpu" here? Under torch.set_default_device("cuda"), AITER would otherwise still build on the GPU and async_tensor_h2d would receive a device tensor. Current serving callers use the normal CPU default, so this is just making the host-only intent robust.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good call, done. checked it under torch.set_default_device("cuda"): AITER now gets a CPU cu_seqlens and builds the schedule on the host.

@mawong-amd mawong-amd added the ready ONLY add when PR is ready to merge/full CI is needed label Nov 9, 2026
Co-authored-by: Claude <noreply@anthropic.com>
Signed-off-by: Andreas Karatzas <akaratza@amd.com>
@AndreasKaratzas

Copy link
Copy Markdown
Member Author

/ci run

@github-actions

github-actions Bot commented Nov 9, 2026

Copy link
Copy Markdown

✅ Triggered Buildkite CI #101244 for commit 0a4cb6184244.

ℹ️ This PR is 68 commits behind upstream main. Merged upstream main into this branch as 0a4cb6184244 before starting CI. Pull the branch before pushing more changes.

@AndreasKaratzas

Copy link
Copy Markdown
Member Author

/ci retry

@github-actions

github-actions Bot commented Nov 9, 2026

Copy link
Copy Markdown

✅ The previous CI build is still running: https://buildkite.com/vllm/ci/builds/101244

@AndreasKaratzas

Copy link
Copy Markdown
Member Author

/ci run

@github-actions

Copy link
Copy Markdown

✅ Triggered Buildkite CI #101347 for commit e21b19d0eb9c.

ℹ️ This PR is 22 commits behind upstream main. Merged upstream main into this branch as e21b19d0eb9c before starting CI. Pull the branch before pushing more changes.

@AndreasKaratzas

Copy link
Copy Markdown
Member Author

/ci retry

@github-actions

Copy link
Copy Markdown

✅ Triggered Buildkite CI #101372 for commit cef800a8cf6a, running 2 failed step(s) from Buildkite CI #101347.

@AndreasKaratzas
AndreasKaratzas merged commit 68776b0 into vllm-project:main Nov 10, 2026
24 checks passed
@github-project-automation github-project-automation Bot moved this from Todo to Done in AMD Nov 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working ready ONLY add when PR is ready to merge/full CI is needed rocm Related to AMD ROCm

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

3 participants