Skip to content

[ROCm][Bugfix] Stash DiffusionGemma logprobs on device without host syncs - #66398

Merged
AndreasKaratzas merged 3 commits into
vllm-project:mainfrom
AndreasKaratzas:akaratza_diffusion_gemma_sync_free_logprobs
Nov 9, 2026
Merged

AndreasKaratzas merged 3 commits into
vllm-project:mainfrom
AndreasKaratzas:akaratza_diffusion_gemma_sync_free_logprobs

Conversation

@AndreasKaratzas

@AndreasKaratzas AndreasKaratzas commented Nov 8, 2026 •

Copy link
Copy Markdown
Member

Overview

Removes the per-step GPU->CPU syncs in DiffusionSampler.__call__'s logprobs stash (DiffusionGemma), which crash the engine under VLLM_GPU_SYNC_CHECK=error in the AMD "Entrypoints Integration (API Server Generate)" group (tests/entrypoints/generate/structured_decisions/test_joint_e2e.py). The logprobs of converging slots now go into a fixed-size device stash, written where the converged mask (known only on device) says so.

Part of the series that replaces the gpu_sync_allowed() wrapping in #64176 with real fixes (following njhill's review there). The final PR in the series re-enables VLLM_GPU_SYNC_CHECK=error in docker/Dockerfile.rocm and depends on this one.

Failing CI: https://buildkite.com/vllm/amd-ci/builds/15532 (mi355-dpx-entrypoints-integration-api-server-generate):

File ".../vllm/model_executor/models/diffusion_gemma.py", line 1598, in __call__
    if just_converged.any():
RuntimeError: GPU<->CPU sync detected - avoid it or wrap with gpu_sync_allowed()
FAILED entrypoints/generate/structured_decisions/test_joint_e2e.py::test_joint_one_canvas - ... 500

CUDA CI also runs entrypoints/generate (docker/Dockerfile sets VLLM_GPU_SYNC_CHECK=error), but on an H200 MIG 35GB slice, where test_joint_e2e.py (large_gpu_mark(min_gb=80)) is skipped. So this path never ran under the sync check there.

Claims

  • With VLLM_GPU_SYNC_CHECK=error, DiffusionGemma decode steps with logprobs requests (top-k, logprob_token_ids, read-only slots, constrained allowed-set reads) no longer sync with the host. The one exception is a top-k wider than 128, which needs a raised --max-logprobs: that path syncs explicitly under gpu_sync_allowed(), only in steps that contain such a request.
  • The CI-failing path (structured decisions: read-only reads with logprob_token_ids, no diffusion_constrained) goes through the new Triton kernels with per-request token ids. The PyTorch fallback (_tile_logprobs + _masked_stash_write) serves only diffusion_constrained steps and non-CUDA platforms.
  • On ROCm, logprob values, argmax, and ranks match compute_topk_scores bit for bit for non-read-only slots. Top-k ids differ from main only in the order of ids with tied values: the kernels order ties by lower id, torch.topk arbitrarily.
  • Read-only slots: logprobs are still reported at temperature 1, and the argmax (column 0) is now taken from the same temperature-1 rows. Main took it from the schedule-tempered rows. On GPU the emitted token is sample_row_stats's argmax of the raw rows, so column 0 now equals the emitted token (except on the wide top-k > 128 path, which keeps main's tempered argmax); main's could differ when the division by the temperature rounds two distinct top logits to the same value.
  • The sampler step with logprobs is faster than main in every measured logprobs scenario (e.g. model defaults, all top-20: 4.72 -> 4.01 ms). Steps with no logprobs request are 10-20 us slower than main (e.g. 3.306 -> 3.326 ms at the defaults).
  • The stash is [max_num_reqs, canvas_length, 129], allocated in DiffusionSampler.__init__ (during load_model, before memory profiling) and never resized: 3.2 MB at the model defaults.
  • Two more syncs on the constrained-read path are fixed: allowed_tensor now uses async_tensor_h2d, and _mask_rows_to_allowed uses index_fill_ instead of keep[allowed] = True (an index_put_ of the Python scalar True).

Validation

All runs: MI355X (gfx950), ROCm torch 2.13, single GPU per server, this branch unless noted. "main" is the merge base fd0c471b3a.

The CI failure, end to end. google/diffusiongemma-26B-A4B-it:

$ cd tests && VLLM_GPU_SYNC_CHECK=error pytest -v -s entrypoints/generate/structured_decisions/test_joint_e2e.py
this PR: 6 passed (10/10 in a loop)
main:    6 failed, 1 error (teardown, after the server died)
  diffusion_gemma.py:1646 in __call__  (if just_converged.any():)
  RuntimeError: GPU<->CPU sync detected - avoid it or wrap with gpu_sync_allowed()

With a temporary print in _stash_tile_logprobs, all 22 stash calls in that run took the kernel branch (0 fallback), most of them with per-request token ids (ids=3..52, k=0).

Fail before / pass after (unit). With main's diffusion_gemma.py and diffusion_gemma_sampler.py swapped in, the three new sampler tests fail with the CI error:

$ cd tests && pytest -q v1/worker/test_gpu_diffusion_sampler.py -k "host_sync or full_vocab"   # main's model files
diffusion_gemma.py:1646: in __call__   (if just_converged.any():)
E   RuntimeError: GPU<->CPU sync detected - avoid it or wrap with gpu_sync_allowed()
diffusion_gemma.py:813: in allowed_tensor   (constrained-read test)
E   RuntimeError: GPU<->CPU sync detected - avoid it or wrap with gpu_sync_allowed()
3 failed

Reverting only the allowed_tensor fix, or only the index_fill_ fix, fails test_constrained_reads_report_their_allowed_ids_without_a_host_sync, while test_joint_e2e.py still passes 6/6 under VLLM_GPU_SYNC_CHECK=error. That unit case is the only coverage for those two fixes.

Unit tests:

$ cd tests && pytest -q v1/worker/test_gpu_diffusion_sampler.py v1/sample/test_diffusion_gemma_reads.py v1/sample/test_diffusion_sampler_stats.py
61 passed
$ VLLM_GPU_SYNC_CHECK=error pytest -q <same files>
61 passed
$ for i in $(seq 10); do pytest -q v1/sample/test_diffusion_sampler_stats.py; done
33 passed, 10/10

test_stashed_logprobs_match_compute_topk_scores runs fp32 and bf16 logits. It checks the values, that every reported id holds its value, the first-index argmax and the rank exactly. The top-k id set is compared only where the cutoff is not tied. Ties are built explicitly: a 201-way tie that straddles every cutoff, and a row with 129 distinct maxima. With torch.topk patched to break ties toward the highest index (another valid choice), the previous version of this test fails 6/12 and this one passes 24/24.

OpenAI server, PR vs main. /v1/completions, 6 prompts x logprobs 5, 20 and 0, max_tokens=128, sent one at a time to a fresh server. Diffusion requests reject temperature and seed, so these use the model's own schedule with the default engine seed. A restarted main server reproduces main bit for bit, and so does main on the other GPU, so any difference below comes from the code.

PR vs main (both GPU assignments)
sampled tokens 1510/1510 identical
sampled-token logprobs bitwise identical (max abs diff 0.0)
top-k id sets identical in 1510/1510 rows
top-k values identical (max abs diff 0.0)
top-k order 543 rows differ, all only between entries with equal values

The first PR server ran with VLLM_GPU_SYNC_CHECK=error, served every request, and logged no sync. /v1/systemone joint answers are identical to main. The non-joint answers differ between two main runs too, so they are not compared.

Serving latency. vllm bench serve --dataset-name random --random-input-len 512 --random-output-len 256 --ignore-eos --num-prompts 320 --max-concurrency 8 --seed 0 --num-warmups 8 (plus --logprobs 5), server defaults (canvas 256). Each build ran each config twice on each of the two GPUs. The runs were interleaved main/PR, and the GPUs were swapped between rounds. A canvas comes out in one piece, so TTFT equals E2E and TPOT is 0.

logprobs metric main (n=4, range) PR (n=4, range) PR vs main
none E2E mean (ms) 2527 (2455-2591) 2515 (2449-2584) -0.5%
none E2E p99 (ms) 2596 (2494-2693) 2556 (2504-2617) -1.5%
none output tok/s 810 (790-834) 814 (792-836) +0.5%
5 E2E mean (ms) 2644 (2585-2701) 2519 (2472-2567) -4.7%
5 E2E p99 (ms) 2704 (2642-2774) 2556 (2507-2596) -5.4%
5 output tok/s 771 (753-787) 812 (797-827) +5.3%

Paired on the same GPU, the E2E mean changes by -0.2% / -0.8% without logprobs and -5.0% / -4.5% with logprobs=5. The 10-20 us/step no-logprobs cost from the sampler benchmark stays inside the run-to-run noise (about ±3%). With logprobs the PR is about as fast as no logprobs, while main pays about 4.5%.

All runs
logprobs build run (GPU) E2E mean E2E p50 E2E p99 out tok/s req/s
none main r1.1 (1) 2522.82 2511.43 2693.26 811.25 3.17
none main r1.2 (1) 2455.21 2461.64 2494.30 833.73 3.26
none main r2.1 (0) 2591.13 2593.37 2615.29 790.28 3.09
none main r2.2 (0) 2539.86 2542.44 2581.12 806.19 3.15
none PR r1.1 (0) 2583.97 2589.47 2616.70 792.29 3.09
none PR r1.2 (0) 2537.90 2539.72 2571.75 806.87 3.15
none PR r2.1 (1) 2489.36 2492.37 2531.35 822.53 3.21
none PR r2.2 (1) 2448.59 2449.81 2503.53 835.94 3.27
5 main r1.1 (1) 2592.82 2596.21 2663.92 785.89 3.07
5 main r1.2 (1) 2584.97 2591.74 2642.39 787.44 3.08
5 main r2.1 (0) 2701.05 2734.31 2774.23 752.99 2.94
5 main r2.2 (0) 2695.15 2703.98 2733.77 757.72 2.96
5 PR r1.1 (0) 2566.73 2572.01 2596.28 797.00 3.11
5 PR r1.2 (0) 2562.16 2566.09 2595.30 798.73 3.12
5 PR r2.1 (1) 2472.38 2474.45 2507.45 827.21 3.23
5 PR r2.2 (1) 2474.64 2485.54 2526.77 824.71 3.22

All 320/320 requests completed in every run.

Sampler-level performance (unchanged from before; sampler alone, V=262144, fp32 logits, mean GPU ms per step):

Table
scenario main, no logprobs main first take this PR
canvas 16, step caps 2..8, 512 rows, all top-20 1.18 4.52 3.41 1.57
same, 1 of 32 requests top-20 1.18 4.51 3.39 1.32
canvas 16, 2048 rows, logprob_token_ids only 3.28 14.47 5.84 3.95
canvas 16, step where nothing converges, all top-20 (median) 1.17 1.53 3.35 1.32
model defaults (canvas 256, 8 seqs, 48 steps, caps 16..48), all top-20 3.31 4.72 9.80 4.01
model defaults, 1 of 8 top-20 3.31 4.63 9.70 3.54
model defaults, all logprob_token_ids 3.31 4.47 5.63 3.75
every slot converges every 2nd step, top-20 3.31 7.64 8.99 3.64
every slot converges every 2nd step, top-128 3.31 7.71 9.00 3.80
no logprobs request, model defaults (main vs this PR) - 3.306 - 3.326
no logprobs request, canvas 16, 2048 rows (main vs this PR) - 3.283 - 3.294

CI steps, run locally. I ran the "V1 Sample" and "V1 Executor + Worker" commands on this branch and on main, side by side on the same machine:

$ cd tests && pytest -v -s v1/sample
this PR: 1 failed, 317 passed, 43 skipped
main:    1 failed, 293 passed, 43 skipped
$ pytest -v -s v1/executor
this PR: 126 passed      main: 126 passed
$ pytest -v -s v1/worker
this PR: 1023 passed, 231 skipped      main: 1020 passed, 231 skipped

The one v1/sample failure, test_logprobs.py::test_prompt_logprob_token_ids_with_chunking_and_preemption, fails the same way on main (1/136 elements off by 0.2498 vs atol=0.2) and does not touch DiffusionGemma. The extra passes are this PR's new tests: 24 cases in v1/sample and 3 in v1/worker. Buildkite has not run on this PR yet.

Not validated: NVIDIA GPUs (the temperature division uses tl.div_rn, which is IEEE like torch's, but the bitwise match was only checked on ROCm), and gfx942/MI300.

Details

Root cause. Which slots converged is only known on the device, after the compiled sample step. The stash read it back on every step that mattered: just_converged.any(), .nonzero(), two .item() calls per converged slot, and num_sampled.cpu() to pick the emitting slots.

Fix.

  • Requests without logprobs are dropped on the host, where their sampling params are mirrored.
  • stash_converged_logprobs (new, diffusion_gemma_sampler.py) runs chunked max/argmax, block-sum/rank, top-k, top-k merge, and write kernels over a (request, position, 16x1024-chunk) grid. Rows of slots that did not converge, and canvas padding, exit at once. The logsumexp is summed in the same 1024-wide blocks and order as _topk_log_softmax_kernel; top-k keys pack order-preserving fp32 bits with the inverted index, and the K-th largest block max bounds which elements are considered. The temperature is applied in the kernels, so no scaled [rows, vocab] copy is made.
  • The output carries a canvas of rows for every logprobs request, with cu_num_generated_tokens known on the host. The scheduler slices each request by the tokens it emitted, so the host never needs to know which slots emitted, and stale rows are never read.
  • Constrained (shared allowed-set) steps and non-CUDA platforms keep a PyTorch pass over the logprobs requests' rows (_tile_logprobs, the existing per-slot code moved into a helper, plus _masked_stash_write).

Why this is so large (+809/-99 in vllm/; the new code in diffusion_gemma_sampler.py is about 340 lines of Triton kernels and 150 of launcher). Simpler designs were measured and rejected:

  • Wrapping the reads in gpu_sync_allowed() ([ROCm][CI] Re-enable VLLM_GPU_SYNC_CHECK=error in the ROCm test image #64176) leaves the syncs in place, which is the feedback this series answers.
  • compute_topk_scores over every row on every step ("first take") is sync-free but about 2x slower than main at the model defaults (9.80 vs 4.72 ms).
  • Kernels for argmax/logsumexp/rank with torch.topk still over every row: 8.51 ms at the defaults, still slower than main, because almost no slot converges on a given step at canvas 256 / 48 steps.

Splitting into a minimal sync-free fix plus a follow-up kernel PR was considered; the minimal fix alone would regress logprobs steps about 2x, so they are kept together.

Tradeoffs and limitations.

  • Wide top-k (> 128). Only with a raised --max-logprobs. A per-slot stash for it would be [canvas, vocab] per slot. Such requests keep a per-slot dict filled only for converged slots and read convergence and emission back under gpu_sync_allowed(), only in steps that contain one (about 100 lines: _pend_wide_logprobs, _emitted_logprobs). Rejecting top-k > 128 for DiffusionGemma would remove them and the last allowed sync; open for reviewer preference.
  • Per-step D2H. Every step with a logprobs request copies a canvas of stash rows per logprobs request to the host, emitting or not: n_lp * canvas * lp_width * 12 B (+8 B rank per row). Bounded by max_num_reqs * canvas * 129 * 12 B: 3.2 MB at the defaults (8 seqs, canvas 256), about 25 MB at 64 seqs with top-128. At the defaults with all top-20 this is 2048 rows/step (516 KB) vs 72 rows/step on main.
  • Steps where nothing converges still cost about 0.15-0.25 ms over no logprobs (early-exit launches plus that D2H).
  • Tie order. Tied top-k ids, including the -inf ties of top_k/top_p-masked rows, now come in lower-id order. Values and id sets are unchanged.
  • Adversarial top-k input. On synthetic logits that ascend within blocks with many near-top block maxima, K=128 over 2048 converged rows at V=262144 took 25.7 ms vs 7.0 ms for torch.topk. Gaussian, bf16-quantized, low-temperature, and top_k-masked rows are 2-4x faster than torch.
  • First use of a top-k bucket pays a Triton JIT compile.
  • Coupling. The ROCm bitwise match depends on _topk_log_softmax_kernel keeping BLOCK=1024 and 4 warps.

Duplicate-work check. Searched open PRs for VLLM_GPU_SYNC_CHECK, gpu_sync_allowed, sync rocm, and DiffusionGemma/logprobs keywords: no open PR addresses this site. #64176 is the submitter's own predecessor, which wrapped this site in gpu_sync_allowed(); this PR replaces that approach here.

AI assistance. This PR was prepared with AI assistance (Claude Code).

  • Model evaluation (test_joint_e2e.py under VLLM_GPU_SYNC_CHECK=error + one logprobs request) filled in above
Pull Request Checklist
  • I used vLLM's /pr-checklist skill. (Mandatory for agents, optional for humans).

  • AI assistance was used during the creation of this PR.

  • Design Fit: Minimizes impact on core components, reuses existing functionality, and justifies added complexity.

  • Testing and Validation: Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI.

  • Code Quality and Style: Keeps code and comments clear and concise, and updates relevant documentation and examples.

  • Pull Request Contents: Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.

BEFORE SUBMITTING, PLEASE READ https://docs.vllm.ai/en/latest/contributing (anything written below this line will be removed by GitHub Actions)

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

…yncs

With VLLM_GPU_SYNC_CHECK=error, the AMD "Entrypoints Integration (API
Server Generate)" structured-decisions tests crash the engine in
DiffusionSampler.__call__ on `if just_converged.any():`. CUDA CI runs
entrypoints/generate on an H200 MIG 35GB slice, where test_joint_e2e.py
(large_gpu_mark(min_gb=80)) is skipped, so this path never ran under the
sync check there.

Which slots converge is only known on the device, after the compiled
sample step. The logprobs stash read it back every step it mattered:
`just_converged.any()`, `.nonzero()`, two `.item()` per converged slot,
and `num_sampled.cpu()` to pick the emitting slots.

Keep the stash as a device tensor per slot instead,
[max_num_reqs, canvas_length, 1 + MAX_LOGPROB_TOKEN_IDS], allocated with
the sampler before memory profiling and never resized (3.2 MB at the
model's defaults). The output carries a canvas of rows for every logprobs
request with host-known cu_num_generated_tokens; the scheduler slices each
request by the tokens it emitted, so stale rows are never read.

Not knowing which slots converged must not cost a logprobs pass over every
row each step:
- Requests without logprobs are dropped on the host (their sampling params
  are mirrored there): they cost nothing and get no output rows.
- New Triton kernels (stash_converged_logprobs) read the converged mask
  on device; rows of other slots and canvas padding exit at once. Rows
  are split into chunks so a few converged rows still use the whole GPU.
  They compute the argmax, the logsumexp (summed block by block in
  _topk_log_softmax_kernel's order), _ranks_kernel's rank and the top-k,
  and write them to the stash. The top-k is per chunk, then merged: keys
  order by value, then by lower id, and the K-th largest 1024-block max
  bounds which elements are looked at. The temperature is applied in the
  kernels, so no scaled [rows, vocab] copy is made.
- On ROCm, logprobs, argmax and ranks match compute_topk_scores bit for
  bit; top-k ids only differ in the order of ids with tied values
  (torch.topk orders those arbitrarily, e.g. bf16 logits). Read-only
  slots take the argmax of their temperature-1 rows, the token the GPU
  sample_row_stats emits; main took it from the schedule-tempered rows.
- A top-k wider than the stash (only with a raised --max-logprobs, up to
  the full vocabulary) keeps the per-slot dict, filled only for the slots
  that converged, and reads that back under gpu_sync_allowed(): a stash
  for it would be [canvas, vocab] per slot and a top-k over all its rows
  every step a full sort.
- Constrained (shared allowed set) steps and non-CUDA runs keep a PyTorch
  pass over the logprobs requests' rows, written in place where converged.

Two more syncs on the same constrained-read path surfaced once the first
was gone: allowed_tensor built the allowed-id tensor with a blocking
torch.tensor(..., device=cuda), and _mask_rows_to_allowed ran index_put_
with the Python scalar True (`keep[allowed] = True`). Use
async_tensor_h2d and index_fill_.

Sampler step on MI355X, V=262144, fp32 logits, interleaved in one process,
mean ms of GPU time (sampler call plus the D2H of its outputs: +0.1 ms).
"host-sync-free" is the first take, which ran argmax and compute_topk_scores
over every row of the batch on every step.

                                        no lp   main  host-sync-  this
                                                      free
  canvas 16, step caps 2..8 staggered:
  (a) 512 rows, all top-20              1.18    4.52    3.41      1.57
  (b) 512 rows, 1 of 32 top-20          1.18    4.51    3.39      1.32
  (c) 2048 rows, logprob_token_ids      3.28   14.47    5.84      3.95
  canvas 16, step where nothing converges (median):
  (a)                                   1.17    1.53    3.35      1.32
  (b)                                   1.17    1.54    3.35      1.31
  (c)                                   3.31    4.16    5.62      3.47
  model defaults (canvas 256, 8 seqs, 48 steps, caps 16..48):
  all top-20                            3.31    4.72    9.80      4.01
  1 of 8 top-20                         3.31    4.63    9.70      3.54
  all logprob_token_ids                 3.31    4.47    5.63      3.75
  every slot converges every 2nd step:
  all top-20                            3.31    7.64    8.99      3.64
  all top-128                           3.31    7.71    9.00      3.80

"no lp" is main without logprobs requests. Such steps take 10-20 us more
with this change (3.306 -> 3.326 ms at the model defaults).

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andreas Karatzas <akaratza@amd.com>
@AndreasKaratzas
AndreasKaratzas force-pushed the akaratza_diffusion_gemma_sync_free_logprobs branch from 1eda73b to 56f1a6d Compare November 8, 2026 21:19

@mawong-amd mawong-amd left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The stash lifecycle and sync-free design look sound. My concern is whether the validation matches the scope: the kernels are the bulk of the diff, but the CI e2e case exercises the PyTorch fallback rather than the kernels, and nothing has run end to end yet. Before merge I'd like to see:

  • test_joint_e2e.py under VLLM_GPU_SYNC_CHECK=error — the original CI failure; this covers the fallback path.
  • The OpenAI-server completion with logprobs=5 from the body's pending list — the one piece that exercises the Triton kernel path end to end. Comparing it against main for the same greedy request would also confirm the disclosed tie-order and read-only argmax changes are the only output differences.
  • A main/PR serving-latency comparison on the same requests, with logprobs enabled and disabled — the kernel scope rests on a single sampler-level measurement, and the no-logprobs path pays a small disclosed regression, so a serving-level comparison would show both sides of the trade.
  • The selected CI steps (V1 Sample, V1 Executor + Worker) — no CI has run on this PR yet.

Co-authored-by: Claude <noreply@anthropic.com>
Signed-off-by: Andreas Karatzas <akaratza@amd.com>
@AndreasKaratzas

Copy link
Copy Markdown
Member Author

/ci run

@github-actions

github-actions Bot commented Nov 9, 2026

Copy link
Copy Markdown

✅ Triggered Buildkite CI #101183 for commit 2aede3550fd9.

ℹ️ This PR is 73 commits behind upstream main. Merged upstream main into this branch as 2aede3550fd9 before starting CI. Pull the branch before pushing more changes.

@AndreasKaratzas

Copy link
Copy Markdown
Member Author

thanks, ran all four on MI355X, body's validation section is updated with the details:

  • test_joint_e2e.py with VLLM_GPU_SYNC_CHECK=error: 6 passed on the PR (10/10 in a loop), 6 failed on main with the just_converged.any() sync error.
  • one correction to my body: that e2e doesn't use the fallback. structured decisions sends read-only reads with logprob_token_ids and no diffusion_constrained, so all 22 stash calls in that run went through the triton kernels (0 fallback). the CI case covers the kernel path with per-request ids, and the fallback only serves diffusion_constrained and non-CUDA. fixed the claim.
  • completions with logprobs 5/20/0, 6 prompts (diffusion rejects temperature, so fresh servers with the default seed): 1510/1510 tokens and sampled logprobs bitwise equal to main, top-k sets and values equal, and 543 rows reorder only among equal values. main vs a restarted main is bit-identical, so none of that is noise.
  • vllm bench serve (random 512/256, 320 prompts, concurrency 8, 2 runs per config on each GPU): no logprobs 2527 -> 2515 ms mean E2E (-0.5%, noise is ±3%), logprobs=5 2644 -> 2519 ms (-4.7%), 771 -> 812 tok/s.
  • V1 Sample and V1 Executor + Worker locally: same as main plus the new tests. the one v1/sample failure (test_prompt_logprob_token_ids_with_chunking_and_preemption) fails identically on main. buildkite still needs a run here.

@mawong-amd mawong-amd left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for running these comparisons. They show that the new path preserves model-visible outputs in the tested configurations, while leaving no-logprobs serving performance within noise and improving the logprobs case end to end. That addresses both my correctness concern and whether the added kernels are justified at serving level.

@mawong-amd mawong-amd added the ready ONLY add when PR is ready to merge/full CI is needed label Nov 9, 2026
@AndreasKaratzas
AndreasKaratzas merged commit fea3c5e into vllm-project:main Nov 9, 2026
127 checks passed
@github-project-automation github-project-automation Bot moved this from Todo to Done in AMD Nov 9, 2026
@AndreasKaratzas
AndreasKaratzas deleted the akaratza_diffusion_gemma_sync_free_logprobs branch November 9, 2026 19:18
vadiklyutiy added a commit to vadiklyutiy/vllm that referenced this pull request Nov 11, 2026
The batch-shard gather returns a rank per column when the batch has
logprob_token_ids, so the logprob_token_ids cases added upstream to
test_gather_sampler_output_logprobs_and_nans expect topk positions after
a 1-D source rank.

The DiffusionGemma per-slot stash (vllm-project#66398) holds only the sampled token's
rank. Its PyTorch fallback now stashes column 0 of the per-entry ranks
instead of failing to reshape them, as the stash kernel does.

Co-authored-by: Claude <noreply@anthropic.com>
Signed-off-by: Vadim Gimpelson <vadim.gimpelson@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working ready ONLY add when PR is ready to merge/full CI is needed rocm Related to AMD ROCm

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

3 participants