Repository navigation
[ROCm][Bugfix] Stash DiffusionGemma logprobs on device without host syncs - #66398
Conversation
…yncs
With VLLM_GPU_SYNC_CHECK=error, the AMD "Entrypoints Integration (API
Server Generate)" structured-decisions tests crash the engine in
DiffusionSampler.__call__ on `if just_converged.any():`. CUDA CI runs
entrypoints/generate on an H200 MIG 35GB slice, where test_joint_e2e.py
(large_gpu_mark(min_gb=80)) is skipped, so this path never ran under the
sync check there.
Which slots converge is only known on the device, after the compiled
sample step. The logprobs stash read it back every step it mattered:
`just_converged.any()`, `.nonzero()`, two `.item()` per converged slot,
and `num_sampled.cpu()` to pick the emitting slots.
Keep the stash as a device tensor per slot instead,
[max_num_reqs, canvas_length, 1 + MAX_LOGPROB_TOKEN_IDS], allocated with
the sampler before memory profiling and never resized (3.2 MB at the
model's defaults). The output carries a canvas of rows for every logprobs
request with host-known cu_num_generated_tokens; the scheduler slices each
request by the tokens it emitted, so stale rows are never read.
Not knowing which slots converged must not cost a logprobs pass over every
row each step:
- Requests without logprobs are dropped on the host (their sampling params
are mirrored there): they cost nothing and get no output rows.
- New Triton kernels (stash_converged_logprobs) read the converged mask
on device; rows of other slots and canvas padding exit at once. Rows
are split into chunks so a few converged rows still use the whole GPU.
They compute the argmax, the logsumexp (summed block by block in
_topk_log_softmax_kernel's order), _ranks_kernel's rank and the top-k,
and write them to the stash. The top-k is per chunk, then merged: keys
order by value, then by lower id, and the K-th largest 1024-block max
bounds which elements are looked at. The temperature is applied in the
kernels, so no scaled [rows, vocab] copy is made.
- On ROCm, logprobs, argmax and ranks match compute_topk_scores bit for
bit; top-k ids only differ in the order of ids with tied values
(torch.topk orders those arbitrarily, e.g. bf16 logits). Read-only
slots take the argmax of their temperature-1 rows, the token the GPU
sample_row_stats emits; main took it from the schedule-tempered rows.
- A top-k wider than the stash (only with a raised --max-logprobs, up to
the full vocabulary) keeps the per-slot dict, filled only for the slots
that converged, and reads that back under gpu_sync_allowed(): a stash
for it would be [canvas, vocab] per slot and a top-k over all its rows
every step a full sort.
- Constrained (shared allowed set) steps and non-CUDA runs keep a PyTorch
pass over the logprobs requests' rows, written in place where converged.
Two more syncs on the same constrained-read path surfaced once the first
was gone: allowed_tensor built the allowed-id tensor with a blocking
torch.tensor(..., device=cuda), and _mask_rows_to_allowed ran index_put_
with the Python scalar True (`keep[allowed] = True`). Use
async_tensor_h2d and index_fill_.
Sampler step on MI355X, V=262144, fp32 logits, interleaved in one process,
mean ms of GPU time (sampler call plus the D2H of its outputs: +0.1 ms).
"host-sync-free" is the first take, which ran argmax and compute_topk_scores
over every row of the batch on every step.
no lp main host-sync- this
free
canvas 16, step caps 2..8 staggered:
(a) 512 rows, all top-20 1.18 4.52 3.41 1.57
(b) 512 rows, 1 of 32 top-20 1.18 4.51 3.39 1.32
(c) 2048 rows, logprob_token_ids 3.28 14.47 5.84 3.95
canvas 16, step where nothing converges (median):
(a) 1.17 1.53 3.35 1.32
(b) 1.17 1.54 3.35 1.31
(c) 3.31 4.16 5.62 3.47
model defaults (canvas 256, 8 seqs, 48 steps, caps 16..48):
all top-20 3.31 4.72 9.80 4.01
1 of 8 top-20 3.31 4.63 9.70 3.54
all logprob_token_ids 3.31 4.47 5.63 3.75
every slot converges every 2nd step:
all top-20 3.31 7.64 8.99 3.64
all top-128 3.31 7.71 9.00 3.80
"no lp" is main without logprobs requests. Such steps take 10-20 us more
with this change (3.306 -> 3.326 ms at the model defaults).
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andreas Karatzas <akaratza@amd.com>
1eda73b to
56f1a6d
Compare
mawong-amd
left a comment
There was a problem hiding this comment.
The stash lifecycle and sync-free design look sound. My concern is whether the validation matches the scope: the kernels are the bulk of the diff, but the CI e2e case exercises the PyTorch fallback rather than the kernels, and nothing has run end to end yet. Before merge I'd like to see:
test_joint_e2e.pyunderVLLM_GPU_SYNC_CHECK=error— the original CI failure; this covers the fallback path.- The OpenAI-server completion with
logprobs=5from the body's pending list — the one piece that exercises the Triton kernel path end to end. Comparing it against main for the same greedy request would also confirm the disclosed tie-order and read-only argmax changes are the only output differences. - A main/PR serving-latency comparison on the same requests, with logprobs enabled and disabled — the kernel scope rests on a single sampler-level measurement, and the no-logprobs path pays a small disclosed regression, so a serving-level comparison would show both sides of the trade.
- The selected CI steps (V1 Sample, V1 Executor + Worker) — no CI has run on this PR yet.
Co-authored-by: Claude <noreply@anthropic.com> Signed-off-by: Andreas Karatzas <akaratza@amd.com>
|
/ci run |
|
✅ Triggered Buildkite CI #101183 for commit ℹ️ This PR is 73 commits behind upstream |
|
thanks, ran all four on MI355X, body's validation section is updated with the details:
|
mawong-amd
left a comment
There was a problem hiding this comment.
Thanks for running these comparisons. They show that the new path preserves model-visible outputs in the tested configurations, while leaving no-logprobs serving performance within noise and improving the logprobs case end to end. That addresses both my correctness concern and whether the added kernels are justified at serving level.
The batch-shard gather returns a rank per column when the batch has logprob_token_ids, so the logprob_token_ids cases added upstream to test_gather_sampler_output_logprobs_and_nans expect topk positions after a 1-D source rank. The DiffusionGemma per-slot stash (vllm-project#66398) holds only the sampled token's rank. Its PyTorch fallback now stashes column 0 of the per-entry ranks instead of failing to reshape them, as the stash kernel does. Co-authored-by: Claude <noreply@anthropic.com> Signed-off-by: Vadim Gimpelson <vadim.gimpelson@gmail.com>
Overview
Removes the per-step GPU->CPU syncs in
DiffusionSampler.__call__'s logprobs stash (DiffusionGemma), which crash the engine underVLLM_GPU_SYNC_CHECK=errorin the AMD "Entrypoints Integration (API Server Generate)" group (tests/entrypoints/generate/structured_decisions/test_joint_e2e.py). The logprobs of converging slots now go into a fixed-size device stash, written where the converged mask (known only on device) says so.Part of the series that replaces the
gpu_sync_allowed()wrapping in #64176 with real fixes (followingnjhill's review there). The final PR in the series re-enablesVLLM_GPU_SYNC_CHECK=errorindocker/Dockerfile.rocmand depends on this one.Failing CI: https://buildkite.com/vllm/amd-ci/builds/15532 (
mi355-dpx-entrypoints-integration-api-server-generate):CUDA CI also runs
entrypoints/generate(docker/DockerfilesetsVLLM_GPU_SYNC_CHECK=error), but on an H200 MIG 35GB slice, wheretest_joint_e2e.py(large_gpu_mark(min_gb=80)) is skipped. So this path never ran under the sync check there.Claims
VLLM_GPU_SYNC_CHECK=error, DiffusionGemma decode steps with logprobs requests (top-k,logprob_token_ids, read-only slots, constrained allowed-set reads) no longer sync with the host. The one exception is a top-k wider than 128, which needs a raised--max-logprobs: that path syncs explicitly undergpu_sync_allowed(), only in steps that contain such a request.logprob_token_ids, nodiffusion_constrained) goes through the new Triton kernels with per-request token ids. The PyTorch fallback (_tile_logprobs+_masked_stash_write) serves onlydiffusion_constrainedsteps and non-CUDA platforms.compute_topk_scoresbit for bit for non-read-only slots. Top-k ids differ from main only in the order of ids with tied values: the kernels order ties by lower id,torch.topkarbitrarily.sample_row_stats's argmax of the raw rows, so column 0 now equals the emitted token (except on the wide top-k > 128 path, which keeps main's tempered argmax); main's could differ when the division by the temperature rounds two distinct top logits to the same value.[max_num_reqs, canvas_length, 129], allocated inDiffusionSampler.__init__(duringload_model, before memory profiling) and never resized: 3.2 MB at the model defaults.allowed_tensornow usesasync_tensor_h2d, and_mask_rows_to_allowedusesindex_fill_instead ofkeep[allowed] = True(anindex_put_of the Python scalarTrue).Validation
All runs: MI355X (gfx950), ROCm torch 2.13, single GPU per server, this branch unless noted. "main" is the merge base
fd0c471b3a.The CI failure, end to end.
google/diffusiongemma-26B-A4B-it:With a temporary print in
_stash_tile_logprobs, all 22 stash calls in that run took the kernel branch (0 fallback), most of them with per-request token ids (ids=3..52, k=0).Fail before / pass after (unit). With main's
diffusion_gemma.pyanddiffusion_gemma_sampler.pyswapped in, the three new sampler tests fail with the CI error:Reverting only the
allowed_tensorfix, or only theindex_fill_fix, failstest_constrained_reads_report_their_allowed_ids_without_a_host_sync, whiletest_joint_e2e.pystill passes 6/6 underVLLM_GPU_SYNC_CHECK=error. That unit case is the only coverage for those two fixes.Unit tests:
test_stashed_logprobs_match_compute_topk_scoresruns fp32 and bf16 logits. It checks the values, that every reported id holds its value, the first-index argmax and the rank exactly. The top-k id set is compared only where the cutoff is not tied. Ties are built explicitly: a 201-way tie that straddles every cutoff, and a row with 129 distinct maxima. Withtorch.topkpatched to break ties toward the highest index (another valid choice), the previous version of this test fails 6/12 and this one passes 24/24.OpenAI server, PR vs main.
/v1/completions, 6 prompts xlogprobs5, 20 and 0,max_tokens=128, sent one at a time to a fresh server. Diffusion requests rejecttemperatureandseed, so these use the model's own schedule with the default engine seed. A restarted main server reproduces main bit for bit, and so does main on the other GPU, so any difference below comes from the code.The first PR server ran with
VLLM_GPU_SYNC_CHECK=error, served every request, and logged no sync./v1/systemonejoint answers are identical to main. The non-joint answers differ between two main runs too, so they are not compared.Serving latency.
vllm bench serve --dataset-name random --random-input-len 512 --random-output-len 256 --ignore-eos --num-prompts 320 --max-concurrency 8 --seed 0 --num-warmups 8(plus--logprobs 5), server defaults (canvas 256). Each build ran each config twice on each of the two GPUs. The runs were interleaved main/PR, and the GPUs were swapped between rounds. A canvas comes out in one piece, so TTFT equals E2E and TPOT is 0.Paired on the same GPU, the E2E mean changes by -0.2% / -0.8% without logprobs and -5.0% / -4.5% with
logprobs=5. The 10-20 us/step no-logprobs cost from the sampler benchmark stays inside the run-to-run noise (about ±3%). With logprobs the PR is about as fast as no logprobs, while main pays about 4.5%.All runs
All 320/320 requests completed in every run.
Sampler-level performance (unchanged from before; sampler alone, V=262144, fp32 logits, mean GPU ms per step):
Table
logprob_token_idsonlylogprob_token_idsCI steps, run locally. I ran the "V1 Sample" and "V1 Executor + Worker" commands on this branch and on main, side by side on the same machine:
The one
v1/samplefailure,test_logprobs.py::test_prompt_logprob_token_ids_with_chunking_and_preemption, fails the same way on main (1/136 elements off by 0.2498 vsatol=0.2) and does not touch DiffusionGemma. The extra passes are this PR's new tests: 24 cases inv1/sampleand 3 inv1/worker. Buildkite has not run on this PR yet.Not validated: NVIDIA GPUs (the temperature division uses
tl.div_rn, which is IEEE like torch's, but the bitwise match was only checked on ROCm), and gfx942/MI300.Details
Root cause. Which slots converged is only known on the device, after the compiled sample step. The stash read it back on every step that mattered:
just_converged.any(),.nonzero(), two.item()calls per converged slot, andnum_sampled.cpu()to pick the emitting slots.Fix.
stash_converged_logprobs(new,diffusion_gemma_sampler.py) runs chunked max/argmax, block-sum/rank, top-k, top-k merge, and write kernels over a (request, position, 16x1024-chunk) grid. Rows of slots that did not converge, and canvas padding, exit at once. The logsumexp is summed in the same 1024-wide blocks and order as_topk_log_softmax_kernel; top-k keys pack order-preserving fp32 bits with the inverted index, and the K-th largest block max bounds which elements are considered. The temperature is applied in the kernels, so no scaled[rows, vocab]copy is made.cu_num_generated_tokensknown on the host. The scheduler slices each request by the tokens it emitted, so the host never needs to know which slots emitted, and stale rows are never read._tile_logprobs, the existing per-slot code moved into a helper, plus_masked_stash_write).Why this is so large (+809/-99 in
vllm/; the new code indiffusion_gemma_sampler.pyis about 340 lines of Triton kernels and 150 of launcher). Simpler designs were measured and rejected:gpu_sync_allowed()([ROCm][CI] Re-enable VLLM_GPU_SYNC_CHECK=error in the ROCm test image #64176) leaves the syncs in place, which is the feedback this series answers.compute_topk_scoresover every row on every step ("first take") is sync-free but about 2x slower than main at the model defaults (9.80 vs 4.72 ms).torch.topkstill over every row: 8.51 ms at the defaults, still slower than main, because almost no slot converges on a given step at canvas 256 / 48 steps.Splitting into a minimal sync-free fix plus a follow-up kernel PR was considered; the minimal fix alone would regress logprobs steps about 2x, so they are kept together.
Tradeoffs and limitations.
--max-logprobs. A per-slot stash for it would be[canvas, vocab]per slot. Such requests keep a per-slot dict filled only for converged slots and read convergence and emission back undergpu_sync_allowed(), only in steps that contain one (about 100 lines:_pend_wide_logprobs,_emitted_logprobs). Rejecting top-k > 128 for DiffusionGemma would remove them and the last allowed sync; open for reviewer preference.n_lp * canvas * lp_width * 12 B(+8 B rank per row). Bounded bymax_num_reqs * canvas * 129 * 12 B: 3.2 MB at the defaults (8 seqs, canvas 256), about 25 MB at 64 seqs with top-128. At the defaults with all top-20 this is 2048 rows/step (516 KB) vs 72 rows/step on main.-infties oftop_k/top_p-masked rows, now come in lower-id order. Values and id sets are unchanged.torch.topk. Gaussian, bf16-quantized, low-temperature, andtop_k-masked rows are 2-4x faster than torch._topk_log_softmax_kernelkeeping BLOCK=1024 and 4 warps.Duplicate-work check. Searched open PRs for
VLLM_GPU_SYNC_CHECK,gpu_sync_allowed,sync rocm, and DiffusionGemma/logprobs keywords: no open PR addresses this site. #64176 is the submitter's own predecessor, which wrapped this site ingpu_sync_allowed(); this PR replaces that approach here.AI assistance. This PR was prepared with AI assistance (Claude Code).
test_joint_e2e.pyunderVLLM_GPU_SYNC_CHECK=error+ one logprobs request) filled in abovePull Request Checklist
I used vLLM's
/pr-checklistskill. (Mandatory for agents, optional for humans).AI assistance was used during the creation of this PR.
Design Fit: Minimizes impact on core components, reuses existing functionality, and justifies added complexity.
Testing and Validation: Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI.
Code Quality and Style: Keeps code and comments clear and concise, and updates relevant documentation and examples.
Pull Request Contents: Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.
BEFORE SUBMITTING, PLEASE READ https://docs.vllm.ai/en/latest/contributing (anything written below this line will be removed by GitHub Actions)