Skip to content

[Bugfix][ROCm] Size the extensible KV cache from exact VRAM use, not 2 MB-rounded free memory - #63401

Merged
njhill merged 9 commits into
vllm-project:mainfrom
GodRishUniverse:rocm_cuda_graph_test
Nov 8, 2026
Merged

njhill merged 9 commits into
vllm-project:mainfrom
GodRishUniverse:rocm_cuda_graph_test

Conversation

@GodRishUniverse

@GodRishUniverse GodRishUniverse commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

Overview

On ROCm, hipMemGetInfo counts every allocation as at least 2 MB, so after warmup the extensible KV cache (#56492) sizes itself from free memory that looks a few GB smaller than it is. This PR adds that memory back once, at the final sizing, using amdgpu's exact VRAM count. Only unpartitioned discrete ROCm GPUs are affected.

Root cause. The amdgpu driver rounds each allocation up to 2 MB in the free memory it reports, on purpose (commit 1ac354beecfd, "drm/amdgpu: Pessimistic availability based on rounded up allocations"), unless the GPU's memory is kept in system memory (apu_prefer_gtt). Its own allocation check uses exact sizes. The many small kernel code objects loaded during warmup therefore hide a few GB. The sysfs counter mem_info_vram_used counts exact sizes for the whole device.

Claims

  • On an unpartitioned discrete ROCm GPU, the extensible KV cache sizes itself from exact device memory use. On Llama 3.1 8B / R9700 the KV cache went from 51,712 to 76,288 tokens, with no OOM and no failed requests (one run per config).
  • The correction is never more than what is physically free and never negative. Other processes' allocations are not added back.
  • Integrated GPUs, partitioned GPUs, a missing or mismatched sysfs count, and all non-ROCm platforms keep the current behaviour (no-op platform hooks).

Validation

Unit tests (the sysfs lookup and correction arithmetic need no GPU; the GPU tests skip off ROCm):

pytest tests/utils_/test_rocm_vram_lookup.py tests/v1/worker/test_extensible_kv_cache.py tests/v1/worker/test_gpu_worker.py tests/v1/worker/test_rocm_device_memory.py

On a ROCm GPU (Radeon AI PRO R9700, gfx1201). --optional adds tests that allocate all but 256 MB of the corrected free memory, both as 64 MB / 2 MB torch blocks and as one ExtensibleTensor chunk (the defragment=True case):

pytest tests/v1/worker/test_rocm_device_memory.py --optional

Serving comparison, main vs this PR, with a fresh compile cache for each run:

vllm serve meta-llama/Llama-3.1-8B-Instruct --max-model-len 8192 --gpu-memory-utilization 0.97

vllm bench serve --backend openai --endpoint /v1/completions --model meta-llama/Llama-3.1-8B-Instruct \
  --dataset-name random --random-input-len 512 --random-output-len 128 --ignore-eos \
  --request-rate inf --seed 0 --max-concurrency 128 --num-prompts 1280
# repeated with --max-concurrency 256 --num-prompts 2560

Llama 3.1 8B, TP1, Radeon AI PRO R9700 (32 GB), --gpu-memory-utilization 0.97. One run per config, so there is no variability figure:

main this PR
KV cache 51,712 tokens 76,288 tokens (+48%)
Output tok/s, concurrency 128 / 256 918.9 / 905.5 1,054.0 / 1,026.9
Median TTFT, concurrency 128 / 256 3,934 / 25,195 ms 1,084 / 17,110 ms
Most memory used / allowed 24.6 / 31.1 GB 27.8 / 31.1 GB
Failed requests / OOM 0 / 0 0 / 0

Not yet done:

  • KV cache size across several starts.
  • Multi-GPU, CDNA (MI300/MI355) and spec decode. I only have access to an R9700.

Limitations.

  • gpu_memory_utilization now bounds exact use on unpartitioned discrete ROCm GPUs.
  • Several instances on one GPU: with the extensible cache on, request_memory skips the startup check. An earlier instance's rounding is already in a later instance's baseline and is never added back, so a later instance gets a smaller cache, depending on start order. This doesn't cause OOM.
  • KFD's page-table reserve and RAS pages are not in the exact count (about 80 MB on R9700). The 3 GB ROCm margin floor ([Bugfix][ROCm] Raise the extensible KV cache margin floor to 3 GiB #65418) covers them.
  • Tested on R9700 (gfx1201) only. CDNA should behave the same according to the driver code, but this is unverified. In AMD CI, tests/v1/worker runs on mi355_dpx (partitioned, so it falls back), but other jobs run on mi300_1 / mi355_1. I don't know whether those pools are partitioned. If they aren't, this path runs there untested, and the correction could be limited to validated GPUs.
  • The V1 runner and --no-enable-extensible-kv-cache are unchanged.

Not a duplicate. I found no other open PR correcting ROCm's 2 MB-rounded free memory. #65413 (merged) corrects UMA free memory and is the fallback inside the new hook. #55828 (open) changes which memory counts as this process's own and only takes effect on NVML. No runtime overlap and either merge order works.

Related: #56492 (extensible KV cache), #65413 (UMA free-memory correction), #65418 (ROCm 3 GiB margin floor), #55828 (process-scoped KV sizing), #60121 (asynchronous VRAM release on ROCm).

Note: From my understanding, amdsmi_get_gpu_memory_usage reads the same mem_info_vram_used so amdsmi and sysfs should be the same but if it can't be read then it will ask KFD and will again use 2MB rounded values that hipMemGetInfo reports.

Pull Request Checklist
  • [] I used vLLM's /pr-checklist skill. (Mandatory for agents, optional for humans).

  • AI assistance was used during the creation of this PR.

  • Design Fit: Minimizes impact on core components, reuses existing functionality, and justifies added complexity.

  • Testing and Validation: Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI.

  • Code Quality and Style: Keeps code and comments clear and concise, and updates relevant documentation and examples.

  • Pull Request Contents: Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.

AI assistance was used. I have reviewed the changes.

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

@GodRishUniverse
GodRishUniverse marked this pull request as draft October 23, 2026 19:43
@mergify mergify Bot added nvidia rocm Related to AMD ROCm mrv2 Model Runner V2 specific labels Oct 23, 2026
@mergify mergify Bot added the bug Something isn't working label Oct 23, 2026
@github-project-automation github-project-automation Bot moved this to Todo in AMD Oct 23, 2026
@mergify

mergify Bot commented Oct 26, 2026

Copy link
Copy Markdown
Contributor

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @GodRishUniverse.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

@mergify mergify Bot added the needs-rebase label Oct 26, 2026
@GodRishUniverse
GodRishUniverse force-pushed the rocm_cuda_graph_test branch 3 times, most recently from c67e642 to 12e41f4 Compare October 26, 2026 14:04
@mergify mergify Bot removed the needs-rebase label Oct 26, 2026
@mergify

mergify Bot commented Oct 27, 2026

Copy link
Copy Markdown
Contributor

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @GodRishUniverse.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

@GodRishUniverse GodRishUniverse changed the title [Bugfix][ROCm][MRV2] Measure CUDA graph memory with the process's own VRAM counter on ROCm rather than using KFD estimate [Bugfix][ROCm] Correct graph memory estimates and KV-cache sizing with per-process VRAM using fdinfo rather than KFD Nov 2, 2026
@mergify

mergify Bot commented Nov 2, 2026

Copy link
Copy Markdown
Contributor

Documentation preview: https://vllm--63401.org.readthedocs.build/en/63401/

@mergify mergify Bot added documentation Improvements or additions to documentation ci/build labels Nov 2, 2026
@GodRishUniverse

Copy link
Copy Markdown
Contributor Author

@njhill
Gemma 4 31B-it, TP4 on R9700, --max-model-len 4096, --gpu-memory-utilization 0.97, 3 starts per side, fresh compile cache each start. Main is 25e7c5728d.

At --max-model-len 8192, main failed to start (KV cache too small for one request) while the PR run did go ahead and start.

main PR
KV cache 4,359 tokens 32,744 tokens
Memory added back per GPU n/a 5.98 GiB
Output tok/s 121.1–121.3 482.8–483.7
Median TTFT 126 s 17.9 s
Median TPOT 54.5 ms 104.3–104.5 ms
Failed requests / OOM 0 / 0 0 / 0
Peak VRAM, worst GPU 19.3 GiB 25.5 GiB

Commands:

HIP_VISIBLE_DEVICES=0,1,2,3 vllm serve google/gemma-4-31B-it --served-model-name gemma4 \
  --tensor-parallel-size 4 --max-model-len 4096 --gpu-memory-utilization 0.97 --port 8000

vllm bench serve --backend openai --endpoint /v1/completions --model gemma4 \
  --tokenizer google/gemma-4-31B-it --dataset-name random --random-input-len 512 \
  --random-output-len 128 --ignore-eos --request-rate inf --seed 0 \
  --max-concurrency 128 --num-prompts 1280

@GodRishUniverse

Copy link
Copy Markdown
Contributor Author

Re the merge order with #55828 (mine): the two PRs overlap in the diff but not at runtime.

* [[Core] Size the KV cache, including the extensible one, from this process's own device memory #55828](https://github.com/vllm-project/vllm/pull/55828)'s process-scoped path only runs where `get_process_memory_usage()` returns a number, and only `NvmlCudaPlatform` implements it. On ROCm it returns `None`, so `measure_kv_cache_blocks` keeps the device-wide free-memory delta, and your correction feeds into `non_kv_used` and the headroom exactly as it does in this PR.

* Only ROCm implements `get_device_memory_used()`, so on CUDA your correction is 0.

* In the diff, both PRs add a `Platform` method at the same spot in `interface.py`, and both edit right after the same free-memory reading in `measure_kv_cache_blocks` (now `device_memory_info` after [[Bugfix] Size the extensible KV cache from UMA-corrected free memory #65413](https://github.com/vllm-project/vllm/pull/65413)).

So either order works for me, and whichever lands second resolves a small conflict. #55828 is already rebased onto #65413.

On sharing one mechanism: #55828 isn't a free-memory correction. It only changes which memory counts as this process's own use and leaves the free-memory reading alone, so I don't think it has anything to share with this PR or #65413.

I think #65413 got merged. Yeah apologies, I meant that the PR is overlapping in diff but runtime is completely separate. The edits now should reduce overlap now as I changed my edits and scoped it to a new file.

@njhill

njhill commented Nov 6, 2026 •

Copy link
Copy Markdown
Member

Thanks @GodRishUniverse. I know I suggested the separate methods before but actually I now feel it's a bit fragile/clunky to make it stateless. WDYT about this simplification: njhill@f91573a

Please feel free to cherry pick if it looks good to you.

Then we can get this merged asap.

@GodRishUniverse

GodRishUniverse commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor Author

Thanks @GodRishUniverse. I know I suggested the separate methods before but actually I now feel it's a bit fragile/clunky to make it stateless. WDYT about this simplification: njhill@f91573a

Please feel free to cherry pick if it looks good to you.

Then we can get this merged asap.

Let me get back to you on this in a few days (I was on leave). Looks simpler from my initial reading.

@njhill

njhill commented Nov 7, 2026

Copy link
Copy Markdown
Member

Thanks @GodRishUniverse. I know I suggested the separate methods before but actually I now feel it's a bit fragile/clunky to make it stateless. WDYT about this simplification: njhill@f91573a
Please feel free to cherry pick if it looks good to you.
Then we can get this merged asap.

Let me get back to you on this in a few days (I was on leave). Looks simpler from my initial reading.

Thanks @GodRishUniverse, would like to make sure we get this in before the next release.

@mergify

mergify Bot commented Nov 7, 2026

Copy link
Copy Markdown
Contributor

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @GodRishUniverse.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

@mergify mergify Bot added the needs-rebase label Nov 7, 2026
njhill and others added 2 commits November 8, 2026 15:18
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Signed-off-by: GodRishUniverse <risagarw@amd.com>
…t and resolved conflicts

Signed-off-by: GodRishUniverse <risagarw@amd.com>
@GodRishUniverse

GodRishUniverse commented Nov 8, 2026 •

Copy link
Copy Markdown
Contributor Author

Cherry-picked, much simpler and less code change! Thanks @njhill ! LGTM, I have resolved the merge conflicts as well.

@mergify mergify Bot removed the needs-rebase label Nov 8, 2026

@njhill njhill left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@github-project-automation github-project-automation Bot moved this to Ready in NVIDIA Nov 8, 2026
@njhill

njhill commented Nov 8, 2026

Copy link
Copy Markdown
Member

/ci run

@github-actions

github-actions Bot commented Nov 8, 2026

Copy link
Copy Markdown

✅ Triggered Buildkite CI #100905 for commit 52d821602c76.

@njhill njhill added the ready ONLY add when PR is ready to merge/full CI is needed label Nov 8, 2026
@github-actions

github-actions Bot commented Nov 8, 2026

Copy link
Copy Markdown

✅ @GodRishUniverse, CI is now available for this PR.

  • /ci run starts upstream CI; /amd-ci run starts AMD CI only.
  • Run commands test your branch with every commit currently on its upstream target branch. If it is behind, the bot merges the target branch into it first (pull before pushing again). Append --allow-stale to a run command to test an outdated branch as is, at your own risk.
  • /ci retry retries failed jobs in the CI build for the current PR head. If the current head has no CI build, it starts a new CI build for the current head containing only jobs that failed in the latest earlier CI build for this PR.
  • /amd-ci retry retries failed jobs in AMD CI for the current PR head. Use /amd-ci run when the current head has no AMD CI build.
  • /ci cancel cancels scheduled or running CI builds for this PR branch; /amd-ci cancel does the same for AMD CI only.

@mergify

mergify Bot commented Nov 8, 2026

Copy link
Copy Markdown
Contributor

Hi @GodRishUniverse, the pre-commit checks have failed. Please run:

uv pip install pre-commit>=4.5.1
pre-commit install
pre-commit run --all-files

Then, commit the changes and push to your branch.

For future commits, pre-commit will run automatically on changed files before each commit.

njhill and others added 2 commits November 8, 2026 11:20
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Resolve the enable_extensible_kv_cache docstring in vllm/config/cache.py:
take main's conditions (no V2 model runner condition, the NIXL IMEX
exception) and keep this PR's ROCm sentence.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
@njhill

njhill commented Nov 8, 2026

Copy link
Copy Markdown
Member

/ci retry

@github-actions

github-actions Bot commented Nov 8, 2026

Copy link
Copy Markdown

✅ Triggered Buildkite CI #100921 for commit 8129bd0c4922, running 3 failed step(s) from Buildkite CI #100905.

@njhill
njhill merged commit fd0c471 into vllm-project:main Nov 8, 2026
18 checks passed
@github-project-automation github-project-automation Bot moved this from Todo to Done in AMD Nov 8, 2026
@github-project-automation github-project-automation Bot moved this from Ready to Done in NVIDIA Nov 8, 2026
khluu added a commit that referenced this pull request Nov 13, 2026
…rning test

#66704's test mocks current_platform.free_device_memory, which only exists
after #63401. That PR isn't on releases/v0.34.0, where measure_kv_cache_blocks
still calls device_memory_info(device), so the mock never applied and the
test called torch.accelerator.get_memory_info on a CPU device. Mock the name
the branch actually calls. Release-branch only; main has #63401.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: khluu <khluu000@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working ci/build documentation Improvements or additions to documentation mrv2 Model Runner V2 specific nvidia ready ONLY add when PR is ready to merge/full CI is needed rocm Related to AMD ROCm

Projects

Status: Done
Status: Done

Development

Successfully merging this pull request may close these issues.

3 participants