Repository navigation
[Bugfix][ROCm] Size the extensible KV cache from exact VRAM use, not 2 MB-rounded free memory - #63401
Conversation
|
This pull request has merge conflicts that must be resolved before it can be |
c67e642 to
12e41f4
Compare
|
This pull request has merge conflicts that must be resolved before it can be |
|
Documentation preview: https://vllm--63401.org.readthedocs.build/en/63401/ |
|
@njhill At
Commands: HIP_VISIBLE_DEVICES=0,1,2,3 vllm serve google/gemma-4-31B-it --served-model-name gemma4 \
--tensor-parallel-size 4 --max-model-len 4096 --gpu-memory-utilization 0.97 --port 8000
vllm bench serve --backend openai --endpoint /v1/completions --model gemma4 \
--tokenizer google/gemma-4-31B-it --dataset-name random --random-input-len 512 \
--random-output-len 128 --ignore-eos --request-rate inf --seed 0 \
--max-concurrency 128 --num-prompts 1280 |
I think #65413 got merged. Yeah apologies, I meant that the PR is overlapping in diff but runtime is completely separate. The edits now should reduce overlap now as I changed my edits and scoped it to a new file. |
|
Thanks @GodRishUniverse. I know I suggested the separate methods before but actually I now feel it's a bit fragile/clunky to make it stateless. WDYT about this simplification: njhill@f91573a Please feel free to cherry pick if it looks good to you. Then we can get this merged asap. |
Let me get back to you on this in a few days (I was on leave). Looks simpler from my initial reading. |
Thanks @GodRishUniverse, would like to make sure we get this in before the next release. |
|
This pull request has merge conflicts that must be resolved before it can be |
Signed-off-by: Nick Hill <nickhill123@gmail.com> Signed-off-by: GodRishUniverse <risagarw@amd.com>
…t and resolved conflicts Signed-off-by: GodRishUniverse <risagarw@amd.com>
|
Cherry-picked, much simpler and less code change! Thanks @njhill ! LGTM, I have resolved the merge conflicts as well. |
|
/ci run |
|
✅ Triggered Buildkite CI #100905 for commit |
|
✅ @GodRishUniverse, CI is now available for this PR.
|
|
Hi @GodRishUniverse, the pre-commit checks have failed. Please run: uv pip install pre-commit>=4.5.1
pre-commit install
pre-commit run --all-filesThen, commit the changes and push to your branch. For future commits, |
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Resolve the enable_extensible_kv_cache docstring in vllm/config/cache.py: take main's conditions (no V2 model runner condition, the NIXL IMEX exception) and keep this PR's ROCm sentence. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nick Hill <nickhill123@gmail.com>
|
/ci retry |
|
✅ Triggered Buildkite CI #100921 for commit |
…rning test #66704's test mocks current_platform.free_device_memory, which only exists after #63401. That PR isn't on releases/v0.34.0, where measure_kv_cache_blocks still calls device_memory_info(device), so the mock never applied and the test called torch.accelerator.get_memory_info on a CPU device. Mock the name the branch actually calls. Release-branch only; main has #63401. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: khluu <khluu000@gmail.com>
Overview
On ROCm,
hipMemGetInfocounts every allocation as at least 2 MB, so after warmup the extensible KV cache (#56492) sizes itself from free memory that looks a few GB smaller than it is. This PR adds that memory back once, at the final sizing, using amdgpu's exact VRAM count. Only unpartitioned discrete ROCm GPUs are affected.Root cause. The amdgpu driver rounds each allocation up to 2 MB in the free memory it reports, on purpose (commit 1ac354beecfd, "drm/amdgpu: Pessimistic availability based on rounded up allocations"), unless the GPU's memory is kept in system memory (
apu_prefer_gtt). Its own allocation check uses exact sizes. The many small kernel code objects loaded during warmup therefore hide a few GB. The sysfs countermem_info_vram_usedcounts exact sizes for the whole device.Claims
Validation
Unit tests (the sysfs lookup and correction arithmetic need no GPU; the GPU tests skip off ROCm):
On a ROCm GPU (Radeon AI PRO R9700, gfx1201).
--optionaladds tests that allocate all but 256 MB of the corrected free memory, both as 64 MB / 2 MB torch blocks and as oneExtensibleTensorchunk (thedefragment=Truecase):Serving comparison, main vs this PR, with a fresh compile cache for each run:
vllm serve meta-llama/Llama-3.1-8B-Instruct --max-model-len 8192 --gpu-memory-utilization 0.97 vllm bench serve --backend openai --endpoint /v1/completions --model meta-llama/Llama-3.1-8B-Instruct \ --dataset-name random --random-input-len 512 --random-output-len 128 --ignore-eos \ --request-rate inf --seed 0 --max-concurrency 128 --num-prompts 1280 # repeated with --max-concurrency 256 --num-prompts 2560Llama 3.1 8B, TP1, Radeon AI PRO R9700 (32 GB),
--gpu-memory-utilization 0.97. One run per config, so there is no variability figure:Not yet done:
Limitations.
gpu_memory_utilizationnow bounds exact use on unpartitioned discrete ROCm GPUs.request_memoryskips the startup check. An earlier instance's rounding is already in a later instance's baseline and is never added back, so a later instance gets a smaller cache, depending on start order. This doesn't cause OOM.tests/v1/workerruns onmi355_dpx(partitioned, so it falls back), but other jobs run onmi300_1/mi355_1. I don't know whether those pools are partitioned. If they aren't, this path runs there untested, and the correction could be limited to validated GPUs.--no-enable-extensible-kv-cacheare unchanged.Not a duplicate. I found no other open PR correcting ROCm's 2 MB-rounded free memory. #65413 (merged) corrects UMA free memory and is the fallback inside the new hook. #55828 (open) changes which memory counts as this process's own and only takes effect on NVML. No runtime overlap and either merge order works.
Related: #56492 (extensible KV cache), #65413 (UMA free-memory correction), #65418 (ROCm 3 GiB margin floor), #55828 (process-scoped KV sizing), #60121 (asynchronous VRAM release on ROCm).
Note: From my understanding,
amdsmi_get_gpu_memory_usagereads the samemem_info_vram_usedsoamdsmiandsysfsshould be the same but if it can't be read then it will ask KFD and will again use 2MB rounded values that hipMemGetInfo reports.Pull Request Checklist
[] I used vLLM's
/pr-checklistskill. (Mandatory for agents, optional for humans).AI assistance was used during the creation of this PR.
Design Fit: Minimizes impact on core components, reuses existing functionality, and justifies added complexity.
Testing and Validation: Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI.
Code Quality and Style: Keeps code and comments clear and concise, and updates relevant documentation and examples.
Pull Request Contents: Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.
AI assistance was used. I have reviewed the changes.