Skip to content

[Core] Size the KV cache, including the extensible one, from this process's own device memory - #55828

Open
twu3202 wants to merge 6 commits into
vllm-project:mainfrom
twu3202:profiling-process-memory
Open

twu3202 wants to merge 6 commits into
vllm-project:mainfrom
twu3202:profiling-process-memory

Conversation

@twu3202

@twu3202 twu3202 commented Sep 8, 2026 •

Copy link
Copy Markdown
Contributor

Purpose

memory_profiling() derives total_consumed (weights + non-torch overhead, the base of the KV cache budget) from the
device-wide free-memory delta between the worker's init snapshot and the end of the profile run. Anything another
process allocates or releases on the same device in that window is charged to, or credited against, this instance:
two instances started at the same time on one GPU shrink each other's KV cache or fail with
No available memory for the cache blocks or To serve at least one request with the model's max seq len .... A
release by another process trips the init_free_memory >= free_gpu_memory assertion when it is larger than this
instance's own growth; a smaller one passes, and the KV cache silently takes the freed memory (on a GB10 with v0.25.1
the instance ended up 22% over --gpu-memory-utilization, see
this comment). Fixes #55827; #55169 hit
the same mechanism with sleep/wake
model swapping (closed as interference from concurrent GPU use), #35920 is the UMA variant, and #56830 is the
assertion side of the same mechanism (a co-tenant releasing memory during profiling, or host memory moving on
a unified-memory part with no co-tenant at all).

Since #56492, the KV cache on CUDA and ROCm with the V2 model runner is extensible by default. It is sized a second
time after warmup, in measure_kv_cache_bytes, again from the drop in free device memory since startup. With only
the profiling fix (the first two commits), both instances in the concurrent run start, but the after-warmup sizing
leaves them at 7.16 / 2.57 GiB instead of 11.49 / 6.60 GiB. This PR covers both steps.

Changes:

  • Platform.get_process_memory_usage(device_id) returns the device memory used by the current process
    (None when the platform cannot attribute memory to processes); NvmlCudaPlatform implements it through
    nvmlDeviceGetComputeRunningProcesses and returns None for any NVML error, missing PID entry (e.g. a container
    whose PID namespace NVML does not see), WDDM/MIG (usedGpuMemory unavailable).
  • MemorySnapshot records it as process_memory; memory_profiling() uses after − before of that value as
    total_consumed when both snapshots have it (MemoryProfilingResult.process_scoped = True) and logs a warning when
    the device-wide delta differs from it by ≥ 128 MiB (other processes changed their usage while we were loading).
    Without per-process data the behaviour is unchanged.
  • Worker.determine_available_memory keeps the release-during-profiling assertion only for the device-level path;
    with per-process accounting a release by others is logged and ignored.
  • measure_kv_cache_bytes (the extensible KV cache's sizing after warmup) takes this process's own growth since the
    init snapshot in place of the drop in free memory when the platform reports it. The cap from what is physically
    free on the device stays as it is.

gpu_memory_utilization keeps its meaning (this instance uses at most that fraction of the device). Co-tenants that
together ask for more than the device has still fail rather than silently. Without the extensible KV cache, the
start-time check stops an instance started after the others have loaded, and instances started together fail when
they allocate the KV cache, as before. The extensible KV cache never grows past what is free on the device after
warmup, less a margin.

Test Plan

  • tests/utils_/test_mem_utils.py: mocked platform — another process allocating between the snapshots is not
    counted (process_scoped), fallback to the device delta when per-process usage is unavailable, snapshot
    subtraction; real device — a spawned helper process allocates 512 MiB while this process profiles, total_consumed
    must reflect only this process's 256 MiB (skipped when the platform cannot report per-process usage). Existing
    test_memory_profiling unchanged (own allocations are counted identically by both methods).
  • tests/v1/worker/test_extensible_kv_cache.py: another process allocating or releasing 2 GiB before the after-warmup
    measurement changes the budget computed from free memory, but not the one computed from this process's own growth.
  • End to end (repro from [Bug]: Starting two vLLM instances on one GPU at the same time makes each count the other's model loading as its own non-torch memory (KV cache shrinks or "No available memory for the cache blocks") #55827): Qwen/Qwen3-ASR-1.7B at util 0.35 and Qwen/Qwen3-ForcedAligner-0.6B at
    util 0.20 started sequentially (reference), concurrently, and the aligner 1 s after killing the ASR, with the
    default (extensible) KV cache and with --no-enable-extensible-kv-cache. Also one ASR instance alone at util
    0.9 and 0.97, where both ways of counting should agree.

Test Result

  • Unit tests (RTX 6000 Ada, main b10a8c2 and this branch rebased there, each file on its own as in CI):
    tests/utils_/test_mem_utils.py 7 passed (main: 3), tests/v1/worker/test_extensible_kv_cache.py 22 passed
    (main: 20). pre-commit (ruff, mypy, typos, SPDX headers, forbidden imports, Buildkite test wiring) passes on the
    changed files.

  • End to end, RTX 6000 Ada 48 GB (bare metal), main @ b10a8c2 vs this branch rebased there, V2 model runner (the
    default), --enforce-eager,
    KV cache size of Qwen/Qwen3-ASR-1.7B (util 0.35) / Qwen/Qwen3-ForcedAligner-0.6B (util 0.20):

    start order KV cache main this PR
    sequential (reference) extensible (default) 11.49 / 6.60 GiB 11.49 / 6.60 GiB
    both at once extensible (default) 9.72 GiB / aligner fails 11.49 / 6.60 GiB
    aligner started 1 s after killing the ASR extensible (default) – / 6.60 GiB – / 6.60 GiB
    sequential (reference) --no-enable-extensible-kv-cache 11.49 / 6.60 GiB 11.49 / 6.60 GiB
    both at once --no-enable-extensible-kv-cache 9.72 GiB / aligner fails 11.49 / 6.60 GiB
    aligner started 1 s after killing the ASR --no-enable-extensible-kv-cache – / 6.60 GiB – / 6.60 GiB

    On main the aligner fails in both modes with To serve at least one request with the model's max seq len (4096), (0.44 GiB KV cache is needed, which is larger than the available KV cache memory (0.16 GiB). With the patch the
    concurrent start logs, e.g. for the ASR, Other processes on cuda:0 changed their device memory usage by 8.39 GiB while this instance was loading and profiling. The KV cache budget is based on this process's own usage (4.0 GiB) rather than the device-wide change (12.39 GiB).

  • One ASR instance alone, extensible KV cache: 37.55 GiB at util 0.9 and 40.73 GiB at util 0.97, the same on main
    and with this PR (startup sizing only, no load).

  • DGX Spark (GB10, unified memory) on 0dcbd1c, typhoon-ai/typhoon2.5-qwen3-4b at util 0.12, V2 model runner,
    measured by spped2000 (comment): 7 starts
    on the host and in a container without --pid=host (2 quiet, 5 with a 4 GiB host process freed or allocated
    during profiling). Every start used the per-process path (inside the container NVML returned the namespace PID,
    matching os.getpid()), and the instance stayed within its 14,953 MiB budget (at most 14,910 MiB; v0.25.1
    reached 18,251 MiB with the same release, see the
    earlier comment). The final KV size
    did not change with the other process; it differs only between a start that compiles (2752 blocks) and one that
    loads the cached graph (2944), which happens on main as well.

Earlier runs, before #56492 (main @ 8a0a7ee and 144e79c, this patch as an overlay)
  • RTX 6000 Ada 48 GB (bare metal, driver 595.84), main @ 8a0a7ee, same models and utilisations. Unit tests: 7 passed.

    start order ASR KV before aligner KV before ASR KV after aligner KV after
    sequential (reference) 10.82 GiB 6.50 GiB 10.82 GiB 6.50 GiB
    both at once 9.01 GiB −0.01 GiB → No available memory for the cache blocks 10.82 GiB 6.50 GiB
    aligner started 1 s after killing the ASR – 6.50 GiB – 6.50 GiB, no assertion
  • RTX 5090 32 GB (docker, driver 595.80; NVML reports the container PID), main @ 144e79c. Unit tests: 7 passed.

    start order ASR KV before aligner KV before ASR KV after aligner KV after
    sequential (reference) 4.72 GiB 2.82 GiB 4.72 GiB 2.82 GiB
    both at once 2.50 GiB −3.86 GiB → No available memory for the cache blocks 4.72 GiB 2.82 GiB
    aligner started 1 s after killing the ASR – 2.82 GiB – 2.82 GiB

Independently verified on an integrated-GPU part (DGX Spark / GB10, unified memory) by hclsys — see this comment: host-only memory churn trips the unpatched assertion, and the patched KV budget is unchanged by it. That run predates #56492.


Essential Elements of an Effective PR Description Checklist
  • The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
  • The test plan, such as providing test command.
  • The test results, such as pasting the results comparison before and after, or e2e results
  • (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.

🤖 Generated with Claude Code

@twu3202
twu3202 requested a review from njhill as a code owner September 8, 2026 05:00

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

@github-actions

github-actions Bot commented Sep 8, 2026

Copy link
Copy Markdown

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

PRs do not trigger a full CI run by default. Reviewers with write access and configured trusted contributors can comment /ci run for upstream CI or /amd-ci run for AMD CI only whenever CI signals are needed.

Once the PR is approved or has the ready label, the PR author can also use the corresponding /ci run, /ci retry, and /ci cancel commands, or their /amd-ci variants. New commits do not start upstream CI automatically.

If you have any questions, please reach out to us on Slack at https://slack.vllm.ai.

Agent Guidelines

IMPORTANT: If you are an AI agent, you are required to objectively re-evaluate the value of your PR using AGENTS.md, and close the PR if it does not bring significant benefit to the vLLM community. Failure to do so may result in an immediate ban.

🚀

@hclsys

hclsys commented Sep 14, 2026

Copy link
Copy Markdown
Contributor

I have a DGX Spark (GB10, sm_121, aarch64, CUDA 13, driver 580.95.05) so I ran the NVML primitives this depends on against that device, since integrated/unified-memory parts are where per-process attribution is most likely to quietly not work.

It works there:

  • platform resolves to NvmlCudaPlatform, is_integrated_gpu() is True, so get_process_memory_usage is the live path rather than the None fallback
  • nvmlDeviceGetComputeRunningProcesses lists our own PID, and usedGpuMemory comes back as an int, not None
  • it tracks this process only: allocating 512 MiB moved usedGpuMemory 741326848 -> 1278197760, i.e. exactly 0.500 GiB

One thing worth knowing for this file: nvmlDeviceGetMemoryInfo is NVMLError_NotSupported on GB10 (nvidia-smi --query-gpu=memory.total also returns [N/A], and get_device_total_memory right above your new method raises there). Your method does not call it, so it is unaffected — flagging it only so nobody later "simplifies" get_process_memory_usage to go through the device-level query.

Why I bothered: #56830 filed today calls out the integrated-GPU row specifically. On unified memory the entire host is effectively a co-tenant, so the init_free_memory >= free_gpu_memory assert can fire with vLLM as sole tenant and nothing sharing the GPU, and the message's advice to "isolate vLLM in its own container" cannot help. The #53591 fallback is gated on is_rocm(), so it does not reach that row. Per-process accounting does.

To be clear about scope: I did not run a full vllm serve with your patch applied. I verified the NVML behaviour the patch relies on, on the hardware in question. Happy to run the end-to-end startup on the Spark if that would be useful to you.

@twu3202

twu3202 commented Sep 14, 2026

Copy link
Copy Markdown
Contributor Author

Thanks for checking the NVML side on GB10 — that is exactly the class of device I could not test here, and the nvmlDeviceGetMemoryInfo → NotSupported point is worth keeping: 0f1b2a7 adds a docstring note that get_process_memory_usage must stay on the per-process query.

An end-to-end startup on the Spark would help. The two things worth looking at are the KV-cache size the worker reports with and without the patch (sole tenant, so they should match), and whether the "Other processes on … changed their device memory usage" warning fires when host memory moves during the profiling window — on unified memory that is #56830's row 3.

@hclsys

hclsys commented Sep 14, 2026

Copy link
Copy Markdown
Contributor

Ran it end to end on the Spark. Your patch at 0f1b2a7 as an overlay on 1a522b6 (only the four vllm/ files; the tests were not part of the overlay), Qwen/Qwen3-0.6B, gpu_memory_utilization=0.20, max_model_len=2048, eager.

For the second question I wanted host memory to move without anything touching the GPU, so the churner is a plain Python process with no CUDA context at all: bytearray(8 GiB), hold ~1.5 s, free, repeat, for the whole startup.

quiet host churn (CPU only, no CUDA)
without patch KV 211,376 tok AssertionError: Error in memory profiling. Initial free memory 66.23 GiB, current free memory 67.62 GiB
with patch KV 212,080 tok KV 212,080 tok

So on this device the assertion is reachable by host allocation churn alone, and your patch is what makes the budget immune to it. The KV size is bit-identical across both patched runs, and the warning reports the perturbation instead of failing:

  • quiet: changed their device memory usage by 0.46 GiB ... own usage (1.48 GiB) rather than the device-wide change (1.94 GiB)
  • churn: changed their device memory usage by 1.66 GiB ... own usage (1.48 GiB) rather than the device-wide change (3.15 GiB)

Own usage stays 1.48 GiB in both; only the device-wide number moves. That is the unified-memory failure mode stated as plainly as I can get it.

Quiet unpatched vs patched is 211,376 vs 212,080, +0.33%. I would not read much into the sign — the device-wide delta on this box also picks up whatever the host did during load.

Two caveats so the numbers are not over-read. This box is not a strict sole tenant: a long-running ASR server of mine holds ~16 GiB on the GPU throughout, steady, and present in all four runs — the churner is the only thing I varied. And all four runs die later at flashinfer.autotuner.set_autotune_process_group (a local flashinfer version gap, 4 occurrences in every log, well after the profiling phase), so the comparison is apples to apples but it is startup-to-KV-sizing, not a served request.

Happy to rerun at a larger model or a different utilisation if a specific configuration would be more convincing.

@twu3202

twu3202 commented Sep 14, 2026

Copy link
Copy Markdown
Contributor Author

Thank you for taking this all the way to an end-to-end run, and for building the churner. If I'm reading the table right, host-only churn is enough to trip the assertion on the unpatched build, while the patched budget comes out the same with and without it, which is what the process-scoped path is meant to buy on a unified-memory part.

On the 0.46 GiB in your quiet run, my guess is it's the same effect at rest: during model load the host's own activity shows up in the device-wide number, which is why the warning reports the device-wide vs own-usage split rather than trying to explain it. I don't think a larger model is needed — this is the configuration that matters. I've linked your run from the PR description.

@GodRishUniverse

Copy link
Copy Markdown
Contributor

Was this stress tested at gpu_memory_utilization = 0.9 or 0.97 @twu3202 ?

@twu3202

twu3202 commented Oct 28, 2026

Copy link
Copy Markdown
Contributor Author

Not at 0.9 or 0.97. The runs in the description use 0.35 and 0.20 because the bug needs two instances on one GPU. With a single instance on a discrete GPU the two counters should agree (the sequential row gives the same KV size before and after), so I don't expect a change at high utilization, but I haven't measured it. I can run 0.9 and 0.97 alone under load on a 48 GB card and post the KV size and peak memory. Is there a case you're worried about?

@GodRishUniverse

GodRishUniverse commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

@twu3202 Not really, just curious about whether OOM behaviour against main should be tested for KV-cache sizing. But 0.35 makes sense for the 2 instances. I was concerned about 1 instance having OOM.

@twu3202
twu3202 force-pushed the profiling-process-memory branch from 0f1b2a7 to 0288ac8 Compare November 2, 2026 07:22
@twu3202 twu3202 changed the title [Core] Size the KV cache from this process's own device memory, not the device-wide delta [Core] Size the KV cache, including the extensible one, from this process's own device memory Nov 2, 2026
@mergify

mergify Bot commented Nov 4, 2026

Copy link
Copy Markdown
Contributor

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @twu3202.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

@mergify mergify Bot added the needs-rebase label Nov 4, 2026
@twu3202
twu3202 force-pushed the profiling-process-memory branch from 0288ac8 to 0dcbd1c Compare November 4, 2026 03:10
@mergify mergify Bot removed the needs-rebase label Nov 4, 2026
@spped2000

Copy link
Copy Markdown

GB10 data for the release direction, when the release is smaller than the instance's own load. The PR description says a release by another process is "credited against" this instance, and #55827 says a release now "fails loudly instead of over-allocating". On a DGX Spark I measured the case that falls between those two statements. The free-memory drop across profiling stays positive, so the assertion passes, and the instance can end up over its --gpu-memory-utilization budget without any error.

Setup.

  • DGX Spark (GB10, sm_121, aarch64), driver 580.142.
  • Stock vllm/vllm-openai:v0.25.1-aarch64, running in its own container (no --pid=host).
  • Model: typhoon-ai/typhoon2.5-qwen3-4b (BF16).
  • Flags: --gpu-memory-utilization 0.12 (14.6 GiB = 14,953 MiB), --max-model-len 4096 --max-num-seqs 4 --no-enable-prefix-caching --seed 0, VLLM_LOGGING_LEVEL=DEBUG.
  • Interferer: a host Python process with no CUDA context that holds a touched 4 GiB bytearray.
    • "Release" runs free the buffer right after the worker init memory snapshot line.
    • "Grab" runs allocate it at that point.
  • Other GPU processes: three unrelated vLLM processes held 24,381, 428 and 46,316 MiB, the same at the end of every run (per nvidia-smi). The hog was the only thing I varied.
run init free → after profile (GiB) free-memory drop non-torch forward increase Available KV cache NVML used_memory for this instance (after /health) NVML minus KV
release 1 31.76 → 27.63 4.13 GiB −3.56 GiB 9.77 GiB 18,251 MiB (+22% over the 14,953 MiB budget) 8.05 GiB
release 2 32.37 → 23.74 8.63 GiB 0.95 GiB 5.27 GiB 13,643 MiB 8.05 GiB
grab 1 32.53 → 19.82 12.71 GiB 5.03 GiB 1.19 GiB 9,467 MiB 8.05 GiB
grab 2 28.52 → 16.56 11.96 GiB 4.27 GiB 1.95 GiB 10,259 MiB 8.07 GiB

Results.

  • Release 1:
    • The 4 GiB release is smaller than what the instance itself takes (about 8 GiB), so free memory still drops overall (4.13 GiB).
    • The init_free_memory >= free_gpu_memory assertion passes.
    • The KV cache is sized as if the instance used 4 GiB less than it does. The process ends up 3.2 GiB (22%) over budget.
    • The only trace in the log is the negative non-torch forward increase memory.
  • Release 2 did not inflate, with the same timing. I have no explanation for that.
  • Grab runs: both shrank the KV cache by an amount consistent with the 4 GiB hog.
  • Per-process NVML usage: minus the KV size, it comes out at 8.05–8.07 GiB in all four runs. Over the same runs the non-KV memory that vLLM measured ranged from 4.83 to 13.41 GiB. The per-process counter this PR uses gives a stable value in exactly the runs where the device-wide delta does not.
  • Baseline drift: quiet starts also vary. Two unperturbed starts with the same CLI args, about 40 minutes apart, got 5.89 and 8.02 GiB of KV cache. Only the log level differed. So I am not quoting a "vs quiet" delta.

Why this matters for the fix:

Repro outline
  1. Wait until the host has at least 20 GiB available, plus the hog size in release mode. For release mode, start the hog first and wait until it reports "ready".
    • Hog: b = bytearray(4 << 30); b[::4096] = b'\x01' * ((4 << 30) // 4096), then sleep.
  2. Start the server: docker run --gpus all --ipc=host -e VLLM_LOGGING_LEVEL=DEBUG vllm/vllm-openai:v0.25.1-aarch64 <model> --gpu-memory-utilization 0.12 --max-model-len 4096 --max-num-seqs 4 --no-enable-prefix-caching --seed 0.
  3. Follow the log. As soon as worker init memory snapshot appears, kill the hog (release mode) or start it (grab mode).
  4. Wait for /health, then take these lines from the log: Initial free memory, Free memory after profiling, Memory profiling takes, Available KV cache memory.
  5. On the host, read per-process usage: nvidia-smi --query-compute-apps=pid,used_memory --format=csv.

Offer to test. hclsys has already covered the KV size and the warning under host churn. Two things are still open, and I can run both on the same GB10 against the current head (0dcbd1c):

  • Partial release: the same release and grab runs as above. Does the KV size stay at the per-process value, and does the "Other processes on … changed their device memory usage" warning fire?
  • PIDs inside the container: does nvmlDeviceGetComputeRunningProcesses list the worker's PID when vLLM runs in the stock image without --pid=host? In release 1 the EngineCore logged pid=148 inside the container, while nvidia-smi on the host listed it as 4095482. If NVML does not match them, get_process_memory_usage returns None and the start falls back to the device-wide path.

(Measurements and this write-up were prepared with AI assistance (Claude Code). The numbers are copied from the run logs.)

@twu3202

twu3202 commented Nov 5, 2026

Copy link
Copy Markdown
Contributor Author

Thanks for checking this. Release 1 is a case the description and #55827 left out. Both only described a release big enough to trip the assertion, but a smaller one passes, and the KV cache quietly takes the freed memory, as your run shows. I've corrected both.

Yes, both runs on 0dcbd1c would help. What I expect:

  • Partial release: if NVML sees the PID, the "Other processes on … changed their device memory usage" warning should fire with a negative value well over its 128 MiB threshold. On GB10 host activity moves free memory too, so it may also show up in quiet starts. The "Other processes released … ignored" line only appears when free memory ends up higher than at init, so it shouldn't show up in a release 1 style run.
  • KV size: with the V2 runner the cache is resized after warmup, and on this branch that step also uses this process's own usage, so the after-warmup sizing you mention after [Bugfix] Size the extensible KV cache from UMA-corrected free memory #65413 is covered too. The final size is in the Memory after warmup: … KV cache blocks (… GiB) line, not Available KV cache memory. I'd expect it to stay about the same across release, grab and quiet starts.
  • PIDs: in a docker container on an RTX 5090 (driver 595.80), NVML reported the container PID and the per-process path ran. That was a rented cloud container, so I don't know how it was started, and the run predates [Core] Extensible KV cache #56492. I haven't checked GB10 or the 580 driver. If the PIDs don't match, the worker init memory snapshot debug line shows process_memory=n/a and startup falls back to the device-wide path, i.e. the v0.25.1 behaviour.

My guess for release 2 is the same host-side noise as in your two quiet starts. A quiet start on head would be a useful third run.

@spped2000

Copy link
Copy Markdown

I ran 7 starts of 0dcbd1c on one GB10.

  • The per-process path ran in all of them, including a container without --pid=host.
  • The warning came out negative in all 3 releases.
  • The final KV size did not move with the hog within a compile state.

Setup.

  • DGX Spark (GB10), driver 580.142.
  • 0dcbd1c installed editable on top of the precompiled cu130 wheel of its merge base e5d9bad (torch 2.14.0+cu130).
  • Model and flags as before: typhoon2.5-qwen3-4b, --gpu-memory-utilization 0.12 (budget 14,953 MiB), --max-model-len 4096 --max-num-seqs 4 --no-enable-prefix-caching --seed 0, DEBUG logging. The V2 model runner was used.
  • Same 4 GiB host-only hog, freed or allocated right after the worker init memory snapshot line.
  • "host" = vllm serve as a host process.
  • "container" = docker run --gpus all --ipc=host --network host --user 1000:1000 --memory=40g --tmpfs /tmp, with the same venv bind-mounted read-only.
    • The container has its own PID namespace (no --pid).
    • nvcr.io/nvidia/vllm:26.05.post1-py3 supplies only the runtime libraries. This is not the stock vllm/vllm-openai image.
  • The three other vLLM processes on the device stayed constant across these runs, at 24,381 / 428 / 46,362 MiB.
  • Scope:
    • one model, one util, one hog size, 1–2 starts per cell;
    • host swap was full throughout;
    • no unpatched run on this base.
run "Other processes … changed by" (own vs device-wide) after warmup kept for transient activations NVML used, this instance
host quiet none (−0.11 GiB: 7.74 vs 7.63, under 128 MiB) 2752 blocks (6.05 GiB) 0.74 GiB 14,479 MiB
host release 1 −3.9 GiB (7.74 vs 3.84) 2944 blocks (6.47 GiB) 0.31 GiB 14,910 MiB
host release 2 −4.1 GiB (7.74 vs 3.64) 2944 blocks (6.47 GiB) 0.31 GiB 14,910 MiB
host grab +4.56 GiB (7.74 vs 12.3) 2944 blocks (6.47 GiB) 0.31 GiB 14,910 MiB
container quiet +0.86 GiB (7.73 vs 8.6) 2752 blocks (6.05 GiB) 0.74 GiB 14,471 MiB
container release −2.97 GiB (7.73 vs 4.76) 2752 blocks (6.05 GiB) 0.74 GiB 14,471 MiB
container grab +4.74 GiB (7.73 vs 12.48) 2752 blocks (6.05 GiB) 0.74 GiB 14,471 MiB

Results.

  • PIDs. process_memory was 0.23 GiB (not n/a) at init in all 7 runs.
    • Inside the container the EngineCore logged pid=149 / 147 / 148, while host nvidia-smi listed 3390654 / 3400494 / 3408712.
    • So NVML inside the container returned the namespace PIDs, and os.getpid() matched. This answers the question from my earlier comment, at least for this driver and setup.
  • Partial release.
    • In each release the device-wide change (3.84 / 3.64 / 4.76 GiB) is what the fallback path would have charged. The budget used 7.73–7.74 GiB instead.
    • Free memory still ended below init (e.g. 30.11 → 26.26 GiB), so, as you said, the "released … ignored" line did not appear.
  • KV size.
    • In the container, quiet, release and grab all compiled from scratch, and all three got 2752 blocks (6.05 GiB).
    • On the host, the three runs that loaded the cached compiled graph got 2944 blocks. The host quiet start compiled and got 2752.
    • The 0.42 GiB gap matches "kept for transient activations" (0.74 vs 0.31 GiB). I don't have a cached-graph quiet start on the host.
  • Budget. NVML for this instance peaked at 14,910 MiB, against the 14,953 MiB budget. On v0.25.1, release 1 reached 18,251 MiB. NVML minus the final KV is 8.08–8.09 GiB in all 7 runs.
  • Quiet start.
    • The container quiet start fired the warning at +0.86 GiB with no hog, as you expected it might. The host quiet start stayed under the threshold.
    • Part of the container offset is probably my harness: the container's compile cache and HOME are on a tmpfs, which is RAM here.
    • That is far smaller than the ~4.5 GiB missing in v0.25.1 release 2, which I still can't explain. It did not recur in the two host releases today.

Three other attempts are excluded. A harness bug left processes behind and ran the host low on memory, which ended in a watchdog abort and one CUDA OOM at init. Logs are available.

(Measurements and this write-up were prepared with AI assistance (Claude Code). The numbers are copied from the run logs.)

@twu3202

twu3202 commented Nov 5, 2026

Copy link
Copy Markdown
Contributor Author

Thanks for running these. I've added the GB10 results to the description.

On the 2752 vs 2944 split: I could reproduce it on an RTX 6000 Ada with Qwen/Qwen3-8B (main c126ebf with an unrelated transcription patch): 3232 blocks with an empty compile cache, 3648 when the compiled graph loads from the cache, with 1.19 vs 0.31 GiB kept for transient activations. The cause:

  • vLLM compiles through torch._inductor.standalone_compile, which turns on triton.autotune_at_compile_time. Inductor then benchmarks the generated kernels during codegen on random example inputs (rand_strided). A memory-history trace of the first compile shows a single 1.16 GiB allocation from this path, exactly the size of the embedding weight (151936 × 4096 × 2 bytes for Qwen3-8B). For typhoon2.5-qwen3-4b (hidden 2560) that is 0.72 GiB, close to your 0.74.
  • The compile runs inside the profiling run, so this tensor sets the torch peak, and the KV sizing keeps that peak free as transient activations. When the graph loads from the cache nothing is compiled, and the warmup steps' own 0.31 GiB is the larger peak.

So on main too, a start with an empty compile cache ends up with a smaller KV cache than later starts. The profile-time Available KV cache memory, which is the final size without the extensible cache, moves the same way (7.21 vs 8.21 GiB here).

@mergify

mergify Bot commented Nov 8, 2026

Copy link
Copy Markdown
Contributor

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @twu3202.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

@mergify mergify Bot added the needs-rebase label Nov 8, 2026
twu3202 and others added 5 commits November 9, 2026 08:56
memory_profiling() derived total_consumed, the base of the KV cache
budget, from the device-wide free-memory delta between the worker's init
snapshot and the end of the profile run, so anything another process
allocated or released on the same device in that window was charged to
or credited against this instance: two instances started at the same
time shrank each other's KV cache (or failed with 'No available memory
for the cache blocks'), and a release by another process tripped the
init_free_memory >= free_gpu_memory assertion.

Add Platform.get_process_memory_usage() (NVML-backed on CUDA, None when
per-process usage is unavailable), record it in MemorySnapshot, and use
its delta as total_consumed when available, warning when the device-wide
delta disagrees. Without per-process data the behaviour is unchanged.

Co-authored-by: Claude <noreply@anthropic.com>
Signed-off-by: Tianyao Wu <rayroy31@gmail.com>
…s query

nvmlDeviceGetMemoryInfo is NVMLError_NotSupported on integrated parts such as
GB10 (DGX Spark, verified on the PR), where the per-process query is exactly what
still works. Say so in the docstring so nobody later routes this method through
the device-level call.

Signed-off-by: Tianyao Wu <rayroy31@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
…mory

The extensible KV cache is sized after warmup from the drop in free device
memory since startup, so memory another process allocates or releases in the
meantime still moves the budget. Use this process's own growth instead when the
platform reports it, as memory profiling does; the headroom cap stays
device-wide.

Co-authored-by: Claude <noreply@anthropic.com>
Signed-off-by: Tianyao Wu <rayroy31@gmail.com>
Rewritten by the check-docstring-style hook from vllm-project#65158.

Co-authored-by: Claude <noreply@anthropic.com>
Signed-off-by: Tianyao Wu <rayroy31@gmail.com>
test_cudagraph_memory_profile_runs_only_when_applied (vllm-project#64622) replaces
memory_profiling with a SimpleNamespace that lacks the field this PR adds.

Co-authored-by: Claude <noreply@anthropic.com>
Signed-off-by: Tianyao Wu <rayroy31@gmail.com>
@twu3202
twu3202 force-pushed the profiling-process-memory branch from 0dcbd1c to 67bf3c1 Compare November 9, 2026 02:12
@mergify mergify Bot removed the needs-rebase label Nov 9, 2026
@mergify

mergify Bot commented Nov 13, 2026

Copy link
Copy Markdown
Contributor

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @twu3202.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

@mergify mergify Bot added the needs-rebase label Nov 13, 2026
Keep both process-scoped usage and torch reservation growth in the final
KV sizing, with coverage for their interaction.

Signed-off-by: Tianyao Wu <rayroy31@gmail.com>
@mergify mergify Bot removed the needs-rebase label Nov 15, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

Status: No status

4 participants