Repository navigation
[ROCm][CI] Keep startup profiling from aborting when free memory grows - #53591
AndreasKaratzas merged 5 commits into
Conversation
The worker sizes the KV cache from the drop in free device memory across startup and aborts when that difference comes out negative. On a shared device it can, when something else releases memory inside that window, which is what flakes the MI300 RayExecutorV2 and MI355 V1 Sample + Logits groups. A negative difference says nothing about this worker, so on ROCm size the cache from what torch reserved for this process instead, which no other process on the device can influence, and log a warning. Off ROCm the existing assertion is left untouched. Signed-off-by: Stefan Koncarevic <stefan.koncarevic@amd.com>
f228452 to
0d420b7
Compare
|
/ci run |
|
✅ Triggered Buildkite CI #85352 for commit |
|
/ci run |
|
✅ CI is already running for this commit: https://buildkite.com/vllm/ci/builds/85352 |
AndreasKaratzas
left a comment
There was a problem hiding this comment.
I don't understand the numbers here to be honest, there are a lot of arbitrary numbers.
|
|
||
| # Memory accounting of the profiling run (Worker.determine_available_memory). | ||
|
|
||
| FREE_BEFORE = 188 * GiB_bytes |
There was a problem hiding this comment.
Isn't that only for mi300 ?
There was a problem hiding this comment.
You are right, the 188 GiB came straight from an MI300 log. The check only reads the sign of the measured drop and what torch reserved for this process, and the free memory readings only show up in the warning, so I named every amount for what it stands for and none of them reads as a device property now.
|
✅ @stefankoncarevic, CI is now available for this PR.
|
The amounts were scaffolding, but taking the free memory level from an MI300 log made it read as a device property. Name each amount for the role it plays, since the fallback only reads the sign of the measured drop and this process's torch reservation. Signed-off-by: Stefan Koncarevic <stefan.koncarevic@amd.com>
|
/ci run |
1 similar comment
|
/ci run |
|
✅ Triggered Buildkite CI #85783 for commit |
|
/ci run |
|
✅ Triggered Buildkite CI #85823 for commit |
With documentation moved to main, codex/deepseek-v4-startup-handover is byte-identical to codex/fastsafetensors-parallel-mtp-share, so record it as redundant rather than leaving the older "do not deploy both" note. Also record that the startup branch's tests/ changes no longer apply to current upstream, which edited tests/v1/worker/test_gpu_worker.py in vllm-project#53591 after this branch's base. The vllm/ code still applies clean, so extraction and the composability check should both scope to -- vllm/ csrc/ rather than taking a whole-branch diff. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Denis <mardel.home@gmail.com>
vllm-project#53591) Signed-off-by: Stefan Koncarevic <stefan.koncarevic@amd.com>
vllm-project#53591) Signed-off-by: Stefan Koncarevic <stefan.koncarevic@amd.com>
vllm-project#53591) Signed-off-by: Stefan Koncarevic <stefan.koncarevic@amd.com> (cherry picked from commit 9236159)
vllm-project#53591) Signed-off-by: Stefan Koncarevic <stefan.koncarevic@amd.com>
Purpose
Two AMD CI groups abort engine startup with this error and pass when the job is retried:
The KV cache is sized from how much free device memory dropped between a snapshot taken when the
worker starts and one taken after profiling. If anything else on the device releases memory inside
that window, the difference comes out negative and the worker refuses to start. That is what happens
in mi300_4 RayExecutorV2 and in mi355_1 V1 Sample + Logits, where two engines share one device.
A negative difference says nothing about this worker. On ROCm the cache is sized from what torch
reserved for this process instead, which no other process can influence, and a warning is logged. That
number can come out slightly low, so the cache ends up marginally larger, which I prefer to a worker
that will not start. Off ROCm nothing changes.
Test Plan
A helper process releases memory inside the profiling window, which reproduces both failures on
demand. Both tests run with and without it, both groups run end to end, and unit tests in
tests/v1/worker/test_gpu_worker.pycover the branches.Test Result
Both tests fail before the change and pass after it, and are unaffected without the helper. Both
groups pass end to end and no run reports a profiling error. Unit tests: 7 passed.
Essential Elements of an Effective PR Description Checklist
supported_models.mdandexamplesfor a new model.