Skip to content

[ROCm][CI] Keep startup profiling from aborting when free memory grows - #53591

Merged
AndreasKaratzas merged 5 commits into
vllm-project:mainfrom
stefankoncarevic:rocm-profiling-free-memory-race
Aug 28, 2026
Merged

AndreasKaratzas merged 5 commits into
vllm-project:mainfrom
stefankoncarevic:rocm-profiling-free-memory-race

Conversation

@stefankoncarevic

@stefankoncarevic stefankoncarevic commented Aug 24, 2026 •

Copy link
Copy Markdown
Collaborator

Purpose

Two AMD CI groups abort engine startup with this error and pass when the job is retried:

Error in memory profiling. Initial free memory 188.29 GiB, current free memory 188.75 GiB.

The KV cache is sized from how much free device memory dropped between a snapshot taken when the
worker starts and one taken after profiling. If anything else on the device releases memory inside
that window, the difference comes out negative and the worker refuses to start. That is what happens
in mi300_4 RayExecutorV2 and in mi355_1 V1 Sample + Logits, where two engines share one device.

A negative difference says nothing about this worker. On ROCm the cache is sized from what torch
reserved for this process instead, which no other process can influence, and a warning is logged. That
number can come out slightly low, so the cache ends up marginally larger, which I prefer to a worker
that will not start. Off ROCm nothing changes.

Test Plan

A helper process releases memory inside the profiling window, which reproduces both failures on
demand. Both tests run with and without it, both groups run end to end, and unit tests in
tests/v1/worker/test_gpu_worker.py cover the branches.

Test Result

Both tests fail before the change and pass after it, and are unaffected without the helper. Both
groups pass end to end and no run reports a profiling error. Unit tests: 7 passed.


Essential Elements of an Effective PR Description Checklist
  • The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
  • The test plan, such as providing test command.
  • The test results, such as pasting the results comparison before and after, or e2e results
  • (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

@mergify mergify Bot added the rocm Related to AMD ROCm label Aug 24, 2026
@github-project-automation github-project-automation Bot moved this to Todo in AMD Aug 24, 2026
The worker sizes the KV cache from the drop in free device memory across
startup and aborts when that difference comes out negative. On a shared
device it can, when something else releases memory inside that window,
which is what flakes the MI300 RayExecutorV2 and MI355 V1 Sample + Logits
groups.

A negative difference says nothing about this worker, so on ROCm size the
cache from what torch reserved for this process instead, which no other
process on the device can influence, and log a warning. Off ROCm the
existing assertion is left untouched.

Signed-off-by: Stefan Koncarevic <stefan.koncarevic@amd.com>
@stefankoncarevic
stefankoncarevic force-pushed the rocm-profiling-free-memory-race branch from f228452 to 0d420b7 Compare August 24, 2026 15:36
@AndreasKaratzas

Copy link
Copy Markdown
Member

/ci run

@github-actions

Copy link
Copy Markdown

✅ Triggered Buildkite CI #85352 for commit 0d420b745904.

@AndreasKaratzas

Copy link
Copy Markdown
Member

/ci run

@github-actions

Copy link
Copy Markdown

✅ CI is already running for this commit: https://buildkite.com/vllm/ci/builds/85352

@AndreasKaratzas AndreasKaratzas left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't understand the numbers here to be honest, there are a lot of arbitrary numbers.

Comment thread tests/v1/worker/test_gpu_worker.py Outdated

# Memory accounting of the profiling run (Worker.determine_available_memory).

FREE_BEFORE = 188 * GiB_bytes

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Isn't that only for mi300 ?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You are right, the 188 GiB came straight from an MI300 log. The check only reads the sign of the measured drop and what torch reserved for this process, and the free memory readings only show up in the warning, so I named every amount for what it stands for and none of them reads as a device property now.

@AndreasKaratzas AndreasKaratzas added the ready ONLY add when PR is ready to merge/full CI is needed label Aug 26, 2026
@github-actions

Copy link
Copy Markdown

✅ @stefankoncarevic, CI is now available for this PR.

  • /ci run starts upstream CI; /amd-ci run starts AMD CI only.
  • /ci retry retries failed jobs in the CI build for the current PR head. If the current head has no CI build, it starts a new CI build for the current head containing only jobs that failed in the latest earlier CI build for this PR.
  • /amd-ci retry retries failed jobs in AMD CI for the current PR head. Use /amd-ci run when the current head has no AMD CI build.
  • /ci cancel cancels scheduled or running CI builds for this PR branch; /amd-ci cancel does the same for AMD CI only.

The amounts were scaffolding, but taking the free memory level from an
MI300 log made it read as a device property. Name each amount for the
role it plays, since the fallback only reads the sign of the measured
drop and this process's torch reservation.

Signed-off-by: Stefan Koncarevic <stefan.koncarevic@amd.com>
@stefankoncarevic

Copy link
Copy Markdown
Collaborator Author

/ci run

1 similar comment
@AndreasKaratzas

Copy link
Copy Markdown
Member

/ci run

@github-actions

Copy link
Copy Markdown

✅ Triggered Buildkite CI #85783 for commit 45cc03dd285d.

@stefankoncarevic

Copy link
Copy Markdown
Collaborator Author

/ci run

@github-actions

Copy link
Copy Markdown

✅ Triggered Buildkite CI #85823 for commit c20cf521594b.

@AndreasKaratzas
AndreasKaratzas merged commit 9236159 into vllm-project:main Aug 28, 2026
101 checks passed
@github-project-automation github-project-automation Bot moved this from Todo to Done in AMD Aug 28, 2026
mardelden added a commit to mardelden/vllm-custom that referenced this pull request Aug 31, 2026
With documentation moved to main, codex/deepseek-v4-startup-handover is byte-identical to codex/fastsafetensors-parallel-mtp-share, so record it as redundant rather than leaving the older "do not deploy both" note.

Also record that the startup branch's tests/ changes no longer apply to current upstream, which edited tests/v1/worker/test_gpu_worker.py in vllm-project#53591 after this branch's base. The vllm/ code still applies clean, so extraction and the composability check should both scope to -- vllm/ csrc/ rather than taking a whole-branch diff.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Signed-off-by: Denis <mardel.home@gmail.com>
am-cohere pushed a commit to am-cohere/vllm that referenced this pull request Sep 1, 2026
vllm-project#53591)

Signed-off-by: Stefan Koncarevic <stefan.koncarevic@amd.com>
mylibrar pushed a commit to tanyuqian/vllm that referenced this pull request Sep 3, 2026
vllm-project#53591)

Signed-off-by: Stefan Koncarevic <stefan.koncarevic@amd.com>
D-G-Dimitrov pushed a commit to D-G-Dimitrov/vllm that referenced this pull request Sep 4, 2026
vllm-project#53591)

Signed-off-by: Stefan Koncarevic <stefan.koncarevic@amd.com>
(cherry picked from commit 9236159)
michael-gpu pushed a commit to practical-inference/vllm-ampere-extended that referenced this pull request Oct 20, 2026
vllm-project#53591)

Signed-off-by: Stefan Koncarevic <stefan.koncarevic@amd.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready ONLY add when PR is ready to merge/full CI is needed rocm Related to AMD ROCm

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

2 participants