Skip to content

feat(vllm): share one engine across multiple HTTP frontends - #5074

Merged
yuki-97 merged 21 commits into
mainfrom
yukih/vllm-multiple-frontends
Nov 14, 2026
Merged

yuki-97 merged 21 commits into
mainfrom
yukih/vllm-multiple-frontends

Conversation

@yuki-97

@yuki-97 yuki-97 commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

What does this PR do ?

Scale HTTP request handling and tokenization with multiple frontend processes sharing one vLLM engine. Single-frontend behavior remains the default and is unchanged.

What changed

1. Add shared-engine HTTP serving. Set vllm_cfg.http_api_server_count above one with async HTTP serving enabled. All frontends share one listening socket.

2. Synchronize frontend state. Broadcast token-capture setup and rollout weight versions to every frontend, and aggregate multiprocess Prometheus metrics.

Validation

1. Functional nightly. Validated grpo-qwen3-0.6b-1n8g-megatron-vllm-tp2-single-controller-gym-api8 on H100. All checks pass. This small-scale nightly validates functionality and shows no performance improvement over a single API server. The SC metric-window issue is addressed by #5155. Until #5155 is merged, the generation_tokens > 0 check is expected to fail without that fix. Validated multiple vLLM frontends together with that fix; the per-step generation_tokens > 0 check passes.
image

2. Offline performance. Offline validation shows a significant improvement at high concurrency. Details are in the offline thread.

Ruff checks and formatting pass.

@copy-pr-bot

copy-pr-bot Bot commented Nov 4, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@yuki-97 yuki-97 added the CI:Lfast Runs a fast test suite and re-use nightly `main` container (but sync dependencies to PRs version) label Nov 4, 2026
@yuki-97
yuki-97 added this pull request to stack #5076 November 4, 2026 09:03
@yuki-97
yuki-97 force-pushed the yukih/vllm-multiple-frontends branch 2 times, most recently from cf2b716 to e092a90 Compare November 10, 2026 09:41

@yuki-97 yuki-97 left a comment •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Self-review via /review-pr-team: 3 collaborating agents covering seven roles (RL analysis, bug finding, design, test coverage, existing comments, independent devil’s advocate and comment editor), plus leader verification.

Verdict — 9 inline comments: cached-frame video initialization (C1), CI test selection (C2), metric collection (C3, C4, C5), feature documentation (C9), API docstrings (C10), an unused startup value (C11) and failure cleanup (C12). C1 needs a fix before using this mode with cached-frame video inputs; the other comments are suggestions.

Dropped by the author — C6 (additional count type validation), C7 (optional-field defaults), and C8 (public numerical performance evidence).

Verified clean

  • Engine ownership — Frontends share one GPU engine and listening socket; client shutdown remains separate from engine-manager shutdown.
  • Single frontend — A count of one retains the existing construction path.
  • Control ordering — Successful broadcasts are serialized and wait for every child acknowledgement.
  • Multimodal cache — The multi-frontend path carries processed tensors without relying on an engine-side receiver cache.

Validation — Static inspection and AST checks completed. Whitespace, Ruff and Taplo hooks passed; pyrefly and configuration minimization could not execute because Python 3.13.14 was unavailable. Nightly dry-run accounting fits the 5241 GPU-hour cap. No pytest, collection, import smoke test or GPU run was performed on this host.

Generated by Codex (#4999 version)

@github-actions github-actions Bot added the Documentation Improvements or additions to documentation label Nov 10, 2026
@yuki-97

yuki-97 commented Nov 10, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test 532e7cd

@yuki-97
yuki-97 force-pushed the yukih/vllm-multiple-frontends branch from 532e7cd to 9b96dc4 Compare November 10, 2026 14:58
@yuki-97
yuki-97 marked this pull request as ready for review November 10, 2026 16:35
@yuki-97

yuki-97 commented Nov 10, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test 908f05f

@svcnvidia-nemo-ci svcnvidia-nemo-ci left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The static review found two correctness issues: the eager HTTP startup path now ignores the configured port range (C13), and the frontend-count validator accepts non-integer values (C14). C15–C17 cover telemetry, configuration defaults, and performance evidence.

The current source addresses the earlier video-loader, test-selection, progress, snapshot, documentation, and command-cleanup findings.

I inspected all 24 changed files and their directly affected paths across three review topics. One child completed without a usable findings report, so I completed its pool-lifecycle inspection from source. No tests, lint checks, configuration loading, or GPU runs were executed in this read-only environment. CI results, the linked image, and the offline benchmark thread weren't independently verified.

Generated by Codex

Note: formal approval is not available for this request, so this review is posted as a comment only.

Review generated by codex. This was a full review. Request a full review with /review reset=true.

@ZhiyuLi-Nvidia ZhiyuLi-Nvidia left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No blocking bug found. 3 low-severity comments: a failed control command leaves a dead pool that looks healthy, the DP=1/IPC rules are checked late, and the child host is built with __new__.

@ZhiyuLi-Nvidia ZhiyuLi-Nvidia left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One low-severity question on the new /metrics route.

ZhiyuLi-Nvidia
ZhiyuLi-Nvidia previously approved these changes Nov 11, 2026

@ZhiyuLi-Nvidia ZhiyuLi-Nvidia left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Just some nits.

@yuki-97

yuki-97 commented Nov 13, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test 4b21722

@yuki-97

yuki-97 commented Nov 13, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test 2f507d6

ZhiyuLi-Nvidia
ZhiyuLi-Nvidia previously approved these changes Nov 13, 2026

@ZhiyuLi-Nvidia ZhiyuLi-Nvidia left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

Signed-off-by: Yuki Huang <yukih@nvidia.com>
Signed-off-by: Yuki Huang <yukih@nvidia.com>
Signed-off-by: Yuki Huang <yukih@nvidia.com>
Signed-off-by: Yuki Huang <yukih@nvidia.com>
Signed-off-by: Yuki Huang <yukih@nvidia.com>
Signed-off-by: Yuki Huang <yukih@nvidia.com>
Signed-off-by: Yuki Huang <yukih@nvidia.com>
Signed-off-by: Yuki Huang <yukih@nvidia.com>
Signed-off-by: Yuki Huang <yukih@nvidia.com>
Signed-off-by: Yuki Huang <yukih@nvidia.com>
Signed-off-by: Yuki Huang <yukih@nvidia.com>
Signed-off-by: Yuki Huang <yukih@nvidia.com>
Signed-off-by: Yuki Huang <yukih@nvidia.com>
@yuki-97
yuki-97 force-pushed the yukih/vllm-multiple-frontends branch from 2f507d6 to 1ff2769 Compare November 13, 2026 12:44
Signed-off-by: Yuki Huang <yukih@nvidia.com>
@yuki-97

yuki-97 commented Nov 13, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test b7227ec

ZhiyuLi-Nvidia
ZhiyuLi-Nvidia previously approved these changes Nov 13, 2026
Signed-off-by: Yuki Huang <yukih@nvidia.com>
@yuki-97

yuki-97 commented Nov 14, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test a05d0c1

@yuki-97 yuki-97 added CI:L0 Run doctests and unit tests and removed CI:Lfast Runs a fast test suite and re-use nightly `main` container (but sync dependencies to PRs version) labels Nov 14, 2026
Signed-off-by: Yuki Huang <yukih@nvidia.com>
@yuki-97

yuki-97 commented Nov 14, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test 97a0d80

@yuki-97
yuki-97 enabled stacked auto-merge (squash) November 14, 2026 09:48
@yuki-97
yuki-97 merged commit effdd9b into main Nov 14, 2026
133 of 137 checks passed
@yuki-97
yuki-97 deleted the yukih/vllm-multiple-frontends branch November 14, 2026 09:54
jthomson04 added a commit to jthomson04/RL that referenced this pull request Nov 15, 2026
vllm_cfg.http_api_server_count (NVIDIA-NeMo#5074) adds HTTP frontend processes to each
engine's native server. With policy.generation.dynamo=true NeMo-Gym's calls
go through the Dynamo frontend, so those processes would serve none of them.
Startup now rejects the combination and points at
dynamo_cfg.frontend_args.render_workers. The native path is unchanged.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CI:L0 Run doctests and unit tests Documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants