Repository navigation
feat(vllm): share one engine across multiple HTTP frontends - #5074
Conversation
cf2b716 to
e092a90
Compare
There was a problem hiding this comment.
Self-review via /review-pr-team: 3 collaborating agents covering seven roles (RL analysis, bug finding, design, test coverage, existing comments, independent devil’s advocate and comment editor), plus leader verification.
Verdict — 9 inline comments: cached-frame video initialization (C1), CI test selection (C2), metric collection (C3, C4, C5), feature documentation (C9), API docstrings (C10), an unused startup value (C11) and failure cleanup (C12). C1 needs a fix before using this mode with cached-frame video inputs; the other comments are suggestions.
Dropped by the author — C6 (additional count type validation), C7 (optional-field defaults), and C8 (public numerical performance evidence).
Verified clean
- Engine ownership — Frontends share one GPU engine and listening socket; client shutdown remains separate from engine-manager shutdown.
- Single frontend — A count of one retains the existing construction path.
- Control ordering — Successful broadcasts are serialized and wait for every child acknowledgement.
- Multimodal cache — The multi-frontend path carries processed tensors without relying on an engine-side receiver cache.
Validation — Static inspection and AST checks completed. Whitespace, Ruff and Taplo hooks passed; pyrefly and configuration minimization could not execute because Python 3.13.14 was unavailable. Nightly dry-run accounting fits the 5241 GPU-hour cap. No pytest, collection, import smoke test or GPU run was performed on this host.
Generated by Codex (#4999 version)
|
/ok to test 532e7cd |
532e7cd to
9b96dc4
Compare
|
/ok to test 908f05f |
svcnvidia-nemo-ci
left a comment
There was a problem hiding this comment.
The static review found two correctness issues: the eager HTTP startup path now ignores the configured port range (C13), and the frontend-count validator accepts non-integer values (C14). C15–C17 cover telemetry, configuration defaults, and performance evidence.
The current source addresses the earlier video-loader, test-selection, progress, snapshot, documentation, and command-cleanup findings.
I inspected all 24 changed files and their directly affected paths across three review topics. One child completed without a usable findings report, so I completed its pool-lifecycle inspection from source. No tests, lint checks, configuration loading, or GPU runs were executed in this read-only environment. CI results, the linked image, and the offline benchmark thread weren't independently verified.
Generated by Codex
Note: formal approval is not available for this request, so this review is posted as a comment only.
Review generated by codex. This was a full review. Request a full review with /review reset=true.
ZhiyuLi-Nvidia
left a comment
There was a problem hiding this comment.
No blocking bug found. 3 low-severity comments: a failed control command leaves a dead pool that looks healthy, the DP=1/IPC rules are checked late, and the child host is built with __new__.
ZhiyuLi-Nvidia
left a comment
There was a problem hiding this comment.
One low-severity question on the new /metrics route.
ZhiyuLi-Nvidia
left a comment
There was a problem hiding this comment.
LGTM! Just some nits.
908f05f to
241628b
Compare
|
/ok to test 4b21722 |
|
/ok to test 2f507d6 |
Signed-off-by: Yuki Huang <yukih@nvidia.com>
Signed-off-by: Yuki Huang <yukih@nvidia.com>
Signed-off-by: Yuki Huang <yukih@nvidia.com>
Signed-off-by: Yuki Huang <yukih@nvidia.com>
Signed-off-by: Yuki Huang <yukih@nvidia.com>
Signed-off-by: Yuki Huang <yukih@nvidia.com>
Signed-off-by: Yuki Huang <yukih@nvidia.com>
Signed-off-by: Yuki Huang <yukih@nvidia.com>
Signed-off-by: Yuki Huang <yukih@nvidia.com>
Signed-off-by: Yuki Huang <yukih@nvidia.com>
Signed-off-by: Yuki Huang <yukih@nvidia.com>
Signed-off-by: Yuki Huang <yukih@nvidia.com>
Signed-off-by: Yuki Huang <yukih@nvidia.com>
Signed-off-by: Yuki Huang <yukih@nvidia.com>
Signed-off-by: Yuki Huang <yukih@nvidia.com>
2f507d6 to
1ff2769
Compare
Signed-off-by: Yuki Huang <yukih@nvidia.com>
|
/ok to test b7227ec |
Signed-off-by: Yuki Huang <yukih@nvidia.com>
|
/ok to test a05d0c1 |
Signed-off-by: Yuki Huang <yukih@nvidia.com>
|
/ok to test 97a0d80 |
vllm_cfg.http_api_server_count (NVIDIA-NeMo#5074) adds HTTP frontend processes to each engine's native server. With policy.generation.dynamo=true NeMo-Gym's calls go through the Dynamo frontend, so those processes would serve none of them. Startup now rejects the combination and points at dynamo_cfg.frontend_args.render_workers. The native path is unchanged. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
What does this PR do ?
Scale HTTP request handling and tokenization with multiple frontend processes sharing one vLLM engine. Single-frontend behavior remains the default and is unchanged.
What changed
1. Add shared-engine HTTP serving. Set
vllm_cfg.http_api_server_countabove one with async HTTP serving enabled. All frontends share one listening socket.2. Synchronize frontend state. Broadcast token-capture setup and rollout weight versions to every frontend, and aggregate multiprocess Prometheus metrics.
Validation
1. Functional nightly. Validated

grpo-qwen3-0.6b-1n8g-megatron-vllm-tp2-single-controller-gym-api8on H100. All checks pass. This small-scale nightly validates functionality and shows no performance improvement over a single API server. The SC metric-window issue is addressed by #5155. Until #5155 is merged, thegeneration_tokens > 0check is expected to fail without that fix. Validated multiple vLLM frontends together with that fix; the per-stepgeneration_tokens > 0check passes.2. Offline performance. Offline validation shows a significant improvement at high concurrency. Details are in the offline thread.
Ruff checks and formatting pass.