Repository navigation
feat(vllm): support generation-prefix cuts with multiple HTTP frontends - #5261
Open
macandro96 wants to merge 1 commit into
Open
macandro96 wants to merge 1 commit into
macandro96 wants to merge 1 commit into
Conversation
With http_api_server_count > 1, every frontend accepts from one shared socket, and a model call's decoding state lives only in the frontend that accepted its request. Gym's single cut call reached one frontend, which could not cut the others' calls, so #5074 rejected prefix cuts there. Each frontend now also listens on a loopback control port. The frontend that receives Gym's cut asks every other one, over that port, to cut the calls it serves (a new claims-only route), cuts its own in parallel, and merges the claims into the one receipt Gym validates: a running call wins over a finished record, and a call no frontend knows fails and regenerates. A frontend that does not answer within the control timeout claims nothing. The owner hands every frontend the others' control URLs after start-up, forwards the cut settings to children with the capture command, and fans out obsolete-row flushes and per-frontend prefix counts. A single frontend keeps the existing cut path. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Anish Mahishi <amahishi@nvidia.com>
Contributor
Author
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Allow
rollout_recovery.target_level: prefixwith multiple vLLM HTTP frontends (vllm_cfg.http_api_server_count > 1, #5074). #5074 rejected this combination because a generation cut could reach only one frontend's calls.Why
All frontends accept from one shared listening socket, and a model call's decoding state (token IDs, logprobs, capture registry, finished record) lives only in the frontend that accepted its request. Gym sends one cut request per backend per checkpoint, and the kernel hands it to one frontend. That frontend could cut only its own calls, so every other call would regenerate after a restore.
How
Topology. All frontends share one listening socket and one engine. Each also listens on a private loopback port, so a frontend can reach a specific sibling:
flowchart LR Gym["Gym policy model server"] Shared(["shared listening socket"]) Gym -->|"one cut per backend per checkpoint"| Shared Shared -->|"kernel picks one,<br/>here the owner"| Owner Shared -.- C1 Shared -.- C2 subgraph Worker["vLLM worker: one EngineCore"] Owner["owner frontend<br/>loopback 127.0.0.1:A"] C1["child 1<br/>loopback 127.0.0.1:B"] C2["child 2<br/>loopback 127.0.0.1:C"] end Owner ==>|"POST claims, loopback"| C1 Owner ==>|"POST claims, loopback"| C2Whichever frontend the kernel hands Gym's request to becomes the coordinator and fans out to the others over loopback; any frontend can take that role.
One cut, with child 1 as the coordinator in this example:
sequenceDiagram participant Gym participant C1 as child 1 (coordinator) participant Own as owner participant C2 as child 2 Gym->>C1: POST /ng-control/v1/generation-cut (inventory) par cut in parallel C1->>C1: cut_claims (its own calls) and C1->>Own: POST /generation-cut/claims (loopback) Own->>Own: cut_claims (its own calls) Own-->>C1: claim per call, or null and C1->>C2: POST /generation-cut/claims (loopback) C2->>C2: cut_claims (its own calls) C2-->>C1: claim per call, or null end C1->>C1: merge_cut_claims (running call wins, unclaimed call fails) C1-->>Gym: one GenerationCutReceiptLoopback control port per frontend. With more than one frontend, each one also binds
127.0.0.1:0and serves the same app on it. After start-up, the owner collects every frontend's control URL (children return theirs fromstart) and sends the full list to all of them (peers).Claims-only cut.
PrefixRecovery.cut_claimscuts only the inventory calls this frontend runs or has a finished record of. It returnsNonefor the rest, with no warning and no failure count. It is served on a new/ng-control/v1/generation-cut/claimsroute, bearer-protected with the same control token.Coordinator merge. The frontend Gym reached runs its own claims and asks every peer at the same time, then
merge_cut_claimsbuilds the one receipt Gym validates:token_capture.control_timeout_sclaims nothing, so its calls regenerate and the checkpoint never fails.Answers are cached per (checkpoint, inventory digest), so a retry that lands on another coordinator gets the same receipt.
Other per-frontend state. Children receive the cut settings with
capture.flush_obsolete_generation_cut_rows_asyncalso flushes children (flush). Prefix counts are collected per frontend (get_generation_prefix_counts_per_frontend) and summed by the driver. Restores need no coordination: any frontend fetches the prefix from TQ.A single frontend keeps the existing
run_cutpath. No Gym change, no new config keys.Validation
Unit tests (all passing):
test_vllm_token_capture_hosting.py: 125 passed (gym lane).test_vllm_captured_call_progress.py: 19 passed (vLLM lane).test_vllm_multi_api.py: 9 passed.test_vllm_generation.py: 13 passed.start/peers/flush/countscommands.Functional:
grpo_async_gym_single_controller_prefix_recovery.shwith++policy.generation.vllm_cfg.http_api_server_count=2 ++async_rl.generation_fleet_health.enabled=falsepassed on H100:Process-tagged logs confirmed both frontends cut their own calls in the same checkpoint (child 6–7 calls, owner 2), and both served restored calls (owner 5, child 3).
Benchmark (one run per setting): Qwen3-1.7B, one TP1 vLLM engine, one Gym backend, 32 prompts × 8 generations of long lists, a snapshot attempt every 3 s.
gym_parked_seconds, four largest snapshots (average)Notes for reviewers
http_api_server_count > 1, even with prefix cuts off, because frontends start beforesetup_token_capturedecides on cuts.🤖 Generated with Claude Code