Skip to content

feat(vllm): support generation-prefix cuts with multiple HTTP frontends - #5261

Open
macandro96 wants to merge 1 commit into
mainfrom
amahishi/multi-frontend-prefix-cuts
Open

macandro96 wants to merge 1 commit into
mainfrom
amahishi/multi-frontend-prefix-cuts

Conversation

@macandro96

@macandro96 macandro96 commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

What does this PR do?

Allow rollout_recovery.target_level: prefix with multiple vLLM HTTP frontends (vllm_cfg.http_api_server_count > 1, #5074). #5074 rejected this combination because a generation cut could reach only one frontend's calls.

Why

All frontends accept from one shared listening socket, and a model call's decoding state (token IDs, logprobs, capture registry, finished record) lives only in the frontend that accepted its request. Gym sends one cut request per backend per checkpoint, and the kernel hands it to one frontend. That frontend could cut only its own calls, so every other call would regenerate after a restore.

How

Topology. All frontends share one listening socket and one engine. Each also listens on a private loopback port, so a frontend can reach a specific sibling:

flowchart LR
    Gym["Gym policy model server"]
    Shared(["shared listening socket"])
    Gym -->|"one cut per backend per checkpoint"| Shared
    Shared -->|"kernel picks one,<br/>here the owner"| Owner
    Shared -.- C1
    Shared -.- C2
    subgraph Worker["vLLM worker: one EngineCore"]
        Owner["owner frontend<br/>loopback 127.0.0.1:A"]
        C1["child 1<br/>loopback 127.0.0.1:B"]
        C2["child 2<br/>loopback 127.0.0.1:C"]
    end
    Owner ==>|"POST claims, loopback"| C1
    Owner ==>|"POST claims, loopback"| C2
Loading

Whichever frontend the kernel hands Gym's request to becomes the coordinator and fans out to the others over loopback; any frontend can take that role.

One cut, with child 1 as the coordinator in this example:

sequenceDiagram
    participant Gym
    participant C1 as child 1 (coordinator)
    participant Own as owner
    participant C2 as child 2
    Gym->>C1: POST /ng-control/v1/generation-cut (inventory)
    par cut in parallel
        C1->>C1: cut_claims (its own calls)
    and
        C1->>Own: POST /generation-cut/claims (loopback)
        Own->>Own: cut_claims (its own calls)
        Own-->>C1: claim per call, or null
    and
        C1->>C2: POST /generation-cut/claims (loopback)
        C2->>C2: cut_claims (its own calls)
        C2-->>C1: claim per call, or null
    end
    C1->>C1: merge_cut_claims (running call wins, unclaimed call fails)
    C1-->>Gym: one GenerationCutReceipt
Loading
  1. Loopback control port per frontend. With more than one frontend, each one also binds 127.0.0.1:0 and serves the same app on it. After start-up, the owner collects every frontend's control URL (children return theirs from start) and sends the full list to all of them (peers).

  2. Claims-only cut. PrefixRecovery.cut_claims cuts only the inventory calls this frontend runs or has a finished record of. It returns None for the rest, with no warning and no failure count. It is served on a new /ng-control/v1/generation-cut/claims route, bearer-protected with the same control token.

  3. Coordinator merge. The frontend Gym reached runs its own claims and asks every peer at the same time, then merge_cut_claims builds the one receipt Gym validates:

    • a running call wins over a finished record of the same call, as within one frontend;
    • a call no frontend claims fails and regenerates;
    • a peer that errors or misses token_capture.control_timeout_s claims nothing, so its calls regenerate and the checkpoint never fails.

    Answers are cached per (checkpoint, inventory digest), so a retry that lands on another coordinator gets the same receipt.

  4. Other per-frontend state. Children receive the cut settings with capture. flush_obsolete_generation_cut_rows_async also flushes children (flush). Prefix counts are collected per frontend (get_generation_prefix_counts_per_frontend) and summed by the driver. Restores need no coordination: any frontend fetches the prefix from TQ.

A single frontend keeps the existing run_cut path. No Gym change, no new config keys.

Validation

Unit tests (all passing):

  • test_vllm_token_capture_hosting.py: 125 passed (gym lane).
  • test_vllm_captured_call_progress.py: 19 passed (vLLM lane).
  • test_vllm_multi_api.py: 9 passed.
  • Cut-route tests in test_vllm_generation.py: 13 passed.
  • New coverage:
    • two frontends cutting only their own calls into one valid receipt;
    • running call over finished record in either order;
    • the deadline path;
    • the coordinator fan-out;
    • a real loopback HTTP round trip, including a peer that fails or answers with the wrong length;
    • the child start / peers / flush / counts commands.
  • Ruff and pyrefly clean.

Functional: grpo_async_gym_single_controller_prefix_recovery.sh with ++policy.generation.vllm_cfg.http_api_server_count=2 ++async_rl.generation_fleet_health.enabled=false passed on H100:

  • 8 calls restored from their prefix;
  • generation logprobs bit-identical to the uninterrupted reference over 9,312 tokens;
  • invalid-row and poisoned-capture metrics 0.

Process-tagged logs confirmed both frontends cut their own calls in the same checkpoint (child 6–7 calls, owner 2), and both served restored calls (owner 5, child 3).

Benchmark (one run per setting): Qwen3-1.7B, one TP1 vLLM engine, one Gym backend, 32 prompts × 8 generations of long lists, a snapshot attempt every 3 s.

1 frontend 4 frontends
Cut wall time, ~100–250 live calls 0.50–1.72 s 0.28–0.32 s
gym_parked_seconds, four largest snapshots (average) 2.39 s 1.55 s
  • Speedup is below 4× because the shared socket spreads calls unevenly; the cut takes as long as the busiest frontend (e.g. 89 / 67 / 41 / 13 calls).
  • Peer round trips took 10–70 ms, almost all of it the peer's own cut.
  • Every frontend acted as coordinator at least once, and each cut sent exactly 3 peer requests.

Notes for reviewers

  • With several frontends, calls no frontend recognizes get one summary warning per cut instead of one warning per call.
  • The loopback port is opened whenever http_api_server_count > 1, even with prefix cuts off, because frontends start before setup_token_capture decides on cuts.

🤖 Generated with Claude Code

With http_api_server_count > 1, every frontend accepts from one shared
socket, and a model call's decoding state lives only in the frontend that
accepted its request. Gym's single cut call reached one frontend, which
could not cut the others' calls, so #5074 rejected prefix cuts there.

Each frontend now also listens on a loopback control port. The frontend
that receives Gym's cut asks every other one, over that port, to cut the
calls it serves (a new claims-only route), cuts its own in parallel, and
merges the claims into the one receipt Gym validates: a running call wins
over a finished record, and a call no frontend knows fails and regenerates.
A frontend that does not answer within the control timeout claims nothing.

The owner hands every frontend the others' control URLs after start-up,
forwards the cut settings to children with the capture command, and fans
out obsolete-row flushes and per-frontend prefix counts. A single frontend
keeps the existing cut path.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Anish Mahishi <amahishi@nvidia.com>
@copy-pr-bot

copy-pr-bot Bot commented Nov 17, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@github-actions github-actions Bot added the Documentation Improvements or additions to documentation label Nov 17, 2026
@macandro96

Copy link
Copy Markdown
Contributor Author

Superseded by #5267, the same change stacked on #5176 (prefix write pipeline and TQ writer actor). With the pipeline, each frontend runs its own pipeline and writer actor; claims-only cuts go through the pipeline as separate jobs. Leaving this open until we decide which lands.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant