Skip to content

feat(sglang-sidecar): serve a node's Rust listeners from one sidecar - #17610

Open
jain-ria wants to merge 17 commits into
jain-ria/sglang-sidecar-api-v1-launchfrom
jain-ria/sglang-sidecar-api-v1-dp
Open

jain-ria wants to merge 17 commits into
jain-ria/sglang-sidecar-api-v1-launchfrom
jain-ria/sglang-sidecar-api-v1-dp

Conversation

@jain-ria

@jain-ria jain-ria commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

Overview

SGLang's Rust server opens one listener (an HTTP and gRPC endpoint) for each data-parallel (DP) rank. The sidecar connected to only one listener, so it refused DP>1. Now one sidecar per node connects to all of that node's listeners and registers them as one Dynamo worker that owns their DP ranks. Dynamo can then serve SGLang attention DP (--attn-dp-size), and replica DP (--dp-size) on aggregated and decode workers. DP=1 serves as before, apart from the request IDs SGLang sees (see User-facing).

Key Changes

Core logic: about 800 added lines (about 750 in lib/sidecar/sglang, 55 in the SGLang Mocker), excluding tests, comments and blank lines. Most of it is the node's listener table, its checks, per-listener startup and per-request dispatch. Nothing changes in lib/llm, lib/sidecar/common or the proto.

  • node.rs: reads the node's listeners from DeploymentInfo.node_listeners and registers one worker that owns their DP ranks (for example ranks 2-3 of 4). The worker reports the smallest KV capacity of its listeners, and startup fails unless every listener publishes KV events or none does.
  • listeners.rs: picks the listener for each request, and makes an abort cancel only that request.
    • A request runs on the listener of the DP rank it names: the KV router's choice, or a rank the client pinned (nvext.dp_rank or x-dynamo-dp-rank with a worker ID). Requests that name no rank (round-robin, random or direct routing, and prefill without KV routing) take the node's listeners in turn. A rank outside the node is InvalidArgument before a stream opens.
    • Each time the sidecar sends a request to SGLang, it gives it its own SGLang request ID (rid), <request id>-<8 hex digits>, and an abort cancels only that rid, on the listener running it. Dynamo retries a migrated request under the same request ID, and SGLang's Abort cancels every request with a given rid, so aborting by request ID could also kill a healthy retry.
    • Each listener has its own GetModelInfo poller. If any listener is lost, the node is lost: the worker is withdrawn and the sidecar exits with one error naming the rank, "SGLang engine lost: listener for DP rank r at : ...".
  • protocol.rs: when the deployment's total DP size is above 1, prefill no longer registers one fixed bootstrap port, the port decode pulls KV through (engine.rs). Instead, prefill's first response carries the bootstrap port of the listener that ran it, and a bootstrap room chosen so that room % dp_size is that listener's rank, which is how SGLang finds the rank. Decode gets both through the frontend's existing first-response handoff. Prefill names its rank to SGLang only when the request named one.

Also:

User-facing: SGLang DP>1 starts instead of failing discovery. SGLang without node_listeners (v0.5.23) still serves DP=1. SGLang sees gRPC requests under <request id>-<8 hex digits>, as it already did for native /generate requests. A prefill sidecar on a node other than node 0 advertises its own address for PD bootstrap, not node 0's --dist-init-addr. No CLI flag, environment variable, proto or frontend change.

Performance: per request, a range check or a round-robin counter picks the listener. Each listener gets its own --grpc-connections pool, so a DP=8 node opens 8 times as many (256 at the default 32). Each listener also gets one GetModelInfo poll a second. An abort is one RPC, to one listener. When the total DP size is above 1, the frontend dispatches decode after prefill's first response instead of alongside it. DP=1 runs as before. Not benchmarked.

Validation

Case Before Now Test
Node serving DP ranks [2, 4) of 4 Refused at discovery One worker for ranks 2-3 with the least KV capacity; prefill registers no bootstrap port node_registration_covers_every_listener
Rank gap or out of range; no listener table at DP>1; KV events on some ranks only Refused at discovery Startup fails and nothing registers rejects_unusable_tables, kv_union_requires_every_rank_or_none
Request names rank r, or names none Any r other than 0 refused Runs on r's listener, and r outside the node is InvalidArgument; with no rank, requests take the listeners in turn routing_rank_selects_listener
Native /generate body with a disagg_prefill_dp_rank from the request Passed to SGLang as is Prefill: the dispatched rank's port and room, and the rank only when the request named it. Aggregated: dropped native_bodies_name_the_dispatched_prefill_rank
Abort on a node of 2 listeners Refused at discovery gRPC and native: the request's own rid, on its own listener only, even after a cancelled stream has ended; nothing once the request is gone sglang_node_sidecar_aborts_on_the_owning_listener, sglang_abort_after_cancelled_stream_ends_reaches_its_dispatch
A retry with the same request ID runs beside the failed attempt gRPC: aborting the failed attempt also cancels the retry Each attempt has its own rid, and an abort names only that rid, on the listener running it repeated_public_ids_have_their_own_abort_targets
PD, round robin over two prefill nodes of 2 listeners (DP 4) Refused at discovery Prefills reach all four ranks; both legs carry the dispatched listener's port and a room with room % 4 == rank; a pinned rank 3 keeps its rank sglang_round_robin_prefill_hands_off_on_both_nodes
Listener 1 of 2 stops answering Refused at discovery The node is lost: the worker is withdrawn, then the sidecar exits 1, naming DP rank 1 sglang_node_engine_lost_names_the_lost_listener, sglang_node_listener_loss_withdraws_worker_and_fails_the_process
GetServerInfo returns UNAVAILABLE while ranks bind Startup fails Retried until the startup deadline; other errors fail at once startup_discovery_retries_only_unavailable
GPU, H200, DP>1, on this PR's first version (7102ad8c0b) Refused at discovery Pass: attention DP2 and replica DP2 with round robin and KV routing; PD at DP2 with round robin and KV routing, including prefill --load-balance-method round_robin; abort on both ranks; SIGTERM and kill -9; sidecar started before the engine; attention DP across 2 nodes Manual cluster run, SGLang with #46637, #45996 and #46181
GPU, 2x H800, attention DP2, 2026-11-05 at 5331f7b2f2 Refused at discovery Pass, with the test's skip removed, on SGLang main 493c857736 tests/serve/test_sidecar.py -k sglang_dep_kv_router
GPU, 4x GH200, PD at DP2 across 4 nodes, 2026-11-05 at 5331f7b2f2 Refused at discovery Pass: round robin 49/49 and KV routing 32/32 requests, also with prefill --load-balance-method round_robin; decode finds each prefill rank's bootstrap on that rank's own node Manual cluster run, SGLang main 493c857736
GPU, 1x RTX 6000 Ada, DP=1, 2026-11-05 at 5331f7b2f2 The 4 gpu_1 SGLang sidecar tests pass on stock v0.5.23 tests/serve/test_sidecar.py

At 4d0cf3352d, built against a local copy of sglang-api-types 0.1.1, on CPU: the 107 SGLang sidecar tests, the SGLang Mocker's 15 and dynamo-sglang-generate's 18, all 55 sidecar_mocker_integration and all 54 router_sidecar_mocker_integration tests (every backend), and the vLLM, TRT-LLM and LPX sidecar tests pass, and clippy is clean. These run in pre-merge's Cargo tests.

None of the GPU runs include the refreshed #17431, the round-robin change (cd4c498784) or the abort change (8c826eaaae), and per-listener engine loss has not run on a GPU. No CI job runs DP>1 on a real SGLang engine: CI's SGLang (v0.5.23) reports no node_listeners, so the 2-GPU sglang_dep_kv_router post-merge job cannot pass there; #17612 skips it until the pin includes #46637.

Issues and Ongoing Work

Pull Request Checklist
  • I used Dynamo's /pr-checklist skill.

🤖 Generated with Claude Code

SGLang's Rust server runs one listener per DP rank, on --port + k and
--grpc-port + k, and the sidecar, which served one listener, refused
DP>1. It now finds every listener of its node in
DeploymentInfo.node_listeners (sgl-project/sglang#46637) and serves
them as one worker that owns the node's DP ranks. Replica DP also
needs sgl-project/sglang#45996. The proto and the frontend are
unchanged.

- Startup connects to every listener under one deadline and registers
  one worker for their DP ranks. A build without the table still
  serves DP=1. Bootstrap discovery retries
  UNAVAILABLE, which GetServerInfo answers until every rank's listener
  is bound (sgl-project/sglang#46181).
- A request runs on the listener of the rank the router names; a rank
  outside the node is an invalid argument. Abort goes to every
  listener.
- At global DP>1 prefill registers no bootstrap port. Its first
  response carries the dispatched listener's bootstrap port and a room
  with room % dp_size == that listener's rank, so decode reaches the
  rank through the frontend's existing handoff. Prefill names its rank
  to SGLang only when the router named one. At DP=1 prefill still
  registers its port. On a node other than 0, prefill no longer
  advertises the rendezvous host.
- The SGLang Mocker library builds one node's listeners, which the
  testkit serves as one node with one listener table. The Mocker
  binary takes no new options.
- The sidecar README's support matrix lists attention DP and replica DP
  (not for prefill) and gives the listener port spacing.

Part of #17218

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: jain-ria <riajain@NVIDIA.com>
@jain-ria
jain-ria requested review from a team as code owners October 29, 2026 19:18
@github-actions github-actions Bot added documentation Improvements or additions to documentation feat labels Oct 29, 2026
@jain-ria jain-ria changed the title feat(sglang-sidecar): serve a node's Rust listeners from one sidecar feat(sglang-sidecar): serve all Rust listeners on a node from one sidecar Oct 29, 2026
@jain-ria
jain-ria added this pull request to stack #17369 October 29, 2026 19:24
Comment thread lib/sidecar/sglang/src/client.rs Resolved
Comment thread lib/sidecar/sglang/src/client.rs Resolved
Comment thread lib/sidecar/sglang/src/client.rs Resolved
Comment thread lib/sidecar/sglang/src/listeners.rs Resolved
Comment thread lib/sidecar/sglang/src/native_http.rs Resolved
Comment thread lib/sidecar/sglang/src/node.rs Resolved
@coderabbitai

coderabbitai Bot commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

Review in Change Stack →

No actionable comments were generated in the recent review.

Walkthrough

The SGLang Rust server and sidecar now represent each node as a set of data-parallel listeners. The sidecar validates listener metadata, starts and routes across listeners, and propagates prefill ranks through gRPC and native HTTP requests. Tests cover multi-listener routing and abort behavior.

Changes

SGLang node listeners

Layer / File(s) Summary
Mocker listener configuration and discovery
lib/mocker/servers/sglang/src/main.rs, lib/mocker/servers/sglang/src/server.rs, lib/sidecar/sglang/README.md
The Mocker configures listener ranks and port offsets, creates one service per listener, and reports listener and deployment metadata. The support matrix documents the listener layout and data-parallel modes.
Node listener metadata and aggregation
lib/sidecar/sglang/src/discovery.rs, lib/sidecar/sglang/src/node.rs, lib/sidecar/sglang/src/lib.rs
Discovery builds and validates a node listener table. Node information combines listener capacity and KV-event metadata and resolves listener endpoints.
Sidecar startup and listener dispatch
lib/sidecar/sglang/src/client.rs, lib/sidecar/sglang/src/engine.rs, lib/sidecar/sglang/src/listeners.rs
Startup connects to node listeners under a shared deadline and builds node-wide state. Requests select a listener, while aborts are sent to the listener set. Discovery retries transient connection failures.
Rank-aware rendezvous and request construction
lib/sidecar/sglang/src/protocol.rs, lib/sidecar/sglang/src/protocol/request_tests.rs, lib/sidecar/sglang/src/native_http.rs, lib/sidecar/sglang/src/args.rs
Request construction carries the selected listener target and optional prefill rank through gRPC, native HTTP, and handoff data. Prefill rooms are rank-congruent when a rank is supplied.
Multi-listener fixtures and integration coverage
lib/sidecar/testkit/tests/support/sglang.rs, lib/sidecar/testkit/tests/support/process.rs, lib/sidecar/testkit/tests/support/mod.rs, lib/sidecar/testkit/tests/*integration.rs, lib/sidecar/testkit/README.md
Fixtures start nodes with multiple listeners and record listener-level requests. Integration tests cover routing, rank and port propagation, and abort behavior. The testkit documentation describes the fixture and scenarios.

Estimated code review effort: 4 (Complex) | ~60 minutes

Merge Risk: Moderate · up to 7102a

With round-robin, random or direct routing, only the first DP rank on each node receives traffic, so the other ranks idle even though the node advertises their capacity. This is a throughput and capacity-accounting problem for DP>1 deployments, not a correctness failure. DP=1 and KV-routed deployments are unaffected. The published fix should be applied or the limitation accepted before merge.

Pre-merge checks: Passed 4

Check name Status Explanation
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Title check Passed The title names the main change: the SGLang sidecar now serves a node's Rust listeners (one per DP rank) through a single sidecar. The changeset matches this: node.rs, listeners.rs, and the protocol/e…
Description check Passed The description follows the template and covers the Overview, Key Changes, Validation, and Issues and Ongoing Work sections. It states the purpose: one sidecar per node serves all SGLang listeners as …
Full details: Title check

Explanation

The title names the main change: the SGLang sidecar now serves a node's Rust listeners (one per DP rank) through a single sidecar. The changeset matches this: node.rs, listeners.rs, and the protocol/engine changes all route requests per listener behind one worker. The title is specific, concise, and follows the conventional-commit format.

Full details: Description check

Explanation

The description follows the template and covers the Overview, Key Changes, Validation, and Issues and Ongoing Work sections. It states the purpose: one sidecar per node serves all SGLang listeners as one worker, enabling attention DP and replica DP while DP=1 stays as before. It lists the changed files and their behavior, the request routing and abort semantics, the user-facing and performance impacts, and a detailed validation table with test names and manual GPU runs. It also names the SGLang dependencies (sgl-project/sglang#46637, #45996, #46181), the CI limitation, and the stacked-PR order. The checklist item is checked. The issue link is present as "Part of #17218" rather than "Closes #XXXX", which is acceptable because the template's link check is advisory today. The description is complete and specific, so it passes.

  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Recent review info

Run configuration

Commits

Reviewing files that changed from the base of the PR and between 466887c and 7102ad8.

Files selected for processing: 19

  • lib/mocker/servers/sglang/src/main.rs
  • lib/mocker/servers/sglang/src/server.rs
  • lib/sidecar/sglang/README.md
  • lib/sidecar/sglang/src/args.rs
  • lib/sidecar/sglang/src/client.rs
  • lib/sidecar/sglang/src/discovery.rs
  • lib/sidecar/sglang/src/engine.rs
  • lib/sidecar/sglang/src/lib.rs
  • lib/sidecar/sglang/src/listeners.rs
  • lib/sidecar/sglang/src/native_http.rs
  • lib/sidecar/sglang/src/node.rs
  • lib/sidecar/sglang/src/protocol.rs
  • lib/sidecar/sglang/src/protocol/request_tests.rs
  • lib/sidecar/testkit/README.md
  • lib/sidecar/testkit/tests/router_sidecar_mocker_integration.rs
  • lib/sidecar/testkit/tests/sidecar_mocker_integration.rs
  • lib/sidecar/testkit/tests/support/mod.rs
  • lib/sidecar/testkit/tests/support/process.rs
  • lib/sidecar/testkit/tests/support/sglang.rs

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.

@devin-ai-integration

devin-ai-integration Bot commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

❌ Dynamo PR CI failed — run 44202551210 (attempt 1) on 80a72f7f82

Gate checks: ❌ backend-status-check · ✅ deploy-status-check · ❌ dynamo-status-check

Framework Build
vLLM ❌ 1
SGLang ❌ 1
TRT-LLM ❌ 1
Other Jobs
dynamo-runtime ❌ 1
dynamo-sidecar ❌ 1
frontend ❌ 1
planner ❌ 1
triton-runtime ❌ 1

Failure details

8 jobs failed, all on the same error. Every image build stops at the Rust build because cargo can't resolve sglang-api-types = "=0.1.1": crates.io only has 0.1.0. The requirement comes from dynamo-sglang-sidecar and dynamo-sglang-mocker. Either publish sglang-api-types 0.1.1, or point these crates at the in-tree path or the published 0.1.0. A rerun will fail the same way until then.

❌ dynamo-sidecar / Build multi-arch cpu: sglang-api-types =0.1.1 not found on crates.io

Job: dynamo-sidecar / Build multi-arch cpu · Failed step: Build and Push Image · Logs: gh run view --job 134460308883 -R ai-dynamo/dynamo --log-failed

error: failed to select a version for the requirement `sglang-api-types = "=0.1.1"`
candidate versions found which didn't match: 0.1.0
location searched: crates.io index
required by package `dynamo-sglang-mocker v1.8.0 (/src/lib/mocker/servers/sglang)`

The sidecar-builder stage's cargo build --release --locked (vllm/sglang/trtllm/lpx sidecars) cannot resolve sglang-api-types = "=0.1.1": crates.io only has 0.1.0. Cargo names dynamo-sglang-mocker (lib/mocker/servers/sglang) as the requiring package.

❌ vllm-runtime / Build multi-arch cuda13.0: sglang-api-types =0.1.1 not found on crates.io

Job: vllm-runtime / Build multi-arch cuda13.0 · Failed step: Build and Push Image · Logs: gh run view --job 134460308943 -R ai-dynamo/dynamo --log-failed

error: failed to select a version for the requirement `sglang-api-types = "=0.1.1"`
candidate versions found which didn't match: 0.1.0
location searched: crates.io index
required by package `dynamo-sglang-sidecar v1.8.0 (/opt/dynamo/lib/sidecar/sglang)`

The wheel/runtime Rust build (cargo ... --locked) fails to resolve sglang-api-types = "=0.1.1": crates.io only has 0.1.0. The requiring package is dynamo-sglang-sidecar. This is the same cause as the other builds.

❌ triton-runtime / Build multi-arch cpu: sglang-api-types =0.1.1 not found on crates.io

Job: triton-runtime / Build multi-arch cpu · Failed step: Build and Push Image · Logs: gh run view --job 134460309089 -R ai-dynamo/dynamo --log-failed

error: failed to select a version for the requirement `sglang-api-types = "=0.1.1"`
candidate versions found which didn't match: 0.1.0
location searched: crates.io index
required by package `dynamo-sglang-sidecar v1.8.0 (/opt/dynamo/lib/sidecar/sglang)`

The wheel/runtime Rust build (cargo ... --locked) fails to resolve sglang-api-types = "=0.1.1": crates.io only has 0.1.0. The requiring package is dynamo-sglang-sidecar. This is the same cause as the other builds.

❌ trtllm-runtime / Build multi-arch cuda13.1: sglang-api-types =0.1.1 not found on crates.io

Job: trtllm-runtime / Build multi-arch cuda13.1 · Failed step: Build and Push Image · Logs: gh run view --job 134460309178 -R ai-dynamo/dynamo --log-failed

error: failed to select a version for the requirement `sglang-api-types = "=0.1.1"`
candidate versions found which didn't match: 0.1.0
location searched: crates.io index
required by package `dynamo-sglang-sidecar v1.8.0 (/opt/dynamo/lib/sidecar/sglang)`

The wheel/runtime Rust build (cargo ... --locked) fails to resolve sglang-api-types = "=0.1.1": crates.io only has 0.1.0. The requiring package is dynamo-sglang-sidecar. This is the same cause as the other builds.

❌ frontend / Build multi-arch cpu: sglang-api-types =0.1.1 not found on crates.io

Job: frontend / Build multi-arch cpu · Failed step: Build EPP Image (For Frontend Only) · Logs: gh run view --job 134460309179 -R ai-dynamo/dynamo --log-failed

error: failed to select a version for the requirement `sglang-api-types = "=0.1.1"`
candidate versions found which didn't match: 0.1.0
location searched: crates.io index
required by package `dynamo-sglang-mocker v1.8.0 (/src/lib/mocker/servers/sglang)`
ERROR: failed to build: failed to solve: failed to compute cache key: failed to calculate checksum of ref ...: "/epp": not found

The EPP image's Rust build hits the same sglang-api-types = "=0.1.1" resolution error from dynamo-sglang-mocker. Because of that, the later COPY of /epp, /sbom-rust-epp.cdx.json and /rust-licenses fails with not found, and the step exits with code 2.

❌ sglang-runtime / Build multi-arch cuda13.0: sglang-api-types =0.1.1 not found on crates.io

Job: sglang-runtime / Build multi-arch cuda13.0 · Failed step: Build and Push Image · Logs: gh run view --job 134460309272 -R ai-dynamo/dynamo --log-failed

error: failed to select a version for the requirement `sglang-api-types = "=0.1.1"`
candidate versions found which didn't match: 0.1.0
location searched: crates.io index
required by package `dynamo-sglang-sidecar v1.8.0 (/opt/dynamo/lib/sidecar/sglang)`

The wheel/runtime Rust build (cargo ... --locked) fails to resolve sglang-api-types = "=0.1.1": crates.io only has 0.1.0. The requiring package is dynamo-sglang-sidecar. This is the same cause as the other builds.

❌ planner / Build multi-arch cpu: sglang-api-types =0.1.1 not found on crates.io

Job: planner / Build multi-arch cpu · Failed step: Build and Push Image · Logs: gh run view --job 134460309526 -R ai-dynamo/dynamo --log-failed

error: failed to select a version for the requirement `sglang-api-types = "=0.1.1"`
candidate versions found which didn't match: 0.1.0
location searched: crates.io index
required by package `dynamo-sglang-sidecar v1.8.0 (/opt/dynamo/lib/sidecar/sglang)`

The wheel/runtime Rust build (cargo ... --locked) fails to resolve sglang-api-types = "=0.1.1": crates.io only has 0.1.0. The requiring package is dynamo-sglang-sidecar. This is the same cause as the other builds.

❌ dynamo-runtime / image / Build multi-arch cuda13.0: sglang-api-types =0.1.1 not found on crates.io

Job: dynamo-runtime / image / Build multi-arch cuda13.0 · Failed step: Build and Push Image · Logs: gh run view --job 134460309677 -R ai-dynamo/dynamo --log-failed

error: failed to select a version for the requirement `sglang-api-types = "=0.1.1"`
candidate versions found which didn't match: 0.1.0
location searched: crates.io index
required by package `dynamo-sglang-sidecar v1.8.0 (/opt/dynamo/lib/sidecar/sglang)`

The wheel/runtime Rust build (cargo ... --locked) fails to resolve sglang-api-types = "=0.1.1": crates.io only has 0.1.0. The requiring package is dynamo-sglang-sidecar. This is the same cause as the other builds.

For agents
{"pr": 17610, "run_id": 44202551210, "run_attempt": 1, "head_sha": "80a72f7f8254280f167ec8486cdbc6ff8bd19103", "failures": [{"job": "dynamo-sidecar / Build multi-arch cpu", "job_id": 134460308883, "failed_step": "Build and Push Image", "signature": "sglang-api-types =0.1.1 not found on crates.io (only 0.1.0)", "tests": [], "log_cmd": "gh run view --job 134460308883 -R ai-dynamo/dynamo --log-failed"}, {"job": "vllm-runtime / Build multi-arch cuda13.0", "job_id": 134460308943, "failed_step": "Build and Push Image", "signature": "sglang-api-types =0.1.1 not found on crates.io (only 0.1.0)", "tests": [], "log_cmd": "gh run view --job 134460308943 -R ai-dynamo/dynamo --log-failed"}, {"job": "triton-runtime / Build multi-arch cpu", "job_id": 134460309089, "failed_step": "Build and Push Image", "signature": "sglang-api-types =0.1.1 not found on crates.io (only 0.1.0)", "tests": [], "log_cmd": "gh run view --job 134460309089 -R ai-dynamo/dynamo --log-failed"}, {"job": "trtllm-runtime / Build multi-arch cuda13.1", "job_id": 134460309178, "failed_step": "Build and Push Image", "signature": "sglang-api-types =0.1.1 not found on crates.io (only 0.1.0)", "tests": [], "log_cmd": "gh run view --job 134460309178 -R ai-dynamo/dynamo --log-failed"}, {"job": "frontend / Build multi-arch cpu", "job_id": 134460309179, "failed_step": "Build EPP Image (For Frontend Only)", "signature": "sglang-api-types =0.1.1 not found on crates.io (only 0.1.0)", "tests": [], "log_cmd": "gh run view --job 134460309179 -R ai-dynamo/dynamo --log-failed"}, {"job": "sglang-runtime / Build multi-arch cuda13.0", "job_id": 134460309272, "failed_step": "Build and Push Image", "signature": "sglang-api-types =0.1.1 not found on crates.io (only 0.1.0)", "tests": [], "log_cmd": "gh run view --job 134460309272 -R ai-dynamo/dynamo --log-failed"}, {"job": "planner / Build multi-arch cpu", "job_id": 134460309526, "failed_step": "Build and Push Image", "signature": "sglang-api-types =0.1.1 not found on crates.io (only 0.1.0)", "tests": [], "log_cmd": "gh run view --job 134460309526 -R ai-dynamo/dynamo --log-failed"}, {"job": "dynamo-runtime / image / Build multi-arch cuda13.0", "job_id": 134460309677, "failed_step": "Build and Push Image", "signature": "sglang-api-types =0.1.1 not found on crates.io (only 0.1.0)", "tests": [], "log_cmd": "gh run view --job 134460309677 -R ai-dynamo/dynamo --log-failed"}]}

Posted automatically by Devin for run 44202551210. Updated on every full-CI run of this PR.

jain-ria and others added 3 commits November 4, 2026 22:49
Brings in origin/main 02331f0, the refreshed #17219 (fc43ab6),
#17368 (4bc0635) and #17431 (b724d66).

- Engine loss: NodeListeners::watch_serving runs one GetModelInfo poller
  per listener; the first lost listener loses the node, reported once as
  "SGLang engine lost: listener for DP rank r at <endpoint>: ...". The
  polls stop on cleanup's cancel token.
- Abort: NodeListeners::abort takes the ctx. A native dispatch rid is
  aborted only on the listener whose NativeHttp holds it; otherwise
  ctx.id() goes to every listener.
- args.rs: keep the routing flags and this branch's bootstrap-host doc.
- native_http.rs: port the rid test to the Target signature.
- Testkit: the node abort test waits for the native dispatch rid. New
  tests shut down listener 1 of 2: engine_lost names DP rank 1, and the
  sidecar process withdraws the worker and exits with code 1.
- READMEs: keep the refreshed support matrix and coverage rows, with this
  branch's DP rows on top.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: jain-ria <riajain@NVIDIA.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: jain-ria <riajain@NVIDIA.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: jain-ria <riajain@NVIDIA.com>
@jain-ria
jain-ria requested a review from a team as a code owner November 5, 2026 17:13
jain-ria added a commit that referenced this pull request Nov 5, 2026
- README and Fern sidecar.md: keep the Attention DP and Replica DP rows
  in the refreshed support matrix; the media row reads "No; the sidecar
  returns 400".
- common.sh: sglang_check_listener_ports accepts digits only and
  compares ports in base 10.
- sidecar.md and multinode_dp.yaml: attention-DP (any DP>1) needs an
  SGLang build with sgl-project/sglang#46637 on SGLang main at or after
  sgl-project/sglang#46181, replacing "SGLang 0.5.22 or later".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: jain-ria <riajain@NVIDIA.com>
jain-ria added a commit that referenced this pull request Nov 5, 2026
…aunch

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: jain-ria <riajain@NVIDIA.com>
jain-ria and others added 3 commits November 5, 2026 09:47
Drop an envelope's disagg_prefill_dp_rank in aggregated mode, as the
other bootstrap keys already are, and fail start when a listener's
DP rank or the node listener table differs from bootstrap. Remove two
redundant doc comments.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: jain-ria <riajain@NVIDIA.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: jain-ria <riajain@NVIDIA.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: jain-ria <riajain@NVIDIA.com>
jain-ria added a commit that referenced this pull request Nov 5, 2026
…aunch

- Keep the DP docs (#46637 requirement, attention-DP example).
- Carry #17431's removal of the decode bootstrap port into the DP
  multinode script.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: jain-ria <riajain@NVIDIA.com>
@jain-ria jain-ria changed the title feat(sglang-sidecar): serve all Rust listeners on a node from one sidecar feat(sglang-sidecar): serve a node's Rust listeners from one sidecar Nov 5, 2026

@tanmayv25 tanmayv25 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work getting DP>1 back on the Rust server. The node table, room_for_rank and the listener-loss watch read well, and the PD tests that check room % dp_size == rank are a good guard.

One thing I'd like fixed before merge: with any router mode except KV, a DP>1 node serves everything on its first rank (comment on listeners.rs:75). I'd suggest spreading unranked requests across the node's ranks. Below that, there's an abort fan-out case and two small cleanups.

Comment thread lib/sidecar/sglang/src/listeners.rs Resolved
Comment thread lib/sidecar/sglang/src/listeners.rs Resolved
Comment thread lib/sidecar/sglang/src/node.rs Resolved
Comment thread lib/sidecar/sglang/src/client.rs Resolved
jain-ria and others added 6 commits November 17, 2026 01:42
Signed-off-by: jain-ria <riajain@NVIDIA.com>
…s listeners

Only the KV router names a DP rank. With round-robin, random or direct
routing, and on the non-KV prefill path, every request on a DP>1 node
ran on its first listener while the node registered all of its ranks.
Send those requests round-robin over the node's listeners; a prefill
still carries the rank it lands on, so its bootstrap room matches.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: jain-ria <riajain@NVIDIA.com>
A migration retry keeps its public request ID. The gRPC path sent that
ID as SGLang's rid and aborted it on every listener of the node, and
SGLang's Abort cancels every generation with that rid, so cancelling a
failed attempt could also cancel its healthy retry.

Each dispatch now gets its own rid, the public ID plus 8 random hex
digits, for gRPC and native /generate alike. NodeListeners records each
dispatch's listener and rid until its stream is dropped, and abort sends
that rid to that listener only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: jain-ria <riajain@NVIDIA.com>
…proto listener table

Discovery and the listener health wait now run one retry_until helper,
which stops when the next try would start at or after the deadline;
RETRY_LOG_INTERVAL is private to client.rs. NodeTable stores
pb::NodeRustListener as SGLang reports it instead of a field-for-field
copy, so a new proto field cannot escape the table-change check.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: jain-ria <riajain@NVIDIA.com>
…case

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: jain-ria <riajain@NVIDIA.com>
Requests name a rank through the KV router or a client pin; the rest go
round-robin over the node's listeners.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: jain-ria <riajain@NVIDIA.com>
jain-ria added a commit that referenced this pull request Nov 17, 2026
Signed-off-by: jain-ria <riajain@NVIDIA.com>

@alec-flowers alec-flowers left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI-assisted review using the repo's graham-code-review and pr-checklist skills. Only findings not already raised in existing threads.

The design holds up: one listener table per node, a shared startup deadline, per-dispatch abort on the owning listener, and losing one listener means losing the node. The round-robin and abort fixes look right. Inline comments are mostly per-request costs and panics. Broader points:

  • Possible perf regression. At DP>1 no prefill bootstrap port is registered, so the frontend can't overlap decode dispatch with prefill, even for KV-routed PD where the router knows prefill_dp_rank. The description says this isn't benchmarked. A mocker run (agg and PD, DP=1 vs DP=2) would show whether it matters, and registering per-rank ports could be a follow-up.
  • Connection count. Connections grow N× per node: 256 at DP=8 with the default of 32. Worth a line in the docs, or scaling --grpc-connections per listener.
  • Validation. Per-listener engine loss and the latest abort and round-robin commits haven't run on GPU, and no CI job runs DP>1 on a real engine (#17612 skips it). Please note that in Validation.

Comment thread lib/sidecar/sglang/src/listeners.rs Resolved
Comment thread lib/sidecar/sglang/src/listeners.rs Resolved
Comment thread lib/sidecar/sglang/src/listeners.rs Resolved
Comment thread lib/sidecar/sglang/src/listeners.rs Resolved
Comment thread lib/sidecar/sglang/src/node.rs Resolved
Comment thread lib/sidecar/sglang/src/protocol.rs Resolved
Comment thread lib/sidecar/sglang/src/protocol.rs Resolved
Comment thread lib/sidecar/sglang/src/engine.rs Resolved
Comment thread lib/sidecar/sglang/src/client.rs Resolved
Comment thread lib/sidecar/sglang/src/listeners.rs Resolved
Signed-off-by: jain-ria <riajain@NVIDIA.com>
jain-ria added a commit that referenced this pull request Nov 17, 2026
Signed-off-by: jain-ria <riajain@NVIDIA.com>
@tanmayv25
tanmayv25 self-requested a review November 17, 2026 20:06
Comment thread lib/sidecar/sglang/src/discovery.rs Resolved
jain-ria and others added 2 commits November 17, 2026 15:29
Signed-off-by: jain-ria <riajain@NVIDIA.com>
…on panics

- The dispatch map uses parking_lot's Mutex and keeps each rid's random
  suffix instead of a copy of the rid, so a dispatch allocates once.
- NodeListeners::new and NodeInfo::from_listeners return errors instead
  of asserting, and watch_serving no longer expects a listener.
- One debug line per request (the dispatch line with rid and rank), and
  the startup retry warning keeps a constant message with the wait as a
  field.
- Say why a prefill pinned only by dp_rank still lands on its rank.
- Tests: the round-robin check uses a fresh node, and replica DP keeps
  the launch-wide max_num_seqs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: jain-ria <riajain@NVIDIA.com>
@jain-ria
jain-ria requested a review from a team as a code owner November 17, 2026 23:42
parking_lot would add a Cargo.lock entry for an uncontended lock that
std's futex-based Mutex already serves as fast. Keep std's Mutex and
handle poisoning in one helper instead of at each call site.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: jain-ria <riajain@NVIDIA.com>
jain-ria added a commit that referenced this pull request Nov 18, 2026
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: jain-ria <riajain@NVIDIA.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation feat size/XXL

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants