Repository navigation
Conversation
jain-ria
requested review from
alec-flowers,
csahithi,
hjjq,
ishandhanani,
kedarpotdar-nv,
kyleliang-nv,
nlevin-ui and
qiching
as code owners
November 3, 2026 01:44
|
|
Open
1 of 7 tasks
…r node Dynamo's SGLang sidecar now speaks sglang.api.v1, which only SGLang's Rust server (SGLANG_RUST_SERVER=1) serves, and serves all of a node's Rust listeners and DP ranks as one worker. Dynamo also removed the follower KV-relay (telemetry-only) sidecar mode. Update the SGLang sidecar launcher: - Set SGLANG_RUST_SERVER=1 in the worker environment of sidecar jobs (overridable from the role or global env, like SGLANG_RUST_BUILD_MODE). - Run one sidecar on every node that hosts Rust listeners (the first scheduler of each DP rank, pipeline stage 0), pointed at the node's first listener, whether or not the role publishes KV events. With attention DP across nodes every listener node gets one; TP-only followers and later PP stages stay engine-only. This replaces the follower relay sidecar. - Listener k binds --port + k and --grpc-port + k, so each listener node reserves one sidecar gRPC port per listener; the HTTP stride already spaces its HTTP ports. - Prefill serves the KV bootstrap registry on each listener's HTTP port and SGLang rejects any other prefill bootstrap port, so drop --disaggregation-bootstrap-port (and its allocation) in sidecar mode. Every prefill node's sidecar now passes its own --bootstrap-host. - Keep --incremental-streaming-output. Update the sidecar tests, metrics targets, docs, example headers and the two SGLang sidecar launch snapshots. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: jain-ria <riajain@NVIDIA.com>
Each Dynamo sidecar registers its own generate instance, and an SGLang worker now runs one on every node that hosts Rust listeners. Readiness still expected one instance per logical worker, so it passed once node 0's sidecar registered, before the other nodes' DP ranks could be routed. In sidecar mode, the Dynamo frontend now expects one instance per process its backend runs a sidecar on (runs_dynamo_sidecar); vLLM keeps its DP-aware count, and TensorRT-LLM's leader-only sidecar keeps one per worker. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: jain-ria <riajain@NVIDIA.com>
jain-ria
force-pushed
the
jain-ria/sglang-apiv1-sidecar
branch
from
November 17, 2026 19:07
afde712 to
fd17e33
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Dynamo's SGLang sidecar is moving to
sglang.api.v1, which only SGLang's Rust server (SGLANG_RUST_SERVER=1) serves. The Rust server runs one listener per DP rank on the node that holds that rank's first scheduler, and the sidecar registers all of its node's listeners as one worker (ai-dynamo/dynamo#17610). Dynamo also removes the telemetry-only follower sidecar that relayed KV events (ai-dynamo/dynamo#17219).srtctl still launches SGLang sidecar jobs for the old model. It starts the Python server, which serves
runtime.v1, so anapi.v1sidecar fails engine discovery. It also runs a serving sidecar only on the endpoint leader, plus a relay sidecar on followers only when the role publishes KV events, so with multi-node attention DP the ranks on follower nodes get no serving sidecar. And prefill passes a--disaggregation-bootstrap-portthat the Rust server rejects.Changes
SGLANG_RUST_SERVER=1in the worker environment of SGLang sidecar jobs. A role or global env can override it, likeSGLANG_RUST_BUILD_MODE.--port + kand--grpc-port + k, so each listener node reserves one sidecar gRPC port per listener.--disaggregation-bootstrap-portand its allocation in sidecar mode (the Rust server serves the bootstrap registry on each listener's--port). Every prefill node's sidecar passes its own--bootstrap-host, so decode fetches KV from the node that holds it.docs/sidecars.mdand the example headers.Effect on the examples (see the snapshot diffs):
sglang-sidecar-multinode-dp.yaml(aggregated, TP2/attention DP2 on two nodes): onlySGLANG_RUST_SERVER=1is added. Node 1's sidecar now serves rank 1 instead of relaying its KV events.sglang-sidecar-multinode-disagg-dp.yaml: decode node 1 gains a sidecar (before, it ran only the engine); prefill drops the bootstrap port and both prefill sidecars advertise their own node.Merge order
Merge this after the Dynamo api.v1 stack lands: ai-dynamo/dynamo#17219, #17368, #17431, #17610 and #17612 (all open). Dynamo main's SGLang sidecar still speaks
runtime.v1, which the Rust server does not serve, so SGLang sidecar jobs on today's Dynamo images would fail engine discovery with this change.Validation
main(2207464):ruff check,ruff format --check,ty check,srtctl schema-docs --checkand the launch snapshots pass, andpytest tests/passes 5582 tests, including new readiness cases intests/test_health_expectations.py(two listener nodes register two instances; a TP-only follower adds none). The one local failure, themlperf-clientlaunch snapshot, comes from this machine's temp path changing the mount order; it fails the same way without this PR.sglang-sidecar-multinode-dp.yaml): node 1's engine got--node-rank 1 --grpc-port 19001and its sidecar registered DP rank 1; the frontend added both DP ranks and both ranks served requests. Four runs; three completed 160/160 requests. One earlier run hung mid-benchmark inside SGLang's attention-DP sync; it did not reproduce in three reruns or with the same engine launched without srtctl or Dynamo.Not run on GPUs: the disaggregated multi-node DP example, several listeners on one node, attention DP groups spanning nodes, and PP. Those layouts are covered by the unit tests and launch snapshots. The GPU runs also predate the Dynamo stack's latest changes (requests without a DP rank spread over a node's listeners, per-request abort); a re-run on a current image is pending.
🤖 Generated with Claude Code