Skip to content

fix(sglang): launch sidecars on SGLang's Rust server, one per listener node - #641

Open
jain-ria wants to merge 2 commits into
NVIDIA:mainfrom
jain-ria:jain-ria/sglang-apiv1-sidecar
Open

jain-ria wants to merge 2 commits into
NVIDIA:mainfrom
jain-ria:jain-ria/sglang-apiv1-sidecar

Conversation

@jain-ria

@jain-ria jain-ria commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

Problem

Dynamo's SGLang sidecar is moving to sglang.api.v1, which only SGLang's Rust server (SGLANG_RUST_SERVER=1) serves. The Rust server runs one listener per DP rank on the node that holds that rank's first scheduler, and the sidecar registers all of its node's listeners as one worker (ai-dynamo/dynamo#17610). Dynamo also removes the telemetry-only follower sidecar that relayed KV events (ai-dynamo/dynamo#17219).

srtctl still launches SGLang sidecar jobs for the old model. It starts the Python server, which serves runtime.v1, so an api.v1 sidecar fails engine discovery. It also runs a serving sidecar only on the endpoint leader, plus a relay sidecar on followers only when the role publishes KV events, so with multi-node attention DP the ranks on follower nodes get no serving sidecar. And prefill passes a --disaggregation-bootstrap-port that the Rust server rejects.

Changes

  • Set SGLANG_RUST_SERVER=1 in the worker environment of SGLang sidecar jobs. A role or global env can override it, like SGLANG_RUST_BUILD_MODE.
  • Run one sidecar on every node that hosts Rust listeners (stage 0 of the pipeline, the first scheduler of each DP rank), pointed at the node's first listener, whether or not the role publishes KV events. TP-only followers and later PP stages stay engine-only.
  • Listener k binds --port + k and --grpc-port + k, so each listener node reserves one sidecar gRPC port per listener.
  • Prefill: drop --disaggregation-bootstrap-port and its allocation in sidecar mode (the Rust server serves the bootstrap registry on each listener's --port). Every prefill node's sidecar passes its own --bootstrap-host, so decode fetches KV from the node that holds it.
  • Readiness: in Dynamo sidecar mode, the Dynamo frontend now expects one generate instance per sidecar, since each sidecar registers its own. A two-node attention-DP2 worker therefore needs both nodes' instances before the benchmark starts; before, one instance passed, so traffic could begin before node 1's DP rank was routable. vLLM keeps its DP-aware count, and TensorRT-LLM's leader-only sidecar still counts once per worker.
  • Update the sidecar and readiness tests, the two SGLang sidecar launch snapshots, docs/sidecars.md and the example headers.

Effect on the examples (see the snapshot diffs):

  • sglang-sidecar-multinode-dp.yaml (aggregated, TP2/attention DP2 on two nodes): only SGLANG_RUST_SERVER=1 is added. Node 1's sidecar now serves rank 1 instead of relaying its KV events.
  • sglang-sidecar-multinode-disagg-dp.yaml: decode node 1 gains a sidecar (before, it ran only the engine); prefill drops the bootstrap port and both prefill sidecars advertise their own node.

Merge order

Merge this after the Dynamo api.v1 stack lands: ai-dynamo/dynamo#17219, #17368, #17431, #17610 and #17612 (all open). Dynamo main's SGLang sidecar still speaks runtime.v1, which the Rust server does not serve, so SGLang sidecar jobs on today's Dynamo images would fail engine discovery with this change.

Validation

  • Rebased on main (2207464): ruff check, ruff format --check, ty check, srtctl schema-docs --check and the launch snapshots pass, and pytest tests/ passes 5582 tests, including new readiness cases in tests/test_health_expectations.py (two listener nodes register two instances; a TP-only follower adds none). The one local failure, the mlperf-client launch snapshot, comes from this machine's temp path changing the mount order; it fails the same way without this PR.
  • On a Slurm cluster with GH200 nodes (one GPU each), on 2026-10-30 and 2026-11-02, with an image built from SGLang d89a1b1 plus sgl-project/sglang#45859, #45883, #46636 and #46637 (all since on SGLang main, with #46636 replaced by #47963; none released yet), and Dynamo ai-dynamo/dynamo#17612 as of 8904ffa9a0 plus #17370 (since merged), using Qwen3-0.6B and recipes shaped like the examples:
    • One node, aggregated TP1: sa-bench 80/80 requests.
    • Two nodes, aggregated TP2 + attention DP2 (like sglang-sidecar-multinode-dp.yaml): node 1's engine got --node-rank 1 --grpc-port 19001 and its sidecar registered DP rank 1; the frontend added both DP ranks and both ranks served requests. Four runs; three completed 160/160 requests. One earlier run hung mid-benchmark inside SGLang's attention-DP sync; it did not reproduce in three reruns or with the same engine launched without srtctl or Dynamo.

Not run on GPUs: the disaggregated multi-node DP example, several listeners on one node, attention DP groups spanning nodes, and PP. Those layouts are covered by the unit tests and launch snapshots. The GPU runs also predate the Dynamo stack's latest changes (requests without a DP rank spread over a node's listeners, per-request abort); a re-run on a current image is pending.

🤖 Generated with Claude Code

@github-actions

github-actions Bot commented Nov 3, 2026 •

Copy link
Copy Markdown

src/srtctl diff: +80 -47

jain-ria and others added 2 commits November 17, 2026 10:49
…r node

Dynamo's SGLang sidecar now speaks sglang.api.v1, which only SGLang's Rust
server (SGLANG_RUST_SERVER=1) serves, and serves all of a node's Rust
listeners and DP ranks as one worker. Dynamo also removed the follower
KV-relay (telemetry-only) sidecar mode. Update the SGLang sidecar launcher:

- Set SGLANG_RUST_SERVER=1 in the worker environment of sidecar jobs
  (overridable from the role or global env, like SGLANG_RUST_BUILD_MODE).
- Run one sidecar on every node that hosts Rust listeners (the first
  scheduler of each DP rank, pipeline stage 0), pointed at the node's first
  listener, whether or not the role publishes KV events. With attention DP
  across nodes every listener node gets one; TP-only followers and later
  PP stages stay engine-only. This replaces the follower relay sidecar.
- Listener k binds --port + k and --grpc-port + k, so each listener node
  reserves one sidecar gRPC port per listener; the HTTP stride already
  spaces its HTTP ports.
- Prefill serves the KV bootstrap registry on each listener's HTTP port and
  SGLang rejects any other prefill bootstrap port, so drop
  --disaggregation-bootstrap-port (and its allocation) in sidecar mode.
  Every prefill node's sidecar now passes its own --bootstrap-host.
- Keep --incremental-streaming-output.

Update the sidecar tests, metrics targets, docs, example headers and the
two SGLang sidecar launch snapshots.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: jain-ria <riajain@NVIDIA.com>
Each Dynamo sidecar registers its own generate instance, and an SGLang
worker now runs one on every node that hosts Rust listeners. Readiness
still expected one instance per logical worker, so it passed once node
0's sidecar registered, before the other nodes' DP ranks could be
routed. In sidecar mode, the Dynamo frontend now expects one instance
per process its backend runs a sidecar on (runs_dynamo_sidecar); vLLM
keeps its DP-aware count, and TensorRT-LLM's leader-only sidecar keeps
one per worker.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: jain-ria <riajain@NVIDIA.com>
@jain-ria
jain-ria force-pushed the jain-ria/sglang-apiv1-sidecar branch from afde712 to fd17e33 Compare November 17, 2026 19:07

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant