Repository navigation
Conversation
SGLang's Rust server runs one listener per DP rank, on --port + k and --grpc-port + k, and the sidecar, which served one listener, refused DP>1. It now finds every listener of its node in DeploymentInfo.node_listeners (sgl-project/sglang#46637) and serves them as one worker that owns the node's DP ranks. Replica DP also needs sgl-project/sglang#45996. The proto and the frontend are unchanged. - Startup connects to every listener under one deadline and registers one worker for their DP ranks. A build without the table still serves DP=1. Bootstrap discovery retries UNAVAILABLE, which GetServerInfo answers until every rank's listener is bound (sgl-project/sglang#46181). - A request runs on the listener of the rank the router names; a rank outside the node is an invalid argument. Abort goes to every listener. - At global DP>1 prefill registers no bootstrap port. Its first response carries the dispatched listener's bootstrap port and a room with room % dp_size == that listener's rank, so decode reaches the rank through the frontend's existing handoff. Prefill names its rank to SGLang only when the router named one. At DP=1 prefill still registers its port. On a node other than 0, prefill no longer advertises the rendezvous host. - The SGLang Mocker library builds one node's listeners, which the testkit serves as one node with one listener table. The Mocker binary takes no new options. - The sidecar README's support matrix lists attention DP and replica DP (not for prefill) and gives the listener port spacing. Part of #17218 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: jain-ria <riajain@NVIDIA.com>
|
No actionable comments were generated in the recent review. WalkthroughThe SGLang Rust server and sidecar now represent each node as a set of data-parallel listeners. The sidecar validates listener metadata, starts and routes across listeners, and propagates prefill ranks through gRPC and native HTTP requests. Tests cover multi-listener routing and abort behavior. ChangesSGLang node listeners
Estimated code review effort: 4 (Complex) | ~60 minutes Merge Risk:
|
❌ Dynamo PR CI failed — run 44202551210 (attempt 1) on
|
| Framework | Build |
|---|---|
| vLLM | ❌ 1 |
| SGLang | ❌ 1 |
| TRT-LLM | ❌ 1 |
| Other | Jobs |
|---|---|
| dynamo-runtime | ❌ 1 |
| dynamo-sidecar | ❌ 1 |
| frontend | ❌ 1 |
| planner | ❌ 1 |
| triton-runtime | ❌ 1 |
Failure details
8 jobs failed, all on the same error. Every image build stops at the Rust build because cargo can't resolve sglang-api-types = "=0.1.1": crates.io only has 0.1.0. The requirement comes from dynamo-sglang-sidecar and dynamo-sglang-mocker. Either publish sglang-api-types 0.1.1, or point these crates at the in-tree path or the published 0.1.0. A rerun will fail the same way until then.
❌ dynamo-sidecar / Build multi-arch cpu: sglang-api-types =0.1.1 not found on crates.io
Job: dynamo-sidecar / Build multi-arch cpu · Failed step: Build and Push Image · Logs: gh run view --job 134460308883 -R ai-dynamo/dynamo --log-failed
error: failed to select a version for the requirement `sglang-api-types = "=0.1.1"`
candidate versions found which didn't match: 0.1.0
location searched: crates.io index
required by package `dynamo-sglang-mocker v1.8.0 (/src/lib/mocker/servers/sglang)`
The sidecar-builder stage's cargo build --release --locked (vllm/sglang/trtllm/lpx sidecars) cannot resolve sglang-api-types = "=0.1.1": crates.io only has 0.1.0. Cargo names dynamo-sglang-mocker (lib/mocker/servers/sglang) as the requiring package.
❌ vllm-runtime / Build multi-arch cuda13.0: sglang-api-types =0.1.1 not found on crates.io
Job: vllm-runtime / Build multi-arch cuda13.0 · Failed step: Build and Push Image · Logs: gh run view --job 134460308943 -R ai-dynamo/dynamo --log-failed
error: failed to select a version for the requirement `sglang-api-types = "=0.1.1"`
candidate versions found which didn't match: 0.1.0
location searched: crates.io index
required by package `dynamo-sglang-sidecar v1.8.0 (/opt/dynamo/lib/sidecar/sglang)`
The wheel/runtime Rust build (cargo ... --locked) fails to resolve sglang-api-types = "=0.1.1": crates.io only has 0.1.0. The requiring package is dynamo-sglang-sidecar. This is the same cause as the other builds.
❌ triton-runtime / Build multi-arch cpu: sglang-api-types =0.1.1 not found on crates.io
Job: triton-runtime / Build multi-arch cpu · Failed step: Build and Push Image · Logs: gh run view --job 134460309089 -R ai-dynamo/dynamo --log-failed
error: failed to select a version for the requirement `sglang-api-types = "=0.1.1"`
candidate versions found which didn't match: 0.1.0
location searched: crates.io index
required by package `dynamo-sglang-sidecar v1.8.0 (/opt/dynamo/lib/sidecar/sglang)`
The wheel/runtime Rust build (cargo ... --locked) fails to resolve sglang-api-types = "=0.1.1": crates.io only has 0.1.0. The requiring package is dynamo-sglang-sidecar. This is the same cause as the other builds.
❌ trtllm-runtime / Build multi-arch cuda13.1: sglang-api-types =0.1.1 not found on crates.io
Job: trtllm-runtime / Build multi-arch cuda13.1 · Failed step: Build and Push Image · Logs: gh run view --job 134460309178 -R ai-dynamo/dynamo --log-failed
error: failed to select a version for the requirement `sglang-api-types = "=0.1.1"`
candidate versions found which didn't match: 0.1.0
location searched: crates.io index
required by package `dynamo-sglang-sidecar v1.8.0 (/opt/dynamo/lib/sidecar/sglang)`
The wheel/runtime Rust build (cargo ... --locked) fails to resolve sglang-api-types = "=0.1.1": crates.io only has 0.1.0. The requiring package is dynamo-sglang-sidecar. This is the same cause as the other builds.
❌ frontend / Build multi-arch cpu: sglang-api-types =0.1.1 not found on crates.io
Job: frontend / Build multi-arch cpu · Failed step: Build EPP Image (For Frontend Only) · Logs: gh run view --job 134460309179 -R ai-dynamo/dynamo --log-failed
error: failed to select a version for the requirement `sglang-api-types = "=0.1.1"`
candidate versions found which didn't match: 0.1.0
location searched: crates.io index
required by package `dynamo-sglang-mocker v1.8.0 (/src/lib/mocker/servers/sglang)`
ERROR: failed to build: failed to solve: failed to compute cache key: failed to calculate checksum of ref ...: "/epp": not found
The EPP image's Rust build hits the same sglang-api-types = "=0.1.1" resolution error from dynamo-sglang-mocker. Because of that, the later COPY of /epp, /sbom-rust-epp.cdx.json and /rust-licenses fails with not found, and the step exits with code 2.
❌ sglang-runtime / Build multi-arch cuda13.0: sglang-api-types =0.1.1 not found on crates.io
Job: sglang-runtime / Build multi-arch cuda13.0 · Failed step: Build and Push Image · Logs: gh run view --job 134460309272 -R ai-dynamo/dynamo --log-failed
error: failed to select a version for the requirement `sglang-api-types = "=0.1.1"`
candidate versions found which didn't match: 0.1.0
location searched: crates.io index
required by package `dynamo-sglang-sidecar v1.8.0 (/opt/dynamo/lib/sidecar/sglang)`
The wheel/runtime Rust build (cargo ... --locked) fails to resolve sglang-api-types = "=0.1.1": crates.io only has 0.1.0. The requiring package is dynamo-sglang-sidecar. This is the same cause as the other builds.
❌ planner / Build multi-arch cpu: sglang-api-types =0.1.1 not found on crates.io
Job: planner / Build multi-arch cpu · Failed step: Build and Push Image · Logs: gh run view --job 134460309526 -R ai-dynamo/dynamo --log-failed
error: failed to select a version for the requirement `sglang-api-types = "=0.1.1"`
candidate versions found which didn't match: 0.1.0
location searched: crates.io index
required by package `dynamo-sglang-sidecar v1.8.0 (/opt/dynamo/lib/sidecar/sglang)`
The wheel/runtime Rust build (cargo ... --locked) fails to resolve sglang-api-types = "=0.1.1": crates.io only has 0.1.0. The requiring package is dynamo-sglang-sidecar. This is the same cause as the other builds.
❌ dynamo-runtime / image / Build multi-arch cuda13.0: sglang-api-types =0.1.1 not found on crates.io
Job: dynamo-runtime / image / Build multi-arch cuda13.0 · Failed step: Build and Push Image · Logs: gh run view --job 134460309677 -R ai-dynamo/dynamo --log-failed
error: failed to select a version for the requirement `sglang-api-types = "=0.1.1"`
candidate versions found which didn't match: 0.1.0
location searched: crates.io index
required by package `dynamo-sglang-sidecar v1.8.0 (/opt/dynamo/lib/sidecar/sglang)`
The wheel/runtime Rust build (cargo ... --locked) fails to resolve sglang-api-types = "=0.1.1": crates.io only has 0.1.0. The requiring package is dynamo-sglang-sidecar. This is the same cause as the other builds.
For agents
{"pr": 17610, "run_id": 44202551210, "run_attempt": 1, "head_sha": "80a72f7f8254280f167ec8486cdbc6ff8bd19103", "failures": [{"job": "dynamo-sidecar / Build multi-arch cpu", "job_id": 134460308883, "failed_step": "Build and Push Image", "signature": "sglang-api-types =0.1.1 not found on crates.io (only 0.1.0)", "tests": [], "log_cmd": "gh run view --job 134460308883 -R ai-dynamo/dynamo --log-failed"}, {"job": "vllm-runtime / Build multi-arch cuda13.0", "job_id": 134460308943, "failed_step": "Build and Push Image", "signature": "sglang-api-types =0.1.1 not found on crates.io (only 0.1.0)", "tests": [], "log_cmd": "gh run view --job 134460308943 -R ai-dynamo/dynamo --log-failed"}, {"job": "triton-runtime / Build multi-arch cpu", "job_id": 134460309089, "failed_step": "Build and Push Image", "signature": "sglang-api-types =0.1.1 not found on crates.io (only 0.1.0)", "tests": [], "log_cmd": "gh run view --job 134460309089 -R ai-dynamo/dynamo --log-failed"}, {"job": "trtllm-runtime / Build multi-arch cuda13.1", "job_id": 134460309178, "failed_step": "Build and Push Image", "signature": "sglang-api-types =0.1.1 not found on crates.io (only 0.1.0)", "tests": [], "log_cmd": "gh run view --job 134460309178 -R ai-dynamo/dynamo --log-failed"}, {"job": "frontend / Build multi-arch cpu", "job_id": 134460309179, "failed_step": "Build EPP Image (For Frontend Only)", "signature": "sglang-api-types =0.1.1 not found on crates.io (only 0.1.0)", "tests": [], "log_cmd": "gh run view --job 134460309179 -R ai-dynamo/dynamo --log-failed"}, {"job": "sglang-runtime / Build multi-arch cuda13.0", "job_id": 134460309272, "failed_step": "Build and Push Image", "signature": "sglang-api-types =0.1.1 not found on crates.io (only 0.1.0)", "tests": [], "log_cmd": "gh run view --job 134460309272 -R ai-dynamo/dynamo --log-failed"}, {"job": "planner / Build multi-arch cpu", "job_id": 134460309526, "failed_step": "Build and Push Image", "signature": "sglang-api-types =0.1.1 not found on crates.io (only 0.1.0)", "tests": [], "log_cmd": "gh run view --job 134460309526 -R ai-dynamo/dynamo --log-failed"}, {"job": "dynamo-runtime / image / Build multi-arch cuda13.0", "job_id": 134460309677, "failed_step": "Build and Push Image", "signature": "sglang-api-types =0.1.1 not found on crates.io (only 0.1.0)", "tests": [], "log_cmd": "gh run view --job 134460309677 -R ai-dynamo/dynamo --log-failed"}]}Posted automatically by Devin for run 44202551210. Updated on every full-CI run of this PR.
Brings in origin/main 02331f0, the refreshed #17219 (fc43ab6), #17368 (4bc0635) and #17431 (b724d66). - Engine loss: NodeListeners::watch_serving runs one GetModelInfo poller per listener; the first lost listener loses the node, reported once as "SGLang engine lost: listener for DP rank r at <endpoint>: ...". The polls stop on cleanup's cancel token. - Abort: NodeListeners::abort takes the ctx. A native dispatch rid is aborted only on the listener whose NativeHttp holds it; otherwise ctx.id() goes to every listener. - args.rs: keep the routing flags and this branch's bootstrap-host doc. - native_http.rs: port the rid test to the Target signature. - Testkit: the node abort test waits for the native dispatch rid. New tests shut down listener 1 of 2: engine_lost names DP rank 1, and the sidecar process withdraws the worker and exits with code 1. - READMEs: keep the refreshed support matrix and coverage rows, with this branch's DP rows on top. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: jain-ria <riajain@NVIDIA.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: jain-ria <riajain@NVIDIA.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: jain-ria <riajain@NVIDIA.com>
- README and Fern sidecar.md: keep the Attention DP and Replica DP rows in the refreshed support matrix; the media row reads "No; the sidecar returns 400". - common.sh: sglang_check_listener_ports accepts digits only and compares ports in base 10. - sidecar.md and multinode_dp.yaml: attention-DP (any DP>1) needs an SGLang build with sgl-project/sglang#46637 on SGLang main at or after sgl-project/sglang#46181, replacing "SGLang 0.5.22 or later". Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: jain-ria <riajain@NVIDIA.com>
…aunch Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: jain-ria <riajain@NVIDIA.com>
Drop an envelope's disagg_prefill_dp_rank in aggregated mode, as the other bootstrap keys already are, and fail start when a listener's DP rank or the node listener table differs from bootstrap. Remove two redundant doc comments. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: jain-ria <riajain@NVIDIA.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: jain-ria <riajain@NVIDIA.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: jain-ria <riajain@NVIDIA.com>
…aunch - Keep the DP docs (#46637 requirement, attention-DP example). - Carry #17431's removal of the decode bootstrap port into the DP multinode script. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: jain-ria <riajain@NVIDIA.com>
tanmayv25
left a comment
There was a problem hiding this comment.
Nice work getting DP>1 back on the Rust server. The node table, room_for_rank and the listener-loss watch read well, and the PD tests that check room % dp_size == rank are a good guard.
One thing I'd like fixed before merge: with any router mode except KV, a DP>1 node serves everything on its first rank (comment on listeners.rs:75). I'd suggest spreading unranked requests across the node's ranks. Below that, there's an abort fan-out case and two small cleanups.
Signed-off-by: jain-ria <riajain@NVIDIA.com>
…s listeners Only the KV router names a DP rank. With round-robin, random or direct routing, and on the non-KV prefill path, every request on a DP>1 node ran on its first listener while the node registered all of its ranks. Send those requests round-robin over the node's listeners; a prefill still carries the rank it lands on, so its bootstrap room matches. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: jain-ria <riajain@NVIDIA.com>
A migration retry keeps its public request ID. The gRPC path sent that ID as SGLang's rid and aborted it on every listener of the node, and SGLang's Abort cancels every generation with that rid, so cancelling a failed attempt could also cancel its healthy retry. Each dispatch now gets its own rid, the public ID plus 8 random hex digits, for gRPC and native /generate alike. NodeListeners records each dispatch's listener and rid until its stream is dropped, and abort sends that rid to that listener only. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: jain-ria <riajain@NVIDIA.com>
…proto listener table Discovery and the listener health wait now run one retry_until helper, which stops when the next try would start at or after the deadline; RETRY_LOG_INTERVAL is private to client.rs. NodeTable stores pb::NodeRustListener as SGLang reports it instead of a field-for-field copy, so a new proto field cannot escape the table-change check. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: jain-ria <riajain@NVIDIA.com>
…case Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: jain-ria <riajain@NVIDIA.com>
Requests name a rank through the KV router or a client pin; the rest go round-robin over the node's listeners. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: jain-ria <riajain@NVIDIA.com>
Signed-off-by: jain-ria <riajain@NVIDIA.com>
alec-flowers
left a comment
There was a problem hiding this comment.
AI-assisted review using the repo's graham-code-review and pr-checklist skills. Only findings not already raised in existing threads.
The design holds up: one listener table per node, a shared startup deadline, per-dispatch abort on the owning listener, and losing one listener means losing the node. The round-robin and abort fixes look right. Inline comments are mostly per-request costs and panics. Broader points:
- Possible perf regression. At DP>1 no prefill bootstrap port is registered, so the frontend can't overlap decode dispatch with prefill, even for KV-routed PD where the router knows
prefill_dp_rank. The description says this isn't benchmarked. A mocker run (agg and PD, DP=1 vs DP=2) would show whether it matters, and registering per-rank ports could be a follow-up. - Connection count. Connections grow N× per node: 256 at DP=8 with the default of 32. Worth a line in the docs, or scaling
--grpc-connectionsper listener. - Validation. Per-listener engine loss and the latest abort and round-robin commits haven't run on GPU, and no CI job runs DP>1 on a real engine (#17612 skips it). Please note that in Validation.
Signed-off-by: jain-ria <riajain@NVIDIA.com>
Signed-off-by: jain-ria <riajain@NVIDIA.com>
Signed-off-by: jain-ria <riajain@NVIDIA.com>
…on panics - The dispatch map uses parking_lot's Mutex and keeps each rid's random suffix instead of a copy of the rid, so a dispatch allocates once. - NodeListeners::new and NodeInfo::from_listeners return errors instead of asserting, and watch_serving no longer expects a listener. - One debug line per request (the dispatch line with rid and rank), and the startup retry warning keeps a constant message with the wait as a field. - Say why a prefill pinned only by dp_rank still lands on its rank. - Tests: the round-robin check uses a fresh node, and replica DP keeps the launch-wide max_num_seqs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: jain-ria <riajain@NVIDIA.com>
parking_lot would add a Cargo.lock entry for an uncontended lock that std's futex-based Mutex already serves as fast. Keep std's Mutex and handle poisoning in one helper instead of at each call site. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: jain-ria <riajain@NVIDIA.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: jain-ria <riajain@NVIDIA.com>
Overview
SGLang's Rust server opens one listener (an HTTP and gRPC endpoint) for each data-parallel (DP) rank. The sidecar connected to only one listener, so it refused DP>1. Now one sidecar per node connects to all of that node's listeners and registers them as one Dynamo worker that owns their DP ranks. Dynamo can then serve SGLang attention DP (
--attn-dp-size), and replica DP (--dp-size) on aggregated and decode workers. DP=1 serves as before, apart from the request IDs SGLang sees (see User-facing).Key Changes
Core logic: about 800 added lines (about 750 in
lib/sidecar/sglang, 55 in the SGLang Mocker), excluding tests, comments and blank lines. Most of it is the node's listener table, its checks, per-listener startup and per-request dispatch. Nothing changes inlib/llm,lib/sidecar/commonor the proto.node.rs: reads the node's listeners fromDeploymentInfo.node_listenersand registers one worker that owns their DP ranks (for example ranks 2-3 of 4). The worker reports the smallest KV capacity of its listeners, and startup fails unless every listener publishes KV events or none does.listeners.rs: picks the listener for each request, and makes an abort cancel only that request.nvext.dp_rankorx-dynamo-dp-rankwith a worker ID). Requests that name no rank (round-robin, random or direct routing, and prefill without KV routing) take the node's listeners in turn. A rank outside the node isInvalidArgumentbefore a stream opens.<request id>-<8 hex digits>, and an abort cancels only that rid, on the listener running it. Dynamo retries a migrated request under the same request ID, and SGLang's Abort cancels every request with a given rid, so aborting by request ID could also kill a healthy retry.GetModelInfopoller. If any listener is lost, the node is lost: the worker is withdrawn and the sidecar exits with one error naming the rank, "SGLang engine lost: listener for DP rank r at : ...".protocol.rs: when the deployment's total DP size is above 1, prefill no longer registers one fixed bootstrap port, the port decode pulls KV through (engine.rs). Instead, prefill's first response carries the bootstrap port of the listener that ran it, and a bootstrap room chosen so thatroom % dp_sizeis that listener's rank, which is how SGLang finds the rank. Decode gets both through the frontend's existing first-response handoff. Prefill names its rank to SGLang only when the request named one.Also:
UNAVAILABLE, which SGLang'sGetServerInforeturns until every rank's listener is up ([Feat] rust-server: Fan control requests out to every DP rank sgl-project/sglang#46181).--grpc-connectionsconnections, and that DP>1 needs SGLang main at or after [Feat] rust/grpc: Report the node's Rust listeners in api.v1 discovery sgl-project/sglang#46637.User-facing: SGLang DP>1 starts instead of failing discovery. SGLang without
node_listeners(v0.5.23) still serves DP=1. SGLang sees gRPC requests under<request id>-<8 hex digits>, as it already did for native/generaterequests. A prefill sidecar on a node other than node 0 advertises its own address for PD bootstrap, not node 0's--dist-init-addr. No CLI flag, environment variable, proto or frontend change.Performance: per request, a range check or a round-robin counter picks the listener. Each listener gets its own
--grpc-connectionspool, so a DP=8 node opens 8 times as many (256 at the default 32). Each listener also gets oneGetModelInfopoll a second. An abort is one RPC, to one listener. When the total DP size is above 1, the frontend dispatches decode after prefill's first response instead of alongside it. DP=1 runs as before. Not benchmarked.Validation
[2, 4)of 4node_registration_covers_every_listenerrejects_unusable_tables,kv_union_requires_every_rank_or_noneInvalidArgument; with no rank, requests take the listeners in turnrouting_rank_selects_listener/generatebody with adisagg_prefill_dp_rankfrom the requestnative_bodies_name_the_dispatched_prefill_ranksglang_node_sidecar_aborts_on_the_owning_listener,sglang_abort_after_cancelled_stream_ends_reaches_its_dispatchrepeated_public_ids_have_their_own_abort_targetsroom % 4 == rank; a pinned rank 3 keeps its ranksglang_round_robin_prefill_hands_off_on_both_nodessglang_node_engine_lost_names_the_lost_listener,sglang_node_listener_loss_withdraws_worker_and_fails_the_processGetServerInforeturnsUNAVAILABLEwhile ranks bindstartup_discovery_retries_only_unavailable7102ad8c0b)--load-balance-method round_robin; abort on both ranks; SIGTERM and kill -9; sidecar started before the engine; attention DP across 2 nodes5331f7b2f2493c857736tests/serve/test_sidecar.py -k sglang_dep_kv_router5331f7b2f2--load-balance-method round_robin; decode finds each prefill rank's bootstrap on that rank's own node493c8577365331f7b2f2gpu_1SGLang sidecar tests pass on stock v0.5.23tests/serve/test_sidecar.pyAt
4d0cf3352d, built against a local copy ofsglang-api-types0.1.1, on CPU: the 107 SGLang sidecar tests, the SGLang Mocker's 15 anddynamo-sglang-generate's 18, all 55sidecar_mocker_integrationand all 54router_sidecar_mocker_integrationtests (every backend), and the vLLM, TRT-LLM and LPX sidecar tests pass, and clippy is clean. These run in pre-merge's Cargo tests.None of the GPU runs include the refreshed #17431, the round-robin change (
cd4c498784) or the abort change (8c826eaaae), and per-listener engine loss has not run on a GPU. No CI job runs DP>1 on a real SGLang engine: CI's SGLang (v0.5.23) reports nonode_listeners, so the 2-GPUsglang_dep_kv_routerpost-merge job cannot pass there; #17612 skips it until the pin includes #46637.sglang-api-typesuntil 0.1.1 is published ([Chore] rust: Bump sglang-api-types to 0.1.1 sgl-project/sglang#50028), as on feat(sglang-sidecar): replace runtime.v1 with sglang.api.v1 #17219.Issues and Ongoing Work
Pull Request Checklist
/pr-checklistskill.🤖 Generated with Claude Code