Repository navigation
[Feat] rust/grpc: Report orchestrator launch facts in api.v1 discovery - #45859
Merged
ishandhanani merged 15 commits intoNov 10, 2026
Merged
ishandhanani merged 15 commits into
ishandhanani merged 15 commits into
Conversation
Out-of-process orchestrators (such as Dynamo's SGLang sidecar) need a few launch facts to reproduce the Rust server's admission limits and sizing, which the legacy runtime.v1 GetServerInfo exposed through its raw server-args dump: - GetServerInfo: allow_auto_truncate, skip_tokenizer_init, and num_reserved_tokens (the speculative draft-token reserve the server already uses in its own total-token check). - GetModelInfo: model_type and architectures; HTTP /model_info now carries them too, matching Python's /model_info. - DeploymentInfo.hicache_host_total_tokens: report the KV host pool's realized capacity instead of null; the tree cache is built before the Rust server launches. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: jain-ria <riajain@NVIDIA.com>
10 of 13 tasks
Contributor
Author
|
/tag-and-rerun-ci |
Closed
4 of 5 tasks
1 task done
jain-ria
added a commit
to ai-dynamo/dynamo
that referenced
this pull request
Oct 27, 2026
The sidecar README and the Fern SGLang pages say the sidecar talks to SGLang's Rust server. The Kubernetes notes name the SGLang build it needs (sgl-project/sglang#45859 and #45883, plus #46635 for an in-process --sidecar), prefill's bootstrap on its HTTP port and KV events from discovery. The support matrix lists DP>1 as not yet supported, replacing the multi-node DP follower section and its diagram. The Mocker and testkit READMEs describe their api.v1 behavior, the proto README lists the source PRs after sgl-project/sglang#45996 was split, and the Slime example notes that it needs SGLang's Python server. Part of #17218 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: jain-ria <riajain@NVIDIA.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: jain-ria <riajain@NVIDIA.com>
Contributor
Author
|
/rerun-failed-ci |
Contributor
Author
|
/rerun-failed-ci |
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: jain-ria <riajain@NVIDIA.com>
Contributor
Author
|
/rerun-failed-ci |
3 similar comments
Contributor
Author
|
/rerun-failed-ci |
Contributor
Author
|
/rerun-failed-ci |
Contributor
Author
|
/rerun-failed-ci |
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: jain-ria <riajain@NVIDIA.com>
Contributor
Author
|
/rerun-failed-ci |
This comment has been minimized.
This comment has been minimized.
Contributor
Author
|
/rerun-failed-ci |
Contributor
Author
|
/rerun-failed-ci |
1 similar comment
Contributor
Author
|
/rerun-failed-ci |
Contributor
Author
|
/rerun-failed-ci |
1 similar comment
Contributor
Author
|
/rerun-failed-ci |
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: jain-ria <riajain@NVIDIA.com>
Contributor
Author
|
/rerun-failed-ci |
3 similar comments
Contributor
Author
|
/rerun-failed-ci |
Contributor
Author
|
/rerun-failed-ci |
Contributor
Author
|
/rerun-failed-ci |
This was referenced Nov 10, 2026
Closed
3 tasks done
Merged
4 of 5 tasks
4 of 5 tasks
2 of 5 tasks
jain-ria
added a commit
to jain-ria/sglang
that referenced
this pull request
Nov 16, 2026
Use 0.1.1 for the release that adds the sgl-project#45859 fields. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: jain-ria <riajain@NVIDIA.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Exposes new discovery fields to
api.v1clients.Modifications
New fields:
GetServerInfoResponse:allow_auto_truncate- to allow truncation of long prompts (instead of rejeciton)skip_tokenizer_init- launch server without tokenizernum_reserved_tokens- exact context-window budget under speculative decoding/server_infoalso gainsnum_reserved_tokens; it already carriedallow_auto_truncateandskip_tokenizer_initfrom the launch args.GetModelInfoResponse:model_type- model familyarchitectures- HF architectureDeploymentInfo.hicache_host_total_tokensreports the KV host pool's realized capacity instead ofnull. The tree cache, host pools included, is built before the Rust server launches.Tests
CPU, on this branch (base
mainat8cd3c494a3):cargo gen-api: no diff.cargo fmt --checkandcargo clippy --workspace --all-targets -- -D warnings: clean.cargo test --workspace: 547 passed, 0 failed, including the newserver_info_reports_launch_facts_for_orchestratorsand the extended/model_inforoute test.cargo package -p sglang-api-types --offline: OK.test/registered/unit/api/v1/test_parity.py: 9 passed (44 subtests).test/registered/unit/entrypoints/test_rust_server_dp_ports.py: 32 passed.pre-commiton the changed files: passed.End to end on H100 (earlier base
7d9bc10493, same change): Dynamo's SGLang sidecar read these fields over api.v1 in its aggregated, disaggregated and KV-router tests, 4/4 passed.Not applicable: this PR adds discovery metadata and does not change model outputs.
Checklist
Review and Merge Process
/tag-and-rerun-ci,/tag-run-ci-label,/rerun-failed-ci🤖 Generated with Claude Code
CI States
Latest PR Test (Base): ✅ Run #42141961059
Latest PR Test (Extra): ❌ Run #42141961110
Latest PR Test (AMD ROCm 10): ✅ Run #42141961066
Tip
Base CI only covers the core cases, and its workload may not exercise your change. Use
/rerun-testto run the tests your change touches, or copy this prompt for an agent: