Skip to content

[Feat] rust/grpc: Report orchestrator launch facts in api.v1 discovery - #45859

Merged
ishandhanani merged 15 commits into
sgl-project:mainfrom
jain-ria:jain-ria/api-v1-discovery-launch-facts
Nov 10, 2026
Merged

ishandhanani merged 15 commits into
sgl-project:mainfrom
jain-ria:jain-ria/api-v1-discovery-launch-facts

Conversation

@jain-ria

@jain-ria jain-ria commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

Motivation

Exposes new discovery fields to api.v1 clients.

Modifications

New fields:

  • In GetServerInfoResponse:
    • allow_auto_truncate - to allow truncation of long prompts (instead of rejeciton)
    • skip_tokenizer_init - launch server without tokenizer
    • num_reserved_tokens - exact context-window budget under speculative decoding
    • Note: HTTP /server_info also gains num_reserved_tokens; it already carried allow_auto_truncate and skip_tokenizer_init from the launch args.
  • In GetModelInfoResponse:
    • model_type - model family
    • architectures - HF architecture
  • DeploymentInfo.hicache_host_total_tokens reports the KV host pool's realized capacity instead of null. The tree cache, host pools included, is built before the Rust server launches.

Tests

CPU, on this branch (base main at 8cd3c494a3):

  • cargo gen-api: no diff. cargo fmt --check and cargo clippy --workspace --all-targets -- -D warnings: clean.
  • cargo test --workspace: 547 passed, 0 failed, including the new server_info_reports_launch_facts_for_orchestrators and the extended /model_info route test.
  • cargo package -p sglang-api-types --offline: OK.
  • test/registered/unit/api/v1/test_parity.py: 9 passed (44 subtests). test/registered/unit/entrypoints/test_rust_server_dp_ports.py: 32 passed.
  • pre-commit on the changed files: passed.

End to end on H100 (earlier base 7d9bc10493, same change): Dynamo's SGLang sidecar read these fields over api.v1 in its aggregated, disaggregated and KV-router tests, 4/4 passed.

Not applicable: this PR adds discovery metadata and does not change model outputs.

Checklist

Review and Merge Process

  1. Ping Merge Oncalls to start the process. See the PR Merge Process.
  2. Get approvals from CODEOWNERS and other reviewers.
  3. Trigger CI tests with comments or contact authorized users to do so.
    • Common commands include /tag-and-rerun-ci, /tag-run-ci-label, /rerun-failed-ci
  4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR.

🤖 Generated with Claude Code


CI States

Latest PR Test (Base): ✅ Run #42141961059
Latest PR Test (Extra): ❌ Run #42141961110
Latest PR Test (AMD ROCm 10): ✅ Run #42141961066

Tip

Base CI only covers the core cases, and its workload may not exercise your change. Use /rerun-test to run the tests your change touches, or copy this prompt for an agent:

Find the registered tests this PR affects and run them with /rerun-test, following https://github.com/sgl-project/sglang/blob/main/.agents/skills/rerun-test/SKILL.md

Out-of-process orchestrators (such as Dynamo's SGLang sidecar) need a few
launch facts to reproduce the Rust server's admission limits and sizing,
which the legacy runtime.v1 GetServerInfo exposed through its raw
server-args dump:

- GetServerInfo: allow_auto_truncate, skip_tokenizer_init, and
  num_reserved_tokens (the speculative draft-token reserve the server
  already uses in its own total-token check).
- GetModelInfo: model_type and architectures; HTTP /model_info now
  carries them too, matching Python's /model_info.
- DeploymentInfo.hicache_host_total_tokens: report the KV host pool's
  realized capacity instead of null; the tree cache is built before the
  Rust server launches.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: jain-ria <riajain@NVIDIA.com>
@jain-ria

Copy link
Copy Markdown
Contributor Author

/tag-and-rerun-ci

jain-ria added a commit to ai-dynamo/dynamo that referenced this pull request Oct 27, 2026
The sidecar README and the Fern SGLang pages say the sidecar talks to
SGLang's Rust server. The Kubernetes notes name the SGLang build it needs
(sgl-project/sglang#45859 and #45883, plus #46635 for an in-process
--sidecar), prefill's bootstrap on its HTTP port and KV events from
discovery. The support matrix lists DP>1 as not yet supported, replacing
the multi-node DP follower section and its diagram.

The Mocker and testkit READMEs describe their api.v1 behavior, the proto
README lists the source PRs after sgl-project/sglang#45996 was split, and
the Slime example notes that it needs SGLang's Python server.

Part of #17218

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: jain-ria <riajain@NVIDIA.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: jain-ria <riajain@NVIDIA.com>
@jain-ria

Copy link
Copy Markdown
Contributor Author

/rerun-failed-ci

@jain-ria

Copy link
Copy Markdown
Contributor Author

/rerun-failed-ci

jain-ria and others added 2 commits October 28, 2026 09:19
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: jain-ria <riajain@NVIDIA.com>
@jain-ria

Copy link
Copy Markdown
Contributor Author

/rerun-failed-ci

3 similar comments
@jain-ria

Copy link
Copy Markdown
Contributor Author

/rerun-failed-ci

@jain-ria

Copy link
Copy Markdown
Contributor Author

/rerun-failed-ci

@jain-ria

Copy link
Copy Markdown
Contributor Author

/rerun-failed-ci

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: jain-ria <riajain@NVIDIA.com>
@jain-ria

Copy link
Copy Markdown
Contributor Author

/rerun-failed-ci

@github-actions

This comment has been minimized.

@jain-ria

jain-ria commented Nov 2, 2026

Copy link
Copy Markdown
Contributor Author

/rerun-failed-ci

@jain-ria

jain-ria commented Nov 3, 2026

Copy link
Copy Markdown
Contributor Author

/rerun-failed-ci

1 similar comment
@jain-ria

jain-ria commented Nov 4, 2026

Copy link
Copy Markdown
Contributor Author

/rerun-failed-ci

@jain-ria

jain-ria commented Nov 4, 2026

Copy link
Copy Markdown
Contributor Author

/rerun-failed-ci

1 similar comment
@jain-ria

jain-ria commented Nov 5, 2026

Copy link
Copy Markdown
Contributor Author

/rerun-failed-ci

jain-ria and others added 2 commits November 5, 2026 09:59
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: jain-ria <riajain@NVIDIA.com>
@jain-ria

jain-ria commented Nov 6, 2026

Copy link
Copy Markdown
Contributor Author

/rerun-failed-ci

3 similar comments
@jain-ria

jain-ria commented Nov 9, 2026

Copy link
Copy Markdown
Contributor Author

/rerun-failed-ci

@jain-ria

jain-ria commented Nov 9, 2026

Copy link
Copy Markdown
Contributor Author

/rerun-failed-ci

@jain-ria

jain-ria commented Nov 9, 2026

Copy link
Copy Markdown
Contributor Author

/rerun-failed-ci

@ishandhanani
ishandhanani merged commit e85fac4 into sgl-project:main Nov 10, 2026
388 of 436 checks passed
jain-ria added a commit to jain-ria/sglang that referenced this pull request Nov 16, 2026
Use 0.1.1 for the release that adds the sgl-project#45859 fields.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: jain-ria <riajain@NVIDIA.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

run-ci CI: run the baseline test suite on this PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants