You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
fix(recipes): drop --prefill-round-robin-balance removed in SGLang 0.5.20 - #18437
The DeepSeek-R1 SGLang disaggregated recipe (recipes/deepseek-r1/sglang/disagg-8gpu) does not become Ready with the Dynamo 1.7.0 RC3 SGLang runtime. The decode worker exits at startup with:
The decode worker passes its arguments to SGLang's own argument parser. SGLang 0.5.20 removed --prefill-round-robin-balance, and the 1.7.0 runtime ships SGLang 0.5.23, so the parser rejects it. SGLang 0.5.14 through 0.5.19 only printed a deprecation warning for it. The defect is in Dynamo's recipes, which still pass a flag that upstream SGLang removed.
Reproduction
Without the fix
With the fix
Note
Exact failure on the reported setup
Not run
—
Needs 16 H200 GPUs in a Kubernetes cluster with the 1.7.0 RC3 image; the coding run had one A100 and no cluster.
Exact failure on a similar setup
❌ Exact failure
✅ Passed
A live Dynamo prefill/decode pair on one A100 with SGLang 0.5.22: with the flag, decode exited at argument parsing and the model never registered; without it, the pair registered and answered a chat request.
Exact error in a test
❌ Exact error
✅ Passed
Dynamo's real SGLang argument parser on the DeepSeek-R1 8-GPU recipe's decode arguments, at the ticket's source commit and at main, rejected the flag with the ticket's message; after the change both workers parsed.
Overview
Remove the obsolete decode flag so the DeepSeek-R1 SGLang disaggregated recipe can start with newer SGLang runtimes. Apply the same removal to the other recipes and the profiler fixture that still pass it.
Key Changes
Decode workers in the SGLang disaggregated recipes no longer get --prefill-round-robin-balance. Prefill workers keep --load-balance-method round_robin. Image tags and all other arguments are unchanged.
Case
Before
After
DeepSeek-R1, DeepSeek-V4.1-Flash, DeepSeek-V4-Pro B200 recipes on SGLang 0.5.20 or newer
Decode exits on the unknown flag
Decode arguments parse
Any edited recipe on its pinned image (SGLang 0.5.19 or older)
Deprecation warning, flag ignored
Unchanged, without the warning
GLM-5 NVFP4 and DeepSeek-V4-Pro GB200 recipes on SGLang 0.5.20 or newer
Fail on other removed flags
Unchanged: still fail
It also removes the flag from a profiler test fixture copied from the DeepSeek-R1 recipe, and a stale comment about it in examples/backends/sglang/launch/disagg.sh.
Not covered: the GLM-5 NVFP4 and DeepSeek-V4-Pro GB200 recipes also pass --cuda-graph-max-bs, and GLM-5 passes --nsa-decode-backend and --nsa-prefill-backend. SGLang 0.5.20 removed or split these too, so those recipes need a separate fix for 1.7.0. The release/1.7.0 cherry-pick follows after this merges.
Core change: about 9 lines (+0/−9). The logic is in 9 files, in reading order:
The rest of the +0/−12 diff is one test fixture line and a stale comment in an example launch script.
Validation
This PR adds no tests. It changes one test fixture, components/src/dynamo/profiler/tests/data/configs/9a_thorough_dsr1_sglang_overrides.yaml, which is used by the existing test_9a_sglang_overrides in components/src/dynamo/profiler/tests/integration/test_profile_sla_dgdr.py; it runs without a GPU.
Behavior checked
Checked by
Without the fix
With the fix
The profiler still forwards the DeepSeek-R1 SGLang overrides
Existing unit test with a changed fixture, plus a parser run over the fixture's override arguments
✅ Passed; the override arguments failed to parse with the removed flag
✅ Passed; the override arguments parse
The other DeepSeek-R1, DeepSeek-V4.1-Flash and DeepSeek-V4-Pro B200 decode arguments start
Parser run with Dynamo's real SGLang parser
❌ Exact error
✅ Passed
The DeepSeek-V4-Pro GB200 decode arguments no longer carry the removed flag
Parser run, with the bare --cuda-graph-max-bs mapped to its decode form in memory
❌ Exact error
✅ Passed
The GLM-5 NVFP4 decode arguments no longer carry the removed flag
Parser run, with the same in-memory mapping
❌ Exact error
❌ Failed: still stops on --nsa-decode-backend and --nsa-prefill-backend, which SGLang also removed in 0.5.20; this change does not touch them
Prefill arguments, which the change leaves alone, still parse
Parser run over the DeepSeek-R1, DeepSeek-V4.1-Flash and DeepSeek-V4-Pro B200 recipes
The other removed SGLang flags and the release/1.7.0 cherry-pick are separate follow-up work, as described above.
Pull Request Checklist
I used Dynamo's /pr-checklist skill.
Summary by CodeRabbit
Configuration Updates
SGLang disaggregated deployment configurations no longer pass the prefill round-robin balance option to decode workers across the affected DeepSeek and GLM recipes.
Prefill workers retain their existing round-robin load-balancing configuration.
…5.20
SGLang 0.5.10.post1 through 0.5.19 accept the flag only as a deprecated
no-op, and SGLang 0.5.20 removed it. On the 1.6.x and 1.7.0 runtimes the
dynamo.sglang decode worker exits during argument parsing with
'unrecognized arguments: --prefill-round-robin-balance'.
Remove the flag from every SGLang disagg recipe, regenerate the
deepseek-v4.1-flash kustomize outputs, update the profiler fixture that
is derived from the DeepSeek-R1 8-GPU recipe, and drop a stale comment
in the SGLang disagg launch example. Image tags and prefill arguments
are unchanged.
Refs DYN-5517
Signed-off-by: svc-glamr@nvidia.com <svc-glamr@nvidia.com>
👋 Hi glamr-agent! Thank you for contributing to ai-dynamo/dynamo.
Just a reminder: The NVIDIA Test Github Validation CI runs an essential subset of the testing framework to quickly catch errors.Your PR reviewers may elect to test the changes comprehensively before approving your changes.
Runs the relevant Python unit tests without requiring a GPU.
Passed (exit 0)
Dynamo's real SGLang 0.5.22 argument parser over every changed manifest, before and after the change; test_9a_sglang_overrides with the changed profiler fixture
Inspects the changed code when the claim cannot be tested with a local command.
Passed (exit 0)
Source scan of the published SGLang 0.5.14, 0.5.19, 0.5.20 and 0.5.23 wheels for the removed flag
Starts separate SGLang prefill and decode workers and checks a disaggregated request.
Passed (exit 0)
One A100, SGLang 0.5.22, Qwen/Qwen3-0.6B, dp=1: decode exits on the flag; without it the pair serves a chat request with HTTP 200
Main gap: the 16-GPU H200 deployment with the 1.7.0 RC3 image was not run, and SGLang 0.5.23 was checked from wheel source only.
Detailed evidence — validation complete
Left out of the table because CI repeats them: building the changed source and importing its compiled extension (Check 1), lint and formatting (Check 2).
Evidence tally: [5/5 validated]
Advisory AI assessment: sound. No blocking findings. One note: the GLM-5 NVFP4 and DeepSeek-V4-Pro GB200 recipes still fail to parse on SGLang 0.5.20 or newer because of other removed or split flags; this change does not address them.
Validation result: pass — the real SGLang parser and a live one-GPU Dynamo pair both show the ticket's unrecognized arguments: --prefill-round-robin-balance failure without the change and a clean start and HTTP 200 with it; wheel scans show the flag was a no-op on every pinned image; the 16-GPU H200 run was not possible here, and three edited recipes (DeepSeek-V4-Pro GB200, both GLM-5) still need separate fixes for other flags SGLang removed in 0.5.20
Evidence audit: complete [5/5 validated] — the command report comes from recorded runs.
Commands and results [5/5 validated]
Generated from the commands recorded during this run.
Check 1
Builds the changed Dynamo source and confirms that Python can import its compiled extension.
Result: Passed (exit 0)
Command:
Not shown because the exact command contained private run data.
Check 2
Checks the changed files with the repository's fast lint and formatting commands.
Result: Passed (exit 0)
Command:
Not shown because the exact command contained private run data.
Check 3
Runs the relevant Python unit tests without requiring a GPU.
Result: Passed (exit 0)
Command:
Not shown because the exact command contained private run data.
Check 4
Inspects the changed code when the claim cannot be tested with a local command.
Result: Passed (exit 0)
Command:
Not shown because the exact command contained private run data.
Check 5
Starts separate SGLang prefill and decode workers and checks a disaggregated request.
Result: Passed (exit 0)
Command:
Not shown because the exact command contained private run data.
No actionable comments were generated in the recent review.
Walkthrough
SGLang decode configurations and deployment commands no longer pass --prefill-round-robin-balance. The launch script also removes a comment about round-robin prefill balancing. Prefill worker settings remain unchanged where noted.
Changes
SGLang decode flag removal
Layer / File(s)
Summary
Remove the decode balance flag components/src/dynamo/profiler/tests/data/configs/*, examples/backends/sglang/launch/disagg.sh, recipes/deepseek-r1/sglang/*/deploy.yaml, recipes/deepseek-v4.1-flash/sglang/disagg-gb200/..., recipes/deepseek-v4/deepseek-v4-pro/sglang/*/deploy.yaml, recipes/glm-5-nvfp4/sglang/disagg/...
The profiler override and SGLang deployment commands no longer include --prefill-round-robin-balance. The launch script removes a comment about round-robin prefill balancing.
This change removes a decode worker flag that newer SGLang versions reject, so the affected recipes should now start. No merge-blocking risk remains. The author did not run the full 16-GPU deployment, and some recipes still need separate fixes for other flags.
Pre-merge checks: 4
Check name
Status
Explanation
Title check
The title names the main change: removing the --prefill-round-robin-balance flag from SGLang recipes because SGLang 0.5.20 removed it. It is short, specific, and matches the changeset, which removes…
Linked Issues check
Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check
Check skipped because no linked issues were found for this pull request.
Description check
The description follows the repository template closely. It has all four required sections (Overview, Key Changes, Validation, Issues and Ongoing Work), a linked issue (Refs DYN-5517), and a checked P…
Full details: Title check
Explanation
The title names the main change: removing the --prefill-round-robin-balance flag from SGLang recipes because SGLang 0.5.20 removed it. It is short, specific, and matches the changeset, which removes that flag from decode arguments across the recipes, a profiler fixture, and a stale comment.
Full details: Description check
Explanation
The description follows the repository template closely. It has all four required sections (Overview, Key Changes, Validation, Issues and Ongoing Work), a linked issue (Refs DYN-5517), and a checked PR checklist item. It explains the failure, the scope of the change, and the validation performed. It is thorough. The author states plainly what was not verified: the exact 16-H200 deployment with the 1.7.0 RC3 image was not run, and the GLM-5 NVFP4 and DeepSeek-V4-Pro GB200 recipes still fail on other removed flags. The 'Closes #XXXX' line from the template is replaced by 'Refs DYN-5517', which is acceptable because the PR issue link check is advisory on this branch. The sections use some non-template content (a reproduction table and a validation table), but they carry the required information. The description is mostly complete, so the check passes.
Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 8 remain after this review.
nursery: I updated the PR description. This changes only GitHub metadata, so there is no new commit. The source stays at 510e37de33136cf916e23e4cb0c83d4a39274224.
This addresses the description check in CodeRabbit's summary. I added Overview, Key Changes, Issues and Ongoing Work, and the Pull Request Checklist from .github/pull_request_template.md. I kept the first line and the validation evidence. I kept the non-closing reference to DYN-5517 because the description still lists release follow-up work.
Absorbs 19 upstream commits (c5680e5..3a8226a). One conflict, the root Cargo.toml: kept upstream's dynamo-tokenizers 4.1.2, dynamo-renderer 9.0.0 and dynamo-protocols 8.5.0 bumps together with the stack's workspace members and path crates; the four lockfiles were regenerated with cargo metadata (no drift from the auto-merge).
- ai-dynamo#17923 perf(kv-router) no block-lookup growth for existing-key updates: lands in lib/router/kv-indexer/src/concurrent_radix_tree_compressed/block_lookup.rs (rename-detected; the stack copy differs from upstream only by the crate::prelude import).
- ai-dynamo#17741 build(deps) dynamo-tokenizers 4.1.2, dynamo-renderer 9.0.0, tokenizers 0.23.2 in lib/llm and lib/bench: root Cargo.toml conflict resolved as above; lib/llm/tests/tokenizer_cache.rs.
- ai-dynamo#17520 feat(vllm) realtime text tools, held-request prefill, chat completions and sampling options: Python vllm, dynamo-protocols 8.5.0 bump.
- ai-dynamo#18380 perf(bindings) direct token_ids/extra_args reads in RoutedEngine.generate and ai-dynamo#18448 refactor(bindings) py_convert: new lib/bindings/python/rust/llm/py_convert.rs, routed_engine.rs, engine.rs, _convert_preprocessed_request registered in lib.rs, _core.pyi, tests/test_preprocessed_request_conversion.py; llm/kv.rs extract_token_ids now delegates to py_convert::read_exact_u32_list with the stack's owner-crate imports kept.
- ai-dynamo#18454 fix(frontend) named passthrough fields to Python chat processors: lib/llm protocols/openai/chat_completions.rs, Python sglang and vllm processors.
- ai-dynamo#18253 fix(frontend) 16-bit image narrowing and EXIF opt-in: lib/llm preprocessor/media decoders, image_reader, loader.
- ai-dynamo#18185 fix(frontend) rendering context in speculative prefill: lib/llm preprocessor/speculative_prefill.rs.
- ai-dynamo#18428 fix(frontend) image step count without accounting: lib/llm protocols/openai/images aggregator.rs and nvext.rs.
- ai-dynamo#17656 fix(frontend) recipe-based DeepSeek encoding with SGLang: Python frontend sglang_deepseek.py, sglang_prompt_renderer.py, sglang_prepost.py, sglang_processor.py.
- ai-dynamo#18378 fix(runtime) TCP_NODELAY on accepted request-plane connections: lib/runtime pipeline/network/ingress/shared_tcp_endpoint.rs.
- ai-dynamo#18007 test(router) routed-worker assertion for cache-salt isolation repeats: tests/router/common.py and helper.py.
- ai-dynamo#18452 feat(backend) embedded engines skip Dynamo logging setup, ai-dynamo#18170 feat(trtllm) per-request metrics opt-in, ai-dynamo#18135 and ai-dynamo#16164 fix(vllm), ai-dynamo#16879 and ai-dynamo#18437 fix(recipes), c5680e5 docs community events: Python, recipes, docs; all auto-merged.
No change to the router kernel, admission, worker-selection, slot-tracker, session-affinity, kv-events or kv-hints crates; no [features] table or Dynamo path-crate dependency edge changed.
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Failure behavior
The DeepSeek-R1 SGLang disaggregated recipe (
recipes/deepseek-r1/sglang/disagg-8gpu) does not become Ready with the Dynamo 1.7.0 RC3 SGLang runtime. The decode worker exits at startup with:The decode worker passes its arguments to SGLang's own argument parser. SGLang 0.5.20 removed
--prefill-round-robin-balance, and the 1.7.0 runtime ships SGLang 0.5.23, so the parser rejects it. SGLang 0.5.14 through 0.5.19 only printed a deprecation warning for it. The defect is in Dynamo's recipes, which still pass a flag that upstream SGLang removed.main, rejected the flag with the ticket's message; after the change both workers parsed.Overview
Remove the obsolete decode flag so the DeepSeek-R1 SGLang disaggregated recipe can start with newer SGLang runtimes. Apply the same removal to the other recipes and the profiler fixture that still pass it.
Key Changes
Decode workers in the SGLang disaggregated recipes no longer get
--prefill-round-robin-balance. Prefill workers keep--load-balance-method round_robin. Image tags and all other arguments are unchanged.It also removes the flag from a profiler test fixture copied from the DeepSeek-R1 recipe, and a stale comment about it in
examples/backends/sglang/launch/disagg.sh.Not covered: the GLM-5 NVFP4 and DeepSeek-V4-Pro GB200 recipes also pass
--cuda-graph-max-bs, and GLM-5 passes--nsa-decode-backendand--nsa-prefill-backend. SGLang 0.5.20 removed or split these too, so those recipes need a separate fix for 1.7.0. Therelease/1.7.0cherry-pick follows after this merges.Validation
This PR adds no tests. It changes one test fixture,
components/src/dynamo/profiler/tests/data/configs/9a_thorough_dsr1_sglang_overrides.yaml, which is used by the existingtest_9a_sglang_overridesincomponents/src/dynamo/profiler/tests/integration/test_profile_sla_dgdr.py; it runs without a GPU.--cuda-graph-max-bsmapped to its decode form in memory--nsa-decode-backendand--nsa-prefill-backend, which SGLang also removed in 0.5.20; this change does not touch themIssues and Ongoing Work
Refs DYN-5517.
The other removed SGLang flags and the
release/1.7.0cherry-pick are separate follow-up work, as described above.Pull Request Checklist
/pr-checklistskill.Summary by CodeRabbit