Skip to content

fix(recipes): drop --prefill-round-robin-balance removed in SGLang 0.5.20 - #18437

Merged
yunzhoul-nv merged 1 commit into
ai-dynamo:mainfrom
glamr-agent:dyn-5517-deepseek-r1-sglang-disagg-recipe-removed-flag-708aa9013dcc
Nov 9, 2026
Merged

yunzhoul-nv merged 1 commit into
ai-dynamo:mainfrom
glamr-agent:dyn-5517-deepseek-r1-sglang-disagg-recipe-removed-flag-708aa9013dcc

Conversation

@glamr-agent

@glamr-agent glamr-agent commented Nov 9, 2026 •

Copy link
Copy Markdown
Collaborator

Failure behavior

The DeepSeek-R1 SGLang disaggregated recipe (recipes/deepseek-r1/sglang/disagg-8gpu) does not become Ready with the Dynamo 1.7.0 RC3 SGLang runtime. The decode worker exits at startup with:

__main__.py: error: unrecognized arguments: --prefill-round-robin-balance

The decode worker passes its arguments to SGLang's own argument parser. SGLang 0.5.20 removed --prefill-round-robin-balance, and the 1.7.0 runtime ships SGLang 0.5.23, so the parser rejects it. SGLang 0.5.14 through 0.5.19 only printed a deprecation warning for it. The defect is in Dynamo's recipes, which still pass a flag that upstream SGLang removed.

Reproduction Without the fix With the fix Note
Exact failure on the reported setup Not run — Needs 16 H200 GPUs in a Kubernetes cluster with the 1.7.0 RC3 image; the coding run had one A100 and no cluster.
Exact failure on a similar setup ❌ Exact failure ✅ Passed A live Dynamo prefill/decode pair on one A100 with SGLang 0.5.22: with the flag, decode exited at argument parsing and the model never registered; without it, the pair registered and answered a chat request.
Exact error in a test ❌ Exact error ✅ Passed Dynamo's real SGLang argument parser on the DeepSeek-R1 8-GPU recipe's decode arguments, at the ticket's source commit and at main, rejected the flag with the ticket's message; after the change both workers parsed.

Overview

Remove the obsolete decode flag so the DeepSeek-R1 SGLang disaggregated recipe can start with newer SGLang runtimes. Apply the same removal to the other recipes and the profiler fixture that still pass it.

Key Changes

Decode workers in the SGLang disaggregated recipes no longer get --prefill-round-robin-balance. Prefill workers keep --load-balance-method round_robin. Image tags and all other arguments are unchanged.

Case Before After
DeepSeek-R1, DeepSeek-V4.1-Flash, DeepSeek-V4-Pro B200 recipes on SGLang 0.5.20 or newer Decode exits on the unknown flag Decode arguments parse
Any edited recipe on its pinned image (SGLang 0.5.19 or older) Deprecation warning, flag ignored Unchanged, without the warning
GLM-5 NVFP4 and DeepSeek-V4-Pro GB200 recipes on SGLang 0.5.20 or newer Fail on other removed flags Unchanged: still fail

It also removes the flag from a profiler test fixture copied from the DeepSeek-R1 recipe, and a stale comment about it in examples/backends/sglang/launch/disagg.sh.

Not covered: the GLM-5 NVFP4 and DeepSeek-V4-Pro GB200 recipes also pass --cuda-graph-max-bs, and GLM-5 passes --nsa-decode-backend and --nsa-prefill-backend. SGLang 0.5.20 removed or split these too, so those recipes need a separate fix for 1.7.0. The release/1.7.0 cherry-pick follows after this merges.

Core change: about 9 lines (+0/−9). The logic is in 9 files, in reading order:

The rest of the +0/−12 diff is one test fixture line and a stale comment in an example launch script.

Validation

This PR adds no tests. It changes one test fixture, components/src/dynamo/profiler/tests/data/configs/9a_thorough_dsr1_sglang_overrides.yaml, which is used by the existing test_9a_sglang_overrides in components/src/dynamo/profiler/tests/integration/test_profile_sla_dgdr.py; it runs without a GPU.

Behavior checked Checked by Without the fix With the fix
The profiler still forwards the DeepSeek-R1 SGLang overrides Existing unit test with a changed fixture, plus a parser run over the fixture's override arguments ✅ Passed; the override arguments failed to parse with the removed flag ✅ Passed; the override arguments parse
The other DeepSeek-R1, DeepSeek-V4.1-Flash and DeepSeek-V4-Pro B200 decode arguments start Parser run with Dynamo's real SGLang parser ❌ Exact error ✅ Passed
The DeepSeek-V4-Pro GB200 decode arguments no longer carry the removed flag Parser run, with the bare --cuda-graph-max-bs mapped to its decode form in memory ❌ Exact error ✅ Passed
The GLM-5 NVFP4 decode arguments no longer carry the removed flag Parser run, with the same in-memory mapping ❌ Exact error ❌ Failed: still stops on --nsa-decode-backend and --nsa-prefill-backend, which SGLang also removed in 0.5.20; this change does not touch them
Prefill arguments, which the change leaves alone, still parse Parser run over the DeepSeek-R1, DeepSeek-V4.1-Flash and DeepSeek-V4-Pro B200 recipes ✅ Passed ✅ Passed

Issues and Ongoing Work

Refs DYN-5517.

The other removed SGLang flags and the release/1.7.0 cherry-pick are separate follow-up work, as described above.

Pull Request Checklist
  • I used Dynamo's /pr-checklist skill.

Summary by CodeRabbit

  • Configuration Updates
    • SGLang disaggregated deployment configurations no longer pass the prefill round-robin balance option to decode workers across the affected DeepSeek and GLM recipes.
    • Prefill workers retain their existing round-robin load-balancing configuration.

…5.20

SGLang 0.5.10.post1 through 0.5.19 accept the flag only as a deprecated
no-op, and SGLang 0.5.20 removed it. On the 1.6.x and 1.7.0 runtimes the
dynamo.sglang decode worker exits during argument parsing with
'unrecognized arguments: --prefill-round-robin-balance'.

Remove the flag from every SGLang disagg recipe, regenerate the
deepseek-v4.1-flash kustomize outputs, update the profiler fixture that
is derived from the DeepSeek-R1 8-GPU recipe, and drop a stale comment
in the SGLang disagg launch example. Image tags and prefill arguments
are unchanged.

Refs DYN-5517

Signed-off-by: svc-glamr@nvidia.com <svc-glamr@nvidia.com>
@glamr-agent
glamr-agent requested review from a team as code owners November 9, 2026 11:36
@copy-pr-bot

copy-pr-bot Bot commented Nov 9, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@glamr-agent
glamr-agent temporarily deployed to external_collaborator November 9, 2026 11:36 — with GitHub Actions Inactive
@glamr-agent
glamr-agent temporarily deployed to external_collaborator November 9, 2026 11:36 — with GitHub Actions Inactive
@github-actions

github-actions Bot commented Nov 9, 2026

Copy link
Copy Markdown
Contributor

👋 Hi glamr-agent! Thank you for contributing to ai-dynamo/dynamo.

Just a reminder: The NVIDIA Test Github Validation CI runs an essential subset of the testing framework to quickly catch errors.Your PR reviewers may elect to test the changes comprehensively before approving your changes.

🚀

@github-actions github-actions Bot added fix external-contribution Pull request is from an external contributor labels Nov 9, 2026
@glamr-agent

Copy link
Copy Markdown
Collaborator Author

factory: Initial validation snapshot.

Validation status: complete

Check Result Scope or reason
Runs the relevant Python unit tests without requiring a GPU. Passed (exit 0) Dynamo's real SGLang 0.5.22 argument parser over every changed manifest, before and after the change; test_9a_sglang_overrides with the changed profiler fixture
Inspects the changed code when the claim cannot be tested with a local command. Passed (exit 0) Source scan of the published SGLang 0.5.14, 0.5.19, 0.5.20 and 0.5.23 wheels for the removed flag
Starts separate SGLang prefill and decode workers and checks a disaggregated request. Passed (exit 0) One A100, SGLang 0.5.22, Qwen/Qwen3-0.6B, dp=1: decode exits on the flag; without it the pair serves a chat request with HTTP 200

Main gap: the 16-GPU H200 deployment with the 1.7.0 RC3 image was not run, and SGLang 0.5.23 was checked from wheel source only.

Detailed evidence — validation complete

Left out of the table because CI repeats them: building the changed source and importing its compiled extension (Check 1), lint and formatting (Check 2).

Evidence tally: [5/5 validated]

Advisory AI assessment: sound. No blocking findings. One note: the GLM-5 NVFP4 and DeepSeek-V4-Pro GB200 recipes still fail to parse on SGLang 0.5.20 or newer because of other removed or split flags; this change does not address them.

Validation result: pass — the real SGLang parser and a live one-GPU Dynamo pair both show the ticket's unrecognized arguments: --prefill-round-robin-balance failure without the change and a clean start and HTTP 200 with it; wheel scans show the flag was a no-op on every pinned image; the 16-GPU H200 run was not possible here, and three edited recipes (DeepSeek-V4-Pro GB200, both GLM-5) still need separate fixes for other flags SGLang removed in 0.5.20

Evidence audit: complete [5/5 validated] — the command report comes from recorded runs.

Commands and results [5/5 validated]

Generated from the commands recorded during this run.

Check 1

Builds the changed Dynamo source and confirms that Python can import its compiled extension.

Result: Passed (exit 0)

Command:

Not shown because the exact command contained private run data.

Check 2

Checks the changed files with the repository's fast lint and formatting commands.

Result: Passed (exit 0)

Command:

Not shown because the exact command contained private run data.

Check 3

Runs the relevant Python unit tests without requiring a GPU.

Result: Passed (exit 0)

Command:

Not shown because the exact command contained private run data.

Check 4

Inspects the changed code when the claim cannot be tested with a local command.

Result: Passed (exit 0)

Command:

Not shown because the exact command contained private run data.

Check 5

Starts separate SGLang prefill and decode workers and checks a disaggregated request.

Result: Passed (exit 0)

Command:

Not shown because the exact command contained private run data.

@github-actions github-actions Bot added backend::sglang Relates to the sglang backend planner labels Nov 9, 2026
@glamr-agent

Copy link
Copy Markdown
Collaborator Author

No description provided.

@coderabbitai

coderabbitai Bot commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

Explain Change →

No actionable comments were generated in the recent review.

Walkthrough

SGLang decode configurations and deployment commands no longer pass --prefill-round-robin-balance. The launch script also removes a comment about round-robin prefill balancing. Prefill worker settings remain unchanged where noted.

Changes

SGLang decode flag removal

Layer / File(s) Summary
Remove the decode balance flag
components/src/dynamo/profiler/tests/data/configs/*, examples/backends/sglang/launch/disagg.sh, recipes/deepseek-r1/sglang/*/deploy.yaml, recipes/deepseek-v4.1-flash/sglang/disagg-gb200/..., recipes/deepseek-v4/deepseek-v4-pro/sglang/*/deploy.yaml, recipes/glm-5-nvfp4/sglang/disagg/...
The profiler override and SGLang deployment commands no longer include --prefill-round-robin-balance. The launch script removes a comment about round-robin prefill balancing.

Estimated code review effort: 2 (Simple) | ~8 minutes

Merge Risk: Minimal · up to 510e3

This change removes a decode worker flag that newer SGLang versions reject, so the affected recipes should now start. No merge-blocking risk remains. The author did not run the full 16-GPU deployment, and some recipes still need separate fixes for other flags.

Pre-merge checks: Passed 4

Check name Status Explanation
Title check Passed The title names the main change: removing the --prefill-round-robin-balance flag from SGLang recipes because SGLang 0.5.20 removed it. It is short, specific, and matches the changeset, which removes…
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Description check Passed The description follows the repository template closely. It has all four required sections (Overview, Key Changes, Validation, Issues and Ongoing Work), a linked issue (Refs DYN-5517), and a checked P…
Full details: Title check

Explanation

The title names the main change: removing the --prefill-round-robin-balance flag from SGLang recipes because SGLang 0.5.20 removed it. It is short, specific, and matches the changeset, which removes that flag from decode arguments across the recipes, a profiler fixture, and a stale comment.

Full details: Description check

Explanation

The description follows the repository template closely. It has all four required sections (Overview, Key Changes, Validation, Issues and Ongoing Work), a linked issue (Refs DYN-5517), and a checked PR checklist item. It explains the failure, the scope of the change, and the validation performed. It is thorough. The author states plainly what was not verified: the exact 16-H200 deployment with the 1.7.0 RC3 image was not run, and the GLM-5 NVFP4 and DeepSeek-V4-Pro GB200 recipes still fail on other removed flags. The 'Closes #XXXX' line from the template is replaced by 'Refs DYN-5517', which is acceptable because the PR issue link check is advisory on this branch. The sections use some non-template content (a reproduction table and a validation table), but they carry the required information. The description is mostly complete, so the check passes.

  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Recent review info

Run configuration

Commits

Reviewing files that changed from the base of the PR and between 02a6ccb and 510e37d.

Files selected for processing: 11

  • components/src/dynamo/profiler/tests/data/configs/9a_thorough_dsr1_sglang_overrides.yaml
  • examples/backends/sglang/launch/disagg.sh
  • recipes/deepseek-r1/sglang/disagg-16gpu/deploy.yaml
  • recipes/deepseek-r1/sglang/disagg-8gpu/deploy.yaml
  • recipes/deepseek-v4.1-flash/sglang/disagg-gb200/deploy-generic.yaml
  • recipes/deepseek-v4.1-flash/sglang/disagg-gb200/deploy-gke-rdma.yaml
  • recipes/deepseek-v4.1-flash/sglang/disagg-gb200/kustomize/base/deploy.yaml
  • recipes/deepseek-v4/deepseek-v4-pro/sglang/disagg-b200/deploy.yaml
  • recipes/deepseek-v4/deepseek-v4-pro/sglang/disagg-gb200/deploy.yaml
  • recipes/glm-5-nvfp4/sglang/disagg/deploy.yaml
  • recipes/glm-5-nvfp4/sglang/disagg/efa/deploy.yaml
Files with no reviewable changes: 11

  • recipes/deepseek-v4.1-flash/sglang/disagg-gb200/kustomize/base/deploy.yaml
  • recipes/deepseek-r1/sglang/disagg-16gpu/deploy.yaml
  • recipes/deepseek-v4/deepseek-v4-pro/sglang/disagg-gb200/deploy.yaml
  • recipes/deepseek-v4.1-flash/sglang/disagg-gb200/deploy-gke-rdma.yaml
  • examples/backends/sglang/launch/disagg.sh
  • recipes/glm-5-nvfp4/sglang/disagg/deploy.yaml
  • recipes/deepseek-r1/sglang/disagg-8gpu/deploy.yaml
  • recipes/glm-5-nvfp4/sglang/disagg/efa/deploy.yaml
  • components/src/dynamo/profiler/tests/data/configs/9a_thorough_dsr1_sglang_overrides.yaml
  • recipes/deepseek-v4.1-flash/sglang/disagg-gb200/deploy-generic.yaml
  • recipes/deepseek-v4/deepseek-v4-pro/sglang/disagg-b200/deploy.yaml

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 8 remain after this review.

@glamr-agent

Copy link
Copy Markdown
Collaborator Author

factory: @dynamo-ops please run full CI for 510e37d

@dynamo-ops

Copy link
Copy Markdown
Contributor

/ok to test 510e37d

@devin-ai-integration

Copy link
Copy Markdown
Contributor

✅ Dynamo PR CI passed — run 42610675379 (attempt 1) on 510e37de33

Gate checks: ✅ backend-status-check · ✅ deploy-status-check · ✅ dynamo-status-check

Posted automatically by Devin for run 42610675379. Updated on every full-CI run of this PR.

@glamr-agent

Copy link
Copy Markdown
Collaborator Author

nursery: I updated the PR description. This changes only GitHub metadata, so there is no new commit. The source stays at 510e37de33136cf916e23e4cb0c83d4a39274224.

This addresses the description check in CodeRabbit's summary. I added Overview, Key Changes, Issues and Ongoing Work, and the Pull Request Checklist from .github/pull_request_template.md. I kept the first line and the validation evidence. I kept the non-closing reference to DYN-5517 because the description still lists release follow-up work.

@yunzhoul-nv yunzhoul-nv left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, approved.

@yunzhoul-nv
yunzhoul-nv merged commit 1ca6113 into ai-dynamo:main Nov 9, 2026
124 of 125 checks passed
nv-nmailhot pushed a commit that referenced this pull request Nov 9, 2026
…removed in SGLang 0.5.20 (#18437) (#18446)

Signed-off-by: svc-glamr@nvidia.com <svc-glamr@nvidia.com>
Signed-off-by: GLAMR <svc-glamr@nvidia.com>
ishandhanani added a commit to ishandhanani/dynamo that referenced this pull request Nov 10, 2026
Absorbs 19 upstream commits (c5680e5..3a8226a). One conflict, the root Cargo.toml: kept upstream's dynamo-tokenizers 4.1.2, dynamo-renderer 9.0.0 and dynamo-protocols 8.5.0 bumps together with the stack's workspace members and path crates; the four lockfiles were regenerated with cargo metadata (no drift from the auto-merge).

- ai-dynamo#17923 perf(kv-router) no block-lookup growth for existing-key updates: lands in lib/router/kv-indexer/src/concurrent_radix_tree_compressed/block_lookup.rs (rename-detected; the stack copy differs from upstream only by the crate::prelude import).
- ai-dynamo#17741 build(deps) dynamo-tokenizers 4.1.2, dynamo-renderer 9.0.0, tokenizers 0.23.2 in lib/llm and lib/bench: root Cargo.toml conflict resolved as above; lib/llm/tests/tokenizer_cache.rs.
- ai-dynamo#17520 feat(vllm) realtime text tools, held-request prefill, chat completions and sampling options: Python vllm, dynamo-protocols 8.5.0 bump.
- ai-dynamo#18380 perf(bindings) direct token_ids/extra_args reads in RoutedEngine.generate and ai-dynamo#18448 refactor(bindings) py_convert: new lib/bindings/python/rust/llm/py_convert.rs, routed_engine.rs, engine.rs, _convert_preprocessed_request registered in lib.rs, _core.pyi, tests/test_preprocessed_request_conversion.py; llm/kv.rs extract_token_ids now delegates to py_convert::read_exact_u32_list with the stack's owner-crate imports kept.
- ai-dynamo#18454 fix(frontend) named passthrough fields to Python chat processors: lib/llm protocols/openai/chat_completions.rs, Python sglang and vllm processors.
- ai-dynamo#18253 fix(frontend) 16-bit image narrowing and EXIF opt-in: lib/llm preprocessor/media decoders, image_reader, loader.
- ai-dynamo#18185 fix(frontend) rendering context in speculative prefill: lib/llm preprocessor/speculative_prefill.rs.
- ai-dynamo#18428 fix(frontend) image step count without accounting: lib/llm protocols/openai/images aggregator.rs and nvext.rs.
- ai-dynamo#17656 fix(frontend) recipe-based DeepSeek encoding with SGLang: Python frontend sglang_deepseek.py, sglang_prompt_renderer.py, sglang_prepost.py, sglang_processor.py.
- ai-dynamo#18378 fix(runtime) TCP_NODELAY on accepted request-plane connections: lib/runtime pipeline/network/ingress/shared_tcp_endpoint.rs.
- ai-dynamo#18007 test(router) routed-worker assertion for cache-salt isolation repeats: tests/router/common.py and helper.py.
- ai-dynamo#18452 feat(backend) embedded engines skip Dynamo logging setup, ai-dynamo#18170 feat(trtllm) per-request metrics opt-in, ai-dynamo#18135 and ai-dynamo#16164 fix(vllm), ai-dynamo#16879 and ai-dynamo#18437 fix(recipes), c5680e5 docs community events: Python, recipes, docs; all auto-merged.

No change to the router kernel, admission, worker-selection, slot-tracker, session-affinity, kv-events or kv-hints crates; no [features] table or Dynamo path-crate dependency edge changed.

Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>

This branch was previously deployed

1 inactive deployment
external_collaborator — 510e37de Deployed Nov 9, 2026 by glamr-agent via pr_reminder #15354
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backend::sglang Relates to the sglang backend external-contribution Pull request is from an external contributor fix planner size/S

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants