Skip to content

fix(recipes): fix SGLang GB200 frontend args — remove --model-path, fix parser flag - #13324

Merged
chw001 merged 5 commits into
mainfrom
fix/sglang-gb200-frontend-args
Aug 21, 2026
Merged

chw001 merged 5 commits into
mainfrom
fix/sglang-gb200-frontend-args

Conversation

@chw001

@chw001 chw001 commented Aug 15, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Fixed the following in the Qwen3.8-2.4T-A95B-FP8 GB200 SGLang frontend manifests that prevent the deployment from starting.

Issue 1: --model-path rejected by dynamo.frontend

dynamo.frontend requires --model-path to be a valid local directory on disk. The manifests passed the HF model ID (Qwen/Qwen3.8-2.4T-A95B-FP8), causing an immediate exit:

model-path must be a valid directory on disk

Fix: Remove --model-path and --model-name from the frontend. The frontend does not load model weights — only the workers (dynamo.sglang) need the model path.

Issue 2: --dyn-tool-call-parser unknown to dynamo.frontend with sglang chat processor

When using --dyn-chat-processor sglang, the correct flag is --tool-call-parser (no dyn- prefix). --dyn-tool-call-parser is a Dynamo-specific wrapper for the vLLM backend and is rejected as unknown in this context.

Fix: --dyn-tool-call-parser qwen3_coder → --tool-call-parser qwen3_coder

Scope

Affects sglang/agg-gb200-chat/deploy.yaml and sglang/disagg-gb200-chat/deploy.yaml. GB300 SGLang recipes are unaffected (their frontends have neither flag).

Validation

Deployed sanity-sgl-agg on GCP GB200 with the fixes applied. Coherence check passed:

  • content: 1 + 1 = 2 ✅
  • reasoning_content populated ✅
  • model: Qwen/Qwen3.8-2.4T-A95B-FP8 ✅

🤖 Generated with Claude Code


Open in Devin Review

Summary by CodeRabbit

  • Improvements
    • Updated Qwen deployment configurations to use the standard tool-call parser setting.
    • Removed redundant model path and model name options from frontend startup configuration.
    • Improved consistency between aggregated and disaggregated deployment modes.
    • Tool-calling behavior remains configured for the Qwen3 Coder format.

@chw001
chw001 requested a review from a team as a code owner August 15, 2026 23:01
@github-actions github-actions Bot added the fix label Aug 15, 2026

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no bugs or issues to report.

Open in Devin Review

@coderabbitai

coderabbitai Bot commented Aug 15, 2026 •

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: bd3c0072-fa17-47b4-a342-6946f40f3e78

📥 Commits

Reviewing files that changed from the base of the PR and between e147cee and 14ab4bd.

📒 Files selected for processing (2)
  • recipes/qwen3.8-2.4t-a95b/sglang/agg-gb200-chat/deploy.yaml
  • recipes/qwen3.8-2.4t-a95b/sglang/disagg-gb200-chat/deploy.yaml

Included review availability: Your plan includes up to 12 reviews per rolling hour; 11 remain after this review.


Walkthrough

The SGLang GB200 chat deployment configurations update the frontend tool-call parser argument to --tool-call-parser with qwen3_coder. They also remove the frontend --model-path and --model-name arguments.

Changes

SGLang frontend argument updates

Layer / File(s) Summary
Update frontend deployment arguments
recipes/qwen3.8-2.4t-a95b/sglang/agg-gb200-chat/deploy.yaml, recipes/qwen3.8-2.4t-a95b/sglang/disagg-gb200-chat/deploy.yaml
Both deployments replace --dyn-tool-call-parser with --tool-call-parser, retain qwen3_coder, and remove --model-path and --model-name.

Estimated code review effort: 1 (Trivial) | ~5 minutes

Merge Risk: ⚪ Minimal · up to 14ab4

This localized recipe change removes invalid frontend arguments and corrects the tool-parser flag for GB200 SGLang deployments; no actionable merge-blocking risk remains beyond normal checks and review.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Description check ⚠️ Warning The description explains the changes and validation, but it omits the required Related Issues section and the Where should the reviewer start section. Add the required Related Issues section with an issue link or no-issue confirmation, and identify the files where reviewers should start.
✅ Passed checks (4 passed)
Check name Status Explanation
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Title check ✅ Passed The title clearly identifies the SGLang GB200 frontend argument fixes and matches the main changes.
✨ Finishing Touches 💡 1
⚔️ Resolve merge conflicts 💡
  • Resolve merge conflict in branch fix/sglang-gb200-frontend-args

Comment @coderabbitai help to get the list of available commands.

@pull-request-size pull-request-size Bot added size/M and removed size/S labels Aug 18, 2026
…ix parser flag

Post-merge validation (NVBug 6610056) found two blockers in the GB200
SGLang frontend:

1. --model-path Qwen/Qwen3.8-2.4T-A95B-FP8: dynamo.frontend requires a
   valid local directory; HF IDs are rejected. Removed: the frontend
   does not load model weights.

2. --dyn-tool-call-parser qwen3_coder: unknown to dynamo.frontend when
   using --dyn-chat-processor sglang. Changed to --tool-call-parser
   qwen3_coder per the sglang pre-processor convention.

Validated on GCP GB200 cluster: both fixes confirmed working, coherence
check passed with correct reasoning_content.

GB300 recipes unaffected: their frontends have neither flag.

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: Cheng Wang <chengwa@nvidia.com>
@chw001
chw001 force-pushed the fix/sglang-gb200-frontend-args branch from 2fea0f2 to 22561c9 Compare August 19, 2026 07:20
@pull-request-size pull-request-size Bot added size/S and removed size/M labels Aug 19, 2026
num_speculative_tokens=1 is not a valid workaround: when the single
draft token is accepted, num_accepted_tokens = draft + target = 2 > 1,
which still triggers the DS conv state assertion. Remove
--speculative-config entirely until vLLM PR#49291 ships in the image.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Signed-off-by: Cheng Wang <chengwa@nvidia.com>
@chw001
chw001 force-pushed the fix/sglang-gb200-frontend-args branch from f5e08a6 to f3b14f9 Compare August 19, 2026 23:42
ynpandey-nv and others added 2 commits August 20, 2026 14:15
…3.8 README

- Wait and port-forward commands used qwen38max-agg / qwen38max-agg-frontend
  (GB200 DGD names); correct to qwen38max-agg-chat / qwen38max-agg-chat-frontend
  which match the agg-gb300-chat deploy.yaml
- Configuration note incorrectly stated the frontend does not mount the PVC and
  referenced envFrom: hf-token-secret; both frontend and workers mount model-cache,
  and HF_HUB_OFFLINE=1 means no token lookup at runtime

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Signed-off-by: Cheng Wang <chengwa@nvidia.com>
@github-actions github-actions Bot added the documentation Improvements or additions to documentation label Aug 21, 2026
@github-actions

github-actions Bot commented Aug 21, 2026 •

Copy link
Copy Markdown
Contributor

…8 README

HF_HOME, HF_HUB_OFFLINE, and TRANSFORMERS_OFFLINE are set on workers only;
the frontend mounts the PVC but does not set these variables.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Signed-off-by: Cheng Wang <chengwa@nvidia.com>
@chw001
chw001 merged commit cfb9622 into main Aug 21, 2026
101 checks passed
@chw001
chw001 deleted the fix/sglang-gb200-frontend-args branch August 21, 2026 16:53
ynpandey-nv added a commit that referenced this pull request Aug 21, 2026
…ix parser flag (#13324)

Signed-off-by: Cheng Wang <chengwa@nvidia.com>
Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: Yogendra Pandey <20666257+ynpandey-nv@users.noreply.github.com>
Signed-off-by: Yogendra Pandey <ypandey@nvidia.com>
shwetha-krishnamurthy added a commit to shwetha-krishnamurthy/dynamo that referenced this pull request Aug 24, 2026
Rebases onto current main and addresses the review on ai-dynamo#13661.

Reviewer comments:

- Landing page reports 88 deployable configurations, not 92. ai-dynamo#13370 cut
  Nemotron 3.5 Lightning from 30 targets to 26; summing catalog targets
  across the 18 carded recipes now gives exactly 88.
- Nemotron 3.5 Lightning is reconciled against the post-ai-dynamo#13370 recipe
  tree. The rebase takes upstream's page as the base, so every BF16
  checkpoint, `-bf16` path and resource name is upstream's; this commit
  re-applies only the structural work: the picker moves from a fused
  backend/topology row to GPU -> Workload -> Backend -> Topology, with
  Disaggregated gated on vLLM because TensorRT-LLM ships no such target.
- GLM-5.2 GPU counts now state per-worker GPUs, worker counts and the
  deployment total. Checking all 91 catalog targets against their
  manifests showed the reviewer's `sglang-disagg-b200` (12 -> 20) was one
  of three: `sglang-agg-b200` 4 -> 16 and `sglang-agg-h200` 8 -> 24 were
  wrong the same way. The landing card repeated the same four numbers.
- Nemotron-3-Super B200 chat and agentic summaries now carry the drafter
  MoE backend. Only those two manifests set `moe_backend=triton` in the
  MTP draft loop; both H200 manifests set no override, so they are left
  alone.

Post-rebase drift, from commits that landed after this branch was written:

- ai-dynamo#13324 dropped `--speculative-config` from qwen3.8 `vllm/agg-gb200-chat`,
  so the shared GB200 vLLM summary, the supported-features bullet and the
  catalog technique list no longer claim MTP for chat. Parser wiring and
  the model-resolution note are corrected to match the rewritten README.
- ai-dynamo#13125 deleted `recipes/kustomize/components/aws-efa-p8d8/` and moved AWS
  EFA to a shared template bundle. qwen3-235b-a22b-fp8 pointed at the dead
  path and claimed 8 EFA per worker; the manifest requests 4 on decode and
  2 on each prefill. Both pages' contributor sections now describe the
  template model.

Reported by review bots and confirmed against the sources:

- RecipeStyles.tsx: the tab-order companion block enumerated 2 of 7
  `recipe-variant` values, so for the other five a hidden label kept a
  focusable radio. All four dimensions now match their label-hiding rules.
  Two rulesets whose bodies held only a comment are removed.
- gpt-oss-120b step 2 applies the PVC; step 3 applies the download Job,
  matching the README instead of applying the whole directory.
- kimi-k2-6 fetches the Git LFS trace objects before staging them.
- nemotron-3-ultra no longer says "skip to step 3", which collided with
  the page's own step 3.
- qwen3-32b-fp8 opens on the catalog's recommended 2-GPU aggregated
  target, with the hint on the row that distinguishes it.

Validation: catalog validator and check_asset_paths pass; all 91 catalog
counts match their manifests; 18/18 pages keep the four-level picker order
with 0 dead ends; div balance, fences, frontmatter, SPDX, casing clean.

Not applied, with reasons in the PR discussion: the DeepSeek helper-pod and
"~25x vs ~20x" comments both mirror their recipe READMEs verbatim, so
changing only the pages would create the drift this PR removes.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: shwetha-krishnamurthy <shwetha.krishnamurthy27@gmail.com>
shwetha-krishnamurthy added a commit to shwetha-krishnamurthy/dynamo that referenced this pull request Aug 24, 2026
Rebases onto current main and addresses the review on ai-dynamo#13661.

Reviewer comments:

- Landing page reports 88 deployable configurations, not 92. ai-dynamo#13370 cut
  Nemotron 3.5 Lightning from 30 targets to 26; summing catalog targets
  across the 18 carded recipes now gives exactly 88.
- Nemotron 3.5 Lightning is reconciled against the post-ai-dynamo#13370 recipe
  tree. The rebase takes upstream's page as the base, so every BF16
  checkpoint, `-bf16` path and resource name is upstream's; this commit
  re-applies only the structural work: the picker moves from a fused
  backend/topology row to GPU -> Workload -> Backend -> Topology, with
  Disaggregated gated on vLLM because TensorRT-LLM ships no such target.
- GLM-5.2 GPU counts now state per-worker GPUs, worker counts and the
  deployment total. Checking all 91 catalog targets against their
  manifests showed the reviewer's `sglang-disagg-b200` (12 -> 20) was one
  of three: `sglang-agg-b200` 4 -> 16 and `sglang-agg-h200` 8 -> 24 were
  wrong the same way. The landing card repeated the same four numbers.
- Nemotron-3-Super B200 chat and agentic summaries now carry the drafter
  MoE backend. Only those two manifests set `moe_backend=triton` in the
  MTP draft loop; both H200 manifests set no override, so they are left
  alone.

Post-rebase drift, from commits that landed after this branch was written:

- ai-dynamo#13324 dropped `--speculative-config` from qwen3.8 `vllm/agg-gb200-chat`,
  so the shared GB200 vLLM summary, the supported-features bullet and the
  catalog technique list no longer claim MTP for chat. Parser wiring and
  the model-resolution note are corrected to match the rewritten README.
- ai-dynamo#13125 deleted `recipes/kustomize/components/aws-efa-p8d8/` and moved AWS
  EFA to a shared template bundle. qwen3-235b-a22b-fp8 pointed at the dead
  path and claimed 8 EFA per worker; the manifest requests 4 on decode and
  2 on each prefill. Both pages' contributor sections now describe the
  template model.

Reported by review bots and confirmed against the sources:

- RecipeStyles.tsx: the tab-order companion block enumerated 2 of 7
  `recipe-variant` values, so for the other five a hidden label kept a
  focusable radio. All four dimensions now match their label-hiding rules.
  Two rulesets whose bodies held only a comment are removed.
- gpt-oss-120b step 2 applies the PVC; step 3 applies the download Job,
  matching the README instead of applying the whole directory.
- kimi-k2-6 fetches the Git LFS trace objects before staging them.
- nemotron-3-ultra no longer says "skip to step 3", which collided with
  the page's own step 3.
- qwen3-32b-fp8 opens on the catalog's recommended 2-GPU aggregated
  target, with the hint on the row that distinguishes it.

Validation: catalog validator and check_asset_paths pass; all 91 catalog
counts match their manifests; 18/18 pages keep the four-level picker order
with 0 dead ends; div balance, fences, frontmatter, SPDX, casing clean.

Not applied, with reasons in the PR discussion: the DeepSeek helper-pod and
"~25x vs ~20x" comments both mirror their recipe READMEs verbatim, so
changing only the pages would create the drift this PR removes.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: shwetha-krishnamurthy <shwetha.krishnamurthy27@gmail.com>
ynpandey-nv added a commit that referenced this pull request Aug 25, 2026
…ix parser flag (#13324)

Signed-off-by: Cheng Wang <chengwa@nvidia.com>
Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: Yogendra Pandey <20666257+ynpandey-nv@users.noreply.github.com>
Signed-off-by: Yogendra Pandey <ypandey@nvidia.com>
shwetha-krishnamurthy added a commit to shwetha-krishnamurthy/dynamo that referenced this pull request Aug 27, 2026
Rebases onto current main and addresses the review on ai-dynamo#13661.

Reviewer comments:

- Landing page reports 88 deployable configurations, not 92. ai-dynamo#13370 cut
  Nemotron 3.5 Lightning from 30 targets to 26; summing catalog targets
  across the 18 carded recipes now gives exactly 88.
- Nemotron 3.5 Lightning is reconciled against the post-ai-dynamo#13370 recipe
  tree. The rebase takes upstream's page as the base, so every BF16
  checkpoint, `-bf16` path and resource name is upstream's; this commit
  re-applies only the structural work: the picker moves from a fused
  backend/topology row to GPU -> Workload -> Backend -> Topology, with
  Disaggregated gated on vLLM because TensorRT-LLM ships no such target.
- GLM-5.2 GPU counts now state per-worker GPUs, worker counts and the
  deployment total. Checking all 91 catalog targets against their
  manifests showed the reviewer's `sglang-disagg-b200` (12 -> 20) was one
  of three: `sglang-agg-b200` 4 -> 16 and `sglang-agg-h200` 8 -> 24 were
  wrong the same way. The landing card repeated the same four numbers.
- Nemotron-3-Super B200 chat and agentic summaries now carry the drafter
  MoE backend. Only those two manifests set `moe_backend=triton` in the
  MTP draft loop; both H200 manifests set no override, so they are left
  alone.

Post-rebase drift, from commits that landed after this branch was written:

- ai-dynamo#13324 dropped `--speculative-config` from qwen3.8 `vllm/agg-gb200-chat`,
  so the shared GB200 vLLM summary, the supported-features bullet and the
  catalog technique list no longer claim MTP for chat. Parser wiring and
  the model-resolution note are corrected to match the rewritten README.
- ai-dynamo#13125 deleted `recipes/kustomize/components/aws-efa-p8d8/` and moved AWS
  EFA to a shared template bundle. qwen3-235b-a22b-fp8 pointed at the dead
  path and claimed 8 EFA per worker; the manifest requests 4 on decode and
  2 on each prefill. Both pages' contributor sections now describe the
  template model.

Reported by review bots and confirmed against the sources:

- RecipeStyles.tsx: the tab-order companion block enumerated 2 of 7
  `recipe-variant` values, so for the other five a hidden label kept a
  focusable radio. All four dimensions now match their label-hiding rules.
  Two rulesets whose bodies held only a comment are removed.
- gpt-oss-120b step 2 applies the PVC; step 3 applies the download Job,
  matching the README instead of applying the whole directory.
- kimi-k2-6 fetches the Git LFS trace objects before staging them.
- nemotron-3-ultra no longer says "skip to step 3", which collided with
  the page's own step 3.
- qwen3-32b-fp8 opens on the catalog's recommended 2-GPU aggregated
  target, with the hint on the row that distinguishes it.

Validation: catalog validator and check_asset_paths pass; all 91 catalog
counts match their manifests; 18/18 pages keep the four-level picker order
with 0 dead ends; div balance, fences, frontmatter, SPDX, casing clean.

Not applied, with reasons in the PR discussion: the DeepSeek helper-pod and
"~25x vs ~20x" comments both mirror their recipe READMEs verbatim, so
changing only the pages would create the drift this PR removes.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: shwetha-krishnamurthy <shwetha.krishnamurthy27@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation fix size/S

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants