Repository navigation
fix(recipes): fix SGLang GB200 frontend args — remove --model-path, fix parser flag - #13324
Conversation
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Enterprise Run ID: 📒 Files selected for processing (2)
Included review availability: Your plan includes up to 12 reviews per rolling hour; 11 remain after this review. WalkthroughThe SGLang GB200 chat deployment configurations update the frontend tool-call parser argument to ChangesSGLang frontend argument updates
Estimated code review effort: 1 (Trivial) | ~5 minutes Merge Risk: ⚪ Minimal · up to This localized recipe change removes invalid frontend arguments and corrects the tool-parser flag for GB200 SGLang deployments; no actionable merge-blocking risk remains beyond normal checks and review. 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches 💡 1⚔️ Resolve merge conflicts 💡
Comment |
…ix parser flag Post-merge validation (NVBug 6610056) found two blockers in the GB200 SGLang frontend: 1. --model-path Qwen/Qwen3.8-2.4T-A95B-FP8: dynamo.frontend requires a valid local directory; HF IDs are rejected. Removed: the frontend does not load model weights. 2. --dyn-tool-call-parser qwen3_coder: unknown to dynamo.frontend when using --dyn-chat-processor sglang. Changed to --tool-call-parser qwen3_coder per the sglang pre-processor convention. Validated on GCP GB200 cluster: both fixes confirmed working, coherence check passed with correct reasoning_content. GB300 recipes unaffected: their frontends have neither flag. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> Signed-off-by: Cheng Wang <chengwa@nvidia.com>
2fea0f2 to
22561c9
Compare
num_speculative_tokens=1 is not a valid workaround: when the single draft token is accepted, num_accepted_tokens = draft + target = 2 > 1, which still triggers the DS conv state assertion. Remove --speculative-config entirely until vLLM PR#49291 ships in the image. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: Cheng Wang <chengwa@nvidia.com>
f5e08a6 to
f3b14f9
Compare
…3.8 README - Wait and port-forward commands used qwen38max-agg / qwen38max-agg-frontend (GB200 DGD names); correct to qwen38max-agg-chat / qwen38max-agg-chat-frontend which match the agg-gb300-chat deploy.yaml - Configuration note incorrectly stated the frontend does not mount the PVC and referenced envFrom: hf-token-secret; both frontend and workers mount model-cache, and HF_HUB_OFFLINE=1 means no token lookup at runtime Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: Cheng Wang <chengwa@nvidia.com>
…8 README HF_HOME, HF_HUB_OFFLINE, and TRANSFORMERS_OFFLINE are set on workers only; the frontend mounts the PVC but does not set these variables. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: Cheng Wang <chengwa@nvidia.com>
…ix parser flag (#13324) Signed-off-by: Cheng Wang <chengwa@nvidia.com> Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> Co-authored-by: Yogendra Pandey <20666257+ynpandey-nv@users.noreply.github.com> Signed-off-by: Yogendra Pandey <ypandey@nvidia.com>
Rebases onto current main and addresses the review on ai-dynamo#13661. Reviewer comments: - Landing page reports 88 deployable configurations, not 92. ai-dynamo#13370 cut Nemotron 3.5 Lightning from 30 targets to 26; summing catalog targets across the 18 carded recipes now gives exactly 88. - Nemotron 3.5 Lightning is reconciled against the post-ai-dynamo#13370 recipe tree. The rebase takes upstream's page as the base, so every BF16 checkpoint, `-bf16` path and resource name is upstream's; this commit re-applies only the structural work: the picker moves from a fused backend/topology row to GPU -> Workload -> Backend -> Topology, with Disaggregated gated on vLLM because TensorRT-LLM ships no such target. - GLM-5.2 GPU counts now state per-worker GPUs, worker counts and the deployment total. Checking all 91 catalog targets against their manifests showed the reviewer's `sglang-disagg-b200` (12 -> 20) was one of three: `sglang-agg-b200` 4 -> 16 and `sglang-agg-h200` 8 -> 24 were wrong the same way. The landing card repeated the same four numbers. - Nemotron-3-Super B200 chat and agentic summaries now carry the drafter MoE backend. Only those two manifests set `moe_backend=triton` in the MTP draft loop; both H200 manifests set no override, so they are left alone. Post-rebase drift, from commits that landed after this branch was written: - ai-dynamo#13324 dropped `--speculative-config` from qwen3.8 `vllm/agg-gb200-chat`, so the shared GB200 vLLM summary, the supported-features bullet and the catalog technique list no longer claim MTP for chat. Parser wiring and the model-resolution note are corrected to match the rewritten README. - ai-dynamo#13125 deleted `recipes/kustomize/components/aws-efa-p8d8/` and moved AWS EFA to a shared template bundle. qwen3-235b-a22b-fp8 pointed at the dead path and claimed 8 EFA per worker; the manifest requests 4 on decode and 2 on each prefill. Both pages' contributor sections now describe the template model. Reported by review bots and confirmed against the sources: - RecipeStyles.tsx: the tab-order companion block enumerated 2 of 7 `recipe-variant` values, so for the other five a hidden label kept a focusable radio. All four dimensions now match their label-hiding rules. Two rulesets whose bodies held only a comment are removed. - gpt-oss-120b step 2 applies the PVC; step 3 applies the download Job, matching the README instead of applying the whole directory. - kimi-k2-6 fetches the Git LFS trace objects before staging them. - nemotron-3-ultra no longer says "skip to step 3", which collided with the page's own step 3. - qwen3-32b-fp8 opens on the catalog's recommended 2-GPU aggregated target, with the hint on the row that distinguishes it. Validation: catalog validator and check_asset_paths pass; all 91 catalog counts match their manifests; 18/18 pages keep the four-level picker order with 0 dead ends; div balance, fences, frontmatter, SPDX, casing clean. Not applied, with reasons in the PR discussion: the DeepSeek helper-pod and "~25x vs ~20x" comments both mirror their recipe READMEs verbatim, so changing only the pages would create the drift this PR removes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: shwetha-krishnamurthy <shwetha.krishnamurthy27@gmail.com>
Rebases onto current main and addresses the review on ai-dynamo#13661. Reviewer comments: - Landing page reports 88 deployable configurations, not 92. ai-dynamo#13370 cut Nemotron 3.5 Lightning from 30 targets to 26; summing catalog targets across the 18 carded recipes now gives exactly 88. - Nemotron 3.5 Lightning is reconciled against the post-ai-dynamo#13370 recipe tree. The rebase takes upstream's page as the base, so every BF16 checkpoint, `-bf16` path and resource name is upstream's; this commit re-applies only the structural work: the picker moves from a fused backend/topology row to GPU -> Workload -> Backend -> Topology, with Disaggregated gated on vLLM because TensorRT-LLM ships no such target. - GLM-5.2 GPU counts now state per-worker GPUs, worker counts and the deployment total. Checking all 91 catalog targets against their manifests showed the reviewer's `sglang-disagg-b200` (12 -> 20) was one of three: `sglang-agg-b200` 4 -> 16 and `sglang-agg-h200` 8 -> 24 were wrong the same way. The landing card repeated the same four numbers. - Nemotron-3-Super B200 chat and agentic summaries now carry the drafter MoE backend. Only those two manifests set `moe_backend=triton` in the MTP draft loop; both H200 manifests set no override, so they are left alone. Post-rebase drift, from commits that landed after this branch was written: - ai-dynamo#13324 dropped `--speculative-config` from qwen3.8 `vllm/agg-gb200-chat`, so the shared GB200 vLLM summary, the supported-features bullet and the catalog technique list no longer claim MTP for chat. Parser wiring and the model-resolution note are corrected to match the rewritten README. - ai-dynamo#13125 deleted `recipes/kustomize/components/aws-efa-p8d8/` and moved AWS EFA to a shared template bundle. qwen3-235b-a22b-fp8 pointed at the dead path and claimed 8 EFA per worker; the manifest requests 4 on decode and 2 on each prefill. Both pages' contributor sections now describe the template model. Reported by review bots and confirmed against the sources: - RecipeStyles.tsx: the tab-order companion block enumerated 2 of 7 `recipe-variant` values, so for the other five a hidden label kept a focusable radio. All four dimensions now match their label-hiding rules. Two rulesets whose bodies held only a comment are removed. - gpt-oss-120b step 2 applies the PVC; step 3 applies the download Job, matching the README instead of applying the whole directory. - kimi-k2-6 fetches the Git LFS trace objects before staging them. - nemotron-3-ultra no longer says "skip to step 3", which collided with the page's own step 3. - qwen3-32b-fp8 opens on the catalog's recommended 2-GPU aggregated target, with the hint on the row that distinguishes it. Validation: catalog validator and check_asset_paths pass; all 91 catalog counts match their manifests; 18/18 pages keep the four-level picker order with 0 dead ends; div balance, fences, frontmatter, SPDX, casing clean. Not applied, with reasons in the PR discussion: the DeepSeek helper-pod and "~25x vs ~20x" comments both mirror their recipe READMEs verbatim, so changing only the pages would create the drift this PR removes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: shwetha-krishnamurthy <shwetha.krishnamurthy27@gmail.com>
…ix parser flag (#13324) Signed-off-by: Cheng Wang <chengwa@nvidia.com> Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> Co-authored-by: Yogendra Pandey <20666257+ynpandey-nv@users.noreply.github.com> Signed-off-by: Yogendra Pandey <ypandey@nvidia.com>
Rebases onto current main and addresses the review on ai-dynamo#13661. Reviewer comments: - Landing page reports 88 deployable configurations, not 92. ai-dynamo#13370 cut Nemotron 3.5 Lightning from 30 targets to 26; summing catalog targets across the 18 carded recipes now gives exactly 88. - Nemotron 3.5 Lightning is reconciled against the post-ai-dynamo#13370 recipe tree. The rebase takes upstream's page as the base, so every BF16 checkpoint, `-bf16` path and resource name is upstream's; this commit re-applies only the structural work: the picker moves from a fused backend/topology row to GPU -> Workload -> Backend -> Topology, with Disaggregated gated on vLLM because TensorRT-LLM ships no such target. - GLM-5.2 GPU counts now state per-worker GPUs, worker counts and the deployment total. Checking all 91 catalog targets against their manifests showed the reviewer's `sglang-disagg-b200` (12 -> 20) was one of three: `sglang-agg-b200` 4 -> 16 and `sglang-agg-h200` 8 -> 24 were wrong the same way. The landing card repeated the same four numbers. - Nemotron-3-Super B200 chat and agentic summaries now carry the drafter MoE backend. Only those two manifests set `moe_backend=triton` in the MTP draft loop; both H200 manifests set no override, so they are left alone. Post-rebase drift, from commits that landed after this branch was written: - ai-dynamo#13324 dropped `--speculative-config` from qwen3.8 `vllm/agg-gb200-chat`, so the shared GB200 vLLM summary, the supported-features bullet and the catalog technique list no longer claim MTP for chat. Parser wiring and the model-resolution note are corrected to match the rewritten README. - ai-dynamo#13125 deleted `recipes/kustomize/components/aws-efa-p8d8/` and moved AWS EFA to a shared template bundle. qwen3-235b-a22b-fp8 pointed at the dead path and claimed 8 EFA per worker; the manifest requests 4 on decode and 2 on each prefill. Both pages' contributor sections now describe the template model. Reported by review bots and confirmed against the sources: - RecipeStyles.tsx: the tab-order companion block enumerated 2 of 7 `recipe-variant` values, so for the other five a hidden label kept a focusable radio. All four dimensions now match their label-hiding rules. Two rulesets whose bodies held only a comment are removed. - gpt-oss-120b step 2 applies the PVC; step 3 applies the download Job, matching the README instead of applying the whole directory. - kimi-k2-6 fetches the Git LFS trace objects before staging them. - nemotron-3-ultra no longer says "skip to step 3", which collided with the page's own step 3. - qwen3-32b-fp8 opens on the catalog's recommended 2-GPU aggregated target, with the hint on the row that distinguishes it. Validation: catalog validator and check_asset_paths pass; all 91 catalog counts match their manifests; 18/18 pages keep the four-level picker order with 0 dead ends; div balance, fences, frontmatter, SPDX, casing clean. Not applied, with reasons in the PR discussion: the DeepSeek helper-pod and "~25x vs ~20x" comments both mirror their recipe READMEs verbatim, so changing only the pages would create the drift this PR removes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: shwetha-krishnamurthy <shwetha.krishnamurthy27@gmail.com>
Summary
Fixed the following in the Qwen3.8-2.4T-A95B-FP8 GB200 SGLang frontend manifests that prevent the deployment from starting.
Issue 1:
--model-pathrejected bydynamo.frontenddynamo.frontendrequires--model-pathto be a valid local directory on disk. The manifests passed the HF model ID (Qwen/Qwen3.8-2.4T-A95B-FP8), causing an immediate exit:Fix: Remove
--model-pathand--model-namefrom the frontend. The frontend does not load model weights — only the workers (dynamo.sglang) need the model path.Issue 2:
--dyn-tool-call-parserunknown todynamo.frontendwith sglang chat processorWhen using
--dyn-chat-processor sglang, the correct flag is--tool-call-parser(nodyn-prefix).--dyn-tool-call-parseris a Dynamo-specific wrapper for the vLLM backend and is rejected as unknown in this context.Fix:
--dyn-tool-call-parser qwen3_coder→--tool-call-parser qwen3_coderScope
Affects
sglang/agg-gb200-chat/deploy.yamlandsglang/disagg-gb200-chat/deploy.yaml. GB300 SGLang recipes are unaffected (their frontends have neither flag).Validation
Deployed
sanity-sgl-aggon GCP GB200 with the fixes applied. Coherence check passed:content: 1 + 1 = 2✅reasoning_contentpopulated ✅model: Qwen/Qwen3.8-2.4T-A95B-FP8✅🤖 Generated with Claude Code
Summary by CodeRabbit