Repository navigation
fix(sglang): Sglang compatbility fixes for get_model_config / use_mla_backend - #14234
Conversation
|
👋 Hi trevor-m! Thank you for contributing to ai-dynamo/dynamo. Just a reminder: The 🚀 |
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Enterprise Run ID: 📒 Files selected for processing (4)
Included review availability: Your plan provides up to 12 included reviews per hour; 11 remain after this review. WalkthroughChangesSGLang compatibility
Estimated code review effort: 2 (Simple) | ~15 minutes Merge Risk: ⚪ Minimal · up to This localized compatibility change supports both legacy and relocated SGLang accessors without introducing a new external interface or deployment behavior. No actionable merge-blocking risk remains beyond normal checks and review. 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
Full details: Description checkExplanation The description explains the SGLang API changes, the compatibility shims, and the resulting error. It includes the required Overview, Details, and Related Issues sections. The reviewer-start section is empty, but this is non-critical. Full details: Title checkExplanation The title clearly identifies SGLang compatibility fixes for
Comment |
Signed-off-by: Trevor Morris <tmorris@nvidia.com>
|
Thanks! I thought the original fix was enough so I closed #14064 but later just found this fix is still needed. Thanks for bringing up the follow up fix! |
|
/ok-to-test 598ace3 |
…l linker / 将 GB300 DeepSeek-V4-Pro AgentX 分离式配置切换到 Mooncake external linker (#3187) * feat(agentx): move GB300 DSV4 AgentX disagg to the Mooncake external linker Replace hierarchical cache with the Mooncake unified-cache external linker on the image's own SGLang and bump the image to nightly-dev-cu13-20260916-c9a8fba9. Decode nodes lend host DRAM to the store through four 180GB standalone mooncake_client daemons each, started by the recipe's setup script and registered with the HTTP metadata server; the job fails unless every daemon serves and resolves. Prefill runs 16k-token chunks per DP rank with MegaMoE sized to match and mem-fraction-static 0.80. The benchmark client moves to the dedicated etcd/nats node, where it no longer OOM-kills prefill_0, and Tachometer is off for this ladder. The gb300-nv srt lane gains role_env, which sets this cluster's RDMA devices on the recipe's prefill and decode roles in place of the removed bash launcher block. * chore(agentx): bump the Dynamo wheel and drop the get_model_config shim Dynamo 1.5.0.dev20260914 falls back when ServerArgs.get_model_config is missing (ai-dynamo/dynamo#14234), matching the B200 recipes on the same image, so the sitecustomize shim and its PYTHONPATH entry go away. * chore(agentx): drop MC_STORE_CLIENT_METRIC, which only restated Mooncake's default * chore(agentx): drop decode store-client env the sidecar setup no longer reads Decode ranks no longer mount a store segment, so MOONCAKE_GLOBAL_SEGMENT_SIZE and MOONCAKE_STANDALONE_STORAGE had no effect there; the daemons take their size from MC_SIDECAR_SEGMENT_SIZE. * refactor(agentx): name the Mooncake RDMA devices in the recipe, not the launcher Set MOONCAKE_DEVICE on the prefill and decode roles directly, as the GB300 vLLM Mooncake recipes already do, and drop the srt lane role_env hook. * chore(agentx): record the Mooncake version the GB300 DSV4 disagg ladder runs The image ships mooncake-transfer-engine-cuda13 0.3.13; CONFIGS.md asks independently versioned offload backends to state their version. * refactor(agentx): run the decode Mooncake stores as srt-slurm services Replace the recipe setup script with four mooncake-store services on the decode nodes (180gb each, ports 8800-8803). srt-slurm injects the master and HTTP metadata server, starts them before the workers, and gates on their readiness ports, so the custom daemon script goes away. * chore(agentx): drop store-client env from the decode role Decode workers no longer act as Mooncake store clients; the decode stores carry their own MOONCAKE_DEVICE/PROTOCOL, and P/D KV transfer does not read them. * feat(agentx): move the GB300 DSV4 4P1D point to c2400, add a DEP32 c480 point, and give decode stores 600 s to start * feat(agentx): replace the GB300 DSV4 agg TP4 c8 point with a 1P1D c120 disagg point and drop the DEP32 point * fix(agentx): give the GB300 DSV4 agg recipe PP_SIZE and PCP_SIZE for the AgentX power topology check * fix(agentx): advertise the GB300 DSV4 Mooncake store on the node address, not a BMC Redfish interface * fix(agentx): keep W4A4 MXFP4 MegaMoE on the GB300 DSV4 prefill role only so the DSpark draft runs at the image's default MoE precision * Update GB300 DSV4 AgentX disagg recipe comments for Mooncake KV offload The override comments still described hierarchical-cache KV offload. Comment-only change; the parsed recipe is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: weireweire <20922698+weireweire@users.noreply.github.com> Co-authored-by: Cam Quilici <cjquilici@gmail.com> Co-authored-by: Po-Han Huang <pohanh@nvidia.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: functionstackx <47992694+functionstackx@users.noreply.github.com>
Overview:
sgl-project/sglang#36972 made the following changes which broke Dynamo usage:
server_args.get_model_config()is nowoverrides.model_config_of()server_args.use_mla_backend()is nowoverrides.use_mla_backend()Details:
Add compatiblity shims for both methods.
Example error when using ToT dynamo with ToT sglang without this fix:
Where should the reviewer start?
Related Issues
🚫 This PR is NOT linked to an issue:
Summary by CodeRabbit