Skip to content

fix(sglang): Sglang compatbility fixes for get_model_config / use_mla_backend - #14234

Merged
ishandhanani merged 1 commit into
ai-dynamo:mainfrom
trevor-m:fix-args
Sep 3, 2026
Merged

ishandhanani merged 1 commit into
ai-dynamo:mainfrom
trevor-m:fix-args

Conversation

@trevor-m

@trevor-m trevor-m commented Sep 2, 2026 •

Copy link
Copy Markdown
Contributor

Overview:

sgl-project/sglang#36972 made the following changes which broke Dynamo usage:

  • server_args.get_model_config() is now overrides.model_config_of()
  • server_args.use_mla_backend() is now overrides.use_mla_backend()

Details:

Add compatiblity shims for both methods.

Example error when using ToT dynamo with ToT sglang without this fix:

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/opt/sglang/lib/python3.12/site-packages/dynamo/sglang/__main__.py", line 18, in <module>
    main()
  File "/opt/sglang/lib/python3.12/site-packages/dynamo/sglang/main.py", line 164, in main
    uvloop.run(worker())
  File "/opt/sglang/lib/python3.12/site-packages/uvloop/__init__.py", line 96, in run
    return __asyncio.run(
           ^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/asyncio/runners.py", line 194, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
  File "/opt/sglang/lib/python3.12/site-packages/uvloop/__init__.py", line 48, in wrapper
    return await main
           ^^^^^^^^^^
  File "/opt/sglang/lib/python3.12/site-packages/dynamo/sglang/main.py", line 44, in worker
    config = await parse_args(argv)
             ^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/sglang/lib/python3.12/site-packages/dynamo/sglang/args.py", line 627, in parse_args
    if server_args.get_model_config().is_multimodal:
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'ServerArgs' object has no attribute 'get_model_config'

Where should the reviewer start?

Related Issues

🚫 This PR is NOT linked to an issue:

  • Confirmed — no related issue

Summary by CodeRabbit

  • Bug Fixes
    • Improved compatibility across supported SGLang versions when retrieving model configuration and detecting MLA backend usage.
    • Preserved support for both current APIs and legacy server argument accessors.
  • Tests
    • Added coverage verifying compatibility behavior and accessor preference across SGLang versions.

@trevor-m
trevor-m requested a review from a team as a code owner September 2, 2026 23:33
@copy-pr-bot

copy-pr-bot Bot commented Sep 2, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@trevor-m
trevor-m temporarily deployed to external_collaborator September 2, 2026 23:33 — with GitHub Actions Inactive
@trevor-m
trevor-m temporarily deployed to external_collaborator September 2, 2026 23:33 — with GitHub Actions Inactive
@github-actions github-actions Bot added the fix label Sep 2, 2026
@github-actions

github-actions Bot commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

👋 Hi trevor-m! Thank you for contributing to ai-dynamo/dynamo.

Just a reminder: The NVIDIA Test Github Validation CI runs an essential subset of the testing framework to quickly catch errors.Your PR reviewers may elect to test the changes comprehensively before approving your changes.

🚀

@github-actions github-actions Bot added external-contribution Pull request is from an external contributor backend::sglang Relates to the sglang backend labels Sep 2, 2026
@coderabbitai

coderabbitai Bot commented Sep 2, 2026 •

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 215028e8-0225-40a6-870b-fa7d73811bf3

📥 Commits

Reviewing files that changed from the base of the PR and between 5e75161 and a5e1e83.

📒 Files selected for processing (4)
  • components/src/dynamo/sglang/_compat.py
  • components/src/dynamo/sglang/args.py
  • components/src/dynamo/sglang/register.py
  • components/src/dynamo/sglang/tests/test_sglang_unit.py

Included review availability: Your plan provides up to 12 included reviews per hour; 11 remain after this review.


Walkthrough

Changes

SGLang compatibility

Layer / File(s) Summary
Compatibility helper implementation
components/src/dynamo/sglang/_compat.py
Adds optional imports and public helpers for model configuration and MLA backend detection. Legacy ServerArgs methods take precedence, with module-level accessors as fallback.
SGLang caller integration
components/src/dynamo/sglang/args.py, components/src/dynamo/sglang/register.py
Updates multimodal detection and Mooncake MLA metadata to use the compatibility helpers.
Compatibility path tests
components/src/dynamo/sglang/tests/test_sglang_unit.py
Tests current and legacy accessor paths and verifies legacy methods avoid current accessors.

Estimated code review effort: 2 (Simple) | ~15 minutes

Merge Risk: ⚪ Minimal · up to a5e1e

This localized compatibility change supports both legacy and relocated SGLang accessors without introducing a new external interface or deployment behavior. No actionable merge-blocking risk remains beyond normal checks and review.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 27.27% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 11 functions across 4 files. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Description check ✅ Passed The description explains the SGLang API changes, the compatibility shims, and the resulting error. It includes the required Overview, Details, and Related Issues sections. The reviewer-start section i…
Title check ✅ Passed The title clearly identifies SGLang compatibility fixes for get_model_config and use_mla_backend. It contains a minor spelling error in “compatbility,” but it remains specific and accurately descr…
Full details: Description check

Explanation

The description explains the SGLang API changes, the compatibility shims, and the resulting error. It includes the required Overview, Details, and Related Issues sections. The reviewer-start section is empty, but this is non-critical.

Full details: Title check

Explanation

The title clearly identifies SGLang compatibility fixes for get_model_config and use_mla_backend. It contains a minor spelling error in “compatbility,” but it remains specific and accurately describes the main change.

  • Fix all pre-merge checks with AI

Comment @coderabbitai help to get the list of available commands.

Signed-off-by: Trevor Morris <tmorris@nvidia.com>
@YAMY1234

YAMY1234 commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

Thanks! I thought the original fix was enough so I closed #14064 but later just found this fix is still needed. Thanks for bringing up the follow up fix!

cc @ishandhanani

@ishandhanani

Copy link
Copy Markdown
Contributor

/ok-to-test 598ace3

@ishandhanani
ishandhanani enabled auto-merge (squash) September 3, 2026 03:26
@ishandhanani
ishandhanani merged commit b1c5147 into ai-dynamo:main Sep 3, 2026
177 of 179 checks passed
@jain-ria jain-ria mentioned this pull request Sep 3, 2026
2 of 4 tasks
functionstackx added a commit to SemiAnalysisAI/InferenceX that referenced this pull request Oct 8, 2026
…l linker / 将 GB300 DeepSeek-V4-Pro AgentX 分离式配置切换到 Mooncake external linker (#3187)

* feat(agentx): move GB300 DSV4 AgentX disagg to the Mooncake external linker

Replace hierarchical cache with the Mooncake unified-cache external linker
on the image's own SGLang and bump the image to
nightly-dev-cu13-20260916-c9a8fba9.

Decode nodes lend host DRAM to the store through four 180GB standalone
mooncake_client daemons each, started by the recipe's setup script and
registered with the HTTP metadata server; the job fails unless every
daemon serves and resolves. Prefill runs 16k-token chunks per DP rank
with MegaMoE sized to match and mem-fraction-static 0.80.

The benchmark client moves to the dedicated etcd/nats node, where it no
longer OOM-kills prefill_0, and Tachometer is off for this ladder.

The gb300-nv srt lane gains role_env, which sets this cluster's RDMA
devices on the recipe's prefill and decode roles in place of the removed
bash launcher block.

* chore(agentx): bump the Dynamo wheel and drop the get_model_config shim

Dynamo 1.5.0.dev20260914 falls back when ServerArgs.get_model_config is
missing (ai-dynamo/dynamo#14234), matching the B200 recipes on the same
image, so the sitecustomize shim and its PYTHONPATH entry go away.

* chore(agentx): drop MC_STORE_CLIENT_METRIC, which only restated Mooncake's default

* chore(agentx): drop decode store-client env the sidecar setup no longer reads

Decode ranks no longer mount a store segment, so MOONCAKE_GLOBAL_SEGMENT_SIZE
and MOONCAKE_STANDALONE_STORAGE had no effect there; the daemons take their
size from MC_SIDECAR_SEGMENT_SIZE.

* refactor(agentx): name the Mooncake RDMA devices in the recipe, not the launcher

Set MOONCAKE_DEVICE on the prefill and decode roles directly, as the GB300
vLLM Mooncake recipes already do, and drop the srt lane role_env hook.

* chore(agentx): record the Mooncake version the GB300 DSV4 disagg ladder runs

The image ships mooncake-transfer-engine-cuda13 0.3.13; CONFIGS.md asks
independently versioned offload backends to state their version.

* refactor(agentx): run the decode Mooncake stores as srt-slurm services

Replace the recipe setup script with four mooncake-store services on the
decode nodes (180gb each, ports 8800-8803). srt-slurm injects the master
and HTTP metadata server, starts them before the workers, and gates on
their readiness ports, so the custom daemon script goes away.

* chore(agentx): drop store-client env from the decode role

Decode workers no longer act as Mooncake store clients; the decode stores
carry their own MOONCAKE_DEVICE/PROTOCOL, and P/D KV transfer does not read
them.

* feat(agentx): move the GB300 DSV4 4P1D point to c2400, add a DEP32 c480 point, and give decode stores 600 s to start

* feat(agentx): replace the GB300 DSV4 agg TP4 c8 point with a 1P1D c120 disagg point and drop the DEP32 point

* fix(agentx): give the GB300 DSV4 agg recipe PP_SIZE and PCP_SIZE for the AgentX power topology check

* fix(agentx): advertise the GB300 DSV4 Mooncake store on the node address, not a BMC Redfish interface

* fix(agentx): keep W4A4 MXFP4 MegaMoE on the GB300 DSV4 prefill role only so the DSpark draft runs at the image's default MoE precision

* Update GB300 DSV4 AgentX disagg recipe comments for Mooncake KV offload

The override comments still described hierarchical-cache KV offload. Comment-only change; the parsed recipe is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: weireweire <20922698+weireweire@users.noreply.github.com>
Co-authored-by: Cam Quilici <cjquilici@gmail.com>
Co-authored-by: Po-Han Huang <pohanh@nvidia.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: functionstackx <47992694+functionstackx@users.noreply.github.com>

This branch was previously deployed

1 inactive deployment
external_collaborator — 598ace3f Deployed Sep 2, 2026 by trevor-m via ok-to-test #14689
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backend::sglang Relates to the sglang backend external-contribution Pull request is from an external contributor fix size/L

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants