Skip to content

[MLX] Fall back to no_buffer mamba radix cache on Apple Silicon - #37355

Open
oyl1998 wants to merge 1 commit into
sgl-project:mainfrom
oyl1998:fix/mlx-mamba-extra-buffer
Open

oyl1998 wants to merge 1 commit into
sgl-project:mainfrom
oyl1998:fix/mlx-mamba-extra-buffer

Conversation

@oyl1998

@oyl1998 oyl1998 commented Sep 1, 2026 •

Copy link
Copy Markdown

Motivation

Running a hybrid Mamba/linear-attention model on the MLX backend (Apple Silicon, SGLANG_USE_MLX=1) fails during server-argument resolution:

AssertionError: extra_buffer needs CUDA/MUSA/NPU/ROCm/XPU (FLA).

With --mamba-radix-cache-strategy auto (the default), hybrid architectures that support the extra_buffer strategy (e.g. FalconH1ForCausalLM, Qwen3NextForCausalLM, NemotronHForCausalLM) resolve to extra_buffer whenever overlap scheduling or paging is desired. But MLX never implements extra_buffer — it has no overlap scheduler or paged extra buffer — so validate_mamba_extra_buffer trips the platform assertion before any model loads. The MLX backend should therefore always fall back to the no_buffer strategy.

Modifications

In python/sglang/srt/arg_groups/overrides.py, supports_mamba_cache_extra_buffer now returns False on the MLX backend (use_mlx(), which is only true when SGLANG_USE_MLX=1 and the MLX runtime validates). The gate is folded into the existing architecture check, so the function body changes by one line only:

# MLX only implements the no_buffer mamba radix-cache strategy.
if model_arch in _MAMBA_EXTRA_BUFFER_ARCHS and not use_mlx():
    return view.linear_attn_backend == "triton"
return False

Because the auto resolution in _mamba_radix_cache_resolution gates on this predicate, MLX automatically selects no_buffer and disables overlap scheduling — the only Mamba radix-cache strategy the MLX backend implements — while CUDA/MUSA/NPU/ROCm/XPU remain unaffected. use_mlx() is lazy, so non-MLX platforms take the exact same path as before.

Added a unit regression test test_mlx_never_uses_mamba_extra_buffer in test/registered/unit/test_model_overrides.py that mocks the MLX backend and asserts _mamba_radix_cache_resolution resolves auto to no_buffer on Qwen3NextForCausalLM.

Accuracy Tests

This change only alters which Mamba radix-cache strategy is selected during argument resolution on the MLX backend; it does not touch any model forward or kernel code, so model outputs are unchanged. Verified locally on Apple Silicon with mlx-community/Falcon-H1-0.5B-Instruct-4bit (SGLANG_USE_MLX=1): the server now gets past resolution (mamba_radix_cache_strategy: no_buffer, disable_overlap_schedule: True) and proceeds to MLX model loading instead of aborting on the extra_buffer assertion.

Speed Tests and Profiling

No speed impact. Previously the server never got past startup argument resolution on MLX for these hybrid models; with the fix it starts and reaches the MLX model runner, where performance is governed by the existing MLX runtime.

Checklist

Review and Merge Process

  1. Ping Merge Oncalls to start the process. See the PR Merge Process.
  2. Get approvals from CODEOWNERS and other reviewers.
  3. Trigger CI tests with comments or contact authorized users to do so.
    • Common commands include /tag-and-rerun-ci, /tag-run-ci-label, /rerun-failed-ci
  4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR.

CI States

Latest PR Test (Base): ❌ Run #33722291655
Latest PR Test (Extra): ❌ Run #33722291513
Latest PR Test (AMD ROCm 7.2): ❌ Run #33722291630

On the MLX backend (Apple Silicon, SGLANG_USE_MLX=1), hybrid Mamba models
resolved --mamba-radix-cache-strategy=auto to extra_buffer, which then
failed the platform assertion in validate_mamba_extra_buffer
("extra_buffer needs CUDA/MUSA/NPU/ROCm/XPU (FLA)") before model load.
MLX implements only the no_buffer strategy (no overlap scheduler or paged
extra buffer), so supports_mamba_cache_extra_buffer now reports no support
on MLX and auto resolution falls back to no_buffer. Adds a unit regression
test asserting auto resolves to no_buffer on MLX.
@oyl1998
oyl1998 force-pushed the fix/mlx-mamba-extra-buffer branch from c1e7f78 to 5b6223d Compare September 3, 2026 06:14
@oyl1998

oyl1998 commented Sep 4, 2026 •

Copy link
Copy Markdown
Author

Hi maintainers, this PR is ready for review when you have time. I’d really appreciate any feedback or suggestions. Thank you! @yeahdongcn

@now-ing

now-ing commented Sep 6, 2026

Copy link
Copy Markdown

Verified the core gate against current main (febb360, Apple Silicon M4 Max): with the and not use_mlx() condition applied to supports_mamba_cache_extra_buffer on top of main, plus a regression test that drives _mamba_radix_cache_resolution under a mocked MLX backend (Qwen3Next, strategy=auto), the resolution now falls back to no_buffer instead of asserting � test_model_overrides.py in full: 91 passed, 1 skipped, 29 subtests (including the new test_mlx_never_uses_mamba_extra_buffer).

Rebase notes for current main:

  1. The natural insertion point is python/sglang/srt/arg_groups/overrides.py � supports_mamba_cache_extra_buffer now lives there (the resolution logic moved from server_args into arg_groups + runtime_context accessors), so the PR's hunks need relocating; the semantic change itself is just the one and not use_mlx() conjunct (with the matching use_mlx import from sglang.srt.hardware_backend.mlx.runtime � it has moved from tensor_bridge since this PR branched).
  2. override_settings_and_env-style tests that pin the resolution may need the SGLANG_USE_MLX environment cleared (or the cached use_mlx reset) � the runtime caches the flag per process, which is exactly what the PR's env-pinning notes anticipated.

The failure mode this prevents (AssertionError: extra_buffer needs CUDA/MUSA/NPU/ROCm/XPU (FLA)) reproduces the issue description exactly, and the no_buffer fallback is the conservative correct default on Apple Silicon. Happy to help rebase.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants