Repository navigation
Conversation
On the MLX backend (Apple Silicon, SGLANG_USE_MLX=1), hybrid Mamba models
resolved --mamba-radix-cache-strategy=auto to extra_buffer, which then
failed the platform assertion in validate_mamba_extra_buffer
("extra_buffer needs CUDA/MUSA/NPU/ROCm/XPU (FLA)") before model load.
MLX implements only the no_buffer strategy (no overlap scheduler or paged
extra buffer), so supports_mamba_cache_extra_buffer now reports no support
on MLX and auto resolution falls back to no_buffer. Adds a unit regression
test asserting auto resolves to no_buffer on MLX.
c1e7f78 to
5b6223d
Compare
|
Hi maintainers, this PR is ready for review when you have time. I’d really appreciate any feedback or suggestions. Thank you! @yeahdongcn |
|
Verified the core gate against current Rebase notes for current main:
The failure mode this prevents ( |
Motivation
Running a hybrid Mamba/linear-attention model on the MLX backend (Apple Silicon,
SGLANG_USE_MLX=1) fails during server-argument resolution:With
--mamba-radix-cache-strategy auto(the default), hybrid architectures that support theextra_bufferstrategy (e.g.FalconH1ForCausalLM,Qwen3NextForCausalLM,NemotronHForCausalLM) resolve toextra_bufferwhenever overlap scheduling or paging is desired. But MLX never implementsextra_buffer— it has no overlap scheduler or paged extra buffer — sovalidate_mamba_extra_buffertrips the platform assertion before any model loads. The MLX backend should therefore always fall back to theno_bufferstrategy.Modifications
In
python/sglang/srt/arg_groups/overrides.py,supports_mamba_cache_extra_buffernow returnsFalseon the MLX backend (use_mlx(), which is only true whenSGLANG_USE_MLX=1and the MLX runtime validates). The gate is folded into the existing architecture check, so the function body changes by one line only:Because the
autoresolution in_mamba_radix_cache_resolutiongates on this predicate, MLX automatically selectsno_bufferand disables overlap scheduling — the only Mamba radix-cache strategy the MLX backend implements — while CUDA/MUSA/NPU/ROCm/XPU remain unaffected.use_mlx()is lazy, so non-MLX platforms take the exact same path as before.Added a unit regression test
test_mlx_never_uses_mamba_extra_bufferintest/registered/unit/test_model_overrides.pythat mocks the MLX backend and asserts_mamba_radix_cache_resolutionresolvesautotono_bufferonQwen3NextForCausalLM.Accuracy Tests
This change only alters which Mamba radix-cache strategy is selected during argument resolution on the MLX backend; it does not touch any model forward or kernel code, so model outputs are unchanged. Verified locally on Apple Silicon with
mlx-community/Falcon-H1-0.5B-Instruct-4bit(SGLANG_USE_MLX=1): the server now gets past resolution (mamba_radix_cache_strategy: no_buffer,disable_overlap_schedule: True) and proceeds to MLX model loading instead of aborting on theextra_bufferassertion.Speed Tests and Profiling
No speed impact. Previously the server never got past startup argument resolution on MLX for these hybrid models; with the fix it starts and reaches the MLX model runner, where performance is governed by the existing MLX runtime.
Checklist
Review and Merge Process
/tag-and-rerun-ci,/tag-run-ci-label,/rerun-failed-ciCI States
Latest PR Test (Base): ❌ Run #33722291655
Latest PR Test (Extra): ❌ Run #33722291513
Latest PR Test (AMD ROCm 7.2): ❌ Run #33722291630