Skip to content

[Feat] sail: support sail fa3 and flashmla [3/N] - #45946

Merged
hzh0425 merged 14 commits into
sgl-project:mainfrom
t-head:ywj/upstream_attention
Oct 29, 2026
Merged

hzh0425 merged 14 commits into
sgl-project:mainfrom
t-head:ywj/upstream_attention

Conversation

@Vinsmoke86

@Vinsmoke86 Vinsmoke86 commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

Motivation

Refs #37519

T-HEAD SAIL exposes a CUDA-compatible torch backend (SailSRTPlatform already exists in platforms/sail.py), but its attention kernels are not the ones sgl_kernel ships:

  • sgl_kernel imports its compiled flash_ops (FA3) at import time, so the import fails on SAIL.
  • SAIL provides its own FA3 interface (flash_attn_interface) and FlashMLA library, with different metadata semantics: a lazily-initialized scheduler-metadata object, a 320 sm_parts cap, and no qv / only_qv support.
  • The FA3 backend's capability assert rejects MLA on SM 8.x, which is the capability SAIL reports.
  • Channel-wise FP8 kv_b_proj weights go through channel_quant_to_tensor_quant + FP8 BMM, which SAIL does not support.

This PR lets MLA models (DeepSeek / GLM family) run on SAIL with --attention-backend fa3 and flashmla, including MTP / EAGLE speculative decoding and CUDA graph. It keeps the existing backends unchanged: SAIL-specific behavior is injected via @plugin_hook and only loaded when SAIL hardware is detected.

Modifications

New: python/sglang/srt/hardware_backend/sail/attention/

File Purpose
flash_attention.py Wraps SAIL's flash_attn_interface: flash_attn_with_kvcache, flash_attn_varlen_func, get_scheduler_metadata, with the same signatures as sgl_kernel.flash_attn. FA4 (ver=4) and only_qv=True raise NotImplementedError with a clear message.
flash_mla.py Wraps SAIL's FlashMLA library: get_mla_metadata, flash_mla_with_kvcache, flash_mla_sparse_fwd.
sail_fa3_hooks.py Hooks flashattention_backend.flash_attn_varlen_func / flash_attn_with_kvcache (REPLACE) and FlashAttentionBackend.__init__ (AROUND), so the FA3 backend calls the SAIL ops.
sail_flashmla_hooks.py Hooks for sgl_kernel.flash_mla.* (REPLACE) and FlashMLABackend: caps the CUDA graph metadata buffer at 320 sm_parts, and replaces _apply_decode_target_verify_metadata (SAIL metadata object, DRAFT_EXTEND_V2 seq_lens) and init_forward_metadata_in_graph, so lazy metadata init runs inside graph capture.
sail_dsa_hooks.py Hooks for DSAFlashMLAMetadata.copy_ / .slice and DeepseekSparseAttnBackend graph-metadata init/replay, so the cached SAIL metadata is never overwritten during replay.

Platform detection and hook loading

  • platforms/sail.py: adds is_sail_available() (cached; true when the CUDA device name contains ZW). When it returns true, imports the three hook modules to register them. They are applied by the existing load_plugins() → HookRegistry.apply_hooks() path.
  • runtime_context.py: adds "is_sail" to _PLATFORM_PROBES, so get_platform().is_sail is available.
  • sgl_kernel/utils.py: adds is_sail(). sgl_kernel/flash_attn.py: skips the flash_ops import on SAIL, so import sgl_kernel.flash_attn no longer fails there.

Attention backend / model changes

  • attention_registry.py: the FA3 capability assert also accepts major == 8 with MLA when _is_sail.

  • flashmla_backend.py: forward_decode / forward_extend accept optional q_rope / k_rope.

    • When k_rope is given, the KV cache is written with set_mla_kv_buffer.
    • When q_rope is given, the backend builds the 576-dim query with concat_mla_absorb_q_general.
    • The ForwardMode.EXTEND branch forwards them to FlashInferMLAAttnBackend.forward_extend.
  • deepseek_common/utils.py: adds "flashmla" to FORWARD_ABSORB_CORE_ATTENTION_BACKENDS, so the model passes q_rope / k_rope separately and does not pre-concatenate. Adds _is_sail.

  • forward_mla.py:

    • Uses concat_mla_absorb_q_general in place of torch.cat for the non-absorb-core q / k concat.
    • On SAIL, skips the FP8 w_kc / w_vc BMM branches; the weights are BF16 there (see below).
  • deepseek_weight_loader.py: on SAIL, dequantizes channel-wise FP8 kv_b_proj to BF16 (w * weight_scale) instead of channel_quant_to_tensor_quant.

  • python/pyproject_other.toml: adds flash-attention-for-sail, flashinfer-for-sail and flashmla-for-sail to srt_sail (no version pins).

  • python/sglang/srt/layers/moe/topk.py: requires get_device_sm() >= 89 before selecting the FlashInfer biased grouped topk kernel.

Server-arg resolution

  • model_override_base.py: on SAIL, MHA models default to fa3, under the same is_no_spec_infer_or_topk_one condition as Hopper.
  • model_hook.py: on SAIL with an MLA backend, sets the prefill phase of cuda_graph_config to DISABLED. A captured prefill graph pins MLA to the absorbed path, which passes qv to FA3, and the SAIL FA3 interface rejects qv.

Impact on non-SAIL platforms

  • All _is_sail / is_sail() / get_platform().is_sail branches are false on non-SAIL devices, and no hook module is imported.
  • One shared behavior change: with "flashmla" in FORWARD_ABSORB_CORE_ATTENTION_BACKENDS, CUDA FlashMLA now receives q_rope / k_rope and does the concat and set_mla_kv_buffer inside the backend. This replaces the concat in the model plus set_kv_buffer. It needs CUDA (Hopper) validation before going upstream.

Known limitations

  • SAIL FA3 does not support qv / only_qv. DSA + fa3 (DeepseekSparseAttnBackend._forward_fa3 always passes qv) is therefore not usable on SAIL; use flashmla_kv for DSA decode.
  • Prefill CUDA graph is disabled for MLA on SAIL (see above).

Accuracy Tests

  • SAIL: DeepSeek-V3 / GLM-5.x, --attention-backend fa3 and flashmla, GSM8K
  • SAIL: MTP / EAGLE (speculative_attention_mode prefill & decode), GSM8K + accept length
  • CUDA (H100/H200): --attention-backend flashmla, DeepSeek-V3, GSM8K (regression check for the FORWARD_ABSORB_CORE_ATTENTION_BACKENDS change)

Speed Tests and Profiling

  • SAIL: bench_serving / bench_one_batch for fa3 and flashmla
  • CUDA: --attention-backend flashmla decode throughput, no regression vs. base

Checklist

Review and Merge Process

  1. Ping Merge Oncalls to start the process. See the PR Merge Process.
  2. Get approvals from CODEOWNERS and other reviewers.
  3. Trigger CI tests with comments or contact authorized users to do so.
    • Common commands include /tag-and-rerun-ci, /tag-run-ci-label, /rerun-failed-ci
  4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR.

CI States

Latest PR Test (Base): ⏳ Run #40833194223
Latest PR Test (Extra): ❌ Run #40833194003
Latest PR Test (AMD ROCm 10): ⏳ Run #40833194689

Comment thread python/sglang/srt/platforms/sail.py Resolved
@Vinsmoke86
Vinsmoke86 force-pushed the ywj/upstream_attention branch from fc04f2f to 0606024 Compare October 23, 2026 09:23
@hzh0425 hzh0425 added run-ci CI: run the baseline test suite on this PR run-ci-ppu labels Oct 23, 2026
@ch-wan ch-wan removed the run-ci-extra CI: also run the extra suite (requires run-ci) label Oct 28, 2026
The SAIL FA3 build only accepts q_v on Hopper, but the absorbed MLA decode path always passes qv=q_nope. FA3 used to be unreachable for MLA on SM8.x; enabling it in the registry left --attention-backend fa3 resolving decode to fa3, which fails on the first decode (graph warmup when decode graphs are on).

Fold the decode case into the existing prefill-side q_v fallback: one pass now normalizes decode to flashmla for MLA models, then judges prefill against the normalized pair and returns both declarations together. A pass cannot see its own declarations through the view, so the decode result is carried in a local variable instead of a second pass.
Adding flashmla to FORWARD_ABSORB_CORE_ATTENTION_BACKENDS moved SAIL MLA off the concat branch, which multiplies the joined q by llama_4_scaling. The split branch only forwards the multiplier for the trtllm/cutedsl fused-rope paths and SailFlashMLABackend takes no such argument, so Mistral-Large-3 silently lost the scaling once positions pass original_max_position_embeddings.

Scale both q halves before dispatch when the backend is flashmla, and assert in a unit test that a non-unit multiplier produces the same q as the concat branch.
@Vinsmoke86
Vinsmoke86 requested a review from ch-wan October 29, 2026 02:57
@AgainstEntropy AgainstEntropy added parallel-stages CI: stages dispatch together instead of waiting on each other highest-priority CI: all three control labels, plus never batch-cancelled or stale-closed labels Oct 29, 2026
SailFlashMLABackend overrides _apply_decode_target_verify_metadata, which reads self.kv_index_translator, but nothing ever set it: FlashMLABackend relies on KVIndexTranslator.bind_and_verify_backends, which only visits the top-level backends, and on SAIL a prefill/decode split wraps this instance inside SailHybridAttnBackend. Decode graph capture therefore crashed on the class-level None.

Read the translator in __init__ instead, and treat a missing one as 'no translation': runners that predate the unified pool keep building flashmla kv indices with the triton kernel.
@Vinsmoke86
Vinsmoke86 force-pushed the ywj/upstream_attention branch from d6cde64 to 2575d5b Compare October 29, 2026 07:05
@hzh0425

hzh0425 commented Oct 29, 2026

Copy link
Copy Markdown
Collaborator

https://github.com/sgl-project/sglang/actions/runs/40819228963/job/123267851596?pr=45946
image

The previous round of CI has completed; The new commit only includes a fix for a change related to single .sail file: 2575d5b

It should be safe to merge.

@hzh0425
hzh0425 merged commit d7522db into sgl-project:main Oct 29, 2026
151 of 191 checks passed
nvrohanv added a commit to nvrohanv/sglang that referenced this pull request Oct 29, 2026
One conflict, in DeepseekSparseAttnBackend of
python/sglang/srt/layers/attention/dsa_backend.py: main
(sgl-project#45946) adds the _call_flash_mla_* wrappers and this
branch adds out_graph_metadata_capturable at the same place, after the
class attributes. Both are kept: out_graph_metadata_capturable first,
next to the attribute comment it refers to, then main's wrappers.
nvrohanv added a commit to nvrohanv/sgl-nvidia-mega that referenced this pull request Oct 29, 2026
sgl-project/sglang#45946 adds SAIL-only fp8 conditions to
forward_absorb_prepare / forward_absorb_core and a FlashMLA-backend-only
llama_4_scaling step to forward_absorb_prepare; the DSA path the spec
serves is unchanged. Pin the new digests next to the old ones.
Vinsmoke86 added a commit to t-head/sglang-for-sail that referenced this pull request Nov 5, 2026
Move the SAIL-only DeepGEMM pieces out of the shared layers tree into
`srt/hardware_backend/sail/`, the layout sgl-project#45946 started for attention:

  sail/moe/deepgemm_ep.py      <- layers/moe/moe_runner/sail_deepgemm_ep.py
  sail/moe/deepgemm_moe.py     <- layers/moe/moe_runner/sail_deepgemm_moe.py
  sail/quantization/compressed_tensors_w8a8_int8.py
                               <- compressed_tensors/schemes/compressed_tensors_w8a8_int8_generic_moe.py

The modules are moved verbatim (they already use absolute imports); only the
callers follow the new paths: the runner factory, the compressed-tensors and
fp8/mxfp4/unquant/w8a8_int8 backends that import the fused funcs for their
register side effects, and the unit test.
Yuxingwang-intel pushed a commit to Yuxingwang-intel/sglang that referenced this pull request Nov 6, 2026
Co-authored-by: shiyunzhi <shiyunzhi.syz@alibaba-inc.com>
Vinsmoke86 added a commit to t-head/sglang-for-sail that referenced this pull request Nov 6, 2026
Move the SAIL-only DeepGEMM pieces out of the shared layers tree into
`srt/hardware_backend/sail/`, the layout sgl-project#45946 started for attention:

  sail/moe/deepgemm_ep.py      <- layers/moe/moe_runner/sail_deepgemm_ep.py
  sail/moe/deepgemm_moe.py     <- layers/moe/moe_runner/sail_deepgemm_moe.py
  sail/quantization/compressed_tensors_w8a8_int8.py
                               <- compressed_tensors/schemes/compressed_tensors_w8a8_int8_generic_moe.py

The modules are moved verbatim (they already use absolute imports); only the
callers follow the new paths: the runner factory, the compressed-tensors and
fp8/mxfp4/unquant/w8a8_int8 backends that import the fused funcs for their
register side effects, and the unit test.
Vinsmoke86 added a commit to t-head/sglang-for-sail that referenced this pull request Nov 6, 2026
Move the SAIL-only DeepGEMM pieces out of the shared layers tree into
`srt/hardware_backend/sail/`, the layout sgl-project#45946 started for attention:

  sail/moe/deepgemm_ep.py      <- layers/moe/moe_runner/sail_deepgemm_ep.py
  sail/moe/deepgemm_moe.py     <- layers/moe/moe_runner/sail_deepgemm_moe.py
  sail/quantization/compressed_tensors_w8a8_int8.py
                               <- compressed_tensors/schemes/compressed_tensors_w8a8_int8_generic_moe.py

The modules are moved verbatim (they already use absolute imports); only the
callers follow the new paths: the runner factory, the compressed-tensors and
fp8/mxfp4/unquant/w8a8_int8 backends that import the fused funcs for their
register side effects, and the unit test.
Vinsmoke86 added a commit to t-head/sglang-for-sail that referenced this pull request Nov 6, 2026
Move the SAIL-only DeepGEMM pieces out of the shared layers tree into
`srt/hardware_backend/sail/`, the layout sgl-project#45946 started for attention:

  sail/moe/deepgemm_ep.py      <- layers/moe/moe_runner/sail_deepgemm_ep.py
  sail/moe/deepgemm_moe.py     <- layers/moe/moe_runner/sail_deepgemm_moe.py
  sail/quantization/compressed_tensors_w8a8_int8.py
                               <- compressed_tensors/schemes/compressed_tensors_w8a8_int8_generic_moe.py

The modules are moved verbatim (they already use absolute imports); only the
callers follow the new paths: the runner factory, the compressed-tensors and
fp8/mxfp4/unquant/w8a8_int8 backends that import the fused funcs for their
register side effects, and the unit test.
Vinsmoke86 added a commit to t-head/sglang-for-sail that referenced this pull request Nov 6, 2026
Move the SAIL-only DeepGEMM pieces out of the shared layers tree into
`srt/hardware_backend/sail/`, the layout sgl-project#45946 started for attention:

  sail/moe/deepgemm_ep.py      <- layers/moe/moe_runner/sail_deepgemm_ep.py
  sail/moe/deepgemm_moe.py     <- layers/moe/moe_runner/sail_deepgemm_moe.py
  sail/quantization/compressed_tensors_w8a8_int8.py
                               <- compressed_tensors/schemes/compressed_tensors_w8a8_int8_generic_moe.py

The modules are moved verbatim (they already use absolute imports); only the
callers follow the new paths: the runner factory, the compressed-tensors and
fp8/mxfp4/unquant/w8a8_int8 backends that import the fused funcs for their
register side effects, and the unit test.
Vinsmoke86 added a commit to t-head/sglang-for-sail that referenced this pull request Nov 9, 2026
Move the SAIL-only DeepGEMM pieces out of the shared layers tree into
`srt/hardware_backend/sail/`, the layout sgl-project#45946 started for attention:

  sail/moe/deepgemm_ep.py      <- layers/moe/moe_runner/sail_deepgemm_ep.py
  sail/moe/deepgemm_moe.py     <- layers/moe/moe_runner/sail_deepgemm_moe.py
  sail/quantization/compressed_tensors_w8a8_int8.py
                               <- compressed_tensors/schemes/compressed_tensors_w8a8_int8_generic_moe.py

The modules are moved verbatim (they already use absolute imports); only the
callers follow the new paths: the runner factory, the compressed-tensors and
fp8/mxfp4/unquant/w8a8_int8 backends that import the fused funcs for their
register side effects, and the unit test.
Vinsmoke86 added a commit to t-head/sglang-for-sail that referenced this pull request Nov 9, 2026
Move the SAIL-only DeepGEMM pieces out of the shared layers tree into
`srt/hardware_backend/sail/`, the layout sgl-project#45946 started for attention:

  sail/moe/deepgemm_ep.py      <- layers/moe/moe_runner/sail_deepgemm_ep.py
  sail/moe/deepgemm_moe.py     <- layers/moe/moe_runner/sail_deepgemm_moe.py
  sail/quantization/compressed_tensors_w8a8_int8.py
                               <- compressed_tensors/schemes/compressed_tensors_w8a8_int8_generic_moe.py

The modules are moved verbatim (they already use absolute imports); only the
callers follow the new paths: the runner factory, the compressed-tensors and
fp8/mxfp4/unquant/w8a8_int8 backends that import the fused funcs for their
register side effects, and the unit test.
YAMY1234 added a commit to sgl-project/sgl-nvidia-mega that referenced this pull request Nov 10, 2026
… NVFP4 MTP target verify and NextN draft (#24)

* [Feat] [B300] glm5_3: NVIDIA megakernel specs for GLM-5.3 NVFP4 MTP target verify and NextN draft on B300 TP8

Add sgl_nvidia_mega/glm5_3: the GLM-5.3 target spec for SGLang's megakernel
layer (MTP verify steps of 1 to 4 requests, 4 to 16 rows, on 8 x B300 at
TP 8), its step functions with stock paths adapted from SGLang's model code,
and the CuTe DSL attention, small-M NVFP4 MoE and all-reduce hop kernels.
With --enable-nvidia-megakernel every target slot runs (attention, moe,
hops, qkv_a_small_m_shared, dnorm); SGLANG_NVIDIA_MEGAKERNEL_SLOTS is a test
and bisect override. Needs SGLang's DSA binding view.

Add the GlmMoeDsaForCausalLMNextN draft spec (slot draft, on by default) and
its CuTe DSL kernels: the embedding gather and eh norms in one kernel, the
column-sharded eh projection gathered by the target's all-reduce hop, the
distributed top-1, and the draft layer's MLA prologue, in every captured
draft graph. Needs embedding replication (with the flag, SGLang's megakernel
layer sets SGLANG_ENABLE_EMBED_REPLICATION when it is unset), greedy draft
acceptance and SGLang's EAGLE draft graph capture check; the contract test
skips the check's marker, naming it, on an SGLang without it.

At startup rank 0 logs the covered verify widths (1-4 requests) and the
derived weight copies' memory per GPU, and warns when
--max-running-requests is set above 8, where the megakernel helps little.

Also the README section, a NOTICE paragraph, a CODEOWNERS line and the
contract test's checks of the DSA binding view reads and the mirrored SGLang
code; the GLM-5.3 tests are filed separately.

The same specs serve GLM-5.2 NVFP4, which has GLM-5.3's architecture
(GlmMoeDsaForCausalLM) and the same value of every config field the target
and draft specs pin. The spec names and the plugin's log, refusal and rank
agreement texts name both models, and the module path stays glm5_3.
test/unit/test_glm5_3_glm5_2_config.py runs the published config of
nvidia/GLM-5.2-NVFP4 (revision 53e0691e21895a3863a606dfd12910c69eba94ab)
through SGLang's requirement check for both specs, and checks that a config
that differs in any pinned value, or lacks one, is still refused.

The package imports SGLang's megakernel interface from its location since
sgl-project/sglang#45389 (sglang.kernels.plugin_kernels.nvidia,
sglang.srt.kernel_plugins.nvidia) and falls back to sglang.srt.megakernels
on an older SGLang, as deepseek_v41 does.

* docs: correct GLM GPU test availability

* [Feat] [B300] glm5_3: serve MTP with 5 draft steps (6 verify rows a request)

The GLM-5.3 / GLM-5.2 specs served MTP / EAGLE with 3 draft steps only, so
the 5 / 1 / 6 setup of SGLang's GLM-5.3 and GLM-5.2 B300 NVFP4 low-latency
recipes stopped at startup with --enable-nvidia-megakernel.

The verify widths, the MoE row map and the draft eh rows are now derived
from the verify rows a request T (speculative_num_draft_tokens =
speculative_num_steps + 1, top-1 drafts), and the specs serve the tested
values 4 and 6 (plan.SERVED_TOKENS_PER_REQUEST); any other value is refused
at startup, naming them.

- Widths: every whole number of requests up to the kernels' 16 rows
  (plan.supported_rows): 4 / 8 / 12 / 16 rows at T 4 as before, 6 / 12 rows
  (1-2 requests) at T 6. Steps of more requests run SGLang's path.
- MoE: the engine takes at most 4 row groups of at most 4 rows, token sets
  over one expert list rather than requests. moe_split groups the rows by 4
  with the remainder last: T 4 keeps (4,) to (4, 4, 4, 4); 6 rows run on the
  two-group engine as (4, 2) and 12 rows on the four-group engine as
  (4, 4, 4). The one-group engine holds 4 rows, so it cannot take 6.
- Draft: the eh_proj shard serves the draft decode's 1-4 rows and the draft
  extend at the covered widths, so T 4 keeps its rows and T 6 adds 6;
  remove the unused max_requests.
- Attention, the hops and the draft attention already take any row count
  up to 16 on their 8- and 16-row forms; no kernel changes.

At T 4 every width, row map, eh row set and kernel form is unchanged.

* [Perf] [B300] glm5_3: the one-request fast forms at 6 verify rows

With 5 draft steps a one-request verify step has 6 rows, past the 4-row
limit of the forms that serve one request of 3 draft steps. Serve them at up
to 6 rows:

- the split MoE fold hop (SPLIT_MAX_ROWS 6; still taken only where the
  cooperative occupancy holds 2 x rows + 1 clusters of 8 CTAs, probed at
  build);
- the plain all-reduce hops on the dense layers (DENSE_MAX_ROWS 6);
- the 16-warp sparse MLA (one CTA per SM; its kernel already takes any row
  count and gives the 8-warp form's bytes).

No width of 4 verify rows a request has 5 or 6 rows, so those steps keep
their forms. The kernels are unchanged.

* docs: glm5_3 MoE groups and eh_proj rows with 6 draft tokens

The README's GLM-5.3 slot table named the MoE engine forms by requests
and listed the eh_proj rows of 4 draft tokens only. Name the forms by row
groups (6 rows run as 4 + 2, 12 as 4 + 4 + 4) and add the eh_proj rows of
6 draft tokens (6 and 12).

* [Fix] glm5_3: pin the SGLang mirrors with main's mirror digest

Main's SGLANG_MIRRORS check (#20) digests a
function's tokens in another encoding than this branch's copy, which the
merge replaced with main's. Pin the 18 GLM-5.3 mirrors with main's
mirror_digest of the same SGLang code: no mirrored body changes, only how
its digest is written.

* [Fix] glm5_3: re-pin four SGLang mirrors for current SGLang main

Four mirrored SGLang bodies changed on SGLang main. Each pin is now a
tuple of the digests before and after the change, so the check passes on
either SGLang:

- DeepseekMLAForwardMixin.forward_absorb_prepare (sgl-project/sglang#46359):
  every changed line is on the --dcp-replicate-q-proj path, which runs only
  with decode context parallelism; the specs refuse it, and the stock
  attention step does not copy those lines.
- DeepseekSparseAttnBackend._forward_trtllm (sgl-project/sglang#46990): the
  KV store and read go through the pool's layer router; on the DSA pool, a
  plain pool, that is the same set_mla_kv_buffer and get_key_buffer call
  with the same arguments, which the stock step makes.
- DeepseekSparseAttnBackend.forward_decode: sgl-project/sglang#46990 as
  above, and sgl-project/sglang#44322, a triton branch for ROCm with
  decode context parallelism.
- LogitsProcessor.forward (sgl-project/sglang#43449): the new lines run only
  with --enable-return-aux-hidden-states, which turns speculative decoding
  off, so never with the NextN draft; the output's new last_hidden_states
  stays None.

The copied steps stay as they are. The other 14 mirrored bodies are
unchanged on SGLang main.

* [Fix] glm5_3: adapt the specs to SGLang's megakernel API version 2

SGLang main's megakernel contract is version 2: a built object may also
offer the EAGLE entries draft_loop, draft_extend and begin_draft_round and
attention graph_variants, which SGLang reads from every built object. The
GLM-5.3 / GLM-5.2 target and draft built objects offer none of them (None:
SGLang's own path, as before), and the specs declare API version 2, as
main's DeepSeek-V4.1 and Qwen3.5 specs do.

* docs: glm5_3 boundary docstring says plain dense hops up to 6 rows

DENSE_MAX_ROWS is 6 since the 5-draft-step change; the module docstring
still said 4.

* test: glm5_3 pins the forward_mla mirrors after sglang#45946

sgl-project/sglang#45946 adds SAIL-only fp8 conditions to
forward_absorb_prepare / forward_absorb_core and a FlashMLA-backend-only
llama_4_scaling step to forward_absorb_prepare; the DSA path the spec
serves is unchanged. Pin the new digests next to the old ones.

* [Feat] [Rubin] glm5_3: serve SM107, TP8 across two trays, and nodes without NVLink multicast

Port the Rubin pieces of #19 (glm_dsa) to the
GLM-5.3 / GLM-5.2 specs. The kernels are the same; only the admission changes.

- spec, draft: admit sm_107 (Rubin / VR NVL72) next to sm_103. The draft
  takes the target's ISAs.
- spec, draft: declare Feature.MULTI_NODE_TP, so TP8 can span the two 4-GPU
  trays of an NVL72 rack (GB300 NVL72, Rubin VR NVL72), as the Qwen3.5 and
  Kimi K3 specs do.
- runtime: SGLANG_NVIDIA_MEGAKERNEL_GLM_ALLOW_NO_AR_FUSION=1 (off by default)
  lets the model check and the per-step coverage pass where SGLang's FlashInfer
  all-reduce + RMSNorm fusion does not apply (no NVLink multicast). The stock
  residual steps then run SGLang's unfused all-reduce, which is what its
  boundaries run there, and slot hops runs its own symmetric-memory all-reduce.
  Without the variable nothing changes.
- README: the GPUs and the two-tray launch. test/unit/test_glm5_3_rubin.py
  covers the three changes on CPU.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* [Perf] [GB300] glm5_3: Run the NextN draft-extend MLP on the select_index rows only

The draft extend runs the NextN layer on every verify row of each request, but
the next draft step reads only the select_index row (one per request); the
logits processor already prunes to those rows before the LM head. After the
post-attention norm the draft step now keeps those rows of the MLP input and of
the residual SGLang's prepare wrote, so the MoE, its all-reduce and the final
norm run on B rows instead of B * tokens (6 -> 1 and 12 -> 2 rows at EAGLE
5/1/6). The attention still writes every row's KV and indexer key. The hidden
states handed to the logits then hold only the selected rows, with a
LogitsMetadata that does not select again.

- Slot draft_rowselect (shipped, a draft slot, needs slot draft).
- Gate (draft_step.select_index_rows): DRAFT_EXTEND_V2, no logprobs, hidden
  capture not FULL, no padded-row mask, a 1-D select_index with 0 < B < N
  rows; the NextN MLP is DeepseekV2MoE without an all-to-all. Host-only facts,
  fixed per captured shape. draft_attention.select_rows raises on any other
  stream layout.
- draft_attention.decoder_layer_forward takes the rows to keep (ffn_rows) and,
  at a width the draft attention family does not serve, SGLang's own attention
  call.
- Rank 0 logs once per captured draft graph its attention and MLP rows.

Numerics: the selection itself is exact (the kept rows equal the full-row
stream's rows bit for bit through SGLang's residual stream and final add +
RMSNorm). The draft MoE and its router pick their kernels by the row count, so
the MLP output on B rows is not bit-identical to the same rows of the N-row run
(draft.rowselect.mlp.* in the draft NUMERICS). Draft-only: target verification
decides every emitted token.

Tests: test/unit/test_glm5_3_draft_rowselect.py (CPU; the logits hand-off
needs CUDA), test/kernels/test_glm5_3_draft_rowselect.py (4 GPUs).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* [Feat] glm5_3: TP4 profiles, verify steps composed per width from kernel units

The spec serves TP8 at 4 or 6 tokens per request up to 16 rows. It now also
accepts TP4 at 3 to 6 tokens per request up to 48 rows (one GB300 tray: the
InferenceX AgentX TP4 aggregate and PD decode-role configurations), and serves
those widths with kernel units composed per step; the TP8 path
(plan.build_providers, the executor, the families) is unchanged.

- profile.py: the facts a build is decided on (TP, heads per rank, tokens per
  request, widths, device, features) and the accepted table: the TP8 envelope
  (plan.py) plus TP4 at 3-6 tokens, 48 rows. Profile.units says which build
  runs.
- registry.py: candidates claim steps of a group atomically, with coverage per
  (profile, rows, mode) and contract tokens; per width the first preferred,
  switched-on, covering unit takes each step, the rest run the stock body. A
  decline (coverage) runs stock and is logged once per width; a family that
  covers a width but fails to build raises on every rank. A mixed group is a
  subclass of the stock group with the claimed steps bound to their units, so
  the stock step bodies and Providers.describe see them.
- units.py: the TP4 preference table. Every unit is a slot (runtime.LL_SLOTS,
  shipped; OPT_IN_SLOTS only when listed), selected by
  SGLANG_NVIDIA_MEGAKERNEL_SLOTS as the TP8 families are; "" is parity (every
  accepted width on the stock bodies).
- runtime/composed.py: the executor (GlmMegakernelVerify: its layer loop,
  first-capture checks and parity switch) on the composed step groups; its
  logits take ll.gemm's lm_head role and share the model processor's gatherer.
  runtime.build hands a TP4 profile to it.
- spec.py: tp_sizes {4, 8}, the (TP, tokens) pairing checked against the
  profile table (profile_reasons), max_rows 48, the units' named tolerances.
- ll/__init__.py, ll/jit.py: the units' package (optional modules, the
  ll.linear GEMM entry point, named tolerances) and the JIT loader for its .cuh
  kernels (SGLang's load_jit; refuses to compile under capture; failures
  raise).

Tests: test/unit/test_glm5_3_{profile,registry,spec,ll_jit,verify_logits}.py.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* [Perf] [GB300] glm5_3: TP4 small-M BF16 GEMMs on PDL split-K and TGV (ll.gemm)

The TP4 roles' small-M BF16 GEMMs that SGLang leaves on cuBLAS launch without
programmatic dependent launch. ll.gemm moves them, at the row counts where a
GB300 crossover bench measured a win in the GEMM's model context, to a PDL
kernel (slot ll.gemm.<role> each):

- router (FP32 logits): split-K at 17-96 rows (the tiny GEMM keeps <= 16);
- indexer wk|weights_proj: split-K at 1-64 rows;
- o_proj TP4: TGV 1-72; dense_down TP4: TGV 1-48; NextN eh_proj: TGV;
- lm_head TP4: TGV 1-60; measured but deferred (run only when listed): the
  TP8 o_proj / q_b shapes.

gemm_plan.py holds the plan and the named tolerances. The plan applies only on
SM103 with 152 SMs; elsewhere, and for quantized or biased weights or unplanned
rows, every role runs SGLang's own GEMM for the module, bit for bit.

Split-K is a vendored copy of FlashInfer v0.7.2's cluster split-K kernel with
FP32 output / any M and synchronization fixes (owner wait at .acquire.cluster,
phase-0 wait before the TMEM release, tcgen05 thread-sync fences; no arithmetic
change). FlashInfer's own kernel is used only when it carries those fixes
(SplitKTactic.SYNC_FIXED); SGLANG_NVIDIA_MEGAKERNEL_GLM_GEMM_SPLITK pins
auto / flashinfer / vendored. TGV is SGLang's cutedsl_bf16_gemm.

Tests: test/unit/test_glm5_3_ll_gemm_plan.py (CPU: tactics, engagement only on
SM103 / 152 SMs, slots, FlashInfer detection, stock dispatch);
test/kernels/test_glm5_3_ll_gemm.py (1 GPU: path per role and row, tolerances
vs cuBLAS and FP64, graph replay, PDL edges, launch stress).
benchmark/bench_glm5_3_ll_gemm.py re-checks the crossover.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* [Perf] [GB300] glm5_3: TP4 sparse MLA flash decode as unit ll.sparse_mla_fd

The sparse_mla and uv_proj steps of the TP4 verify attention and of the NextN
draft layer's attention on a flash-decoding CUDA kernel over the top-k gathered
FP8 KV rows, instead of TRTLLM-Gen's sparse MLA decode + torch.bmm. The step
reads the running forward's ForwardView (view.dsa, view.kv_pool,
view.metadata), as the stock step does.

- Kernels (ll/csrc/attention/glm_sparse_mla_fd_*.cuh, adapted from
  TensorRT-LLM's sparse MLA flash decode): one 16-CTA cluster per row, 8 or 16
  heads; a v_b fold instance (8 heads, bitwise cuBLAS's BMM); a multi-tile mode
  (clusters of 2-8 CTAs, 8 heads).
- Launch plan (GB300, 152 SMs): one-tile up to 7 rows; built at compile; the
  co-resident cluster counts are agreed over the TP group (MIN); a build
  failure on any rank raises on every rank.
- Covers sm_103 with 152 SMs, 16 heads (TP4), <= 7 rows; declines with
  TRTLLM-Gen skip-softmax decode on.
- Tolerance ll.sparse_mla_fd.max_abs = 0.0625 against TRTLLM-Gen (measured
  0.052 / 0.055 at 8 / 16 heads).

16 heads: -2.1 to -2.8 us per layer at 5 / 6 rows
(benchmark/bench_glm5_3_sparse_mla.py).

Tests: numerics vs TRTLLM-Gen and an FP64 oracle, the provider steps vs the
stock steps, CUDA-graph replay and the PDL chain (1 GPU); the launch plan and
rank agreement (CPU).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* [Perf] [GB300] glm5_3: TP4 attention front on low-latency fused-A and a one-launch MLA KV half

The unit ll.attention (groups attention and draft_attention) serves the
attention front of a TP4 rank (16 heads, SM103, 1-48 rows, every step mode) by
claiming qkv_a / prologue / index_k / rope_cache of AttentionStep:

- qkv_a: SGLang's min-latency fused-A kernel with a larger stage ring and a
  pipelined MMA loop from 9 rows, up to 16 rows (bit for bit SGLang's kernel); a
  split-K form at 17-24 rows when one wave fits (ll.attention.qkv_a_wide.max_rel
  of fp32); stock above 24 rows.
- prologue: stock q_a norm; q_b on the pipelined form at 9-16 rows (bitwise);
  the kv_a norm deferred into rope_cache when folded.
- index_k: the stock body with its key / head-gate projection through
  ll.linear("wk_weights_proj").
- rope_cache: one PDL launch for the kv_a norm (folded), RoPE and FP8 of q and
  k and the MLA cache append, bit for bit FlashInfer's norm and
  mla_rope_quantize_fp8 and SGLang's set_mla_kv_buffer (the cache through the
  step's ForwardView).

The fold (slot ll.attention.kv_norm_fold) engages only when FlashInfer's norm
backend, read from its public surface, is CuTe DSL and a probe at build gives
the stock kv_a norm's bits; the family logs which way it went. fused_a_ll.cuh is
adapted from SGLang's dsv3_fused_a_gemm.cuh.

Tests (1 GPU): fused-A forms bitwise at 1-16 rows and within tolerance at
17-24, graph replay; the KV half bitwise (q and every cache byte); the stock
AttentionStep vs ll.attention on one TP4 layer at 5-48 rows, eager and
replayed. CPU: the norm-backend decision, coverage and composition.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* [Perf] [GB300] glm5_3: TP4 all-reduce + residual + RMSNorm hops on a plugin-owned Lamport buffer

The unit ll.hops serves the three hop steps (HopSteps) of the TP4 verify step:

- csrc/distributed/glm_ll_mnnvl_oneshot.cuh: one-shot / two-shot MNNVL
  all-reduce + residual + RMSNorm with a MoE-finalize prologue and an NVFP4
  epilogue (builds with nvcc 13.0 and 13.4). Every wait on a peer is bounded:
  past the deadline it writes a host-mapped error record and traps (fail-stop).
- ll/transport.py: the Lamport buffer and flags are the plugin's own (torch
  symmetric memory, or FlashInfer's McastGPUBuffer) in FlashInfer's MNNVL
  layout; no FlashInfer workspace or flags are shared. Modules load through
  ll.jit; a build failure raises on every rank.
- ll/hops.py: the GB300 launch plan per world size and row count (W=4: one-shot
  up to 36 rows, two-shot 37-48), the family (deadline min(20 s, the build
  context's), o_proj through ll.linear) and its step group. Covers the TP4
  verify step on sm_103 up to 48 rows; provides hops:finalize and
  mlp_input:nvfp4 (written only when a taken MLP unit consumes it).

Tests: the plan tables (CPU); at W=4 every row 1-48 bit-identical to
FlashInfer's one-shot / two-shot, the finalize hop bit-identical to
moe_finalize_fuse_shared + the hop, a protocol mix eager and in a CUDA graph;
fail-stop of every cross-GPU wait; a two-tray multicast probe.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* [Perf] [GB300] glm5_3: TP4 MoE and dense MLP steps with the deferred finalize (ll.moe)

The units ll.moe.nvfp4 and ll.moe serve the MLP step (MlpSteps) of the TP4
verify step and leave the MoE finalize to the next hop:

- MoE layers: the router GEMM (ll.linear("router"), FP32 logits), then
  FlashInfer's public trtllm_fp4_block_scale_moe with do_finalize=False and the
  layer's processed NVFP4 weights (SGLang's
  fused_experts_none_to_flashinfer_trtllm_fp4 call for this configuration), and
  the shared expert on the layer's alternate stream; returns a deferred
  finalize for the next hop.
- ll.moe.nvfp4 takes the routed experts' FC1 input from the post-attention
  hop's NVFP4 epilogue; ll.moe quantizes its own input.
- Dense layers: the stock gate_up_proj and SiLU-and-mul, then
  ll.linear("dense_down"), un-reduced.
- Opt-in slot ll.moe.alt_stream_priority: the shared expert on a stream of the
  greatest priority at <= 6 rows. It gains only with FlashInfer's small-batch
  routing kernel (TP4 5/1/6: -1.82% step at 6 rows); on stock FlashInfer 0.7.2
  it loses (+1.2% at 5 rows, TP4 4/1/5, n = 2; +2.4% at 6 rows), so it runs only
  when listed in SGLANG_NVIDIA_MEGAKERNEL_SLOTS.
- Both require hops:finalize. A layer that does not fit declines the family
  with the layer and the reason.

Tests (4 GPUs): the copied call equals SGLang's runner bit for bit at 5-48 rows;
the layer (hop + MoE + finalize hop) equals the stock layer bit for bit on the
hop's FP4 input inside FlashInfer's one-shot band; CUDA-graph replays equal
eager; the priority stream keeps the bits.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* [Perf] [GB300] glm5_3: TP4 NextN draft composed from units (P6a eh_proj, LL attention, row select)

The draft step (draft_step.FusedDraft) serves TP8. At a TP4 profile the draft
spec now builds draft_units.ComposedBuiltDraft: each draft graph (draft decode
at one row per request, draft extend at the verify rows) is composed through
the registry from the draft step groups (draft_front, draft_eh,
draft_attention, draft_mlp, draft_top1):

- draft.front (slot draft): the draft step's fused embedding + eh norms (the
  kernel does not depend on the TP size);
- ll.draft_eh (P6a, ll/draft.py): eh_proj row-parallel, each rank's K window
  times its weight columns. With the target's LL hops (found through the
  shared-table registry, runtime.register_target_hops) the partials are summed
  and the draft layer's input norm applied in one hop launch, and the layer
  starts from that (draft_attention.PreparedInput: the attention prepare is
  skipped); otherwise SGLang's all-reduce and the stock norm. Per draft forward
  at TP4, L2 flushed: 20-26 us vs 39-41 us for the replicated GEMM;
- the draft layer's attention on ll.attention and ll.sparse_mla_fd (the stock
  DraftAttention on the step's ForwardView elsewhere);
- draft_rowselect in the draft extend (draft_step.select_index_rows);
- the LM head on ll.gemm's lm_head role; the top-1 stays SGLang's (at TP4 its
  one-tray multimem gather is as fast as the distributed argmax or faster).

The composed draft is built when the target shares its embedding and LM head
(set_embed_and_head, before the KV pool is sized), so the LM-head role is built
on the target's head. Draft graphs outside the profile's widths run SGLang's
draft forward. The TP8 draft keeps the TP8 widths (<= 16 rows).

Tests: test/unit/test_glm5_3_draft_units.py and test_glm5_3_draft_eh_prepared.py
(CPU: widths, compositions, slots, the hand-off to the logits processor),
test/kernels/test_glm5_3_draft_eh_proj.py (4 GPUs: numerics at 1-48 rows, the
hop sum + norm, graph replay).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* [Test] glm5_3: Serving tests and decode A/B configuration at the AgentX TP4 / TP8 points

- test/e2e/glm5_3_server.py, test_glm5_3_server.py: servers at the InferenceX
  AgentX configurations (tp4-416: aggregated TP4, EAGLE 4/1/5, HiCache;
  tp8-516: aggregated TP8 on two trays, EAGLE 5/1/6, HiCache; tp4-516: the
  disaggregated decode role, TP4, EAGLE 5/1/6, standalone): on / off (the
  active line, covered decode graphs, GSM8K-200 within 3 points), parity
  (SGLANG_NVIDIA_MEGAKERNEL_SLOTS="" vs flag off, greedy ids and logprobs
  bitwise) and HiCache load-back (load-back output equals the device-hit
  output).
- benchmark/README.md: the GLM decode A/B protocol at those configurations.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* [Feat] [GB300] glm5_3: Serve the TP4 decode role of a PD-disaggregated deployment

A PD deployment of GLM-5.2 on GB300 (the InferenceX AgentX 1p5d-c30 / 1p3d-c40
points) runs a TP1 PP4 prefill role and TP4 EAGLE 5/1/6 decode roles. The spec now
lists Feature.PD_DISAGGREGATION, so a decode role serves its verify steps on the
TP4 kernel units: the KV arrives through the transfer backend (NIXL) into pool
slots before a request's first verify step, outside the covered step, and the
decode radix cache is not a framework feature. The TP8 envelope keeps PD refused
(pd_reasons). The prefill role runs without the flag (TP1 and PP are refused
anyway).

Tests: test/unit/test_glm5_3_spec.py (the decode role accepted at TP4, refused at
TP8); test/e2e/test_glm5_3_server.py TestGlm53PDDecode (GSM8K through a running
1P1D router, decode engaged, prefill not, no NIXL transfer failure).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* [Test] glm5_3: Remove the tests this PR added (repo policy: no tests in sgl-nvidia-mega)

Per review, sgl-nvidia-mega carries no tests. Remove the 28 test files this
PR added under test/e2e, test/kernels and test/unit (the GLM-5.3 TP4, draft
row select and PD decode tests and their helpers glm5_3_server.py and
glm5_3_slots.py), and drop the benchmark README's pointer to the removed
test/e2e/glm5_3_server.py. Tests that already exist on the base branch are
left untouched.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* glm5_3: Keep SGLang's copyright notice in the headers of files adapted from SGLang

Apache-2.0 requires derivative files to retain the copyright notices of the
work they adapt. Following the OSRB guidance, add
"Portions Copyright 2023-2024 SGLang Team (Apache-2.0)" after the NVIDIA SPDX
lines of the glm5_3 files adapted from SGLang, and name the origin of
ll/moe.py, whose routed-expert call copies SGLang's. fused_a_ll.cuh copies
most of SGLang's dsv3_fused_a_gemm.cuh, so it keeps that file's header
unchanged and adds the NVIDIA and SGLang lines below it. Header lines only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* [Fix] glm5_3: pin the SGLang mirrors for main's SGLang (cf6133a)

main's unit tests check out SGLang cf6133a, where eight glm5_3 mirrors no
longer matched. Each change keeps the covered steps as mirrored, so the new
digest is added beside the old ones (both SGLangs pass):

- forward_absorb_prepare / forward_absorb_core (#44503, #38943): the indexer,
  the unfused RoPE and the W_UV projection moved into the run_mla_indexer,
  prepare_mla_qkv and project_mla_values hooks, whose defaults are the old
  calls (no SGLang class overrides them); the DCP extend gather is out of the
  spec's setups.
- Indexer.forward_cuda, DeepseekSparseAttnBackend._forward_trtllm (#46890):
  prefill-CP only.
- DeepseekV2DecoderLayer.forward (#46857), in the hops, the MoE step and the
  draft attention: an assert under CP decode attention TP, which the spec
  refuses.
- LogitsProcessor.forward (#47200): vocab-sharded logits for a target verify
  only; the draft steps the mirror covers take the gather as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* [Doc] glm5_3: Document the TP4 split-K GEMM source switch in the README

SGLANG_NVIDIA_MEGAKERNEL_GLM_GEMM_SPLITK picks the split-K source of the TP4
small-M GEMMs (auto, flashinfer or vendored); document it next to the TP4
units, as the README documents Qwen3.5's SGLANG_NVIDIA_MEGAKERNEL_MOE_TACTICS.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Yangmin Li <yangminl@nvidia.com>
Co-authored-by: shuw <shuw@nvidia.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Kaixi Matteo Chen <kaiximatteoc@nvidia.com>
yuhe-drinks-soda pushed a commit to t-head/sglang-for-sail that referenced this pull request Nov 15, 2026
Keep the ZW810E identification helper used by the later tensor-core query
and Marlin/SBO patches. Reuse the target release SAIL FA3 subclasses,
backend admission, defaults and safety fallbacks from upstream sgl-project#45946.
Do not restore the superseded global FA3 hooks.

(cherry picked from commit f3fb1dd)

Migration: Migrate only the board helper; retain the target SAIL attention architecture.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bypass-fail-fast CI: a failing job no longer aborts its siblings (lint still gates) deepseek dependencies Pull requests that update a dependency file highest-priority CI: all three control labels, plus never batch-cancelled or stale-closed parallel-stages CI: stages dispatch together instead of waiting on each other run-ci CI: run the baseline test suite on this PR run-ci-ppu sgl-kernel unified-radix-cache

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants