Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
28 commits
Select commit Hold shift + click to select a range
912963d
[Feat] [GB300] qwen3_5: NVIDIA megakernel plugin for Qwen3.5-397B-A17…
YAMY1234 Oct 21, 2026
ef31703
[Perf] [GB300] qwen3_5: Slot gdn_push: the GatedDeltaNet out_proj as …
YAMY1234 Oct 21, 2026
c696d45
[Perf] [GB300] qwen3_5: Slot gdn_front: the GatedDeltaNet projections…
YAMY1234 Oct 21, 2026
d147d3d
[Perf] [GB300] qwen3_5: Slot route_shared: the shared expert's gate/u…
YAMY1234 Oct 21, 2026
f8b4b09
[Perf] [GB300] qwen3_5: Slot skv2: the full attention's core on the s…
YAMY1234 Oct 21, 2026
6c6142e
[Perf] [GB300] qwen3_5: Slot mka: the full attention's fused prologue…
YAMY1234 Oct 21, 2026
5508ef9
[Perf] [GB300] qwen3_5: Slot mka_push: the attention epilogue pushes …
YAMY1234 Oct 21, 2026
258cbe2
[Feat] qwen3_5: Entries draft_loop and draft_extend: the MTP draft's …
YAMY1234 Oct 21, 2026
7a839b9
[Perf] [GB300] qwen3_5: Draft slot draft_moe: the MTP draft's MoE in …
YAMY1234 Oct 21, 2026
256406d
[Perf] [GB300] qwen3_5: Draft slot draft_q1: the draft steps' q1 spli…
YAMY1234 Oct 21, 2026
4b56c1f
[Perf] [GB300] qwen3_5: Draft slot draft_xchg: the draft step boundar…
YAMY1234 Oct 21, 2026
3c99a88
[Perf] [GB300] qwen3_5: Draft slot draft_hop: the draft's all-reduce …
YAMY1234 Oct 21, 2026
55b8ea0
[Perf] [GB300] qwen3_5: Draft slot draft_front: the draft attention's…
YAMY1234 Oct 21, 2026
71974f5
[Perf] [GB300] qwen3_5: Draft slot draft_skv2: the draft extend's att…
YAMY1234 Oct 21, 2026
c679a3b
[Perf] qwen3_5: The draft extend writes its token and selected rows s…
YAMY1234 Oct 21, 2026
a7b9463
[Perf] [GB300] qwen3_5: The GDN front takes each projection CTA's wei…
YAMY1234 Oct 21, 2026
eb57195
[Perf] [GB300] qwen3_5: SKV2's producer and merge launch as programma…
YAMY1234 Oct 21, 2026
a183562
[Perf] [GB300] qwen3_5: The draft q1 decode and its merge launch as p…
YAMY1234 Oct 21, 2026
0b0edc2
[Perf] [GB300] qwen3_5: The final hop releases the draft head at its …
YAMY1234 Oct 21, 2026
90e5c47
[Perf] [GB300] qwen3_5: On hidden ranks the draft head leaves SMs to …
YAMY1234 Oct 21, 2026
0d052c8
[Perf] [GB300] qwen3_5: The draft step's front launches as a programm…
YAMY1234 Oct 21, 2026
6c2a69f
[Perf] [GB300] qwen3_5: The draft o_proj GEMV applies the attention o…
YAMY1234 Oct 21, 2026
a9c8b8d
[Feat] [GB300] qwen3_5: Serve TP4 on one tray, with slot mkm's engine…
YAMY1234 Oct 21, 2026
43b40a0
[Perf] [GB300] qwen3_5: Opt-in multi_request: cover TP8's verify step…
YAMY1234 Oct 21, 2026
01d1711
[Perf] [GB300] qwen3_5: Slot mkm_wide: the fused routed experts of 8-…
YAMY1234 Oct 21, 2026
c5c714d
[Perf] [GB300] qwen3_5: Entry draft_loop serves two requests at TP8 w…
YAMY1234 Oct 21, 2026
88180fb
Merge remote-tracking branch 'upstream/main' into qwen35-gb300-full
YAMY1234 Oct 26, 2026
3bd8c0b
Adapt megakernel specs to SGLang API v2
YAMY1234 Oct 26, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/CODEOWNERS
Original file line number Diff line number Diff line change
Expand Up @@ -4,4 +4,5 @@
# Per model: one owner for its step code, kernels and tests. Each new model's PR
# adds its line.
**/*deepseek_v41* @YAMY1234
**/*qwen3_5* @YAMY1234
**/*kimi_k3* @leejnau
11 changes: 11 additions & 0 deletions NOTICE
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,17 @@ decode step of DeepSeek-V4.1 (the layer loop, the mHC boundaries, the attention,
MoE, Engram and the head) follows SGLang's own model code (Apache License 2.0,
Copyright SGLang Team). Each adapted file names its origin in its header.

Portions are adapted from SGLang (https://github.com/sgl-project/sglang): the stock
verify step of Qwen3.5 (the layer loop, the GatedDeltaNet, the attention and the MoE)
follows SGLang's own model code (Apache License 2.0, Copyright SGLang Team). Qwen3.5's
GatedDeltaNet verify slots follow SGLang's linear-attention backend; its causal conv
update (Apache License 2.0, Copyright contributors to the vLLM project, Copyright (c)
2024 Tri Dao); its gated RMSNorm Triton kernel, which SGLang adapted from
flash-linear-attention (MIT License, Copyright (c) 2024 Tri Dao); and its ReplaySSM
verify kernel, which SGLang ported from the vLLM ReplaySSM reference implementation
(Apache License 2.0, Copyright contributors to the vLLM project). Each adapted file
names its origin in its header.

Portions are adapted from FlashInfer (https://github.com/flashinfer-ai/flashinfer):
the Lamport all-reduce primitives and the low-latency all-reduce (Apache License
2.0, Copyright (c) 2026 by FlashInfer team), and the epilogue of the wo_b GEMM
Expand Down
54 changes: 49 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,12 +15,14 @@ The package loads as an SGLang [general plugin](https://github.com/sgl-project/s
| Model | GPU | Tensor parallel | Coverage |
| --- | --- | --- | --- |
| DeepSeek-V4.1-Flash | GB300 (SM103) | 4 | DSpark target verify steps of 1 to 384 rows (batch 1 to 64 at 6 rows a request) |
| Qwen3.5-397B-A17B (NVFP4) | GB300 (SM103) | 8 (two trays, one NVLink domain) | MTP target verify steps of 7 rows (batch 1) |
| Kimi K3 (packed MXFP4) | GB300 (SM103) | 8 | One-token decode steps of 1–8, 16, 32 and 64 rows |

For DeepSeek-V4.1-Flash, larger verify steps, prefill and extend run SGLang's
standard path. The DSpark draft step runs on the plugin's draft spec. For Kimi
K3, the plugin covers the 93-layer language model (24 MLA and 69 KDA layers) at
the listed widths; every other call runs the standard path.
For DeepSeek-V4.1-Flash and Qwen3.5, larger verify steps, prefill and extend run
SGLang's standard path. Their draft specs provide the DSpark proposal and the
Qwen3.5 EAGLE entries, respectively. For Kimi K3, the plugin covers the 93-layer
language model (24 MLA and 69 KDA layers) at the listed widths; every other call
runs the standard path.

The Kimi K3 adapter is experimental. On the pinned GB300/TP8 configuration
below, every listed CUDA-graph width and padding boundary passed dummy-weight
Expand Down Expand Up @@ -48,6 +50,26 @@ The step is the plugin's own code over the weights SGLang loaded (`deepseek_v41/

With the megakernel on, the default 39 slots run: the ones above, the attention chain, hop B's triggers, and the DSpark draft step's 16 slots. `SGLANG_NVIDIA_MEGAKERNEL_SLOTS` (comma-separated) overrides that list for tests and bisecting; empty runs every step function's stock code. The opt-in slots run only when listed there: `target_lm_head` (the target's LM head on the low-M kernel at 1-8 rows), `lm_head_pinned_tactic` (both low-M LM heads on one fixed tactic per row count, untuned), `hop_b_quad_trigger` (hop B's finalize-publish launches its consumer right after its own wait, batch 1 only), `hop_b_cap16` and `draft_embed`.

Qwen3.5 (`qwen3_5/runtime`) runs the same way: per layer the stage boundary (`boundary/`), the GatedDeltaNet or the attention (`gdn/`, `attention/`) and the MoE (`moe/`), each with a stock implementation that reproduces SGLang's verify step bit for bit. It reads the running forward's metadata and pools through SGLang's EAGLE binding (`sglang.srt.kernel_plugins.nvidia.eagle_binding`). Its slots run only when listed in `SGLANG_NVIDIA_MEGAKERNEL_SLOTS`, and each is bit for bit SGLang's:

| Slot | Rows | Kernel |
| --- | --- | --- |
| `route` | 7 | The router GEMM and the NVFP4 quantization of the MoE input in one launch |
| `mkm` (with `route`) | 7 | On `route`'s outputs, the top-10, FC1, SwiGLU, the NVFP4 requantization and FC2 in one launch |
| `gdn_verify` | 7 | The GatedDeltaNet verify chain (conv update, split, ReplaySSM verify, gated norm with the FP8 `out_proj` input) with its launch choices changed |
| `gdn_pdl` (with `gdn_verify`) | 7 | Its ReplaySSM verify launched as a programmatic dependent of the conv |
| `hop` | 7 | The layer boundary's fused all-reduce + residual add + RMSNorm points on an NVLink multicast hop |

These slots change the arithmetic (numeric: each sums in another order than SGLang's op, within the tolerances `spec.py` declares):

| Slot | Rows | Kernel |
| --- | --- | --- |
| `gdn_push` (with `gdn_verify`, `hop`) | 7 | The GatedDeltaNet `out_proj` as the hop's GEMM, whose epilogue pushes the partial into the hop's mailbox |
| `gdn_front` (with `gdn_verify`, `gdn_pdl`) | 7 (one request) | The GatedDeltaNet `in_proj_qkvz` (static FP8) and `in_proj_ba` (BF16) projections, the conv update and an L2 prefetch of the verify's state in one launch, then the ReplaySSM verify and the gated norm writing `out_proj`'s FP8 input, each launched dependent on the one before |
| `route_shared` (with `route`) | 1–8 | The shared expert's static-FP8 gate/up in the router front's launch, so the shared expert on the alternate stream waits for that launch only |

The sampling and acceptance after the verify, and the GatedDeltaNet extend of prefill, stay SGLang's own code. In an SGLang that has the switches, `SGLANG_ENABLE_EAGLE_POST_VERIFY_CUDA_GRAPH=1` runs the sampling and acceptance in CUDA graphs, and `SGLANG_ENABLE_GDN_PREFILL_GRAPH_EXTEND=1` captures the GatedDeltaNet extend inside the breakable prefill CUDA graphs.

### Design

- **Built at load.** Buffers, kernels and tactics for every covered width are allocated and compiled when the model is loaded, before the KV cache is sized. Nothing is compiled or allocated inside a step.
Expand All @@ -63,9 +85,10 @@ With the megakernel on, the default 39 slots run: the ones above, the attention

### Requirements

- An SGLang with the megakernels framework (`sglang.kernels.plugin_kernels.nvidia`, or `sglang.srt.megakernels` before sgl-project/sglang#45389; `--enable-nvidia-megakernel`)
- An SGLang with the megakernels framework (`sglang.kernels.plugin_kernels.nvidia`, or `sglang.srt.megakernels` before sgl-project/sglang#45389; `--enable-nvidia-megakernel`); Qwen3.5 also needs `sglang.srt.kernel_plugins.nvidia.eagle_binding`
- SGLang's CUDA 13 image: FlashInfer (its low-latency block-scaled GEMM as in 0.7.0 to 0.7.2), CuTe DSL (`nvidia-cutlass-dsl`) and the CUDA Python bindings
- DeepSeek-V4.1-Flash: 4 x GB300
- Qwen3.5: 8 x GB300, two trays of one NVLink domain
- Kimi K3: 8 x GB300 (SM103), BF16, context length 8192, TP8 and DP1,
FP8 E4M3 KV cache with page size 64, `trtllm_mla` for prefill and decode,
`flashinfer_mxfp4` with default precision, and full decode CUDA graphs
Expand Down Expand Up @@ -98,6 +121,24 @@ python -m sglang.launch_server \

The three environment variables are required by the DeepSeek-V4.1 spec; without them the server stops at startup, naming each.

Qwen3.5, on each of the two trays (`--node-rank 0` and `1`):

```bash
SGLANG_NVIDIA_MEGAKERNEL_SLOTS=route,mkm,gdn_verify,gdn_pdl,hop \
python -m sglang.launch_server \
--model-path nvidia/Qwen3.5-397B-A17B-NVFP4-V2 \
--tp 8 --nnodes 2 --node-rank 0 --dist-init-addr <tray-0 host>:<port> \
--kv-cache-dtype fp8_e4m3 \
--speculative-algorithm NEXTN --speculative-num-steps 6 \
--speculative-eagle-topk 1 --speculative-num-draft-tokens 7 \
--attention-backend trtllm_mha \
--linear-attn-decode-backend triton \
--enable-linear-replayssm-spec --mamba-ssm-dtype bfloat16 \
--moe-runner-backend flashinfer_trtllm \
--flashinfer-allreduce-fusion-backend cutedsl \
--enable-nvidia-megakernel
```

## Repository Structure

```text
Expand All @@ -115,6 +156,7 @@ sgl-nvidia-mega/
│ │ ├── engram/ # the Engram layers
│ │ ├── logits/ # the head and the logits
│ │ └── sm103/ # kernels and the build-time runner; no SGLang imports
│ ├── qwen3_5/ # spec, plan, runtime, boundary, GDN, attention, MoE and SM103 kernels
│ └── kimi_k3/ # the Kimi K3 spec, adapter, executor and CuTe DSL kernels
├── benchmark/ # the static decode table behind the speed numbers
└── test/
Expand All @@ -131,6 +173,8 @@ pytest test/kernels # DeepSeek-V4.1-Flash: 4 x GB300
pytest test/e2e # DeepSeek-V4.1-Flash: 4 x GB300, with SGLANG_NVIDIA_MEGAKERNEL_E2E_MODEL_PATH set
```

Qwen3.5 runs TP8, which spans two GB300 trays: its e2e tests read a server launched on them through `SGLANG_NVIDIA_MEGAKERNEL_E2E_URL` and the files its ranks log to (`SGLANG_NVIDIA_MEGAKERNEL_E2E_LOGS`, comma-separated).

Kimi K3 GPU and end-to-end suites are not yet part of these commands.

## Contributing
Expand Down
4 changes: 4 additions & 0 deletions sgl_nvidia_mega/deepseek_v41/draft.py
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,10 @@ class DraftProposal:
"""The draft spec's built object: SGLang's forward on every step, and the proposal."""

widths: tuple[int, ...] = ()
draft_loop = None
draft_extend = None
begin_draft_round = None
graph_variants = None

def __init__(self, lm, *, deadline_s: float, slots: frozenset[str]):
self.lm, self.deadline_s = lm, deadline_s
Expand Down
6 changes: 4 additions & 2 deletions sgl_nvidia_mega/deepseek_v41/moe/step.py
Original file line number Diff line number Diff line change
Expand Up @@ -63,7 +63,7 @@
"moe_runner_config",
),
"srt/layers/moe/utils.py": ("sum_post_experts_output",),
"srt/layers/layer_boundary/fusions/cutedsl.py": (
"srt/layers/layer_boundary/fusions/moe_finalize.py": (
"MoeDeferredFinalize",
"from_flashinfer",
),
Expand Down Expand Up @@ -324,7 +324,9 @@ def _dual_stream(
current.wait_stream(mlp.alt_stream)

if deferred and fwd.defer_moe_finalize:
from sglang.srt.layers.layer_boundary.fusions.cutedsl import MoeDeferredFinalize
from sglang.srt.layers.layer_boundary.fusions.moe_finalize import (
MoeDeferredFinalize,
)
from sglang.srt.layers.moe.utils import sum_post_experts_output

return MoeDeferredFinalize.from_flashinfer(
Expand Down
6 changes: 5 additions & 1 deletion sgl_nvidia_mega/deepseek_v41/runtime/executor.py
Original file line number Diff line number Diff line change
Expand Up @@ -107,8 +107,12 @@ class MegakernelDecode:
"""One rank's covered steps: the model, each width's slots and each package's
build result."""

# the target offers no draft sampler (BuiltMegakernel)
# the target offers no speculative entries or graph variants (BuiltMegakernel)
draft_sampler = None
draft_loop = None
draft_extend = None
begin_draft_round = None
graph_variants = None

def __init__(
self,
Expand Down
2 changes: 1 addition & 1 deletion sgl_nvidia_mega/deepseek_v41/spec.py
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,7 @@

ARCHITECTURE = "DeepseekV4ForCausalLM"
# the SGLang megakernel contract this spec is written against
API_VERSION = 1
API_VERSION = 2
# The FlashInfer low-latency GEMM file the wo_b publish epilogue subclasses, by the
# sha256 of its source; it is byte-identical in FlashInfer 0.7.0, 0.7.1 and 0.7.2
FLASHINFER_LL_GEMM = "gemm/kernels/cute_dsl/low_latency_blockscaled_gemm.py"
Expand Down
4 changes: 4 additions & 0 deletions sgl_nvidia_mega/kimi_k3/spec.py
Original file line number Diff line number Diff line change
Expand Up @@ -147,6 +147,10 @@ class _Built:
"""Top-level model-forward replacement for the exact decode shapes served."""

draft_sampler = None
draft_loop = None
draft_extend = None
begin_draft_round = None
graph_variants = None

def __init__(self, model: Any, resources: Any, widths: tuple[int, ...]):
from .executor import KimiK3MegakernelExecutor
Expand Down
3 changes: 3 additions & 0 deletions sgl_nvidia_mega/plugin.py
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,9 @@
("DeepseekV4ForCausalLM", "sgl_nvidia_mega.deepseek_v41.spec:SPEC"),
# the DSpark draft: SGLang's forward, the megakernel's folded proposal
("DeepseekV4ForCausalLMDSpark", "sgl_nvidia_mega.deepseek_v41.draft:SPEC"),
("Qwen3_5MoeForConditionalGeneration", "sgl_nvidia_mega.qwen3_5.spec:SPEC"),
# The MTP draft offers EAGLE draft-loop and draft-extend entries.
("Qwen3_5ForCausalLMMTP", "sgl_nvidia_mega.qwen3_5.draft:SPEC"),
("KimiK3ForConditionalGeneration", "sgl_nvidia_mega.kimi_k3.spec:SPEC"),
)

Expand Down
31 changes: 31 additions & 0 deletions sgl_nvidia_mega/qwen3_5/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
"""Qwen3.5 (397B-A17B, NVFP4) megakernel: its spec and the setups it serves (``spec``),
the slot plan (``plan``), the executor of a covered target-verify step and its build
(``runtime``), the step code per package (``boundary``, ``gdn``, ``attention``, ``moe``,
``logits``), and the SM103 (GB300) kernels (``sm103``). The MTP draft runs SGLang's own
forward."""

# Qwen3.5-397B-A17B's text_config (the HF config nests it under text_config), which
# the kernels are built for
TEXT_CONFIG = dict(
model_type="qwen3_5_moe_text",
hidden_size=4096,
num_hidden_layers=60,
full_attention_interval=4,
num_attention_heads=32,
num_key_value_heads=2,
head_dim=256,
attn_output_gate=True,
linear_num_key_heads=16,
linear_num_value_heads=64,
linear_key_head_dim=128,
linear_value_head_dim=128,
linear_conv_kernel_dim=4,
num_experts=512,
num_experts_per_tok=10,
moe_intermediate_size=1024,
shared_expert_intermediate_size=1024,
hidden_act="silu",
rms_norm_eps=1e-6,
)
126 changes: 126 additions & 0 deletions sgl_nvidia_mega/qwen3_5/attention/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,126 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# Adapted from SGLang (Apache-2.0, SGLang Team):
# python/sglang/srt/models/qwen3_5.py (Qwen3_5AttentionDecoderLayer.self_attention).
"""The gated full attention of a covered target-verify step, over the layer's loaded
weights, SGLang's attention backend and its KV cache.

Stock: the CUDA arm of ``Qwen3_5AttentionDecoderLayer.self_attention`` with SGLang's own
ops in its order: the qkv projection with the fused q / k Gemma RMSNorm + RoPE + gate
split, the attention through the backend (which writes this step's KV rows), the
sigmoid output gate and ``o_proj``. ``step`` returns the TP-partial output, whose
all-reduce the boundary runs. Slot ``skv2`` (``split_kv.py``) runs the attention's core
through the split-KV attention of SGLang branch nvidia/qwen35-agentx-gb300, and slot
``mka`` (``mka.py``) the rest of the layer through that branch's fused prologue and
epilogue.
"""

from __future__ import annotations

import msgspec

# The SGLang names this module reads (test/unit/test_contract.py pins them)
SGLANG_READS = {
"srt/models/qwen3_vl.py": ("is_mrope_enabled",),
"srt/models/qwen3_5.py": (
"Qwen3_5AttentionDecoderLayer",
"layers",
"start_layer",
"end_layer",
"_prepare_qkv_gate",
"_prepare_qkv_gate(positions)",
"_prepare_qkv_gate(hidden_states)",
"_prepare_qkv_gate(forward_batch)",
"attn",
"attn_output_gate",
"o_proj",
),
"kernels/ops/elementwise/elementwise.py": (
"fused_sigmoid_mul",
"fused_sigmoid_mul(inplace)",
),
}

# The SGLang functions whose body this module mirrors, by the digest of their code
# (test/unit/test_contract.py recomputes them on the installed SGLang)
SGLANG_MIRRORS = {
"srt/models/qwen3_5.py:Qwen3_5AttentionDecoderLayer.self_attention": "db0f4d98e0fb1abf",
}

ATTENTION_SLOTS = ("skv2", "mka", "mka_push")


class Bound(msgspec.Struct, frozen=True):
"""The attention package's build result (``st.bound["attention"]``)."""

skv2: bool = False # slot skv2 serves every full-attention layer
mka: bool = False
mka_push: bool = False
reason: str = "" # why a requested slot is not built


def build(model, ctx) -> Bound:
"""The requested attention slots for every full-attention layer, or why a layer
cannot take them."""
from sglang.srt.models.qwen3_5 import Qwen3_5AttentionDecoderLayer

from sgl_nvidia_mega.qwen3_5.attention import mka, split_kv
from sgl_nvidia_mega.qwen3_5.runtime import SLOTS

if not SLOTS & set(ATTENTION_SLOTS):
return Bound(reason="stock (SGLANG_NVIDIA_MEGAKERNEL_SLOTS)")
m = model.model
layers = [
(i, m.layers[i])
for i in range(m.start_layer, m.end_layer)
if isinstance(m.layers[i], Qwen3_5AttentionDecoderLayer)
]
for i, layer in layers:
reason = ""
if "skv2" in SLOTS:
reason = split_kv.unsupported(layer.attn, ctx.tokens_per_request)
if not reason and "mka" in SLOTS:
reason = (
mka.unsupported(layer)
if model.is_mrope_enabled
else "no M-RoPE positions (MKA reads them)"
)
if reason:
return Bound(reason=f"layer {i}: {reason}")
if "mka" in SLOTS:
for _, layer in layers:
mka.pack(layer)
return Bound(skv2="skv2" in SLOTS, mka="mka" in SLOTS, mka_push="mka_push" in SLOTS)


def pool_reasons(view, binding, skv2: bool = True, mka: bool = False) -> list[str]:
"""Why the attention slots cannot read this forward's KV pools ([] when they can)."""
from sgl_nvidia_mega.qwen3_5.attention import mka as fused
from sgl_nvidia_mega.qwen3_5.attention import split_kv

reasons = split_kv.pool_reasons(view.kv, binding.page_size) if skv2 else []
return reasons + (fused.pool_reasons(view.kv) if mka else [])


def step(layer, x, st):
"""``layer``'s attention on the normed input ``x`` [rows, hidden]: the TP-partial
output [rows, hidden]."""
from sglang.kernels.ops.elementwise.elementwise import fused_sigmoid_mul

if st.slots.mka:
from sgl_nvidia_mega.qwen3_5.attention import mka

return mka.step(layer, x, st)
q, k, v, gate = layer._prepare_qkv_gate(
positions=st.positions, hidden_states=x, forward_batch=st.fb
)
if st.slots.skv2:
from sgl_nvidia_mega.qwen3_5.attention import split_kv

o = split_kv.attend(layer.attn, q, k, v, st)
else:
o = layer.attn(q, k, v, st.fb)
if layer.attn_output_gate:
o = fused_sigmoid_mul(o, gate, inplace=True)
out, _ = layer.o_proj(o)
return out
Loading
Loading