Skip to content

[Feat] attention: Serve several window sub-pools per runner in the Triton backend - #48014

Merged
ch-wan merged 1 commit into
mainfrom
cheng/refactor/unified-kv-host-window-pools
Nov 9, 2026
Merged

ch-wan merged 1 commit into
mainfrom
cheng/refactor/unified-kv-host-window-pools

Conversation

@ch-wan

@ch-wan ch-wan commented Nov 3, 2026 •

Copy link
Copy Markdown
Collaborator

This PR is part of a stack (oldest at bottom):

Motivation

With --enable-unified-memory, a model whose layers attend through windows of different lengths is refused at startup ("multiple target windows require named model declarations"): the plan holds one target window, and every attention backend carries the write ids of one window sub-pool (KVWriteLoc.swa_loc). The storage below already keeps a sub-pool, a prefix-tree slot, a host pool and a PD state entry per window, and a fused draft's window of another length already lives in a sub-pool of its own. What is missing is the attention side: a runner whose layers read several window sub-pools, and a backend that builds window metadata for each of them.

This PR lets a plan declare several target windows and serves them in the Triton backend. Other backends keep serving one window per runner, and a runner whose layers need more is refused when its backend is bound, naming the backend.

Modifications

  • Declarations and plan (mem_cache/kv/decl.py, kv/plan.py, kv/build.py): kv_layer_decls(windows=[WindowLayers(...)]) declares one layer group per window length, MHA or MLA; the refusal of several target windows is removed. Routes and the router's sub-pools carry each window sub-pool's attention window (PartRoute.window, PublicSubPool.window).
  • Write ids (mem_cache/memory_pool.py, kv/router.py, kv/routed_cache.py, swa_memory_pool.py): KVWriteLoc.window_locs carries every window sub-pool's write ids by sub-pool name and replaces swa_loc; the fields after loc are keyword-only, and each window's ids must be contiguous, as loc and full_loc must. Backends key the ids they resolved for their window storage with window_write_locs; draft writers that address a layer directly (DFlash, DSpark) use layer_window_locs. FlashAttention, FlashInfer, Triton, TRT-LLM MHA, torch native, aiter, XPU, Intel AMX, MUSA and Ascend backends, the CP strategies and the DFlash worker follow.
  • Runner ids (kv/alloc/runner_ids.py, layers/attention/base_attn_backend.py): RunnerIds.window_sub_pools lists the window sub-pools a runner's layers read; RunnerIds.window_storage names the one a single-rail backend reads (it replaces window_kv_kind). Binding a backend whose max_window_sub_pools is smaller than the runner's window count raises NotImplementedError naming the leaf and the wrappers in front of it (check_window_capacity); wrappers answer through the backends they run (AttentionBackend.window_capacity_children: HybridAttnBackend's prefill and decode backends, two-batch overlap's primary and children, HybridLinearAttnBackend's full-attention backend; and a multi-step draft's step backends), and the smallest capacity wins (TestWindowCapacityThroughWrappers). Triton takes each window's length from the router's declared windows, the scalar sliding window only filling an unstated one, and refuses a window neither states; a fused draft's pre-load window rule reads the text config (gemma4_mtp.py).
  • Triton (layers/attention/triton_backend.py, dllm/attention.py, speculative/spec_utils.py): one WindowRail per window sub-pool; each window layer reads and writes through its own rail. A target runner builds several rails for ordinary MHA execution, in eager extend and decode and in decode CUDA graphs, when every window's length is stated (_builds_rails_per_window); draft runners, speculative decoding, decode context parallelism, deterministic inference, dLLM, MLA, skip-prefill backends and an externally supplied kv_indptr buffer are excluded; elsewhere it builds one rail, and a runner over several window sub-pools is refused at binding. A unified runner takes its rails from the window sub-pools its own layers read (RunnerIds.window_sub_pools), each with its own window, and refuses a window sub-pool that states none; it builds no rail over another runner's band (a fused draft's), and one over the full sub-pool only when the model states a scalar window. Decode scratch holds one logits buffer per distinct value width, taken from the sub-pools' geometry, and the ROCm Lean partial-output buffer is sized to the largest width rounded up to a power of two, as its kernel addresses it (ForwardMetadata.extra_attn_logits).
  • Fused drafts (layout/fused_draft.py, kv_cache_configurator.py, model_runner_components/load_model_utils.py, draft model classes, unified_draft_pool.py): the placement context carries every window the target declares (PlacementContext.target_windows), and a draft window rides in the target window sub-pool of its length, whichever it is. The draft's window is the one its own runner will attend over, stated from its config (resolve_sliding_layer_window_size, sliding_layer_window_size on DFlash, Dots3, Gemma 4 MTP, Inkling MTP and Llama Eagle3); a draft class whose rule needs the built model is refused with a message. A draft pool routes its window layers by their host sub-pool's name.
  • Tests:
    • new layers/attention/test_triton_several_windows.py (each window layer reads and writes its own rail) and, nightly, test_triton_several_windows_nightly.py (decode CUDA graph replay and value widths);
    • new layers/attention/test_window_id_space_resolution.py (a draft window band resolves to its own storage) and models/test_draft_sliding_window_rules.py;
    • several target windows in test_unified_window_sub_pools.py, disaggregation/test_unified_window_lengths.py, disaggregation/test_unified_window_wire.py, test_unified_chain_realize.py (three window floats) and managers/test_prefill_demand_agreement.py, which below this PR build the target's window and a fused draft's window instead;
    • write-id updates in the backend, CP and pool tests.

Behaviour changes

  • A plan with several target windows serves under --enable-unified-memory with the Triton backend in the modes above; elsewhere, and with another backend, it is refused when the backend is bound, naming the backend and its capacity. The serving configurator still declares one target window: several reach a plan through a model declaration.
  • A fused draft's window is placed by its own architecture's window rule; a draft whose rule is known only once built is refused at placement.
  • No change for a model with one target window, or without --enable-unified-memory.

Accuracy and Speed Tests

All results below are at the top of this stack (this PR plus a test-only commit, 78123128d7) on main a4b8801197, except where a row names another revision. The serving rows are at earlier tops on main 5e1774d137: the openai/gpt-oss-120b throughput runs at c03f2e2cf2, and the full GSM8K, PD and MTP rows at 5e3702a3e5. Since those runs the stack changed allocator host bookkeeping and the capacity rules, and was rebased.

Unit tests (B300, torch 2.14.1+cu130, flashinfer 0.7.2, sglang-kernel 0.4.10), each file in its own process, at the default invariant level and with SGLANG_INVARIANT_CHECK=2 (strict): 222 files (every unit test file this stack adds or changes, the Triton-backend tests, and related main tests). Failures, all also on main or from the environment: test_srt_endpoint.py, test_disaggregation_optimistic_prefill.py and test_input_embeds_chunked.py (gated models unavailable), test_minimax_verify_topk_idx.py (fp8e4b8 unsupported on B300), test_eagle_binding.py, test_unified_hicache_transfer_lifecycle.py (a torch stream type error with a GPU visible), test_unified_scheduler_hooks.py (a scheduler stub without PauseState, failing on main too), and test_prefill_send_protection.py at strict. test_inkling_checkpoint_indices.py passes 8 of 8 runs at both levels. These files ran one revision before the current stack. The current revision changes continuation chunk fitting, the window branch tail of the request demand, the cross-stage state-entry check, the window metric label and the host-receive startup check; the affected unit test files (11 files, 187 tests, B200, torch 2.14.1+cu130, flashinfer 0.7.2) pass at the default and strict invariant levels at this PR. The whole test/registered/unit tree at the default level (each subdirectory in its own process, about 15,500 passing tests) fails the same tests at this PR as on main a4b8801197, all from the environment (gated models, kernels and toolchains unavailable on the machine). The stack's comments and docstrings are condensed and its unit tests trimmed (duplicate and mirror cases removed, a few files merged); production code is unchanged by that, as a comparison of every changed file's syntax tree without docstrings shows, and every changed test file passes at both invariant levels, apart from test_unified_hicache_transfer_lifecycle.py with a GPU visible (as above) and the weekly test_unified_page_safety_sweep_cuda.py, which passes on its own but not after the other files in one process.

Page-safety stress harness (test/manual/mem_cache/unified_page_safety_stress.py, CPU), --seeds 0:64 --ops 300 at the default and strict invariant levels: 0 failures each. The CPU suite and the longer sweeps (--seeds 0:512 --ops 400, --seeds 0:32 --ops 3000, 0 failures) are at an earlier top on main 6d1ce5b894.

Servers start and answer a request (B300): Qwen/Qwen3-0.6B with the default backend, flashinfer, and Triton prefill with FlashInfer decode; openai/gpt-oss-20b with --enable-unified-memory on the default and Triton backends.

Rust (rust/): cargo fmt --check and cargo clippy --workspace -D warnings pass.

Serving (H200, driver 595.71.05, torch 2.14.1+cu130, flashinfer 0.7.2, sglang-kernel 0.4.10, Mooncake transfer engine 0.3.13.post1), this stack's top against main 5e1774d137, --enable-unified-memory on every server. Each case ran one round, except openai/gpt-oss-120b with two (both shown, top / main):

Case Top of this stack main
openai/gpt-oss-120b TP1, overlap scheduling, --max-total-tokens 120000 --attention-backend fa3: bench_serving random 4096 in / 512 out, 200 prompts, concurrency 64, output throughput 1470.1 / 1473.1 tok/s 1470.1 / 1468.8 tok/s
same: mean TPOT 32.66 / 32.60 ms 32.69 / 32.72 ms
same: GSM8K, 200 questions 0.825 / 0.835 0.825 / 0.830
openai/gpt-oss-120b, --max-total-tokens 30000 (full pool, with retractions): GSM8K, all 1319 questions 0.8484 / 0.8431 0.8469 / 0.8431
Qwen/Qwen3.8-27B PD 1P1D on Mooncake, unified memory on both sides: GSM8K, 200 questions 0.975 0.975
same: random 2048 in / 256 out, 100 prompts, concurrency 32, output throughput 1052.2 tok/s 1083.7 tok/s
same: shared-prefix workload, output throughput 1248.0 tok/s 1238.4 tok/s
Qwen/Qwen3.8-27B MTP (EAGLE, 3 steps, top-k 1, 4 draft tokens), TP2, overlap scheduling: GSM8K, 200 questions 0.970 0.980
same: random 1024 in / 256 out, 100 prompts, concurrency 32, output throughput 2111.6 tok/s 2165.7 tok/s
same: accept length 3.50 3.49

On openai/gpt-oss-120b the top matches main in throughput and TPOT. On Qwen/Qwen3.8-27B the PD random workload is about 3% slower and the shared-prefix one about 1% faster, and MTP is about 2.5% slower; those cases ran one round each. GSM8K differences are within what one run moves at these sizes.

These models declare one target window; a model with several target windows was not served. Not run: PD with HiCache, staging or HiSparse; other models and hardware.

Checklist

  • Format with pre-commit on the changed files.
  • Unit tests added and updated.

CI States

Latest PR Test (Base): 🚫 Run #42609003129
Latest PR Test (Extra): 🚫 Run #42609003251
Latest PR Test (AMD ROCm 10): ❌ Run #42609003050

Tip

Base CI only covers the core cases, and its workload may not exercise your change. Use /rerun-test to run the tests your change touches, or copy this prompt for an agent:

Find the registered tests this PR affects and run them with /rerun-test, following https://github.com/sgl-project/sglang/blob/main/.agents/skills/rerun-test/SKILL.md

@ch-wan
ch-wan force-pushed the cheng/refactor/unified-kv-pd-window-entries branch from f82cc57 to ceff909 Compare November 4, 2026 06:18
@ch-wan
ch-wan force-pushed the cheng/refactor/unified-kv-host-window-pools branch from 6021a63 to 3f73c10 Compare November 4, 2026 06:18
@ch-wan
ch-wan force-pushed the cheng/refactor/unified-kv-pd-window-entries branch from ceff909 to a91493b Compare November 4, 2026 07:42
@ch-wan
ch-wan force-pushed the cheng/refactor/unified-kv-host-window-pools branch from 3f73c10 to e3f8fb6 Compare November 4, 2026 07:42
@ch-wan
ch-wan force-pushed the cheng/refactor/unified-kv-pd-window-entries branch from a91493b to f0221ef Compare November 4, 2026 08:02
@ch-wan
ch-wan force-pushed the cheng/refactor/unified-kv-host-window-pools branch from e3f8fb6 to a6a6227 Compare November 4, 2026 08:02
@ch-wan
ch-wan force-pushed the cheng/refactor/unified-kv-pd-window-entries branch from f0221ef to 18c97dd Compare November 4, 2026 09:14
@ch-wan
ch-wan force-pushed the cheng/refactor/unified-kv-host-window-pools branch from a6a6227 to 3ca176e Compare November 4, 2026 09:14
@ch-wan
ch-wan force-pushed the cheng/refactor/unified-kv-pd-window-entries branch from 18c97dd to ceb2bef Compare November 4, 2026 09:31
@ch-wan
ch-wan force-pushed the cheng/refactor/unified-kv-host-window-pools branch from 3ca176e to 158b380 Compare November 4, 2026 09:31
This was referenced Nov 4, 2026
@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

…iton backend

With --enable-unified-memory, a model whose layers attend through windows of
different lengths was refused at startup ("multiple target windows require
named model declarations"): the plan held one target window, and every
attention backend carried the write ids of one window sub-pool
(`KVWriteLoc.swa_loc`). The storage already keeps a sub-pool, a prefix-tree
slot, a host pool and a PD state entry per window; this adds the attention
side.

- Declarations and plan: `kv_layer_decls(windows=[WindowLayers(...)])`
  declares one layer group per window length, MHA or MLA, and the refusal of
  several target windows is removed. Routes and the router's sub-pools carry
  each window sub-pool's attention window.
- Write ids: `KVWriteLoc.window_locs` carries every window sub-pool's write
  ids by sub-pool name and replaces `swa_loc`. Backends key the ids resolved
  for their window storage with `window_write_locs`; draft writers that
  address a layer directly (DFlash, DSpark) use `layer_window_locs`. Every
  backend, the CP strategies and the DFlash worker follow.
- Runner ids: `RunnerIds.window_sub_pools` lists the window sub-pools a
  runner's layers read and `RunnerIds.window_storage` names the one a
  single-rail backend reads. Binding a backend whose `max_window_sub_pools`
  is smaller than the runner's window count raises, naming the backend; a
  multi-step draft wrapper answers through its step backends.
- Triton: one `WindowRail` per window sub-pool, built alike in extend,
  decode, target verify, draft extend, multi-step draft and every CUDA graph
  mode; each window layer reads and writes through its own rail.
  The rails, and the decode scratch (one `attn_logits` buffer per value
  width), come from the runner's own window sub-pools and their geometry; a
  runner that owns none builds no rail over another runner's window band. The
  model's sliding window applies only to pools without window sub-pools.
- Fused drafts: the placement context carries every target window, and a
  draft window rides in the target window sub-pool of its length. The
  draft's window is stated from its config by its architecture's rule
  (`sliding_layer_window_size` on DFlash, Dots3, Gemma 4 MTP, Inkling MTP and
  Llama Eagle3); a draft whose rule needs the built model is refused. A draft
  pool routes its window layers by their host sub-pool's name.

Other backends keep serving one window per runner.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

blackwell SM100/SM120 bypass-fail-fast CI: a failing job no longer aborts its siblings (lint still gates) hicache Hierarchical Caching for SGLang max-concurrency CI: a suite fans out to every shard; apply before triggering the run memory-pool mooncake mthreads npu parallel-stages CI: stages dispatch together instead of waiting on each other run-ci-nv-extra CI: NVIDIA/shared Extra suite (requires run-ci) unified-radix-cache

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant