Repository navigation
[Feat] attention: Serve several window sub-pools per runner in the Triton backend - #48014
Merged
Merged
Conversation
ch-wan
force-pushed
the
cheng/refactor/unified-kv-pd-window-entries
branch
from
November 4, 2026 06:18
f82cc57 to
ceff909
Compare
ch-wan
force-pushed
the
cheng/refactor/unified-kv-host-window-pools
branch
from
November 4, 2026 06:18
6021a63 to
3f73c10
Compare
ch-wan
force-pushed
the
cheng/refactor/unified-kv-pd-window-entries
branch
from
November 4, 2026 07:42
ceff909 to
a91493b
Compare
ch-wan
force-pushed
the
cheng/refactor/unified-kv-host-window-pools
branch
from
November 4, 2026 07:42
3f73c10 to
e3f8fb6
Compare
ch-wan
force-pushed
the
cheng/refactor/unified-kv-pd-window-entries
branch
from
November 4, 2026 08:02
a91493b to
f0221ef
Compare
ch-wan
force-pushed
the
cheng/refactor/unified-kv-host-window-pools
branch
from
November 4, 2026 08:02
e3f8fb6 to
a6a6227
Compare
ch-wan
force-pushed
the
cheng/refactor/unified-kv-pd-window-entries
branch
from
November 4, 2026 09:14
f0221ef to
18c97dd
Compare
ch-wan
force-pushed
the
cheng/refactor/unified-kv-host-window-pools
branch
from
November 4, 2026 09:14
a6a6227 to
3ca176e
Compare
ch-wan
force-pushed
the
cheng/refactor/unified-kv-pd-window-entries
branch
from
November 4, 2026 09:31
18c97dd to
ceb2bef
Compare
ch-wan
force-pushed
the
cheng/refactor/unified-kv-host-window-pools
branch
from
November 4, 2026 09:31
3ca176e to
158b380
Compare
This was referenced Nov 4, 2026
Merged
Merged
Merged
Merged
ch-wan
requested review from
ClawSeven,
iforgetmyname,
rwang5203,
whybeyoung and
yeahdongcn
as code owners
November 5, 2026 10:16
ch-wan
force-pushed
the
cheng/refactor/unified-kv-pd-window-entries
branch
from
November 5, 2026 11:25
4591cce to
13d9991
Compare
ch-wan
requested review from
Duyi-Wang,
Jialin,
Lzy17,
ShangmingCai,
fzyzcjy,
kkHuang-amd,
rainj-me,
stmatengss,
sufeng-buaa and
yuan-luo
as code owners
November 5, 2026 11:25
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
4 of 5 tasks
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
…iton backend
With --enable-unified-memory, a model whose layers attend through windows of
different lengths was refused at startup ("multiple target windows require
named model declarations"): the plan held one target window, and every
attention backend carried the write ids of one window sub-pool
(`KVWriteLoc.swa_loc`). The storage already keeps a sub-pool, a prefix-tree
slot, a host pool and a PD state entry per window; this adds the attention
side.
- Declarations and plan: `kv_layer_decls(windows=[WindowLayers(...)])`
declares one layer group per window length, MHA or MLA, and the refusal of
several target windows is removed. Routes and the router's sub-pools carry
each window sub-pool's attention window.
- Write ids: `KVWriteLoc.window_locs` carries every window sub-pool's write
ids by sub-pool name and replaces `swa_loc`. Backends key the ids resolved
for their window storage with `window_write_locs`; draft writers that
address a layer directly (DFlash, DSpark) use `layer_window_locs`. Every
backend, the CP strategies and the DFlash worker follow.
- Runner ids: `RunnerIds.window_sub_pools` lists the window sub-pools a
runner's layers read and `RunnerIds.window_storage` names the one a
single-rail backend reads. Binding a backend whose `max_window_sub_pools`
is smaller than the runner's window count raises, naming the backend; a
multi-step draft wrapper answers through its step backends.
- Triton: one `WindowRail` per window sub-pool, built alike in extend,
decode, target verify, draft extend, multi-step draft and every CUDA graph
mode; each window layer reads and writes through its own rail.
The rails, and the decode scratch (one `attn_logits` buffer per value
width), come from the runner's own window sub-pools and their geometry; a
runner that owns none builds no rail over another runner's window band. The
model's sliding window applies only to pools without window sub-pools.
- Fused drafts: the placement context carries every target window, and a
draft window rides in the target window sub-pool of its length. The
draft's window is stated from its config by its architecture's rule
(`sliding_layer_window_size` on DFlash, Dots3, Gemma 4 MTP, Inkling MTP and
Llama Eagle3); a draft whose rule needs the built model is refused. A draft
pool routes its window layers by their host sub-pool's name.
Other backends keep serving one window per runner.
This was referenced Nov 9, 2026
Draft
5 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR is part of a stack (oldest at bottom):
Motivation
With
--enable-unified-memory, a model whose layers attend through windows of different lengths is refused at startup ("multiple target windows require named model declarations"): the plan holds one target window, and every attention backend carries the write ids of one window sub-pool (KVWriteLoc.swa_loc). The storage below already keeps a sub-pool, a prefix-tree slot, a host pool and a PD state entry per window, and a fused draft's window of another length already lives in a sub-pool of its own. What is missing is the attention side: a runner whose layers read several window sub-pools, and a backend that builds window metadata for each of them.This PR lets a plan declare several target windows and serves them in the Triton backend. Other backends keep serving one window per runner, and a runner whose layers need more is refused when its backend is bound, naming the backend.
Modifications
mem_cache/kv/decl.py,kv/plan.py,kv/build.py):kv_layer_decls(windows=[WindowLayers(...)])declares one layer group per window length, MHA or MLA; the refusal of several target windows is removed. Routes and the router's sub-pools carry each window sub-pool's attention window (PartRoute.window,PublicSubPool.window).mem_cache/memory_pool.py,kv/router.py,kv/routed_cache.py,swa_memory_pool.py):KVWriteLoc.window_locscarries every window sub-pool's write ids by sub-pool name and replacesswa_loc; the fields afterlocare keyword-only, and each window's ids must be contiguous, aslocandfull_locmust. Backends key the ids they resolved for their window storage withwindow_write_locs; draft writers that address a layer directly (DFlash, DSpark) uselayer_window_locs. FlashAttention, FlashInfer, Triton, TRT-LLM MHA, torch native, aiter, XPU, Intel AMX, MUSA and Ascend backends, the CP strategies and the DFlash worker follow.kv/alloc/runner_ids.py,layers/attention/base_attn_backend.py):RunnerIds.window_sub_poolslists the window sub-pools a runner's layers read;RunnerIds.window_storagenames the one a single-rail backend reads (it replaceswindow_kv_kind). Binding a backend whosemax_window_sub_poolsis smaller than the runner's window count raisesNotImplementedErrornaming the leaf and the wrappers in front of it (check_window_capacity); wrappers answer through the backends they run (AttentionBackend.window_capacity_children:HybridAttnBackend's prefill and decode backends, two-batch overlap's primary and children,HybridLinearAttnBackend's full-attention backend; and a multi-step draft's step backends), and the smallest capacity wins (TestWindowCapacityThroughWrappers). Triton takes each window's length from the router's declared windows, the scalar sliding window only filling an unstated one, and refuses a window neither states; a fused draft's pre-load window rule reads the text config (gemma4_mtp.py).layers/attention/triton_backend.py,dllm/attention.py,speculative/spec_utils.py): oneWindowRailper window sub-pool; each window layer reads and writes through its own rail. A target runner builds several rails for ordinary MHA execution, in eager extend and decode and in decode CUDA graphs, when every window's length is stated (_builds_rails_per_window); draft runners, speculative decoding, decode context parallelism, deterministic inference, dLLM, MLA, skip-prefill backends and an externally suppliedkv_indptrbuffer are excluded; elsewhere it builds one rail, and a runner over several window sub-pools is refused at binding. A unified runner takes its rails from the window sub-pools its own layers read (RunnerIds.window_sub_pools), each with its own window, and refuses a window sub-pool that states none; it builds no rail over another runner's band (a fused draft's), and one over the full sub-pool only when the model states a scalar window. Decode scratch holds one logits buffer per distinct value width, taken from the sub-pools' geometry, and the ROCm Lean partial-output buffer is sized to the largest width rounded up to a power of two, as its kernel addresses it (ForwardMetadata.extra_attn_logits).layout/fused_draft.py,kv_cache_configurator.py,model_runner_components/load_model_utils.py, draft model classes,unified_draft_pool.py): the placement context carries every window the target declares (PlacementContext.target_windows), and a draft window rides in the target window sub-pool of its length, whichever it is. The draft's window is the one its own runner will attend over, stated from its config (resolve_sliding_layer_window_size,sliding_layer_window_sizeon DFlash, Dots3, Gemma 4 MTP, Inkling MTP and Llama Eagle3); a draft class whose rule needs the built model is refused with a message. A draft pool routes its window layers by their host sub-pool's name.layers/attention/test_triton_several_windows.py(each window layer reads and writes its own rail) and, nightly,test_triton_several_windows_nightly.py(decode CUDA graph replay and value widths);layers/attention/test_window_id_space_resolution.py(a draft window band resolves to its own storage) andmodels/test_draft_sliding_window_rules.py;test_unified_window_sub_pools.py,disaggregation/test_unified_window_lengths.py,disaggregation/test_unified_window_wire.py,test_unified_chain_realize.py(three window floats) andmanagers/test_prefill_demand_agreement.py, which below this PR build the target's window and a fused draft's window instead;Behaviour changes
--enable-unified-memorywith the Triton backend in the modes above; elsewhere, and with another backend, it is refused when the backend is bound, naming the backend and its capacity. The serving configurator still declares one target window: several reach a plan through a model declaration.--enable-unified-memory.Accuracy and Speed Tests
All results below are at the top of this stack (this PR plus a test-only commit,
78123128d7) onmaina4b8801197, except where a row names another revision. The serving rows are at earlier tops onmain5e1774d137: theopenai/gpt-oss-120bthroughput runs atc03f2e2cf2, and the full GSM8K, PD and MTP rows at5e3702a3e5. Since those runs the stack changed allocator host bookkeeping and the capacity rules, and was rebased.Unit tests (B300, torch 2.14.1+cu130, flashinfer 0.7.2, sglang-kernel 0.4.10), each file in its own process, at the default invariant level and with
SGLANG_INVARIANT_CHECK=2(strict): 222 files (every unit test file this stack adds or changes, the Triton-backend tests, and relatedmaintests). Failures, all also onmainor from the environment:test_srt_endpoint.py,test_disaggregation_optimistic_prefill.pyandtest_input_embeds_chunked.py(gated models unavailable),test_minimax_verify_topk_idx.py(fp8e4b8 unsupported on B300),test_eagle_binding.py,test_unified_hicache_transfer_lifecycle.py(a torch stream type error with a GPU visible),test_unified_scheduler_hooks.py(a scheduler stub withoutPauseState, failing onmaintoo), andtest_prefill_send_protection.pyat strict.test_inkling_checkpoint_indices.pypasses 8 of 8 runs at both levels. These files ran one revision before the current stack. The current revision changes continuation chunk fitting, the window branch tail of the request demand, the cross-stage state-entry check, the window metric label and the host-receive startup check; the affected unit test files (11 files, 187 tests, B200, torch 2.14.1+cu130, flashinfer 0.7.2) pass at the default and strict invariant levels at this PR. The wholetest/registered/unittree at the default level (each subdirectory in its own process, about 15,500 passing tests) fails the same tests at this PR as onmaina4b8801197, all from the environment (gated models, kernels and toolchains unavailable on the machine). The stack's comments and docstrings are condensed and its unit tests trimmed (duplicate and mirror cases removed, a few files merged); production code is unchanged by that, as a comparison of every changed file's syntax tree without docstrings shows, and every changed test file passes at both invariant levels, apart fromtest_unified_hicache_transfer_lifecycle.pywith a GPU visible (as above) and the weeklytest_unified_page_safety_sweep_cuda.py, which passes on its own but not after the other files in one process.Page-safety stress harness (
test/manual/mem_cache/unified_page_safety_stress.py, CPU),--seeds 0:64 --ops 300at the default and strict invariant levels: 0 failures each. The CPU suite and the longer sweeps (--seeds 0:512 --ops 400,--seeds 0:32 --ops 3000, 0 failures) are at an earlier top onmain6d1ce5b894.Servers start and answer a request (B300):
Qwen/Qwen3-0.6Bwith the default backend,flashinfer, and Triton prefill with FlashInfer decode;openai/gpt-oss-20bwith--enable-unified-memoryon the default and Triton backends.Rust (
rust/):cargo fmt --checkandcargo clippy --workspace -D warningspass.Serving (H200, driver 595.71.05, torch 2.14.1+cu130, flashinfer 0.7.2, sglang-kernel 0.4.10, Mooncake transfer engine 0.3.13.post1), this stack's top against
main5e1774d137,--enable-unified-memoryon every server. Each case ran one round, exceptopenai/gpt-oss-120bwith two (both shown, top / main):mainopenai/gpt-oss-120bTP1, overlap scheduling,--max-total-tokens 120000 --attention-backend fa3:bench_servingrandom 4096 in / 512 out, 200 prompts, concurrency 64, output throughputopenai/gpt-oss-120b,--max-total-tokens 30000(full pool, with retractions): GSM8K, all 1319 questionsQwen/Qwen3.8-27BPD 1P1D on Mooncake, unified memory on both sides: GSM8K, 200 questionsQwen/Qwen3.8-27BMTP (EAGLE, 3 steps, top-k 1, 4 draft tokens), TP2, overlap scheduling: GSM8K, 200 questionsOn
openai/gpt-oss-120bthe top matchesmainin throughput and TPOT. OnQwen/Qwen3.8-27Bthe PD random workload is about 3% slower and the shared-prefix one about 1% faster, and MTP is about 2.5% slower; those cases ran one round each. GSM8K differences are within what one run moves at these sizes.These models declare one target window; a model with several target windows was not served. Not run: PD with HiCache, staging or HiSparse; other models and hardware.
Checklist
pre-commiton the changed files.CI States
Latest PR Test (Base): 🚫 Run #42609003129
Latest PR Test (Extra): 🚫 Run #42609003251
Latest PR Test (AMD ROCm 10): ❌ Run #42609003050
Tip
Base CI only covers the core cases, and its workload may not exercise your change. Use
/rerun-testto run the tests your change touches, or copy this prompt for an agent: