Repository navigation
[Fix] swa: Start the SWA KV pool at its first layer under pipeline parallelism - #46200
Merged
Merged
Conversation
A later pipeline stage's SWAKVPool holds global layer ids but reported start_layer 0, so TritonAttnBackend's get_value_buffer(start_layer) raised KeyError at startup for hybrid-SWA models. Report the smallest layer id the pool holds, as the full-attention pool reports its stage's first layer.
This was referenced Oct 24, 2026
Test the fix against main at 1ced9a9, after the KV pool router and routed-cache refactors. The merge has no conflicts.
Read the value buffer at start_layer first, as TritonAttnBackend does, so the case fails on main with the KeyError the fix removes.
Jiminator
marked this pull request as ready for review
November 3, 2026 00:30
There was a problem hiding this comment.
Claude Code Review
This repository is configured for manual code reviews. Comment @claude review for a one-time review, or @claude review always to subscribe this PR to a review on every future push.
Tip: disable this comment in your organization's Code Review settings.
Jiminator
requested review from
Ying1123,
alphabetc1,
hanming-lu,
hnyls2002,
huangtingwei9988,
hzh0425,
ispobock,
merrymercy,
xiezhq-hermann and
yizhang2077
as code owners
November 3, 2026 00:30
ispobock
approved these changes
Nov 3, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Under pipeline parallelism, a later stage's
SWAKVPoolholds global layer ids, but it reportsstart_layer = 0. Code that reads the pool's first layer throughstart_layerthen asks for a layer the stage doesn't have.For example,
TritonAttnBackend.__init__readsget_value_buffer(token_to_kv_pool.start_layer)when the model isn't mambaish and the SWA and full value head dims match. On a hybrid-SWA model with--attention-backend tritonand--pp-size 2, the second stage fails at startup (main at1ced9a9eb0):Split out of #45256 at review; found while adding Beam (#44747).
Modifications
SWAKVPool.start_layeris now the smallest layer id the pool holds, asrouted_draft_poolalready does withmin(layer_ids). On the tested pipeline split this is the stage's first global layer, the value plain pools get from the configurator. Pools that hold layer 0 keep the same start, inner-pool indexing is unchanged, and a pool with no layers keeps 0._swa_layer_mappingsand the load waits (KVLayerRouter._wait) subtract the samestart_layer, so they stay consistent. HiCache already worked with the global ids, because the layer counter is sized to the largest id plus one and earlier stages' ids are skipped.k. It now reportsk, with or without pipeline parallelism, whichlayer_info.start_layer(0 there) wouldn't. A CPU check of window-only and full-only pools at depth 3 shows that on mainget_value_buffer(start_layer)raisesKeyError: 0, and that herestart_layeris 3 and reads the same storage as the router, withget_v_head_dimunchanged. From reading the code, draft pools get no load counter. I didn't run an Inkling model.trtllm_mha's native FP4 buffers,aiter,get_v_head_dim, the HiCache host pools) use that sub-pool's ownstart_layer, which is still 0.TestSWAPoolVHeadDimmoves toCustomTestCase, which the contribution guide requires for registered tests.Not fixed by this PR:
PD with prefill
--pp-size 2and decode--pp-size 1on a non-MLA SWA model._transfer_start_layerreturnsstart_layer, but the peer's KV list holds only full-attention layers. On the model below the second stage needs offset 6, the full-attention layers before it. Main gives 0 and this PR gives 12. A CPU probe of the real pool,_transfer_start_layerand Mooncake's transfer builder, with a recording transport, shows:IndexErrorbefore any block is issued.No PD server pair was run, and NIXL and Mori weren't exercised. From reading the code, that
IndexErrorwould stop Mooncake's transfer thread, and peers at the same--pp-sizewith the same KV entries skip the offset because their lists match. The fix needs layer-aware pairing for the full-attention and SWA-state entries, asHybridLinearKVPoolhas, so I left it to a follow-up.openai/gpt-oss-20b(MXFP4) under--pp-size 2. It fails earlier on main, in_load_mxfp4_experts_weights, withAttributeError: PPMissingLayer has no attribute mlp. That's a separate loader bug, and it's why the runs below use the bf16 conversion.Accuracy Tests
Unit and neighbouring tests
test_swa_pool_v_head_dim.pyadds a case for a second-stage pool (layers 2 and 3). On main it fails withKeyError: 0atswa_memory_pool.py:262, the same line as the startup crash. It passes here.test_kv_cache_configurator_draft_gate.py.--tp-size 2with one GPU exposed, and its weights weren't cached. It doesn't use PP. The skips and that error give no coverage.Server runs: NVIDIA B300, two GPUs for PP2 and one for PP1;
lmsys/gpt-oss-20b-bf16@8de8d3dfa4The model has 24 layers alternating sliding (window 128) and full. Requests are greedy
/generatecalls with frozeninput_idsof 24, 320, 1200 and 3400 tokens,temperature=0,max_new_tokens=64,ignore_eos.--attention-backend triton --pp-size 2KeyError: 0--pp-size 1--attention-backend triton,--pp-size 1--pp-size 2, default backend (trtllm_mha),--enable-hierarchical-cache --hicache-size 4 --max-total-tokens 8192 --mem-fraction-static 0.6: serve, evict with four filler prompts, serve againtransfer_layer_id_maxlogged by that run, first / second stageLimits on these runs:
--pp-size 1, this PR at--pp-size 1and2), so that prompt says little about PP. Their 64 tokens match. Thetrtllm_mharuns first differ at the third token, but they also differ in memory flags and page size (64 vs 1), so the cause isn't isolated. The other three prompts, including the 320- and 1200-token ones that exceed the window, give the same tokens on both backends.--pp-size 2.Speed Tests and Profiling
Not applicable.
Checklist
CI States
Latest PR Test (Base): ✅ Run #42141919227
Latest PR Test (Extra): ✅ Run #42141919450
Latest PR Test (AMD ROCm 10): ✅ Run #42141919375
Tip
Base CI only covers the core cases, and its workload may not exercise your change. Use
/rerun-testto run the tests your change touches, or copy this prompt for an agent: