Skip to content

[Fix] swa: Start the SWA KV pool at its first layer under pipeline parallelism - #46200

Merged
Jiminator merged 7 commits into
mainfrom
swa-pool-pp-start-layer
Nov 6, 2026
Merged

Jiminator merged 7 commits into
mainfrom
swa-pool-pp-start-layer

Conversation

@Jiminator

@Jiminator Jiminator commented Oct 24, 2026 •

Copy link
Copy Markdown
Collaborator

Motivation

Under pipeline parallelism, a later stage's SWAKVPool holds global layer ids, but it reports start_layer = 0. Code that reads the pool's first layer through start_layer then asks for a layer the stage doesn't have.

For example, TritonAttnBackend.__init__ reads get_value_buffer(token_to_kv_pool.start_layer) when the model isn't mambaish and the SWA and full value head dims match. On a hybrid-SWA model with --attention-backend triton and --pp-size 2, the second stage fails at startup (main at 1ced9a9eb0):

File ".../layers/attention/triton_backend.py", line 311, in __init__
    self.v_head_dim = model_runner.token_to_kv_pool.get_value_buffer(
File ".../mem_cache/swa_memory_pool.py", line 262, in get_value_buffer
    layer_id_pool, is_swa_layer = self.layers_mapping[layer_id]
KeyError: 0

Split out of #45256 at review; found while adding Beam (#44747).

Modifications

  • SWAKVPool.start_layer is now the smallest layer id the pool holds, as routed_draft_pool already does with min(layer_ids). On the tested pipeline split this is the stage's first global layer, the value plain pools get from the configurator. Pools that hold layer 0 keep the same start, inner-pool indexing is unchanged, and a pool with no layers keeps 0.
  • HiCache on later stages becomes stage-local. _swa_layer_mappings and the load waits (KVLayerRouter._wait) subtract the same start_layer, so they stay consistent. HiCache already worked with the global ids, because the layer counter is sized to the largest id plus one and earlier stages' ids are skipped.
  • A draft pool for an Inkling MTP depth holds only layer k. It now reports k, with or without pipeline parallelism, which layer_info.start_layer (0 there) wouldn't. A CPU check of window-only and full-only pools at depth 3 shows that on main get_value_buffer(start_layer) raises KeyError: 0, and that here start_layer is 3 and reads the same storage as the router, with get_v_head_dim unchanged. From reading the code, draft pools get no load counter. I didn't run an Inkling model.
  • Readers that go through a sub-pool (trtllm_mha's native FP4 buffers, aiter, get_v_head_dim, the HiCache host pools) use that sub-pool's own start_layer, which is still 0.
  • TestSWAPoolVHeadDim moves to CustomTestCase, which the contribution guide requires for registered tests.

Not fixed by this PR:

  • PD with prefill --pp-size 2 and decode --pp-size 1 on a non-MLA SWA model. _transfer_start_layer returns start_layer, but the peer's KV list holds only full-attention layers. On the model below the second stage needs offset 6, the full-attention layers before it. Main gives 0 and this PR gives 12. A CPU probe of the real pool, _transfer_start_layer and Mooncake's transfer builder, with a recording transport, shows:

    • Main: the second stage pairs with the first stage's entries, with no error (rc 0).
    • This PR: IndexError before any block is issued.
    • Offset 6 pairs correctly on both.

    No PD server pair was run, and NIXL and Mori weren't exercised. From reading the code, that IndexError would stop Mooncake's transfer thread, and peers at the same --pp-size with the same KV entries skip the offset because their lists match. The fix needs layer-aware pairing for the full-attention and SWA-state entries, as HybridLinearKVPool has, so I left it to a follow-up.

  • openai/gpt-oss-20b (MXFP4) under --pp-size 2. It fails earlier on main, in _load_mxfp4_experts_weights, with AttributeError: PPMissingLayer has no attribute mlp. That's a separate loader bug, and it's why the runs below use the bf16 conversion.

Accuracy Tests

Unit and neighbouring tests

  • test_swa_pool_v_head_dim.py adds a case for a second-stage pool (layers 2 and 3). On main it fails with KeyError: 0 at swa_memory_pool.py:262, the same line as the startup crash. It passes here.
  • Neighbouring tests give the same results on main and here:
    • All 17 related CPU-registered test files pass, including test_kv_cache_configurator_draft_gate.py.
    • 9 GPU-registered files, on one GPU, give 339 passed, 57 skipped, 1 error on both. The error is a Gemma 4 server test that couldn't start on either tree: it asks for --tp-size 2 with one GPU exposed, and its weights weren't cached. It doesn't use PP. The skips and that error give no coverage.

Server runs: NVIDIA B300, two GPUs for PP2 and one for PP1; lmsys/gpt-oss-20b-bf16 @ 8de8d3dfa4

The model has 24 layers alternating sliding (window 128) and full. Requests are greedy /generate calls with frozen input_ids of 24, 320, 1200 and 3400 tokens, temperature=0, max_new_tokens=64, ignore_eos.

Run main This PR
--attention-backend triton --pp-size 2 startup fails with KeyError: 0 starts; 4/4 prompts give the same tokens as this PR at --pp-size 1
--attention-backend triton, --pp-size 1 4/4 prompts give the same tokens as this PR 4/4 prompts give the same tokens as main
--pp-size 2, default backend (trtllm_mha), --enable-hierarchical-cache --hicache-size 4 --max-total-tokens 8192 --mem-fraction-static 0.6: serve, evict with four filler prompts, serve again second pass host hits 0/256/1152/3392, device 0; 4/4 prompts on both passes give the same tokens as without HiCache same host hits; 4/4 prompts on both passes give the same tokens as without HiCache, and the second pass matches main
transfer_layer_id_max logged by that run, first / second stage 12 / 24 12 / 12

Limits on these runs:

  • The 3400-token prompt's output has long runs of token 1131 in all three Triton runs (main at --pp-size 1, this PR at --pp-size 1 and 2), so that prompt says little about PP. Their 64 tokens match. The trtllm_mha runs first differ at the third token, but they also differ in memory flags and page size (64 vs 1), so the cause isn't isolated. The other three prompts, including the 320- and 1200-token ones that exceed the window, give the same tokens on both backends.
  • The 24-token prompt is shorter than the 64-token page, so it gets no host hit. The second-pass check rests on the other three prompts.
  • These are four-prompt parity checks; I ran no accuracy benchmark.
  • I didn't run Triton with HiCache under --pp-size 2.

Speed Tests and Profiling

Not applicable.

Checklist


CI States

Latest PR Test (Base): ✅ Run #42141919227
Latest PR Test (Extra): ✅ Run #42141919450
Latest PR Test (AMD ROCm 10): ✅ Run #42141919375

Tip

Base CI only covers the core cases, and its workload may not exercise your change. Use /rerun-test to run the tests your change touches, or copy this prompt for an agent:

Find the registered tests this PR affects and run them with /rerun-test, following https://github.com/sgl-project/sglang/blob/main/.agents/skills/rerun-test/SKILL.md

A later pipeline stage's SWAKVPool holds global layer ids but reported start_layer 0, so TritonAttnBackend's get_value_buffer(start_layer) raised KeyError at startup for hybrid-SWA models. Report the smallest layer id the pool holds, as the full-attention pool reports its stage's first layer.
Test the fix against main at 1ced9a9, after the KV pool router and
routed-cache refactors. The merge has no conflicts.
Read the value buffer at start_layer first, as TritonAttnBackend does,
so the case fails on main with the KeyError the fix removes.
@Jiminator
Jiminator marked this pull request as ready for review November 3, 2026 00:30

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This repository is configured for manual code reviews. Comment @claude review for a one-time review, or @claude review always to subscribe this PR to a review on every future push.

Tip: disable this comment in your organization's Code Review settings.

@ispobock ispobock added run-ci CI: run the baseline test suite on this PR run-ci-nv-extra CI: NVIDIA/shared Extra suite (requires run-ci) bypass-fail-fast CI: a failing job no longer aborts its siblings (lint still gates) labels Nov 3, 2026
@Jiminator
Jiminator merged commit c2580a7 into main Nov 6, 2026
233 of 273 checks passed
@Jiminator
Jiminator deleted the swa-pool-pp-start-layer branch November 6, 2026 22:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bypass-fail-fast CI: a failing job no longer aborts its siblings (lint still gates) memory-pool run-ci CI: run the baseline test suite on this PR run-ci-nv-extra CI: NVIDIA/shared Extra suite (requires run-ci)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants