Skip to content

[Feat] KVShard: Support kv cache sharding for MTP - #40929

Merged
hzh0425 merged 14 commits into
sgl-project:mainfrom
Shunkangz:mtp_kvshard
Oct 23, 2026
Merged

hzh0425 merged 14 commits into
sgl-project:mainfrom
Shunkangz:mtp_kvshard

Conversation

@Shunkangz

@Shunkangz Shunkangz commented Sep 23, 2026 •

Copy link
Copy Markdown
Contributor

Motivation

This PR adds pool-level MTP KV-cache sharding support for dense MLA and MHA models. The draft reuses the target’s allocator and shard layout while maintaining separate KV storage and scratch buffers. MTP-specific compatibility checks currently run during pool construction; we will move them into argument validation when those checks are added.

Modifications

Accuracy Tests

Speed Tests and Profiling

Checklist

Review and Merge Process

  1. Ping Merge Oncalls to start the process. See the PR Merge Process.
  2. Get approvals from CODEOWNERS and other reviewers.
  3. Trigger CI tests with comments or contact authorized users to do so.
    • Common commands include /tag-and-rerun-ci, /tag-run-ci-label, /rerun-failed-ci
  4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR.

CI States

Latest PR Test (Base): ❌ Run #39891565092
Latest PR Test (Extra): 🚫 Run #39891565015
Latest PR Test (AMD ROCm 10): ⏳ Run #39891565052

@Shunkangz

Copy link
Copy Markdown
Contributor Author

/tag-and-rerun-ci

@github-actions github-actions Bot added the run-ci CI: run the baseline test suite on this PR label Sep 28, 2026
@huangtingwei9988 huangtingwei9988 self-assigned this Sep 28, 2026
@Shunkangz
Shunkangz force-pushed the mtp_kvshard branch 2 times, most recently from a259236 to 156b0db Compare September 29, 2026 06:55
@Shunkangz

Copy link
Copy Markdown
Contributor Author

/rerun-failed-ci

1 similar comment
@Shunkangz

Copy link
Copy Markdown
Contributor Author

/rerun-failed-ci

Shunkangz and others added 2 commits October 8, 2026 11:19
The main merge brought in sgl-project#35594, which threads the resolved MLA pool
class into _build_mla_kv_pool. Expect that kwarg in the fallback-path test.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Shunkangz

Copy link
Copy Markdown
Contributor Author

/rerun-failed-ci

@Shunkangz

Copy link
Copy Markdown
Contributor Author

/rerun-failed-ci

Main sgl-project#43503 dropped the always-skipped 2-GPU section from the CPU-registered
test_page_interleave_shard.py. Move it, including the MTP gather tests, into
test_page_interleave_gather_multi_gpu.py registered on 2-gpu-large so it runs
in CI. kv_cache_configurator.py: union of both sides' page_interleave imports.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Shunkangz

Copy link
Copy Markdown
Contributor Author

/rerun-failed-ci

Preserve MTP shared-pool imports alongside upstream DSA indexer sharding. Keep CPU, two-GPU MTP, and four-GPU DSA tests in their registered suites.
@Shunkangz

Copy link
Copy Markdown
Contributor Author

/rerun-failed-ci

@github-actions

This comment has been minimized.

@Shunkangz Shunkangz changed the title Support kv cache sharding for MTP [Feat] KVShard: Support kv cache sharding for MTP Oct 19, 2026
@Shunkangz

Copy link
Copy Markdown
Contributor Author

/rerun-failed-ci

@Shunkangz

Copy link
Copy Markdown
Contributor Author

/rerun-failed-ci

Preserve upstream shard footprint and P/D send-filter tests while keeping MTP coverage and the split multi-GPU suites.

@hzh0425 hzh0425 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Others looks good

Keep upstream page-interleave test pruning while preserving the MTP scratch-budget regression test.
@hzh0425 hzh0425 added the run-ci-extra CI: also run the extra suite (requires run-ci) label Oct 22, 2026
Preserve the dedicated MLA/MHA/MTP gather suite while keeping upstream's expanded DSA tests. Remove the superseded indexer test copy and update the CPU shard suite's coverage references.
@hzh0425

hzh0425 commented Oct 23, 2026

Copy link
Copy Markdown
Collaborator
image https://github.com/sgl-project/sglang/actions/runs/39714079531 Most of tests have passed in previous run, just fix a single conflict test

@hzh0425

hzh0425 commented Oct 23, 2026

Copy link
Copy Markdown
Collaborator

/rerun-test test/registered/unit/mem_cache/test_page_interleave_gather_multi_gpu.py test/registered/unit/mem_cache/test_page_interleave_shard.py test/registered/unit/managers/test_prefill_adder.py

@github-actions

github-actions Bot commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

🚀 2-gpu-h100 (1 test): ✅ View workflow run

cd test/ && python3 registered/unit/mem_cache/test_page_interleave_gather_multi_gpu.py

🚀 ubuntu-latest (2 tests): ✅ View workflow run

cd test/ && python3 registered/unit/mem_cache/test_page_interleave_shard.py
cd test/ && python3 registered/unit/managers/test_prefill_adder.py

@hzh0425
hzh0425 merged commit 79ab8f6 into sgl-project:main Oct 23, 2026
174 of 235 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

deepseek run-ci CI: run the baseline test suite on this PR run-ci-extra CI: also run the extra suite (requires run-ci)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants