Repository navigation
[Fix] server_args: Refuse a sharded-padded CP KV layout with unified memory - #48002
Merged
Merged
Conversation
ch-wan
force-pushed
the
cheng/hot-fix/unified-kv-cp-sharded-padded-refusal
branch
from
November 4, 2026 07:42
8d8acbf to
43fc645
Compare
ch-wan
force-pushed
the
cheng/hot-fix/unified-kv-routed-draft-context-writes
branch
from
November 4, 2026 07:42
230206e to
2716b7e
Compare
This was referenced Nov 4, 2026
Merged
Merged
ch-wan
marked this pull request as ready for review
November 4, 2026 09:34
There was a problem hiding this comment.
Claude Code Review
This repository is configured for manual code reviews. Comment @claude review for a one-time review, or @claude review always to subscribe this PR to a review on every future push.
Tip: disable this comment in your organization's Code Review settings.
This was referenced Nov 4, 2026
Merged
Merged
Closed
Closed
[Refactor] mem_cache: Choose reclaim in one place, keep one static storage, joint LRU reclaim
#48025
Closed
ch-wan
force-pushed
the
cheng/hot-fix/unified-kv-cp-sharded-padded-refusal
branch
from
November 4, 2026 11:33
43fc645 to
5458116
Compare
ch-wan
force-pushed
the
cheng/hot-fix/unified-kv-routed-draft-context-writes
branch
from
November 4, 2026 20:27
2716b7e to
4778fdd
Compare
ch-wan
requested review from
Qiaolin-Yu,
Ying1123,
alphabetc1,
hanming-lu,
hnyls2002,
huangtingwei9988,
hzh0425,
ispobock,
merrymercy,
xiezhq-hermann and
yizhang2077
as code owners
November 4, 2026 20:27
Base automatically changed from
cheng/hot-fix/unified-kv-routed-draft-context-writes
to
main
November 4, 2026 20:28
…ded-padded CP KV layout with unified memory A context-parallel KV layout that keeps each rank's KV sharded and padded (`--cp-kv-layout`, a strategy with `cp_kv_sharded`) sizes and indexes KV per rank, which the unified memory pool's placement and transfers do not model. The combination was never run, so it is refused at startup, right after the CP strategy is bound, instead of failing later. Static pools keep the sharded-padded accounting. Test: with a sharded-padded strategy bound, `handle_context_parallelism` refuses `--enable-unified-memory` and accepts the static pool.
ch-wan
force-pushed
the
cheng/hot-fix/unified-kv-cp-sharded-padded-refusal
branch
from
November 4, 2026 20:29
5458116 to
0c76b4a
Compare
4 of 5 tasks
Yuxingwang-intel
pushed a commit
to Yuxingwang-intel/sglang
that referenced
this pull request
Nov 6, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR is part of a stack (oldest at bottom):
Motivation
A context-parallel KV layout that keeps each rank's KV sharded and padded (
--cp-kv-layoutwith a strategy that setscp_kv_sharded) sizes and indexes KV per rank. The unified memory pool's placement and PD transfers do not model that layout, so the combination could only fail later, at allocation or transfer time.Modifications
arg_groups/parallel_hook.py::handle_context_parallelism: right after the CP strategy is bound, a sharded-padded strategy together with--enable-unified-memoryis refused with a clear error. Static pools keep the sharded-padded accounting.test/registered/unit/server_args/test_server_args.py: with a sharded-padded strategy bound, unified memory is refused and the static pool is accepted.Behavior change:
--enable-unified-memorywith a sharded-padded--cp-kv-layoutnow fails at startup instead of later.Accuracy Tests
Startup-only check; no model output changes. Covered by the unit test above. Stack-level serving validation is described in the top PR of the stack.
Speed Tests and Profiling
Not applicable (startup argument validation).
Checklist
Review and Merge Process
/tag-and-rerun-ci,/tag-run-ci-label,/rerun-failed-ciCI States
Latest PR Test (Base): 🚫 Run #41907045240
Latest PR Test (Extra): 🚫 Run #41907045365
Latest PR Test (AMD ROCm 10): ❌ Run #41907045496
Tip
Base CI only covers the core cases, and its workload may not exercise your change. Use
/rerun-testto run the tests your change touches, or copy this prompt for an agent: