Skip to content

[Fix] server_args: Refuse a sharded-padded CP KV layout with unified memory - #48002

Merged
ch-wan merged 1 commit into
mainfrom
cheng/hot-fix/unified-kv-cp-sharded-padded-refusal
Nov 4, 2026
Merged

ch-wan merged 1 commit into
mainfrom
cheng/hot-fix/unified-kv-cp-sharded-padded-refusal

Conversation

@ch-wan

@ch-wan ch-wan commented Nov 3, 2026 •

Copy link
Copy Markdown
Collaborator

This PR is part of a stack (oldest at bottom):

Motivation

A context-parallel KV layout that keeps each rank's KV sharded and padded (--cp-kv-layout with a strategy that sets cp_kv_sharded) sizes and indexes KV per rank. The unified memory pool's placement and PD transfers do not model that layout, so the combination could only fail later, at allocation or transfer time.

Modifications

  • arg_groups/parallel_hook.py::handle_context_parallelism: right after the CP strategy is bound, a sharded-padded strategy together with --enable-unified-memory is refused with a clear error. Static pools keep the sharded-padded accounting.
  • test/registered/unit/server_args/test_server_args.py: with a sharded-padded strategy bound, unified memory is refused and the static pool is accepted.

Behavior change: --enable-unified-memory with a sharded-padded --cp-kv-layout now fails at startup instead of later.

Accuracy Tests

Startup-only check; no model output changes. Covered by the unit test above. Stack-level serving validation is described in the top PR of the stack.

Speed Tests and Profiling

Not applicable (startup argument validation).

Checklist

Review and Merge Process

  1. Ping Merge Oncalls to start the process. See the PR Merge Process.
  2. Get approvals from CODEOWNERS and other reviewers.
  3. Trigger CI tests with comments or contact authorized users to do so.
    • Common commands include /tag-and-rerun-ci, /tag-run-ci-label, /rerun-failed-ci
  4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR.

CI States

Latest PR Test (Base): 🚫 Run #41907045240
Latest PR Test (Extra): 🚫 Run #41907045365
Latest PR Test (AMD ROCm 10): ❌ Run #41907045496

Tip

Base CI only covers the core cases, and its workload may not exercise your change. Use /rerun-test to run the tests your change touches, or copy this prompt for an agent:

Find the registered tests this PR affects and run them with /rerun-test, following https://github.com/sgl-project/sglang/blob/main/.agents/skills/rerun-test/SKILL.md

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This repository is configured for manual code reviews. Comment @claude review for a one-time review, or @claude review always to subscribe this PR to a review on every future push.

Tip: disable this comment in your organization's Code Review settings.

This was referenced Nov 4, 2026
@ch-wan
ch-wan force-pushed the cheng/hot-fix/unified-kv-cp-sharded-padded-refusal branch from 43fc645 to 5458116 Compare November 4, 2026 11:33
@ch-wan
ch-wan force-pushed the cheng/hot-fix/unified-kv-routed-draft-context-writes branch from 2716b7e to 4778fdd Compare November 4, 2026 20:27
Base automatically changed from cheng/hot-fix/unified-kv-routed-draft-context-writes to main November 4, 2026 20:28
…ded-padded CP KV layout with unified memory

A context-parallel KV layout that keeps each rank's KV sharded and padded
(`--cp-kv-layout`, a strategy with `cp_kv_sharded`) sizes and indexes KV per
rank, which the unified memory pool's placement and transfers do not
model. The combination was never run, so it is refused at startup, right
after the CP strategy is bound, instead of failing later. Static pools keep
the sharded-padded accounting.

Test: with a sharded-padded strategy bound, `handle_context_parallelism`
refuses `--enable-unified-memory` and accepts the static pool.
@ch-wan
ch-wan force-pushed the cheng/hot-fix/unified-kv-cp-sharded-padded-refusal branch from 5458116 to 0c76b4a Compare November 4, 2026 20:29
@ch-wan
ch-wan merged commit 1666971 into main Nov 4, 2026
24 of 30 checks passed
@ch-wan
ch-wan deleted the cheng/hot-fix/unified-kv-cp-sharded-padded-refusal branch November 4, 2026 20:29
Yuxingwang-intel pushed a commit to Yuxingwang-intel/sglang that referenced this pull request Nov 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant