Skip to content

Use a shared byte budget for unified hybrid-SWA memory - #36729

Merged
ch-wan merged 17 commits into
sgl-project:mainfrom
ZYHowell:yonghao/ump-shared-capacity
Sep 15, 2026
Merged

ch-wan merged 17 commits into
sgl-project:mainfrom
ZYHowell:yonghao/ump-shared-capacity

Conversation

@ZYHowell

@ZYHowell ZYHowell commented Aug 27, 2026 •

Copy link
Copy Markdown
Collaborator

Motivation

Unified full-attention and sliding-window pools share one byte arena, but static per-pool capacity gates can reject allocations that fit by borrowing unused bytes from the other side. Capacity reporting must follow the same dynamic allocation state.

Builds on #36403, which is now merged. This is the first change in a follow-up stack.

Modifications

  • Reconcile the shared-byte admission logic with upstream's split allocators, Mamba donor interface, DCP page semantics, config bags, and SWA request-ring accounting. UMP-off/static-pool behavior stays on the upstream path.
  • Size unified hybrid-SWA pools from an explicit shared byte budget.
  • Use joint full/SWA admission and reclaim checks in scheduling and decode preallocation.
  • Report each side's current realizable capacity for metrics and invariants.
  • Remove the stale split-pool recovery helper and add mutation-sensitive derived-property coverage for unified sizing and packed-byte admission.

Accuracy Tests

Not applicable; this changes cache capacity accounting rather than model math.

Speed Tests and Profiling

Not run; this does not change kernels.

Test Plan

  • Updated head: f604efb38f. Merged OSS main a63efd9056 into the bottom PR and propagated normal merges; no rebase or force-push. Final merge previews also pass against main dc5f59c3a2.
  • Existing focused suite: 331 passed; 722 subtests passed.
  • Adapted the existing HiSparse fixture to config bags and Python 3.10-compatible with syntax after the first CPU CI exposed unittest.enterContext being unavailable on Python 3.10. Its 8 tests pass locally. No new tests were added for this conflict refresh.
  • Full changed-file pre-commit passes, including test registry validation.
  • Exact-head Base CI has started; results are not yet claimed green. Extra CI is not opted in (run-ci-extra is absent). The initial separate MLX run failed in the upstream, unchanged test_scheduler_mixin.py mock-ingestion contract.
  • Full-model serving, multi-node PD, and accuracy/performance benchmarks were not run locally.
Local validation commands

Python commands below were executed through an existing uv run environment (Python 3.12, Torch 2.11); environment-specific paths are omitted.

PYTHONPATH=python python -m pytest -q \
  test/registered/unit/mem_cache/test_unified_byte_budget_sizing.py \
  test/registered/unit/mem_cache/test_unified_byte_accounting.py \
  test/registered/unit/mem_cache/test_unified_capacity_memo.py \
  test/registered/unit/mem_cache/test_unified_swa_shared_virtual_ids.py \
  test/registered/unit/mem_cache/test_unified_tri_pool.py \
  test/registered/unit/mem_cache/test_swa_locked_full_recover_unified.py \
  test/registered/unit/mem_cache/test_unified_free_no_host_sync.py \
  test/registered/unit/mem_cache/test_unified_npool_sweep.py \
  test/registered/unit/mem_cache/test_swa_ring_page_return.py \
  test/registered/unit/managers/test_prefill_adder.py \
  test/registered/unit/managers/test_schedule_policy.py \
  test/registered/unit/managers/test_kv_page_invariants.py \
  test/registered/unit/managers/scheduler_components/test_invariant_checker.py \
  test/registered/unit/mem_cache/test_multi_ended_allocator.py \
  test/registered/unit/model_executor/test_pool_configurator.py \
  test/registered/unit/mem_cache/test_hisparse_max_token_pool_size.py \
  test/registered/unit/managers/test_scheduler_init_req_max_new_tokens.py \
  --disable-warnings --maxfail=4

PYTHONPATH=python python test/registered/unit/mem_cache/test_hisparse_max_token_pool_size.py -q

GITHUB_BASE_REF=a63efd9056b33a3d4a32dfba6262fac1d62b959a \
  python -m pre_commit run \
  --from-ref a63efd9056b33a3d4a32dfba6262fac1d62b959a --to-ref HEAD

Original commits

  • b3a8bc7130bd5990adc8e0b70ef0fcf2c68f3208
  • b0ce2cc0debbe57ab07b4f43e453bbba03046b84
  • 76bdccc03fae9ffbeb5f780706077e31d952e42c
  • 8f948e4b3f0c69a4afb05ef75e0c3093e4371399
  • b9d422f867ec869c0af055b3158b41554c2507ed
  • 80ce849eb45825b2c2d2087245622c2199f7d2df
  • 958587ef0718aba81113d6a3260fc48b863b1c47
  • c06e90922bc7b6115d7416c5c2050a749c008f7b

Checklist


CI States

Latest PR Test (Base): ❌ Run #35011826222
Latest PR Test (Extra): ❌ Run #35011825963
Latest PR Test (AMD ROCm 10): ❌ Run #35011826207

@ch-wan ch-wan left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Summary

This PR switches unified hybrid-SWA from static FULL/SWA token partitions to a shared byte arena, then admits and reclaims against a joint can_reserve predicate and reports each side as available + allocated. The core accounting is coherent: empty-pool slack still matches the historical one-token admission bound, leak checks still reduce to “allocated tokens are owned,” and the eviction search (min FULL reclaim with all SWA evictable, then min SWA) is ordered so can_reserve stays monotonic. Residual risk is less “this cannot work” than “the new math is barely tested” plus a couple of sharp edges in the SWA estimator API and newly hard-failed scheduler modes.

Issue counts by severity

  • bugs: 0
  • suggestions: 3
  • nits: 0

Comment thread python/sglang/srt/server_args.py Resolved
@ZYHowell
ZYHowell force-pushed the yonghao/ump-shared-capacity branch from f722068 to 1599357 Compare September 1, 2026 22:24

@ch-wan ch-wan left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Summary

Prior issue 1 is fixed: estimate_swa_kv_tokens now takes include_allocated_tail, and PrefillAdder's reservation path passes False instead of overloading allocation_limit=0. Prior issue 3 is fixed: the PP assert is scoped to PD in handle_unified_memory_pool, and mixed-chunk is no longer blanket-rejected; unified admission folds num_mixed_decode_tokens into rem_swa_token_offset. Prior issue 2 is still open in substance: the author kept the patch test-neutral, and the new joint-capacity / can_reserve math still has no derived-property guard. Eviction-policy churn was restored (net unified_radix_cache.py is just _finish_eviction); remaining risk is leftover dead SWA-recovery API and untested byte-budget predicates, not a restored eviction algorithm.

Issue counts by severity

  • bugs: 0
  • suggestions: 2
  • nits: 0

@ZYHowell

ZYHowell commented Sep 4, 2026

Copy link
Copy Markdown
Collaborator Author

/tag-and-rerun-ci

@github-actions github-actions Bot added the run-ci CI: run the baseline test suite on this PR label Sep 4, 2026
@ZYHowell ZYHowell mentioned this pull request Sep 4, 2026
5 tasks done
@ZYHowell

Copy link
Copy Markdown
Collaborator Author

/tag-and-rerun-ci

@ch-wan ch-wan left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Summary

Reviewed 70312b502dff against its base. The shared-byte admission and reclaim ordering are consistent in the CPU probes, but enqueue clipping still rounds FULL demand differently from PrefillAdder and can reduce a feasible generation to zero tokens. Add permanent coverage for reclaim planning with live pages and blocked compaction.

Validation: executed the actual scheduler and reservation methods in isolation; compared reclaim_plan with exhaustive ordered search in 1,080 live/fragmented states. No GPU serving run.

Issue counts by severity

  • bugs: 1
  • suggestions: 1
  • nits: 0

Comment thread python/sglang/srt/managers/scheduler.py Resolved

@merrymercy merrymercy left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we have a e2e unified hybrid swa memory pool test with an engine launch?

Comment thread python/sglang/srt/managers/scheduler.py Resolved
Comment thread python/sglang/srt/mem_cache/common.py Resolved
Comment thread python/sglang/srt/mem_cache/common.py Resolved

@ZYHowell ZYHowell left a comment •

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

generated by agent:

Focused follow-up on the original allocator if/else feedback at ab2ea29ea348.

The main duplication is now addressed: PrefillAdder delegates admission to allocator-created budgets, eviction goes through evict_to_free_tokens, paired capacity reporting goes through swa_capacity_and_available, and the two-pool / tri-pool allocators are sibling policies. The old chunk-helper parameter is also gone. I found no further substantial unified/static dispatch duplication to request here. The remaining shared-byte, fixed-partition, ring-slot, and tri-pool policy differences should remain explicit; reversing guard clauses would not simplify them.

This is a focused style/refactoring conclusion, not a correctness approval. No GPU or end-to-end tests were rerun. The upper PRs still inherit the earlier capacity history ending at 089673a0e6, so their findings are reviewed against their own heads.

ZYHowell pushed a commit to ZYHowell/sglang that referenced this pull request Sep 14, 2026
@merrymercy

Copy link
Copy Markdown
Collaborator

/rerun-test test_gemma4_unified_swa_virtual_ids.py

@github-actions

github-actions Bot commented Sep 14, 2026 •

Copy link
Copy Markdown
Contributor

Results for /rerun-test test_gemma4_unified_swa_virtual_ids.py:

🚀 1-gpu-h100 (1 test): ✅ View workflow run

cd test/ && python3 registered/attention/test_gemma4_unified_swa_virtual_ids.py

@ZYHowell

Copy link
Copy Markdown
Collaborator Author

/rerun-test test_pool_configurator.py test_unified_byte_budget_sizing.py test_prefill_memory_budget.py test_prefill_adder.py test_scheduler_init_req_max_new_tokens.py test_multi_ended_allocator.py test_unified_tri_pool.py test_kv_index_translator.py test_unified_radix_allocation_eviction.py test_unified_free_no_host_sync.py test_hisparse_max_token_pool_size.py test_unified_memory_move_gate.py test_unified_radix_cache_unittest.py test_page_major_gpt_oss.py test_inkling_unified.py test_kimi_linear_models.py::TestKimiLinearUnifiedMemory test_page_major_qwen_hybrid.py test_disaggregation_unified_memory_swa.py

@github-actions

github-actions Bot commented Sep 14, 2026 •

Copy link
Copy Markdown
Contributor

Results for /rerun-test test_pool_configurator.py test_unified_byte_budget_sizing.py test_prefill_memory_budget.py test_prefill_adder.py test_scheduler_init_req_max_new_tokens.py test_multi_ended_allocator.py test_unified_tri_pool.py test_kv_index_translator.py test_unified_radix_allocation_eviction.py test_unified_free_no_host_sync.py test_hisparse_max_token_pool_size.py test_unified_memory_move_gate.py test_unified_radix_cache_unittest.py test_page_major_gpt_oss.py test_inkling_unified.py test_kimi_linear_models.py::TestKimiLinearUnifiedMemory test_page_major_qwen_hybrid.py test_disaggregation_unified_memory_swa.py:

🚀 ubuntu-latest (12 tests): ✅ View workflow run

cd test/ && python3 registered/unit/model_executor/test_pool_configurator.py
cd test/ && python3 registered/unit/mem_cache/test_unified_byte_budget_sizing.py
cd test/ && python3 registered/unit/mem_cache/test_prefill_memory_budget.py
cd test/ && python3 registered/unit/managers/test_prefill_adder.py
cd test/ && python3 registered/unit/managers/test_scheduler_init_req_max_new_tokens.py
cd test/ && python3 registered/unit/mem_cache/test_multi_ended_allocator.py
cd test/ && python3 registered/unit/mem_cache/test_unified_tri_pool.py
cd test/ && python3 registered/unit/mem_cache/test_kv_index_translator.py
cd test/ && python3 registered/unit/mem_cache/test_unified_radix_allocation_eviction.py
cd test/ && python3 registered/unit/mem_cache/test_unified_free_no_host_sync.py
cd test/ && python3 registered/unit/mem_cache/test_hisparse_max_token_pool_size.py
cd test/ && python3 registered/unit/disaggregation/test_unified_memory_move_gate.py

🚀 1-gpu-5090 (1 test): ✅ View workflow run

cd test/ && python3 registered/unit/mem_cache/test_unified_radix_cache_unittest.py

🚀 1-gpu-h100 (3 tests): ✅ View workflow run

cd test/ && python3 registered/page_major/test_page_major_gpt_oss.py
cd test/ && python3 registered/e2e/models/test_inkling_unified.py
cd test/ && python3 registered/page_major/test_page_major_qwen_hybrid.py

🚀 2-gpu-h100 (2 tests): ✅ View workflow run

cd test/ && python3 registered/e2e/models/test_kimi_linear_models.py TestKimiLinearUnifiedMemory
cd test/ && python3 registered/e2e/disaggregation/test_disaggregation_unified_memory_swa.py

@ZYHowell

Copy link
Copy Markdown
Collaborator Author

/rerun-test test_pool_configurator.py test_unified_byte_budget_sizing.py test_prefill_memory_budget.py test_prefill_adder.py test_scheduler_init_req_max_new_tokens.py test_multi_ended_allocator.py test_unified_tri_pool.py test_kv_index_translator.py test_unified_radix_allocation_eviction.py test_unified_free_no_host_sync.py test_hisparse_max_token_pool_size.py test_unified_memory_move_gate.py test_unified_radix_cache_unittest.py test_page_major_gpt_oss.py test_inkling_unified.py test_kimi_linear_models.py::TestKimiLinearUnifiedMemory test_page_major_qwen_hybrid.py test_disaggregation_unified_memory_swa.py

@github-actions

github-actions Bot commented Sep 15, 2026 •

Copy link
Copy Markdown
Contributor

Results for /rerun-test test_pool_configurator.py test_unified_byte_budget_sizing.py test_prefill_memory_budget.py test_prefill_adder.py test_scheduler_init_req_max_new_tokens.py test_multi_ended_allocator.py test_unified_tri_pool.py test_kv_index_translator.py test_unified_radix_allocation_eviction.py test_unified_free_no_host_sync.py test_hisparse_max_token_pool_size.py test_unified_memory_move_gate.py test_unified_radix_cache_unittest.py test_page_major_gpt_oss.py test_inkling_unified.py test_kimi_linear_models.py::TestKimiLinearUnifiedMemory test_page_major_qwen_hybrid.py test_disaggregation_unified_memory_swa.py:

🚀 ubuntu-latest (12 tests): ✅ View workflow run

cd test/ && python3 registered/unit/model_executor/test_pool_configurator.py
cd test/ && python3 registered/unit/mem_cache/test_unified_byte_budget_sizing.py
cd test/ && python3 registered/unit/mem_cache/test_prefill_memory_budget.py
cd test/ && python3 registered/unit/managers/test_prefill_adder.py
cd test/ && python3 registered/unit/managers/test_scheduler_init_req_max_new_tokens.py
cd test/ && python3 registered/unit/mem_cache/test_multi_ended_allocator.py
cd test/ && python3 registered/unit/mem_cache/test_unified_tri_pool.py
cd test/ && python3 registered/unit/mem_cache/test_kv_index_translator.py
cd test/ && python3 registered/unit/mem_cache/test_unified_radix_allocation_eviction.py
cd test/ && python3 registered/unit/mem_cache/test_unified_free_no_host_sync.py
cd test/ && python3 registered/unit/mem_cache/test_hisparse_max_token_pool_size.py
cd test/ && python3 registered/unit/disaggregation/test_unified_memory_move_gate.py

🚀 1-gpu-5090 (1 test): ✅ View workflow run

cd test/ && python3 registered/unit/mem_cache/test_unified_radix_cache_unittest.py

🚀 1-gpu-h100 (3 tests): ✅ View workflow run

cd test/ && python3 registered/page_major/test_page_major_gpt_oss.py
cd test/ && python3 registered/e2e/models/test_inkling_unified.py
cd test/ && python3 registered/page_major/test_page_major_qwen_hybrid.py

🚀 2-gpu-h100 (2 tests): ✅ View workflow run

cd test/ && python3 registered/e2e/models/test_kimi_linear_models.py TestKimiLinearUnifiedMemory
cd test/ && python3 registered/e2e/disaggregation/test_disaggregation_unified_memory_swa.py

@ch-wan
ch-wan merged commit 2929a39 into sgl-project:main Sep 15, 2026
152 of 200 checks passed
ZYHowell added a commit that referenced this pull request Sep 15, 2026
Use the shared-capacity implementation merged in #36729 as the base, retaining only the PD page-envelope delta. Update the paged-SWA resume fixture for refreshed FULL budgets.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

memory-pool run-ci CI: run the baseline test suite on this PR unified-radix-cache

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants