Repository navigation
Use a shared byte budget for unified hybrid-SWA memory - #36729
Conversation
ch-wan
left a comment
There was a problem hiding this comment.
Summary
This PR switches unified hybrid-SWA from static FULL/SWA token partitions to a shared byte arena, then admits and reclaims against a joint can_reserve predicate and reports each side as available + allocated. The core accounting is coherent: empty-pool slack still matches the historical one-token admission bound, leak checks still reduce to “allocated tokens are owned,” and the eviction search (min FULL reclaim with all SWA evictable, then min SWA) is ordered so can_reserve stays monotonic. Residual risk is less “this cannot work” than “the new math is barely tested” plus a couple of sharp edges in the SWA estimator API and newly hard-failed scheduler modes.
Issue counts by severity
- bugs: 0
- suggestions: 3
- nits: 0
f722068 to
1599357
Compare
ch-wan
left a comment
There was a problem hiding this comment.
Summary
Prior issue 1 is fixed: estimate_swa_kv_tokens now takes include_allocated_tail, and PrefillAdder's reservation path passes False instead of overloading allocation_limit=0. Prior issue 3 is fixed: the PP assert is scoped to PD in handle_unified_memory_pool, and mixed-chunk is no longer blanket-rejected; unified admission folds num_mixed_decode_tokens into rem_swa_token_offset. Prior issue 2 is still open in substance: the author kept the patch test-neutral, and the new joint-capacity / can_reserve math still has no derived-property guard. Eviction-policy churn was restored (net unified_radix_cache.py is just _finish_eviction); remaining risk is leftover dead SWA-recovery API and untested byte-budget predicates, not a restored eviction algorithm.
Issue counts by severity
- bugs: 0
- suggestions: 2
- nits: 0
|
/tag-and-rerun-ci |
|
/tag-and-rerun-ci |
ch-wan
left a comment
There was a problem hiding this comment.
Summary
Reviewed 70312b502dff against its base. The shared-byte admission and reclaim ordering are consistent in the CPU probes, but enqueue clipping still rounds FULL demand differently from PrefillAdder and can reduce a feasible generation to zero tokens. Add permanent coverage for reclaim planning with live pages and blocked compaction.
Validation: executed the actual scheduler and reservation methods in isolation; compared reclaim_plan with exhaustive ordered search in 1,080 live/fragmented states. No GPU serving run.
Issue counts by severity
- bugs: 1
- suggestions: 1
- nits: 0
merrymercy
left a comment
There was a problem hiding this comment.
do we have a e2e unified hybrid swa memory pool test with an engine launch?
There was a problem hiding this comment.
generated by agent:
Focused follow-up on the original allocator if/else feedback at ab2ea29ea348.
The main duplication is now addressed: PrefillAdder delegates admission to allocator-created budgets, eviction goes through evict_to_free_tokens, paired capacity reporting goes through swa_capacity_and_available, and the two-pool / tri-pool allocators are sibling policies. The old chunk-helper parameter is also gone. I found no further substantial unified/static dispatch duplication to request here. The remaining shared-byte, fixed-partition, ring-slot, and tri-pool policy differences should remain explicit; reversing guard clauses would not simplify them.
This is a focused style/refactoring conclusion, not a correctness approval. No GPU or end-to-end tests were rerun. The upper PRs still inherit the earlier capacity history ending at 089673a0e6, so their findings are reviewed against their own heads.
Cherry-picked from a6eb82d.
|
/rerun-test test_gemma4_unified_swa_virtual_ids.py |
|
Results for 🚀 |
|
/rerun-test test_pool_configurator.py test_unified_byte_budget_sizing.py test_prefill_memory_budget.py test_prefill_adder.py test_scheduler_init_req_max_new_tokens.py test_multi_ended_allocator.py test_unified_tri_pool.py test_kv_index_translator.py test_unified_radix_allocation_eviction.py test_unified_free_no_host_sync.py test_hisparse_max_token_pool_size.py test_unified_memory_move_gate.py test_unified_radix_cache_unittest.py test_page_major_gpt_oss.py test_inkling_unified.py test_kimi_linear_models.py::TestKimiLinearUnifiedMemory test_page_major_qwen_hybrid.py test_disaggregation_unified_memory_swa.py |
|
Results for 🚀 🚀 🚀 🚀 |
|
/rerun-test test_pool_configurator.py test_unified_byte_budget_sizing.py test_prefill_memory_budget.py test_prefill_adder.py test_scheduler_init_req_max_new_tokens.py test_multi_ended_allocator.py test_unified_tri_pool.py test_kv_index_translator.py test_unified_radix_allocation_eviction.py test_unified_free_no_host_sync.py test_hisparse_max_token_pool_size.py test_unified_memory_move_gate.py test_unified_radix_cache_unittest.py test_page_major_gpt_oss.py test_inkling_unified.py test_kimi_linear_models.py::TestKimiLinearUnifiedMemory test_page_major_qwen_hybrid.py test_disaggregation_unified_memory_swa.py |
|
Results for 🚀 🚀 🚀 🚀 |
Use the shared-capacity implementation merged in #36729 as the base, retaining only the PD page-envelope delta. Update the paged-SWA resume fixture for refreshed FULL budgets.
Motivation
Unified full-attention and sliding-window pools share one byte arena, but static per-pool capacity gates can reject allocations that fit by borrowing unused bytes from the other side. Capacity reporting must follow the same dynamic allocation state.
Builds on #36403, which is now merged. This is the first change in a follow-up stack.
Modifications
Accuracy Tests
Not applicable; this changes cache capacity accounting rather than model math.
Speed Tests and Profiling
Not run; this does not change kernels.
Test Plan
f604efb38f. Merged OSS maina63efd9056into the bottom PR and propagated normal merges; no rebase or force-push. Final merge previews also pass against maindc5f59c3a2.withsyntax after the first CPU CI exposedunittest.enterContextbeing unavailable on Python 3.10. Its 8 tests pass locally. No new tests were added for this conflict refresh.run-ci-extrais absent). The initial separate MLX run failed in the upstream, unchangedtest_scheduler_mixin.pymock-ingestion contract.Local validation commands
Python commands below were executed through an existing
uv runenvironment (Python 3.12, Torch 2.11); environment-specific paths are omitted.Original commits
b3a8bc7130bd5990adc8e0b70ef0fcf2c68f3208b0ce2cc0debbe57ab07b4f43e453bbba03046b8476bdccc03fae9ffbeb5f780706077e31d952e42c8f948e4b3f0c69a4afb05ef75e0c3093e4371399b9d422f867ec869c0af055b3158b41554c2507ed80ce849eb45825b2c2d2087245622c2199f7d2df958587ef0718aba81113d6a3260fc48b863b1c47c06e90922bc7b6115d7416c5c2050a749c008f7bChecklist
CI States
Latest PR Test (Base): ❌ Run #35011826222
Latest PR Test (Extra): ❌ Run #35011825963
Latest PR Test (AMD ROCm 10): ❌ Run #35011826207