Skip to content

feat(aginfer): reclaim Dead KV on session end - #3

Merged
rucnyz merged 6 commits into
aginfer-syncedfrom
dead-kv-session-end
Aug 24, 2026
Merged

rucnyz merged 6 commits into
aginfer-syncedfrom
dead-kv-session-end

Conversation

@rucnyz

@rucnyz rucnyz commented Aug 24, 2026 •

Copy link
Copy Markdown
Owner

Summary

  • add synchronous and asynchronous end_program APIs plus POST /aginfer/session_end
  • propagate a terminal lifecycle event across DP and synchronize the reclaim decision across TP/CP ranks
  • detach the ended program from radix-tree holders and drop only exclusive HBM/DRAM KV, leaf to root
  • defer cleanup while requests, locks, or cache transfers are in flight, and return success only after all ranks complete
  • preserve shared prefixes, make repeated END calls idempotent, and fix program ownership across radix splits
  • add a reproducible default-LRU vs. LRU+SESSION_END benchmark, result summary, and direct E2E verifier

Validation

  • clean-checkout d5 direct SGLang TP4 E2E: passed
  • clean-checkout d5 paired A/B smoke: passed; baseline retained 28.22 MiB Dead KV and SESSION_END retained 0
  • focused Dead-KV and communicator tests: 24 passed
  • standalone communicator tests: 7 passed
  • radix split regression: passed
  • unified radix suite: 1063 passed, 626 skipped; two pre-existing SWA JIT/environment failures remain
  • benchmark and E2E mock tests: passed
  • new-file pre-commit hooks, py_compile, secret scan, and git diff --check: passed

Measured result

On d5 (4x GB300, TP4, Qwen3-0.6B), five paired ABBA trials under a controlled cache-pressure workload observed:

  • terminal Dead KV: 479.72 MiB to 0 in the reported TP-rank state view
  • peak HBM occupancy: -22.58%
  • live-prefix cache hit: 0/768 to 704/768 tokens
  • live-probe TTFT: -88.06%
  • inference throughput: +9.88%
  • lifecycle throughput including END RPCs: +6.44%

The benchmark is a lifecycle-signal ablation on the same patched binary, not a comparison against a separate upstream binary. Full methodology and limitations are in dev/aginfer/wherewewin/s3-drop-on-death/RESULTS.md.

Scope

Validated for a single-node TP4 deployment with UnifiedRadixCache and HBM+DRAM HiCache. PP greater than one and decode-side asynchronous KV offload fail closed. SSD/NIXL per-key deletion, context compaction, multi-node disaggregation, and production DeepSeek-V4 traces are not included.


CI States

Latest PR Test (Base): 🚫 Run #32684582438
Latest PR Test (Extra): ❌ Run #32684582258

rucnyz added 5 commits August 23, 2026 19:33
Signed-off-by: Yuzhou Nie <rucnyz@gmail.com>
Signed-off-by: Yuzhou Nie <rucnyz@gmail.com>
Signed-off-by: Yuzhou Nie <rucnyz@gmail.com>
Signed-off-by: Yuzhou Nie <rucnyz@gmail.com>
Signed-off-by: Yuzhou Nie <rucnyz@gmail.com>
@github-actions github-actions Bot added the documentation Improvements or additions to documentation label Aug 24, 2026
@rucnyz rucnyz added the run-ci label Aug 24, 2026
Signed-off-by: Yuzhou Nie <rucnyz@gmail.com>
@rucnyz
rucnyz merged commit 96813d7 into aginfer-synced Aug 24, 2026
51 of 74 checks passed
@rucnyz
rucnyz deleted the dead-kv-session-end branch August 24, 2026 02:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation run-ci

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant