Repository navigation
[PD Disaggregation] Prefill and decode support cache storage - #6768
Conversation
|
Thanks for your contribution! |
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## develop #6768 +/- ##
==========================================
Coverage ? 72.01%
==========================================
Files ? 395
Lines ? 54999
Branches ? 8660
==========================================
Hits ? 39609
Misses ? 12578
Partials ? 2812
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
There was a problem hiding this comment.
Pull request overview
该 PR 旨在在 PD Disaggregation(Prefill/Decode 分离部署) 场景下补齐 KV cache 写回到外部存储(storage backend) 的能力,尤其是让 Decode 实例在不依赖 Radix Tree 的情况下也能完成 cache 落盘,从而支持跨实例/跨轮次复用缓存。
Changes:
- 在
PrefixCacheManager新增 Decode 场景的简化写回方法write_cache_to_storage_decode(),通过 token_ids 直接计算链式 hash keys 并写入 storage。 - 在
ResourceManagerV1中按 splitwise role 增加保护条件,避免 Decode 实例执行依赖 Radix Tree 的 prefix/output cache 更新与释放逻辑,并在请求结束时调用 decode 写回方法。 - 移除 decode 角色下强制关闭
enable_prefix_caching的参数后处理逻辑,并新增 PD + storage 的示例脚本。
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 7 comments.
| File | Description |
|---|---|
| fastdeploy/engine/sched/resource_manager_v1.py | 为 splitwise decode 增加 cache 相关逻辑的 role 保护,并在 finish 时区分 P/D 写回路径 |
| fastdeploy/engine/args_utils.py | 移除 decode 角色强制关闭 enable_prefix_caching 的逻辑,以允许 decode 侧启用 storage backend |
| fastdeploy/cache_manager/prefix_cache_manager.py | 新增 write_cache_to_storage_decode(),让 decode 侧不依赖 Radix Tree 也能生成 keys 并写回 storage |
| examples/cache_storage/run_03b_pd.sh | 增加 PD 分离 + Mooncake storage 的端到端示例脚本 |
eb9d579 to
571d172
Compare
| break # Do not cache incomplete block | ||
|
|
||
| # Calculate hash key for current block | ||
| key = get_hash_str(block_token_ids, prefix_block_key) |
There was a problem hiding this comment.
这里比较好的做法是不是传P节点最后一个block的hash值,把这个hash + block_token_ids 一起计算新hash比较好?不然P节点的一些多摸信息会漏掉
There was a problem hiding this comment.
en,这样成本最小,下个pr可以支持
…addle#6768) * Prefill and decode support cache storage * up * up * update docs and refine mooncake store * up
…addle#6768) * Prefill and decode support cache storage * up * up * update docs and refine mooncake store * up
Motivation
Prefill and decode support cache storage
Modifications
在 PrefixCacheManager 新增 Decode 场景的简化写回方法 write_cache_to_storage_decode(),通过 token_ids 直接计算链式 hash keys 并写入 storage。
在 ResourceManagerV1 中按 splitwise role 增加保护条件,避免 Decode 实例执行依赖 Radix Tree 的 prefix/output cache 更新与释放逻辑,并在请求结束时调用 decode 写回方法。
移除 decode 角色下强制关闭 enable_prefix_caching 的参数后处理逻辑,并新增 PD + storage 的示例脚本。
新增相关文档。
Usage or Command
Refer to examples/cache_storage/run_03b_pd.sh
Accuracy Tests
None
Checklist
[FDConfig],[APIServer],[Engine],[Scheduler],[PD Disaggregation],[Executor],[Graph Optimization],[Speculative Decoding],[RL],[Models],[Quantization],[Loader],[OP],[KVCache],[DataProcessor],[BugFix],[Docs],[CI],[Optimization],[Feature],[Benchmark],[Others],[XPU],[HPU],[GCU],[DCU],[Iluvatar],[Metax]]pre-commitbefore commit.releasebranch, make sure the PR has been submitted to thedevelopbranch, then cherry-pick it to thereleasebranch with the[Cherry-Pick]PR tag.