Skip to content

[PD Disaggregation] Prefill and decode support cache storage - #6768

Merged
Jiang-Jia-Jun merged 5 commits into
PaddlePaddle:developfrom
juncaipeng:pd_cache_storage
Mar 16, 2026
Merged

Jiang-Jia-Jun merged 5 commits into
PaddlePaddle:developfrom
juncaipeng:pd_cache_storage

Conversation

@juncaipeng

@juncaipeng juncaipeng commented Mar 10, 2026 •

Copy link
Copy Markdown
Collaborator

Motivation

Prefill and decode support cache storage

💡 If this PR is a Cherry Pick, the PR title needs to follow the format by adding the [Cherry-Pick] label at the very beginning and appending the original PR ID at the end. For example, [Cherry-Pick][CI] Add check trigger and logic(#5191)

💡 如若此PR是Cherry Pick,PR标题需遵循格式,在最开始加上[Cherry-Pick]标签,以及最后面加上原PR ID,例如[Cherry-Pick][CI] Add check trigger and logic(#5191)

Modifications

在 PrefixCacheManager 新增 Decode 场景的简化写回方法 write_cache_to_storage_decode(),通过 token_ids 直接计算链式 hash keys 并写入 storage。
在 ResourceManagerV1 中按 splitwise role 增加保护条件,避免 Decode 实例执行依赖 Radix Tree 的 prefix/output cache 更新与释放逻辑,并在请求结束时调用 decode 写回方法。
移除 decode 角色下强制关闭 enable_prefix_caching 的参数后处理逻辑,并新增 PD + storage 的示例脚本。
新增相关文档。

Usage or Command

Refer to examples/cache_storage/run_03b_pd.sh

Accuracy Tests

None

Checklist

  • Add at least a tag in the PR title.
    • Tag list: [[FDConfig],[APIServer],[Engine], [Scheduler], [PD Disaggregation], [Executor], [Graph Optimization], [Speculative Decoding], [RL], [Models], [Quantization], [Loader], [OP], [KVCache], [DataProcessor], [BugFix], [Docs], [CI], [Optimization], [Feature], [Benchmark], [Others], [XPU], [HPU], [GCU], [DCU], [Iluvatar], [Metax]]
    • You can add new tags based on the PR content, but the semantics must be clear.
  • Format your code, run pre-commit before commit.
  • Add unit tests. Please write the reason in this PR if no unit tests.
  • Provide accuracy results.
  • If the current PR is submitting to the release branch, make sure the PR has been submitted to the develop branch, then cherry-pick it to the release branch with the [Cherry-Pick] PR tag.

@paddle-bot

paddle-bot Bot commented Mar 10, 2026

Copy link
Copy Markdown

Thanks for your contribution!

@codecov-commenter

codecov-commenter commented Mar 10, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 9.75610% with 37 lines in your changes missing coverage. Please review.
⚠️ Please upload report for BASE (develop@250ce40). Learn more about missing BASE report.

Files with missing lines Patch % Lines
fastdeploy/cache_manager/prefix_cache_manager.py 3.44% 28 Missing ⚠️
fastdeploy/engine/sched/resource_manager_v1.py 30.00% 0 Missing and 7 partials ⚠️
.../transfer_factory/mooncake_store/mooncake_store.py 0.00% 2 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             develop    #6768   +/-   ##
==========================================
  Coverage           ?   72.01%           
==========================================
  Files              ?      395           
  Lines              ?    54999           
  Branches           ?     8660           
==========================================
  Hits               ?    39609           
  Misses             ?    12578           
  Partials           ?     2812           
Flag Coverage Δ
GPU 72.01% <9.75%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

该 PR 旨在在 PD Disaggregation(Prefill/Decode 分离部署) 场景下补齐 KV cache 写回到外部存储(storage backend) 的能力,尤其是让 Decode 实例在不依赖 Radix Tree 的情况下也能完成 cache 落盘,从而支持跨实例/跨轮次复用缓存。

Changes:

  • 在 PrefixCacheManager 新增 Decode 场景的简化写回方法 write_cache_to_storage_decode(),通过 token_ids 直接计算链式 hash keys 并写入 storage。
  • 在 ResourceManagerV1 中按 splitwise role 增加保护条件,避免 Decode 实例执行依赖 Radix Tree 的 prefix/output cache 更新与释放逻辑,并在请求结束时调用 decode 写回方法。
  • 移除 decode 角色下强制关闭 enable_prefix_caching 的参数后处理逻辑,并新增 PD + storage 的示例脚本。

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 7 comments.

File Description
fastdeploy/engine/sched/resource_manager_v1.py 为 splitwise decode 增加 cache 相关逻辑的 role 保护,并在 finish 时区分 P/D 写回路径
fastdeploy/engine/args_utils.py 移除 decode 角色强制关闭 enable_prefix_caching 的逻辑,以允许 decode 侧启用 storage backend
fastdeploy/cache_manager/prefix_cache_manager.py 新增 write_cache_to_storage_decode(),让 decode 侧不依赖 Radix Tree 也能生成 keys 并写回 storage
examples/cache_storage/run_03b_pd.sh 增加 PD 分离 + Mooncake storage 的端到端示例脚本

Comment thread examples/cache_storage/run_03b_pd.sh Resolved
Comment thread examples/cache_storage/run_03b_pd.sh Resolved
Comment thread examples/cache_storage/run_03b_pd.sh Resolved
break # Do not cache incomplete block

# Calculate hash key for current block
key = get_hash_str(block_token_ids, prefix_block_key)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

这里比较好的做法是不是传P节点最后一个block的hash值,把这个hash + block_token_ids 一起计算新hash比较好?不然P节点的一些多摸信息会漏掉

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

en,这样成本最小,下个pr可以支持

@Jiang-Jia-Jun
Jiang-Jia-Jun merged commit 04fde3b into PaddlePaddle:develop Mar 16, 2026
58 of 70 checks passed
mattheliu pushed a commit to mattheliu/FastDeploy that referenced this pull request Apr 1, 2026
…addle#6768)

* Prefill and decode support cache storage

* up

* up

* update docs and refine mooncake store

* up
xiaoguoguo626807 pushed a commit to xiaoguoguo626807/FastDeploy that referenced this pull request May 7, 2026
…addle#6768)

* Prefill and decode support cache storage

* up

* up

* update docs and refine mooncake store

* up

This branch had an error being deployed

1 failed deployment
Metax_ci — 2f9a92f7 Deployed Mar 12, 2026 by juncaipeng via Trigger Jenkins for PR #5197
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants