[Feature] Mooncake CPP (Chunked Pipeline Parallelism)

[kingzevin]

Checklist

• 1. If the issue you raised is not a feature but a question, please raise a discussion at https://github.com/sgl-project/sglang/discussions/new/choose Otherwise, it will be closed.
• 2. Please use English, otherwise it will be closed.

Motivation

Hi team,

I found that the sequence parallelism (Meta CP based on https://www.arxiv.org/pdf/2411.01783) is in the roadmap (#4042). For long-context inference, I'm wondering do you have plans to support chunked pipeline parallelism from Mooncake or sequence pipeline parallelism from Mnemosyne?
If there are no plans, could you please leave some comments about your thoughts on the technology to combine chunked prefill + pp for long-context prefilling, compared with Context Parallelism? Really appreciate your attention~

Related resources

No response

[yinfan98]

really need this

[yinfan98]

Need pp firstly @zhyncs

[Ying1123]

@yinfan98 Supported in

• [PP] Add pipeline parallelism #5724

[kingzevin]

This is amazing — thank you so much for implementing this feature so quickly!

@yinfan98 Supported in

• [PP] Add pipeline parallelism #5724

[jifa513]

How effective is CPP?

[jifa513]

Need pp firstly @zhyncs

How effective is CPP?

[Zhangmj0621]

@kingzevin @Ying1123 Why close this feature? It seems that vllm only support pipeline parallelism not chunked pp.

[kingzevin]

@kingzevin @Ying1123 Why close this feature? It seems that vllm only support pipeline parallelism not chunked pp.

Already supported in SGLang.

[Zhangmj0621]

@kingzevin @Ying1123 Why close this feature? It seems that vllm only support pipeline parallelism not chunked pp.

Already supported in SGLang.

Sorry I mistake vllm and sglang. But I wonder In the pr above, it seems that only pipeline parallelism is supported in sglang, not chunked pipeline parallelism of pd disaggregation. Could you show me some pr related with chunked pipeline parallelism? I would highly appreciate any feedbacks from you.

[kingzevin]

CPP is not supported in PD disaggregation but only in PD colocation mode. To enable it, you can just look at the arguments about chunked-prefill-size and pp.

[Zhangmj0621]

CPP is not supported in PD disaggregation but only in PD colocation mode. To enable it, you can just look at the arguments about chunked-prefill-size and pp.

In fact, chunked prefill is mainly used for PD colocation mode. However, CPP is presented for PD disaggregation in mooncake to address long context. Sglang now support pp(used for request-level parallelism). I wonder if the parallelsim of long contexts(divided to several chunks) hasn't been implemented by sglang.

[kingzevin]

I agree with you that CPP is important for PD disaggregation.
If I understand it correctly, the current chunked-prefill + PP can already do the task of 'parallelism + divided to several chunks' you've mentioned. I think that IS CPP.
What's the difference between CPP in PD colocation vs in disaggregation?