Skip to content

[docker] Distributed Serving with k8s Statefulset ( good example for DeepSeek-R1) - #3631

Merged
zhyncs merged 1 commit into
sgl-project:mainfrom
panpan0000:sts-yaml
Mar 9, 2025
Merged

zhyncs merged 1 commit into
sgl-project:mainfrom
panpan0000:sts-yaml

Conversation

@panpan0000

@panpan0000 panpan0000 commented Feb 17, 2025 •

Copy link
Copy Markdown
Contributor

DeepSeek-R1 is good candidate

Motivation

There're already a single node k8s serving(docker/k8s-sglang-service.yaml).
But for deepseek-r1, distributed serving is required.

So here we provide an vanilla k8s deployment, without dependency (most suitable for small scale and easy for users who are not experts of k8s),
although there was a PR for LWS solution #3540, but it requires LWS installed as prerequisites.

Modifications

  • We leverage k8s native statefulset
  • Use k8s native label apps.kubernetes.io/pod-index to automatically fill the --node-rank for each node
  • Show expert-parallel arguments example for those who will try DeepSeek-R1
  • Two nodes example mostly as the minimal scale for DS-R1
  • Consider using the RDMA/IB network to speed up TP/EP
  • KINDLY NOTE: --enable-ep-moe is still not working for DeepSeek till 0.4.3-post2 , due to [Bug] DeepSeek-V3 load weights failed with --enable-ep-moe #3075

Checklist

@panpan0000 panpan0000 changed the title [docker] Distributed Serving with k8s statefulset [docker] Distributed Serving with k8s Statefulset ( good example for DeepSeek-R1) Feb 17, 2025
Comment thread docker/k8s-sglang-distributed-sts.yaml Resolved
DeepSeek-R1 is good candidate

Co-authored-by: Peter Pan <Peter.Pan@daocloud.io>
Co-authored-by: Kebe <kebe.liu@daocloud.io>
Signed-off-by: Peter Pan <Peter.Pan@daocloud.io>
@panpan0000

Copy link
Copy Markdown
Contributor Author

@whybeyoung since your PR has been merged, can we continue with this PR ?

@zhaochenyang20

Copy link
Copy Markdown
Collaborator

Thanks! We will merge this

@whybeyoung whybeyoung left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good for for k8s< 1.26

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants