Repository navigation
feat(examples/sglang): add Intel XPU deployment configs - #10023
Conversation
|
👋 Hi yao531441! Thank you for contributing to ai-dynamo/dynamo. Just a reminder: The 🚀 |
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@examples/backends/sglang/deploy/xpu/disagg_planner_xpu_dra.yaml`:
- Around line 5-7: Update the header warning comment to match the manifest's
replica config: change the note from "Requires 4+ GPU devices (2 prefill + 2
decode)" to "Requires 3 GPU devices (1 prefill + 2 decode)" and remove or adjust
the subsequent line that suggests a 3-GPU fallback; ensure the top comment
reflects the actual spec of 1 prefill + 2 decode replicas.
- Around line 109-110: The UCX_TLS environment entry in
disagg_planner_xpu_dra.yaml is missing the "ze_ipc" transport; update the
UCX_TLS value string for the existing env var named UCX_TLS so it matches the
other disaggregated XPU templates by changing the value to
"ze_ipc,ze_copy,tcp,cma,sysv,posix,self" (ensure the UCX_TLS env block that
currently has "ze_copy,tcp,cma,sysv,posix,self" is replaced with the new
string).
In `@examples/backends/sglang/deploy/xpu/README.md`:
- Line 24: The secret creation command currently uses the key `token` which
mismatches runtime expectations; update the kubectl command that creates
`hf-token-secret` to use the key `HF_TOKEN` (e.g., create secret generic
hf-token-secret --from-literal=HF_TOKEN=<your-token>) so the runtime
templates/env variables that read `HF_TOKEN` find the token; also scan
references to `hf-token-secret`/`HF_TOKEN` in deployment manifests and env
templates to ensure they consistently use `HF_TOKEN`.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: fdbbdd7e-6225-4dd4-b215-da100d882730
📒 Files selected for processing (6)
examples/backends/sglang/deploy/README.mdexamples/backends/sglang/deploy/xpu/README.mdexamples/backends/sglang/deploy/xpu/agg_xpu_dra.yamlexamples/backends/sglang/deploy/xpu/disagg_planner_xpu_dra.yamlexamples/backends/sglang/deploy/xpu/disagg_xpu.yamlexamples/backends/sglang/deploy/xpu/disagg_xpu_dra.yaml
|
/ok to test 93d6e69 |
788220d to
a8a442b
Compare
fb6d9e5 to
a6dac87
Compare
Add Kubernetes deployment templates for SGLang on Intel XPU GPUs: - agg_xpu_dra.yaml: Aggregated deployment with DRA - disagg_xpu_dra.yaml: Disaggregated deployment with DRA - disagg_planner_xpu_dra.yaml: Disaggregated with Planner - disagg_xpu.yaml: Traditional device plugin (non-DRA) Key configurations for XPU disaggregated KV transfer: - hostIPC: true for ZE_IPC shared memory - UCX_TLS=ze_ipc,ze_copy,... for UCX transport - ResourceClaimTemplate for GPU allocation via DRA Aligns with vLLM XPU configs structure from PR ai-dynamo#9253. Signed-off-by: Yao, Qing <qing.yao@intel.com>
Add agg_router_xpu_dra.yaml for KV-aware aggregated serving on Intel XPU: - 2 workers behind KV router, each on separate GPU via DRA - KV events via ZMQ for routing optimization - XPU-specific flags: --device xpu, --disable-cuda-graph, --grammar-backend none Aligns with existing XPU deployment configs structure. Tested on XPU-817225 kind cluster (8x Intel Arc Pro B60). Signed-off-by: Yao, Qing <qing.yao@intel.com>
Signed-off-by: Yao, Qing <qing.yao@intel.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Yao, Qing <qing.yao@intel.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Yao, Qing <qing.yao@intel.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Yao, Qing <qing.yao@intel.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Yao, Qing <qing.yao@intel.com>
- remove disagg_planner_xpu_dra.yaml: forward pass metrics are off on XPU, so the Planner has no performance model and never scales; re-add once the XPU SGLang pin includes sgl-project/sglang#33739 - set --page-size 64; the intel_xpu attention backend accepts only 64 or 128 AICE-1084 Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: pallavi jaini <pallavi.jaini@intel.com>
|
/ok to test cdb9601 |
dmitry-tokarev-nv
left a comment
There was a problem hiding this comment.
No P0, P1 or P2 found. This review read head cdb960121ca8a9f48f46d30b6e3739d3d97b4a89. It found three P3 items.
- [P3] The test command forwards to a Deployment, which does not exist on a cluster that runs Grove.
examples/backends/sglang/deploy/xpu/README.md:85. See the inline comment, which has a suggestion. - [P3] The table of differences does not name the page size (16 to 64).
examples/backends/sglang/deploy/xpu/README.md:56. See the inline comment, which has a suggestion. - [P3] The PR description is still out of date, as the review on 2026-10-26 said. It now has a Validation section. But it still names
disagg_planner_xpu_dra.yaml, says "All five manifests", and says "The Planner template ships no profile data". This head has four manifests and no Planner template. Please update the description.
The image tag rule of the review on 2026-10-26 is fixed. Its thread has the measurement. The --skip-tokenizer-init question is answered: no worker sets the flag now.
Open items: 14 threads of other reviewers and bots are still unresolved. main requires resolved threads before merge.
What this review measured, and where it stopped.
- Change since
24c592cd38: five content commits and five merges. Each merge equals a clean redo of its parents withgit merge-tree. This head differs frommainebad592 in exactly the 6 files of this pull request (+488/-0). - Shell wrapper: this review rendered each component of the four manifests with the operator code of
main, on the DCD path and on the Grove path. For all six workers, the command is["/bin/bash", "-c"], and the script reaches the pod unchanged. The operator setsDYN_FORWARDPASS_METRIC_PORT=20380on every worker (component_worker.go:114). If that variable is not empty,dynamo.sglangturns on forward pass metrics (components/src/dynamo/sglang/args.py:223). Bash ran each script with a stubpython3: the variable was unset, and each argument list matched the manifest. A copy without theunsetline kept 20380. - For one node, the SGLang backend of the operator adds no flag (
backend_sglang.go:54). It changes nothing for routing or disaggregation. With two nodes, it puts--dist-init-addr,--nnodesand--node-rankright afterpython3 -m dynamo.sglanginside the script. - SGLang flags, read at v0.5.19:
intel_xpuis a valid--attention-backend, and--page-size 64is valid with it for Qwen3. All six workers set the same page size, attention backend, device and wrapper. Prefill and decode differ only in--disaggregation-mode. - CRD and webhook: the four DGD objects pass the v1beta1 schema, the CEL rules and strict decoding at
main. The threeResourceClaimTemplateobjects decode strictly asresource.k8s.io/v1. Six planted errors fail as expected, one each: a misspelledresourceClaims, a stringreplicas, an unknown component type, a misspelleddeviceClassName, a stringhostIPC, and a duplicate component name. - Storage: each worker requests 8Gi.
Qwen/Qwen3-0.6Bis 1,519,209,243 bytes (1.41 GiB) on Hugging Face. - README:
container/render.py --framework=sglang --device=xpu --target=runtimewritescontainer/sglang-runtime-xpu-amd64-rendered.Dockerfile, the name in the README. That image copiesexamplesto/workspace/examplesand puts thesglangenvironment first inPATH. The secret name and key match the sixenvFromreferences. The model name incurlmatches--served-model-name.docs_lint.pyanddetect_broken_links.pyfind 0 errors in the two READMEs, and each one catches a planted broken link. The two new links use github.com, and both return 200. At this head, only the vLLM XPU README namesdisagg_planner_xpu_dra.yaml, and it names its own file. - A merge of this head with
main9f88a47 has no conflict and differs from it only in the 6 files of this pull request. The one new commit onmainchanges only Rust crates. - CI on this head at 17:02Z: Pre Merge passed, including
pre-commitandRecipe Check. It skipped Docs Lint (that job scansdocs/only), the Fern, operator and Rust jobs. Docs link check, Copyright Checks, codeowners and DCO passed. Onpull-request/10023, Fern Docs passed, and PR-XPU passed with no test lane: its only lane, vLLM, was skipped.PRwas still running, with 25 jobs passed, 44 skipped and 2 running. No CI job deploys these manifests, because the deploy tests read onlyexamples/backends/<framework>/deploy/*.yaml. - Not tested: no cluster, no Intel GPU, no image build and no SGLang run. The
torch.cuda.Eventcause in the README note comes from reading SGLang v0.5.19 (python/sglang/srt/utils/device_timer.py:93), not from a run.
Co-authored-by: Dmitry Tokarev <dtokarev@nvidia.com> Signed-off-by: Pallavi Jaini <pallavi.jaini@intel.com>
Co-authored-by: Dmitry Tokarev <dtokarev@nvidia.com> Signed-off-by: Pallavi Jaini <pallavi.jaini@intel.com>
dmitry-tokarev-nv
left a comment
There was a problem hiding this comment.
No P0, P1 or P2 found at head 6622c853ec6de4dc904566a54c44930567db051b. The two new commits apply the two suggestions of my previous review as written: the test command now forwards to svc/sglang-agg-xpu-dra-frontend, and the table of differences names the page size. One P3 item is open: the PR description still names the removed Planner manifest. The image tag rule and the --skip-tokenizer-init question of the earlier reviews are fixed. The operator passes the bash -c wrapper to all six workers unchanged, and SGLang v0.5.19 accepts --page-size 64 with intel_xpu. No CI job deploys these manifests, and no Intel GPU ran them for this review. The unresolved threads of other reviewers must be resolved before merge.
|
/ok to test 6622c85 |
Rebasing onto main brought in upstream's merged diffusion templates (ai-dynamo#14660) and new example sources. Reflect them in the catalog: - Describe the merged image, language-model, and Qwen3-TTS templates as validated on Dynamo 1.5.1, drop the obsolete SGLang shareProcessNamespace note, and explain the vLLM-Omni canary health check workaround. The main Qwen3-TTS template is no longer experimental; the experimental/ manifests keep their pinned builds. - Add the SGLang Intel XPU DGDs (ai-dynamo#10023) to the aggregated, KV-routing, and disaggregated pages, the SGLang LoRA router launcher (ai-dynamo#15746) to LoRA Serving, and the vLLM transcription launcher (ai-dynamo#16208) to Audio and Realtime Serving. - Note that the KVCR memory-service DGD now mounts a content-addressed startup ConfigMap that deploy.sh applies first. Signed-off-by: Jonathan Tong <jt572@cornell.edu>
Rebasing onto main brought in upstream's merged diffusion templates (ai-dynamo#14660) and new example sources. Reflect them in the catalog: - Describe the merged image, language-model, and Qwen3-TTS templates as validated on Dynamo 1.5.1, drop the obsolete SGLang shareProcessNamespace note, and explain the vLLM-Omni canary health check workaround. The main Qwen3-TTS template is no longer experimental; the experimental/ manifests keep their pinned builds. - Add the SGLang Intel XPU DGDs (ai-dynamo#10023) to the aggregated, KV-routing, and disaggregated pages, the SGLang LoRA router launcher (ai-dynamo#15746) to LoRA Serving, and the vLLM transcription launcher (ai-dynamo#16208) to Audio and Realtime Serving. - Note that the KVCR memory-service DGD now mounts a content-addressed startup ConfigMap that deploy.sh applies first. Signed-off-by: Jonathan Tong <jt572@cornell.edu>
Rebasing onto main brought in upstream's merged diffusion templates (ai-dynamo#14660) and new example sources. Reflect them in the catalog: - Describe the merged image, language-model, and Qwen3-TTS templates as validated on Dynamo 1.5.1, drop the obsolete SGLang shareProcessNamespace note, and explain the vLLM-Omni canary health check workaround. The main Qwen3-TTS template is no longer experimental; the experimental/ manifests keep their pinned builds. - Add the SGLang Intel XPU DGDs (ai-dynamo#10023) to the aggregated, KV-routing, and disaggregated pages, the SGLang LoRA router launcher (ai-dynamo#15746) to LoRA Serving, and the vLLM transcription launcher (ai-dynamo#16208) to Audio and Realtime Serving. - Note that the KVCR memory-service DGD now mounts a content-addressed startup ConfigMap that deploy.sh applies first. Signed-off-by: Jonathan Tong <jt572@cornell.edu>
Rebasing onto main brought in upstream's merged diffusion templates (ai-dynamo#14660) and new example sources. Reflect them in the catalog: - Describe the merged image, language-model, and Qwen3-TTS templates as validated on Dynamo 1.5.1, drop the obsolete SGLang shareProcessNamespace note, and explain the vLLM-Omni canary health check workaround. The main Qwen3-TTS template is no longer experimental; the experimental/ manifests keep their pinned builds. - Add the SGLang Intel XPU DGDs (ai-dynamo#10023) to the aggregated, KV-routing, and disaggregated pages, the SGLang LoRA router launcher (ai-dynamo#15746) to LoRA Serving, and the vLLM transcription launcher (ai-dynamo#16208) to Audio and Realtime Serving. - Note that the KVCR memory-service DGD now mounts a content-addressed startup ConfigMap that deploy.sh applies first. Signed-off-by: Jonathan Tong <jt572@cornell.edu>
Rebasing onto main brought in upstream's merged diffusion templates (ai-dynamo#14660) and new example sources. Reflect them in the catalog: - Describe the merged image, language-model, and Qwen3-TTS templates as validated on Dynamo 1.5.1, drop the obsolete SGLang shareProcessNamespace note, and explain the vLLM-Omni canary health check workaround. The main Qwen3-TTS template is no longer experimental; the experimental/ manifests keep their pinned builds. - Add the SGLang Intel XPU DGDs (ai-dynamo#10023) to the aggregated, KV-routing, and disaggregated pages, the SGLang LoRA router launcher (ai-dynamo#15746) to LoRA Serving, and the vLLM transcription launcher (ai-dynamo#16208) to Audio and Realtime Serving. - Note that the KVCR memory-service DGD now mounts a content-addressed startup ConfigMap that deploy.sh applies first. Signed-off-by: Jonathan Tong <jt572@cornell.edu>
Rebasing onto main brought in upstream's merged diffusion templates (ai-dynamo#14660) and new example sources. Reflect them in the catalog: - Describe the merged image, language-model, and Qwen3-TTS templates as validated on Dynamo 1.5.1, drop the obsolete SGLang shareProcessNamespace note, and explain the vLLM-Omni canary health check workaround. The main Qwen3-TTS template is no longer experimental; the experimental/ manifests keep their pinned builds. - Add the SGLang Intel XPU DGDs (ai-dynamo#10023) to the aggregated, KV-routing, and disaggregated pages, the SGLang LoRA router launcher (ai-dynamo#15746) to LoRA Serving, and the vLLM transcription launcher (ai-dynamo#16208) to Audio and Realtime Serving. - Note that the KVCR memory-service DGD now mounts a content-addressed startup ConfigMap that deploy.sh applies first. Signed-off-by: Jonathan Tong <jt572@cornell.edu>
Rebasing onto main brought in upstream's merged diffusion templates (ai-dynamo#14660) and new example sources. Reflect them in the catalog: - Describe the merged image, language-model, and Qwen3-TTS templates as validated on Dynamo 1.5.1, drop the obsolete SGLang shareProcessNamespace note, and explain the vLLM-Omni canary health check workaround. The main Qwen3-TTS template is no longer experimental; the experimental/ manifests keep their pinned builds. - Add the SGLang Intel XPU DGDs (ai-dynamo#10023) to the aggregated, KV-routing, and disaggregated pages, the SGLang LoRA router launcher (ai-dynamo#15746) to LoRA Serving, and the vLLM transcription launcher (ai-dynamo#16208) to Audio and Realtime Serving. - Note that the KVCR memory-service DGD now mounts a content-addressed startup ConfigMap that deploy.sh applies first. Signed-off-by: Jonathan Tong <jt572@cornell.edu>
Rebasing onto main brought in upstream's merged diffusion templates (ai-dynamo#14660) and new example sources. Reflect them in the catalog: - Describe the merged image, language-model, and Qwen3-TTS templates as validated on Dynamo 1.5.1, drop the obsolete SGLang shareProcessNamespace note, and explain the vLLM-Omni canary health check workaround. The main Qwen3-TTS template is no longer experimental; the experimental/ manifests keep their pinned builds. - Add the SGLang Intel XPU DGDs (ai-dynamo#10023) to the aggregated, KV-routing, and disaggregated pages, the SGLang LoRA router launcher (ai-dynamo#15746) to LoRA Serving, and the vLLM transcription launcher (ai-dynamo#16208) to Audio and Realtime Serving. - Note that the KVCR memory-service DGD now mounts a content-addressed startup ConfigMap that deploy.sh applies first. Signed-off-by: Jonathan Tong <jt572@cornell.edu>
Rebasing onto main brought in upstream's merged diffusion templates (ai-dynamo#14660) and new example sources. Reflect them in the catalog: - Describe the merged image, language-model, and Qwen3-TTS templates as validated on Dynamo 1.5.1, drop the obsolete SGLang shareProcessNamespace note, and explain the vLLM-Omni canary health check workaround. The main Qwen3-TTS template is no longer experimental; the experimental/ manifests keep their pinned builds. - Add the SGLang Intel XPU DGDs (ai-dynamo#10023) to the aggregated, KV-routing, and disaggregated pages, the SGLang LoRA router launcher (ai-dynamo#15746) to LoRA Serving, and the vLLM transcription launcher (ai-dynamo#16208) to Audio and Realtime Serving. - Note that the KVCR memory-service DGD now mounts a content-addressed startup ConfigMap that deploy.sh applies first. Signed-off-by: Jonathan Tong <jt572@cornell.edu>
Rebasing onto main brought in upstream's merged diffusion templates (ai-dynamo#14660) and new example sources. Reflect them in the catalog: - Describe the merged image, language-model, and Qwen3-TTS templates as validated on Dynamo 1.5.1, drop the obsolete SGLang shareProcessNamespace note, and explain the vLLM-Omni canary health check workaround. The main Qwen3-TTS template is no longer experimental; the experimental/ manifests keep their pinned builds. - Add the SGLang Intel XPU DGDs (ai-dynamo#10023) to the aggregated, KV-routing, and disaggregated pages, the SGLang LoRA router launcher (ai-dynamo#15746) to LoRA Serving, and the vLLM transcription launcher (ai-dynamo#16208) to Audio and Realtime Serving. - Note that the KVCR memory-service DGD now mounts a content-addressed startup ConfigMap that deploy.sh applies first. Signed-off-by: Jonathan Tong <jt572@cornell.edu>
Details:
Adds Intel XPU deployment templates for SGLang on Kubernetes under
examples/backends/sglang/deploy/xpu/, using Dynamic Resource Allocation (DRA)plus one device-plugin variant:
agg_xpu_dra.yaml: aggregated, one workeragg_router_xpu_dra.yaml: aggregated, two workers behind the KV routerdisagg_xpu_dra.yaml: disaggregated prefill and decode (NIXL)disagg_planner_xpu_dra.yaml: disaggregated with the Plannerdisagg_xpu.yaml: disaggregated with the Intel device plugin (gpu.intel.com/xe)xpu/README.md, and a link fromsglang/deploy/README.mdNotes on the XPU launch:
DYN_FORWARDPASS_METRIC_PORTbefore starting. The operatorinjects it, and the XPU SGLang build crashes on the first request when forward
pass metrics are on (they use
torch.cuda.Event).--attention-backend intel_xpu. The default Triton backendreturned garbage tokens (
!!!!) in testing.my-registry/...:my-tag). The operator reads theDynamo version from the image tag, so the manifests set no
runtimeVersionOverride.pre_deployment_sweeping_mode: none).Validation
Tested on an Intel XPU node (single node, Kubernetes with DRA) with a
dynamo_xpuruntime image:agg_xpu_dra.yaml: pods ready;/v1/completionsand/v1/chat/completionsreturn correct text.
disagg_xpu_dra.yaml: frontend, prefill and decode ready; completions correct.kubectl apply --dry-run=server, and pre-commit passes.Related Issues
#16321