Skip to content
Merged
48 changes: 38 additions & 10 deletions docs/fern/pages/recipes/model-recipes/kimi-k3-1-pro.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ body:has(#recipe-framework-vllm:checked):has(#recipe-sku-vr200:checked):has(#rec
}
`}</style>

Serve `moonshotai/Kimi-K3.1-Pro` with NVIDIA Dynamo using vLLM or SGLang. Pick your framework, GPU, and serving topology; deployment commands update to match. SGLang targets are experimental, text-only, and functional-only; no performance claims are made.
Serve `moonshotai/Kimi-K3.1-Pro` (currently private; authorized access required) with NVIDIA Dynamo using vLLM or SGLang. Pick your framework, GPU, and serving topology; deployment commands update to match. SGLang targets are experimental, text-only, and functional-only; no performance claims are made.
Comment thread
NVShreyas marked this conversation as resolved.
Resolved

<div className="dynamo-target-picker">
<p className="dynamo-target-picker-title">Choose your deployment target</p>
Expand Down Expand Up @@ -282,6 +282,23 @@ kubectl apply -f recipes/kimi-k3.1-pro/vllm/disagg-vr200-agentx/deploy-vr200.yam

</div>

You can also apply the B200 or VR200 target overlay directly:

```bash
kubectl apply -k recipes/kimi-k3.1-pro/vllm/agg-b200-agentx/kustomize/overlays/b200 -n ${NAMESPACE}
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Resolved
# Or choose one VR200 topology:
kubectl apply -k recipes/kimi-k3.1-pro/vllm/agg-vr200-agentx/kustomize/overlays/vr200 -n ${NAMESPACE}
kubectl apply -k recipes/kimi-k3.1-pro/vllm/disagg-vr200-agentx/kustomize/overlays/vr200 -n ${NAMESPACE}
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Resolved
```

To render and apply a generic overlay, for example:

```bash
kubectl kustomize recipes/kimi-k3.1-pro/sglang/agg-gb300-agentic/kustomize/overlays/generic | kubectl apply -f - -n ${NAMESPACE}
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Resolved
```

Use the SKU overlay (for example `gb300`) to add scheduling, registry credentials, and ComputeDomain settings for your cluster.

Each target also has a `generic` Kustomize overlay for customization. It inherits the base model paths and hardware requirements. B200 and VR200 targets enable DSpark; separate `*-synthetic-al` overlays change draft acceptance for benchmarking.

## Smoke Test
Expand Down Expand Up @@ -360,15 +377,15 @@ All results below measure text-only traffic; the selected target is highlighted.

<div className="dynamo-variant-table-wrap">
<table className="dynamo-variant-table dynamo-kimi-comparison">
<thead><tr><th>Workload</th><th>Recipe</th><th>Framework</th><th>SKU</th><th>GPUs</th><th>Concurrency</th><th>System Output Tokens/s</th><th>System Output Tokens/s/GPU</th><th>User Output Tokens/s (P50)</th><th>TTFT P50 (ms)</th></tr></thead>
<thead><tr><th>Workload</th><th>Recipe</th><th>Framework</th><th>SKU</th><th>Concurrency</th><th>System Output Tokens/s/GPU</th><th>User Output Tokens/s (P50)</th><th>TTFT P50 (ms)</th></tr></thead>
<tbody>
<tr data-highlight-target="vllm-vr200-agg"><td>AgentX</td><td>aggregated, 2 replicas</td><td>vLLM</td><td>VR200</td><td>16</td><td>180</td><td>2,448.08</td><td>153.01</td><td>57.71</td><td>1,534</td></tr>
<tr data-highlight-target="vllm-vr200-disagg"><td>AgentX</td><td>disaggregated 2P1D</td><td>vLLM</td><td>VR200</td><td>24</td><td>174</td><td>2,932.63</td><td>122.19</td><td>73.93</td><td>1,761</td></tr>
<tr data-highlight-target="vllm-b200-agg"><td>AgentX</td><td>aggregated</td><td>vLLM</td><td>B200</td><td>16</td><td>48</td><td>422.50</td><td>≈26.41</td><td>64.02</td><td>550.45</td></tr>
<tr data-highlight-target="vllm-gb300-agg"><td>AgentX</td><td>aggregated TP8</td><td>vLLM</td><td>GB300</td><td>8 (2 nodes)</td><td>32</td><td>≈319.12</td><td>39.89</td><td>76.7</td><td>428</td></tr>
<tr data-highlight-target="vllm-gb300-disagg"><td>AgentX</td><td>disaggregated 2P1D</td><td>vLLM</td><td>GB300</td><td>24</td><td>48</td><td>≈553.92</td><td>23.08</td><td>87.1</td><td>656</td></tr>
<tr data-highlight-target="vllm-gb300-agg"><td>AgentX</td><td>aggregated multimodal overlay (pre-release)</td><td>vLLM</td><td>GB300</td><td>8 (2 nodes)</td><td>32</td><td>≈316.48</td><td>39.56</td><td>70.9</td><td>440</td></tr>
<tr data-highlight-target="vllm-gb300-disagg"><td>AgentX</td><td>disaggregated multimodal overlay (pre-release)</td><td>vLLM</td><td>GB300</td><td>24</td><td>48</td><td>≈561.12</td><td>23.38</td><td>85.7</td><td>693</td></tr>
<tr data-highlight-target="vllm-vr200-agg"><td>AgentX</td><td>aggregated, 2 replicas</td><td>vLLM</td><td>VR200</td><td>180</td><td>153.01</td><td>57.71</td><td>1,534</td></tr>
<tr data-highlight-target="vllm-vr200-disagg"><td>AgentX</td><td>disaggregated 2P1D</td><td>vLLM</td><td>VR200</td><td>174</td><td>122.19</td><td>73.93</td><td>1,761</td></tr>
<tr data-highlight-target="vllm-b200-agg"><td>AgentX</td><td>aggregated</td><td>vLLM</td><td>B200</td><td>48</td><td>≈26.41</td><td>64.02</td><td>550.45</td></tr>
<tr data-highlight-target="vllm-gb300-agg"><td>AgentX</td><td>aggregated TP8</td><td>vLLM</td><td>GB300</td><td>32</td><td>39.89</td><td>76.7</td><td>428</td></tr>
<tr data-highlight-target="vllm-gb300-disagg"><td>AgentX</td><td>disaggregated 2P1D</td><td>vLLM</td><td>GB300</td><td>48</td><td>23.08</td><td>87.1</td><td>656</td></tr>
<tr data-highlight-target="vllm-gb300-agg"><td>AgentX</td><td>aggregated multimodal overlay (pre-release)</td><td>vLLM</td><td>GB300</td><td>32</td><td>39.56</td><td>70.9</td><td>440</td></tr>
<tr data-highlight-target="vllm-gb300-disagg"><td>AgentX</td><td>disaggregated multimodal overlay (pre-release)</td><td>vLLM</td><td>GB300</td><td>48</td><td>23.38</td><td>85.7</td><td>693</td></tr>
</tbody>
</table>
</div>
Expand All @@ -378,6 +395,18 @@ All results below measure text-only traffic; the selected target is highlighted.

## Compare All Targets

| Target | GPUs | Parallelism | Validation |
| --- | ---: | --- | --- |
| [GB300 vLLM aggregated](https://github.com/ai-dynamo/dynamo/blob/main/recipes/kimi-k3.1-pro/vllm/agg-gb300-agentic/deploy-gb300.yaml) | 8 | TP8 per worker | Text-only benchmark |
| [GB300 vLLM aggregated multimodal](https://github.com/ai-dynamo/dynamo/blob/main/recipes/kimi-k3.1-pro/vllm/agg-gb300-agentic/deploy-gb300-multimodal.yaml) | 8 | TP8 per worker | Text-only benchmark with pre-release weights; image input validated |
| [GB300 vLLM disaggregated](https://github.com/ai-dynamo/dynamo/blob/main/recipes/kimi-k3.1-pro/vllm/disagg-gb300-agentic/deploy-gb300.yaml) | 24 | TP8 per worker | Text-only benchmark |
| [GB300 vLLM disaggregated multimodal](https://github.com/ai-dynamo/dynamo/blob/main/recipes/kimi-k3.1-pro/vllm/disagg-gb300-agentic/deploy-gb300-multimodal.yaml) | 24 | TP8 per worker | Text-only benchmark with pre-release weights; image input validated |
| [GB300 SGLang aggregated](https://github.com/ai-dynamo/dynamo/blob/main/recipes/kimi-k3.1-pro/sglang/agg-gb300-agentic/deploy-gb300.yaml) | 8 | TP8 per worker | Functional only; no performance claim |
| [GB300 SGLang disaggregated](https://github.com/ai-dynamo/dynamo/blob/main/recipes/kimi-k3.1-pro/sglang/disagg-gb300-agentic/deploy-gb300.yaml) | 16 | TP8 per worker | Functional only; no performance claim |
| [B200 vLLM aggregated](https://github.com/ai-dynamo/dynamo/blob/main/recipes/kimi-k3.1-pro/vllm/agg-b200-agentx/deploy-b200.yaml) | 16 | TP8 × PP2 | Text-only benchmark |
| [VR200 vLLM aggregated](https://github.com/ai-dynamo/dynamo/blob/main/recipes/kimi-k3.1-pro/vllm/agg-vr200-agentx/deploy-vr200.yaml) | 16 | TP8 per worker | Text-only benchmark |
| [VR200 vLLM disaggregated](https://github.com/ai-dynamo/dynamo/blob/main/recipes/kimi-k3.1-pro/vllm/disagg-vr200-agentx/deploy-vr200.yaml) | 24 | TP8 per worker | Text-only benchmark |

GPU counts below describe the checked-in manifests. P counts prefill workers; D counts decode workers.

<div className="dynamo-variant-table-wrap">
Expand All @@ -398,7 +427,6 @@ GPU counts below describe the checked-in manifests. P counts prefill workers; D
## Notes

- SGLang targets are experimental, text-only, and functional-only. No SGLang performance claims are made. Use the vLLM multimodal overlays for image and video input.
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Resolved
The SGLang frontends run the Dynamo chat processor (`--dyn-chat-processor dynamo`, `DYN_SGL_FRONTEND_DECODING=true`). With the SGLang chat processor, strict `json_schema` output with thinking enabled was returned in `reasoning_content`; if you switch processors, disable thinking for `json_schema` requests (`"chat_template_kwargs": {"thinking": false}`).
- Synthetic draft acceptance is for benchmarking. Use the natural-acceptance variants for normal serving.
- Disaggregated performance depends on your fabric; RDMA tuning is cluster-specific.

Expand Down
49 changes: 2 additions & 47 deletions recipes/kimi-k3.1-pro/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,51 +3,6 @@ SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES.
SPDX-License-Identifier: Apache-2.0
-->

# Kimi-K3.1-Pro Recipe
# Kimi-K3.1-Pro recipes

Recipes for `moonshotai/Kimi-K3.1-Pro` (currently private; authorized access required) on Dynamo.

## Deployment Targets

| Target | GPUs | Parallelism | Validation |
| --- | ---: | --- | --- |
| [GB300 vLLM aggregated](vllm/agg-gb300-agentic/deploy-gb300.yaml) | 8 | TP8 per worker | Text-only benchmark |
| [GB300 vLLM aggregated multimodal](vllm/agg-gb300-agentic/deploy-gb300-multimodal.yaml) | 8 | TP8 per worker | Text-only benchmark with pre-release weights; image input validated |
| [GB300 vLLM disaggregated](vllm/disagg-gb300-agentic/deploy-gb300.yaml) | 24 | TP8 per worker | Text-only benchmark |
| [GB300 vLLM disaggregated multimodal](vllm/disagg-gb300-agentic/deploy-gb300-multimodal.yaml) | 24 | TP8 per worker | Text-only benchmark with pre-release weights; image input validated |
| [GB300 SGLang aggregated](sglang/agg-gb300-agentic/deploy-gb300.yaml) | 8 | TP8 per worker | Functional only; no performance claim |
| [GB300 SGLang disaggregated](sglang/disagg-gb300-agentic/deploy-gb300.yaml) | 16 | TP8 per worker | Functional only; no performance claim |
| [B200 vLLM aggregated](vllm/agg-b200-agentx/deploy-b200.yaml) | 16 | TP8 × PP2 | Text-only benchmark |
| [VR200 vLLM aggregated](vllm/agg-vr200-agentx/deploy-vr200.yaml) | 16 | TP8 per worker | Text-only benchmark |
| [VR200 vLLM disaggregated](vllm/disagg-vr200-agentx/deploy-vr200.yaml) | 24 | TP8 per worker | Text-only benchmark |

Weights: all recipes expect the model at `/shared-model-cache/moonshotai/Kimi-K3.1-Pro`, on the `shared-model-cache` PVC. Create the PVC and download the weights with `model-cache/model-cache.yaml` and `model-cache/model-download.yaml`. The download Job needs a `hf-token-secret` Secret with your Hugging Face token. Set `storageClassName` in the PVC to a ReadWriteMany class on your cluster.

The B200 recipe also needs the draft checkpoint at its `SPECULATIVE_CONFIG` path on the PVC. VR200 uses the same PVC for main weights and expects the `Inferact/Kimi-K3-DSpark` draft checkpoint at `/shared-model-cache/Inferact/Kimi-K3-DSpark`. Stage the draft checkpoint there before deploying. The shared download Job downloads only the main weights onto the PVC.

Deploy B200 or VR200 with the target overlay:

```bash
kubectl apply -k recipes/kimi-k3.1-pro/vllm/agg-b200-agentx/kustomize/overlays/b200 -n ${NAMESPACE}
# Or choose one VR200 topology:
kubectl apply -k recipes/kimi-k3.1-pro/vllm/agg-vr200-agentx/kustomize/overlays/vr200 -n ${NAMESPACE}
kubectl apply -k recipes/kimi-k3.1-pro/vllm/disagg-vr200-agentx/kustomize/overlays/vr200 -n ${NAMESPACE}
```

Set `NAMESPACE` to your namespace and review storage, scheduling, and registry settings first. These targets use DSpark speculative decoding; their `*-synthetic-al` overlays reproduce the performance configuration with synthetic acceptance length 3.84 and seven draft tokens. GB300 and VR200 hardware overlays include ComputeDomain channel claims and require the NVIDIA DRA driver.

Deploy: render a recipe with kustomize and apply it. Each recipe has a `generic` overlay for customization; review inherited storage paths and hardware requirements, for example:

kubectl kustomize recipes/kimi-k3.1-pro/sglang/agg-gb300-agentic/kustomize/overlays/generic | kubectl apply -f - -n ${NAMESPACE}

Use the SKU overlay (for example `gb300`) to add scheduling, registry credentials, and ComputeDomain settings for your cluster.

Modes and limits:

- Performance results are for text-only traffic.
- Both SGLang recipes are experimental and functional-only. No SGLang performance claims are made.

Docs and benchmark:

- Docs page: `docs/fern/pages/recipes/model-recipes/kimi-k3-1-pro.mdx` (deploy, smoke test, benchmarks, and target comparison).
- Benchmark: `perf/perf.yaml` runs AIPerf 0.13.0 with the AgentX workload against a deployed frontend. See the docs page for benchmark deployment commands.
Refer to the [documentation](https://docs.nvidia.com/dynamo/dev/recipes/kimi-k3-1-pro) for instructions and details.
Original file line number Diff line number Diff line change
Expand Up @@ -26,9 +26,6 @@
apiVersion: nvidia.com/v1beta1
kind: DynamoGraphDeployment
metadata:
labels:
app.kubernetes.io/name: kimi-k3-1-pro-sglang-aggregate-gb300
app.kubernetes.io/part-of: dynamo
name: kimi-k3-1-pro-sglang-aggregate-gb300
spec:
backendFramework: sglang
Expand All @@ -44,8 +41,6 @@ spec:
- "8000"
- --dyn-chat-processor
- dynamo
- --router-mode
- round-robin
- --router-session-affinity-ttl-secs
- "900"
- --trust-remote-code
Expand All @@ -54,6 +49,8 @@ spec:
- -m
- dynamo.frontend
env:
- name: DYN_ROUTER_MODE
value: round-robin
- name: DYN_SGL_FRONTEND_DECODING
value: "true"
- name: HF_HUB_OFFLINE
Expand All @@ -64,8 +61,6 @@ spec:
value: "3600"
- name: DYN_TOKENIZER_CACHE_BYTES
value: "8589934592"
- name: MODEL_PATH
value: /shared-model-cache/moonshotai/Kimi-K3.1-Pro
image: nvcr.io/nvidia/ai-dynamo/sglang-runtime:1.7.0-kimi-k3.1-pro-dev.1
imagePullPolicy: IfNotPresent
name: main
Expand Down Expand Up @@ -144,8 +139,6 @@ spec:
value: "true"
- name: HOME
value: /tmp/k31-home
- name: PYTHONUNBUFFERED
value: "1"
- name: SGLANG_CUSTOM_ALL_REDUCE_V2_MAX_SIZE_KB
value: "32768"
- name: SGLANG_UNBALANCED_MODEL_LOADING_TIMEOUT_S
Expand Down Expand Up @@ -195,7 +188,6 @@ spec:
volumeMounts:
- mountPath: /shared-model-cache
name: shared-model-cache
readOnly: true
imagePullSecrets:
- name: ngc-pull-secret
- name: acr-pull-secret
Expand All @@ -214,7 +206,6 @@ spec:
- name: shared-model-cache
persistentVolumeClaim:
claimName: shared-model-cache
readOnly: true
replicas: 1
runtimeVersionOverride: 1.7.0
sharedMemorySize: 64Gi
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -26,9 +26,6 @@
apiVersion: nvidia.com/v1beta1
kind: DynamoGraphDeployment
metadata:
labels:
app.kubernetes.io/name: kimi-k3-1-pro-sglang-aggregate-gb300
app.kubernetes.io/part-of: dynamo
name: kimi-k3-1-pro-sglang-aggregate-gb300
spec:
backendFramework: sglang
Expand All @@ -44,8 +41,6 @@ spec:
- "8000"
- --dyn-chat-processor
- dynamo
- --router-mode
- round-robin
- --router-session-affinity-ttl-secs
- "900"
- --trust-remote-code
Expand All @@ -54,6 +49,8 @@ spec:
- -m
- dynamo.frontend
env:
- name: DYN_ROUTER_MODE
value: round-robin
- name: DYN_SGL_FRONTEND_DECODING
value: "true"
- name: HF_HUB_OFFLINE
Expand All @@ -64,8 +61,6 @@ spec:
value: "3600"
- name: DYN_TOKENIZER_CACHE_BYTES
value: "8589934592"
- name: MODEL_PATH
value: /shared-model-cache/moonshotai/Kimi-K3.1-Pro
image: nvcr.io/nvidia/ai-dynamo/sglang-runtime:1.7.0-kimi-k3.1-pro-dev.1
imagePullPolicy: IfNotPresent
name: main
Expand Down Expand Up @@ -134,8 +129,6 @@ spec:
value: "true"
- name: HOME
value: /tmp/k31-home
- name: PYTHONUNBUFFERED
value: "1"
- name: SGLANG_CUSTOM_ALL_REDUCE_V2_MAX_SIZE_KB
value: "32768"
- name: SGLANG_UNBALANCED_MODEL_LOADING_TIMEOUT_S
Expand Down Expand Up @@ -183,13 +176,11 @@ spec:
volumeMounts:
- mountPath: /shared-model-cache
name: shared-model-cache
readOnly: true
terminationGracePeriodSeconds: 60
volumes:
- name: shared-model-cache
persistentVolumeClaim:
claimName: shared-model-cache
readOnly: true
replicas: 1
runtimeVersionOverride: 1.7.0
sharedMemorySize: 64Gi
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -18,9 +18,6 @@ apiVersion: nvidia.com/v1beta1
kind: DynamoGraphDeployment
metadata:
name: kimi-k3-1-pro-sglang-aggregate-gb300
labels:
app.kubernetes.io/name: kimi-k3-1-pro-sglang-aggregate-gb300
app.kubernetes.io/part-of: dynamo
spec:
backendFramework: sglang
components:
Expand All @@ -35,8 +32,6 @@ spec:
- '8000'
- --dyn-chat-processor
- dynamo
- --router-mode
- round-robin
- --router-session-affinity-ttl-secs
- '900'
- --trust-remote-code
Expand All @@ -45,6 +40,8 @@ spec:
- -m
- dynamo.frontend
env:
- name: DYN_ROUTER_MODE
value: round-robin
- name: DYN_SGL_FRONTEND_DECODING
value: 'true'
- name: HF_HUB_OFFLINE
Expand All @@ -55,8 +52,6 @@ spec:
value: '3600'
- name: DYN_TOKENIZER_CACHE_BYTES
value: '8589934592'
- name: MODEL_PATH
value: /shared-model-cache/moonshotai/Kimi-K3.1-Pro
image: nvcr.io/nvidia/ai-dynamo/sglang-runtime:1.7.0-kimi-k3.1-pro-dev.1
imagePullPolicy: IfNotPresent
name: main
Expand Down Expand Up @@ -125,8 +120,6 @@ spec:
value: 'true'
- name: HOME
value: /tmp/k31-home
- name: PYTHONUNBUFFERED
value: '1'
- name: SGLANG_CUSTOM_ALL_REDUCE_V2_MAX_SIZE_KB
value: '32768'
- name: SGLANG_UNBALANCED_MODEL_LOADING_TIMEOUT_S
Expand Down Expand Up @@ -174,13 +167,11 @@ spec:
volumeMounts:
- mountPath: /shared-model-cache
name: shared-model-cache
readOnly: true
terminationGracePeriodSeconds: 60
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Resolved
volumes:
- name: shared-model-cache
persistentVolumeClaim:
claimName: shared-model-cache
readOnly: true
replicas: 1
runtimeVersionOverride: 1.7.0
sharedMemorySize: 64Gi
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,3 @@
path: /metadata
value:
name: kimi-k3-1-pro-sglang-aggregate-gb300
labels:
app.kubernetes.io/name: kimi-k3-1-pro-sglang-aggregate-gb300
app.kubernetes.io/part-of: dynamo
Loading
Loading