Repository navigation
feat: add Kimi K3.1 Pro release support with published frontend crates - #18124
Conversation
Signed-off-by: Indrajit Bhosale <iamindrajitb@gmail.com>
…LLM PD Published Dynamo wheels cannot load the kimi_k31 tokenizer, so the PD frontend 404s. Pin ai-dynamo/dynamo#18124, which pulls the K3.1 frontend crates, and record it as the router version. The PD roles also use the MXFP4 indexer cache, which the week-1 rules allow and vLLM's K3.1 DSA path supports with NIXL.
Published Dynamo wheels cannot load the kimi_k31 tokenizer, so the PD frontend 404s. Pin ai-dynamo/dynamo#18124, which pulls the K3.1 frontend crates, and record it as the router version. Every role also uses the MXFP4 indexer cache, which the week-1 rules allow and vLLM's K3.1 DSA path supports with DCP8, NIXL, MegaMoE and the latency-specialized decode.
Published Dynamo wheels cannot load the kimi_k31 tokenizer and their SGLang argument check rejects DCP decode for K3.1's DSA layers. ai-dynamo/dynamo#18124 fixes both; pin it and record it as the router version.
The SGLang image ships no Dynamo, and published Dynamo wheels cannot load the kimi_k31 tokenizer or accept DCP decode for K3.1's DSA layers. Install ai-dynamo/dynamo#18124, which fixes both, and record it as the router version.
❌ Dynamo PR CI failed — run 42110113452 (attempt 1) on
|
| Framework | 1-GPU amd64 | 1-GPU arm64 | Multi-GPU amd64 | Deploy | Snapshot |
|---|---|---|---|---|---|
| vLLM | ❌ 1 | ✅ 1 | ❌ 1 | ⏹️ 2 ✅ 2 | ⏹️ 1 ✅ 2 |
| SGLang | ❌ 1 | ✅ 1 | ✅ 1 | ✅ 3 | ✅ 3 |
| TRT-LLM | ✅ 1 | ❌ 1 | ✅ 1 | ✅ 2 | ✅ 3 |
Failure details
4 jobs failed: 2 SGLang multimodal router tests get a 0% KV hit rate, 6 vLLM 2-GPU serve tests fail because the server exits at startup, the vLLM 1-GPU sequential stage selects no tests (pytest exit 5), and a TRT-LLM arm64 NIXL telemetry test times out. The 3 cancelled vLLM deploy/snapshot jobs hit their job time limit (25m/30m), not fail-fast.
❌ sglang-runtime / Test cuda13.0, amd64: router_kv_hit_rate 0.000 in MM router tests
Job: sglang-runtime / Test cuda13.0, amd64 · Failed step: Run GPU tests (parallel) · Logs: gh run view --job 127542808893 -R ai-dynamo/dynamo --log-failed
[w4] FAILED tests/serve/test_sglang.py::test_sglang_deployment[video_agg_fd_qwen-4] - AssertionError: router_kv_hit_rate: mean over R2+ (0.000) below required min (0.9). delta_n=2.0, delta_sum=0.000. Router-side block hashes did not match the worker — MM-aware routing degraded silently.
[w6] FAILED tests/serve/test_sglang.py::test_sglang_deployment[mm_agg_router_qwen3.5-0.8b-4] - AssertionError: router_kv_hit_rate: mean over R2+ (0.000) below required min (0.7). delta_n=2.0, delta_sum=0.000. Router-side block hashes did not match the worker — MM-aware routing degraded silently.
== 2 failed, 14 passed, 2 skipped in 1231.70s (20:31) ==
Both multimodal KV-router deployments failed the same way on every retry: the router's KV hit rate was 0.000 on repeat requests, meaning router-side block hashes did not match the worker's for multimodal (image/video) requests.
❌ vllm-runtime / 2-GPU Test cuda13.0, amd64: server exits with code 1 before health check (6 tests)
Job: vllm-runtime / 2-GPU Test cuda13.0, amd64 · Failed step: Run GPU tests (sequential) · Logs: gh run view --job 127543346267 -R ai-dynamo/dynamo --log-failed
FAILED tests/serve/test_vllm.py::test_lora_aggregated_router[2] - RuntimeError: Main server process exited with code 1 while waiting for health check
FAILED tests/serve/test_vllm.py::test_serve_deployment[agg-router-3] - RuntimeError: Main server process exited with code 1 while waiting for health check
FAILED tests/serve/test_vllm.py::test_serve_deployment[agg-router-approx-3] - RuntimeError: Main server process exited with code 1 while waiting for health check
FAILED tests/serve/test_vllm.py::test_serve_deployment[disaggregated-3] - RuntimeError: Main server process exited with code 1 while waiting for health check
FAILED tests/serve/test_vllm.py::test_serve_deployment[multi_node_tp_headless-3] - Failed: Timeout (>300.0s) from pytest-timeout.
FAILED tests/vllm_self_benchmark/test_self_benchmark_gpu.py::test_self_benchmark_disagg_serves_after_bench - RuntimeError: Main server process exited with code 1 while waiting for health check
[BASH] [rank1]: ... TCPStore check for dump key "exception_dump" failed (store unavailable, not absent key) ... Error: Broken pipe
===== 6 failed, 3 passed, 7 skipped, 6163 deselected in 1923.84s (0:32:03) =====
All multi-worker vLLM launches (agg_router.sh, disagg, LoRA router, self-benchmark) exited with code 1 within ~50 s, before /v1/models came up; multi_node_tp_headless instead hung until the 300 s pytest timeout with repeated torch c10d sendBytes / TCPStore broken-pipe errors. The failing-launch stderr is not in the failed-step log; check the per-test bash.log.txt output in the full job log.
❌ vllm-runtime / Test cuda13.0, amd64: pre_merge_gpu stage selected 0 tests (pytest exit 5)
Job: vllm-runtime / Test cuda13.0, amd64 · Failed step: Run GPU tests (sequential) · Logs: gh run view --job 127543346301 -R ai-dynamo/dynamo --log-failed
▶️ Executing (cwd=/workspace): python -m pytest -n 0 ... -m (pre_merge and vllm and gpu_1 and not sidecar) and not profiled_vram_gib ...
collecting ... collected 6172 items / 6172 deselected / 7 skipped / 0 selected
===================== 7 skipped, 6172 deselected in 16.35s =====================
🧪 Tests completed with exit code: 5
✅ pre_merge_cpu: pytest exited 0
✅ pre_merge_cpu_e2e: pytest exited 0
✅ pre_merge_gpu_parallel: pytest exited 0
❌ pre_merge_gpu: step outcome 'failure'
##[error]A pytest stage did not record a passing result. See this step's log for the stage.
No test failed: every other stage passed, but the sequential 1-GPU stage's marker expression matched zero tests, so pytest exited 5 ("no tests collected") and the step was treated as failed. This happens when every pre_merge and vllm and gpu_1 test is skipped or moved to the parallel stage.
❌ trtllm-runtime / Test cuda13.1, arm64: NIXL agent subprocess TimeoutExpired
Job: trtllm-runtime / Test cuda13.1, arm64 · Failed step: Run CPU-only tests (parallelized) · Logs: gh run view --job 127544015463 -R ai-dynamo/dynamo --log-failed
[gw0] [ 98%] FAILED tests/dependencies/test_nixl_file_telemetry.py::test_colocated_agents_start_with_occupied_telemetry_port[file-sink]
raise TimeoutExpired(
E subprocess.TimeoutExpired: Command '['/opt/dynamo/venv/bin/python', '-c', '\nimport json\nimport sys\nfrom nixl._api import nixl_agent, nixl_agent_config\n\nagent = nixl_agent(sys.argv[1], nixl_agent_config(backends=[]))\nprint(json.dumps({"name": agent.name}))\n', 'rank-0']' timed out after 30 seconds
============ 1 failed, 277 passed, 27 skipped in 171.59s (0:02:51) =============
One test failed on all retries: starting colocated NIXL agents with the telemetry port already taken hung, and the nixl_agent subprocess timed out after 30 s. (test_cuda_version_consistency also failed once but passed on retry.)
⏹️ vllm Deploy Test / agg, vllm Deploy Test / agg_router and vLLM Snapshot Deploy Test were cancelled because they hit their max execution time (25m / 30m) during the deploy test step. This was a job timeout, not a fail-fast or manual cancel.
For agents
{"pr": 18124, "run_id": 42110113452, "run_attempt": 1, "head_sha": "6dd4bd37d8cdbb1092f0fd665bd4e6bde5097319",
"failures": [
{"job": "sglang-runtime / Test cuda13.0, amd64", "job_id": 127542808893, "failed_step": "Run GPU tests (parallel)", "signature": "router_kv_hit_rate 0.000 below min; MM router block hashes mismatch", "tests": ["tests/serve/test_sglang.py::test_sglang_deployment[video_agg_fd_qwen-4]", "tests/serve/test_sglang.py::test_sglang_deployment[mm_agg_router_qwen3.5-0.8b-4]"], "log_cmd": "gh run view --job 127542808893 -R ai-dynamo/dynamo --log-failed"},
{"job": "vllm-runtime / 2-GPU Test cuda13.0, amd64", "job_id": 127543346267, "failed_step": "Run GPU tests (sequential)", "signature": "Main server process exited with code 1 while waiting for health check", "tests": ["tests/serve/test_vllm.py::test_lora_aggregated_router[2]", "tests/serve/test_vllm.py::test_serve_deployment[agg-router-3]", "tests/serve/test_vllm.py::test_serve_deployment[agg-router-approx-3]", "tests/serve/test_vllm.py::test_serve_deployment[disaggregated-3]", "tests/serve/test_vllm.py::test_serve_deployment[multi_node_tp_headless-3]", "tests/vllm_self_benchmark/test_self_benchmark_gpu.py::test_self_benchmark_disagg_serves_after_bench"], "log_cmd": "gh run view --job 127543346267 -R ai-dynamo/dynamo --log-failed"},
{"job": "vllm-runtime / Test cuda13.0, amd64", "job_id": 127543346301, "failed_step": "Run GPU tests (sequential)", "signature": "pre_merge_gpu stage selected 0 tests; pytest exit code 5", "tests": [], "log_cmd": "gh run view --job 127543346301 -R ai-dynamo/dynamo --log-failed"},
{"job": "trtllm-runtime / Test cuda13.1, arm64", "job_id": 127544015463, "failed_step": "Run CPU-only tests (parallelized)", "signature": "subprocess.TimeoutExpired: nixl_agent start timed out after 30s", "tests": ["tests/dependencies/test_nixl_file_telemetry.py::test_colocated_agents_start_with_occupied_telemetry_port[file-sink]"], "log_cmd": "gh run view --job 127544015463 -R ai-dynamo/dynamo --log-failed"}
]}Posted automatically by Devin for run 42110113452. Updated on every full-CI run of this PR.
No buildable Dynamo can load the kimi_k31 tokenizer: published wheels lack it and ai-dynamo/dynamo#18124 does not compile. The disaggregated points now run direct vllm serve pools behind vllm-router 0.1.14 with consistent-hash session routing and static NIXL prefill bootstrap ports, so both keys use framework vllm and drop dynamo from their names. Engine args, the MXFP4 indexer, DCP interleave, offload, concurrency points and forced acceptance are unchanged.
Signed-off-by: Indrajit Bhosale <iamindrajitb@gmail.com>
Signed-off-by: Indrajit Bhosale <iamindrajitb@gmail.com>
Signed-off-by: Indrajit Bhosale <iamindrajitb@gmail.com>
* feat(kimik3.1): add Kimi K3.1 Pro GB200 SGLang AgentX configs Aggregated TP16 DCP16 and 1P1D TP16 DCP16 dynamo-sglang points on gb200-nv with RadixArk DFLASH block verification, HiCache CPU offload and Mooncake PD transfer, at Kimi K3's GB200 concurrencies. * chore(kimik3.1): link the GB200 SGLang changelog entry to its PR * fix(kimik3.1): keep only documented or common recipe settings Remove the DCP comm backend and the custom all-reduce and DCP push-combine env vars, which the SGLang cookbook's GB200 command does not use. * fix(kimik3.1): install Dynamo #18124 for the GB200 SGLang frontend The SGLang image ships no Dynamo, and published Dynamo wheels cannot load the kimi_k31 tokenizer or accept DCP decode for K3.1's DSA layers. Install ai-dynamo/dynamo#18124, which fixes both, and record it as the router version. * fix(kimik3.1): say the GB200 SGLang image ships no Dynamo * feat(kimik3.1): front the GB200 SGLang workers with the SGLang router No Dynamo revision both builds and loads the kimi_k31 tokenizer, so the workers run sglang.launch_server behind the image's sglang-router 0.3.2. The configs become plain sglang (keys drop dynamo-), the Dynamo services and router args go, the reasoning and tool-call parsers move to SGLang's own flags, and the gb200-nv AgentX power rule accepts sglang. Engine args, topology, offload and forced acceptance are unchanged. * fix(kimik3.1): use SGLang's replicated K3 MoE front at TP16 on GB200 At TP16 the K3 sharded MoE front builds an N=1048 slice GEMM that the warp split-K kernel has no output tile for, so every worker capturing graphs above 17 tokens crashed. SGLANG_K3_SHARDED_MOE=0 selects the engine's replicated fused front; the front stays BF16 and K3 all-reduce fusion stays on. * feat(kimik3.1): serve the GB200 SGLang points through the image's Dynamo The K3.1 SGLang image ships Dynamo 1.6.0, whose SGLang argument check lets Kimi-K3.1 decode with DCP on its DSA backend. Following Po-Han Huang's GB300 recipes, the frontend runs the KV router with SGLang's chat processor, which renders kimi_k31 requests the frontend tokenizer cannot load; the aggregated and prefill workers publish KV events and advertise session affinity. The configs return to dynamo-sglang with the image's dynamo-router 1.6.0. Engine args, topology, offload and forced acceptance are unchanged. * fix(kimik3.1): read the GB200 DFLASH draft from the shared Hub cache The workers had no HF_HUB_CACHE, so every node downloaded the DFlash2 draft into its container on each start. Load times varied from seconds to minutes, and agg c1 and c24 failed when one node finished more than SGLang's 480 s post-load barrier after the others. Point the workers at the mounted shared cache, as Po-Han's GB300 recipes and other GB300 SGLang recipes on main do. * chore: refresh PR #4804 for sweep reuse [skip-sweep]
Signed-off-by: Indrajit Bhosale <iamindrajitb@gmail.com>
|
approving pending sglang license fixes |
06e34ad
into
release/1.7.0-kimi-k3.1-pro-dev.1
* feat(kimik3.1): add day-0 vLLM AgentX on GB200 Aggregated TP8 x PP2 x DCP8 and disaggregated 1P1D TP16 Kimi K3.1 Pro points on gb200-nv with the K3.1 DSpark draft and Simple CPU KV offload, plus the lane's shared run root and AgentX power rule. * chore(kimik3.1): link the GB200 vLLM changelog entry to its PR * fix(kimik3.1): keep only documented or common recipe settings Remove settings that no active recipe uses and vLLM's K3.1 recipe does not document: the DCP comm backend, prompt-token details, and the FlashInfer all-reduce, execute-timeout, UCX and AIPerf failure-threshold env vars. * fix(kimik3.1): pin Dynamo #18124 and use the MXFP4 indexer on GB200 vLLM PD Published Dynamo wheels cannot load the kimi_k31 tokenizer, so the PD frontend 404s. Pin ai-dynamo/dynamo#18124, which pulls the K3.1 frontend crates, and record it as the router version. The PD roles also use the MXFP4 indexer cache, which the week-1 rules allow and vLLM's K3.1 DSA path supports with NIXL. * fix(kimik3.1): front the GB200 vLLM PD row with the vLLM Router No published Dynamo frontend loads the kimi_k31 tokenizer and the K3.1 Dynamo branch does not build, so the 1P1D row runs direct vLLM workers behind vllm-router 0.1.14 with the same engine arguments, NIXL transfer and MXFP4 indexer. The GB200 Kimi-K3.1 AgentX power rule now also covers vllm. * fix(kimik3.1): label the GB200 vLLM rows as vllm Neither GB200 row runs Dynamo: the aggregated row uses the direct vLLM frontend and the PD row the vLLM Router. Name both keys and their framework vllm, drop the Dynamo-only recipe settings, and scope the Kimi-K3.1 AgentX power rule to vllm. * fix(kimik3.1): keep GB200 vLLM PD worker connections alive for the router vllm-router reuses pooled connections that Uvicorn closes after its 5 s default keep-alive, and with retries disabled the reset aborts the request. Hold PD worker connections for 600 s, as main's GB200 vLLM PD recipe does. * fix(synthetic-acceptance): drop Kimi vLLM parsers on synthetic throughput servers Synthetic acceptance streams unverified draft tokens. The kimi_k3 tool and reasoning parsers stop streaming on some of them, which empties or truncates the response the client sees. Evals and acceptance gates verify for real and keep their parsers. * fix(kimik3.1-gb200-vllm): run disagg decode with expert parallelism Decode at TP16/EP1 holds 131.5 GiB/GPU of weights and leaves 12-19 GiB of KV, less than the 18.17 GiB one 1M-token request needs at c32 and c48. EP16, as on prefill, cuts decode weights to about 103.5 GiB/GPU.
Summary
Port Kimi K3.1 Pro serving fixes and Vera Rubin container support into
release/1.7.0-kimi-k3.1-pro-dev.1.8.2.2-kimi-k3.1-pro-dev.1, tokenizers3.1.4-kimi-k3.1-pro-dev.1, protocols8.5.0, and legacy parsers10.2.2; remove private-crate authentication requirements.Related: ai-dynamo/frontend-crates#546.
Validation
61c008c.