Skip to content

feat: add Kimi K3.1 Pro release support with published frontend crates - #18124

Merged
nv-nmailhot merged 8 commits into
release/1.7.0-kimi-k3.1-pro-dev.1from
ibhosale/kimi-k31-release
Nov 6, 2026
Merged

nv-nmailhot merged 8 commits into
release/1.7.0-kimi-k3.1-pro-dev.1from
ibhosale/kimi-k31-release

Conversation

@indrajit96

@indrajit96 indrajit96 commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Port Kimi K3.1 Pro serving fixes and Vera Rubin container support into release/1.7.0-kimi-k3.1-pro-dev.1.

  • Carry backend compatibility, multimodal, request-validation, and container compliance changes.
  • Pin published renderer 8.2.2-kimi-k3.1-pro-dev.1, tokenizers 3.1.4-kimi-k3.1-pro-dev.1, protocols 8.5.0, and legacy parsers 10.2.2; remove private-crate authentication requirements.
  • Use public Kimi-specific vLLM and SGLang base images pinned by digest: CUDA 13.0 on AMD64 and CUDA 13.4 on ARM64 for Rubin. Update architecture selection and matching compliance baselines.

Related: ai-dynamo/frontend-crates#546.

Validation

  • Locked dependency resolution, formatting, YAML, and diff checks passed.
  • Matching x86_64 Python/runtime wheels built and imported locally at 61c008c.
  • 44 Dockerfile render configurations and all four updated registry drift checks passed.
  • CI built the multi-architecture vLLM runtime and EFA images; their license checks passed.
  • CI blockers: SGLang AMD64 runtime license policy reports 135 findings; vLLM backend tests have 11 failures per architecture (missing media/audio dependencies, compatibility assertions, and one timeout). The operator lint failure also occurred before the base-image change. No fresh model-serving GPU validation was run for this port.

Signed-off-by: Indrajit Bhosale <iamindrajitb@gmail.com>
@github-actions github-actions Bot added feat documentation Improvements or additions to documentation backend::vllm Relates to the vllm backend backend::sglang Relates to the sglang backend frontend `python -m dynamo.frontend` and `dynamo-run in=http|text|grpc` multimodal container actions labels Nov 4, 2026
@indrajit96
indrajit96 marked this pull request as ready for review November 4, 2026 20:42
@indrajit96
indrajit96 requested review from a team as code owners November 4, 2026 20:42
@github-actions

github-actions Bot commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

cquil11 added a commit to SemiAnalysisAI/InferenceX that referenced this pull request Nov 5, 2026
…LLM PD

Published Dynamo wheels cannot load the kimi_k31 tokenizer, so the PD frontend 404s. Pin ai-dynamo/dynamo#18124, which pulls the K3.1 frontend crates, and record it as the router version. The PD roles also use the MXFP4 indexer cache, which the week-1 rules allow and vLLM's K3.1 DSA path supports with NIXL.
cquil11 added a commit to SemiAnalysisAI/InferenceX that referenced this pull request Nov 5, 2026
Published Dynamo wheels cannot load the kimi_k31 tokenizer, so the PD frontend 404s. Pin ai-dynamo/dynamo#18124, which pulls the K3.1 frontend crates, and record it as the router version. Every role also uses the MXFP4 indexer cache, which the week-1 rules allow and vLLM's K3.1 DSA path supports with DCP8, NIXL, MegaMoE and the latency-specialized decode.
cquil11 added a commit to SemiAnalysisAI/InferenceX that referenced this pull request Nov 5, 2026
Published Dynamo wheels cannot load the kimi_k31 tokenizer and their SGLang argument check rejects DCP decode for K3.1's DSA layers. ai-dynamo/dynamo#18124 fixes both; pin it and record it as the router version.
cquil11 added a commit to SemiAnalysisAI/InferenceX that referenced this pull request Nov 5, 2026
The SGLang image ships no Dynamo, and published Dynamo wheels cannot load the kimi_k31 tokenizer or accept DCP decode for K3.1's DSA layers. Install ai-dynamo/dynamo#18124, which fixes both, and record it as the router version.
@devin-ai-integration

devin-ai-integration Bot commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

❌ Dynamo PR CI failed — run 42110113452 (attempt 1) on 6dd4bd37d8

Gate checks: ❌ backend-status-check · ❌ deploy-status-check · ✅ dynamo-status-check

Framework 1-GPU amd64 1-GPU arm64 Multi-GPU amd64 Deploy Snapshot
vLLM ❌ 1 ✅ 1 ❌ 1 ⏹️ 2 ✅ 2 ⏹️ 1 ✅ 2
SGLang ❌ 1 ✅ 1 ✅ 1 ✅ 3 ✅ 3
TRT-LLM ✅ 1 ❌ 1 ✅ 1 ✅ 2 ✅ 3

Failure details

4 jobs failed: 2 SGLang multimodal router tests get a 0% KV hit rate, 6 vLLM 2-GPU serve tests fail because the server exits at startup, the vLLM 1-GPU sequential stage selects no tests (pytest exit 5), and a TRT-LLM arm64 NIXL telemetry test times out. The 3 cancelled vLLM deploy/snapshot jobs hit their job time limit (25m/30m), not fail-fast.

❌ sglang-runtime / Test cuda13.0, amd64: router_kv_hit_rate 0.000 in MM router tests

Job: sglang-runtime / Test cuda13.0, amd64 · Failed step: Run GPU tests (parallel) · Logs: gh run view --job 127542808893 -R ai-dynamo/dynamo --log-failed

[w4] FAILED tests/serve/test_sglang.py::test_sglang_deployment[video_agg_fd_qwen-4] - AssertionError: router_kv_hit_rate: mean over R2+ (0.000) below required min (0.9). delta_n=2.0, delta_sum=0.000. Router-side block hashes did not match the worker — MM-aware routing degraded silently.
[w6] FAILED tests/serve/test_sglang.py::test_sglang_deployment[mm_agg_router_qwen3.5-0.8b-4] - AssertionError: router_kv_hit_rate: mean over R2+ (0.000) below required min (0.7). delta_n=2.0, delta_sum=0.000. Router-side block hashes did not match the worker — MM-aware routing degraded silently.
== 2 failed, 14 passed, 2 skipped in 1231.70s (20:31) ==

Both multimodal KV-router deployments failed the same way on every retry: the router's KV hit rate was 0.000 on repeat requests, meaning router-side block hashes did not match the worker's for multimodal (image/video) requests.

❌ vllm-runtime / 2-GPU Test cuda13.0, amd64: server exits with code 1 before health check (6 tests)

Job: vllm-runtime / 2-GPU Test cuda13.0, amd64 · Failed step: Run GPU tests (sequential) · Logs: gh run view --job 127543346267 -R ai-dynamo/dynamo --log-failed

FAILED tests/serve/test_vllm.py::test_lora_aggregated_router[2] - RuntimeError: Main server process exited with code 1 while waiting for health check
FAILED tests/serve/test_vllm.py::test_serve_deployment[agg-router-3] - RuntimeError: Main server process exited with code 1 while waiting for health check
FAILED tests/serve/test_vllm.py::test_serve_deployment[agg-router-approx-3] - RuntimeError: Main server process exited with code 1 while waiting for health check
FAILED tests/serve/test_vllm.py::test_serve_deployment[disaggregated-3] - RuntimeError: Main server process exited with code 1 while waiting for health check
FAILED tests/serve/test_vllm.py::test_serve_deployment[multi_node_tp_headless-3] - Failed: Timeout (>300.0s) from pytest-timeout.
FAILED tests/vllm_self_benchmark/test_self_benchmark_gpu.py::test_self_benchmark_disagg_serves_after_bench - RuntimeError: Main server process exited with code 1 while waiting for health check
[BASH] [rank1]: ... TCPStore check for dump key "exception_dump" failed (store unavailable, not absent key) ... Error: Broken pipe
===== 6 failed, 3 passed, 7 skipped, 6163 deselected in 1923.84s (0:32:03) =====

All multi-worker vLLM launches (agg_router.sh, disagg, LoRA router, self-benchmark) exited with code 1 within ~50 s, before /v1/models came up; multi_node_tp_headless instead hung until the 300 s pytest timeout with repeated torch c10d sendBytes / TCPStore broken-pipe errors. The failing-launch stderr is not in the failed-step log; check the per-test bash.log.txt output in the full job log.

❌ vllm-runtime / Test cuda13.0, amd64: pre_merge_gpu stage selected 0 tests (pytest exit 5)

Job: vllm-runtime / Test cuda13.0, amd64 · Failed step: Run GPU tests (sequential) · Logs: gh run view --job 127543346301 -R ai-dynamo/dynamo --log-failed

▶️ Executing (cwd=/workspace): python -m pytest -n 0 ... -m (pre_merge and vllm and gpu_1 and not sidecar) and not profiled_vram_gib ...
collecting ... collected 6172 items / 6172 deselected / 7 skipped / 0 selected
===================== 7 skipped, 6172 deselected in 16.35s =====================
🧪 Tests completed with exit code: 5
✅ pre_merge_cpu: pytest exited 0
✅ pre_merge_cpu_e2e: pytest exited 0
✅ pre_merge_gpu_parallel: pytest exited 0
❌ pre_merge_gpu: step outcome 'failure'
##[error]A pytest stage did not record a passing result. See this step's log for the stage.

No test failed: every other stage passed, but the sequential 1-GPU stage's marker expression matched zero tests, so pytest exited 5 ("no tests collected") and the step was treated as failed. This happens when every pre_merge and vllm and gpu_1 test is skipped or moved to the parallel stage.

❌ trtllm-runtime / Test cuda13.1, arm64: NIXL agent subprocess TimeoutExpired

Job: trtllm-runtime / Test cuda13.1, arm64 · Failed step: Run CPU-only tests (parallelized) · Logs: gh run view --job 127544015463 -R ai-dynamo/dynamo --log-failed

[gw0] [ 98%] FAILED tests/dependencies/test_nixl_file_telemetry.py::test_colocated_agents_start_with_occupied_telemetry_port[file-sink]
    raise TimeoutExpired(
E   subprocess.TimeoutExpired: Command '['/opt/dynamo/venv/bin/python', '-c', '\nimport json\nimport sys\nfrom nixl._api import nixl_agent, nixl_agent_config\n\nagent = nixl_agent(sys.argv[1], nixl_agent_config(backends=[]))\nprint(json.dumps({"name": agent.name}))\n', 'rank-0']' timed out after 30 seconds
============ 1 failed, 277 passed, 27 skipped in 171.59s (0:02:51) =============

One test failed on all retries: starting colocated NIXL agents with the telemetry port already taken hung, and the nixl_agent subprocess timed out after 30 s. (test_cuda_version_consistency also failed once but passed on retry.)

⏹️ vllm Deploy Test / agg, vllm Deploy Test / agg_router and vLLM Snapshot Deploy Test were cancelled because they hit their max execution time (25m / 30m) during the deploy test step. This was a job timeout, not a fail-fast or manual cancel.

For agents
{"pr": 18124, "run_id": 42110113452, "run_attempt": 1, "head_sha": "6dd4bd37d8cdbb1092f0fd665bd4e6bde5097319",
 "failures": [
  {"job": "sglang-runtime / Test cuda13.0, amd64", "job_id": 127542808893, "failed_step": "Run GPU tests (parallel)", "signature": "router_kv_hit_rate 0.000 below min; MM router block hashes mismatch", "tests": ["tests/serve/test_sglang.py::test_sglang_deployment[video_agg_fd_qwen-4]", "tests/serve/test_sglang.py::test_sglang_deployment[mm_agg_router_qwen3.5-0.8b-4]"], "log_cmd": "gh run view --job 127542808893 -R ai-dynamo/dynamo --log-failed"},
  {"job": "vllm-runtime / 2-GPU Test cuda13.0, amd64", "job_id": 127543346267, "failed_step": "Run GPU tests (sequential)", "signature": "Main server process exited with code 1 while waiting for health check", "tests": ["tests/serve/test_vllm.py::test_lora_aggregated_router[2]", "tests/serve/test_vllm.py::test_serve_deployment[agg-router-3]", "tests/serve/test_vllm.py::test_serve_deployment[agg-router-approx-3]", "tests/serve/test_vllm.py::test_serve_deployment[disaggregated-3]", "tests/serve/test_vllm.py::test_serve_deployment[multi_node_tp_headless-3]", "tests/vllm_self_benchmark/test_self_benchmark_gpu.py::test_self_benchmark_disagg_serves_after_bench"], "log_cmd": "gh run view --job 127543346267 -R ai-dynamo/dynamo --log-failed"},
  {"job": "vllm-runtime / Test cuda13.0, amd64", "job_id": 127543346301, "failed_step": "Run GPU tests (sequential)", "signature": "pre_merge_gpu stage selected 0 tests; pytest exit code 5", "tests": [], "log_cmd": "gh run view --job 127543346301 -R ai-dynamo/dynamo --log-failed"},
  {"job": "trtllm-runtime / Test cuda13.1, arm64", "job_id": 127544015463, "failed_step": "Run CPU-only tests (parallelized)", "signature": "subprocess.TimeoutExpired: nixl_agent start timed out after 30s", "tests": ["tests/dependencies/test_nixl_file_telemetry.py::test_colocated_agents_start_with_occupied_telemetry_port[file-sink]"], "log_cmd": "gh run view --job 127544015463 -R ai-dynamo/dynamo --log-failed"}
 ]}

Posted automatically by Devin for run 42110113452. Updated on every full-CI run of this PR.

cquil11 added a commit to SemiAnalysisAI/InferenceX that referenced this pull request Nov 5, 2026
No buildable Dynamo can load the kimi_k31 tokenizer: published wheels lack it and ai-dynamo/dynamo#18124 does not compile. The disaggregated points now run direct vllm serve pools behind vllm-router 0.1.14 with consistent-hash session routing and static NIXL prefill bootstrap ports, so both keys use framework vllm and drop dynamo from their names. Engine args, the MXFP4 indexer, DCP interleave, offload, concurrency points and forced acceptance are unchanged.
Signed-off-by: Indrajit Bhosale <iamindrajitb@gmail.com>
Signed-off-by: Indrajit Bhosale <iamindrajitb@gmail.com>
Signed-off-by: Indrajit Bhosale <iamindrajitb@gmail.com>
cquil11 added a commit to SemiAnalysisAI/InferenceX that referenced this pull request Nov 5, 2026
* feat(kimik3.1): add Kimi K3.1 Pro GB200 SGLang AgentX configs

Aggregated TP16 DCP16 and 1P1D TP16 DCP16 dynamo-sglang points on gb200-nv
with RadixArk DFLASH block verification, HiCache CPU offload and Mooncake
PD transfer, at Kimi K3's GB200 concurrencies.

* chore(kimik3.1): link the GB200 SGLang changelog entry to its PR

* fix(kimik3.1): keep only documented or common recipe settings

Remove the DCP comm backend and the custom all-reduce and DCP push-combine env vars, which the SGLang cookbook's GB200 command does not use.

* fix(kimik3.1): install Dynamo #18124 for the GB200 SGLang frontend

The SGLang image ships no Dynamo, and published Dynamo wheels cannot load the kimi_k31 tokenizer or accept DCP decode for K3.1's DSA layers. Install ai-dynamo/dynamo#18124, which fixes both, and record it as the router version.

* fix(kimik3.1): say the GB200 SGLang image ships no Dynamo

* feat(kimik3.1): front the GB200 SGLang workers with the SGLang router

No Dynamo revision both builds and loads the kimi_k31 tokenizer, so the workers run
sglang.launch_server behind the image's sglang-router 0.3.2. The configs become plain
sglang (keys drop dynamo-), the Dynamo services and router args go, the reasoning and
tool-call parsers move to SGLang's own flags, and the gb200-nv AgentX power rule accepts
sglang. Engine args, topology, offload and forced acceptance are unchanged.

* fix(kimik3.1): use SGLang's replicated K3 MoE front at TP16 on GB200

At TP16 the K3 sharded MoE front builds an N=1048 slice GEMM that the warp
split-K kernel has no output tile for, so every worker capturing graphs above
17 tokens crashed. SGLANG_K3_SHARDED_MOE=0 selects the engine's replicated
fused front; the front stays BF16 and K3 all-reduce fusion stays on.

* feat(kimik3.1): serve the GB200 SGLang points through the image's Dynamo

The K3.1 SGLang image ships Dynamo 1.6.0, whose SGLang argument check lets
Kimi-K3.1 decode with DCP on its DSA backend. Following Po-Han Huang's GB300
recipes, the frontend runs the KV router with SGLang's chat processor, which
renders kimi_k31 requests the frontend tokenizer cannot load; the aggregated
and prefill workers publish KV events and advertise session affinity. The
configs return to dynamo-sglang with the image's dynamo-router 1.6.0. Engine
args, topology, offload and forced acceptance are unchanged.

* fix(kimik3.1): read the GB200 DFLASH draft from the shared Hub cache

The workers had no HF_HUB_CACHE, so every node downloaded the DFlash2 draft
into its container on each start. Load times varied from seconds to minutes,
and agg c1 and c24 failed when one node finished more than SGLang's 480 s
post-load barrier after the others. Point the workers at the mounted shared
cache, as Po-Han's GB300 recipes and other GB300 SGLang recipes on main do.

* chore: refresh PR #4804 for sweep reuse [skip-sweep]
Signed-off-by: Indrajit Bhosale <iamindrajitb@gmail.com>
@indrajit96
indrajit96 requested a review from a team as a code owner November 5, 2026 18:12
@github-actions github-actions Bot added the deployment::k8s Relates to dynamo deployment in kubernetes label Nov 5, 2026
@nv-nmailhot

Copy link
Copy Markdown
Contributor

approving pending sglang license fixes

Signed-off-by: Indrajit Bhosale <iamindrajitb@gmail.com>
Signed-off-by: Indrajit Bhosale <iamindrajitb@gmail.com>
Signed-off-by: Indrajit Bhosale <iamindrajitb@gmail.com>
@nv-nmailhot
nv-nmailhot merged commit 06e34ad into release/1.7.0-kimi-k3.1-pro-dev.1 Nov 6, 2026
118 of 125 checks passed
@nv-nmailhot
nv-nmailhot deleted the ibhosale/kimi-k31-release branch November 6, 2026 06:58
cquil11 added a commit to SemiAnalysisAI/InferenceX that referenced this pull request Nov 6, 2026
* feat(kimik3.1): add day-0 vLLM AgentX on GB200

Aggregated TP8 x PP2 x DCP8 and disaggregated 1P1D TP16 Kimi K3.1 Pro
points on gb200-nv with the K3.1 DSpark draft and Simple CPU KV offload,
plus the lane's shared run root and AgentX power rule.

* chore(kimik3.1): link the GB200 vLLM changelog entry to its PR

* fix(kimik3.1): keep only documented or common recipe settings

Remove settings that no active recipe uses and vLLM's K3.1 recipe does not document: the DCP comm backend, prompt-token details, and the FlashInfer all-reduce, execute-timeout, UCX and AIPerf failure-threshold env vars.

* fix(kimik3.1): pin Dynamo #18124 and use the MXFP4 indexer on GB200 vLLM PD

Published Dynamo wheels cannot load the kimi_k31 tokenizer, so the PD frontend 404s. Pin ai-dynamo/dynamo#18124, which pulls the K3.1 frontend crates, and record it as the router version. The PD roles also use the MXFP4 indexer cache, which the week-1 rules allow and vLLM's K3.1 DSA path supports with NIXL.

* fix(kimik3.1): front the GB200 vLLM PD row with the vLLM Router

No published Dynamo frontend loads the kimi_k31 tokenizer and the K3.1
Dynamo branch does not build, so the 1P1D row runs direct vLLM workers
behind vllm-router 0.1.14 with the same engine arguments, NIXL transfer and
MXFP4 indexer. The GB200 Kimi-K3.1 AgentX power rule now also covers vllm.

* fix(kimik3.1): label the GB200 vLLM rows as vllm

Neither GB200 row runs Dynamo: the aggregated row uses the direct vLLM
frontend and the PD row the vLLM Router. Name both keys and their framework
vllm, drop the Dynamo-only recipe settings, and scope the Kimi-K3.1 AgentX
power rule to vllm.

* fix(kimik3.1): keep GB200 vLLM PD worker connections alive for the router

vllm-router reuses pooled connections that Uvicorn closes after its 5 s
default keep-alive, and with retries disabled the reset aborts the request.
Hold PD worker connections for 600 s, as main's GB200 vLLM PD recipe does.

* fix(synthetic-acceptance): drop Kimi vLLM parsers on synthetic throughput servers

Synthetic acceptance streams unverified draft tokens. The kimi_k3 tool and reasoning parsers stop streaming on some of them, which empties or truncates the response the client sees. Evals and acceptance gates verify for real and keep their parsers.

* fix(kimik3.1-gb200-vllm): run disagg decode with expert parallelism

Decode at TP16/EP1 holds 131.5 GiB/GPU of weights and leaves 12-19 GiB of KV,
less than the 18.17 GiB one 1M-token request needs at c32 and c48. EP16, as on
prefill, cuts decode weights to about 103.5 GiB/GPU.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

actions backend::sglang Relates to the sglang backend backend::vllm Relates to the vllm backend container deployment::k8s Relates to dynamo deployment in kubernetes documentation Improvements or additions to documentation feat frontend `python -m dynamo.frontend` and `dynamo-run in=http|text|grpc` multimodal size/XXL

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants