Repository navigation
[K3.1-Pro] Day-0 support SGLang on GB200 #4804
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
15 commits
Select commit
Hold shift + click to select a range
1c1bed1
feat(kimik3.1): add Kimi K3.1 Pro GB200 SGLang AgentX configs
cquil11 07b5d3c
chore(kimik3.1): link the GB200 SGLang changelog entry to its PR
cquil11 3f7acfd
fix(kimik3.1): keep only documented or common recipe settings
cquil11 285a068
fix(kimik3.1): install Dynamo #18124 for the GB200 SGLang frontend
cquil11 8df7beb
Merge remote-tracking branch 'origin/main' into cam/kimik3.1-gb200-sg…
cquil11 e5f7507
fix(kimik3.1): say the GB200 SGLang image ships no Dynamo
cquil11 bfa2d14
feat(kimik3.1): front the GB200 SGLang workers with the SGLang router
cquil11 f07f9f1
Merge remote-tracking branch 'origin/main' into cam/kimik3.1-gb200-sg…
cquil11 d186491
fix(kimik3.1): use SGLang's replicated K3 MoE front at TP16 on GB200
cquil11 dec30fb
feat(kimik3.1): serve the GB200 SGLang points through the image's Dynamo
cquil11 5832e0b
Merge remote-tracking branch 'origin/main' into cam/kimik3.1-gb200-sg…
cquil11 11f2c4f
fix(kimik3.1): read the GB200 DFLASH draft from the shared Hub cache
cquil11 9f29be4
Merge remote-tracking branch 'origin/main' into cam/kimik3.1-gb200-sg…
cquil11 7d0925f
Merge remote-tracking branch 'origin/main' into cam/kimik3.1-gb200-sg…
cquil11 298f44c
chore: refresh PR #4804 for sweep reuse [skip-sweep]
cquil11 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
191 changes: 191 additions & 0 deletions
191
...enchmarks/multi_node/srt-slurm-recipes/kimik3.1/sglang/gb200-fp4/agentx/agg-variants.yaml
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,191 @@ | ||
| schema: 2 | ||
|
|
||
| base: | ||
| name: kimik3.1-fp4-gb200-dynamo-sglang-agentx-agg | ||
| identity: | ||
| model: {} | ||
| container: {} | ||
| frameworks: | ||
| dynamo: "1.6.0" | ||
| sglang: "0.0.0.dev1+ge79de62d2.d20261104" | ||
| resources: | ||
| gpu_type: gb200 | ||
| gpus_per_node: 4 | ||
| # The K3.1 image's Dynamo 1.6.0 lets Kimi-K3.1 decode with DCP on its DSA backend. | ||
| dynamo: | ||
| install: false | ||
| request_plane: tcp | ||
| event_plane: zmq | ||
| # The DFlash2 draft is read from the shared Hub cache. | ||
| environment: | ||
| HF_HUB_CACHE: /hf_hub_cache | ||
| # Tachometer's per-node exporters slow decode steps and add host memory; | ||
| # InferenceX does not read its output. | ||
| observability: | ||
| tachometer: | ||
| enabled: false | ||
| frontend: | ||
| type: dynamo | ||
| enable_multiple_frontends: false | ||
| env: | ||
| DYN_ROUTER_TEMPERATURE: '0' | ||
| DYN_ROUTER_QUEUE_THRESHOLD: None | ||
| DYN_ROUTER_ACTIVE_REQUEST_EXPIRY_SECS: '3600' | ||
| DYN_TCP_CHANNEL_BUFFER: '128' | ||
| DYN_TCP_REQUEST_TIMEOUT: '60' | ||
| DYN_TOKENIZER_CACHE: '1' | ||
| DYN_TOKENIZER_CACHE_BYTES: '8000000000' | ||
| args: | ||
| router-mode: kv | ||
| # The frontend's own tokenizer cannot load kimi_k31; SGLang's chat processor renders requests. | ||
| dyn-chat-processor: sglang | ||
| dyn-preprocess-workers: '4' | ||
| trust-remote-code: true | ||
| active-decode-blocks-threshold: None | ||
| active-prefill-tokens-threshold: None | ||
| active-prefill-tokens-threshold-frac: None | ||
| engine: sglang | ||
| roles: | ||
| agg: | ||
| nodes: 4 | ||
| workers: 1 | ||
| gpus: 16 | ||
| env: | ||
| TORCH_DISTRIBUTED_DEFAULT_TIMEOUT: '1800' | ||
| PYTHONUNBUFFERED: '1' | ||
| PYTHONNOUSERSITE: '1' | ||
| PIP_BREAK_SYSTEM_PACKAGES: '1' | ||
| SGLANG_TIMEOUT_KEEP_ALIVE: '900' | ||
| # TP16 spans four trays; the cookbook's GB200 cell disables the per-rank memory check | ||
| # and turns on multi-node NVLink. | ||
| SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK: '0' | ||
| NCCL_MNNVL_ENABLE: '1' | ||
| NCCL_CUMEM_ENABLE: '1' | ||
| # SGLang's K3 sharded MoE front has no output tile for TP16's N=1048 (k3_ar_fusion); this selects the engine's replicated fused front. | ||
| SGLANG_K3_SHARDED_MOE: '0' | ||
| args: | ||
| served-model-name: moonshotai/Kimi-K3.1-Pro | ||
| trust-remote-code: true | ||
| tensor-parallel-size: 16 | ||
| dcp-size: 16 | ||
| expert-parallel-size: 1 | ||
| kv-cache-dtype: fp8_e4m3 | ||
| # Week-1 K3.1 precision policy: FP32 KDA recurrent state. | ||
| mamba-ssm-dtype: float32 | ||
| mem-fraction-static: 0.85 | ||
| chunked-prefill-size: 16384 | ||
| max-prefill-tokens: 16384 | ||
| weight-loader-prefetch-checkpoints: true | ||
| model-loader-extra-config: '{"enable_multithread_load":true}' | ||
| dyn-reasoning-parser: kimi_k3 | ||
| dyn-tool-call-parser: kimi_k3 | ||
| speculative-algorithm: DFLASH | ||
| speculative-draft-model-path: RadixArk/Kimi-K3.1-Dflash2-Preview | ||
| speculative-draft-kv-cache-dtype: fp8_e4m3 | ||
| speculative-draft-attention-backend: cutedsl_mla | ||
| speculative-use-block-verification: true | ||
| # Without ReplaySSM, DFLASH keeps one KDA state per draft token and the KV pool collapses. | ||
| enable-linear-replayssm-spec: true | ||
| enable-hierarchical-cache: true | ||
| hicache-size: 120 | ||
| hicache-io-backend: direct | ||
| hicache-mem-layout: page_first_direct | ||
| hicache-write-policy: write_back | ||
| mamba-max-states-per-path: 1 | ||
| # A session's turns stay on one aggregated worker. | ||
| router-mode: kv | ||
| router-host-cache-hit-weight: 1 | ||
| router-kv-overlap-score-credit: 4 | ||
| router-session-affinity-ttl-secs: '14400' | ||
| watchdog-timeout: 3600 | ||
| enable-metrics: true | ||
| kv_events: true | ||
| health_check: | ||
| max_attempts: 1440 | ||
| interval_seconds: 10 | ||
| benchmark: | ||
| env: | ||
| AIPERF_HTTP_X_DYNAMO_SESSION_ID_FROM_CORRELATION_ID: 'true' | ||
| AIPERF_USE_DYNAMO_CONV_AWARE_ROUTING: '0' | ||
| AIPERF_REQUIRED_SERVER_METRIC_PREFIX: 'sglang:' | ||
| AIPERF_HTTP_TCP_USER_TIMEOUT: '900000' | ||
| services: | ||
| - name: etcd | ||
| type: etcd | ||
| placement: | ||
| node: infra | ||
| # Disables Dynamo's per-request lifecycle span export; InferenceX does not read it. | ||
| - name: otel-collector | ||
| type: otel-collector | ||
| enabled: false | ||
| sbatch_directives: | ||
| mem: '0' | ||
| cpus-per-task: '144' | ||
| srun_options: | ||
| mem: '0' | ||
| container-remap-root: '' | ||
|
|
||
| override_tp16_c1: | ||
| name: kimik3.1-fp4-gb200-dynamo-sglang-agentx-agg-tp16-c1-dflash7 | ||
| roles: | ||
| agg: | ||
| args: | ||
| speculative-num-draft-tokens: 8 | ||
| max-running-requests: 2 | ||
| cuda-graph-max-bs-decode: 2 | ||
|
|
||
| override_tp16_c4: | ||
| name: kimik3.1-fp4-gb200-dynamo-sglang-agentx-agg-tp16-c4-dflash7 | ||
| roles: | ||
| agg: | ||
| args: | ||
| speculative-num-draft-tokens: 8 | ||
| max-running-requests: 8 | ||
| cuda-graph-max-bs-decode: 8 | ||
|
|
||
| override_tp16_c8: | ||
| name: kimik3.1-fp4-gb200-dynamo-sglang-agentx-agg-tp16-c8-dflash7 | ||
| roles: | ||
| agg: | ||
| args: | ||
| speculative-num-draft-tokens: 8 | ||
| max-running-requests: 16 | ||
| cuda-graph-max-bs-decode: 16 | ||
|
|
||
| override_tp16_c24: | ||
| name: kimik3.1-fp4-gb200-dynamo-sglang-agentx-agg-tp16-c24-dflash5 | ||
| roles: | ||
| agg: | ||
| args: | ||
| # Above CONC 8 the verify rows dominate a step: 5 drafts (width 6), the cookbook's | ||
| # throughput setting. | ||
| speculative-num-draft-tokens: 6 | ||
| max-running-requests: 48 | ||
| cuda-graph-max-bs-decode: 48 | ||
|
|
||
| override_tp16_c32: | ||
| name: kimik3.1-fp4-gb200-dynamo-sglang-agentx-agg-tp16-c32-dflash5 | ||
| roles: | ||
| agg: | ||
| args: | ||
| speculative-num-draft-tokens: 6 | ||
| max-running-requests: 64 | ||
| cuda-graph-max-bs-decode: 64 | ||
|
|
||
| override_tp16_c48: | ||
| name: kimik3.1-fp4-gb200-dynamo-sglang-agentx-agg-tp16-c48-dflash5 | ||
| roles: | ||
| agg: | ||
| args: | ||
| speculative-num-draft-tokens: 6 | ||
| max-running-requests: 96 | ||
| cuda-graph-max-bs-decode: 96 | ||
|
|
||
| override_tp16_c56: | ||
| name: kimik3.1-fp4-gb200-dynamo-sglang-agentx-agg-tp16-c56-dflash5 | ||
| roles: | ||
| agg: | ||
| args: | ||
| speculative-num-draft-tokens: 6 | ||
| max-running-requests: 112 | ||
| cuda-graph-max-bs-decode: 112 | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🔴 Kimi K3.1 GB200 worker startups can fail with Hub 429 errors, because every rank fetches the DFLASH draft from Hugging Face at launch. The roles name
RadixArk/Kimi-K3.1-Dflash2-Previewbut set neitherHF_HUB_OFFLINEnorHF_HUB_CACHE. Muse Spark DFlash2 recipes added both after concurrent launches hit the Hub 429 limit. Fix: give every speculating roleHF_HUB_OFFLINE: '1'andHF_HUB_CACHE: /hf_hub_cache, and seed the draft in gb200-nv's shared hf-hub-cache; this covers the 3 sites listed. Same pattern at 3 sites (agg-variants.yaml:68, disagg-variants.yaml:75, disagg-variants.yaml:128).Why this was flagged
The trigger is a sweep that launches several of the 10 new points at once (7 agg and 3 disagg), or gb200-nv compute nodes with no Hub access. With no
HF_HUB_OFFLINE, every rank resolvesspeculative-draft-model-path: RadixArk/Kimi-K3.1-Dflash2-Preview(agg-variants.yaml:68, disagg-variants.yaml:75 and :128) through Hub API calls. The role env blocks set noHF_HUB_CACHE. As a result the draft downloads into each container's default cache, not the shared/hf_hub_cachemount that gb200-nv'sagentic-volume-mountsprovides (runners.yaml:623-625). The launcher passesHF_HUB_CACHEonly to the benchmark client (config.py:236-252). perf-changelog.yaml:10160-10205 records that the Muse Spark DFlash2 points neededHF_HUB_OFFLINE=1because concurrent launches otherwise hit the Hub 429 API rate limit. Neither safeguard applies to these new recipes, so a rate-limited or offline rank fails worker startup and the point produces no result.Verification: Every speculating role names a Hub repo id as its draft:
speculative-draft-model-path: RadixArk/Kimi-K3.1-Dflash2-Preview(agg-variants.yaml:68, disagg-variants.yaml:75 and :128). None of the roleenvblocks setsHF_HUB_OFFLINEorHF_HUB_CACHE, and neither file has a baseenvironment:block. The srt driver does not add them for workers either.