Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,191 @@
schema: 2

base:
name: kimik3.1-fp4-gb200-dynamo-sglang-agentx-agg
identity:
model: {}
container: {}
frameworks:
dynamo: "1.6.0"
sglang: "0.0.0.dev1+ge79de62d2.d20261104"
resources:
gpu_type: gb200
gpus_per_node: 4
# The K3.1 image's Dynamo 1.6.0 lets Kimi-K3.1 decode with DCP on its DSA backend.
dynamo:
install: false
request_plane: tcp
event_plane: zmq
# The DFlash2 draft is read from the shared Hub cache.
environment:
HF_HUB_CACHE: /hf_hub_cache
# Tachometer's per-node exporters slow decode steps and add host memory;
# InferenceX does not read its output.
observability:
tachometer:
enabled: false
frontend:
type: dynamo
enable_multiple_frontends: false
env:
DYN_ROUTER_TEMPERATURE: '0'
DYN_ROUTER_QUEUE_THRESHOLD: None
DYN_ROUTER_ACTIVE_REQUEST_EXPIRY_SECS: '3600'
DYN_TCP_CHANNEL_BUFFER: '128'
DYN_TCP_REQUEST_TIMEOUT: '60'
DYN_TOKENIZER_CACHE: '1'
DYN_TOKENIZER_CACHE_BYTES: '8000000000'
args:
router-mode: kv
# The frontend's own tokenizer cannot load kimi_k31; SGLang's chat processor renders requests.
dyn-chat-processor: sglang
dyn-preprocess-workers: '4'
trust-remote-code: true
active-decode-blocks-threshold: None
active-prefill-tokens-threshold: None
active-prefill-tokens-threshold-frac: None
engine: sglang
roles:
agg:
nodes: 4
workers: 1
gpus: 16
env:
TORCH_DISTRIBUTED_DEFAULT_TIMEOUT: '1800'
PYTHONUNBUFFERED: '1'
PYTHONNOUSERSITE: '1'
PIP_BREAK_SYSTEM_PACKAGES: '1'
SGLANG_TIMEOUT_KEEP_ALIVE: '900'
# TP16 spans four trays; the cookbook's GB200 cell disables the per-rank memory check
# and turns on multi-node NVLink.
SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK: '0'
NCCL_MNNVL_ENABLE: '1'
NCCL_CUMEM_ENABLE: '1'
# SGLang's K3 sharded MoE front has no output tile for TP16's N=1048 (k3_ar_fusion); this selects the engine's replicated fused front.
SGLANG_K3_SHARDED_MOE: '0'
args:
served-model-name: moonshotai/Kimi-K3.1-Pro
trust-remote-code: true
tensor-parallel-size: 16
dcp-size: 16
expert-parallel-size: 1
kv-cache-dtype: fp8_e4m3
# Week-1 K3.1 precision policy: FP32 KDA recurrent state.
mamba-ssm-dtype: float32
mem-fraction-static: 0.85
chunked-prefill-size: 16384
max-prefill-tokens: 16384
weight-loader-prefetch-checkpoints: true
model-loader-extra-config: '{"enable_multithread_load":true}'
dyn-reasoning-parser: kimi_k3
dyn-tool-call-parser: kimi_k3
speculative-algorithm: DFLASH
speculative-draft-model-path: RadixArk/Kimi-K3.1-Dflash2-Preview

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Kimi K3.1 GB200 worker startups can fail with Hub 429 errors, because every rank fetches the DFLASH draft from Hugging Face at launch. The roles name RadixArk/Kimi-K3.1-Dflash2-Preview but set neither HF_HUB_OFFLINE nor HF_HUB_CACHE. Muse Spark DFlash2 recipes added both after concurrent launches hit the Hub 429 limit. Fix: give every speculating role HF_HUB_OFFLINE: '1' and HF_HUB_CACHE: /hf_hub_cache, and seed the draft in gb200-nv's shared hf-hub-cache; this covers the 3 sites listed. Same pattern at 3 sites (agg-variants.yaml:68, disagg-variants.yaml:75, disagg-variants.yaml:128).

Why this was flagged

The trigger is a sweep that launches several of the 10 new points at once (7 agg and 3 disagg), or gb200-nv compute nodes with no Hub access. With no HF_HUB_OFFLINE, every rank resolves speculative-draft-model-path: RadixArk/Kimi-K3.1-Dflash2-Preview (agg-variants.yaml:68, disagg-variants.yaml:75 and :128) through Hub API calls. The role env blocks set no HF_HUB_CACHE. As a result the draft downloads into each container's default cache, not the shared /hf_hub_cache mount that gb200-nv's agentic-volume-mounts provides (runners.yaml:623-625). The launcher passes HF_HUB_CACHE only to the benchmark client (config.py:236-252). perf-changelog.yaml:10160-10205 records that the Muse Spark DFlash2 points needed HF_HUB_OFFLINE=1 because concurrent launches otherwise hit the Hub 429 API rate limit. Neither safeguard applies to these new recipes, so a rate-limited or offline rank fails worker startup and the point produces no result.

Verification: Every speculating role names a Hub repo id as its draft: speculative-draft-model-path: RadixArk/Kimi-K3.1-Dflash2-Preview (agg-variants.yaml:68, disagg-variants.yaml:75 and :128). None of the role env blocks sets HF_HUB_OFFLINE or HF_HUB_CACHE, and neither file has a base environment: block. The srt driver does not add them for workers either.

speculative-draft-kv-cache-dtype: fp8_e4m3
speculative-draft-attention-backend: cutedsl_mla
speculative-use-block-verification: true
# Without ReplaySSM, DFLASH keeps one KDA state per draft token and the KV pool collapses.
enable-linear-replayssm-spec: true
enable-hierarchical-cache: true
hicache-size: 120
hicache-io-backend: direct
hicache-mem-layout: page_first_direct
hicache-write-policy: write_back
mamba-max-states-per-path: 1
# A session's turns stay on one aggregated worker.
router-mode: kv
router-host-cache-hit-weight: 1
router-kv-overlap-score-credit: 4
router-session-affinity-ttl-secs: '14400'
watchdog-timeout: 3600
enable-metrics: true
kv_events: true
health_check:
max_attempts: 1440
interval_seconds: 10
benchmark:
env:
AIPERF_HTTP_X_DYNAMO_SESSION_ID_FROM_CORRELATION_ID: 'true'
AIPERF_USE_DYNAMO_CONV_AWARE_ROUTING: '0'
AIPERF_REQUIRED_SERVER_METRIC_PREFIX: 'sglang:'
AIPERF_HTTP_TCP_USER_TIMEOUT: '900000'
services:
- name: etcd
type: etcd
placement:
node: infra
# Disables Dynamo's per-request lifecycle span export; InferenceX does not read it.
- name: otel-collector
type: otel-collector
enabled: false
sbatch_directives:
mem: '0'
cpus-per-task: '144'
srun_options:
mem: '0'
container-remap-root: ''

override_tp16_c1:
name: kimik3.1-fp4-gb200-dynamo-sglang-agentx-agg-tp16-c1-dflash7
roles:
agg:
args:
speculative-num-draft-tokens: 8
max-running-requests: 2
cuda-graph-max-bs-decode: 2

override_tp16_c4:
name: kimik3.1-fp4-gb200-dynamo-sglang-agentx-agg-tp16-c4-dflash7
roles:
agg:
args:
speculative-num-draft-tokens: 8
max-running-requests: 8
cuda-graph-max-bs-decode: 8

override_tp16_c8:
name: kimik3.1-fp4-gb200-dynamo-sglang-agentx-agg-tp16-c8-dflash7
roles:
agg:
args:
speculative-num-draft-tokens: 8
max-running-requests: 16
cuda-graph-max-bs-decode: 16

override_tp16_c24:
name: kimik3.1-fp4-gb200-dynamo-sglang-agentx-agg-tp16-c24-dflash5
roles:
agg:
args:
# Above CONC 8 the verify rows dominate a step: 5 drafts (width 6), the cookbook's
# throughput setting.
speculative-num-draft-tokens: 6
max-running-requests: 48
cuda-graph-max-bs-decode: 48

override_tp16_c32:
name: kimik3.1-fp4-gb200-dynamo-sglang-agentx-agg-tp16-c32-dflash5
roles:
agg:
args:
speculative-num-draft-tokens: 6
max-running-requests: 64
cuda-graph-max-bs-decode: 64

override_tp16_c48:
name: kimik3.1-fp4-gb200-dynamo-sglang-agentx-agg-tp16-c48-dflash5
roles:
agg:
args:
speculative-num-draft-tokens: 6
max-running-requests: 96
cuda-graph-max-bs-decode: 96

override_tp16_c56:
name: kimik3.1-fp4-gb200-dynamo-sglang-agentx-agg-tp16-c56-dflash5
roles:
agg:
args:
speculative-num-draft-tokens: 6
max-running-requests: 112
cuda-graph-max-bs-decode: 112
Loading
Loading