Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
32 commits
Select commit Hold shift + click to select a range
06e3ea7
feat(kimik3.1): add Kimi K3.1 Pro vLLM AgentX on MI355X
cquil11 Nov 4, 2026
de1f23a
chore(kimik3.1): link the MI355X vLLM changelog entry to its PR
cquil11 Nov 4, 2026
bc5c564
fix(kimik3.1): keep only documented or common recipe settings
cquil11 Nov 4, 2026
6a8fc96
Merge remote-tracking branch 'origin/main' into cam/kimik3.1-mi355x-vllm
cquil11 Nov 5, 2026
bda25c5
fix(kimik3.1): leave GPU headroom for pinned scratch on MI355X
cquil11 Nov 5, 2026
aa2bd20
perf(kimik3.1): match the apex-1104 sweep's points and batching
billishyahao Nov 5, 2026
4594456
Merge remote-tracking branch 'origin/main' into cam/kimik3.1-mi355x-vllm
cquil11 Nov 5, 2026
de31629
fix(kimik3.1): resolve the MI355X DRAM budget to the recipe's 2577 GB
cquil11 Nov 5, 2026
2821c4b
perf(kimik3.1): use 7 DSpark tokens and the specialized decode at conc 1
billishyahao Nov 5, 2026
24f26ac
perf(kimik3.1): prefill with the AITER FA MLA backend at conc 48 only
billishyahao Nov 5, 2026
ba19e2a
perf(kimik3.1): set breakable cudagraphs and scratch reclaim per point
billishyahao Nov 5, 2026
f7e7056
Merge remote-tracking branch 'origin/main' into cam/kimik3.1-mi355x-vllm
cquil11 Nov 5, 2026
800b3ca
Merge remote-tracking branch 'origin/main' into cam/kimik3.1-mi355x-vllm
cquil11 Nov 5, 2026
d3f324b
fix(kimik3.1): keep scratch reclaim off on the MI355X DRAM points
cquil11 Nov 5, 2026
3e1771f
perf(kimik3.1): trim the conc-32 and conc-48 points on MI355X
billishyahao Nov 5, 2026
44c11f8
perf(kimik3.1): keep scratch allocated and step the draft count down
billishyahao Nov 5, 2026
fcbdb39
Merge remote-tracking branch 'inf/main' into cam/kimik3.1-mi355x-vllm
billishyahao Nov 5, 2026
641e8f5
Merge branch 'main' into cam/kimik3.1-mi355x-vllm
billishyahao Nov 5, 2026
9d80dc7
Merge branch 'main' into cam/kimik3.1-mi355x-vllm
billishyahao Nov 5, 2026
d843db2
Merge remote-tracking branch 'origin/main' into cam/kimik3.1-mi355x-vllm
cquil11 Nov 5, 2026
a164a99
fix(synthetic-acceptance): drop Kimi vLLM parsers on synthetic throug…
cquil11 Nov 5, 2026
db2754b
Merge remote-tracking branch 'origin/main' into cam/kimik3.1-mi355x-vllm
cquil11 Nov 5, 2026
18303be
fix(kimik3.1-mi355x-vllm): lower gpu-memory-utilization to 0.86 at co…
cquil11 Nov 5, 2026
33d17a1
Merge remote-tracking branch 'origin/main' into cam/kimik3.1-mi355x-vllm
cquil11 Nov 6, 2026
7596fb6
fix
billishyahao Nov 6, 2026
6cdc164
Merge branch 'main' into cam/kimik3.1-mi355x-vllm
billishyahao Nov 6, 2026
14a3dc8
fix
billishyahao Nov 6, 2026
e1beac3
perf(kimik3.1): refresh the day-0 image and map the offload pool with…
billishyahao Nov 6, 2026
740ee08
Merge branch 'main' into cam/kimik3.1-mi355x-vllm
billishyahao Nov 6, 2026
cf9f666
docs(changelog): sync Kimi-K3.1-Pro MI355X vLLM block with the diff […
seungrokj Nov 6, 2026
312a967
Merge branch 'main' into cam/kimik3.1-mi355x-vllm [skip-sweep]
seungrokj Nov 6, 2026
b1c4fce
chore: refresh PR #4806 for sweep reuse [skip-sweep]
cquil11 Nov 6, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,198 @@
base:
schema: 2
name: kimik3.1-fp4-mi355x-vllm-agentic
resources:
gpu_type: mi355x
gpus_per_node: 8
frontend:
type: vllm
enable_multiple_frontends: false
observability:
enabled: false
tachometer:
enabled: false
engine:
type: vllm
connector: null
# A node's first run downloads ~1.56 TB of MXFP4 shards into its local HF cache before loading.
health_check:
interval_seconds: 10
max_attempts: 720
Comment on lines +17 to +20

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 On every MI355X node without the ~1.56 TB checkpoint already cached, the first K3.1 point can fail its readiness check. The download, load and graph capture all fit in the 7200 s budget only if the node downloads at about 217 MB/s or faster. Nothing routes this model to the shared cache. The budget is max_attempts: 720 x 10 s, matching VLLM_ENGINE_READY_TIMEOUT_S: '7200'. Fix: put the checkpoint where each node does not re-download it before serving. For example, add moonshotai/Kimi-K3.1-* to SHARED_HF_CACHE_LANES["mi355x-amds"], or stage a models.entries row. Alternatively, size the readiness budget for a 1.56 TB cold download.

Why this was flagged

The trigger is a sweep point landing on an mi355x-amds node whose node-local /var/lib/hf-hub-cache (runners.yaml:789) does not yet hold Kimi-K3.1-Pro. Every node starts that way, because no models.entries row exists for this model on mi355x-amds (runners.yaml:776-780). single_node_hf_cache (infx/launch/drivers/srt/models.py:198-201) only picks shared-hf-hub-cache for models matched in SHARED_HF_CACHE_LANES, and models.py:187-194 has no Kimi rule. As a result, vLLM pulls about 1.56 TB from HF inside server startup. The sibling MI355X recipe uses the same 720 x 10 s health check and notes at minimaxm31flash/vllm/mi355x-fp4/agentic.yaml:26-27 that a cold load of only 240 GB can use up the whole VLLM_ENGINE_READY_TIMEOUT_S. On a slow link, the first point on each node exceeds 7200 s and the job fails. Under full-sweep-fail-fast, that one failure cancels the sweep. The 10 points spread across nodes each pay the download separately. The recipe's comment at line 17 acknowledges the download, but nothing bounds or avoids it.

Verification: The readiness budget is 7200 s: agentic.yaml:18-20 sets interval_seconds: 10 and max_attempts: 720. No shared-cache rule matches moonshotai/Kimi-K3.1-Pro (models.py:187-194), so it falls through to node-local /var/lib/hf-hub-cache (runners.yaml:789), and vLLM downloads it during startup. 1.56 TB / 7200 s ≈ 217 MB/s. A node below that rate fails its first point.

roles:
agg:
nodes: 1
workers: 1
gpus: 8
args:
served-model-name: moonshotai/Kimi-K3.1-Pro
tensor-parallel-size: 8
decode-context-parallel-size: 8
dcp-comm-backend: a2a
cp-kv-cache-interleave-size: 1
trust-remote-code: true
moe-backend: auto
seed: 7
language-model-only: true
load-format: fastsafetensors
enable-auto-tool-choice: true
tool-call-parser: kimi_k3
reasoning-parser: kimi_k3
max-model-len: 1048576
enable-prefix-caching: true
prefix-match-unit: 64
kv-cache-dtype: fp8
# The K3.1 week-1 precision policy requires an FP32 KDA recurrent state.
mamba-ssm-cache-dtype: float32
# The policy's FP4 indexer cache; the image's AITER pads the FP4 indexer pages it needs.
attention-config: '{"indexer_kv_dtype":"mxfp4"}'
additional-config: '{"kda_prefill_backend":"fused"}'
# Prefix-cached prefill chunks end on K3.1's 1152-token packed DSA block, so the budget floors to a multiple of 1152.
max-num-batched-tokens: 16384
# Kernel scratch, up to HSA_SCRATCH_SINGLE_LIMIT_ASYNC per dispatch, is allocated outside this budget, so leave headroom.
gpu-memory-utilization: 0.90
stream-interval: 10
disable-uvicorn-access-log: true
# 224.875 GB per rank; TOTAL_CPU_DRAM_GB (1799) split across the 8 TP ranks.
kv-transfer-config: '{"kv_connector":"SimpleCPUOffloadConnector","kv_role":"kv_both","kv_connector_extra_config":{"cpu_bytes_to_use_per_rank":224875000000,"lazy_offload":false}}'
env:
VLLM_ROCM_USE_AITER: '1'
VLLM_ROCM_USE_AITER_MOE_SITUV2: '1'
SAFETENSORS_FAST_GPU: '1'
VLLM_USE_V2_MODEL_RUNNER: '1'
VLLM_FASTSAFETENSORS_QUEUE_SIZE: '-1'
# K3.1 has no torch.compile submodule, so only breakable graphs capture the prefill
# sizes the throughput points list. The conc-1 override turns this off.
VLLM_USE_BREAKABLE_CUDAGRAPH: '1'
VLLM_ROCM_AITER_MLA_ASM_PADDING: asm
# Verify the draft tokens with the AITER assembly DCP kernel.
VLLM_ROCM_AITER_MLA_DCP_VERIFY: asm
VLLM_ROCM_DCP_AITER_ALLGATHER: '1'
VLLM_ROCM_USE_AITER_FLYDSL_SPARSE_MLA: '1'
# AITER reads its own variable for the same kernels.
AITER_FLYDSL_SPARSE_MLA: '1'
VLLM_ROCM_DSA_DCP_GLOBAL_KEY_PREFILL: '1'
VLLM_ROCM_DSA_DCP_ROWSHARD_PREFILL: '1'
VLLM_USE_DIRECT_DCP_A2A: '0'
VLLM_USE_DIRECT_DCP_Q_GATHER: '0'
VLLM_USE_DIRECT_DCP_KV_GATHER: '0'
# The K3.1 policy and MODELS.md keep all-reduce in BF16, so quick reduce must not quantize.
VLLM_ROCM_QUICK_REDUCE_QUANTIZATION: NONE
AITER_FLYDSL_STAGE2_FP8: '0'
AITER_BF16_FP8_MOE_BOUND: '0'
HSA_NO_SCRATCH_RECLAIM: '1'
# Back the offload pool with transparent huge pages. Torch's CPU allocator
# only calls madvise(MADV_HUGEPAGE) when this is set, and the host is in
# madvise mode, so without it the pool is all 4 KiB pages.
THP_MEM_ALLOC_ENABLE: '1'
VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS: '1200'
PYTHONNOUSERSITE: '1'
VLLM_ENGINE_READY_TIMEOUT_S: '7200'
# Identical prefixes must hash to identical block keys run-to-run.
PYTHONHASHSEED: '42'
benchmark:
env:
AIPERF_HTTP_TCP_USER_TIMEOUT: '900000'
AIPERF_REQUIRED_SERVER_METRIC_PREFIX: 'vllm:'

# max-num-seqs stays below CONC from c4 to c32, and equals it at c1 and c48. The capture list is
# the decode sizes, multiples of one plus the draft token count, then the prefill ladder.
# Throughput runs replace block rejection with the golden acceptance length.
override_gpu_kv_c1:
roles:
agg:
args:
max-num-seqs: 1
compilation-config: '{"mode":3,"cudagraph_mode":"FULL_AND_PIECEWISE","max_cudagraph_capture_size":16,"custom_ops":["+fused_rms_norm_gated"],"cudagraph_capture_sizes":[2,4,6,8,10,12,14,16]}'
additional-config: '{"decode_backend":"latency_specialized","kda_prefill_backend":"fused"}'
speculative-config: '{"method":"dspark","model":"Inferact/Kimi-K3.1-Pro-DSpark-preview","num_speculative_tokens":7,"attention_backend":"ROCM_AITER_MLA","kv_cache_dtype":"fp8","draft_sample_method":"probabilistic","rejection_sample_method":"block"}'
kv-transfer-config: null
env:
# This point captures only the decode sizes, so breakable graphs buy nothing.
VLLM_USE_BREAKABLE_CUDAGRAPH: '0'
benchmark:
env:
CONC: '1'
KV_OFFLOADING: 'none'

override_dram_c4:
roles:
agg:
args:
max-num-seqs: 2
compilation-config: '{"mode":3,"cudagraph_mode":"FULL_AND_PIECEWISE","max_cudagraph_capture_size":16384,"custom_ops":["+fused_rms_norm_gated"],"cudagraph_capture_sizes":[6,12,16,24,32,48,64,96,128,160,192,224,256,288,320,352,384,416,448,480,512,640,768,896,1024,1280,1536,1792,2048,2560,3072,3584,4096,4608,5120,6144,6400,7168,8192,8704,10240,12288,14336,16384]}'
speculative-config: '{"method":"dspark","model":"Inferact/Kimi-K3.1-Pro-DSpark-preview","num_speculative_tokens":5,"attention_backend":"ROCM_AITER_MLA","kv_cache_dtype":"fp8","draft_sample_method":"probabilistic","rejection_sample_method":"block"}'
benchmark:
env:
CONC: '4'
KV_OFFLOADING: 'dram'
TOTAL_CPU_DRAM_GB: '1799'

override_dram_c10:
roles:
agg:
args:
max-num-seqs: 6
compilation-config: '{"mode":3,"cudagraph_mode":"FULL_AND_PIECEWISE","max_cudagraph_capture_size":16384,"custom_ops":["+fused_rms_norm_gated"],"cudagraph_capture_sizes":[6,12,18,24,30,36,48,64,96,128,160,192,224,256,288,320,352,384,416,448,480,512,640,768,896,1024,1280,1536,1792,2048,2560,3072,3584,4096,4608,5120,6144,6400,7168,8192,8704,10240,12288,14336,16384]}'
speculative-config: '{"method":"dspark","model":"Inferact/Kimi-K3.1-Pro-DSpark-preview","num_speculative_tokens":5,"attention_backend":"ROCM_AITER_MLA","kv_cache_dtype":"fp8","draft_sample_method":"probabilistic","rejection_sample_method":"block"}'
benchmark:
env:
CONC: '10'
KV_OFFLOADING: 'dram'
TOTAL_CPU_DRAM_GB: '1799'

override_dram_c16:
roles:
agg:
args:
max-num-seqs: 10
compilation-config: '{"mode":3,"cudagraph_mode":"FULL_AND_PIECEWISE","max_cudagraph_capture_size":16384,"custom_ops":["+fused_rms_norm_gated"],"cudagraph_capture_sizes":[5,10,15,20,25,30,35,40,45,50,64,96,128,160,192,224,256,288,320,352,384,416,448,480,512,640,768,896,1024,1280,1536,1792,2048,2560,3072,3584,4096,4608,5120,6144,6400,7168,8192,8704,10240,12288,14336,16384]}'
speculative-config: '{"method":"dspark","model":"Inferact/Kimi-K3.1-Pro-DSpark-preview","num_speculative_tokens":4,"attention_backend":"ROCM_AITER_MLA","kv_cache_dtype":"fp8","draft_sample_method":"probabilistic","rejection_sample_method":"block"}'
benchmark:
env:
CONC: '16'
KV_OFFLOADING: 'dram'
TOTAL_CPU_DRAM_GB: '1799'

override_dram_c24:
roles:
agg:
args:
max-num-seqs: 16
compilation-config: '{"mode":3,"cudagraph_mode":"FULL_AND_PIECEWISE","max_cudagraph_capture_size":16384,"custom_ops":["+fused_rms_norm_gated"],"cudagraph_capture_sizes":[5,10,15,20,25,30,35,40,45,50,55,60,65,70,75,80,96,128,160,192,224,256,288,320,352,384,416,448,480,512,640,768,896,1024,1280,1536,1792,2048,2560,3072,3584,4096,4608,5120,6144,6400,7168,8192,8704,10240,12288,14336,16384]}'
speculative-config: '{"method":"dspark","model":"Inferact/Kimi-K3.1-Pro-DSpark-preview","num_speculative_tokens":4,"attention_backend":"ROCM_AITER_MLA","kv_cache_dtype":"fp8","draft_sample_method":"probabilistic","rejection_sample_method":"block"}'
benchmark:
env:
CONC: '24'
KV_OFFLOADING: 'dram'
TOTAL_CPU_DRAM_GB: '1799'

override_dram_c32:
roles:
agg:
args:
max-num-seqs: 24
max-num-batched-tokens: 10240
gpu-memory-utilization: 0.91
compilation-config: '{"mode":3,"cudagraph_mode":"FULL_AND_PIECEWISE","max_cudagraph_capture_size":10240,"custom_ops":["+fused_rms_norm_gated"],"cudagraph_capture_sizes":[4,8,12,16,20,24,28,32,36,40,44,48,52,56,60,64,68,72,76,80,84,88,92,96,128,160,192,224,256,288,320,352,384,416,448,480,512,640,768,896,1024,1280,1536,1792,2048,2560,3072,3584,4096,4608,5120,6144,6400,7168,8192,8704,10240]}'
speculative-config: '{"method":"dspark","model":"Inferact/Kimi-K3.1-Pro-DSpark-preview","num_speculative_tokens":3,"attention_backend":"ROCM_AITER_MLA","kv_cache_dtype":"fp8","draft_sample_method":"probabilistic","rejection_sample_method":"block"}'
benchmark:
env:
CONC: '32'
KV_OFFLOADING: 'dram'
TOTAL_CPU_DRAM_GB: '1799'

override_dram_c48:
roles:
agg:
args:
max-num-seqs: 48
max-num-batched-tokens: 10240
gpu-memory-utilization: 0.91
compilation-config: '{"mode":3,"cudagraph_mode":"FULL_AND_PIECEWISE","max_cudagraph_capture_size":10240,"custom_ops":["+fused_rms_norm_gated"],"cudagraph_capture_sizes":[4,8,12,16,20,24,28,32,36,40,44,48,52,56,60,64,68,72,76,80,84,88,92,96,100,104,108,112,116,120,124,128,132,136,140,144,148,152,156,160,164,168,172,176,180,184,188,192,224,256,288,320,352,384,416,448,480,512,640,768,896,1024,1280,1536,1792,2048,2560,3072,3584,4096,4608,5120,6144,6400,7168,8192,8704,10240]}'
speculative-config: '{"method":"dspark","model":"Inferact/Kimi-K3.1-Pro-DSpark-preview","num_speculative_tokens":3,"attention_backend":"ROCM_AITER_MLA","kv_cache_dtype":"fp8","draft_sample_method":"probabilistic","rejection_sample_method":"block"}'
benchmark:
env:
CONC: '48'
KV_OFFLOADING: 'dram'
TOTAL_CPU_DRAM_GB: '1799'

18 changes: 18 additions & 0 deletions inferencex-e2e/configs/amd-master.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -926,6 +926,24 @@ minimaxm31flash-fp4-mi355x-vllm-agentic-dspark:
- { tp: 4, kv-offloading: none, spec-decoding: mtp, conc-list: [1, 2, 4, 8, 16, 32], srt-recipe: agentic.yaml }
- { tp: 4, kv-offloading: dram, kv-offload-backend: { name: vllm-simple }, spec-decoding: mtp, conc-list: [64], srt-recipe: agentic.yaml:override_tp4_dspark3_c64_offload }

kimik3.1-fp4-mi355x-vllm-agentic-dspark:
image: vllm/vllm-openai-rocm:kimi-k31@sha256:5f47cc1004d9a7ac65f8a913725edb20c62c99a07e31cbf4ba21741bd0b4f9b1
model: moonshotai/Kimi-K3.1-Pro
model-prefix: kimik3.1
runner: cluster:mi355x-amds
precision: fp4
framework: vllm
multinode: false
srt-recipe-dir: kimik3.1/vllm/mi355x-fp4-mtp
scenarios:
agentic-coding:
# 0.6 resolves to 1799 GB of host DRAM, which the recipe's Simple CPU offload splits across the 8 TP ranks.
- dram-utilization: 0.6
search-space:
# The ~1.56 TB MXFP4 checkpoint needs all 8 GPUs. c1 keeps the KV cache on the GPUs.
- { tp: 8, ep: 1, dcp-size: 8, spec-decoding: mtp, kv-offloading: none, conc-list: [1], srt-recipe: agentic.yaml }
- { tp: 8, ep: 1, dcp-size: 8, spec-decoding: mtp, kv-offloading: dram, kv-offload-backend: { name: vllm-simple }, conc-list: [4, 10, 16, 24, 32, 48], srt-recipe: agentic.yaml }

# MiniMax-M3.1-Flash AgentX on MI355X with SGLang: day-zero image from the SGLang
# cookbook (https://docs.sglang.io/cookbook/autoregressive/MiniMax/MiniMax-M3.1),
# manifest digest verified 2026-10-24. Serve flags follow the cookbook's
Expand Down
15 changes: 15 additions & 0 deletions inferencex-e2e/perf-changelog.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -12035,6 +12035,21 @@
- "Add Kimi-K3.1-Pro day-0 vLLM AgentX on B300: TP8 x DCP8 with DSpark (Inferact/Kimi-K3.1-Pro-DSpark-preview) and Simple CPU KV offload, conc 1-70."
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/4796

- config-keys:
- kimik3.1-fp4-mi355x-vllm-agentic-dspark
scenario-type:
- agentic-coding
description:
- "Add Kimi-K3.1-Pro vLLM AgentX on one 8x MI355X node with vLLM's day-0 image vllm/vllm-openai-rocm:kimi-k31@sha256:5f47cc1004d9a7ac65f8a913725edb20c62c99a07e31cbf4ba21741bd0b4f9b1 (vLLM 0.0.0+mum.14d1ec2336, AITER 52846bc2)."
- "TP8 x DCP8 (a2a DCP backend) with the Inferact/Kimi-K3.1-Pro-DSpark-preview draft: GPU-resident KV at concurrency 1, and SimpleCPUOffloadConnector DRAM offload at concurrency 4, 10, 16, 24, 32 and 48 (224.875 GB per rank, 1799 GB per node, which is what dram-utilization 0.6 resolves to)."
- "Per point at concurrency 1/4/10/16/24/32/48: max-num-seqs 1/2/6/10/16/24/48 and DSpark draft tokens 7/5/5/4/4/3/3. max-num-batched-tokens is 16384 and gpu-memory-utilization 0.90 through concurrency 24; both change to 10240 and 0.91 at concurrency 32 and 48."
- "Every point compiles in mode 3 with FULL_AND_PIECEWISE CUDA graphs. Concurrency 1 runs the latency_specialized decode backend, captures sizes 2-16 and turns breakable CUDA graphs off; the offload points keep VLLM_USE_BREAKABLE_CUDAGRAPH=1 and capture max-num-seqs multiples of (1 + draft tokens) plus a prefill ladder up to max-num-batched-tokens."
- "Common settings: FP8 KV cache, FP32 KDA state (mamba-ssm-cache-dtype float32), MXFP4 indexer cache (indexer_kv_dtype mxfp4), fused KDA prefill, prefix caching with prefix-match-unit 64, max-model-len 1048576, fastsafetensors loading, stream-interval 10, AITER ASM DCP verify, AITER DCP allgather, FlyDSL sparse MLA, the image's HSA_NO_SCRATCH_RECLAIM=1, transparent huge pages for the offload pool (THP_MEM_ALLOC_ENABLE=1), and 1200 s execute-model and 7200 s engine-ready timeouts."
- "K3.1 week-1 precision policy: linear layers and shared experts stay BF16 (no quantization-config); routed experts run A4W4, MXFP4 weights with FP4 activations, through the SiTUv2 FlyDSL MoE (VLLM_ROCM_USE_AITER_MOE_SITUV2=1, the same as the image's auto default); MoE stage 2 stays BF16 (AITER_FLYDSL_STAGE2_FP8=0); all-reduce stays BF16 (VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=NONE)."
- "Verification is block rejection with probabilistic draft sampling. Throughput runs replace it with the golden acceptance length for the draft length: 3.69 at 7 tokens, 3.41 at 5, 3.19 at 4 and 2.88 at 3. Evals and AL gates verify for real."
- "No change to the DSpark draft's precision: Inferact/Kimi-K3.1-Pro-DSpark-preview (revision 6a2ef20a) ships all-BF16 weights with no quantization_config, the speculative config sets no quantization, so get_draft_quant_config() returns None and K3DSparkModel loads its layers in BF16 with the target's BF16 embedding and LM head. Its KV cache is FP8, the same as the target's."
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/4806

- config-keys:
- kimik3.1-fp4-mi355x-sglang-agentic-dflash
scenario-type:
Expand Down
Loading