Repository navigation
[AgentX] DeepSeek-V4.1-Flash GB300 vLLM: move to vLLM main a2bcb565 with specialized decode and retune TP4/DEP4 / [AgentX] DeepSeek-V4.1-Flash GB300 vLLM:升级到 vLLM main a2bcb565,启用 specialized decode 并重新调优 TP4/DEP4 #4844
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from 1 commit
10b1812
58bb586
7725ed5
41ba8f7
310784e
ad3713a
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -14,10 +14,10 @@ base: | |
| engine: | ||
| type: vllm | ||
| connector: null | ||
| # Cold loads of the shared-cache checkpoint run past half an hour. | ||
| # The engine may take the full VLLM_ENGINE_READY_TIMEOUT_S to become ready. | ||
| health_check: | ||
| interval_seconds: 5 | ||
| max_attempts: 1440 | ||
| interval_seconds: 10 | ||
| max_attempts: 720 | ||
| roles: | ||
| agg: | ||
| nodes: 1 | ||
|
|
@@ -29,19 +29,12 @@ base: | |
| tool-call-parser: deepseek_v41 | ||
| enable-auto-tool-choice: true | ||
| reasoning-parser: deepseek_v41 | ||
| engram-config: '{"cpu_offload":true}' | ||
| kernel-config: '{"enable_flashinfer_autotune":true}' | ||
| attention-config: '{"backend":"FLASHINFER_MLA_SPARSE_DSV41","indexer_kv_dtype":"mxfp4","indexer_sparse_logits":true}' | ||
| kv-cache-dtype: fp8 | ||
| # Five-token DSpark with probabilistic drafting. Throughput runs replace | ||
| # block rejection with the golden acceptance length. | ||
| speculative-config: '{"method":"dspark","num_speculative_tokens":5,"draft_sample_method":"probabilistic","rejection_sample_method":"block","enable_adaptive_verification":false}' | ||
| max-model-len: 1048576 | ||
| # Piecewise graph capture sizes are multiples of the six-token DSpark | ||
| # verification block, denser for small batches. | ||
| compilation-config: '{"mode":"VLLM_COMPILE","cudagraph_mode":"FULL_AND_PIECEWISE","cudagraph_capture_sizes":[6,12,18,24,30,36,48,60,72,96,120,144,192,240,288,384,480,576,768,1020,1536,2046,3072,4092,6144,8190]}' | ||
| max-cudagraph-capture-size: 8190 | ||
| max-num-batched-tokens: 8192 | ||
| gpu-memory-utilization: 0.93 | ||
| disable-uvicorn-access-log: true | ||
| env: | ||
| VLLM_USE_V2_MODEL_RUNNER: '1' | ||
|
|
@@ -54,29 +47,36 @@ base: | |
|
|
||
| # Native zip expansion creates one variant per concurrency. | ||
|
|
||
| # TP4 covers low concurrency with THP Engram tables. | ||
| zip_override_tp4: | ||
| roles: | ||
| agg: | ||
| gpus: 4 | ||
| args: | ||
| tensor-parallel-size: 4 | ||
| attention-config: '{"backend":"FLASHINFER_MLA_SPARSE_DSV41","indexer_kv_dtype":"mxfp4","indexer_sparse_logits":true}' | ||
| engram-config: '{"cpu_offload":true,"use_thp":true}' | ||
| max-num-seqs: [8, 8, 16, 32] | ||
| kernel-config: '{"enable_flashinfer_autotune":true}' | ||
| # Specialized decode: the SM103 latency-specialized TP4 decode executor. | ||
| additional-config: '{"decode_backend":"latency_specialized"}' | ||
| max-num-seqs: [1, 8, 16, 64, 64, 64, 64] | ||
| # Keeps decode batches small; max-num-seqs still sizes the runner and graphs. | ||
| max-num-active-seqs: [null, 1, 2, 3, 4, 6, 8] | ||
| prefill-schedule-interval: [1, 16, 96, 96, 96, 96, 96] | ||
| max-num-batched-tokens: [4096, 4096, 8192, 8192, 8192, 8192, 8192] | ||
| compilation-config: | ||
| - '{"mode":"VLLM_COMPILE","cudagraph_mode":"FULL_AND_PIECEWISE","cudagraph_capture_sizes":[6,12,18,24,30,36,48,60,72,96,120,144,192,240,288,384,480,576,768,1020,1536,2046,3072,4092]}' | ||
| - '{"mode":"VLLM_COMPILE","cudagraph_mode":"FULL_AND_PIECEWISE","cudagraph_capture_sizes":[6,12,18,24,30,36,48,60,72,96,120,144,192,240,288,384,480,576,768,1020,1536,2046,3072,4092]}' | ||
| - '{"mode":"VLLM_COMPILE","cudagraph_mode":"FULL_AND_PIECEWISE","cudagraph_capture_sizes":[6,12,18,24,30,36,48,60,72,96,120,144,192,240,288,384,480,576,768,1020,1536,2046,3072,4092,6144,8190]}' | ||
| - '{"mode":"VLLM_COMPILE","cudagraph_mode":"FULL_AND_PIECEWISE","cudagraph_capture_sizes":[6,12,18,24,30,36,48,60,72,96,120,144,192,240,288,384,480,576,768,1020,1536,2046,3072,4092,6144,8190],"decoder_replay_cudagraph_capture_sizes":[6,12,18,24,30,36,48,60,72,96,120,144,192,240,288,384,480,576,768,1020,1536,2048,2304,2560,2816,3072,3328,3584,3840,4096]}' | ||
| max-cudagraph-capture-size: [4092, 4092, 8190, 8190] | ||
| max-num-batched-tokens: [4096, 4096, 8192, 8192] | ||
| - '{"mode":"VLLM_COMPILE","cudagraph_mode":"FULL_AND_PIECEWISE","cudagraph_capture_sizes":[5,6,12,18,24,30,36,48,60,72,96,120,144,192,240,288,384,480,576,768,1020,1536,2046,3072,4092]}' | ||
| - '{"mode":"VLLM_COMPILE","cudagraph_mode":"FULL_AND_PIECEWISE","cudagraph_capture_sizes":[5,6,12,18,24,30,36,48,60,72,96,120,144,192,240,288,384,480,576,768,1020,1536,2046,3072,4092]}' | ||
| - '{"mode":"VLLM_COMPILE","cudagraph_mode":"FULL_AND_PIECEWISE","cudagraph_capture_sizes":[5,6,12,18,24,30,36,48,60,72,96,120,144,192,240,288,384,480,576,768,1020,1536,2046,3072,4092,6144,8190]}' | ||
| - '{"mode":"VLLM_COMPILE","cudagraph_mode":"FULL_AND_PIECEWISE","cudagraph_capture_sizes":[5,6,12,18,24,30,36,48,60,72,96,120,144,192,240,288,384,480,576,768,1020,1536,2046,3072,4092,6144,8190],"decoder_replay_cudagraph_capture_sizes":[6,12,18,24,30,36,48,60,72,96,120,144,192,240,288,384,480,576,768,1020,1536,2048,2304,2560,2816,3072,3328,3584,3840,4096]}' | ||
| - '{"mode":"VLLM_COMPILE","cudagraph_mode":"FULL_AND_PIECEWISE","cudagraph_capture_sizes":[5,6,12,18,24,30,36,48,60,72,96,120,144,192,240,288,384,480,576,768,1020,1536,2046,3072,4092,6144,8190],"decoder_replay_cudagraph_capture_sizes":[6,12,18,24,30,36,48,60,72,96,120,144,192,240,288,384,480,576,768,1020,1536,2048,2304,2560,2816,3072,3328,3584,3840,4096]}' | ||
| - '{"mode":"VLLM_COMPILE","cudagraph_mode":"FULL_AND_PIECEWISE","cudagraph_capture_sizes":[5,6,12,18,24,30,36,48,60,72,96,120,144,192,240,288,384,480,576,768,1020,1536,2046,3072,4092,6144,8190],"decoder_replay_cudagraph_capture_sizes":[6,12,18,24,30,36,48,60,72,96,120,144,192,240,288,384,480,576,768,1020,1536,2048,2304,2560,2816,3072,3328,3584,3840,4096]}' | ||
| - '{"mode":"VLLM_COMPILE","cudagraph_mode":"FULL_AND_PIECEWISE","cudagraph_capture_sizes":[5,6,12,18,24,30,36,48,60,72,96,120,144,192,240,288,384,480,576,768,1020,1536,2046,3072,4092,6144,8190],"decoder_replay_cudagraph_capture_sizes":[6,12,18,24,30,36,48,60,72,96,120,144,192,240,288,384,480,576,768,1020,1536,2048,2304,2560,2816,3072,3328,3584,3840,4096]}' | ||
| max-cudagraph-capture-size: [4092, 4092, 8190, 8190, 8190, 8190, 8190] | ||
| benchmark: | ||
| env: | ||
| CONC: ['1', '4', '8', '16'] | ||
| CONC: ['1', '4', '10', '16', '24', '32', '48'] | ||
|
|
||
| # DEP2: TP1 x DP2 + EP2 with MegaMoE behind a consistent-hash router; | ||
| # MegaAttention at CONC 128 and above. | ||
| zip_override_dep2: | ||
| zip_override_dep4: | ||
| frontend: | ||
| type: vllm-router | ||
| args: | ||
|
|
@@ -88,30 +88,38 @@ zip_override_dep2: | |
| setup_script: vllm-router.sh | ||
| roles: | ||
| agg: | ||
| gpus: 2 | ||
| gpus: 4 | ||
| args: | ||
| tensor-parallel-size: 1 | ||
| data-parallel-size: 2 | ||
| data-parallel-size: 4 | ||
| enable-expert-parallel: true | ||
| kernel-config: '{"enable_flashinfer_autotune":true,"moe_backend":"deep_gemm_mega_moe"}' | ||
| engram-config: '{"cpu_offload":true,"embedding_across_dp":true}' | ||
| max-num-seqs: [16, 16, 32, 64, 128, 192] | ||
| attention-config: | ||
| - '{"backend":"FLASHINFER_MLA_SPARSE_DSV41","indexer_kv_dtype":"mxfp4","indexer_sparse_logits":true}' | ||
| - '{"backend":"FLASHINFER_MLA_SPARSE_DSV41","indexer_kv_dtype":"mxfp4","indexer_sparse_logits":true}' | ||
| - '{"backend":"FLASHINFER_MLA_SPARSE_DSV41","indexer_kv_dtype":"mxfp4","indexer_sparse_logits":true}' | ||
| - '{"backend":"FLASHINFER_MLA_SPARSE_DSV41","indexer_kv_dtype":"mxfp4","indexer_sparse_logits":true}' | ||
| - '{"backend":"FLASHMLA_MEGA_ATTN_DSV41","indexer_kv_dtype":"mxfp4","indexer_sparse_logits":true}' | ||
| - '{"backend":"FLASHMLA_MEGA_ATTN_DSV41","indexer_kv_dtype":"mxfp4","indexer_sparse_logits":true}' | ||
| compilation-config: | ||
| - '{"mode":"VLLM_COMPILE","cudagraph_mode":"FULL_AND_PIECEWISE","cudagraph_capture_sizes":[6,12,18,24,30,36,48,60,72,96,120,144,192,240,288,384,480,576,768,1020,1536,2046,3072,4092,6144,8190]}' | ||
| - '{"mode":"VLLM_COMPILE","cudagraph_mode":"FULL_AND_PIECEWISE","cudagraph_capture_sizes":[6,12,18,24,30,36,48,60,72,96,120,144,192,240,288,384,480,576,768,1020,1536,2046,3072,4092,6144,8190]}' | ||
| - '{"mode":"VLLM_COMPILE","cudagraph_mode":"FULL_AND_PIECEWISE","cudagraph_capture_sizes":[6,12,18,24,30,36,48,60,72,96,120,144,192,240,288,384,480,576,768,1020,1536,2046,3072,4092,6144,8190],"decoder_replay_cudagraph_capture_sizes":[6,12,18,24,30,36,48,60,72,96,120,144,192,240,288,384,480,576,768,1020,1536,2048,2304,2560,2816,3072,3328,3584,3840,4096]}' | ||
| - '{"mode":"VLLM_COMPILE","cudagraph_mode":"FULL_AND_PIECEWISE","cudagraph_capture_sizes":[6,12,18,24,30,36,48,60,72,96,120,144,192,240,288,384,480,576,768,1020,1536,2046,3072,4092,6144,8190],"decoder_replay_cudagraph_capture_sizes":[6,12,18,24,30,36,48,60,72,96,120,144,192,240,288,384,480,576,768,1020,1536,2048,2304,2560,2816,3072,3328,3584,3840,4096]}' | ||
| - '{"mode":"VLLM_COMPILE","cudagraph_mode":"FULL_AND_PIECEWISE","cudagraph_capture_sizes":[6,12,18,24,30,36,48,60,72,96,120,144,192,240,288,384,480,576,768,1020,1536,2046,3072,4092,6144,8190],"decoder_replay_cudagraph_capture_sizes":[6,12,18,24,30,36,48,60,72,96,120,144,192,240,288,384,480,576,768,1020,1536,2048,2304,2560,2816,3072,3328,3584,3840,4096]}' | ||
| - '{"mode":"VLLM_COMPILE","cudagraph_mode":"FULL_AND_PIECEWISE","cudagraph_capture_sizes":[6,12,18,24,30,36,48,60,72,96,120,144,192,240,288,384,480,576,768,1020,1536,2046,3072,4092],"decoder_replay_cudagraph_capture_sizes":[6,12,18,24,30,36,48,60,72,96,120,144,192,240,288,384,480,576,768,1020,1536,2048]}' | ||
| max-cudagraph-capture-size: [8190, 8190, 8190, 8190, 8190, 4092] | ||
| max-num-batched-tokens: [8192, 8192, 8192, 8192, 16384, 16384] | ||
| enable-cumem-allocator: true | ||
| enable-ep-weight-filter: true | ||
| attention-config: '{"backend":"FLASHMLA_MEGA_ATTN_DSV41","indexer_kv_dtype":"mxfp4","indexer_sparse_logits":true,"sparse_mla_prefill_workspace_factor":4}' | ||
| engram-config: '{"cpu_offload":true,"embedding_across_dp":true,"use_thp":false}' | ||
| kernel-config: '{"enable_flashinfer_autotune":true,"moe_backend":"deep_gemm_mega_moe","sparse_indexer_topk_backend":"deep_select"}' | ||
| max-num-seqs: [128, 128, 128, 128, 128, 128, 192, 224, 256, 320] | ||
| prefill-schedule-interval: 96 | ||
| max-num-batched-tokens: 65536 | ||
| # gpu-memory-utilization sizing reserves this prefill budget's activation peak and leaves too little KV. | ||
| kv-cache-memory: [25769803776, 30064771072, 34359738368, 42949672960, 51539607552, 60129542144, 77309411328, 103079215104, 120259084288, 137438953472] | ||
| # A cold long prompt on one DP rank would otherwise stall the other ranks' decode for a full budget step. | ||
| long-prefill-token-threshold: 8192 | ||
| compilation-config: '{"mode":"VLLM_COMPILE","cudagraph_mode":"FULL_AND_PIECEWISE","cudagraph_capture_sizes":[6,12,18,24,30,36,48,60,72,96,120,144,192,240,288,384,480,576,768,1020,1536,2046,3072,4092,6144,8190],"decoder_replay_cudagraph_capture_sizes":[6,12,18,24,30,36,48,60,72,96,120,144,192,240,288,384,480,576,768,1020,1536,2048,2304,2560,2816,3072,3328,3584,3840,4096]}' | ||
| max-cudagraph-capture-size: 8190 | ||
| # Binds each DP rank's EngineCore and worker to its GPU's NUMA node. | ||
| numa-bind: true | ||
| numa-bind-nodes: [[0, 0, 1, 1]] | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🔴 GB300 DeepSeek-V4.1-Flash operatorx layer cases that fall back to this recipe would get server args that Why this was flaggedThe operatorx CI calls Verification: Severity: normal. Trigger: the recipe image's vLLM declares |
||
| env: | ||
| VLLM_RPC_TIMEOUT: '600000' | ||
| VLLM_LOG_STATS_INTERVAL: '1' | ||
| VLLM_SERVER_DEV_MODE: '1' | ||
| VLLM_ALLREDUCE_USE_SYMM_MEM: '0' | ||
| VLLM_RANDOMIZE_DP_DUMMY_INPUTS: '1' | ||
| NCCL_CUMEM_ENABLE: '1' | ||
| NCCL_MNNVL_ENABLE: '1' | ||
| NCCL_NVLS_ENABLE: '1' | ||
| PYTORCH_CUDA_ALLOC_CONF: expandable_segments:True | ||
| benchmark: | ||
| env: | ||
| CONC: ['8', '16', '32', '64', '128', '192'] | ||
| CONC: ['136', '170', '200', '256', '320', '384', '512', '640', '768', '1024'] | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🟡 nit (optional): The comment above
additional-configonly repeats what the next line says, which AGENTS.md's comment rules forbid. Fix: delete the comment, or swap it for the reason TP4 needs this executor (for example, a measured decode-latency gain or an SM103-only constraint).Why this was flagged
agentic.yaml:59 adds
# Specialized decode: the SM103 latency-specialized TP4 decode executor.directly aboveadditional-config: '{"decode_backend":"latency_specialized"}'at line 60. Root CLAUDE.md points to AGENTS.md. Under its mandatory comment rules, a comment may only briefly explain a specific line or block, such as why a value was chosen or a hardware quirk, and must never restate the code. This comment only names what the value already says, so it adds a convention violation the base recipe did not have. There is no runtime impact.Verification: nit. Line 59 of the new agentic.yaml in zip_override_tp4 is
# Specialized decode: the SM103 latency-specialized TP4 decode executor., directly above line 60,additional-config: '{"decode_backend":"latency_specialized"}'. AGENTS.md lines 54-58 say to "never restate the code." Only "SM103" adds a detail, and it gives no reason. Nothing at runtime is affected.