Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -65,7 +65,7 @@ override_c1:
agg:
args:
max-running-requests: 2
cuda-graph-max-bs: 2
cuda-graph-max-bs-decode: 2
speculative-num-steps: 5
speculative-num-draft-tokens: 6
benchmark:
Expand All @@ -77,7 +77,7 @@ override_c4:
agg:
args:
max-running-requests: 8
cuda-graph-max-bs: 8
cuda-graph-max-bs-decode: 8
speculative-num-steps: 5
speculative-num-draft-tokens: 6
benchmark:
Expand All @@ -89,7 +89,7 @@ override_c8:
agg:
args:
max-running-requests: 16
cuda-graph-max-bs: 16
cuda-graph-max-bs-decode: 16
speculative-num-steps: 5
speculative-num-draft-tokens: 6
benchmark:
Expand All @@ -101,7 +101,7 @@ override_c12:
agg:
args:
max-running-requests: 24
cuda-graph-max-bs: 24
cuda-graph-max-bs-decode: 24
benchmark:
env:
CONC: '12'
Expand All @@ -111,7 +111,7 @@ override_c16:
agg:
args:
max-running-requests: 32
cuda-graph-max-bs: 32
cuda-graph-max-bs-decode: 32
benchmark:
env:
CONC: '16'
Expand All @@ -121,7 +121,7 @@ override_c20:
agg:
args:
max-running-requests: 40
cuda-graph-max-bs: 40
cuda-graph-max-bs-decode: 40
speculative-num-steps: 5
speculative-num-draft-tokens: 6
benchmark:
Expand All @@ -133,7 +133,7 @@ override_c24:
agg:
args:
max-running-requests: 48
cuda-graph-max-bs: 48
cuda-graph-max-bs-decode: 48
benchmark:
env:
CONC: '24'
Expand All @@ -143,7 +143,7 @@ override_c28:
agg:
args:
max-running-requests: 56
cuda-graph-max-bs: 56
cuda-graph-max-bs-decode: 56
benchmark:
env:
CONC: '28'
Expand All @@ -153,7 +153,7 @@ override_c32:
agg:
args:
max-running-requests: 64
cuda-graph-max-bs: 64
cuda-graph-max-bs-decode: 64
benchmark:
env:
CONC: '32'
Expand All @@ -163,7 +163,7 @@ override_c40:
agg:
args:
max-running-requests: 80
cuda-graph-max-bs: 64
cuda-graph-max-bs-decode: 64
benchmark:
env:
CONC: '40'
Expand All @@ -173,7 +173,7 @@ override_c48:
agg:
args:
max-running-requests: 96
cuda-graph-max-bs: 64
cuda-graph-max-bs-decode: 64
benchmark:
env:
CONC: '48'
2 changes: 1 addition & 1 deletion inferencex-e2e/configs/nvidia-master.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -6958,7 +6958,7 @@ glm5.2-fp8-h200-dynamo-sglang-agentic-mtp-agg:
# HiCache host-DRAM offload is required because the working set outgrows HBM
# past conc 8 without it and the radix hit rate collapses to <0.1.
glm5.2-fp4-b300-sglang-agentic-mtp:
image: lmsysorg/sglang:nightly-dev-cu13-20260901-07c8f729
image: lmsysorg/sglang:v0.5.22-cu130@sha256:e3a4f06001a29542fc4a8638536d9ffc5eb9c9942b08603c09b19a56eae4a1ae
model: nvidia/GLM-5.2-NVFP4
model-prefix: glm5.2
runner: cluster:b300-dsxe
Expand Down
6 changes: 6 additions & 0 deletions inferencex-e2e/perf-changelog.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -10298,3 +10298,9 @@
- "The image carries vllm-project/vllm#62131: --monokernel now also runs the three DSpark draft layers as the two persistent mono decode launches (128 experts / top-3, window-only attention over the block's non-causal SWA rows) instead of vLLM's regular path. The step limit is unchanged at 48 rows, i.e. concurrency 1-8 with five DSpark drafts."
- "No recipe or point changes: concurrency 1, 2, 4, 8, 16, 32 and 64 on TP2 and TP4 (14 points), all re-measured on the new image."
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/4319

- config-keys:
- glm5.2-fp4-b300-sglang-agentic-mtp
description:
- "Update SGLang image to v0.5.22-cu130 and rename cuda-graph-max-bs to cuda-graph-max-bs-decode for the removed alias."
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/4325
Loading