Skip to content

[Perf] [AMD] Qwen3.5/Qwen3.8: Opt-in AITER FlyDSL causal conv1d prefill - #41879

Open
amd-nprotaso wants to merge 15 commits into
sgl-project:mainfrom
amd-nprotaso:pr/gdn-conv-flydsl
Open

amd-nprotaso wants to merge 15 commits into
sgl-project:mainfrom
amd-nprotaso:pr/gdn-conv-flydsl

Conversation

@amd-nprotaso

@amd-nprotaso amd-nprotaso commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor

Requires AITER with ROCm/aiter#5992 (merged), which adds causal_conv1d_prefill_flydsl_fn.

Motivation

On ROCm, the GDN layers of Qwen3.5/Qwen3.8 run the prefill causal conv1d through Triton. AITER has a FlyDSL kernel for it on gfx950 that is 1.1x to 1.8x faster. This PR adds it behind an opt-in flag. Triton stays the default and the fallback.

Modifications

layers/attention/linear/gdn_backend.py only:

  • The prefill convolution dispatches through _prefill_causal_conv1d. With SGLANG_USE_AITER=1 and SGLANG_AITER_GDN_CONV=1 it calls AITER's causal_conv1d_prefill_flydsl_fn, which has the same contract as causal_conv1d_fn: one output, conv_states updated in place.
  • Fallback: an AITER ValueError (unsupported shape or dtype) falls back to Triton causal_conv1d_fn. When the flags are off or the import fails, that choice is cached and Triton is used.
  • Block-table launches (conv_block_table set) stay on Triton.

Nothing changes on CUDA or when the flags are unset.

Accuracy Tests

Kernel level, on Qwen3.8 shapes, against Triton causal_conv1d_fn:

  • Output: max abs diff 3.1e-2, which is 1 bf16 ULP.
  • conv_states: bit-identical.

Speed Tests and Profiling

MI355X (gfx950), BF16, Qwen3.8 TP1 shapes (dim 10240, width 4), eager timing:

tokens in batch Triton (µs) FlyDSL (µs) speedup
1 × 1024 22.5 19.6 1.14x
1 × 4096 62.6 47.6 1.32x
1 × 16384 313.8 181.2 1.73x
16 × 1024 324.9 179.8 1.81x

End to end

Setup:

  • Hardware and model: MI355X, TP1, amd/Qwen3.8-Flash-Next-Quark-MXFP4-PLEFP8.
    E2E performance on InferenceX agentixX benchmark, c16
Metric Baseline PR Δ
Total tput/GPU (tok/s) 78,936 79,180 +0.3%
Interactivity p50 (tok/s/user) 126.75 127.45 +0.5%
Interactivity p90 (tok/s/user) 76.37 80.56 +5.5% ✅
ITL mean / p50 / p90 (ms) 9.30 / 7.89 / 13.09 9.05 / 7.85 / 12.41 −2.7% ✅ / −0.5% / −5.2% ✅
E2E latency p50 / p90 (s) 3.50 / 18.93 3.44 / 18.64 −1.7 / −1.5% ✅
Error-dropped requests 1 2 +1
Mean MTP accept length 2.320 2.322 +0.1%
aiperf profile
### main e5739babc1
                                                      NVIDIA AIPerf | LLM Metrics                                                      
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━┓
┃                                         Metric ┃        avg ┃    min ┃        max ┃        p99 ┃        p90 ┃       p50 ┃       std ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━┩
│                       Time to First Token (ms) │     565.11 │ 147.76 │   7,604.23 │   4,076.88 │     873.86 │    397.98 │    671.08 │
│                      Time to Second Token (ms) │     477.59 │   0.09 │   7,880.91 │   3,252.95 │     629.90 │    370.10 │    530.07 │
│                Time to First Output Token (ms) │   5,579.19 │ 413.37 │  41,047.81 │  25,890.62 │  10,817.85 │  4,059.41 │  4,946.70 │
│                           Request Latency (ms) │   8,313.37 │ 226.45 │ 258,400.83 │  75,641.24 │  18,934.24 │  3,503.18 │ 15,397.11 │
│                           Decode Duration (ms) │   7,748.27 │   0.00 │ 258,205.96 │  74,568.62 │  17,956.04 │  2,905.31 │ 15,270.95 │
│                      Full Decode Duration (ms) │   7,748.70 │   0.33 │ 258,206.89 │  74,569.21 │  17,956.49 │  2,905.63 │ 15,270.98 │
│                       Inter Token Latency (ms) │       9.30 │   0.00 │     363.50 │      28.56 │      13.09 │      7.89 │     10.73 │
│              Full-Response Inter Token Latency │       9.30 │   0.03 │     363.53 │      28.57 │      13.09 │      7.89 │     10.73 │
│                                           (ms) │            │        │            │            │            │           │           │
│               Output Token Throughput Per User │     124.95 │   2.75 │     474.21 │     207.73 │     165.20 │    126.75 │     35.44 │
│                              (tokens/sec/user) │            │        │            │            │            │           │           │
│  E2E Output Token Throughput (tokens/sec/user) │     101.96 │   0.39 │     212.74 │     181.10 │     142.97 │    103.11 │     32.92 │
│ Full-Response Output Token Throughput Per User │     137.94 │   2.75 │  35,049.51 │     207.89 │     165.30 │    126.72 │    675.43 │
│                              (tokens/sec/user) │            │        │            │            │            │           │           │
│                Output Sequence Length (tokens) │     930.04 │   1.00 │  32,370.00 │   9,265.02 │   2,212.60 │    326.00 │  1,895.36 │
│                 Input Sequence Length (tokens) │ 105,266.37 │ 194.00 │ 256,228.00 │ 246,472.66 │ 192,871.00 │ 94,508.00 │ 58,608.78 │
│           Output Token Throughput (tokens/sec) │     691.30 │    N/A │        N/A │        N/A │        N/A │       N/A │       N/A │
│            Input Token Throughput (tokens/sec) │  78,244.82 │    N/A │        N/A │        N/A │        N/A │       N/A │       N/A │
│              Request Throughput (requests/sec) │       0.74 │    N/A │        N/A │        N/A │        N/A │       N/A │       N/A │
│                         Request Error Rate (%) │       0.00 │    N/A │        N/A │        N/A │        N/A │       N/A │       N/A │
│                       Request Count (requests) │   2,683.00 │    N/A │        N/A │        N/A │        N/A │       N/A │       N/A │
│               Theoretical Prefix Cache Hit (%) │      96.34 │    N/A │        N/A │        N/A │        N/A │       N/A │       N/A │
└────────────────────────────────────────────────┴────────────┴────────┴────────────┴────────────┴────────────┴───────────┴───────────┘

 NVIDIA AIPerf | Server Metrics: Speculative Decoding 
┏━━━━━━━━━━━━━━━━━┳━━━━━━┳━━━━━━┳━━━━━━┳━━━━━━┳━━━━━━┓
┃ Metric          ┃ mean ┃  min ┃  max ┃  p50 ┃  p90 ┃
┡━━━━━━━━━━━━━━━━━╇━━━━━━╇━━━━━━╇━━━━━━╇━━━━━━╇━━━━━━┩
│ Accept Rate (%) │ 44.0 │  0.0 │ 53.5 │ 44.1 │ 47.4 │
│ Accept Length   │ 2.32 │ 0.00 │ 2.60 │ 2.32 │ 2.42 │
└─────────────────┴──────┴──────┴──────┴──────┴──────┘

CLI Command: aiperf profile --url 'http://localhost:8100' --model
'amd/Qwen3.8-Flash-Next-Quark-MXFP4-AttnFP8-PLEFP8' --tokenizer
'amd/Qwen3.8-Flash-Next-Quark-MXFP4-AttnFP8-PLEFP8' --concurrency 16 --benchmark-duration 3600
--tokenizer-trust-remote-code --server-metrics 'http://localhost:8100/metrics' --public-dataset
'semianalysis_cc_traces_weka_062126_256k' --apply-chat-template
Benchmark Duration: 3609.56 sec

### #41879 5a2bc3760b, SGLANG_AITER_GDN_CONV=1
                                                      NVIDIA AIPerf | LLM Metrics                                                      
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━┓
┃                                         Metric ┃        avg ┃    min ┃        max ┃        p99 ┃        p90 ┃       p50 ┃       std ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━┩
│                       Time to First Token (ms) │     567.11 │ 149.83 │   7,420.32 │   3,747.48 │     897.32 │    394.15 │    669.29 │
│                      Time to Second Token (ms) │     457.78 │   0.04 │   7,660.20 │   2,279.87 │     615.13 │    369.68 │    444.31 │
│                Time to First Output Token (ms) │   5,818.27 │ 518.17 │  58,087.32 │  31,982.75 │  10,907.25 │  4,007.48 │  6,146.30 │
│                           Request Latency (ms) │   8,263.79 │ 218.72 │ 259,666.73 │  75,090.30 │  18,644.91 │  3,442.11 │ 15,348.24 │
│                           Decode Duration (ms) │   7,696.67 │   0.00 │ 259,493.96 │  73,909.67 │  17,800.82 │  2,889.01 │ 15,236.19 │
│                      Full Decode Duration (ms) │   7,697.13 │   0.36 │ 259,494.83 │  73,910.30 │  17,801.27 │  2,889.49 │ 15,236.27 │
│                       Inter Token Latency (ms) │       9.05 │   2.92 │     362.23 │      24.43 │      12.41 │      7.85 │      9.37 │
│              Full-Response Inter Token Latency │       9.06 │   2.92 │     362.26 │      24.43 │      12.41 │      7.85 │      9.37 │
│                                           (ms) │            │        │            │            │            │           │           │
│               Output Token Throughput Per User │     126.03 │   2.76 │     342.34 │     209.16 │     165.51 │    127.45 │     34.52 │
│                              (tokens/sec/user) │            │        │            │            │            │           │           │
│  E2E Output Token Throughput (tokens/sec/user) │     102.76 │   2.27 │     214.70 │     180.78 │     142.26 │    104.63 │     32.97 │
│ Full-Response Output Token Throughput Per User │     125.98 │   2.76 │     341.92 │     208.69 │     165.43 │    127.36 │     34.49 │
│                              (tokens/sec/user) │            │        │            │            │            │           │           │
│                Output Sequence Length (tokens) │     930.65 │   1.00 │  32,370.00 │   9,257.85 │   2,218.00 │    327.00 │  1,894.61 │
│                 Input Sequence Length (tokens) │ 105,399.39 │ 196.00 │ 256,230.00 │ 247,022.80 │ 192,951.50 │ 94,567.00 │ 58,701.04 │
│           Output Token Throughput (tokens/sec) │     693.03 │    N/A │        N/A │        N/A │        N/A │       N/A │       N/A │
│            Input Token Throughput (tokens/sec) │  78,487.28 │    N/A │        N/A │        N/A │        N/A │       N/A │       N/A │
│              Request Throughput (requests/sec) │       0.74 │    N/A │        N/A │        N/A │        N/A │       N/A │       N/A │
│                         Request Error Rate (%) │       0.00 │    N/A │        N/A │        N/A │        N/A │       N/A │       N/A │
│                       Request Count (requests) │   2,686.00 │    N/A │        N/A │        N/A │        N/A │       N/A │       N/A │
│               Theoretical Prefix Cache Hit (%) │      96.35 │    N/A │        N/A │        N/A │        N/A │       N/A │       N/A │
└────────────────────────────────────────────────┴────────────┴────────┴────────────┴────────────┴────────────┴───────────┴───────────┘

 NVIDIA AIPerf | Server Metrics: Speculative Decoding 
┏━━━━━━━━━━━━━━━━━┳━━━━━━┳━━━━━━┳━━━━━━┳━━━━━━┳━━━━━━┓
┃ Metric          ┃ mean ┃  min ┃  max ┃  p50 ┃  p90 ┃
┡━━━━━━━━━━━━━━━━━╇━━━━━━╇━━━━━━╇━━━━━━╇━━━━━━╇━━━━━━┩
│ Accept Rate (%) │ 44.0 │  0.0 │ 54.1 │ 44.2 │ 47.5 │
│ Accept Length   │ 2.32 │ 0.00 │ 2.62 │ 2.33 │ 2.42 │
└─────────────────┴──────┴──────┴──────┴──────┴──────┘

CLI Command: aiperf profile --url 'http://localhost:8102' --model
'amd/Qwen3.8-Flash-Next-Quark-MXFP4-AttnFP8-PLEFP8' --tokenizer
'amd/Qwen3.8-Flash-Next-Quark-MXFP4-AttnFP8-PLEFP8' --concurrency 16 --benchmark-duration 3600
--tokenizer-trust-remote-code --server-metrics 'http://localhost:8102/metrics' --public-dataset
'semianalysis_cc_traces_weka_062126_256k' --apply-chat-template
Benchmark Duration: 3606.98 sec


CI States

Latest PR Test (Base): ✅ Run #43865775653
Latest PR Test (Extra): ❌ Run #43865775433
Latest PR Test (AMD ROCm 10): ❌ Run #43865775510

Tip

Base CI only covers the core cases, and its workload may not exercise your change. Use /rerun-test to run the tests your change touches, or copy this prompt for an agent:

Find the registered tests this PR affects and run them with /rerun-test, following https://github.com/sgl-project/sglang/blob/main/.agents/skills/rerun-test/SKILL.md

ChangLiu0709 added a commit to ChangLiu0709/InferenceX that referenced this pull request Oct 15, 2026
…10 image with patches

Image: lmsysorg/sglang-rocm:v0.5.21-rocm10-mi35x-20261014 (pinned by digest).
The recipe's setup_script patches its editable AITER (e7d2453f) and SGLang
(bab9c81f) checkouts before the server starts:
  - ROCm/aiter#5992: FlyDSL causal conv1d prefill and GDN decode/verify
  - sgl-project/sglang#41879, ported to bab9c81f: opt-in
    SGLANG_AITER_GDN_CONV/_DECODE hooks
  - fused Triton QK Gemma-RMSNorm + M-RoPE + gate on HIP for Qwen3.5/Qwen3.8
A patch that no longer applies fails the worker.

Local check on MI355X (TP1, recipe server args at conc 16, CI's lm-eval
GSM8K, 5-shot, full test set): 0.9636 with GDN conv on and GDN decode off,
as in the recipe; 0.9697 with GDN decode on too.

Co-authored-by: Cursor <cursoragent@cursor.com>
@1am9trash 1am9trash added amd run-ci CI: run the baseline test suite on this PR labels Oct 23, 2026
@1am9trash

1am9trash commented Oct 23, 2026 •

Copy link
Copy Markdown
Collaborator

Hi, @amd-nprotaso

I am a bit confused by the decode part change. I am not sure the decode kernel will be called or not?
Because supports_packed_decode may be True, so the decode part runs into packed_decode() and returns there. It may not reach decode().

# gdn_triton.py
supports_packed_decode = not is_cpu() and not is_npu()
# gdn_backend.py
# early return
if self.kernel_dispatcher.supports_packed_decode:
    core_attn_out = self.kernel_dispatcher.packed_decode(
        ......
    )
    return
# your change is inside this decode()
core_attn_out = self.kernel_dispatcher.decode(
    ......
)

Also, the current results look kernel-level only, so we cannot tell whether sglang actually runs these kernels. Could we add an e2e test? We may see improvement on TTFT for prefill(), ITL for decode(), and ITL with speculative decoding for verify().

@1am9trash

Copy link
Copy Markdown
Collaborator

Hi, @amd-nprotaso

Some other notes:

  • This branch conflicts with main. I added the run-ci tag, but CI needs the conflict resolved before it can run.
  • Could SGLANG_AITER_GDN_CONV and SGLANG_AITER_GDN_DECODE go in environ.py, default on for HIP? I know you mentioned that decode regresses at high conc, but users may not know where that threshold is. A batch-size guard in the dispatch may be easier than leaving it opt-in.

Thanks.

@github-actions

This comment has been minimized.

@amd-nprotaso amd-nprotaso changed the title [AMD] Opt-in AITER FlyDSL causal conv1d prefill and GDN decode for Qwen3.5/Qwen3.8Pr/gdn conv flydsl [Perf] [AMD] Qwen3.5/Qwen3.8: Opt-in AITER FlyDSL causal conv1d prefill and GDN decode Oct 24, 2026
Resolve conflict in gdn_backend.py: route the FlyDSL prefill conv dispatch
through upstream's new _convolve_prefill helper, keeping the Triton-only
block-table launch on causal_conv1d_fn.

Co-Authored-By: Claude <noreply@anthropic.com>
@amd-nprotaso

Copy link
Copy Markdown
Contributor Author

Hi, @amd-nprotaso

I am a bit confused by the decode part change. I am not sure the decode kernel will be called or not? Because supports_packed_decode may be True, so the decode part runs into packed_decode() and returns there. It may not reach decode().

# gdn_triton.py
supports_packed_decode = not is_cpu() and not is_npu()
# gdn_backend.py
# early return
if self.kernel_dispatcher.supports_packed_decode:
    core_attn_out = self.kernel_dispatcher.packed_decode(
        ......
    )
    return
# your change is inside this decode()
core_attn_out = self.kernel_dispatcher.decode(
    ......
)

Also, the current results look kernel-level only, so we cannot tell whether sglang actually runs these kernels. Could we add an e2e test? We may see improvement on TTFT for prefill(), ITL for decode(), and ITL with speculative decoding for verify().

Correct. I move it to TritonGDNKernel.packed_decode, so new kernels, will be use even for packed scenario. End-to-end gains are small (1–2%), for conc 16, ISL=32k OSL=32k

@1am9trash

1am9trash commented Oct 29, 2026 •

Copy link
Copy Markdown
Collaborator

Hi, @amd-nprotaso

I mentioned this in previous comments. Maybe overlook. XD!

Could we add a table for e2e perf in PR head?
It may help to clear ttft/itl change before/after the PR. Currently we only have kernel-side comparison, hard to know e2e impact of this PR.

Could SGLANG_AITER_GDN_CONV and SGLANG_AITER_GDN_DECODE go in environ.py, default on for HIP? I know you mentioned that decode regresses at high conc, but users may not know where that threshold is. A batch-size guard in the dispatch may be easier than leaving it opt-in.

Thanks.

@amd-nprotaso

amd-nprotaso commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor Author

. Maybe overlook. XD!

I measure again with updated SGLang:

ISL OSL Conc Interactivity (tok/s/user) Throughput (tok/s/GPU) Mean TTFT (ms) Runs
32k 100 8 – 35,921 → 36,540 (+1.7%) 4,104 → 4,030 (−1.8%) 3
32k 100 12 – 36,655 → 37,297 (+1.8%) 5,786 → 5,678 (−1.9%) 3
32k 100 16 – 36,879 → 37,508 (+1.7%) 7,576 → 7,465 (−1.5%) 3
64k 100 8 – 35,288 → 35,890 (+1.7%) 8,378 → 8,211 (−2.0%) 3
64k 100 12 – 35,790 → 36,274 (+1.4%) 11,889 → 11,733 (−1.3%) 3
64k 100 16 – 35,928 → 36,429 (+1.4%) 15,591 → 15,398 (−1.2%) 3
32k 32k 8 146.8 → 148.6 (+1.2%) 2,356 → 2,386 (+1.3%) 4,107 → 4,010 (−2.4%) 1
32k 32k 12 124.4 → 123.8 (−0.5%) 2,969 → 2,973 (+0.1%) 5,843 → 5,674 (−2.9%) 1
32k 32k 16 † 113.1 → 113.5 (+0.3%) 2,843 → 2,874 (+1.1%) 24,953 → 24,720 (−0.9%) 1
64k 32k 8 140.6 → 142.7 (+1.4%) 3,290 → 3,326 (+1.1%) 8,358 → 8,258 (−1.2%) 1
64k 32k 12 120.0 → 118.9 (−1.0%) 4,130 → 4,131 (0.0%) 11,935 → 11,681 (−2.1%) 1
64k 32k 16 † 108.0 → 108.5 (+0.4%) 3,936 → 3,982 (+1.2%) 33,786 → 33,430 (−1.1%) 1

And see Interactivity degradation for conc=12, so maybe will close this PR.

@1am9trash

Copy link
Copy Markdown
Collaborator

@1am9trash 1am9trash left a comment •

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wrong approval.
Didn't see the regression issue.
Please feel free to close or modify the PR.

End-to-end A/B on MI355X showed no decode-side gain from SGLANG_AITER_GDN_DECODE,
and it only ran with a bf16 SSM state, which is not the model default. The PR now
carries just the opt-in FlyDSL prefill causal conv (SGLANG_AITER_GDN_CONV).

Co-Authored-By: Claude <noreply@anthropic.com>
@amd-nprotaso amd-nprotaso changed the title [Perf] [AMD] Qwen3.5/Qwen3.8: Opt-in AITER FlyDSL causal conv1d prefill and GDN decode [Perf] [AMD] Qwen3.5/Qwen3.8: Opt-in AITER FlyDSL causal conv1d prefill Nov 2, 2026
@amd-nprotaso

Copy link
Copy Markdown
Contributor Author

Wrong approval. Didn't see the regression issue. Please feel free to close or modify the PR.

Updated PR, now it's only about convolution kernel. Add e2e results for different scenarios.

@ChangLiu0709

ChangLiu0709 commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

@amd-nprotaso

amd-nprotaso commented Nov 8, 2026 •

Copy link
Copy Markdown
Contributor Author

E2E performance on InferenceX agentixX benchmark, c16

Metric Baseline PR Δ
Total tput/GPU (tok/s) 78,936 79,180 +0.3%
Interactivity p50 (tok/s/user) 126.75 127.45 +0.5%
Interactivity p90 (tok/s/user) 76.37 80.56 +5.5% ✅
ITL mean / p50 / p90 (ms) 9.30 / 7.89 / 13.09 9.05 / 7.85 / 12.41 −2.7% ✅ / −0.5% / −5.2% ✅
E2E latency p50 / p90 (s) 3.50 / 18.93 3.44 / 18.64 −1.7 / −1.5% ✅
Error-dropped requests 1 2 +1
Mean MTP accept length 2.320 2.322 +0.1%

@chuyeh

chuyeh commented Nov 10, 2026

Copy link
Copy Markdown
Contributor

Hi @amd-nprotaso . There's already test_gdn_prefill_backend_policy.py nearby. Adding a tiny CPU test for the fallback would be nice, e.g. a fake aiter fn that raises ValueError should get you the Triton result.

Fine to skip if you'd rather not.

@amd-nprotaso

Copy link
Copy Markdown
Contributor Author

Hi @amd-nprotaso . There's already test_gdn_prefill_backend_policy.py nearby. Adding a tiny CPU test for the fallback would be nice, e.g. a fake aiter fn that raises ValueError should get you the Triton result.

Fine to skip if you'd rather not.

Added unit test, better to have

@amd-nprotaso
amd-nprotaso requested a review from chuyeh November 10, 2026 12:14

@chuyeh chuyeh left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. This adds an opt-in AITER FlyDSL causal conv1d for GDN prefill on gfx950 (Qwen3.5/Qwen3.8), with Triton as the fallback.

@yichiche

Copy link
Copy Markdown
Collaborator

@ChangLiu0709 The InferenceX result seems to combine many PRs, so it cannot show how much performance gain this PR brings. Please remove it, or replace it with a correct InferenceX result that includes only PR #41879. c.c @amd-nprotaso

@yichiche

yichiche commented Nov 16, 2026 •

Copy link
Copy Markdown
Collaborator

@amd-nprotaso Please resolve conflict issue, code change LGTM.

@github-actions

This comment has been minimized.

@amd-nprotaso

amd-nprotaso commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor Author

@ChangLiu0709 The InferenceX result seems to combine many PRs, so it cannot show how much performance gain this PR brings. Please remove it, or replace it with a correct InferenceX result that includes only PR #41879. c.c @amd-nprotaso

Fresh mainline results for conc 16 and 48:

Conc Metric (AgentX, 1 h) GDN_CONV=0 GDN_CONV=1 Δ
16 Total throughput per GPU (tok/s) 79,607 80,283 +0.85%
16 Output throughput (tok/s) 697.3 703.3 +0.86%
16 TTFT mean / p50 / p90 (s) 0.531 / 0.389 / 0.872 0.539 / 0.397 / 0.883 +1.6% / +1.9% / +1.3%
16 TPOT p50 / p90 (ms) 7.89 / 12.03 7.78 / 11.92 −1.4% / −0.9%
16 Interactivity p50 / p90 (tok/s per user) 126.7 / 83.1 128.5 / 83.9 +1.4% / +0.9%
16 End-to-end latency p50 / p90 (s) 3.34 / 17.44 3.30 / 17.52 −1.3% / +0.5%
16 GPU prefix-cache hit rate 95.5% 95.5% —
16 Successful requests 2,708 2,717 —
48 Total throughput per GPU (tok/s) 157,372 157,591 +0.14%
48 Output throughput (tok/s) 1,504.4 1,505.9 +0.03%
48 TTFT mean / p50 / p90 (s) 0.818 / 0.548 / 1.570 0.808 / 0.550 / 1.501 −1.3% / +0.4% / −4.4%
48 TPOT p50 / p90 (ms) 17.77 / 29.68 17.59 / 28.64 −1.0% / −3.5%
48 Interactivity p50 / p90 (tok/s per user) 56.3 / 33.7 56.9 / 34.9 +1.1% / +3.6%
48 End-to-end latency p50 / p90 (s) 8.06 / 37.55 8.17 / 37.98 +1.3% / +1.1%
48 GPU prefix-cache hit rate 94.1% 94.1% —
48 Successful requests 6,007 6,027 —

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

amd run-ci CI: run the baseline test suite on this PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants