Skip to content

CUDA: small PP performance improvement for MoE models#589

Merged
ikawrakow merged 4 commits into
mainfrom
ik/cuda_quantized_fmoe
Jul 7, 2025
Merged

CUDA: small PP performance improvement for MoE models#589
ikawrakow merged 4 commits into
mainfrom
ik/cuda_quantized_fmoe

Conversation

@ikawrakow

Copy link
Copy Markdown
Owner

This PR brings a small (2-3%) prompt processing performance improvement on CUDA for quantized MoE models (when -fmoe is used).

Instead of first copying activations to contiguous memory and the quantizing, quantization is done directly using the row mapping IDs, thus saving the associated kernel launch overhead.

Here is a performance comparison for Q4_0 quantized DeepSeek-Lite on RTX-4080 using -mla 3 -fa -fmoe -b 4096 -ub 4096

Main branch

PP TG N_KV T_PP s S_PP t/s T_TG s S_TG t/s
4096 1024 0 0.480 8532.52 5.640 181.55
4096 1024 4096 0.566 7240.62 5.904 173.43
4096 1024 8192 0.674 6073.99 6.143 166.68
4096 1024 12288 0.789 5189.61 6.421 159.47

PR

PP TG N_KV T_PP s S_PP t/s T_TG s S_TG t/s
4096 1024 0 0.469 8738.41 5.638 181.61
4096 1024 4096 0.554 7388.85 5.909 173.29
4096 1024 8192 0.670 6117.30 6.148 166.57
4096 1024 12288 0.779 5256.86 6.435 159.14

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant