TG improvements for MoE models#404
Merged
Merged
Conversation
added 3 commits
May 10, 2025 09:49
We get 3-4% TG speed improvement for DeepSeek-Lite just from that.
With smart experts reduction (SER), one potentially uses fewer experts than specified by the model. This is accomplished by setting the ID of the not seected tensors to -1. Most of the necessary stuff was implemented when I added the SER option, but I forgot to update get_rows() for not quantized tensors. As a result, we get random garbage for the weights of the not-selected epxerts, which leads to garbage output. This commit fixes it on the CPU. I'm not quite sure yet why the GPU is not working.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR does 3 things:
GGML_OP_GET_ROWSop implementation did not consider disabled experts for float tensors. As a result, when combining the results of the experts garbage weights were used for the disabled experts, which could lead to NaNs.ggml_cuda_op_mul_mat_vec_q_idfunction did not consider that an expert may be disabled, and needlessly calculated the matrix-vector multiplication for disabled experts.Prompt processing is not eaffected by these changes.
Here is a graph obtained with
sweep-benchshowing TG performance as a function of the number of tokens in the KV cacheN_KV. The model is DeepSeek-Lite quantized toQ4_0. The GPU is RTX-4080. Black symbols are without using SER, red symbols are with-ser 4,1. The command line is