Conversation
Co-authored-by: Stanisław Szymczyk <sszymczy@gmail.com>
Merged
ikawrakow
approved these changes
Feb 9, 2025
ikawrakow
left a comment
ikawrakow
left a comment
Owner
There was a problem hiding this comment.
Looks good. I added a minor change to check if wk_b and wv_b are available before turning on MLA (so we don't crash if someone is using an old model and asked for MLA).
PP-4096 for Q8_0_R8 quantized DeepSeek-Lite with -mla goes up to 292 t/s from 275 t/s with this change.
ikawrakow
added a commit
that referenced
this pull request
Feb 9, 2025
* Deepseek MLA Optimizations Co-authored-by: Stanisław Szymczyk <sszymczy@gmail.com> * Make MLA optional * Remove some unnecessary copies in the MLA attention * Deepseek MLA Optimizations V2 (#195) * Avoid allocating MHA KV cache when MLA is turned on * Added missing gguf-py file * Added final optimizations Co-authored-by: Stanisław Szymczyk <sszymczy@gmail.com> * Make sure we do have wk_b and wv_b before enabling MLA --------- Co-authored-by: Stanisław Szymczyk <sszymczy@gmail.com> Co-authored-by: Iwan Kawrakow <iwan.kawrakow@gmail.com> * Use type_k and type_v to set the types of the MLA caches They were hard-coded at f16. On my Ryzen-7950X with native bf16 support I get a fairly significant PP performance boost with bf16 KV-cache: PP-4096 = 320 t/s up from 292 t/s with fp16 KV-cache. * Better gemm strategy when nth > nhead It gives a ~10% PP performance boost for DeepSeek-Lite with 32 threads (with or without MLA). Before this commit, when nth > nhead heads were processed sequentially with all nth threads participating in each matrix multiplication. Now we ind the gcd of nhead and nth and split threads into nth/gcd groups, each group processing nhead/gcd heads. --------- Co-authored-by: Saood Karim <saood05@gmail.com> Co-authored-by: Stanisław Szymczyk <sszymczy@gmail.com> Co-authored-by: Iwan Kawrakow <iwan.kawrakow@gmail.com>
slartibardfast
pushed a commit
to slartibardfast/ik_llama.cpp
that referenced
this pull request
Jun 9, 2026
…0 GEMV (v0) + gate
Implements the small-batch (ne11<=K) read-once GEMV carve-out, byte-identical to
production MMQ mul_mat_q_split_k<Q4_0,...,split_k=4>. v0 is one thread per output
element (correctness-first); the float-accumulation order reproduces MMQ's exactly:
4 contiguous ascending split-K chunks (boundaries nb*s/4 aligned to blocks_per_iter=8,
tail dropped), one (float(dot32)*dA)*dB add per 32-block (NOT per-8-group — MMQ's k01
loop steps per-block for Q4_0 since dA,dB are one scale per 32), combined
p0+((p1+p2)+p3) via the split-K fixup. Default-off (env GGML_CUDA_M1_GEMV); wired in
ggml_cuda_op_mul_mat_q behind a runtime toggle + carve-out threshold K (default 8).
test-m1-gemv-byte-identity: A/B's MMQ vs GEMV through the identical dispatcher,
asserts max_abs_diff=0.0 per column across ne11 in {1,2,3,4,8} x K in
{1024,1280,5120,17408} (all real Q4_0 K-dims). ALL OK.
Finding (pre-existing MMQ bug, noted not fixed per CLAUDE.md 3): MMQ split_k=4
leaves dst uninitialized for K<1024 (nb<32) — slice-0 returns early without writing
dst when the first K-quarter rounds to empty after blocks_per_iter alignment, then the
fixup adds onto garbage (MMQ returns +/-3e38/inf at K=256/288/768; the GEMV stays
finite). All production K are multiples of 256 and >=5120, outside the buggy domain.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
slartibardfast
pushed a commit
to slartibardfast/ik_llama.cpp
that referenced
this pull request
Jun 9, 2026
…sced read, ordered fold) — byte-identical, 16.7 t/s Still byte-identical (FMA-contraction fix: shuffle X=dot*dA and dB separately, fold p+=X*dB so it contracts to fma(X,dB,p) matching MMQ; materializing X*dB before the add diverged ~1 ULP). Perf 16.71 vs MMQ 27.09 tg128 — the ordered fold on lane 0 serializes the warp (31 lanes idle). Checkpoint before v2 (lane-per-row parallel folds + SMEM-staged coalesced load), which removes the serial-fold bottleneck. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
slartibardfast
pushed a commit
to slartibardfast/ik_llama.cpp
that referenced
this pull request
Jun 9, 2026
…staged, parallel folds) — byte-identical, 7.1 t/s v2 removes v1's serial-fold bottleneck (each lane folds its own row in parallel) and stages weights through SMEM for coalesced global reads, but is SLOWER (7.1 vs MMQ 27.1): lane-strided 18-byte SMEM block reads cause heavy bank conflicts, the per-row cooperative load is partly serial, and 18KB/block SMEM caps occupancy. Gate now also requires ne00%256==0 (the tiled load needs nb%8==0). Three byte-identical designs measured, all < MMQ (v0 9.4 / v1 16.7 / v2 7.1 vs 27.1). Strong evidence the byte-identity constraint (MMQ's exact serial reduction order) forbids the lean warp-reduce that gave stock mmvq its +27% (ikawrakow#194). Decode-GEMV A-vs-B fork re-opens with data. Byte-identity itself remains PROVEN+correct (test ALL OK). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
@ikawrakow
This PR contains the following things
I will follow up with: