Enable AVX-VNNI 256-bit path for Q6_K R4 matmul#1482
Merged
Conversation
Add a separate HAVE_VNNI256 code path using _mm256_dpwssd_epi32 and _mm256_dpbusd_epi32 for the Q6_K R4 kernel. The existing HAVE_FANCY_SIMD (AVX-512 VNNI) path is preserved unchanged.
Nexesenex
added a commit
to Nexesenex/ik_llama.cpp.nxs
that referenced
this pull request
Mar 21, 2026
accaldwell
marked this pull request as ready for review
March 21, 2026 03:17
ikawrakow
approved these changes
Mar 23, 2026
4 tasks
AndrewMoryakov
added a commit
to AndrewMoryakov/ik_llama-pr
that referenced
this pull request
May 5, 2026
Adds a third value (`auto`) to the `--run-time-repack` / `-rtr` CLI flag, along with a matching `-rtra` / `--run-time-repack-auto` alias, that turns the existing repack-on-load behaviour into something safe to leave on by default. Motivation ---------- `-rtr 1` (run-time repack into row-interleaved variants such as `Q4_K_R4`) is a clear win on in-RAM MoE — it activates the AVX-VNNI 256-bit GEMM kernels merged earlier (ikawrakow#1467, ikawrakow#1474, ikawrakow#1482, ikawrakow#1472, ikawrakow#1578), giving a measurable PP boost on Zen4 / Sapphire Rapids+ hardware. On models that overflow physical RAM the same flag is a footgun: the repack pass reads every tensor sequentially, which in turn page-faults swapped tensors back into RAM and evicts others, causing thrashing. On Ryzen 9 7950X (96 GiB RAM) we see TG drop ~50% on a 151 GiB MiniMax quant when `-rtr 1` is on vs off. Users hit this regularly because the flag is a recommended performance knob with no in-product warning. Behaviour --------- `-rtr` now accepts an optional value: -rtr 0 / off disable run-time repack -rtr 1 / on enable, no safety check (legacy) -rtr auto enable, but auto-disable when the model is a MoE that exceeds 90% of physical RAM (probed at load time) The bare `-rtr` form keeps the legacy behaviour (`= 1`). When `auto` decides to disable repack it logs: llama_model_load: --run-time-repack auto: disabled (MoE model 151.0 GiB > 90% of RAM 95.1 GiB) Implementation -------------- * `gpt_params::repack_tensors_auto` and matching field on `llama_model_params` carry the request through to the model loader. * `llama_get_total_ram_bytes()` is a small per-OS helper: - Windows: `GlobalMemoryStatusEx().ullTotalPhys` - Linux: `sysconf(_SC_PHYS_PAGES) * sysconf(_SC_PAGE_SIZE)` - macOS: `sysctl({CTL_HW, HW_MEMSIZE})` - other: returns 0 → caller treats it as "unknown, do nothing" * `llama_rtr_auto_should_disable()` probes the GGUF metadata-only via a short-lived `llama_model_loader` (mmap, no repack), reads arch and hparams, and returns true only if all of these hold: - `repack_tensors_auto` was actually requested, - GPU offload is *not* in play (`n_gpu_layers <= 0`); see "GPU offload" below, - we successfully read total physical RAM, - the probe identifies the model as a MoE (`n_expert > 0 && n_expert_used > 0`), - on-disk model size > 90% of physical RAM. Any failure path (probe exception, RAM unknown, dense model, fits in RAM) returns false and leaves the user-supplied `-rtr` setting alone. * `llama_model_load` consults `llama_rtr_auto_should_disable()` once before constructing the real loader and flips `params.repack_tensors` off in-place when the policy fires. GPU offload ----------- When the user is doing GPU+CPU split inference (`-ngl > 0`, `-cmoe`/`-ncmoe`, etc.) the on-disk model size is no longer a good proxy for the CPU-side RAM footprint, so the auto policy steps out of the way and lets the user's explicit `-rtr` choice stand. This is the conservative call: we'd rather miss an opportunity to auto-disable than auto-disable a build where the CPU side easily fits. Validation ---------- Tested on Ryzen 9 7950X (96 GiB), MSVC 2022 build: * In-RAM Qwen3-30B Q4_K_M (17 GiB) with `-rtr auto` keeps repack enabled. PP/TG numbers indistinguishable from `-rtr 1` (within r=3 stddev). * MiniMax-M2.5 Variant-A (123.6 GiB) with `-rtr auto` correctly auto-disables and logs the reason. * Qwen3-30B with `-rtr auto -ngl 1` keeps repack enabled (GPU offload skip path). * File-not-found probe falls back to the user's setting and prints a single warning. Linux and macOS paths are code-only (compile-tested) — happy to follow up with a Docker-based smoke test if useful. Threshold rationale ------------------- The 0.9 buffer is a deliberate compromise: too tight (e.g. 0.95) and we miss models that are already swap-bound after kernel/other-process RAM use; too loose (e.g. 0.8) and we flip off rtr on plenty of models that would still have benefited. On Linux/Windows desktops "background overhead" tends to be in the 5–10% range, which is what 0.9 covers. The value is hard-coded for now; happy to expose it via env var if there's demand. Out of scope ------------ This PR does not change the default value of `-rtr` (still `0`), and does not alter the bare `-rtr` form (still `= 1`). Adding `auto` as the new default could be a separate follow-up once the policy has been in the field for a while.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add a
HAVE_VNNI256code path for the Q6_K R4 kernel, replacing AVX2_mm256_maddubs_epi16+_mm256_madd_epi16with_mm256_dpbusd_epi32, and_mm256_madd_epi16with_mm256_dpwssd_epi32for the bsums correction. The existingHAVE_FANCY_SIMD(AVX-512 VNNI) path is preserved unchanged.This follows the v2 approach used in PR #1472 (Q3_K R4 VNNI256), which is currently awaiting review.
Benchmark
Qwen3.5-2B Q6_K, rtr=1:
QA
Qwen3.5-2B Q6_K, --run-time-repack, comparing baseline (56e026f) and PR builds (ce35079):