Enable AVX-VNNI 256-bit path for IQ4_NL R4 matmul#1467
Merged
Conversation
ikawrakow
approved these changes
Mar 20, 2026
4 tasks
AndrewMoryakov
added a commit
to AndrewMoryakov/ik_llama-pr
that referenced
this pull request
May 5, 2026
Adds a third value (`auto`) to the `--run-time-repack` / `-rtr` CLI flag, along with a matching `-rtra` / `--run-time-repack-auto` alias, that turns the existing repack-on-load behaviour into something safe to leave on by default. Motivation ---------- `-rtr 1` (run-time repack into row-interleaved variants such as `Q4_K_R4`) is a clear win on in-RAM MoE — it activates the AVX-VNNI 256-bit GEMM kernels merged earlier (ikawrakow#1467, ikawrakow#1474, ikawrakow#1482, ikawrakow#1472, ikawrakow#1578), giving a measurable PP boost on Zen4 / Sapphire Rapids+ hardware. On models that overflow physical RAM the same flag is a footgun: the repack pass reads every tensor sequentially, which in turn page-faults swapped tensors back into RAM and evicts others, causing thrashing. On Ryzen 9 7950X (96 GiB RAM) we see TG drop ~50% on a 151 GiB MiniMax quant when `-rtr 1` is on vs off. Users hit this regularly because the flag is a recommended performance knob with no in-product warning. Behaviour --------- `-rtr` now accepts an optional value: -rtr 0 / off disable run-time repack -rtr 1 / on enable, no safety check (legacy) -rtr auto enable, but auto-disable when the model is a MoE that exceeds 90% of physical RAM (probed at load time) The bare `-rtr` form keeps the legacy behaviour (`= 1`). When `auto` decides to disable repack it logs: llama_model_load: --run-time-repack auto: disabled (MoE model 151.0 GiB > 90% of RAM 95.1 GiB) Implementation -------------- * `gpt_params::repack_tensors_auto` and matching field on `llama_model_params` carry the request through to the model loader. * `llama_get_total_ram_bytes()` is a small per-OS helper: - Windows: `GlobalMemoryStatusEx().ullTotalPhys` - Linux: `sysconf(_SC_PHYS_PAGES) * sysconf(_SC_PAGE_SIZE)` - macOS: `sysctl({CTL_HW, HW_MEMSIZE})` - other: returns 0 → caller treats it as "unknown, do nothing" * `llama_rtr_auto_should_disable()` probes the GGUF metadata-only via a short-lived `llama_model_loader` (mmap, no repack), reads arch and hparams, and returns true only if all of these hold: - `repack_tensors_auto` was actually requested, - GPU offload is *not* in play (`n_gpu_layers <= 0`); see "GPU offload" below, - we successfully read total physical RAM, - the probe identifies the model as a MoE (`n_expert > 0 && n_expert_used > 0`), - on-disk model size > 90% of physical RAM. Any failure path (probe exception, RAM unknown, dense model, fits in RAM) returns false and leaves the user-supplied `-rtr` setting alone. * `llama_model_load` consults `llama_rtr_auto_should_disable()` once before constructing the real loader and flips `params.repack_tensors` off in-place when the policy fires. GPU offload ----------- When the user is doing GPU+CPU split inference (`-ngl > 0`, `-cmoe`/`-ncmoe`, etc.) the on-disk model size is no longer a good proxy for the CPU-side RAM footprint, so the auto policy steps out of the way and lets the user's explicit `-rtr` choice stand. This is the conservative call: we'd rather miss an opportunity to auto-disable than auto-disable a build where the CPU side easily fits. Validation ---------- Tested on Ryzen 9 7950X (96 GiB), MSVC 2022 build: * In-RAM Qwen3-30B Q4_K_M (17 GiB) with `-rtr auto` keeps repack enabled. PP/TG numbers indistinguishable from `-rtr 1` (within r=3 stddev). * MiniMax-M2.5 Variant-A (123.6 GiB) with `-rtr auto` correctly auto-disables and logs the reason. * Qwen3-30B with `-rtr auto -ngl 1` keeps repack enabled (GPU offload skip path). * File-not-found probe falls back to the user's setting and prints a single warning. Linux and macOS paths are code-only (compile-tested) — happy to follow up with a Docker-based smoke test if useful. Threshold rationale ------------------- The 0.9 buffer is a deliberate compromise: too tight (e.g. 0.95) and we miss models that are already swap-bound after kernel/other-process RAM use; too loose (e.g. 0.8) and we flip off rtr on plenty of models that would still have benefited. On Linux/Windows desktops "background overhead" tends to be in the 5–10% range, which is what 0.9 covers. The value is hard-coded for now; happy to expose it via env var if there's demand. Out of scope ------------ This PR does not change the default value of `-rtr` (still `0`), and does not alter the bare `-rtr` form (still `= 1`). Adding `auto` as the new default could be a separate follow-up once the policy has been in the field for a while.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
IQ4_NL has a special kernel in repacked (R4) mode (mul_mat_iq4_nl_r4_q8_2).
It currently has a FANCY_SIMD path that requires AVX-512, here we update the fallback AVX2 path to have a conditionally VNNI accelerated path on AVX-VNNI CPUs.
Benchmarks
Model: Qwen3.5-2B IQ4_NL, pp512
rtr 0 (control - different kernel that is already VNNI optimized)
rtr 1 (runtime repack - uses the newly optimized kernel)
Big improvement here, though rtr 0 is still faster on my hardware for this quant.
Text generation QA
Text generation QA with llama-cli across multiple prompts shows bit-identical results. Full perplexity against wikitest-2 is unchanged as well (13.1025 +/- 0.09740).