Skip to content

Enable AVX-VNNI 256-bit path for IQ3_XXS and IQ3_S R4 matmul#1474

Merged
ikawrakow merged 1 commit into
ikawrakow:mainfrom
accaldwell:ac/vnni_iq3_r4
Mar 20, 2026
Merged

Enable AVX-VNNI 256-bit path for IQ3_XXS and IQ3_S R4 matmul#1474
ikawrakow merged 1 commit into
ikawrakow:mainfrom
accaldwell:ac/vnni_iq3_r4

Conversation

@accaldwell

@accaldwell accaldwell commented Mar 20, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Adding HAVE_VNNI256 optimized paths for mul_mat_iq3_xxs_r4_q8_k and mul_mat_iq3_s_r4_q8_k kernels.

The method we take here is very similar to some of my previous PRs, adding and conditionally using an optimized dpbusd path instead of the multi-instruction AVX2 alternative.

FANCY is doing its own thing above the respective new VNNI code blocks and shouldn't reach this new code.

Performance

Sweep bench (Qwen3.5-2B IQ3_XS, 6 P-core threads, --run-time-repack, N_KV 0-16384):

sweep

Small but solid boost to pp and the data is suggestive of a tiny increase in tg as well

QA

  • Token generation: identical output across 4 prompts (deterministic, seed=42)
  • Perplexity: identical, 14.0944 +/- 0.10305

@accaldwell accaldwell marked this pull request as ready for review March 20, 2026 09:38
@ikawrakow ikawrakow merged commit ac4d6b9 into ikawrakow:main Mar 20, 2026
@AndrewMoryakov AndrewMoryakov mentioned this pull request May 4, 2026
Open
4 tasks
AndrewMoryakov added a commit to AndrewMoryakov/ik_llama-pr that referenced this pull request May 5, 2026
@AndrewMoryakov
runtime : add --run-time-repack auto mode for swap-bound MoE safety
0115ace 
Adds a third value (`auto`) to the `--run-time-repack` / `-rtr` CLI flag,
along with a matching `-rtra` / `--run-time-repack-auto` alias, that
turns the existing repack-on-load behaviour into something safe to leave
on by default.

Motivation
----------

`-rtr 1` (run-time repack into row-interleaved variants such as
`Q4_K_R4`) is a clear win on in-RAM MoE — it activates the AVX-VNNI
256-bit GEMM kernels merged earlier (ikawrakow#1467, ikawrakow#1474, ikawrakow#1482, ikawrakow#1472, ikawrakow#1578),
giving a measurable PP boost on Zen4 / Sapphire Rapids+ hardware.

On models that overflow physical RAM the same flag is a footgun: the
repack pass reads every tensor sequentially, which in turn page-faults
swapped tensors back into RAM and evicts others, causing thrashing.
On Ryzen 9 7950X (96 GiB RAM) we see TG drop ~50% on a 151 GiB MiniMax
quant when `-rtr 1` is on vs off. Users hit this regularly because the
flag is a recommended performance knob with no in-product warning.

Behaviour
---------

`-rtr` now accepts an optional value:

  -rtr 0 / off    disable run-time repack
  -rtr 1 / on     enable, no safety check (legacy)
  -rtr auto       enable, but auto-disable when the model is a MoE that
                  exceeds 90% of physical RAM (probed at load time)

The bare `-rtr` form keeps the legacy behaviour (`= 1`).

When `auto` decides to disable repack it logs:

  llama_model_load: --run-time-repack auto: disabled
                    (MoE model 151.0 GiB > 90% of RAM 95.1 GiB)

Implementation
--------------

* `gpt_params::repack_tensors_auto` and matching field on
  `llama_model_params` carry the request through to the model loader.
* `llama_get_total_ram_bytes()` is a small per-OS helper:
  - Windows: `GlobalMemoryStatusEx().ullTotalPhys`
  - Linux:   `sysconf(_SC_PHYS_PAGES) * sysconf(_SC_PAGE_SIZE)`
  - macOS:   `sysctl({CTL_HW, HW_MEMSIZE})`
  - other:   returns 0 → caller treats it as "unknown, do nothing"
* `llama_rtr_auto_should_disable()` probes the GGUF metadata-only via a
  short-lived `llama_model_loader` (mmap, no repack), reads arch and
  hparams, and returns true only if all of these hold:
  - `repack_tensors_auto` was actually requested,
  - GPU offload is *not* in play (`n_gpu_layers <= 0`); see "GPU offload"
    below,
  - we successfully read total physical RAM,
  - the probe identifies the model as a MoE (`n_expert > 0 &&
    n_expert_used > 0`),
  - on-disk model size > 90% of physical RAM.
  Any failure path (probe exception, RAM unknown, dense model, fits in
  RAM) returns false and leaves the user-supplied `-rtr` setting alone.
* `llama_model_load` consults `llama_rtr_auto_should_disable()` once
  before constructing the real loader and flips `params.repack_tensors`
  off in-place when the policy fires.

GPU offload
-----------

When the user is doing GPU+CPU split inference (`-ngl > 0`,
`-cmoe`/`-ncmoe`, etc.) the on-disk model size is no longer a good
proxy for the CPU-side RAM footprint, so the auto policy steps out of
the way and lets the user's explicit `-rtr` choice stand. This is the
conservative call: we'd rather miss an opportunity to auto-disable than
auto-disable a build where the CPU side easily fits.

Validation
----------

Tested on Ryzen 9 7950X (96 GiB), MSVC 2022 build:

* In-RAM Qwen3-30B Q4_K_M (17 GiB) with `-rtr auto` keeps repack
  enabled. PP/TG numbers indistinguishable from `-rtr 1` (within r=3
  stddev).
* MiniMax-M2.5 Variant-A (123.6 GiB) with `-rtr auto` correctly
  auto-disables and logs the reason.
* Qwen3-30B with `-rtr auto -ngl 1` keeps repack enabled (GPU offload
  skip path).
* File-not-found probe falls back to the user's setting and prints a
  single warning.

Linux and macOS paths are code-only (compile-tested) — happy to follow
up with a Docker-based smoke test if useful.

Threshold rationale
-------------------

The 0.9 buffer is a deliberate compromise: too tight (e.g. 0.95) and we
miss models that are already swap-bound after kernel/other-process RAM
use; too loose (e.g. 0.8) and we flip off rtr on plenty of models that
would still have benefited. On Linux/Windows desktops "background
overhead" tends to be in the 5–10% range, which is what 0.9 covers. The
value is hard-coded for now; happy to expose it via env var if there's
demand.

Out of scope
------------

This PR does not change the default value of `-rtr` (still `0`), and
does not alter the bare `-rtr` form (still `= 1`). Adding `auto` as the
new default could be a separate follow-up once the policy has been in
the field for a while.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ikawrakow ikawrakow ikawrakow approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@accaldwell @ikawrakow