runtime : add --run-time-repack auto mode for swap-bound MoE safety

[AndrewMoryakov]

• I have read the contributing guidelines
• Self-reported review complexity:
  • Low
  • Medium
  • High

What

Adds a third value (auto) to --run-time-repack / -rtr, plus an -rtra / --run-time-repack-auto alias.

-rtr auto enables run-time repack only when the metadata-only safety policy can determine that it is safe for the current load. If the policy determines that repack would likely turn a mmap-friendly swap-bound MoE load into an anonymous-memory load, it disables repack and leaves the user/default mmap setting intact.

The bare -rtr form is unchanged and still means legacy -rtr 1.

Behaviour

Value	Behaviour
0 / off	disable run-time repack
1 / on	enable, no safety check (legacy)
auto	enable, but disable when the safety policy says CPU-resident repack would exceed available memory, or when the policy cannot decide safely

The default when -rtr is absent is unchanged (0).

Policy

The auto policy now follows maintainer feedback from the PR discussion:

• uses available/effective memory, not installed physical RAM;
• accounts for tensor overrides;
• uses quantization type eligibility via iqk_repacked_type();
• counts CPU-resident repackable tensor bytes instead of full on-disk model bytes.

The policy probes GGUF metadata only, walks probe.weights, resolves simple CPU/GPU placement, and compares against 90% of currently available memory.

Placement resolution is intentionally bounded:

• manual tensor_buft_overrides are applied first, using the same std::regex_search / first-match semantics as the loader;
• simple LLAMA_SPLIT_MODE_LAYER / LLAMA_SPLIT_MODE_NONE placement is modeled from n_gpu_layers, output-layer rules, and already-populated model.devices;
• -ncmoe is handled for known simple cases, including all/first-N expert CPU overrides and single-device layer split;
• --fit, merged tensor modes, graph/attention split modes, and multi-device -ncmoe layer distribution are treated as unknown and therefore safety-disable repack.

Memory Gates

Primary gate:

• CPU-resident repackable bytes > 90% available memory => disable repack.

Secondary gate:

• total CPU-resident tensor bytes > 90% available memory => disable repack.

The secondary gate exists because the current loader still forces use_mmap=false whenever repack_tensors=true; even if only a subset of tensors is actually repackable, keeping repack enabled can make all CPU-resident tensors lose mmap.

Unknown also disables repack. This is intentional for explicit -rtr auto: a safety mode should not fall back to the same failure mode as plain -rtr 1.

Available Memory Helper

• Windows: MEMORYSTATUSEX::ullAvailPhys
• Linux: /proc/meminfo MemAvailable, fallback to _SC_AVPHYS_PAGES, capped by cgroup v2/v1 headroom when present
• macOS: host_statistics64 using (free_count + inactive_count) * page_size

Mmap Caveat

This PR fixes the repack_tensors=true => use_mmap=false path for -rtr auto, but it does not fully decouple repack from mmap internally.

There are still independent loader paths that can disable mmap. In particular, CPU tensor overrides can disable mmap unless GGML_CUDA_NO_PINNED=1 is set. The auto policy now logs a warning for that case when it disables repack.

A full design where only repackable CPU tensors are copied to writable memory while non-repackable CPU tensors remain mmap-backed is larger loader work and intentionally out of scope here.

Validation

Current pushed head: 1441ee9a9.

Validated locally on Windows / MSVC 2022:

• nmake llama-cli succeeds;
• llama-cli --help shows -rtr, --run-time-repack [0|1|auto].

I do not have a local GPU machine, so the exact -ngl 99 -ot exps=CPU runtime reported by dmaivel is not runtime-validated here. The code path is covered by the placement resolver through the manual tensor override handling.

Out of Scope

• Changing the default value of -rtr.
• Making legacy -rtr self-protective.
• Fully decoupling runtime repack from mmap.
• Replacing the offline-repack recommendation for low-RAM users who want repacked quantization types with mmap.

[Ph0rk0z]

I guess only part is annoyingly having to add 1 after -rtr

[AndrewMoryakov]

@Ph0rk0z thanks for the early flag — to make sure I'm reading you right:

The bare -rtr form (no value after it) is preserved in this PR — it still means -rtr 1 exactly like before, so existing scripts and READMEs do not need to change. The optional value is only consumed when the next argv token is one of 0/1/off/on/auto; anything else (or end-of-args) leaves params.repack_tensors = true, params.repack_tensors_auto = false, which is the legacy fast path.

You can verify in the diff: common/common.cpp — the parser only does ++i to consume the value when is_mode_token is true.

So in practice:

Command	Behaviour
-rtr	legacy on (= 1)
-rtr 0 / -rtr off	off
-rtr 1 / -rtr on	on
-rtr auto	new safety-net mode
-rtra / --run-time-repack-auto	shortcut for -rtr auto

If you meant something different — e.g. you'd like the default of -rtr to be auto rather than 0, or you'd like a different short-form spelling — happy to adjust. The default (when -rtr is absent entirely) is unchanged on purpose in this PR; making auto the new default would be a worthwhile follow-up once the policy has been in the field for a bit.

[AndrewMoryakov]

Self-review found a real use_mmap regression — force-pushed a fix.

The bug: the parser was unconditionally setting params.use_mmap = false before parsing the optional value, which leaks two ways:

1. -rtr 0 (explicit disable) was also disabling mmap as a side effect — surprising because the user said "no repack".
2. -rtr auto was setting use_mmap = false at parse time, but if the auto policy then disabled repack at load (the whole point of the safety net on swap-bound models!), the load would still proceed with mmap = false — which on a model that already overflows RAM is exactly the wrong thing.

The fix (commit 0115ace21):

• common/common.cpp: rework the -rtr parser to determine the final state first, and only force use_mmap = false when repack is actually going to run unconditionally (legacy -rtr / -rtr 1). For -rtr 0 and -rtr auto we now leave params.use_mmap alone.
• src/llama.cpp: in llama_model_load, after the auto policy decides:
  • If auto keeps repack enabled → set use_mmap = false to match the legacy -rtr 1 coupling (repack pass writes back into tensor buffers).
  • If auto disables repack → leave use_mmap as the user configured it, so swap-bound MoE actually loads via mmap as intended.

End result is the four-way matrix that I think people would expect:

Command	repack	use_mmap (after load)
(no -rtr)	off	default true
-rtr 0	off	default true (was: false 🐛)
-rtr 1	on	false (legacy)
-rtr auto (in-RAM)	on (auto kept it)	false (matches -rtr 1)
-rtr auto (swap-bound)	off (auto disabled it)	default true (was: false 🐛)

Diff is +24 / -11 on top of the original commit; rebuilt cleanly on MSVC 2022 (Zen4); smoke-tested both the in-RAM and swap-bound paths via llama-bench/llama-cli.

Sorry for the noise — no changes to the API surface, only the internal coupling between repack and mmap got tightened.

[dmaivel]

Given that -rtr requires mmap to be disabled, I’m not sure users running swap-bounded MoE models are likely to specify it in the first place.

I tested -rtr auto with a model that does not fit in RAM, and it behaved the same as -rtr on, which seems problematic. This was on a Linux system with 64 GB of RAM and a ~120 GB model:

GGML_CUDA_NO_PINNED=1 ./llama-server \
    --model ~/.../Qwen3.5-397B-A17B-IQ2_XS-00001-of-00004.gguf \
    -c 4096 -ngl 99 -ot exps=CPU \
    -b 4096 -ub 4096 \
    -rtr auto

EDIT: For the above, I read your note about GPU+CPU split inference; if the path can't determine whether repacking should be enabled or disabled, isn't the wiser choice to disable? Otherwise, the user is left with the same failure message they would have gotten if they just used -rtr in its current form.

Also, checking total system memory is not sufficient because the allocation can still fail if other processes are consuming a significant amount of RAM. In that case, checking available memory would be more relevant.

I'm not sure it's necessary, but, if we do want to guard -rtr, would it make more sense to check whether there is enough available RAM and disable it automatically when there is not? That would avoid changing the argument to require on/off/auto; instead, we could simply print a message saying that rtr was disabled due to insufficient memory.

[AndrewMoryakov]

Thanks for catching this on a real workload. The reproducer is correct, and the current policy is wrong on it.

There are two concrete bugs in the code at 0115ace.

First, the auto path skips the check whenever n_gpu_layers > 0. The intent was to avoid false positives when GPU offload moves most weights out of CPU RAM, but the check only looks at n_gpu_layers and ignores tensor_buft_overrides. In your command, -ngl 99 offloads the non-expert parts, while -ot exps=CPU moves the expert weights back to CPU. That is still a swap-bound CPU-side load. The early skip returns false, the auto path keeps repack enabled, and llama_model_load then forces use_mmap=false, so it behaves like plain -rtr.

Second, the policy uses total physical RAM. That is not the right value for an allocation-safety decision. A busy system, page cache pressure, or container limits can make the effective available memory much lower than MEMORYSTATUSEX::ullTotalPhys or sysconf(_SC_PHYS_PAGES). The check should use available/effective memory instead.

I also agree with your edited point about uncertainty. If the auto path cannot determine that repack is safe, keeping repack enabled leaves the user with the same failure mode as -rtr. For an explicit safety mode, unknown should disable repack, preserve mmap, and log a warning.

If we keep the explicit -rtr auto design, the focused fix is:

1. use available/effective memory instead of total RAM;
2. remove the n_gpu_layers > 0 skip;
3. distinguish keep / disable / unknown instead of returning a bool;
4. make unknown safety-first for -rtr auto;
5. update the log/help text to say available memory.

There is still a conservative false-positive risk because the current probe compares full GGUF tensor bytes, not exact CPU-resident bytes after placement. That is safer than the current false negative, but a fully precise solution would need a later placement-aware estimator.

On the larger question: making -rtr itself self-protective is reasonable, but it changes legacy behavior and is not just a small patch on top of the current PR. The current legacy parser sets use_mmap=false for -rtr at parse time. If we add a load-time safety check without moving that coupling, we can still end up with repack disabled but mmap also disabled. A self-protective -rtr design should leave mmap untouched in the parser and resolve mmap only after the safety decision.

@ikawrakow, which direction would you prefer?

(a) keep the explicit -rtr auto mode and apply the focused fix above;
(b) rewrite this so existing -rtr becomes self-protective;
(c) defer this to a larger placement-aware design based on estimated CPU-resident bytes.

If there is no strong preference, I will go with (a) as the smallest reviewable change.

[ikawrakow]

In order this to be useful, it needs to consider actually available RAM and take into account tensor overrides and quantization types (i.e., use only the tensors that will get repacked when computing required memory).

Also, if I was a user with a low-RAM system where the model that I want to use does not fit in RAM, but I want to use the repacked quantization types, I would just offline repack the model using

./bin/llama-quantize --repack $model_in $model_out type

This would allow me to use the repacked model with mmap, which would be much better than swapping in and out.

[AndrewMoryakov]

Thanks for the direction. Plan is to walk probe.weights, filter by iqk_repacked_type() and resolve placement against tensor_buft_overrides / n_gpu_layers / -ncmoe, then compare CPU-resident repackable bytes against available memory.

One thing I want to flag. The loader forces use_mmap = false whenever repack_tensors=true (src/llama-model-loader.cpp:585), so a KEEP based on repackable bytes alone can still trigger swap when total CPU-resident bytes exceed available memory. I plan to AND-check total CPU-resident bytes as a secondary gate against the same threshold; the repackable-bytes metric stays primary and matches your guidance. Say so if you'd rather drop the total check or decouple repack from mmap separately.

For context: I run a CPU-only Zen4 box with 96 GiB RAM and no GPU. I cannot runtime-validate the -ngl 99 -ot exps=CPU configuration dmaivel reported, since exercising that path needs an actual GPU. The placement resolver handles it in code based on tensor_buft_overrides regex matching, but I have no way to run the case locally. Smoke covers in-RAM, swap-bound, and multi-shard cases on CPU only.

[AndrewMoryakov]

Pushed an update based on the maintainer feedback and dmaivel's reproducer. Current head is 1441ee9a9.

What changed since 0115ace21:

• the policy now uses available/effective memory instead of installed RAM;
• the decision is tri-state-ish: unsafe disables repack, unknown also disables repack with a WARN;
• the check walks probe.weights and filters by iqk_repacked_type() so the primary gate is CPU-resident repackable bytes, not full GGUF size;
• manual tensor_buft_overrides are applied with loader-compatible regex first-match semantics, so -ngl 99 -ot exps=CPU is no longer skipped just because n_gpu_layers > 0;
• simple layer placement is modeled for LAYER / NONE split modes; complex placement modes (--fit, merge modes, graph/attention split, multi-device -ncmoe distribution) are treated as unknown and safety-disable repack;
• there is also a secondary total CPU-resident bytes gate because the current loader still forces use_mmap=false whenever repack_tensors=true.

I also updated the PR description because the original body still described the old total-RAM / GPU-skip heuristic.

Local validation: MSVC 2022 nmake llama-cli succeeds, and llama-cli --help shows the updated -rtr [0|1|auto] syntax. I still cannot runtime-test the exact GPU reproducer locally; that path is covered by the override placement logic, not by hardware validation on my side.

[AndrewMoryakov]

Pushed a small follow-up cleanup at 15e04decd.

Three changes addressing reviewer feedback on 1441ee9a9:

1. Dropped the dense if (!is_moe) return NOT_APPLICABLE fast-exit. Dense models now go through the same byte-count walk as MoE. If the dense quant has repackable types and fits, the policy keeps repack; if it has none, the policy disables repack instead of leaving repack=true and silently forcing mmap=false in the loader. Closes the swap-thrash edge case on large dense models that lack repackable types in the current build.

2. Moved the unsupported-mode UNKNOWN exits (--fit, --merge-qkv, --merge-up-gate-experts, split_mode ATTN/GRAPH, available-memory query, and regex compile) before the metadata probe. The probe is real I/O on multi-shard models; bailing out early avoids paying it just to return UNKNOWN.

3. Removed a redundant if (cpu_repackable_bytes <= threshold) wrapper around the KEEP return and the unreachable trailing UNKNOWN. After the two upstream > checks the condition is always true.

Also dropped NOT_APPLICABLE from the decision enum and its caller-side case (now unreachable). The decision is a clean tri-state: KEEP / DISABLE / UNKNOWN.

Smoke matrix on CPU-only Zen4 96 GiB, all green:

Setup	Decision
Qwen3-30B Q4_K_M (17 GiB in-RAM)	KEEP
Qwen3.5-27B Q8_0 dense (29 GiB)	KEEP (was NOT_APPLICABLE before)
MiniMax M2.5 (90 GiB)	DISABLE via total-bytes gate
Qwen3-30B + --merge-qkv	UNKNOWN, early exit (no probe)
Qwen3-30B + -ot 'attn.*=CPU'	KEEP, override placement applied
gpt-oss-20b MXFP4 (11 GiB)	KEEP, repackable 1.2 GiB only

The -ngl 99 -ot exps=CPU reproducer still cannot be runtime-validated on my CPU-only box; the override placement logic was exercised in the -ot 'attn.*=CPU' smoke above.