Tried -ser to cut expert streaming on a 1T MoE running from Optane. It did not help much, and -ser looks like a no-op on main. What are we missing?

[mb8565]

We run large MoE from Optane on a box with 376 GiB RAM, CPU only. Kimi-K2.7 (~378 GiB) does not fit, so its routed experts stream per token while the fixed weights (attention, shared expert, dense, output) and the KV cache stay resident. Cutting active experts looked like the obvious speed lever, so we tried -ser. (We also tested GLM-5.2 IQ4_XS at ~337 GiB, which fits in RAM, so we only force-streamed it with a memory cgroup where noted.) Two things we are unsure about.

1. -ser seems to be a no-op on current main. Output and timings are identical with and without it. Various fused ops around expert selection #840 moved expert selection to the fused ggml_top_k path and the ggml_top_k_thresh call in llm_build_moe_ffn looks left commented out, so the flag parses but the graph appears unchanged. Intentional, or did we misread it?

2. When we patched it back on to test the idea, it did not pan out. We restored the thresh path and guarded two ops that do not skip the -1 dropped-expert sentinel (add_id gpt-oss, mul_multi_add Gemma4). Then we measured top-1 token agreement on a private 2026 codebase (n=24 chunks), not wikitext, where dropping experts had looked misleadingly free:

experts kept (of 8)	Kimi-K2.7	GLM-5.2
7 (drop 1)	92.8%	-
6 (drop 2)	89.6%	91.2%
4 (drop 4)	81.9%	83.3%
2 (drop 6)	55.3%	56.8%

Even the mildest setting cost ~7-10 points, on both what looks like a peaked router (Kimi) and a flatter one (GLM). On speed, for the model that genuinely streams (Kimi at 378 GiB, over our 376 GiB RAM), the mild settings bought only ~+8-19% TG (roughly 1.5 to 1.8 tok/s). We could get more, around 2.2x at the aggressive keep-2 setting, but only there, where agreement fell to ~55% and the code clearly degraded. For our goal of faster streamed 1T without hurting code quality, the safe settings barely helped, and the ones that were actually fast weren't safe.

Could easily be us:

• Is there a better way to cut streamed-expert I/O, or are we just using -ser wrong?
• Should we expect the wikitext-free vs code-costly gap?
• And is the Various fused ops around expert selection #840 no-op intentional?

Happy to share the commands, the harness, and the raw numbers if that helps.

[dmaivel]

If faster streamed 1T is your goal, would you mind testing with #2101's --prefetch-experts? It probably won't help with TG, but it would be interesting to see if/how much it improves PP performance for you.

I am not sure if adding -ser support back is necessary; can't you achieve the same behavior using --override-kv <arch>.expert_used_count=int:N where N is the # of experts kept (where in your case, arch is deepseek2 for Kimi and glm-dsa for GLM)? or is -ser doing something more here?

In my own testing, I don't think one should drop below N=6, the quality loss isn't really worth it, especially with quantization. Though YMMV.

[ikawrakow]

The -ser option has been deprecated for two reasons

1. I have observed that people just use -ser N,0. This brings absolutely nothing beyond just -override-kv <arch>.expert_used_count=int:N mentioned by @dmaivel
2. SER was not supported by the new (and much more efficient for small batches) CUDA matrix multiplication implementation. Given 1, I did not feel like adding the support (quite involved due to the variable number of experts per token).

If you just re-enable it, you are basically getting lucky that it did not crash. If it did not crash, the inference result may or may not have anything to do with the expected result.

[mb8565]

Thanks both, that clears it up. Confirmed on our end.

--override-kv deepseek2.expert_used_count=int:N does what we were after, it reduces the active experts
(verified n_expert_used = 8 -> 7 -> 6) with no -ser involved. On our streamed Kimi-K2.7-Code IQ3_S
(~378 GiB, mmap'd from an Optane drive, CPU inference) that shows up directly in decode throughput, since
fewer active experts means fewer expert pages streamed per token:

  experts   TG (t/s)      top-1 agreement vs keep-8 (held-out code)
  8         3.14          baseline
  7         3.45 (+10%)   92.4%
  6         3.58 (+14%)   89.4%

(TG is the mean of 2 sweep-bench runs, --interleave=all.) For quality we leaned on top-1 token agreement
against the full 8-expert model on a private held-out code corpus, since PPL felt like an awkward gate
here. The wikitext number looks off to us: Kimi scores about 1.6 on it, which seems impossibly low, and
our best guess is it has largely memorized that corpus. On the held-out code the PPL deltas are small
(about +1.0% at keep-7, +3.4% at keep-6), and we weren't sure that told us much about how much the output
actually shifts. Agreement was easier for us to reason about: keep-7 changes about 8% of tokens, keep-6
about 11%. That cost looks modest but real, so override-kv lets us dial the tradeoff, and in our runs
keep-7 was the mildest cut and the one we'd reach for first. We agree with not going below 6. No -ser
needed, and since it's deprecated on purpose we won't.

On --prefetch-experts (#2101), we tried it on the same streamed Kimi (cold page cache, -ngl 0, mmap from
Optane). In our sweep-bench run it went the other way for PP, steady-state PP dropped ~23.6 -> 17.4 t/s and
TG nudged up ~3.24 -> 3.45. Looks like the read-ahead populate engine is competing with the prefill compute
for CPU and memory bandwidth in steady state. I don't think our throughput metric is the right one here
though, prefetch is presumably about overlapping the one-time cold expert faults on the first prefill
(TTFT), and a multi-chunk steady-state PP number hides that. Is there a fairer way you'd measure it, TTFT
on a cold model, or a specific batch/thread setup? Happy to re-run.

[dmaivel]

If you do wish to do a re-run for the expert prefetching, it would be helpful to know more about the hardware you're using (cpu, drive, etc..) and the command you use before I can recommend anything.

[mb8565]

Sure, here's the setup.

Hardware is a dual-socket Intel Xeon Platinum 8260 (Cascade Lake, 24 cores / 48 threads per socket, 96
threads total, AVX-512), with 376 GiB of DDR4 split across the two NUMA nodes (about 192 GiB each). The
model sits on an Intel Optane persistent-memory drive (/dev/pmem1, App Direct, ext4 with noatime) and is
mmap'd from there. It's Kimi-K2.7-Code IQ3_S at about 378 GiB, so it's just over total RAM (376 GiB) and
genuinely streams from the Optane rather than fitting in page cache, which is why the expert paging shows
up at all.

The run is CPU-only (-ngl 0) with llama-sweep-bench, dropping caches before each run so the model starts
cold:

sync; echo 3 > /proc/sys/vm/drop_caches
numactl --interleave=all ./llama-sweep-bench -m Kimi-K2.7-Code-IQ3_S-00001-of-00010.gguf \
  -ngl 0 -t 48 -fa on -ctk q8_0 -ctv q8_0 -c 2048 -ub 512 -b 2048 --prefetch-experts

The baseline is the same line without --prefetch-experts. Happy to change any of it (thread count, batch
shape, or run it through llama-cli/llama-server instead of sweep-bench) if a different setup would isolate
the prefetch effect better. Thanks for taking a look.

[dmaivel]

Considering you don't have offloading (and are using unique hardware), I'm not sure how much prefetching will improve your performance. You can try experimenting with GGML_MOE_PREFETCH_DEBUG=1 and trying combinations of:

• disabling lookahead (GGML_MOE_PREFETCH_AHEAD=0)
• changing the prefetch thread count (--prefetch-experts-threads N, where you might wanna try N=1, N=2, ..)
• increasing batch size (I aim for atleast -b 4096 -ub 4096 but I always use hybrid so unsure if this will help with your setup)

Though, if none of these really help for you I think its just a hardware setup problem. The feature is useful in hybrid setups because prefetching occurs on the CPU while the GPU is busy computing the current batch, making page cache warming practically free. On CPU-only you don't really have idle headroom, so the prefetch threads are likely competing with your computing threads for the same cores and memory bandwidth rather than overlapping with anything.