Gemma4 moe pr v3

[cavusmustafa]

PR: Gemma-4 26B MoE support on the llama.cpp OpenVINO backend

Branch: gemma4_moe_pr_v3 (github.com/cavusmustafa/llama.cpp)
Target: ravi9/llama.cpp:dev_backend_openvino
Base commit: c6311c391 (Merge PR #249, build doc change)
Change: 1 commit, 10 files, +569 / −75, all under ggml/src/ggml-openvino/

---

1. What this PR does

Adds support for the Gemma-4 26B-A4B model to the OpenVINO backend. This is a
hybrid-FFN Mixture-of-Experts model that previously did not run on the backend.
After this PR it runs end-to-end on the OpenVINO CPU device (greedy
"Paris is the capital of" → "France"), and the smaller dense Gemma-4 E2B variant
runs on CPU, GPU, and NPU.

The model (why it needs special handling)

• 30 layers, each with a dense shared FFN + a 128-expert top-8 MoE FFN in parallel.
• Interleaved 5×SWA : 1×global attention, QK-norm, sandwich norms, final logit soft-cap.
• Expert weights are 3-D quantized tensors (the hard part):
  • ffn_gate_up_exps.weight → Q4_K, shape [k=2816, m=1408, n_expert=128]
  • ffn_down_exps.weight → Q5_1, shape [k=704, m=2816, n_expert=128]
• Expert weights are ~14.4 GB of the 15.8 GB model — MoE saves compute
  (only 8 experts run per token) but not memory (all 128 must be resident).

---

2. The changes, grouped by problem solved

All changes live in ggml/src/ggml-openvino/. The single commit bundles the work
below (originally developed as separate commits; squashed for the rebase onto the
current dev base).

2.1 Memory: keep 3-D expert weights compressed (avoid OOM)

Files: ggml-quants.cpp/.h, ggml-openvino-extra.cpp/.h, ggml-openvino.cpp,
ggml-decoder.cpp, openvino/op/mul_mat_id.cpp

The naive path decompresses all 128 experts to f32 at compile time (~100 GB → OOM).
The fix builds the expert dequant as a rank-2 [n_expert, m*k] subgraph
(Constant(u4)→Convert→Subtract→Multiply→Reshape) and gathers experts directly on
that rank-2 node, so the plugin folds Gather + dequant into a single
GatherCompressed op — weights stay compressed, only the selected experts are
decompressed at runtime (mul_mat_id.cpp).

• f16 zero-point (ggml-quants.cpp): Q4_K/Q5_1 store an fp min that isn't a
  multiple of scale. Using an integer zero-point corrupts every weight. The fix keeps
  the fusable Subtract form but makes the zero-point an exact f16 value
  zp = −min/scale — algebraically w·scale + min, yet still GatherCompressed-matchable.
• In-place extraction (ggml-openvino-extra.cpp, ggml-openvino.cpp): the extracted
  u4/u8 + f16 scale/zp are written into the backend buffer (not a heap duplicate),
  keeping the compile peak down.

2.2 Correctness: per-expert VIEW handling

Files: openvino/op/view.cpp, ggml-decoder.cpp

The MoE output is 8 ggml_view_2d expert planes summed by a chain of ADDs.

• VIEW not sliced on the non-static path → each "plane" was the whole tensor.
  Fixed: slice the single indexed expert axis and reshape (view.cpp).

(A second bug — 8 same-named expert views colliding in the name-keyed tensor_map —
was fixed during development with a unique VIEW name, but the current dev base now
disambiguates all same-named tensors via get_tensor_ov_name() (#<hash_pos> suffix),
which subsumes it, so this PR no longer carries that change.)

2.3 Dynamic token dim (decode + GPU prefill)

Files: ggml-decoder.cpp, openvino/op/get_rows.cpp, openvino/op/view.cpp,
openvino/utils.cpp

The backend baked the captured prefill token count into the MoE graph as a static
dimension, which broke multi-token decode (Broadcast mismatch) and corrupted the GPU
KV-cache (static RoPE concats tripped an in-place-concat bug). Fix keeps the token dim
dynamic through the routing-norm ops (SUM_ROWS/DIV/CLAMP now tracked in
compute_node_dynamic_dims), the per-expert scale, and the expert-plane view — so all
60 RoPE concats stay dynamic. get_rows.cpp broadcasts the batched-gather index batch
dim from the data's ShapeOf.

2.4 GPU full-MoE path

Files: ggml-openvino.cpp, ggml-openvino-extra.cpp/.h

• Full-MoE on one submodel (gpu_full_moe_enabled()): keeps the whole MoE on a single
  OV submodel instead of fragmenting at every MoE node (fragmentation caused cross-submodel
  rank-5 / garbage-index bugs). Fully auto-enabled for quant-MoE models on GPU by
  latching on the MUL_MAT_ID op at placement time (device==GPU && saw MUL_MAT_ID). No env
  var or manual flag — CPU/NPU take the unchanged path (the gates it controls are GPU-guarded).

(A related fix — dodging OpenVINO's rms_fusion pass, whose fused GPU kernel collapsed the
MoE router RMSNorm and flipped expert selection — was developed here as a gated
Multiply(x,x) square, but the current dev base now uses Multiply(x,x) unconditionally,
so this PR no longer carries a rms_norm.cpp change.)

2.5 op gating (is_op_unsupported_case in ggml-openvino.cpp)

Decline ops the backend can't run correctly, so the scheduler falls back to CPU:

• TOPK_MOE (fused): declined via the selected_experts VIEW guard — the model uses
  ARGSORT-based routing, not the fused op.
• i64 CONCAT, borderline q4_1/q5_1 n=256 GET_ROWS (ERR ~1.1e-7, right at the
  tolerance), and oversized non-expert MUL_MAT_ID (the large-tmp cap is bypassed only
  for the real ffn_moe_gate_up/ffn_moe_down experts, not arbitrary MUL_MAT_ID).

2.6 Null-guard (needed for gemma4 to run at all)

File: ggml-decoder.cpp — compute_llm_params called is_kvcache(node->view_src, …)
on a GGML_OP_SCALE op without checking view_src != nullptr. gemma4's inp_scaled
SCALE has a null view_src → segfault. Added the guard (latent bug in the base;
surfaced by gemma4).

---

3. Verification

Built against OpenVINO 2026.2.1. Compared against ravi9/dev_backend_openvino.

Test	Result
gemma4 26B MoE — CPU greedy	✅ "Paris is the capital of France"
gemma4 26B MoE — CPU multi-token decode (n=12)	✅ "...Paris."
gemma4 E2B — CPU / GPU / NPU	✅ all "Paris. [end of text]"
dense Llama-3.2-1B — cli / server / bench / perplexity, CPU+GPU	✅ works on both devices; PPL ~1.04–1.05
test-backend-ops (OPENVINO) — CPU	✅ 2622 tests, 0 fail (incl. all MoE ops: MUL_MAT_ID, MUL_MAT_ID_FUSION, TOPK_MOE, GET_ROWS, CONCAT)
test-backend-ops (OPENVINO) — GPU	✅ 0 fail across MoE ops
llama-perplexity (OV backend, on Llama-3.2-1B)	✅ works — PPL 1.0525 ±0.0077

No regressions in dense models or existing tooling — the MoE-specific gates are all
conditioned on MoE op names / the auto-detected full-MoE path, so non-MoE models take the
unchanged code path. This PR is deliberately scoped to MoE only: an unrelated
token_embd extraction tweak (a general dense-model quality change, plus its
GGML_OPENVINO_EMBD_REQUANT env var) was removed from this branch so the diff contains
nothing outside the MoE feature — dense models keep the base behavior byte-for-byte.

Pre-existing CI failures on the base (NOT introduced by this PR)

These fail identically on plain dev_backend_openvino with no MoE commit. This PR
only rebases on top of the current tip and inherits them; it changes none of the code
involved. They will fail on that base's CI for any PR until fixed upstream.

• test-thread-safety (CI test 33) — fails with GGML OpenVINO backend std::exception: map::at / "One or more threads failed". Bisected to the base's own
  commit 49852ce11 ("remove the unique name in llama.cpp; add new ov name in ov bk"),
  which introduced get_tensor_ov_name() (a ggml_hash_find on the cgraph
  visited_hash_set). Verified:

  • 4102b17e1 (before 49852ce11) → ✅ "All threads finished without errors"
  • c6311c391 (current base HEAD) → ❌ map::at
  • this PR (c6311c391 + MoE commit) → ❌ identical map::at

  Not concurrency-specific (also fails at -np 1 -t 1); it's the multi-model/multi-context
  path hitting a map::at lookup miss in the new hash-based naming. Our MoE commit does not
  touch that code (the base's get_tensor_ov_name replaced this PR's earlier unique-view
  naming during rebase). Fix belongs in the base (49852ce11), not here.

• GATED_DELTA_NET op test stalls the full test-backend-ops sweep on the current
  dev base (a base/qwen3next op, untouched by this PR — all MoE ops run and pass before it).

Known non-issues (hardware/environment, not this PR)

• 26B on GPU / 26B perplexity: the test box has only 30 GB RAM and an iGPU
  (~14 GB shared budget). The 26B monolithic OV compile peaks ~30 GB and the ~14.4 GB of
  resident experts exceed the iGPU budget, so 26B-on-GPU and 26B-perplexity OOM on this
  box. These are capacity limits, not code defects — the GPU code path is validated by
  E2B-on-GPU and test-backend-ops GPU 2597/2597, and perplexity is validated on the 1B.
  They pass on larger-RAM machines (prior 64 GB box).

---

4. Files changed

File	What
ggml-quants.cpp/.h	f16 zero-point fusable dequant; in-place extraction sizing
ggml-openvino-extra.cpp/.h	3D-expert layout, f16 zp size, gpu_full_moe_enabled() helper
ggml-openvino.cpp	3D set_tensor/alloc, MoE op gates, MUL_MAT_ID latch + large-tmp scope
ggml-decoder.cpp	3D create_weight_node, dynamic-dim tracking (SUM_ROWS/DIV/CLAMP), null-guard
openvino/op/mul_mat_id.cpp	gather-then-split on rank-2 compressed experts
openvino/op/view.cpp	per-expert slice + dynamic view target
openvino/op/get_rows.cpp	batched-gather index broadcast
openvino/utils.cpp	dynamic-axis guard in process_view_input_new

---

5. How to build & run

# Build (point OpenVINO_DIR at your OV dist's runtime/cmake)
cmake -B build/ReleaseOV -G Ninja -DCMAKE_BUILD_TYPE=Release -DGGML_OPENVINO=ON \
  -DOpenVINO_DIR=<ov>/runtime/cmake
cmake --build build/ReleaseOV -j --target llama-completion test-backend-ops

# Run (CPU device)
export LD_LIBRARY_PATH=<ov>/runtime/lib/intel64:<ov>/runtime/3rdparty/tbb/lib
export GGML_OPENVINO_DEVICE=CPU
./build/ReleaseOV/bin/llama-completion \
  -m gemma-4-26B-A4B-it-UD-Q4_K_M.gguf \
  -p "Paris is the capital of" -n 1 -c 256 -t 8 -no-cnv --temp 0 --no-warmup
# → "Paris is the capital of France"

Env vars added by this PR: none. The GPU full-MoE path is fully auto-detected
(device==GPU && the model has 3D quantized MoE experts); there is no new flag to set.

[wine99]

Does the Compressed Matmul kernel/pattern support (w_f16 - zp_f16) * scale as well, similar to GatherCompressed here? If so, can we remove the two paths for zp_int and bias_f16 and use zp_f16 uniformly?

[cavusmustafa]

It makes sense but it may impact a larger scope of models. We can make it another PR maybe?

Integer zp was causing round error for moe expert weights. For MatMul, int zp seems to be working fine but it makes sense to try fp16 and see how it impacts accuracy and memory usage.

[wine99]

If I understand it correctly, this PR enables mulmat_id as GatherCompressed + Dequantize selected experts + fp Matmul in runtime.

I found in ov there is also CompressedGatherMatmul which does the above steps in one op. But it seems like this op can only be constructed inside a complete MOE pattern. Should we investigate into this op?

[cavusmustafa]

If I understand it correctly, this PR enables mulmat_id as GatherCompressed + Dequantize selected experts + fp Matmul in runtime.

I found in ov there is also CompressedGatherMatmul which does the above steps in one op. But it seems like this op can only be constructed inside a complete MOE pattern. Should we investigate into this op?

Yes, it totally makes sense to investigate on this. But GatherMatmul seems to be an internal op. We need to either introduce internal ops into frontend (this could be beneficial for Rope as well) or translate this into a pattern that plugin matchers can capture and generate GatherMatmul which we can do with public ops like GroupedMatmul.

For the first option, I created a branch which shows significant performance gains on GPU: https://github.com/cavusmustafa/llama.cpp/tree/gathermatmul-pr

Need to investigate the second option more so we can achieve the same result without internal op access.