llama: disable fused up-gate when an offload backend lacks GGML_OP_FUSED_UP_GATE (fixes silent 3.5x Metal slowdown)

[hchengit]

Problem

GGML_OP_FUSED_UP_GATE is implemented in the CUDA backend (ggml_cuda_up_gate_unary) but not in the Metal backend. With -ngl 99 on Apple Silicon, the backend scheduler silently assigns every fused up-gate node to the CPU backend, forcing a GPU→CPU→GPU synchronization stall on every FFN layer — 24 times per token on Qwen 2.5-0.5B.

There is no error and no warning; inference is just silently slower. Measured on an M2 (Qwen 2.5-0.5B, Q5_0, full offload):

	decode speed
default (fused_up_gate = true)	36 tok/s
--no-fused-up-gate	125 tok/s (3.5x)

This affects all standard transformer models (Qwen 2.5, Llama, DeepSeek, ...) on all Apple Silicon Macs (M1–M4). Hybrid DeltaNet models are unaffected (their FFN path doesn't emit this op).

Fix

At context creation, after the backends are initialized, build a dummy GGML_OP_FUSED_UP_GATE node and probe each non-CPU backend with ggml_backend_supports_op(). If any offload backend lacks the op, disable cparams.fused_up_gate and log a warning. The graph then builds the decomposed form (MUL_MAT + FUSED_MUL_UNARY), which Metal handles natively, and the whole graph stays on GPU.

This deliberately does not special-case Metal: any backend without the kernel (Vulkan, SYCL, ...) gets the same fallback, and once someone adds a Metal kernel for the op the probe simply stops triggering.

--no-fused-up-gate still works as an explicit override; the probe only ever turns the option off, never on.

Testing

• Linux x86-64 AVX2, CPU-only: builds clean; probe is a no-op when the only backend is CPU (fused path unchanged, verified fused_up_gate = 1 in the context log and normal generation).
• The behavioral fix (forcing fused_up_gate = false under Metal full offload) is the same mitigation we have been running manually via an eval callback on M2 since April — that is where the 36→125 tok/s numbers come from.
• I don't have Apple hardware in CI reach for this branch, so the probe path on Metal is verified by inspection. Update: we verified it on M2 (details in comment below): this branch built with GGML_METAL=ON, Qwen 2.5-0.5B Q4_K_M at -ngl 99 — the probe warning fires at context creation and default decode runs at 112.5 t/s, matching the explicit --no-fused-up-gate run (115.3 t/s, identical output) instead of the previous ~36 t/s CPU-fallback behavior.

[hchengit]

We verified it on M2.

We built this branch with Metal (Apple M2, macOS) and ran Qwen 2.5-0.5B-Instruct Q4_K_M with full offload (-ngl 99, -n 128, greedy):

• Default settings: the probe fires at context creation —
  llama_init_from_model: backend Metal does not implement FUSED_UP_GATE - disabling fused up-gate (decomposing into mul_mat + fused_mul_unary)
  — and decode runs at 112.5 t/s (8.89 ms/token), i.e. the fast decomposed path, not the previous ~36 t/s CPU-fallback behavior.
• Explicit --no-fused-up-gate: fused_up_gate = 0 from the start, no probe warning (correct — the option was already off), 115.3 t/s. Output is identical to the default run, so the two paths are at parity within noise.

Two pre-existing Metal/ARM build issues we hit while compiling this branch on Xcode clang 21, both unrelated to this PR:

• the iqk sources fail with requires target feature 'fullfp16' unless configured with -DGGML_NATIVE=OFF -DGGML_ARCH_FLAGS=-mcpu=apple-m2 (the -march=native added by GGML_NATIVE overrides the cpu flags);
• src/llama-dflash.cpp calls ggml_backend_is_metal / ggml_backend_metal_set_n_cb without the #ifdef GGML_USE_METAL include of ggml-metal.h that llama.cpp has, so GGML_METAL=ON builds fail to compile there.

[ikawrakow]

The Metal back-end has not been maintained for a while. There are several other new ops that are missing in addition to the fused up/gate op. Are you interested in bringing the Metal back-end up to speed and helping maintain it?

The way out of the missing GGML_OP_FUSED_UP_GATE is to add -no-fmoe to the command line (or --no-fused-moe if you prefer the long form).