ggml-cpu: fuse RMS_NORM + MUL on CPU backend by zzzzwc · Pull Request #22423 · ggml-org/llama.cpp

zzzzwc · 2026-04-27T07:23:55Z

Overview

Add a fused RMS_NORM + MUL kernel on the CPU backend that computes the output in a single pass, avoiding materialization of the intermediate rms_norm result to improve performance of the CPU backend. (Fusion can be disabled via GGML_CPU_DISABLE_FUSION.)

Extend test-backend-ops perf mode with perf_group_size() to support accurate multi-op benchmarking.

Additional information

Discussion: #22315

The benchmark result on my Mac:

Env
MacBook Air M2 24GB

Before (fusion disabled):

$ GGML_CPU_DISABLE_FUSION=on ./build/bin/test-backend-ops perf -o RMS_NORM_MUL_ADD -b CPU -t 1 --perf-duration 10
ggml_metal_device_init: tensor API disabled for pre-M5 and pre-A19 devices
ggml_metal_library_init: loading '/Users/zwc/CLionProjects/llama.cpp/build/bin/default.metallib'
ggml_metal_library_init: loaded in 0.001 sec
ggml_metal_rsets_init: creating a residency set collection (keep_alive = 180 s)
ggml_metal_device_init: GPU name:   MTL0 (Apple M2)
ggml_metal_device_init: GPU family: MTLGPUFamilyApple8  (1008)
ggml_metal_device_init: GPU family: MTLGPUFamilyCommon3 (3003)
ggml_metal_device_init: GPU family: MTLGPUFamilyMetal3  (5001)
ggml_metal_device_init: simdgroup reduction   = true
ggml_metal_device_init: simdgroup matrix mul. = true
ggml_metal_device_init: has unified memory    = true
ggml_metal_device_init: has bfloat            = true
ggml_metal_device_init: has tensor            = false
ggml_metal_device_init: use residency sets    = true
ggml_metal_device_init: use shared buffers    = true
ggml_metal_device_init: recommendedMaxWorkingSetSize  = 17179.89 MB
Testing 3 devices

Backend 1/3: MTL0
  Skipping
Backend 2/3: BLAS
  Skipping
Backend 3/3: CPU
  Device description: Apple M2
  Device memory: 24576 MB (24576 MB free)

  RMS_NORM_MUL_ADD(type=f32,ne=[1024,512,1,1],eps=0.000001,broadcast=0,multi_add=0):                    9234 runs -  1116.29 us/run -    16384 kB/run -   14.00 GB/s
  RMS_NORM_MUL_ADD(type=f32,ne=[1024,512,1,1],eps=0.000001,broadcast=0,multi_add=1):                   11286 runs -   919.52 us/run -    16384 kB/run -   16.99 GB/s
  Backend CPU: OK
3/3 backends passed
OK

After (fusion enabled):

$ ./build/bin/test-backend-ops perf -o RMS_NORM_MUL_ADD -b CPU -t 1 --perf-duration 10
ggml_metal_device_init: tensor API disabled for pre-M5 and pre-A19 devices
ggml_metal_library_init: loading '/Users/zwc/CLionProjects/llama.cpp/build/bin/default.metallib'
ggml_metal_library_init: loaded in 0.001 sec
ggml_metal_rsets_init: creating a residency set collection (keep_alive = 180 s)
ggml_metal_device_init: GPU name:   MTL0 (Apple M2)
ggml_metal_device_init: GPU family: MTLGPUFamilyApple8  (1008)
ggml_metal_device_init: GPU family: MTLGPUFamilyCommon3 (3003)
ggml_metal_device_init: GPU family: MTLGPUFamilyMetal3  (5001)
ggml_metal_device_init: simdgroup reduction   = true
ggml_metal_device_init: simdgroup matrix mul. = true
ggml_metal_device_init: has unified memory    = true
ggml_metal_device_init: has bfloat            = true
ggml_metal_device_init: has tensor            = false
ggml_metal_device_init: use residency sets    = true
ggml_metal_device_init: use shared buffers    = true
ggml_metal_device_init: recommendedMaxWorkingSetSize  = 17179.89 MB
Testing 3 devices

Backend 1/3: MTL0
  Skipping
Backend 2/3: BLAS
  Skipping
Backend 3/3: CPU
  Device description: Apple M2
  Device memory: 24576 MB (24576 MB free)

  RMS_NORM_MUL_ADD(type=f32,ne=[1024,512,1,1],eps=0.000001,broadcast=0,multi_add=0):                   18981 runs -   538.98 us/run -    16384 kB/run -   28.99 GB/s
  RMS_NORM_MUL_ADD(type=f32,ne=[1024,512,1,1],eps=0.000001,broadcast=0,multi_add=1):                   18981 runs -   533.96 us/run -    16384 kB/run -   29.26 GB/s
  Backend CPU: OK
3/3 backends passed
OK

variant	before	after	speedup
`multi_add=0`	1116.29 us/run (14.00 GB/s)	538.98 us/run (28.99 GB/s)	2.07x
`multi_add=1`	919.52 us/run (16.99 GB/s)	533.96 us/run (29.26 GB/s)	1.72x

Requirements

I have read and agree with the contributing guidelines
AI usage disclosure: YES, used for testing, code cleanup and commit message drafting

am17an · 2026-04-27T08:27:27Z

Some quick performance numbers

CPU	Model	Microbatch size	Test	t/s `0f1bb60`	t/s `c7340e1`	Speedup
AMD EPYC 7742 64-Core Processor	llama 8B Q4_K_M	1	pp256	8.94	9.33	1.04
AMD EPYC 7742 64-Core Processor	llama 8B Q4_K_M	2	pp256	18.15	18.67	1.03
AMD EPYC 7742 64-Core Processor	llama 8B Q4_K_M	4	pp256	34.22	34.29	1.00
AMD EPYC 7742 64-Core Processor	llama 8B Q4_K_M	8	pp256	46.85	48.75	1.04
AMD EPYC 7742 64-Core Processor	llama 8B Q4_K_M	16	pp256	52.92	54.63	1.03
AMD EPYC 7742 64-Core Processor	llama 8B Q4_K_M	32	pp256	90.19	96.74	1.07
AMD EPYC 7742 64-Core Processor	llama 8B Q4_K_M	64	pp256	177.99	190.80	1.07
AMD EPYC 7742 64-Core Processor	llama 8B Q4_K_M	128	pp256	345.74	370.07	1.07
AMD EPYC 7742 64-Core Processor	llama 8B Q4_K_M	256	pp256	652.94	697.51	1.07

zzzzwc · 2026-04-28T02:44:20Z

Thanks for the review, @am17an Addressed in the latest commit. PTAL

zzzzwc · 2026-04-29T08:15:59Z

@ggml-org/maintainers need second approval please.

CISC · 2026-04-30T07:25:48Z

@zzzzwc Sorry, as usual GitHub UI messed up newlines, please normalize to \n:
https://github.com/ggml-org/llama.cpp/actions/runs/25149770819/job/73717845312?pr=22423

zzzzwc · 2026-05-04T07:13:58Z

@zzzzwc Sorry, as usual GitHub UI messed up newlines, please normalize to \n: https://github.com/ggml-org/llama.cpp/actions/runs/25149770819/job/73717845312?pr=22423

Done, Sorry for the delay.

zzzzwc · 2026-05-05T04:45:13Z

Is there anything else I should address before merging?

Add a fused rms_norm + mul kernel that computes the output in a single pass, avoiding materialization of the intermediate rms_norm result. The graph compute loop detects the pattern and dispatches to the fused path. fusion can be disabled via the GGML_CPU_DISABLE_FUSION environment variable.

- inline get_disable_fusion() into ggml_cpu_try_fuse_ops - fix env check: only disable on "1"/"on"/"true" (case-insensitive) - return fused node count (int) instead of bool, for future N-op fusion - add TODO to move fusion detection into ggml_graph_plan

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>

- normalize to LF (\n) in ggml-cpu.c - rename variables in ops.cpp from xxx_dst to dst_xxx

- rename GGML_RMS_NORM_FUSE_XXX to GGML_RMS_NORM_FUSE_OP_xxx

(cherry picked from commit f08f20a)

…026-07-12) 4 parallele Subagents (Vulkan/AMD, CUDA/MoE, arXiv, Multi-GPU/Batching) konsolidiert zu 25 neuen Items in ROADMAP: Tier 1 Quick Wins (3 neu): - AtomicBot-ai#31 K-Quant A-Matrix Transpose CM1 (PR ggml-org#22970, +5-15% PP auf Mars) - AtomicBot-ai#32 Pascal L1 Cache Tuning (-Xptxas -dlcm=ca, Styx) - TheTom#33 Per-Quant MMVQ/MMQ Batch Threshold (AMD MFMA, Mars/Venus) Tier 2 (7 neu): - TheTom#34 UBBoost, TheTom#35 Row-Packing DMMV, TheTom#36 Auto Param Fitting TP - TheTom#37 LFRU Expert Caching, TheTom#38 Conf-KV, TheTom#39 Talon, TheTom#40 MoE Load Balancing Tier 3 (5 neu): - TheTom#41 GRKV, TheTom#42 CapKV, TheTom#43 SliderQuant, TheTom#44 Alloc-MoE, TheTom#45 CUDA Streams QKV Tier 4 (10 neu): - TheTom#46-55: SpecMD, QUICK, FluxMoE, STAR-KV, VQKV, CompilerKV, SliceMoE, MoBiE, DASH-Q, GOOSE 7 PRs als bereits im Fork identifiziert (nicht erneut vorschlagen): ggml-org#21472, ggml-org#23764, ggml-org#22299, ggml-org#21611, ggml-org#22423, ggml-org#18749, Warp Shuffle, Constant Memory

zzzzwc requested a review from ggerganov as a code owner April 27, 2026 07:23

github-actions Bot added testing Everything test related ggml changes relating to the ggml tensor library for machine learning labels Apr 27, 2026

am17an reviewed Apr 27, 2026

View reviewed changes

zzzzwc force-pushed the zwc/cpu-fusion branch from c7340e1 to 5dc3409 Compare April 28, 2026 02:07

am17an reviewed Apr 28, 2026

View reviewed changes

Comment thread ggml/src/ggml-cpu/ggml-cpu.c Outdated

am17an reviewed Apr 28, 2026

View reviewed changes

Comment thread ggml/src/ggml-cpu/ggml-cpu.c Outdated

Comment thread ggml/src/ggml-cpu/ops.cpp Outdated

Comment thread ggml/src/ggml-cpu/ops.cpp

Comment thread ggml/src/ggml-cpu/ops.cpp

am17an approved these changes Apr 28, 2026

View reviewed changes

ggerganov reviewed Apr 29, 2026

View reviewed changes

Comment thread ggml/src/ggml-cpu/ggml-cpu.c Outdated

Comment thread ggml/src/ggml-cpu/ops.h Outdated

zzzzwc force-pushed the zwc/cpu-fusion branch from d35f7a5 to da89c25 Compare April 30, 2026 05:26

CISC reviewed Apr 30, 2026

View reviewed changes

Comment thread ggml/src/ggml-cpu/ggml-cpu.c Outdated

Comment thread ggml/src/ggml-cpu/ggml-cpu.c Outdated

Comment thread ggml/src/ggml-cpu/ggml-cpu.c Outdated

ggerganov reviewed Apr 30, 2026

View reviewed changes

Comment thread ggml/src/ggml-cpu/ops.cpp Outdated

CISC approved these changes May 4, 2026

View reviewed changes

zzzzwc force-pushed the zwc/cpu-fusion branch from 87c6601 to 1602e65 Compare May 5, 2026 02:10

ggerganov reviewed May 5, 2026

View reviewed changes

Comment thread ggml/src/ggml-cpu/ops.cpp

zzzzwc and others added 8 commits May 6, 2026 10:08

ggml-cpu : simplify fusion-disable env check with atoi

c545efa

ggml-cpu: remove redundant comments

7012a88

ggml-cpu : address review comments

036bbac

Apply suggestions from code review

ada950d

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>

ggml-cpu : address review comments

d2c7072

- normalize to LF (\n) in ggml-cpu.c - rename variables in ops.cpp from xxx_dst to dst_xxx

ggml-cpu : address review comments

41eb57a

- rename GGML_RMS_NORM_FUSE_XXX to GGML_RMS_NORM_FUSE_OP_xxx

zzzzwc force-pushed the zwc/cpu-fusion branch from 6975343 to 41eb57a Compare May 6, 2026 02:09

ggerganov approved these changes May 6, 2026

View reviewed changes

am17an merged commit f08f20a into ggml-org:master May 6, 2026
45 of 46 checks passed

cetarthoriphros pushed a commit to cetarthoriphros/llama.cpp that referenced this pull request May 9, 2026

ggml-cpu: fuse RMS_NORM + MUL on CPU backend (ggml-org#22423)

b875d36

meh pushed a commit to meh/llama.cpp that referenced this pull request May 10, 2026

ggml-cpu: fuse RMS_NORM + MUL on CPU backend (ggml-org#22423)

a3a0d8b

baramofme pushed a commit to baramofme/llama-cpp-turboquant that referenced this pull request May 23, 2026

ggml-cpu: fuse RMS_NORM + MUL on CPU backend (ggml-org#22423)

128a286

carlosfundora pushed a commit to carlosfundora/llama.cpp-1-bit-turbo that referenced this pull request May 24, 2026

ggml-cpu: fuse RMS_NORM + MUL on CPU backend (ggml-org#22423)

50f4daf

(cherry picked from commit f08f20a)

winstonma pushed a commit to winstonma/llama.cpp that referenced this pull request May 27, 2026

ggml-cpu: fuse RMS_NORM + MUL on CPU backend (ggml-org#22423)

d7e07b1

fewtarius pushed a commit to fewtarius/CachyLLama that referenced this pull request May 30, 2026

ggml-cpu: fuse RMS_NORM + MUL on CPU backend (ggml-org#22423)

5468d9a

Uh oh!

Conversation

zzzzwc commented Apr 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overview

Additional information

Requirements

Uh oh!

am17an commented Apr 27, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

zzzzwc commented Apr 28, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

zzzzwc commented Apr 29, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

CISC commented Apr 30, 2026

Uh oh!

zzzzwc commented May 4, 2026

Uh oh!

zzzzwc commented May 5, 2026

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

zzzzwc commented Apr 27, 2026 •

edited

Loading