Skip to content

[CUDA][Improvement] RMSNorm forward speed up#3850

Merged
nastya236 merged 3 commits into
mainfrom
rmsnorm
Jul 17, 2026
Merged

[CUDA][Improvement] RMSNorm forward speed up#3850
nastya236 merged 3 commits into
mainfrom
rmsnorm

Conversation

@nastya236

@nastya236 nastya236 commented Jul 16, 2026

Copy link
Copy Markdown
Collaborator

We have 2 variants of RNSNorm and for majority of workloads we were using res_norm_small:

  • each thread does 1 vectorised load of width 8 (for bf16)
  • 1 block per row, so block size 1024 for N = 8192

Now block size is 128, we do multiple vectorised loads per thread.
The issue is the register pressure when using safe load_vector with loop fall back. Therefore in a new version there is an alignment check. But in majority of cases x, w and output should be 16 bytes aligned.

b200, pytorch 2.13, cuda 13.3

N M Torch (µs) MLX (µs, before) MLX (µs, after)
1024 32768 51.7 44.9 39.2
2048 32768 57.7 77.9 53.0
3072 32768 83.9 142.5 70.2
4096 32768 92.7 146.4 89.5
7168 32768 160.3 307.0 146.1
8192 32768 180.7 318.2 170.6

@angeloskath angeloskath left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚀

@nastya236
nastya236 merged commit b7c3dd6 into main Jul 17, 2026
28 checks passed
@nastya236
nastya236 deleted the rmsnorm branch July 17, 2026 16:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants