opencl: ragged-tile MoE prefill FP16 GEMM optimization (skip padded expert tiles)#25433
Merged
Merged
Conversation
The MoE prefill GEMM groups tokens into TILESIZE_N=32 per-expert tiles; at low tokens-per-expert most tiles are mostly padding. When a tile's upper 16 slots are all padding (router index 0xFFFFFFFF), skip the second dotx16_reduce8 half. Numerically identical (skipped lanes are padding). Applied to all eight *_f32_ns MoE GEMMs; default on, opt out with GGML_OPENCL_MOE_RAGGED_FP16=0.
Replace the two half-tile dotx16_reduce8 calls in the 8 *_f32_ns MoE GEMMs with
four dotx8_reduce4 (8-column) calls, skipping each empty trailing skip-group
independently. Padding is always trailing, so the kernel rounds the valid count
up to the skip granularity and skips fully-padding groups. Byte-identical to the
non-skipped path. New env GGML_OPENCL_MOE_RAGGED_GRAN={8,16,32} (quarter/half/
off); default quarter.
github-actions
Bot
added
ggml
lhez
approved these changes
Jul 8, 2026
Contributor
X2-90With longer prompts. gpt-oss-20b-mxfp4this PR master Qwen3.6-35B-A3B-MXFP4_MOEthis PR master |
max-krasnyansky
approved these changes
Jul 8, 2026
wanghqc
added a commit
to qualcomm/llama.cpp
that referenced
this pull request
Jul 15, 2026
The ragged MoE GEMM change (ggml-org#25433) added two kernel args (adreno_use_moe_ragged, adreno_moe_ragged_skip_gran) to the ns MoE GEMM paths, set unconditionally. When GGML_OPENCL_USE_ADRENO_BIN_KERNELS is enabled and the prebuilt kernel library is present, these paths select the prebuilt *_ns_bin kernels, whose signatures do not include the ragged args, so clSetKernelArg fails with CL_INVALID_ARG_INDEX and any MoE model aborts on GGML_ASSERT. Only set the ragged args when the built-in source kernel is selected (q4_0, q4_1, q4_K, mxfp4 - the four paths with prebuilt variants; the prebuilt kernels have no ragged support). Verified on Adreno X2-90 with and without the prebuilt library on gpt-oss-20b MXFP4, Qwen3-30B-A3B Q4_K_M and Q4_0.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
Additional information
Gains on X2 GPU: Qwen3-30B-A3B Q4_K_M pp128 +24% / pp512 +10%; Q4_0 pp128 +26%; gpt-oss-20b MXFP4 pp128 +20%
Requirements
Tested with Adreno GPUs on Windows on Snapdragon (WoS) X1 and X2 devices.