Repository navigation
[Perf] gemma3: Pool the projector input in channels-last - #44231
Draft
desmond-intel wants to merge 1 commit into
Draft
desmond-intel wants to merge 1 commit into
desmond-intel wants to merge 1 commit into
Conversation
desmond-intel
force-pushed
the
xpu/gemma-rmsnorm-strided-input
branch
from
October 15, 2026 21:32
df3f4aa to
9b987c9
Compare
Contributor
Author
|
End-to-end CUDA check on 1x NVIDIA H200 NVL (tp 1), default attention backend (fa3). Both runs use the same image (
5347 of the 5349 answers are identical. sglang serve --model-path google/gemma-3-4b-it --tp-size 1 --mem-fraction-static 0.70 --trust-remote-code --enable-multimodal
python3 -m lmms_eval --model openai --model_args model_version=google/gemma-3-4b-it,num_concurrent=16 \
--tasks docvqa_val --batch_size 16 --seed 42 |
desmond-intel
force-pushed
the
xpu/gemma-rmsnorm-strided-input
branch
from
November 2, 2026 19:39
9b987c9 to
3dfbe95
Compare
desmond-intel
force-pushed
the
xpu/gemma-rmsnorm-strided-input
branch
from
November 11, 2026 07:37
3dfbe95 to
495d735
Compare
desmond-intel
force-pushed
the
xpu/gemma-rmsnorm-strided-input
branch
from
November 17, 2026 17:51
495d735 to
5e06d42
Compare
Gemma3MultiModalProjector copied the vision output to NCHW before avg_pool, so after flatten(2).transpose(1, 2) the pooled tokens had stride(-1) != 1 and the norm that follows saw a transposed input. The transposed view is already channels-last; pooling it directly keeps that layout, so the pooled tokens come out contiguous and the copy is gone. The output is bit-identical. Tested on Arc Pro B60 (bf16/fp16/fp32) and CPU: bit-equal pooled output; pool + copy for the norm 660.6 -> 377.8 us at B=8 on B60.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Gemma3MultiModalProjectormakes the vision output NCHW-contiguous beforeavg_pool. Afterflatten(2).transpose(1, 2)the pooled tokens are[B, 256, hidden]withstride(-1) = 256, somm_soft_emb_normgets a transposed input and the XPU path needs a copy beforegemma_rmsnorm(#44225).The view before that
.contiguous()is already channels-last, since it comes from the token-major vision output. Pooling it directly keeps that layout, so the pooled tokens come out contiguous.Modifications
models/gemma3_mm.py: drop the.contiguous()beforeavg_pool.With this, the
stride(-1) != 1copy inGemma3RMSNorm.forward_xpu(#44225) no longer triggers for Gemma-3.Test
Pooled output vs the current code, for a
[B, 4096, 1152]vision output:stride (294912, 1152, 1)).End-to-end accuracy run pending.
Checklist
pre-commit runon the changed file: passed.CI States
Latest PR Test (Base): ❌ Run #44141599733
Latest PR Test (Extra): ❌ Run #44141599487
Latest PR Test (AMD ROCm 10): ❌ Run #44141599419
Tip
Base CI only covers the core cases, and its workload may not exercise your change. Use
/rerun-testto run the tests your change touches, or copy this prompt for an agent: