Skip to content

[Perf] gemma3: Pool the projector input in channels-last - #44231

Draft
desmond-intel wants to merge 1 commit into
sgl-project:mainfrom
desmond-intel:xpu/gemma-rmsnorm-strided-input
Draft

desmond-intel wants to merge 1 commit into
sgl-project:mainfrom
desmond-intel:xpu/gemma-rmsnorm-strided-input

Conversation

@desmond-intel

@desmond-intel desmond-intel commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

Motivation

Gemma3MultiModalProjector makes the vision output NCHW-contiguous before avg_pool. After flatten(2).transpose(1, 2) the pooled tokens are [B, 256, hidden] with stride(-1) = 256, so mm_soft_emb_norm gets a transposed input and the XPU path needs a copy before gemma_rmsnorm (#44225).

The view before that .contiguous() is already channels-last, since it comes from the token-major vision output. Pooling it directly keeps that layout, so the pooled tokens come out contiguous.

Modifications

  • models/gemma3_mm.py: drop the .contiguous() before avg_pool.

With this, the stride(-1) != 1 copy in Gemma3RMSNorm.forward_xpu (#44225) no longer triggers for Gemma-3.

Test

Pooled output vs the current code, for a [B, 4096, 1152] vision output:

  • Intel Arc Pro B60 and CPU, bf16/fp16/fp32: bit-equal, and now contiguous (stride (294912, 1152, 1)).
  • B60, bf16, B=8: pooling plus the copy for the norm, 660.6 µs → 377.8 µs.

End-to-end accuracy run pending.

Checklist

  • pre-commit run on the changed file: passed.

CI States

Latest PR Test (Base): ❌ Run #44141599733
Latest PR Test (Extra): ❌ Run #44141599487
Latest PR Test (AMD ROCm 10): ❌ Run #44141599419

Tip

Base CI only covers the core cases, and its workload may not exercise your change. Use /rerun-test to run the tests your change touches, or copy this prompt for an agent:

Find the registered tests this PR affects and run them with /rerun-test, following https://github.com/sgl-project/sglang/blob/main/.agents/skills/rerun-test/SKILL.md

@desmond-intel desmond-intel reopened this Oct 15, 2026
@desmond-intel desmond-intel changed the title [XPU] Drop the gemma_rmsnorm last-dim copy in Gemma3RMSNorm [Perf] gemma3: Pool the projector input in channels-last Oct 15, 2026
@desmond-intel
desmond-intel force-pushed the xpu/gemma-rmsnorm-strided-input branch from df3f4aa to 9b987c9 Compare October 15, 2026 21:32
@desmond-intel

Copy link
Copy Markdown
Contributor Author

End-to-end CUDA check on 1x NVIDIA H200 NVL (tp 1), default attention backend (fa3). Both runs use the same image (lmsysorg/sglang:nightly-dev-20261015-a6a8c560) with the sglang Python source taken from each commit:

  • before: merge-base 91f6ef2322
  • after: this PR 9b987c96e8
docvqa_val, all 5349 docs anls
Before 0.5933
After 0.5933

5347 of the 5349 answers are identical.

sglang serve --model-path google/gemma-3-4b-it --tp-size 1 --mem-fraction-static 0.70 --trust-remote-code --enable-multimodal

python3 -m lmms_eval --model openai --model_args model_version=google/gemma-3-4b-it,num_concurrent=16 \
  --tasks docvqa_val --batch_size 16 --seed 42

Gemma3MultiModalProjector copied the vision output to NCHW before avg_pool,
so after flatten(2).transpose(1, 2) the pooled tokens had stride(-1) != 1
and the norm that follows saw a transposed input. The transposed view is
already channels-last; pooling it directly keeps that layout, so the pooled
tokens come out contiguous and the copy is gone. The output is bit-identical.

Tested on Arc Pro B60 (bf16/fp16/fp32) and CPU: bit-equal pooled output;
pool + copy for the norm 660.6 -> 377.8 us at B=8 on B60.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant