[hmellor]

Overview

Let the Transformers modeling backend fuse a GLU activation with its down projection's input quantization, as native Llama does with maybe_fused_act_quant. This closes the NVFP4 throughput gap listed in #66547 (serving Llama with the backend).

Claims

• The GLU fuser rewrites down(act(gate_up(x))) to down(act(gate_up(x), down)) when the activation feeds the down projection directly, and wraps the activation in ActAndMulQuant, which calls maybe_fused_act_quant(act, x, down). For HF's LlamaMLP:

  down_proj = self.down_proj(self.act_fn(self.gate_up_proj(x), self.down_proj))

• NVFP4 Llama on the backend uses the fused SiLU-and-mul + NVFP4 quant kernel, and its throughput matches native (it was 1.8% slower).
• No change without a fused kernel: maybe_fused_act_quant falls back to the plain activation.

Validation

B200, RedHatAI/Llama-3.1-8B-Instruct-NVFP4, 1024 in / 256 out, 1000 prompts, mean of 4 runs (2 GPUs × 2 rounds, alternating):

	req/s	vs native
Native	74.44
Backend, main	73.10	-1.8%
Backend, this PR	74.85	+0.6%

Prompt logprobs over 1577 tokens: mean NLL native 2.1295, backend main 2.1301, this PR 2.1268; mean |Δlogprob| vs native is 0.2011 for backend main and 0.2046 for this PR, so the fused path is as close to native as the unfused one. GSM8K (1319 questions, 5-shot, greedy): native 0.689, backend main 0.688, this PR 0.673. Given the matching logprobs, I read the GSM8K gap as greedy paths diverging under NVFP4 rather than a numerical regression.

vllm bench throughput --model RedHatAI/Llama-3.1-8B-Instruct-NVFP4 --model-impl {vllm,transformers} --input-len 1024 --output-len 256 --num-prompts 1000
pytest tests/models/transformers/fusers/test_linear.py

test_linear.py: 58 passed. The new assertion in test_detects_and_rewrites_glu fails on main and passes here. tests/models/transformers/test_backend.py and Gemma test_common pass on B200 (test_distributed fails the same way on main on that host).

Details

The down projection is passed to the activation, rather than the rewritten forward calling maybe_fused_act_quant as a global: Dynamo resolves a function's globals through its module when building guards, and the rewritten forward's module is the Transformers modeling module. The rewrite only applies when the down projection's single argument is the activation call itself; anything in between (casts, dropout) leaves the forward as before.

No open PR covers this (searched for maybe_fused_act_quant, GLU fuser and Transformers backend NVFP4).

---

Pull Request Checklist

• I used vLLM's /pr-checklist skill. (Mandatory for agents, optional for humans).

• AI assistance was used during the creation of this PR.

• Design Fit: Minimizes impact on core components, reuses existing functionality, and justifies added complexity.

• Testing and Validation: Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI.

• Code Quality and Style: Keeps code and comments clear and concise, and updates relevant documentation and examples.

• Pull Request Contents: Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.

🤖 Generated with Claude Code

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[hmellor]

/ci run

[github-actions]

✅ Triggered Buildkite CI #101477 for commit 8469470f9ebc.

ℹ️ This PR is 1 commit behind upstream main. Merged upstream main into this branch as 8469470f9ebc before starting CI. Pull the branch before pushing more changes.

[hmellor]

/ci retry

[github-actions]

✅ Queued 6 failed job(s) for retry in Buildkite CI #101477.

[hmellor]

/ci run

[github-actions]

✅ Triggered Buildkite CI #101494 for commit 957ca647bdda.

[hmellor]

The :amd: (MI355 DPX) Basic Models (Extra Initialization) failure is test_can_initialize_large_subset[DomynEdgeForCausalLM], which also fails on main (native Qwen2 sliding-window assertion with the test's 1-layer override). It should be fixed by #66819.

[hmellor]

/ci retry

[github-actions]

✅ Triggered Buildkite CI #101740 for commit 287736d8085c, running 2 failed step(s) from Buildkite CI #101494.