Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
e42f6a8
Gemma 4 31B QB2: faster decode step with the same outputs
mvasiljevicTT Nov 13, 2026
db4fb3d
Gemma 4 31B QB2: prefix caching with restored sliding history
mvasiljevicTT Nov 13, 2026
eb6a090
Gemma 4 31B QB2: sliding groups use the plugin's anchored sliding-win…
mvasiljevicTT Nov 13, 2026
730fa63
Gemma 4 31B QB2: BF8 projection weights as the fixed weight format
mvasiljevicTT Nov 13, 2026
d0bdbed
Gemma 4 31B QB2: keep one copy of the projection weights
mvasiljevicTT Nov 13, 2026
fc7f44f
Gemma 4 31B QB2: the KV pool follows --max-model-len
mvasiljevicTT Nov 13, 2026
5638638
Gemma 4 31B QB2: compile every prompt length at start-up; retire deco…
mvasiljevicTT Nov 13, 2026
dee4b48
Gemma 4 31B QB2: read the cached sliding history by page id without c…
mvasiljevicTT Nov 13, 2026
0a41e72
Gemma 4 31B QB2: decode with the smallest trace that covers the live …
mvasiljevicTT Nov 13, 2026
199c795
Gemma 4 31B QB2: keep the decode entries when the traces are retired;…
mvasiljevicTT Nov 13, 2026
815caa3
Gemma 4 31B QB2: one DRAM reader per bank for the o projection decode…
mvasiljevicTT Nov 13, 2026
3c7364d
Gemma 4 31B QB2: n-gram speculative decoding, verified in the smalles…
mvasiljevicTT Nov 13, 2026
622cbc9
Gemma 4 31B QB2: continuation prefill without per-signature programs
mvasiljevicTT Nov 13, 2026
aed71cc
Gemma 4 31B QB2: pack the full-attention KV into 256-token blocks ove…
mvasiljevicTT Nov 13, 2026
77e8ecd
Gemma 4 31B QB2: smaller Q chunk for the chunked full-attention SDPA …
mvasiljevicTT Nov 14, 2026
daf2cfe
Gemma 4 31B QB2: size the KV pool from the DRAM budget and stop start…
mvasiljevicTT Nov 14, 2026
65cb2ee
Gemma 4 31B QB2: slice the prefill gate/up and q/k weights straight f…
mvasiljevicTT Nov 14, 2026
a527e16
Gemma 4 31B QB2: rebuild the interleaved prefill weights once per lay…
mvasiljevicTT Nov 15, 2026
f9d540e
Gemma 4 31B QB2 README: show the serving profiles with prefix caching
mvasiljevicTT Nov 15, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 12 additions & 6 deletions models/demos/gemma4_31b_qb2/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,15 +4,15 @@ Text generation for `google/gemma-4-31B-it` on one Blackhole QuietBox 2: four ch

## Design

The 60 decoder layers alternate five sliding-attention layers with one full-attention layer. Each chip holds a quarter of the projection weights. A ring fabric carries tensor-parallel reductions; completed sliding-attention sums retain BF16 precision. Decode replays a device trace with resident token, position, and sampling state. Scheduler updates replace inputs or page tables without resetting unrelated requests.
The 60 decoder layers alternate five sliding-attention layers with one full-attention layer. Each chip holds a quarter of the projection weights. A ring fabric carries tensor-parallel reductions; completed sliding-attention sums retain BF16 precision. Decode replays a device trace with resident token, position, and sampling state. A server keeps decode traces of 1, 4, 8, 16 and 32 rows (those below its slot count, plus the full one) over one KV pool and replays the smallest that covers the live requests, so a lone request decodes at the one-slot speed. Scheduler updates replace inputs or page tables without resetting unrelated requests. Prefill runs on whole cache pages with runtime-valued positions (rope rows, cache-fill pages, chunked-attention start, last-token row), so after start-up warm-up a continuation chunk at a new start or length compiles no program and leaves the decode traces alone (the exceptions are listed under start-up below).

Projection weights use BF4 and LoFi matmuls. The LM head uses BF8 and HiFi2. Residuals and normalization use BF16; KV storage uses BF8. Scheduler-driven chunked prefill uses the plugin's standard `SlidingWindowSpec` and paged full-attention history. Each sliding layer retains a BF16 K/V tail between scheduler calls, including one extra page for an unaligned continuation. Request completion, preemption and slot moves preserve or release that history through the existing plugin lifecycle. Sliding storage is bounded by the window and in-flight chunk budget.
Projection weights use BF8 and LoFi matmuls (one fixed format: BF4 projections break long agent runs). The LM head uses BF8 and HiFi2. Residuals and normalization use BF16; KV storage uses BF8. A layer keeps one copy of its projection weights: decode reads the DRAM-sharded copy and prefill rebuilds its interleaved weights from it once per layer and prefill span (the chunks of one span share them). Scheduler-driven chunked prefill uses the plugin's `AnchoredSlidingWindowSpec` (a request's cache-hit window stays until the request ends) and paged full-attention history. Each sliding layer retains a BF16 K/V tail between scheduler calls, including one extra page for an unaligned continuation. Request completion, preemption and slot moves preserve or release that history through the existing plugin lifecycle. The KV pool is packed: a pool position (one full-attention layer and its five sliding layers) shares one backing of wide rows, the scheduler gives the full-attention group 256-token blocks and the sliding groups 128-token blocks over one block-ID space, and a full block reads its row as two native 128-token half-pages, so the same DRAM holds twice the full-attention tokens and the separate full-attention tensors (about 0.7 GB per DRAM bank) are gone; prefix hits of the full-attention group are 256-token aligned. Sliding storage is bounded by the window and in-flight chunk budget.

Greedy decoding and stochastic sampling with `top_k=1..32` are supported. Launch with `--override-generation-config '{"top_k":20}'` so requests that omit top-k use a supported default. Request seeds must be integers in `0..2³¹−1`. Larger/unbounded top-k and wider seeds are outside this serving contract until the independent plugin fixes land. Penalties, logprobs and other host-only parameters use the existing host sampler. Host and device samplers need not produce the same random sequence.

## Run

Build TT-Metal with its supported toolchain and install the [vLLM TT plugin](https://github.com/tenstorrent/vllm-tt-plugin/blob/main/docs/install-vllm-tt.sh) in the TT-Metal environment. Use unmodified plugin `main`; no companion plugin PR is required. Set `HF_HOME` to a cache containing checkpoint revision `842da3794eaa0b77d5f08bae87a17459d91ff475`.
Build TT-Metal with its supported toolchain and install the [vLLM TT plugin](https://github.com/tenstorrent/vllm-tt-plugin/blob/main/docs/install-vllm-tt.sh) in the TT-Metal environment. Use the plugin with the state-slots fix (draft P1, branch `gemma31b-2-state-slots`: a continuing prefill keeps its device state slot when its batch row moves), the anchored sliding-window KV spec (draft P2, branch `gemma31b-3-anchor`: module `vllm_tt_plugin.kv_cache_anchor`, not on plugin `main`) and the speculative cap-zero change (draft P3, branch `gemma31b-11-spec-cap-zero-main`: only for speculation with more than four live requests); unmodified plugin `main` is not enough. Set `HF_HOME` to a cache containing checkpoint revision `842da3794eaa0b77d5f08bae87a17459d91ff475`.

```bash
export TT_METAL_HOME=$PWD
Expand All @@ -29,13 +29,19 @@ python -m vllm.entrypoints.openai.api_server \
--hf-overrides '{"architectures":["TTGemma4QB2ForCausalLM"]}' \
--host 127.0.0.1 --port 8000 --block-size 128 --max-num-seqs 32 \
--max-model-len 262144 --max-num-batched-tokens 8192 --max-logprobs -1 \
--async-scheduling --no-enable-prefix-caching --enable-chunked-prefill \
--enable-prefix-caching --async-scheduling --enable-chunked-prefill \
--override-generation-config '{"top_k":20}' \
--reasoning-parser gemma4 --default-chat-template-kwargs '{"enable_thinking":true}' \
--additional-config '{"tt":{"sample_on_device_mode":"all","trace_region_size":536870912,"l1_small_size":16384}}'
```

`HF_MODEL`, if set before checkpoint resolution, must name a local checkpoint directory. Use a new cache directory for different weights. The maximum context setting is allocation capacity; it does not establish accuracy at 262,144 tokens. Keep the default decode interleaving enabled and use a batched-token budget no greater than 8,192. Disabling interleaving exhausted DRAM in the tested 32-user configuration because more BF16 prefill histories remained live. Images, audio, speculative decoding and prefix caching are unsupported.
This is the many-agent profile (`gemma4_31b_qb2_c32` in tt-inference-server: prefix caching, async scheduling, no speculation). The few-user profile (`gemma4_31b_qb2`) is the same server with n-gram speculation and synchronous scheduling: replace `--async-scheduling` by `--no-async-scheduling --speculative-config '{"method":"ngram","num_speculative_tokens":5,"prompt_lookup_max":4,"prompt_lookup_min":2}'` and set `NUMBA_NUM_THREADS=1` in the server environment (vLLM refuses async scheduling with the n-gram method).

`HF_MODEL`, if set before checkpoint resolution, must name a local checkpoint directory. Use a new cache directory for different weights. The maximum context setting is allocation capacity; it does not establish accuracy at 262,144 tokens. Keep the default decode interleaving enabled and use a batched-token budget no greater than 8,192. Disabling interleaving exhausted DRAM in the tested 32-user configuration because more BF16 prefill histories remained live. Images and audio are unsupported.

Prefix caching is on in both profiles (`--enable-prefix-caching`). A fresh cached request reconstructs each sliding layer's preceding window from vLLM-owned BF8 pages, then retains the usual BF16 tail for its later chunks. The scheduler owns prefix hashes, page residency and eviction; the sliding groups use the plugin's `AnchoredSlidingWindowSpec`, which keeps a prompt's cache-hit window until the request ends so a long reply does not make it the first thing evicted. Request history uses private identities so concurrent requests sharing pages remain independent. Cached continuation is numerically approximate relative to a cold BF16 prefill; greedy output need not be identical. Targeted sliding-layer HF checks pass PCC ≥ 0.995 at 128-, 1024- and 1152-token boundaries. Scored agentic quality and performance validation is pending; the benchmark table below is an earlier measurement without prefix caching, speculation or the KV packing described above.

Speculative decoding with the plugin's host n-gram drafter is selected by the launch (`--speculative-config`, synchronous scheduling) and follows the number of live requests: a lone request verifies up to K = 5 drafts in a 6-row trace, up to four requests K = 3 in a 16-row trace, and more requests decode without drafts (the plugin's draft cap for those batch sizes is 0, which needs the plugin's cap-zero change). The model verifies every draft token, but verify runs more rows than a plain step, so a speculative server's streams need not be bit-identical to a plain server's. Real agent text accepts about 1.3-1.7 tokens per step, so it pays only with few concurrent users; the serving profile decides.

## Validation and weekly CI

Expand Down Expand Up @@ -77,4 +83,4 @@ The original experiment source scored **167/198 (84.34%) on full GPQA Diamond**,

Each server has a **20-minute wall-clock readiness deadline**, within a **40-minute total recipe allowance**. These are safety limits, not measured runtimes. Gemma and Llama account for **52 minutes** of the shared QB2 budget. Public phase timings contain only fixed stage names and elapsed seconds. Startup snapshots report capacity, elapsed time, a fixed stage label, a bounded layer index and an exception flag. Private evaluator/server diagnostics remain excluded. The [model PR](https://github.com/tenstorrent/tt-metal/pull/56765) tracks final integrated validation against plugin main.

Prefill compiles on demand. An unseen logical prefill shape retires decode traces before creating persistent program buffers; decode then warms and captures again. The first request for that shape can therefore include compilation and recapture latency. Reported performance uses a warmup burst per shape and does not measure this cold-request cost. The 512 MiB serving and 32 MB decoder-test trace reservations are validated budgets, not measured trace footprints or quantified safety margins.
Start-up compiles every page-padded prompt length from position 0 (`warmup_model_prefill`, 128-token steps), the logical lengths that select the last-token row, a continuation after a full sliding window for every padded length, the prefix-cache history restore for every page count and every prefill batch size; a second warm-up pass of the same cache is skipped. Prefill takes its position-dependent inputs (rope rows, cache-fill pages, chunked-attention start, last-token tile) as runtime values, so a continuation chunk at a new start or length compiles no program and the decode traces survive it. A prefill that still compiles new programs retires the decode traces, since new programs create persistent buffers; decode then warms and captures again, so that request includes compilation and recapture latency (about 1 s typically). That happens for a continuation after fewer than 1024 sliding tokens (a history of 128..896 rows) and for a continuation chunk of more than 7040 padded rows. The serving context must be a multiple of 1024 tokens. Reported performance uses a warmup burst per shape and does not measure this cold-request cost. The 512 MiB serving and 32 MB decoder-test trace reservations are validated budgets, not measured trace footprints or quantified safety margins.
29 changes: 28 additions & 1 deletion models/demos/gemma4_31b_qb2/tests/test_chunked_prefill.py
Original file line number Diff line number Diff line change
@@ -1,6 +1,9 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: 2026 Tenstorrent USA, Inc.
"""Resume checkpoint-weight prefills across aligned and partial cache pages."""

from types import SimpleNamespace

import pytest
import torch
from transformers import AutoConfig
Expand All @@ -9,6 +12,7 @@
import ttnn
from models.demos.gemma4_31b_qb2.tests.test_decoder import ReferenceCache, check_rows, reference
from models.demos.gemma4_31b_qb2.tt.decoder import Decoder
from models.demos.gemma4_31b_qb2.tt.generator import Gemma4Generator
from models.demos.gemma4_31b_qb2.tt.model import WeightReader, checkpoint_path


Expand All @@ -18,7 +22,7 @@
[(1024, 2048, 2177), (127, 1023, 1025, 2177), (1024, 2049, 3073, 4098), (1201, 2317, 3467, 4571)],
ids=["aligned", "partial-pages", "after-full-window", "ragged-windows"],
)
def test_resumed_prefill_matches_hf(qb2_mesh, layer_index, ends):
def test_resumed_prefill_matches_hf(qb2_mesh, layer_index, ends, cached_prefix=False):
torch.set_num_threads(8)
rng = torch.Generator().manual_seed(3600 + layer_index)
path = checkpoint_path()
Expand All @@ -36,6 +40,8 @@ def test_resumed_prefill_matches_hf(qb2_mesh, layer_index, ends):
layer._prepare_prefill_norm_stats()
pages = (ends[-1] + 128) // 128
table = torch.randperm(pages + 2, generator=rng)[2:].reshape(1, pages).int()
if cached_prefix:
table = (torch.randperm(pages, generator=rng) + 1).reshape(1, pages).int()
cache = layer.allocate_cache(physical_pages=pages + 2)
ref_cache = ReferenceCache(1, layer.kv_heads * 4, pages * 128, layer.head_dim)
embedding = reader.get("model.language_model.embed_tokens.weight")
Expand All @@ -57,6 +63,22 @@ def test_resumed_prefill_matches_hf(qb2_mesh, layer_index, ends):
valid = history[0].shape[2] - (previous_end - aligned_start)
begin = max(0, valid - 1024)
previous = tuple(t[:, :, begin:valid, :] for t in history) if valid else None
if cached_prefix and previous_end:
generator = object.__new__(Gemma4Generator)
# _restore_prefill_history uploads its slice bounds through the model's tensor helper.
generator.model = SimpleNamespace(
layers=[layer],
tensor=lambda x, dtype=None, layout=ttnn.TILE_LAYOUT: ttnn.from_torch(
x.contiguous(),
device=qb2_mesh,
dtype=dtype,
layout=layout,
mesh_mapper=ttnn.ReplicateTensorToMesh(qb2_mesh),
memory_config=ttnn.DRAM_MEMORY_CONFIG,
),
)
state = SimpleNamespace(kv=[cache])
previous = generator._restore_prefill_history(state, {0: table}, 0, aligned_start)[0]
x = inputs[:, aligned_start:end]
sharded = layer.prefill_input_width(end - aligned_start) != config.hidden_size
tx = ttnn.from_torch(
Expand Down Expand Up @@ -99,3 +121,8 @@ def test_resumed_prefill_matches_hf(qb2_mesh, layer_index, ends):
actual = ttnn.to_torch(ttnn.get_device_tensors(out)[0]).reshape(1, 1, -1)
expected = reference(hf, rope, dx, positions[:, None], ref_cache, layer.kind, layer.window)
check_rows(expected, actual)


@pytest.mark.parametrize("ends", [(128, 257), (1024, 1153), (1152, 2177)])
def test_cached_sliding_prefill_matches_hf(qb2_mesh, ends):
test_resumed_prefill_matches_hf(qb2_mesh, 0, ends, cached_prefix=True)
Loading
Loading