diff --git a/models/demos/gemma4_31b_qb2/README.md b/models/demos/gemma4_31b_qb2/README.md index ae6b4ea7be33..ca8753667ff8 100644 --- a/models/demos/gemma4_31b_qb2/README.md +++ b/models/demos/gemma4_31b_qb2/README.md @@ -4,15 +4,15 @@ Text generation for `google/gemma-4-31B-it` on one Blackhole QuietBox 2: four ch ## Design -The 60 decoder layers alternate five sliding-attention layers with one full-attention layer. Each chip holds a quarter of the projection weights. A ring fabric carries tensor-parallel reductions; completed sliding-attention sums retain BF16 precision. Decode replays a device trace with resident token, position, and sampling state. Scheduler updates replace inputs or page tables without resetting unrelated requests. +The 60 decoder layers alternate five sliding-attention layers with one full-attention layer. Each chip holds a quarter of the projection weights. A ring fabric carries tensor-parallel reductions; completed sliding-attention sums retain BF16 precision. Decode replays a device trace with resident token, position, and sampling state. A server keeps decode traces of 1, 4, 8, 16 and 32 rows (those below its slot count, plus the full one) over one KV pool and replays the smallest that covers the live requests, so a lone request decodes at the one-slot speed. Scheduler updates replace inputs or page tables without resetting unrelated requests. Prefill runs on whole cache pages with runtime-valued positions (rope rows, cache-fill pages, chunked-attention start, last-token row), so after start-up warm-up a continuation chunk at a new start or length compiles no program and leaves the decode traces alone (the exceptions are listed under start-up below). -Projection weights use BF4 and LoFi matmuls. The LM head uses BF8 and HiFi2. Residuals and normalization use BF16; KV storage uses BF8. Scheduler-driven chunked prefill uses the plugin's standard `SlidingWindowSpec` and paged full-attention history. Each sliding layer retains a BF16 K/V tail between scheduler calls, including one extra page for an unaligned continuation. Request completion, preemption and slot moves preserve or release that history through the existing plugin lifecycle. Sliding storage is bounded by the window and in-flight chunk budget. +Projection weights use BF8 and LoFi matmuls (one fixed format: BF4 projections break long agent runs). The LM head uses BF8 and HiFi2. Residuals and normalization use BF16; KV storage uses BF8. A layer keeps one copy of its projection weights: decode reads the DRAM-sharded copy and prefill rebuilds its interleaved weights from it once per layer and prefill span (the chunks of one span share them). Scheduler-driven chunked prefill uses the plugin's `AnchoredSlidingWindowSpec` (a request's cache-hit window stays until the request ends) and paged full-attention history. Each sliding layer retains a BF16 K/V tail between scheduler calls, including one extra page for an unaligned continuation. Request completion, preemption and slot moves preserve or release that history through the existing plugin lifecycle. The KV pool is packed: a pool position (one full-attention layer and its five sliding layers) shares one backing of wide rows, the scheduler gives the full-attention group 256-token blocks and the sliding groups 128-token blocks over one block-ID space, and a full block reads its row as two native 128-token half-pages, so the same DRAM holds twice the full-attention tokens and the separate full-attention tensors (about 0.7 GB per DRAM bank) are gone; prefix hits of the full-attention group are 256-token aligned. Sliding storage is bounded by the window and in-flight chunk budget. Greedy decoding and stochastic sampling with `top_k=1..32` are supported. Launch with `--override-generation-config '{"top_k":20}'` so requests that omit top-k use a supported default. Request seeds must be integers in `0..2³¹−1`. Larger/unbounded top-k and wider seeds are outside this serving contract until the independent plugin fixes land. Penalties, logprobs and other host-only parameters use the existing host sampler. Host and device samplers need not produce the same random sequence. ## Run -Build TT-Metal with its supported toolchain and install the [vLLM TT plugin](https://github.com/tenstorrent/vllm-tt-plugin/blob/main/docs/install-vllm-tt.sh) in the TT-Metal environment. Use unmodified plugin `main`; no companion plugin PR is required. Set `HF_HOME` to a cache containing checkpoint revision `842da3794eaa0b77d5f08bae87a17459d91ff475`. +Build TT-Metal with its supported toolchain and install the [vLLM TT plugin](https://github.com/tenstorrent/vllm-tt-plugin/blob/main/docs/install-vllm-tt.sh) in the TT-Metal environment. Use the plugin with the state-slots fix (draft P1, branch `gemma31b-2-state-slots`: a continuing prefill keeps its device state slot when its batch row moves), the anchored sliding-window KV spec (draft P2, branch `gemma31b-3-anchor`: module `vllm_tt_plugin.kv_cache_anchor`, not on plugin `main`) and the speculative cap-zero change (draft P3, branch `gemma31b-11-spec-cap-zero-main`: only for speculation with more than four live requests); unmodified plugin `main` is not enough. Set `HF_HOME` to a cache containing checkpoint revision `842da3794eaa0b77d5f08bae87a17459d91ff475`. ```bash export TT_METAL_HOME=$PWD @@ -29,13 +29,19 @@ python -m vllm.entrypoints.openai.api_server \ --hf-overrides '{"architectures":["TTGemma4QB2ForCausalLM"]}' \ --host 127.0.0.1 --port 8000 --block-size 128 --max-num-seqs 32 \ --max-model-len 262144 --max-num-batched-tokens 8192 --max-logprobs -1 \ - --async-scheduling --no-enable-prefix-caching --enable-chunked-prefill \ + --enable-prefix-caching --async-scheduling --enable-chunked-prefill \ --override-generation-config '{"top_k":20}' \ --reasoning-parser gemma4 --default-chat-template-kwargs '{"enable_thinking":true}' \ --additional-config '{"tt":{"sample_on_device_mode":"all","trace_region_size":536870912,"l1_small_size":16384}}' ``` -`HF_MODEL`, if set before checkpoint resolution, must name a local checkpoint directory. Use a new cache directory for different weights. The maximum context setting is allocation capacity; it does not establish accuracy at 262,144 tokens. Keep the default decode interleaving enabled and use a batched-token budget no greater than 8,192. Disabling interleaving exhausted DRAM in the tested 32-user configuration because more BF16 prefill histories remained live. Images, audio, speculative decoding and prefix caching are unsupported. +This is the many-agent profile (`gemma4_31b_qb2_c32` in tt-inference-server: prefix caching, async scheduling, no speculation). The few-user profile (`gemma4_31b_qb2`) is the same server with n-gram speculation and synchronous scheduling: replace `--async-scheduling` by `--no-async-scheduling --speculative-config '{"method":"ngram","num_speculative_tokens":5,"prompt_lookup_max":4,"prompt_lookup_min":2}'` and set `NUMBA_NUM_THREADS=1` in the server environment (vLLM refuses async scheduling with the n-gram method). + +`HF_MODEL`, if set before checkpoint resolution, must name a local checkpoint directory. Use a new cache directory for different weights. The maximum context setting is allocation capacity; it does not establish accuracy at 262,144 tokens. Keep the default decode interleaving enabled and use a batched-token budget no greater than 8,192. Disabling interleaving exhausted DRAM in the tested 32-user configuration because more BF16 prefill histories remained live. Images and audio are unsupported. + +Prefix caching is on in both profiles (`--enable-prefix-caching`). A fresh cached request reconstructs each sliding layer's preceding window from vLLM-owned BF8 pages, then retains the usual BF16 tail for its later chunks. The scheduler owns prefix hashes, page residency and eviction; the sliding groups use the plugin's `AnchoredSlidingWindowSpec`, which keeps a prompt's cache-hit window until the request ends so a long reply does not make it the first thing evicted. Request history uses private identities so concurrent requests sharing pages remain independent. Cached continuation is numerically approximate relative to a cold BF16 prefill; greedy output need not be identical. Targeted sliding-layer HF checks pass PCC ≥ 0.995 at 128-, 1024- and 1152-token boundaries. Scored agentic quality and performance validation is pending; the benchmark table below is an earlier measurement without prefix caching, speculation or the KV packing described above. + +Speculative decoding with the plugin's host n-gram drafter is selected by the launch (`--speculative-config`, synchronous scheduling) and follows the number of live requests: a lone request verifies up to K = 5 drafts in a 6-row trace, up to four requests K = 3 in a 16-row trace, and more requests decode without drafts (the plugin's draft cap for those batch sizes is 0, which needs the plugin's cap-zero change). The model verifies every draft token, but verify runs more rows than a plain step, so a speculative server's streams need not be bit-identical to a plain server's. Real agent text accepts about 1.3-1.7 tokens per step, so it pays only with few concurrent users; the serving profile decides. ## Validation and weekly CI @@ -77,4 +83,4 @@ The original experiment source scored **167/198 (84.34%) on full GPQA Diamond**, Each server has a **20-minute wall-clock readiness deadline**, within a **40-minute total recipe allowance**. These are safety limits, not measured runtimes. Gemma and Llama account for **52 minutes** of the shared QB2 budget. Public phase timings contain only fixed stage names and elapsed seconds. Startup snapshots report capacity, elapsed time, a fixed stage label, a bounded layer index and an exception flag. Private evaluator/server diagnostics remain excluded. The [model PR](https://github.com/tenstorrent/tt-metal/pull/56765) tracks final integrated validation against plugin main. -Prefill compiles on demand. An unseen logical prefill shape retires decode traces before creating persistent program buffers; decode then warms and captures again. The first request for that shape can therefore include compilation and recapture latency. Reported performance uses a warmup burst per shape and does not measure this cold-request cost. The 512 MiB serving and 32 MB decoder-test trace reservations are validated budgets, not measured trace footprints or quantified safety margins. +Start-up compiles every page-padded prompt length from position 0 (`warmup_model_prefill`, 128-token steps), the logical lengths that select the last-token row, a continuation after a full sliding window for every padded length, the prefix-cache history restore for every page count and every prefill batch size; a second warm-up pass of the same cache is skipped. Prefill takes its position-dependent inputs (rope rows, cache-fill pages, chunked-attention start, last-token tile) as runtime values, so a continuation chunk at a new start or length compiles no program and the decode traces survive it. A prefill that still compiles new programs retires the decode traces, since new programs create persistent buffers; decode then warms and captures again, so that request includes compilation and recapture latency (about 1 s typically). That happens for a continuation after fewer than 1024 sliding tokens (a history of 128..896 rows) and for a continuation chunk of more than 7040 padded rows. The serving context must be a multiple of 1024 tokens. Reported performance uses a warmup burst per shape and does not measure this cold-request cost. The 512 MiB serving and 32 MB decoder-test trace reservations are validated budgets, not measured trace footprints or quantified safety margins. diff --git a/models/demos/gemma4_31b_qb2/tests/test_chunked_prefill.py b/models/demos/gemma4_31b_qb2/tests/test_chunked_prefill.py index 1346cf8b55dd..13db33171050 100644 --- a/models/demos/gemma4_31b_qb2/tests/test_chunked_prefill.py +++ b/models/demos/gemma4_31b_qb2/tests/test_chunked_prefill.py @@ -1,6 +1,9 @@ # SPDX-License-Identifier: Apache-2.0 # SPDX-FileCopyrightText: 2026 Tenstorrent USA, Inc. """Resume checkpoint-weight prefills across aligned and partial cache pages.""" + +from types import SimpleNamespace + import pytest import torch from transformers import AutoConfig @@ -9,6 +12,7 @@ import ttnn from models.demos.gemma4_31b_qb2.tests.test_decoder import ReferenceCache, check_rows, reference from models.demos.gemma4_31b_qb2.tt.decoder import Decoder +from models.demos.gemma4_31b_qb2.tt.generator import Gemma4Generator from models.demos.gemma4_31b_qb2.tt.model import WeightReader, checkpoint_path @@ -18,7 +22,7 @@ [(1024, 2048, 2177), (127, 1023, 1025, 2177), (1024, 2049, 3073, 4098), (1201, 2317, 3467, 4571)], ids=["aligned", "partial-pages", "after-full-window", "ragged-windows"], ) -def test_resumed_prefill_matches_hf(qb2_mesh, layer_index, ends): +def test_resumed_prefill_matches_hf(qb2_mesh, layer_index, ends, cached_prefix=False): torch.set_num_threads(8) rng = torch.Generator().manual_seed(3600 + layer_index) path = checkpoint_path() @@ -36,6 +40,8 @@ def test_resumed_prefill_matches_hf(qb2_mesh, layer_index, ends): layer._prepare_prefill_norm_stats() pages = (ends[-1] + 128) // 128 table = torch.randperm(pages + 2, generator=rng)[2:].reshape(1, pages).int() + if cached_prefix: + table = (torch.randperm(pages, generator=rng) + 1).reshape(1, pages).int() cache = layer.allocate_cache(physical_pages=pages + 2) ref_cache = ReferenceCache(1, layer.kv_heads * 4, pages * 128, layer.head_dim) embedding = reader.get("model.language_model.embed_tokens.weight") @@ -57,6 +63,22 @@ def test_resumed_prefill_matches_hf(qb2_mesh, layer_index, ends): valid = history[0].shape[2] - (previous_end - aligned_start) begin = max(0, valid - 1024) previous = tuple(t[:, :, begin:valid, :] for t in history) if valid else None + if cached_prefix and previous_end: + generator = object.__new__(Gemma4Generator) + # _restore_prefill_history uploads its slice bounds through the model's tensor helper. + generator.model = SimpleNamespace( + layers=[layer], + tensor=lambda x, dtype=None, layout=ttnn.TILE_LAYOUT: ttnn.from_torch( + x.contiguous(), + device=qb2_mesh, + dtype=dtype, + layout=layout, + mesh_mapper=ttnn.ReplicateTensorToMesh(qb2_mesh), + memory_config=ttnn.DRAM_MEMORY_CONFIG, + ), + ) + state = SimpleNamespace(kv=[cache]) + previous = generator._restore_prefill_history(state, {0: table}, 0, aligned_start)[0] x = inputs[:, aligned_start:end] sharded = layer.prefill_input_width(end - aligned_start) != config.hidden_size tx = ttnn.from_torch( @@ -99,3 +121,8 @@ def test_resumed_prefill_matches_hf(qb2_mesh, layer_index, ends): actual = ttnn.to_torch(ttnn.get_device_tensors(out)[0]).reshape(1, 1, -1) expected = reference(hf, rope, dx, positions[:, None], ref_cache, layer.kind, layer.window) check_rows(expected, actual) + + +@pytest.mark.parametrize("ends", [(128, 257), (1024, 1153), (1152, 2177)]) +def test_cached_sliding_prefill_matches_hf(qb2_mesh, ends): + test_resumed_prefill_matches_hf(qb2_mesh, 0, ends, cached_prefix=True) diff --git a/models/demos/gemma4_31b_qb2/tests/test_continuation_programs.py b/models/demos/gemma4_31b_qb2/tests/test_continuation_programs.py new file mode 100644 index 000000000000..740c43b3d15c --- /dev/null +++ b/models/demos/gemma4_31b_qb2/tests/test_continuation_programs.py @@ -0,0 +1,188 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: 2026 Tenstorrent USA, Inc. +"""A continuation prefill compiles no program for a (length, start) it has not seen, and its layer output equals the per-length one bit for bit. + +The position-dependent ops of a prefill chunk (rope rows, cache-fill page ids, chunked-SDPA start, last-token tile) take +runtime values and the prompt is padded to whole cache pages, so the programs depend only on the padded chunk length. +warm_prefill_shapes compiles those; any later signature then adds nothing and cannot retire the decode traces. +""" + +import pytest +import torch + +import ttnn +from models.demos.gemma4_31b_qb2.tt.generator import build_generator + +CAPACITY = 5120 # 40 pages: the chunked-SDPA page-table row must be a multiple of 32 bytes +# (length, start): unaligned lengths, aligned and unaligned starts, one and several chunks, never seen by the warm-up. +SIGNATURES = [(2327, 1152), (2701, 2304), (3000, 1152), (1300, 1152), (4100, 2048), (1537, 1024)] + + +@pytest.fixture +def generator(qb2_mesh): + gen = build_generator(qb2_mesh, layer_indices=[0, 5], max_seq_len=CAPACITY) + yield gen + gen.close() + + +def test_novel_continuation_signatures_add_no_programs(generator, qb2_mesh): + gen = generator + state = gen.allocate_cache(batch=1, capacity=CAPACITY) + tokens = torch.randint(1000, 50000, (1, CAPACITY), generator=torch.Generator().manual_seed(1)) + gen.warm_prefill_shapes(state, state.page_tables, CAPACITY) + key = iter(range(1, 1000)) + + def run(length, start, **kwargs): + history = next(key) + if start: + gen.prefill_forward( + tokens, page_table=state.page_tables, kv_cache=state, prompt_lens=[start], history_keys=[history] + ) + return gen.prefill_forward( + tokens, + page_table=state.page_tables, + kv_cache=state, + prompt_lens=[length], + start_positions=[start], + history_keys=[history], + **kwargs, + ) + + for length, start in SIGNATURES: + programs = qb2_mesh.num_program_cache_entries() + run(length, start) + assert qb2_mesh.num_program_cache_entries() == programs, f"({length}, {start}) compiled programs" + + +@pytest.mark.parametrize( + "layer_index", [0, 1], ids=["sliding", "full"] +) # the loaded layers are checkpoint layers 0 and 5 +@pytest.mark.parametrize("length, start", [(1175, 1152), (1300, 1280), (300, 1024)]) +def test_runtime_chunk_inputs_give_the_same_layer_output_bit_for_bit(generator, qb2_mesh, layer_index, length, start): + """Layer level: the continuation chunk with runtime-valued inputs (padded to whole pages) equals the per-length one.""" + layer = generator.model.layers[layer_index] + layer._prepare_prefill_norm_stats() + # The chunked-SDPA page-table row must be a multiple of 32 bytes: a multiple of 8 pages. + pages = ((start + length + 127) // 128 + 2 + 7) // 8 * 8 + torch.manual_seed(layer_index) + table = (torch.randperm(pages) + 1).reshape(1, pages).int() + cache = layer.allocate_cache(physical_pages=pages + 2) + replicate = ttnn.ReplicateTensorToMesh(qb2_mesh) + pt = ttnn.from_torch(table, device=qb2_mesh, dtype=ttnn.int32, layout=ttnn.ROW_MAJOR_LAYOUT, mesh_mapper=replicate) + + def hidden(rows): + x = torch.randn(1, 1, rows, layer.hidden_size).to(torch.bfloat16) + return x + + def upload(x): + sharded = layer.prefill_input_width(x.shape[2]) != layer.hidden_size + return ( + ttnn.from_torch( + x, + device=qb2_mesh, + dtype=ttnn.bfloat16, + layout=ttnn.TILE_LAYOUT, + mesh_mapper=ttnn.ShardTensorToMesh(qb2_mesh, dim=3) if sharded else replicate, + ), + sharded, + ) + + def read(out, sharded): + parts = [ttnn.to_torch(t) for t in ttnn.get_device_tensors(out)] + return torch.cat(parts, dim=-1) if sharded else parts[0] + + # The preceding chunk: its cache pages and (sliding) history tail. + first, sharded = upload(hidden(start)) + _, history = layer.prefill_forward(first, page_table=pt, kv_cache=cache, start_pos=0, return_history=True) + previous = None + if history is not None: + valid = history[0].shape[2] + begin = max(0, valid - 1024) + previous = tuple(t[:, :, begin:valid, :] for t in history) + x = hidden(length) + padded = (length + 127) // 128 * 128 + x_padded = torch.zeros(1, 1, padded, layer.hidden_size, dtype=torch.bfloat16) + x_padded[:, :, :length] = x + + tx, sharded = upload(x) + expected, _ = layer.prefill_forward( + tx, page_table=pt, kv_cache=cache, start_pos=start, history=previous, return_history=True + ) + expected = read(expected, sharded)[:, :, :length] + + tx, sharded = upload(x_padded) + # As Gemma4Model.prefill_device builds them: rope rows per layer kind, fill pages and start per page-table row. + rope = layer.prefill_chunk_inputs(padded, start) + fills = layer.prefill_chunk_inputs(padded, start, table) + inputs = [dict(f, rope=r["rope"]) for r, f in zip(rope, fills)] + actual, _ = layer.prefill_forward( + tx, + page_table=pt, + kv_cache=cache, + start_pos=start, + history=previous, + return_history=True, + chunk_inputs=inputs, + padded_tails=True, + ) + actual = read(actual, sharded)[:, :, :length] + assert torch.equal(actual, expected), f"layer {layer_index} ({length}, {start}): outputs differ" + + +# Scheduler chunk ends of one prompt, as chunked prefill produces them: unaligned starts, chunks of different length, one +# chunk of several 1024-token sliding chunks, an aligned start after an unaligned one. +CHAINS = [(1000, 2100, 3000), (300, 1500, 2560, 3500), (2600, 3500)] + + +def _kv_rows(state, tables, layer, positions): + """K and V cache rows of the given absolute positions (every device), as one float tensor per cache.""" + kind = tables[layer] + rows = [] + for cache in state.kv[layer]: + per_device = [ttnn.to_torch(t).float() for t in ttnn.get_device_tensors(cache)] + page = state.page_tables[kind][0] + rows.append(torch.stack([torch.stack([d[page[p // 128], :, p % 128] for p in positions]) for d in per_device])) + return rows + + +@pytest.mark.parametrize("ends", CHAINS, ids=lambda ends: "-".join(map(str, ends))) +def test_unaligned_chunk_chains_match_the_per_length_path(generator, ends): + """Chunk chains with unaligned starts: last-row logits and the KV pages of the valid rows against the per-length path. + + The runtime path (padded chunks, page-padded saved tails, rewind to the aligned start) runs on one cache; the all-logits + path (per-length slices and programs) on another. The logits differ only by the head's matmul shape (every row against + one row), so they are compared by PCC; the KV rows of the last window come from the same projections and are equal. + """ + gen = generator + runtime, reference = (gen.allocate_cache(batch=1, capacity=CAPACITY) for _ in range(2)) + tokens = torch.randint(1000, 50000, (1, CAPACITY), generator=torch.Generator().manual_seed(7)) + previous = 0 + for end in ends: + last = gen.prefill_forward( + tokens, + page_table=runtime.page_tables, + kv_cache=runtime, + prompt_lens=[end], + start_positions=[previous], + history_keys=[1], + ) + everything = gen.prefill_forward( + tokens, + page_table=reference.page_tables, + kv_cache=reference, + prompt_lens=[end], + start_positions=[previous], + history_keys=[2], + return_all_logits=True, + ) + row = end - previous // 128 * 128 - 1 + actual, expected = last.float().flatten(), everything[:, row].float().flatten() + pcc = torch.corrcoef(torch.stack([actual, expected]))[0, 1].item() + assert pcc >= 0.9999, f"chunk ending at {end} (start {previous}): PCC {pcc}" + previous = end + for layer, layer_object in enumerate(gen.model.layers): + window = 1024 if layer_object.sliding else previous + positions = list(range(max(0, previous - window), previous)) + tables = {layer: layer_object.kind} + for a, b in zip(_kv_rows(runtime, tables, layer, positions), _kv_rows(reference, tables, layer, positions)): + assert torch.equal(a, b), f"layer {layer}: KV rows differ after the chain {ends}" diff --git a/models/demos/gemma4_31b_qb2/tests/test_decode_columns.py b/models/demos/gemma4_31b_qb2/tests/test_decode_columns.py new file mode 100644 index 000000000000..4c0b86379b75 --- /dev/null +++ b/models/demos/gemma4_31b_qb2/tests/test_decode_columns.py @@ -0,0 +1,87 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: 2026 Tenstorrent USA, Inc. +"""A speculative block is exact: draft columns of one user in one decode call equal the same steps run one by one. + +The columns write K/V at consecutive positions, which share a cache tile; the decoder updates the cache once per +column so the writes do not race. +""" + +import pytest +import torch +from transformers import AutoConfig + +import ttnn +from models.demos.gemma4_31b_qb2.tt.decoder import Decoder +from models.demos.gemma4_31b_qb2.tt.model import WeightReader, checkpoint_path + +COLUMNS = 4 +FIRST_POSITION = 1500 # inside a cache tile, so the block's positions share it + + +@pytest.mark.parametrize("layer_index", [0, 5], ids=["sliding", "full"]) +def test_draft_columns_equal_sequential_decode_steps(qb2_mesh, layer_index): + rng = torch.Generator().manual_seed(5) + path = checkpoint_path() + config = AutoConfig.from_pretrained(path, local_files_only=True).text_config + reader = WeightReader(path) + layer = Decoder.from_state_dict( + reader.layer(layer_index), hf_config=config, layer_idx=layer_index, mesh_device=qb2_mesh + ) + embedding = reader.get("model.language_model.embed_tokens.weight") + pages = (FIRST_POSITION + COLUMNS + 127) // 128 + 1 + physical = pages + COLUMNS + 2 + cache = layer.allocate_cache(physical_pages=physical) + history = [torch.randn(tuple(t.shape), generator=rng).bfloat16() * 0.5 for t in cache] + table = (torch.randperm(physical - COLUMNS - 2, generator=rng)[:pages] + COLUMNS + 2).int() + ids = torch.randint(100, config.vocab_size, (COLUMNS,), generator=rng) + hidden = (embedding[ids] * torch.tensor(config.hidden_size**0.5, dtype=embedding.dtype)).float() + replicate = ttnn.ReplicateTensorToMesh(qb2_mesh) + + def reset_cache(): + for tensor, host in zip(cache, history): + ttnn.copy_host_to_device_tensor( + ttnn.from_torch(host, dtype=tensor.dtype, layout=ttnn.TILE_LAYOUT, mesh_mapper=replicate), tensor + ) + + def device(values, integer=False): + return ttnn.from_torch( + values.contiguous(), + device=qb2_mesh, + dtype=ttnn.int32 if integer else ttnn.bfloat16, + layout=ttnn.ROW_MAJOR_LAYOUT if integer else ttnn.TILE_LAYOUT, + mesh_mapper=replicate, + ) + + def decode(rows, positions, *, per_column_updates=False): + updates = None + if per_column_updates: + updates = [ + device(torch.where(torch.arange(COLUMNS) == c, positions, -1).int(), True) for c in range(COLUMNS) + ] + out = layer.decode_forward( + device(rows.reshape(1, 1, COLUMNS, -1)), + positions=device(positions, True), + page_table=device(table.unsqueeze(0).repeat(COLUMNS, 1), True), + kv_cache=cache, + cyclic_cache=False, + update_positions=updates, + ) + return ( + ttnn.to_torch(out, mesh_composer=ttnn.ConcatMeshToTensor(qb2_mesh, dim=0)) + .float() + .reshape(4, COLUMNS, -1)[0] + ) + + reset_cache() + block = decode( + hidden, torch.arange(FIRST_POSITION, FIRST_POSITION + COLUMNS, dtype=torch.int32), per_column_updates=True + ) + reset_cache() + for column in range(COLUMNS): + rows = torch.zeros(COLUMNS, hidden.shape[1]) + rows[0] = hidden[column] + positions = torch.full((COLUMNS,), -1, dtype=torch.int32) + positions[0] = FIRST_POSITION + column + assert torch.equal( + block[column], decode(rows, positions)[0] + ), f"column {column} differs from its sequential step" diff --git a/models/demos/gemma4_31b_qb2/tests/test_decode_geometry.py b/models/demos/gemma4_31b_qb2/tests/test_decode_geometry.py new file mode 100644 index 000000000000..d5dbe6a311ef --- /dev/null +++ b/models/demos/gemma4_31b_qb2/tests/test_decode_geometry.py @@ -0,0 +1,37 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: 2026 Tenstorrent USA, Inc. +"""Decode matmul geometry (input cores, readers per DRAM bank) only moves data: the result does not change.""" + +import pytest +import torch +from transformers import AutoConfig + +import ttnn +from models.demos.gemma4_31b_qb2.tt import decoder as decoder_module +from models.demos.gemma4_31b_qb2.tt.decoder import Decoder +from models.demos.gemma4_31b_qb2.tt.model import WeightReader, checkpoint_path + + +@pytest.mark.parametrize("layer_index", [0, 5], ids=["sliding", "full"]) +def test_o_projection_readers_per_bank_do_not_change_the_result(qb2_mesh, layer_index, monkeypatch): + path = checkpoint_path() + config = AutoConfig.from_pretrained(path, local_files_only=True).text_config + weights = WeightReader(path).layer(layer_index) + cores, _, k_block = decoder_module.DECODE_GEOMETRY["o"] + outputs = {} + for readers in (1, 2): + monkeypatch.setitem(decoder_module.DECODE_GEOMETRY, "o", (cores, readers, k_block)) + layer = Decoder.from_state_dict(weights, hf_config=config, layer_idx=layer_index, mesh_device=qb2_mesh) + k = layer.decode_weights["o"].shape[-2] + x = torch.randn(1, 1, 32, k, generator=torch.Generator().manual_seed(1)).bfloat16() + device_x = ttnn.from_torch( + x, + device=qb2_mesh, + dtype=ttnn.bfloat16, + layout=ttnn.TILE_LAYOUT, + mesh_mapper=ttnn.ReplicateTensorToMesh(qb2_mesh), + memory_config=ttnn.DRAM_MEMORY_CONFIG, + ) + y = layer._dlinear(device_x, "o") + outputs[readers] = ttnn.to_torch(y, mesh_composer=ttnn.ConcatMeshToTensor(qb2_mesh, dim=0)).float() + assert torch.equal(outputs[1], outputs[2]) diff --git a/models/demos/gemma4_31b_qb2/tests/test_decode_rows_host.py b/models/demos/gemma4_31b_qb2/tests/test_decode_rows_host.py new file mode 100644 index 000000000000..d5f8d25add83 --- /dev/null +++ b/models/demos/gemma4_31b_qb2/tests/test_decode_rows_host.py @@ -0,0 +1,66 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: 2026 Tenstorrent USA, Inc. +"""Choice of the decode trace row counts and of the trace for a step (host only).""" + +import pytest + +from models.demos.gemma4_31b_qb2.tt.decode_rows import ( + ROW_SIZES, + decode_row_sizes, + rows_covering, + verify_shapes, + verify_users, +) + + +@pytest.mark.parametrize( + "slots, expected", + [(1, ()), (2, (1,)), (4, (1,)), (5, (1, 4)), (8, (1, 4)), (16, (1, 4, 8)), (32, (1, 4, 8, 16)), (12, (1, 4, 8))], +) +def test_extra_trace_sizes_are_the_standard_ones_below_the_slot_count(slots, expected): + assert decode_row_sizes(slots) == expected + + +@pytest.mark.parametrize( + "slots, k, expected", + [ + (32, 5, ((1, 5), (4, 3))), + (4, 5, ((1, 5), (4, 3))), + (2, 5, ((1, 5), (2, 3))), + (1, 5, ((1, 5),)), + (32, 2, ((1, 2), (4, 2))), + (32, 0, ()), + ], +) +def test_verify_traces_follow_the_users_and_the_requested_draft_length(slots, k, expected): + assert verify_shapes(slots, k) == expected + + +def test_every_verify_trace_fits_the_decode_tile(): + for slots in (1, 2, 4, 8, 32): + for k in range(1, 8): + assert all(users * (1 + kk) <= 32 for users, kk in verify_shapes(slots, k)) + + +@pytest.mark.parametrize( + "live, expected", + [(0, 1), (1, 1), (2, 4), (4, 4), (5, 8), (8, 8), (9, 16), (16, 16), (17, 32), (32, 32)], +) +def test_smallest_covering_trace(live, expected): + assert rows_covering(live, decode_row_sizes(32), 32) == expected + + +def test_a_small_server_falls_back_to_its_full_trace(): + sizes = decode_row_sizes(6) + assert sizes == (1, 4) + assert rows_covering(5, sizes, 6) == 6 + assert rows_covering(4, sizes, 6) == 4 + + +def test_sizes_are_sorted_and_the_default_set_is_fixed(): + assert ROW_SIZES == tuple(sorted(ROW_SIZES)) == (1, 4, 8, 16, 32) + + +@pytest.mark.parametrize("live, expected", [(1, 1), (2, 4), (4, 4), (5, None), (8, None), (32, None)]) +def test_verify_users_is_none_when_more_requests_are_live_than_any_verify_trace_covers(live, expected): + assert verify_users(live, verify_shapes(32, 5)) == expected diff --git a/models/demos/gemma4_31b_qb2/tests/test_decode_rows_layer.py b/models/demos/gemma4_31b_qb2/tests/test_decode_rows_layer.py new file mode 100644 index 000000000000..a84930601c92 --- /dev/null +++ b/models/demos/gemma4_31b_qb2/tests/test_decode_rows_layer.py @@ -0,0 +1,82 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: 2026 Tenstorrent USA, Inc. +"""A decode step of a layer with few rows agrees with the 32-row step to within the layer accuracy bound. + +The decode SDPA and the sharded memory configs depend on the row count (a lone row spreads its KV range over more cores), +so the results are not bit-identical across traces; they agree to PCC >= PCC_MIN, well inside the PCC >= 0.995 bound the +decoder tests hold against the HF reference. The end-to-end check is the real-prompt panel (greedy and seeded outputs +of the 32-slot server against the 1-slot server) and the teacher-forced KL over recorded agent prompts. +""" + +import pytest +import torch +from transformers import AutoConfig + +import ttnn +from models.demos.gemma4_31b_qb2.tt.decoder import Decoder +from models.demos.gemma4_31b_qb2.tt.model import WeightReader, checkpoint_path + +PCC_MIN = 0.998 +ROWS = (1, 4, 8, 16, 32) + + +@pytest.mark.parametrize("layer_index", [0, 5], ids=["sliding", "full"]) +def test_decode_rows_agree_with_the_32_row_trace(qb2_mesh, layer_index): + rng = torch.Generator().manual_seed(11) + path = checkpoint_path() + config = AutoConfig.from_pretrained(path, local_files_only=True).text_config + reader = WeightReader(path) + layer = Decoder.from_state_dict( + reader.layer(layer_index), hf_config=config, layer_idx=layer_index, mesh_device=qb2_mesh + ) + embedding = reader.get("model.language_model.embed_tokens.weight") + pages = 16 + physical = pages + 40 + cache = layer.allocate_cache(physical_pages=physical) + history = [torch.randn(tuple(t.shape), generator=rng).bfloat16() * 0.5 for t in cache] + table = (torch.randperm(physical - 40, generator=rng)[:pages] + 34).int() + replicate = ttnn.ReplicateTensorToMesh(qb2_mesh) + + def device(values, integer=False): + return ttnn.from_torch( + values.contiguous(), + device=qb2_mesh, + dtype=ttnn.int32 if integer else ttnn.bfloat16, + layout=ttnn.ROW_MAJOR_LAYOUT if integer else ttnn.TILE_LAYOUT, + mesh_mapper=replicate, + ) + + def decode(rows, hidden, position): + for tensor, host in zip(cache, history): + ttnn.copy_host_to_device_tensor( + ttnn.from_torch(host, dtype=tensor.dtype, layout=ttnn.TILE_LAYOUT, mesh_mapper=replicate), tensor + ) + x = torch.zeros(1, 1, rows, config.hidden_size) + x[0, 0, 0] = hidden + positions = torch.full((rows,), -1, dtype=torch.int32) + positions[0] = position + out = layer.decode_forward( + device(x), + positions=device(positions, True), + rope_positions=device(positions.clamp_min(0), True), + page_table=device(table.unsqueeze(0).repeat(rows, 1), True), + kv_cache=cache, + cyclic_cache=False, + ) + return ( + ttnn.to_torch(out, mesh_composer=ttnn.ConcatMeshToTensor(qb2_mesh, dim=0))[0] + .reshape(-1, config.hidden_size)[0] + .float() + ) + + worst = 1.0 + for _ in range(4): + token = int(torch.randint(100, config.vocab_size, (1,), generator=rng)) + position = int(torch.randint(150, 1900, (1,), generator=rng)) + hidden = (embedding[token] * torch.tensor(config.hidden_size**0.5, dtype=embedding.dtype)).float() + reference = decode(32, hidden, position) + for rows in ROWS[:-1]: + pcc = float(torch.corrcoef(torch.stack([decode(rows, hidden, position), reference]))[0, 1]) + worst = min(worst, pcc) + assert pcc >= PCC_MIN, f"{rows}-row trace vs 32-row trace: PCC {pcc:.5f} at position {position}" + print(f"layer {layer_index}: worst PCC vs the 32-row trace {worst:.5f}") diff --git a/models/demos/gemma4_31b_qb2/tests/test_decoder.py b/models/demos/gemma4_31b_qb2/tests/test_decoder.py index 9b45c97ddf9e..2f6427bb7e14 100644 --- a/models/demos/gemma4_31b_qb2/tests/test_decoder.py +++ b/models/demos/gemma4_31b_qb2/tests/test_decoder.py @@ -9,6 +9,7 @@ import ttnn from models.common.utility_functions import comp_pcc +from models.demos.gemma4_31b_qb2.tt import decoder as decoder_module from models.demos.gemma4_31b_qb2.tt.decoder import Decoder from models.demos.gemma4_31b_qb2.tt.model import WeightReader, checkpoint_path @@ -55,9 +56,10 @@ def check_rows(expected, actual): assert passed, message +@pytest.mark.parametrize("weight_dtype", [ttnn.bfloat4_b, ttnn.bfloat8_b], ids=["bf4", "bf8"]) @pytest.mark.parametrize("layer_index", [0, 5], ids=["sliding", "full"]) @pytest.mark.parametrize("length,batch", [(1, 13), (127, 1), (129, 3), (1025, 2)]) -def test_prefill_and_live_decode_trace(qb2_mesh, layer_index, length, batch): +def test_prefill_and_live_decode_trace(qb2_mesh, layer_index, length, batch, weight_dtype, monkeypatch): """Every logical row is scored; changed inputs, positions and slot maps reach the trace.""" torch.set_num_threads(8) rng = torch.Generator().manual_seed(3600 + layer_index + length) @@ -72,6 +74,8 @@ def test_prefill_and_live_decode_trace(qb2_mesh, layer_index, length, batch): hf.load_state_dict({k[len(prefix) :]: v for k, v in weights.items()}, assign=True, strict=True) hf.eval() rope = Gemma4TextRotaryEmbedding(config) + for name in decoder_module.WEIGHT_DTYPES: + monkeypatch.setitem(decoder_module.WEIGHT_DTYPES, name, weight_dtype) layer = Decoder.from_state_dict(weights, hf_config=config, layer_idx=layer_index, mesh_device=qb2_mesh) layer._prepare_prefill_norm_stats() capacity = ((length + 4 + 127) // 128) * 128 diff --git a/models/demos/gemma4_31b_qb2/tests/test_history_restore.py b/models/demos/gemma4_31b_qb2/tests/test_history_restore.py new file mode 100644 index 000000000000..a65cf19a0205 --- /dev/null +++ b/models/demos/gemma4_31b_qb2/tests/test_history_restore.py @@ -0,0 +1,65 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: 2026 Tenstorrent USA, Inc. +"""Restoring a cached continuation's sliding history from KV pages: same values, and new block ids compile no programs. + +The page id used to be part of a Python slice (cache[p : p + 1]) and so of the program hash: every request with new block +ids compiled new programs, and a prefill that compiles programs retires the decode traces. +""" + +import torch + +import ttnn +from models.demos.gemma4_31b_qb2.tt.decoder import Decoder +from models.demos.gemma4_31b_qb2.tt.generator import build_generator + + +def _reference(generator, state, tables, slot, start): + """The previous implementation: Python slices by page id.""" + histories = [] + for i, layer in enumerate(generator.model.layers): + if not layer.sliding: + histories.append(None) + continue + first = max(0, start - layer.window) // Decoder.PAGE_SIZE + pages = tables[i][slot, first : start // Decoder.PAGE_SIZE].tolist() + pair = [] + for cache in state.kv[i]: + parts = [cache[p : p + 1, :, :, :] for p in pages] + tail = parts[0] if len(parts) == 1 else ttnn.concat(parts, dim=2) + pair.append(ttnn.typecast(tail, ttnn.bfloat16)) + histories.append(tuple(pair)) + return histories + + +def _host(tensor, mesh): + return ttnn.to_torch(tensor, mesh_composer=ttnn.ConcatMeshToTensor(mesh, dim=0)).float() + + +def test_restore_reads_the_same_pages_and_new_block_ids_add_no_programs(qb2_mesh): + generator = build_generator(qb2_mesh, layer_indices=[0], max_seq_len=4096) + state = generator.allocate_cache(batch=1, capacity=4096) + # A toy pool with room for the block ids below (the serving pool is much larger than the default test cache). + state.kv[0] = generator.model.layers[0].allocate_cache(physical_pages=40) + rng = torch.Generator().manual_seed(5) + pages_total = state.kv[0][0].shape[0] + for tensor in state.kv[0]: + host = torch.randn(tuple(tensor.shape), generator=rng).bfloat16() + ttnn.copy_host_to_device_tensor( + ttnn.from_torch( + host, dtype=tensor.dtype, layout=ttnn.TILE_LAYOUT, mesh_mapper=ttnn.ReplicateTensorToMesh(qb2_mesh) + ), + tensor, + ) + start = 1152 + counts = [] + for ids in ([1, 2, 3, 4, 5, 6, 7, 8, 9], [10, 12, 14, 16, 18, 20, 22, 24, 26], [3, 5, 7, 9, 11, 13, 15, 17, 19]): + assert max(ids) < pages_total + tables = {0: torch.tensor([ids + [0] * 4], dtype=torch.int32)} + before = qb2_mesh.num_program_cache_entries() + restored = generator._restore_prefill_history(state, tables, 0, start) + counts.append(qb2_mesh.num_program_cache_entries() - before) + expected = _reference(generator, state, tables, 0, start) + for got, want in zip(restored[0], expected[0]): + assert torch.equal(_host(got, qb2_mesh), _host(want, qb2_mesh)) + # The first restore compiles the slice (and concat/typecast) once; later restores with other block ids add nothing. + assert counts[1] == 0 and counts[2] == 0, counts diff --git a/models/demos/gemma4_31b_qb2/tests/test_kv_packing.py b/models/demos/gemma4_31b_qb2/tests/test_kv_packing.py new file mode 100644 index 000000000000..46dd31a40218 --- /dev/null +++ b/models/demos/gemma4_31b_qb2/tests/test_kv_packing.py @@ -0,0 +1,171 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: 2026 Tenstorrent USA, Inc. +"""KV packing: full-attention blocks of 256 tokens packed into the shared wide rows. + +A pool position holds one full-attention layer and five sliding layers that share one backing of wide rows (sliding +geometry [P, 4, 128, 256]); the scheduler gives every block ID to one group at a time. A sliding block (128 tokens) is one +wide row, a full-attention block (256 tokens) is the same row read as two native 128-token half-pages 2b and 2b + 1. +""" + +from types import SimpleNamespace + +import torch + +import ttnn +from models.demos.gemma4_31b_qb2.tt.generator_vllm import Gemma4ForCausalLM + +LAYERS = 12 # two pool positions of five sliding layers and one full-attention layer +FULL = frozenset(range(5, LAYERS, 6)) + + +def hf_config(): + return SimpleNamespace( + layer_types=["full_attention" if i in FULL else "sliding_attention" for i in range(LAYERS)], + num_key_value_heads=16, + head_dim=256, + num_global_key_value_heads=4, + global_head_dim=512, + sliding_window=1024, + ) + + +def vllm_config(block_size=128): + return SimpleNamespace( + max_in_flight_tokens=8192, + model_config=SimpleNamespace(hf_text_config=hf_config()), + cache_config=SimpleNamespace(block_size=block_size), + ) + + +def test_full_attention_blocks_are_256_tokens_with_the_page_size_of_a_sliding_block(): + specs = Gemma4ForCausalLM.get_kv_cache_spec(vllm_config()) + sizes = {i: specs[f"model.layers.{i}.self_attn"].block_size for i in range(LAYERS)} + assert {i: s for i, s in sizes.items() if i in FULL} == {i: 256 for i in FULL} + assert {s for i, s in sizes.items() if i not in FULL} == {128} + # One page size across the hybrid groups: one global ID space, one wide row per ID. + assert {spec.page_size_bytes for spec in specs.values()} == {2 * 128 * 16 * 256 * 2} + + +def test_the_block_size_stays_128_tokens(expect_error): + with expect_error(ValueError, "128-token cache pages"): + Gemma4ForCausalLM.get_kv_cache_spec(vllm_config(block_size=256)) + + +def test_half_pages_of_full_attention_blocks_and_the_null_block(): + blocks = torch.tensor([[3, 5, 0, 0], [1, 0, 0, 0]]) + pages = Gemma4ForCausalLM._half_pages(blocks, 8) + assert pages.tolist() == [[6, 7, 10, 11, 0, 0, 0, 0], [2, 3, 0, 0, 0, 0, 0, 0]] + + +def test_half_pages_of_a_narrow_table_and_a_table_padded_to_the_page_width(expect_error): + assert Gemma4ForCausalLM._half_pages(torch.tensor([[4]]), 8).tolist() == [[8, 9]] + padded = torch.tensor([[2, 0, 0, 0, 0, 0, 0, 0]]) # as wide as for 128-token blocks, live entries in the first half + assert Gemma4ForCausalLM._half_pages(padded, 8).tolist() == [[4, 5, 0, 0, 0, 0, 0, 0]] + with expect_error(ValueError, "past its 256-token capacity"): + Gemma4ForCausalLM._half_pages(torch.tensor([[1, 0, 0, 0, 0, 0, 0, 2]]), 8) + + +def table_adapter(columns=1, batch=4, width=8): + model = object.__new__(Gemma4ForCausalLM) + model.sub_caches, model.verify_caches, model.full_layers = {}, {}, frozenset({1}) + model.cache = SimpleNamespace( + batch=batch, + kv=[None, None], + page_tables={i: torch.zeros(batch, width, dtype=torch.int32) for i in range(2)}, + columns=columns, + ) + return model + + +def test_tables_expand_only_the_full_attention_layers(): + model = table_adapter() + sliding_source, full_source = torch.tensor([[7, 9], [3, 0]]), torch.tensor([[7, 9], [3, 0]]) + tables = model._tables(model.cache, [sliding_source, full_source], slots=[2, 0]) + assert tables[0][2, :2].tolist() == [7, 9] and tables[0][0, :2].tolist() == [3, 0] + assert tables[1][2, :4].tolist() == [14, 15, 18, 19] and tables[1][0, :4].tolist() == [6, 7, 0, 0] + assert tables[1][1].count_nonzero() == 0 and tables[1][3].count_nonzero() == 0 + + +def test_tables_translate_padding_rows_and_replicate_draft_columns(): + model = table_adapter(columns=2, batch=4) + source = torch.tensor([[11], [-1]]) + tables = model._tables(model.cache, [source, source]) + assert tables[1][0, :2].tolist() == [22, 23] and tables[1][1].count_nonzero() == 0 + assert tables[1][2:].equal(tables[1][:2]), "draft columns repeat the live rows" + + +def fake_tensor(shape, buffer_id): + return SimpleNamespace( + shape=shape, + dtype=ttnn.bfloat8_b, + layout=ttnn.TILE_LAYOUT, + buffer_unique_id=lambda: buffer_id, + buffer_address=lambda: 4096 * buffer_id, + buffer_aligned_page_size=lambda: 1088, + ) + + +def allocation_adapter(monkeypatch): + allocated = [] + + def view(root, shape): + assert tuple(root.shape)[1:] == (4, 128, 256) and shape == [2 * root.shape[0], 1, 128, 512] + return fake_tensor(tuple(shape), root.buffer_unique_id()) + + monkeypatch.setattr(ttnn.experimental, "view", view) + + def layer(i): + sliding = i not in FULL + + def allocate_cache(*, physical_pages): + allocated.append(physical_pages) + return tuple(fake_tensor((physical_pages, 4, 128, 256), 10 * len(allocated) + k) for k in range(2)) + + return SimpleNamespace( + sliding=sliding, + kv_heads=4 if sliding else 1, + head_dim=256 if sliding else 512, + allocate_cache=allocate_cache, + ) + + model = object.__new__(Gemma4ForCausalLM) + model.generator = SimpleNamespace(model=SimpleNamespace(layers=[layer(i) for i in range(LAYERS)])) + model.full_layers, model.rows, model.max_seq_len = FULL, 4, 1024 + model.verify_shapes, model.row_sizes = (), () + return model, allocated + + +def specs(blocks, bad_layer=None): + out = [] + for i in range(LAYERS): + shape = (blocks, 1, 256, 512) if i in FULL else (blocks, 4, 128, 256) + if i == bad_layer: + shape = (blocks, 4, 128, 512) + out.append((shape, torch.bfloat16, i // 6)) + return out + + +def test_each_pool_position_allocates_one_wide_backing_and_views_it_for_its_full_layer(monkeypatch): + model, allocated = allocation_adapter(monkeypatch) + cache = model.allocate_kv_cache_per_layer(specs(40)) + assert allocated == [44, 44], "one backing per pool position, n blocks plus the scratch rows" + for pool in (0, 1): + sliding = [cache.kv[i] for i in range(6 * pool, 6 * pool + 5)] + full = cache.kv[6 * pool + 5] + assert all(s is sliding[0] for s in sliding) + assert full[0].shape == (88, 1, 128, 512) and full[0].buffer_unique_id() == sliding[0][0].buffer_unique_id() + assert full[1].buffer_unique_id() == sliding[0][1].buffer_unique_id() + assert cache.scratch_pages[0] == [40, 41, 42, 43] and cache.scratch_pages[5] == [80, 82, 84, 86] + assert cache.page_tables[5].shape == (4, 8) + + +def test_the_allocator_checks_the_geometry_and_the_pool_composition(monkeypatch, expect_error): + model, _ = allocation_adapter(monkeypatch) + with expect_error(ValueError, "Unexpected allocator geometry"): + model.allocate_kv_cache_per_layer(specs(40, bad_layer=2)) + model, _ = allocation_adapter(monkeypatch) + bad = specs(40) + bad[5] = (bad[5][0], bad[5][1], 0) # a second full-attention layer in pool 0 and none in pool 1 + bad[11] = (bad[11][0], bad[11][1], 0) + with expect_error(ValueError, "one full-attention layer"): + model.allocate_kv_cache_per_layer(bad) diff --git a/models/demos/gemma4_31b_qb2/tests/test_kv_packing_simulation.py b/models/demos/gemma4_31b_qb2/tests/test_kv_packing_simulation.py new file mode 100644 index 000000000000..968c2a6eb59b --- /dev/null +++ b/models/demos/gemma4_31b_qb2/tests/test_kv_packing_simulation.py @@ -0,0 +1,276 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: 2026 Tenstorrent USA, Inc. +"""CPU simulation of KV packing with the real vLLM KV cache manager and the real table translation. + +Gemma4 groups (five sliding groups of 128-token blocks, one full-attention group of 256-token blocks) share one global +block ID space; every ID is one wide backing row. The page tables the manager produces go through the model's own +`_tables`, and a shadow model of the wide rows checks that every attention read (the whole cached or computed prefix for full +attention, the window for sliding attention) returns exactly the content written for that position, under pool pressure, +preemption, prefix hits, eviction and reuse. A full block owns both half-pages of its row, a sliding page covers both halves. +""" + +import math +import random +from types import SimpleNamespace + +import numpy as np +import torch + +from models.demos.gemma4_31b_qb2.tt.generator_vllm import Gemma4ForCausalLM + +PAGE, WINDOW, LAYERS, MAX_SEQ = 128, 1024, 60, 262144 +PAGE_BYTES = 2 * PAGE * 16 * 256 * 2 + + +class Memory: + """Shadow content of the wide rows: [row, half, offset] -> (writer group, prefix fingerprint).""" + + def __init__(self, rows): + self.tag = np.zeros((rows, 2, PAGE), dtype=np.int64) + self.group = np.full((rows, 2, PAGE), -1, dtype=np.int8) + self.reads = 0 + + @staticmethod + def locate(table_row, positions, full): + pages, offsets = table_row[positions // PAGE], positions % PAGE + return (pages // 2, pages % 2, offsets) if full else (pages, None, offsets) + + def write(self, table_row, positions, values, group, full): + rows, halves, offsets = self.locate(table_row, positions, full) + assert (rows > 0).all(), "write into the null block" + for half in [None] if full else [0, 1]: + hh = halves if full else np.full_like(rows, half) + self.tag[rows, hh, offsets] = values + self.group[rows, hh, offsets] = group + + def check(self, table_row, positions, values, group, full, label): + if len(positions) == 0: + return + rows, halves, offsets = self.locate(table_row, positions, full) + assert (rows > 0).all(), f"{label}: read of the null block" + for half in [None] if full else [0, 1]: + hh = halves if full else np.full_like(rows, half) + ok = (self.tag[rows, hh, offsets] == values) & (self.group[rows, hh, offsets] == group) + assert ok.all(), f"{label}: corrupt KV at position {int(positions[np.flatnonzero(~ok)[0]])}, group {group}" + self.reads += len(positions) + + +def fingerprints(tokens): + out, h = np.empty(len(tokens), dtype=np.int64), 1469598103934665603 + for i, t in enumerate(tokens): + h = (h * 1099511628211 + t + 1) & 0x7FFFFFFFFFFFFFFF + out[i] = h + return out + + +def build(num_blocks, batch): + from vllm.utils.hashing import sha256 + from vllm.v1.core import kv_cache_utils as U + from vllm.v1.core.kv_cache_manager import KVCacheManager + from vllm.v1.core.single_type_kv_cache_manager import register_all_kvcache_specs + from vllm.v1.kv_cache_interface import FullAttentionSpec, KVCacheConfig + from vllm_tt_plugin.kv_cache_anchor import AnchoredSlidingWindowSpec + + # Importing the plugin registers its spec first, which makes vLLM skip its lazy registration of the built-in ones. + register_all_kvcache_specs(None) + specs = {} + for i in range(LAYERS): + if i % 6 == 5: + spec = FullAttentionSpec( + block_size=2 * PAGE, num_kv_heads=4, head_size=512, dtype=torch.bfloat16, page_size_padded=PAGE_BYTES + ) + else: + spec = AnchoredSlidingWindowSpec( + block_size=PAGE, + num_kv_heads=16, + head_size=256, + dtype=torch.bfloat16, + page_size_padded=PAGE_BYTES, + sliding_window=WINDOW, + ) + specs[f"model.layers.{i}.self_attn"] = spec + assert len({s.page_size_bytes for s in specs.values()}) == 1 + groups = U._get_kv_cache_groups_uniform_page_size(specs) + sizes = [g.kv_cache_spec.block_size for g in groups] + assert sorted(sizes) == [PAGE] * 5 + [2 * PAGE] + manager = KVCacheManager( + KVCacheConfig(num_blocks=num_blocks, kv_cache_tensors=[], kv_cache_groups=groups), + max_model_len=MAX_SEQ, + scheduler_block_size=math.lcm(*sizes), + hash_block_size=math.gcd(*sizes), + max_in_flight_tokens=16384, + enable_caching=True, + ) + layer_group = { + int(n.split(".")[2]): g for g, group in enumerate(groups) for name in group.layer_names for n in [name] + } + U.init_none_hash(sha256) + hasher = U.get_request_block_hasher(math.gcd(*sizes), sha256) + model = object.__new__(Gemma4ForCausalLM) + model.sub_caches, model.verify_caches = {}, {} + model.full_layers = frozenset(i for i in range(LAYERS) if i % 6 == 5) + model.cache = SimpleNamespace( + batch=batch, + kv=[None] * LAYERS, + page_tables={i: torch.zeros(batch, MAX_SEQ // PAGE, dtype=torch.int32) for i in range(LAYERS)}, + columns=1, + ) + return manager, groups, layer_group, hasher, model + + +def simulate(seed, blocks=500, conversations=10, admitted=8, steps=20000, max_context=14000): + from vllm.sampling_params import SamplingParams + from vllm.v1.request import Request, RequestStatus + + rng, vocab = random.Random(seed), 50000 + manager, groups, layer_group, hasher, model = build(blocks, admitted) + memory, sp = Memory(blocks), SamplingParams(max_tokens=1 << 20) + convs = [ + dict( + id=c, + tokens=[rng.randrange(vocab) for _ in range(rng.randint(1500, 6000))], + turn=0, + turns=rng.randint(3, 8), + ready_at=rng.randint(0, 200), + done=False, + active=None, + ) + for c in range(conversations) + ] + stats = dict(steps=0, finished_turns=0, hit_tokens=0, preemptions=0, admission_waits=0, tables=0) + waiting, running, meta = [], [], {} + + def new_request(conv): + rid = f"c{conv['id']}-t{conv['turn']}" + meta[rid] = dict(conv=conv, fp=fingerprints(conv["tokens"]), out=rng.randint(20, 300)) + return Request(rid, list(conv["tokens"]), sp, None, block_hasher=hasher) + + def tables_of(r): + ids, per_group = manager.get_block_ids(r.request_id), [] + for g in range(len(groups)): + t = torch.zeros(1, MAX_SEQ // PAGE, dtype=torch.int32) + t[0, : len(ids[g])] = torch.tensor(ids[g], dtype=torch.int32) if ids[g] else t[0, :0] + per_group.append(t) + stats["tables"] += 1 + return model._tables(model.cache, [per_group[layer_group[i]] for i in range(LAYERS)], slots=[0]) + + def execute(r, start, end): + m = meta[r.request_id] + if len(m["fp"]) < end: + m["fp"] = fingerprints(r.all_token_ids) + tables = tables_of(r) + for layer in range(6): # one pool position: layer 5 is its full-attention layer + g, full = layer_group[layer], layer % 6 == 5 + row = tables[layer][0].numpy().astype(np.int64) + first = 0 if full else max(0, start - (WINDOW - 1)) + memory.check(row, np.arange(first, start, dtype=np.int64), m["fp"][first:start], g, full, r.request_id) + memory.write(row, np.arange(start, end, dtype=np.int64), m["fp"][start:end], g, full) + pad_end = min(-(-end // 32) * 32, len(row) * PAGE) # tile padding past the chunk, inside its page + if pad_end > end and row[end // PAGE] > 0: + memory.write( + row, np.arange(end, pad_end, dtype=np.int64), np.full(pad_end - end, -7, dtype=np.int64), g, full + ) + + for step in range(steps): + stats["steps"] = step + for conv in convs: + if ( + not conv["done"] + and conv["active"] is None + and conv["ready_at"] <= step + and not any(meta[w.request_id]["conv"] is conv for w in waiting) + ): + waiting.append(new_request(conv)) + conv["active"] = True + budget = 8192 + if waiting and len(running) < admitted: + r = waiting[0] + blocks_hit, hits, _ = manager.get_computed_blocks(r) if r.num_computed_tokens == 0 else (None, 0, 0) + chunk = min(budget, r.num_tokens - r.num_computed_tokens - hits) + if ( + manager.allocate_slots( + r, + chunk, + num_new_computed_tokens=hits, + new_computed_blocks=blocks_hit, + has_scheduled_reqs=bool(running), + ) + is not None + ): + waiting.pop(0) + r.status = RequestStatus.RUNNING + stats["hit_tokens"] += hits + start = r.num_computed_tokens + hits + execute(r, start, start + chunk) + r.num_computed_tokens = start + chunk + running.append(r) + budget -= chunk + if r.num_computed_tokens == r.num_tokens: + r.append_output_token_ids(rng.randrange(vocab)) + else: + stats["admission_waits"] += 1 + for r in list(running): + if r not in running or r.num_tokens - r.num_computed_tokens <= 0: + continue + remaining = r.num_tokens - r.num_computed_tokens + n = min(remaining, budget) if remaining > 1 else 1 + if n <= 0: + continue + while manager.allocate_slots(r, n) is None: + victim = running.pop() + manager.free(victim) + victim.status, victim.num_computed_tokens = RequestStatus.PREEMPTED, 0 + waiting.insert(0, victim) + stats["preemptions"] += 1 + if victim is r: + break + if r not in running: + continue + start = r.num_computed_tokens + execute(r, start, start + n) + r.num_computed_tokens = start + n + budget -= n if remaining > 1 else 0 + if r.num_computed_tokens == r.num_tokens: + m = meta[r.request_id] + if r.num_output_tokens >= m["out"] or r.num_tokens >= max_context: + conv = m["conv"] + running.remove(r) + manager.free(r) + conv["tokens"] = list(r.all_token_ids) + [ + rng.randrange(vocab) for _ in range(rng.randint(50, 1500)) + ] + conv["turn"] += 1 + conv["active"] = None + conv["ready_at"] = step + rng.randint(1, 200) + conv["done"] = conv["turn"] >= conv["turns"] or len(conv["tokens"]) >= max_context + stats["finished_turns"] += 1 + else: + r.append_output_token_ids(rng.randrange(vocab)) + if all(c["done"] for c in convs) and not running and not waiting: + break + stats.update(reads_checked=memory.reads, all_done=all(c["done"] for c in convs)) + return stats + + +def test_packed_blocks_never_corrupt_a_read_under_pressure_preemption_and_reuse(): + for seed in (3, 11): + stats = simulate(seed) + assert stats["all_done"], stats + assert stats["reads_checked"] > 10_000_000, stats + assert stats["hit_tokens"] > 0, "prefix hits must be exercised" + assert stats["preemptions"] + stats["admission_waits"] > 0, "the pool must be under pressure" + + +def test_the_simulation_detects_a_wrong_translation(monkeypatch, expect_error): + """Control: a full block mapped to one native page instead of its two half-pages makes rows shared between blocks.""" + + def wrong(blocks, width): + blocks = blocks[:, : width // 2] + out = torch.zeros(blocks.shape[0], 2 * blocks.shape[1], dtype=blocks.dtype) + out[:, ::2] = blocks + out[:, 1::2] = blocks + return out + + monkeypatch.setattr(Gemma4ForCausalLM, "_half_pages", staticmethod(wrong)) + with expect_error(AssertionError, "corrupt KV|null block"): + simulate(3) diff --git a/models/demos/gemma4_31b_qb2/tests/test_prefill_sdpa_chunks.py b/models/demos/gemma4_31b_qb2/tests/test_prefill_sdpa_chunks.py new file mode 100644 index 000000000000..fab7223183a6 --- /dev/null +++ b/models/demos/gemma4_31b_qb2/tests/test_prefill_sdpa_chunks.py @@ -0,0 +1,87 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: 2026 Tenstorrent USA, Inc. +"""The Q chunk of the chunked full-attention prefill SDPA depends on the number of query rows, and never changes a result.""" + +from types import SimpleNamespace + +import pytest +import torch + +import ttnn +from models.demos.gemma4_31b_qb2.tt.decoder import Decoder +from models.demos.gemma4_31b_qb2.tt.generator import build_generator + + +def full_layer(): + layer = object.__new__(Decoder) + layer.sliding = False + layer.pref_compute = SimpleNamespace(fp32_dest_acc_en=True) + layer.pref_sdpa_by_q = {q: SimpleNamespace(q_chunk_size=q) for q in (32, 64, 128, 256)} + layer.pref_sdpa_selections = {} + return layer + + +def chosen(layer, rows, start=8192): + q = SimpleNamespace(shape=(1, 8, rows, 512), dtype=ttnn.bfloat8_b) + cache = SimpleNamespace(dtype=ttnn.bfloat8_b) + page_table = SimpleNamespace(buffer_aligned_page_size=lambda: 160) + return layer._prefill_sdpa_config(q, cache, page_table, start=start).q_chunk_size + + +@pytest.mark.parametrize("rows", [256, 512, 768, 1024]) +def test_short_suffixes_with_an_even_number_of_128_row_chunks_use_q_chunk_64(rows): + assert Decoder._short_suffix_q_chunk(rows) == 64 + assert chosen(full_layer(), rows) == 64 + + +@pytest.mark.parametrize("rows", [128, 384, 640, 896, 1280, 1536, 2048, 4096, 6656, 8192]) +def test_other_sizes_keep_the_default_choice(rows, monkeypatch): + assert Decoder._short_suffix_q_chunk(rows) is None + layer = full_layer() + assert chosen(layer, rows) == chosen(layer, rows) # deterministic + assert chosen(layer, rows) in (128, 256) + assert chosen(layer, rows) != 64 + + +def test_the_choice_never_depends_on_the_start_and_sliding_layers_are_unchanged(): + layer = full_layer() + assert {chosen(layer, 512, start=s) for s in (128, 8192, 49152)} == {64} + sliding = full_layer() + sliding.sliding = True + sliding.pref_sdpa_by_q = { + q: SimpleNamespace(q_chunk_size=q) for q in (32, 64, 128) + } # sliding layers offer up to 128 + assert chosen(sliding, 512) == 128 + + +@pytest.fixture +def generator(qb2_mesh): + gen = build_generator(qb2_mesh, layer_indices=[0, 5], max_seq_len=16384) + yield gen + gen.close() + + +@pytest.mark.parametrize("rows", [256, 384, 512, 768, 1024, 2048]) +def test_every_q_chunk_choice_gives_bit_identical_outputs(generator, monkeypatch, rows): + """The rule against the default choice (largest chunk that fits L1), same inputs, continuation after 8K cached tokens.""" + gen = generator + state = gen.allocate_cache(batch=1, capacity=16384) + tokens = torch.randint(1000, 50000, (1, 16384), generator=torch.Generator().manual_seed(3)) + key = iter(range(1, 100)) + + def run(): + k = next(key) + gen.prefill_forward(tokens, page_table=state.page_tables, kv_cache=state, prompt_lens=[8192], history_keys=[k]) + return gen.prefill_forward( + tokens, + page_table=state.page_tables, + kv_cache=state, + prompt_lens=[8192 + rows], + start_positions=[8192], + history_keys=[k], + ) + + with_rule = run() + monkeypatch.setattr(Decoder, "_short_suffix_q_chunk", staticmethod(lambda rows: None)) + default = run() + assert torch.equal(with_rule, default) diff --git a/models/demos/gemma4_31b_qb2/tests/test_prefill_warmup.py b/models/demos/gemma4_31b_qb2/tests/test_prefill_warmup.py new file mode 100644 index 000000000000..9a3a34a6c431 --- /dev/null +++ b/models/demos/gemma4_31b_qb2/tests/test_prefill_warmup.py @@ -0,0 +1,173 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: 2026 Tenstorrent USA, Inc. +"""Prefill lengths are compiled up front, and a kept decode trace gives the same tokens as a rebuilt one. + +A prefill of a length class the program cache has never seen compiles new programs (seconds) and retires the decode +traces; after warm_prefill_shapes it does neither, so the trace survives the prefills of an agent loop. +""" + +import time + +import pytest +import torch + +from models.demos.gemma4_31b_qb2.tt.generator import Gemma4Generator, build_generator + +CAPACITY = 2048 +NEW_TOKENS = 24 + + +@pytest.fixture +def generator(qb2_mesh): + gen = build_generator(qb2_mesh, layer_indices=[0, 5], max_seq_len=CAPACITY) + yield gen + gen.close() + + +def _generate(gen, prompt): + return gen.generate(prompt, NEW_TOKENS, temperature=0.0, top_k=1, stop_on_eos=False) + + +def test_warmup_removes_compilation_and_keeps_the_trace(generator, qb2_mesh): + gen = generator + torch.manual_seed(0) + prompt = torch.randint(1000, 50000, (700,)).tolist() + other = torch.randint(1000, 50000, (300,)).tolist() + + # Cold: a new length class compiles programs. + programs = qb2_mesh.num_program_cache_entries() + cold = _generate(gen, prompt) + cold_ttft = gen.last_ttft_s + assert qb2_mesh.num_program_cache_entries() > programs + + # Warm every class up to the capacity, then serve classes that were never run. + start = time.perf_counter() + state = gen._owned(1, CAPACITY) + gen.warm_prefill_shapes(state, state.page_tables, CAPACITY) + warm_seconds = time.perf_counter() - start + programs = qb2_mesh.num_program_cache_entries() + _generate(gen, prompt) # builds the decode trace again after warm-up's retire + entry = next(iter(gen.traces.values())) + warm_ttfts = [] + for length in (320, 1120, 1920): + ids = torch.randint(1000, 50000, (length,)).tolist() + _generate(gen, ids) + warm_ttfts.append(round(gen.last_ttft_s, 3)) + assert gen.traces and next(iter(gen.traces.values())) is entry, f"length {length} retired the trace" + assert qb2_mesh.num_program_cache_entries() == programs, "a warmed class compiled new programs" + # 1000 tokens pad to the warmed 1024: nothing compiles, and its time-to-first-token stays near the warmed ones. + _generate(gen, torch.randint(1000, 50000, (1000,)).tolist()) + odd_ttft = gen.last_ttft_s + assert odd_ttft < 10.0 + _generate(gen, prompt) # rebuild the trace for the identity checks below + print(f"odd-length ttft(1000)={odd_ttft:.2f}s") + print(f"cold ttft(700)={cold_ttft:.2f}s warm-up={warm_seconds:.1f}s warm ttfts(320,1120,1920)={warm_ttfts}") + + # The kept trace decodes exactly what a rebuilt one does. + kept = _generate(gen, prompt) + _generate(gen, other) + kept_after_other = _generate(gen, prompt) + gen._retire_traces() + rebuilt = _generate(gen, prompt) + assert kept == cold == kept_after_other == rebuilt + + +def test_warmup_covers_padded_lengths_last_rows_and_continuations_then_retires_the_traces(): + from types import SimpleNamespace + + generator = object.__new__(Gemma4Generator) + events = [] + generator.prefill_forward = lambda tokens, **kw: events.append( + (int(tokens.shape[1]), kw["prompt_lens"], kw.get("start_positions"), kw.get("history_keys")) + ) + generator._retire_traces = lambda **kwargs: events.append("retire") + generator.model = SimpleNamespace(tensor=lambda *a, **k: object(), config=SimpleNamespace(vocab_size=8)) + generator._merge_prefill_rows = lambda rows: events.append(("merge", len(rows))) + state = SimpleNamespace(prefill_history={-1: object()}) + generator.warm_prefill_shapes(state, None, 1408) + from_zero = [(n, [n], None, None) for n in range(128, 1409, 128)] + last_rows = [(n, [n], None, None) for n in (*range(1, 33), *range(129, 161))] + continuations = [] + for chunk in (128, 256): + for previous_end in (1152, 1153): + continuations.append((1152 + chunk, [previous_end], None, [-1])) + continuations.append((1152 + chunk, [1152 + chunk], [previous_end], [-1])) + merges = [("merge", b) for b in range(2, 33)] + assert events == from_zero + last_rows + continuations + merges + ["retire"] + assert state.prefill_history == {} + + +def _serving_adapter(columns): + from types import SimpleNamespace + + from models.demos.gemma4_31b_qb2.tt.generator_vllm import Gemma4ForCausalLM + + model = object.__new__(Gemma4ForCausalLM) + calls = [] + model.generator = SimpleNamespace( + warm_prefill_shapes=lambda state, tables, max_tokens: calls.append((state, max_tokens, tables)) + ) + state = SimpleNamespace(page_tables={0: torch.zeros(1, columns, dtype=torch.int32)}, capacity=columns * 128) + return model, state, calls + + +def test_serving_warmup_stays_within_the_cache_capacity(): + model, state, calls = _serving_adapter(columns=16) # --max-model-len 2048 + model.warmup_model_prefill(state, True) + ((_, max_tokens, tables),) = calls + assert max_tokens == 2048 + assert tables[0][0].tolist() == list(range(1, 17)) + model, state, calls = _serving_adapter(columns=2048) # 262144: the prefill limit applies + model.warmup_model_prefill(state, True) + assert calls[0][1] == 8192 and calls[0][2][0][0, :64].tolist() == list(range(1, 65)) + + +def test_second_warm_up_of_the_same_cache_is_skipped(): + model, state, calls = _serving_adapter(columns=64) + model.warmup_model_prefill(state, True) + model.warmup_model_prefill(state, True) # the plugin's second phase + assert len(calls) == 1 + other = _serving_adapter(columns=64)[1] + model.warmup_model_prefill(other, True) + assert len(calls) == 2 + + +def _sampling_generator(monkeypatch, grows): + import collections + from types import SimpleNamespace + + import models.demos.gemma4_31b_qb2.tt.generator as module + + generator = object.__new__(Gemma4Generator) + events, programs = [], [0] + generator.mesh = SimpleNamespace(num_program_cache_entries=lambda: programs[0]) + generator.model = SimpleNamespace(config=SimpleNamespace(vocab_size=8)) + generator.counters = collections.defaultdict(int) + generator.configure_sampling = lambda **kwargs: events.append("configure") + + def merge(outputs): + programs[0] += grows + events.append("merge") + return "merged" + + generator._merge_prefill_rows = merge + generator._entry = lambda state: events.append("entry") or {"logits": "l", "sampling_trace": "t", "tokens": "k"} + generator._retire_traces = lambda **kwargs: events.append("retire") + generator._read_tokens = lambda tokens, batch: torch.zeros(batch, dtype=torch.int32) + monkeypatch.setattr(module.ttnn, "Shape", lambda dims: dims) + monkeypatch.setattr(module.ttnn, "reshape", lambda tensor, *args: tensor) + monkeypatch.setattr(module.ttnn, "copy", lambda source, destination: events.append("copy")) + monkeypatch.setattr(module.ttnn, "execute_trace", lambda *args, **kwargs: events.append("execute")) + return generator, events + + +def test_sampling_a_known_batch_size_compiles_nothing_and_keeps_the_traces(monkeypatch): + generator, events = _sampling_generator(monkeypatch, grows=0) + generator.sample_prefill_outputs([object(), object()], object(), {}) + assert events == ["configure", "merge", "entry", "copy", "execute"] + + +def test_a_program_compiled_while_sampling_retires_the_traces_before_the_sampling_trace_runs(monkeypatch): + generator, events = _sampling_generator(monkeypatch, grows=2) + generator.sample_prefill_outputs([object(), object()], object(), {}) + assert events == ["configure", "merge", "entry", "copy", "retire", "entry", "copy", "execute"] diff --git a/models/demos/gemma4_31b_qb2/tests/test_prefill_weight_slice.py b/models/demos/gemma4_31b_qb2/tests/test_prefill_weight_slice.py new file mode 100644 index 000000000000..02e664591a48 --- /dev/null +++ b/models/demos/gemma4_31b_qb2/tests/test_prefill_weight_slice.py @@ -0,0 +1,46 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: 2026 Tenstorrent USA, Inc. +"""The interleaved prefill weights sliced straight from the sharded decode weight equal the copy-then-slice rebuild bit for bit.""" + +import pytest +import torch + +import ttnn +from models.demos.gemma4_31b_qb2.tt.generator import build_generator + + +def host(mesh, tensor): + return ttnn.to_torch(tensor, mesh_composer=ttnn.ConcatMeshToTensor(mesh, dim=0)) + + +def copy_then_slice(layer, name): + """The earlier rebuild: copy the whole fused decode weight to interleaved DRAM, then slice the copy.""" + full = ttnn.to_memory_config(layer.decode_weights["qkv" if name == "qk" else "gate_up"], ttnn.DRAM_MEMORY_CONFIG) + shape = list(full.shape) + zeros = [0] * len(shape) + n = shape[-1] + if name == "qk": + width = n - layer.kv_heads * layer.head_dim + return (ttnn.slice(full, zeros, shape[:-1] + [width]),) + half = n // 2 + return ttnn.slice(full, zeros, shape[:-1] + [half]), ttnn.slice(full, zeros[:-1] + [half], shape[:-1] + [n]) + + +@pytest.fixture +def generator(qb2_mesh): + gen = build_generator(qb2_mesh, layer_indices=[0, 5], max_seq_len=8192) + yield gen + + +@pytest.mark.parametrize("layer_index, name", [(0, "gate_up"), (1, "gate_up"), (1, "qk")]) +def test_sliced_prefill_weights_equal_the_copied_ones(generator, qb2_mesh, layer_index, name): + layer = generator.model.layers[layer_index] + assert layer.sliding == (layer_index == 0) + new = layer._prefill_weight(name) + new = new if isinstance(new, tuple) else (new,) + old = copy_then_slice(layer, name) + assert len(new) == len(old) + for a, b in zip(new, old): + assert a.memory_config() == ttnn.DRAM_MEMORY_CONFIG + assert tuple(a.shape) == tuple(b.shape) + assert torch.equal(host(qb2_mesh, a), host(qb2_mesh, b)) diff --git a/models/demos/gemma4_31b_qb2/tests/test_prefill_weights_once.py b/models/demos/gemma4_31b_qb2/tests/test_prefill_weights_once.py new file mode 100644 index 000000000000..beced297d35c --- /dev/null +++ b/models/demos/gemma4_31b_qb2/tests/test_prefill_weights_once.py @@ -0,0 +1,69 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: 2026 Tenstorrent USA, Inc. +"""A prefill of several chunks rebuilds the layer's interleaved weights once; the outputs equal the per-chunk rebuild bit for bit.""" + +import pytest +import torch + +import ttnn +from models.demos.gemma4_31b_qb2.tt.generator import build_generator + + +@pytest.fixture +def generator(qb2_mesh): + gen = build_generator(qb2_mesh, layer_indices=[0, 5], max_seq_len=16384) + yield gen + gen.close() + + +def run(layer, qb2_mesh, rows, once): + """The layer over `rows` fixed random rows on its own cache pages: (output, names of the weight rebuilds).""" + layer._prepare_prefill_norm_stats() + pages = (rows // 128 + 2 + 7) // 8 * 8 + table = (torch.randperm(pages, generator=torch.Generator().manual_seed(3)) + 1).reshape(1, pages).int() + cache = layer.allocate_cache(physical_pages=pages + 2) + replicate = ttnn.ReplicateTensorToMesh(qb2_mesh) + pt = ttnn.from_torch(table, device=qb2_mesh, dtype=ttnn.int32, layout=ttnn.ROW_MAJOR_LAYOUT, mesh_mapper=replicate) + sharded = layer.prefill_input_width(rows) != layer.hidden_size + x = torch.randn(1, 1, rows, layer.hidden_size, generator=torch.Generator().manual_seed(5)).to(torch.bfloat16) + tx = ttnn.from_torch( + x, + device=qb2_mesh, + dtype=ttnn.bfloat16, + layout=ttnn.TILE_LAYOUT, + mesh_mapper=ttnn.ShardTensorToMesh(qb2_mesh, dim=3) if sharded else replicate, + ) + rebuilds = [] + original = layer._prefill_weight + layer._prefill_weight = lambda name: rebuilds.append(name) or original(name) + if not once: + layer._rebuild_weights_once = lambda chunks: {} + try: + out = layer.prefill_forward(tx, page_table=pt, kv_cache=cache, start_pos=0) + parts = [ttnn.to_torch(t) for t in ttnn.get_device_tensors(out)] + finally: + del layer._prefill_weight + if not once: + del layer._rebuild_weights_once + assert layer.pref_weights == {} + return (torch.cat(parts, dim=-1) if sharded else parts[0]), rebuilds + + +@pytest.mark.parametrize( + "layer_index, rows", [(0, 3 * 1024), (0, 7168), (1, 7168)], ids=["sliding-3", "sliding-7", "full-2"] +) # the loaded layers are checkpoint layers 0 (sliding, 1024-row chunks) and 5 (full, 6656-row chunks) +def test_chunks_share_one_weight_rebuild_with_identical_output(generator, qb2_mesh, layer_index, rows): + layer = generator.model.layers[layer_index] + chunks = -(-rows // layer.PREFILL_CHUNK) + shared, rebuilt_once = run(layer, qb2_mesh, rows, once=True) + separate, rebuilt_per_chunk = run(layer, qb2_mesh, rows, once=False) + assert torch.equal(shared, separate) + names = {"qkv" if layer.sliding else "qk", "o", "gate_up", "down"} + assert sorted(rebuilt_once) == sorted(names) # one rebuild of each weight for the whole span + assert len(rebuilt_per_chunk) == len(names) * chunks and chunks > 1 + + +def test_a_single_chunk_rebuilds_each_weight_once_as_before(generator, qb2_mesh): + layer = generator.model.layers[0] + _, rebuilds = run(layer, qb2_mesh, 1024, once=True) + assert len(rebuilds) == 4 diff --git a/models/demos/gemma4_31b_qb2/tests/test_serving_inputs.py b/models/demos/gemma4_31b_qb2/tests/test_serving_inputs.py index a5fccc768f29..df3c5367cea9 100644 --- a/models/demos/gemma4_31b_qb2/tests/test_serving_inputs.py +++ b/models/demos/gemma4_31b_qb2/tests/test_serving_inputs.py @@ -8,6 +8,7 @@ import torch import ttnn +from models.demos.gemma4_31b_qb2.tt.decode_rows import verify_shapes from models.demos.gemma4_31b_qb2.tt.decoder import Decoder from models.demos.gemma4_31b_qb2.tt.generator import CacheState, Gemma4Generator from models.demos.gemma4_31b_qb2.tt.generator_vllm import Gemma4ForCausalLM @@ -15,8 +16,10 @@ def adapter(): model = object.__new__(Gemma4ForCausalLM) + model.sub_caches, model.verify_caches, model.row_sizes, model.verify_shapes, model.cur_state = {}, {}, (), (), None + model.full_layers = frozenset() model.cache = SimpleNamespace( - batch=4, kv=[None, None], page_tables={i: torch.zeros(4, 8, dtype=torch.int32) for i in range(2)} + batch=4, kv=[None, None], page_tables={i: torch.zeros(4, 8, dtype=torch.int32) for i in range(2)}, columns=1 ) return model @@ -127,3 +130,442 @@ def test_persistent_capture_release_discards_all_prefill_history(): assert model.cache.prefill_history == {} assert model.entry is None assert closed == [True] + + +@pytest.mark.parametrize("start", [128, 1024, 1152, 8192]) +def test_prefix_history_gathers_retained_pages_in_logical_order(monkeypatch, expect_error, start): + generator = object.__new__(Gemma4Generator) + generator.model = SimpleNamespace( + layers=[SimpleNamespace(sliding=True, window=1024), SimpleNamespace(sliding=False)] + ) + table = torch.arange(start // 128, 0, -1).reshape(1, -1) + cache = torch.arange((start // 128 + 1) * 128).reshape(-1, 1, 128, 1).float() + state = SimpleNamespace(kv=[(cache, cache + 1), None]) + monkeypatch.setattr(ttnn, "concat", torch.cat) + monkeypatch.setattr(ttnn, "typecast", lambda tensor, dtype: tensor) + # The page id travels as device tensors (host tensors here); the slice takes the page [start, end) along dim 0. + generator._upload = lambda values, dtype: values + monkeypatch.setattr( + ttnn, "slice", lambda cache, begin, end, slice_dim, num_devices: cache[int(begin[0]) : int(end[0])] + ) + histories = generator._restore_prefill_history(state, {0: table}, 0, start) + expected = torch.cat([cache[p : p + 1] for p in table[0, -8:]], dim=2) + assert torch.equal(histories[0][0], expected) + assert torch.equal(histories[0][1], expected + 1) + assert histories[1] is None + table[0, -1] = 0 + with expect_error(ValueError, "Missing sliding prefix pages"): + generator._restore_prefill_history(state, {0: table}, 0, start) + + +def test_concurrent_shared_prefixes_keep_private_history_and_follow_moves(): + model = adapter() + model.prefill_slot_keys = {} + model.next_prefill_key = 0 + model.cache.prefill_history = {} + submitted = [] + model.generator = SimpleNamespace(prefill_forward=lambda *args, **kwargs: submitted.append(kwargs["history_keys"])) + shared = torch.tensor([[11, 12], [11, 13]]) + tokens = torch.zeros(2, 256, dtype=torch.int32) + model.prefill_forward( + tokens, + None, + model.cache, + [256, 256], + page_tables_per_layer=[shared, shared], + start_pos=[128, 128], + empty_slots=[0, 1], + ) + first, second = submitted[-1] + assert first != second + model.cache.prefill_history.update({first: object(), second: object()}) + model.note_state_slots_moved({0: 2, 1: 0}) + model.release_request(0) + assert set(model.cache.prefill_history) == {first} + model.prefill_forward( + tokens[:1], None, model.cache, [256], page_tables_per_layer=[shared, shared], start_pos=[128], empty_slots=[2] + ) + assert submitted[-1] == [first] + model.prefill_forward( + tokens[:1], None, model.cache, [256], page_tables_per_layer=[shared, shared], start_pos=[128], empty_slots=[0] + ) + assert submitted[-1][0] not in (first, second) + + +@pytest.mark.parametrize("remap", [None, [1, 0, 2, 3]]) +def test_decode_releases_only_its_slot_history_despite_page_id_collision(remap): + model = adapter() + model.max_batch_size = model.rows = 4 + model.row_sizes, model.sub_caches = (), {} + model.prefill_slot_keys = {0: 1, 1: 2} + model.cache.prefill_history = {1: object(), 2: object()} + model.generator = SimpleNamespace( + model=SimpleNamespace(layers=[SimpleNamespace(sliding=False)]), + prepare_decode=lambda *args, **kwargs: {"logits": torch.zeros(1, 1, 4, 10)}, + replay=lambda *args, **kwargs: None, + _host_logits=lambda tensor: tensor, + ) + # Physical page 2 belongs to decoding row 0. Private history key 2 belongs + # to row 1, whose prefill is still in progress and must remain live. + tables = torch.tensor([[2, 3], [2, 4], [0, 0], [0, 0]]) + model.decode_forward( + torch.zeros(4, 1, dtype=torch.int32), + torch.tensor([256, -1, -1, -1]), + None, + model.cache, + page_tables_per_layer=[tables, tables], + slot_remap=None if remap is None else torch.tensor(remap), + ) + remaining_slot = 1 if remap is None else 0 + assert model.prefill_slot_keys == {remaining_slot: remaining_slot + 1} + assert set(model.cache.prefill_history) == {remaining_slot + 1} + if remap is not None: + model.note_state_slots_moved({source: dest for dest, source in enumerate(remap)}) + assert model.prefill_slot_keys == {1: 1} + + +def spec_adapter(slots, spec_k): + model = object.__new__(Gemma4ForCausalLM) + model.spec_k = spec_k + model.max_batch_size = model.rows = slots + model.sub_caches, model.verify_caches, model.row_sizes, model.cur_state = {}, {}, (), None + model.full_layers = frozenset() + model.verify_shapes = verify_shapes(slots, spec_k) + model.prefill_slot_keys = {} + model.cache = SimpleNamespace(prefill_history={}) + return model + + +@pytest.fixture +def spec_k_restored(monkeypatch): + # spec_plan records the draft length on the class; restore it after the test. + monkeypatch.setattr(Gemma4ForCausalLM, "spec_k", Gemma4ForCausalLM.spec_k) + + +@pytest.mark.parametrize( + "slots,requested,effective,k_by_rows", + [ + (1, 7, 5, ((1, 5),)), + (4, 7, 5, ((1, 5), (4, 3))), + (8, 7, 5, ((1, 5), (4, 3), (8, 0))), + (8, 2, 2, ((1, 2), (4, 2), (8, 0))), + (32, 7, 5, ((1, 5), (4, 3), (32, 0))), + (32, 1, 1, ((1, 1), (4, 1), (32, 0))), + ], +) +def test_spec_plan_follows_the_live_requests_and_fits_the_decode_trace( + spec_k_restored, slots, requested, effective, k_by_rows +): + plan = Gemma4ForCausalLM.spec_plan(None, slots, requested) + assert plan.effective_k == effective and plan.k_by_rows == k_by_rows + assert all(users * (1 + k) <= 32 for users, k in verify_shapes(slots, requested)) + assert plan.accept_modes == ("argmax_ids",) and plan.supports_narrow_decode + + +def test_spec_plan_refuses_a_draft_length_of_zero(spec_k_restored): + assert Gemma4ForCausalLM.spec_plan(None, 8, 0).supported_k == () + + +def test_draft_columns_ride_in_rows_after_the_plain_decode_rows(): + slots, spec_k = 2, 2 + model = spec_adapter(slots, spec_k) + state = SimpleNamespace( + batch=slots * (1 + spec_k), + kv=[None], + page_tables={0: torch.zeros(slots * (1 + spec_k), 4, dtype=torch.int32)}, + columns=1 + spec_k, + ) + model.verify_caches = {slots: state} + model.cache = SimpleNamespace( + batch=slots, kv=[None], page_tables={0: torch.zeros(slots, 4, dtype=torch.int32)}, columns=1 + ) + source = torch.tensor([[5, 6], [7, 8]]) + tables = model._tables(state, [source])[0] + assert tables.tolist() == [[5, 6, 0, 0], [7, 8, 0, 0]] * (1 + spec_k) + assert model._pad_rows(torch.tensor([[11]]), 0).reshape(-1).tolist() == [11, 0] + assert model._pad_rows(torch.tensor([3]), -1).tolist() == [3, -1] + + +def test_verify_block_is_flattened_by_column_and_answers_per_slot(): + slots, spec_k = 2, 2 + model = spec_adapter(slots, spec_k) + state = SimpleNamespace( + batch=slots * (1 + spec_k), + kv=[None], + page_tables={0: torch.zeros(slots * (1 + spec_k), 4, dtype=torch.int32)}, + columns=1 + spec_k, + ) + model.verify_caches = {slots: state} + model.cache = SimpleNamespace( + batch=slots, kv=[None], page_tables={0: torch.zeros(slots, 4, dtype=torch.int32)}, columns=1 + ) + seen = {} + + def prepare_decode(tokens, positions, **kwargs): + seen["tokens"], seen["positions"] = tokens.reshape(-1).tolist(), positions.tolist() + return {"tokens": ttnn.from_torch(torch.arange(32, dtype=torch.int32).reshape(1, 1, 1, 32), dtype=ttnn.uint32)} + + model.generator = SimpleNamespace( + prepare_decode=prepare_decode, + configure_sampling=lambda **kw: seen.setdefault("batch", kw["batch"]), + replay=lambda entry: None, + ) + model.read_decode_output = lambda output: output + block = torch.tensor([[10, 11, 12], [20, 999, 999]]) + positions = torch.tensor([[7, 8, 9], [3, -1, -1]]) + out = model._verify(block, positions, model.cache, [torch.tensor([[1, 2], [3, 4]])], None) + assert seen["tokens"] == [10, 20, 11, 0, 12, 0] and seen["positions"] == [7, 3, 8, -1, 9, -1] + assert seen["batch"] == 6 + # device row j * users + u answers slot u, column j + assert out.argmax_ids.tolist() == [[0, 2, 4], [1, 3, 5]] + + +def test_kv_pool_follows_max_model_len(): + """The pool is at least one request of the served context plus the sliding windows and in-flight prefill tokens.""" + needed = lambda length: length + 5 * ( + (Decoder.SLIDING_WINDOW_PAGES * Decoder.PAGE_SIZE * 32 + (Decoder.SLIDING_WINDOW_PAGES - 1) * Decoder.PAGE_SIZE) + + 2 * Gemma4ForCausalLM.MAX_PREFILL_TOKENS + ) + # a context so long that one request needs more than the DRAM budget still gets its pool (vLLM then refuses or fits it) + huge = 8 * 262144 + assert Gemma4ForCausalLM.get_max_tokens_all_users(max_num_seqs=32, max_model_len=huge) == needed(huge) + assert needed(262144) == 533504 + + +def test_kv_pool_takes_the_dram_budget(): + """At 32 slots the pool is the DRAM budget in rows of the wide backing: bank - reserve - non-KV, minus the plugin's output pages.""" + cls = Gemma4ForCausalLM + row = 10 * 2 * 4 * 128 * 256 * 17 // 16 // 8 + assert cls.KV_ROW_BYTES == row == 348160 + tokens = cls.get_max_tokens_all_users(max_num_seqs=32, max_model_len=262144) + assert tokens == cls.get_max_tokens_all_users(max_num_seqs=32) + assert tokens == 688000 and tokens % Decoder.PAGE_SIZE == 0 and tokens > 533504 + # shorter contexts use the same budget (more cache), they no longer shrink the pool + assert cls.get_max_tokens_all_users(max_num_seqs=32, max_model_len=131072) == tokens + # the rows the plugin allocates (tokens / 128 + one output page per request) leave the reserve free after start-up + rows = tokens // Decoder.PAGE_SIZE + 32 + free = cls.DRAM_BANK_BYTES - cls.NON_KV_BYTES - rows * row + assert cls.DRAM_RESERVE_BYTES <= free < cls.DRAM_RESERVE_BYTES + row + assert free >= cls.MIN_FREE_AFTER_STARTUP_BYTES + (200 << 20) + # calibration points measured on the device (speculative profile; the plain profile is 3 MiB lower: 2696 and 3095) + for blocks, measured_mib in ((4200, 2699), (5400, 3098)): + predicted = (cls.NON_KV_BYTES + blocks * row) >> 20 + assert abs(predicted - measured_mib) <= 4 + + +def test_free_dram_guard_stops_startup(expect_error): + cls = Gemma4ForCausalLM + cls.check_free_dram(cls.MIN_FREE_AFTER_STARTUP_BYTES) + cls.check_free_dram(900 << 20) + with expect_error(RuntimeError, "free after start-up"): + cls.check_free_dram((699 << 20)) + + +def stub_state(rows, columns, slots, table_width=4): + return SimpleNamespace( + batch=rows, + kv=[None], + page_tables={0: torch.zeros(rows, table_width, dtype=torch.int32)}, + columns=columns, + ) + + +def stub_generator(seen, ids): + """Records each prepare_decode (tokens, positions, state) and answers every row with `ids` rows in order.""" + + def prepare_decode(tokens, positions, *, page_table, kv_cache): + seen.append({"tokens": tokens.reshape(-1).tolist(), "positions": positions.tolist(), "state": kv_cache}) + rows = len(positions) + return { + "tokens": ttnn.from_torch( + torch.tensor(ids(rows), dtype=torch.int32).reshape(1, 1, 1, -1), dtype=ttnn.uint32 + ), + "logits": torch.zeros(1, 1, 32, 10), + } + + return SimpleNamespace( + prepare_decode=prepare_decode, + configure_sampling=lambda **kw: None, + replay=lambda entry, **kw: None, + _host_logits=lambda tensor: tensor, + ) + + +def serving_model(slots, spec_k, row_sizes): + """A model with the full set of decode and verify traces of a `slots`-slot server (stubbed device).""" + model = spec_adapter(slots, spec_k) + model.row_sizes = row_sizes + model.cache = stub_state(slots, 1, slots) + model.cache.prefill_history = {} + model.sub_caches = {n: stub_state(n, 1, slots) for n in row_sizes} + model.verify_caches = {users: stub_state(users * (1 + k), 1 + k, slots) for users, k in model.verify_shapes} + model.read_decode_output = lambda output: output + return model + + +def test_decode_runs_the_smallest_trace_that_covers_the_live_rows(): + model = serving_model(8, 0, (1, 4)) + model.verify_shapes = () + seen = [] + model.generator = stub_generator(seen, lambda rows: list(range(32))) + tables = [torch.ones(8, 4, dtype=torch.int32)] + for live, expected in [(1, 1), (3, 4), (4, 4), (5, 8)]: + positions = torch.tensor([10] * live + [-1] * (8 - live)) + model.decode_forward( + torch.ones(8, 1, dtype=torch.int32), positions, None, model.cache, page_tables_per_layer=tables + ) + state = seen[-1]["state"] + assert state.batch == expected and len(seen[-1]["positions"]) == expected + assert state is (model.cache if expected == 8 else model.sub_caches[expected]) + + +def test_verify_with_fewer_users_than_slots_uses_the_smallest_verify_trace(): + model = serving_model(8, 5, (1, 4)) + seen = [] + model.generator = stub_generator(seen, lambda rows: list(range(100, 100 + rows))) + block = torch.arange(48).reshape(8, 6) + positions = torch.full((8, 6), -1) + positions[0] = torch.arange(7, 13) + out = model._verify(block, positions, model.cache, [torch.ones(8, 4, dtype=torch.int32)], None) + assert seen[-1]["state"] is model.verify_caches[1] and len(seen[-1]["positions"]) == 6 + # one user: row j answers column j; slots without a live request answer -1 + assert out.argmax_ids[0].tolist() == [100, 101, 102, 103, 104, 105] + assert (out.argmax_ids[1:] == -1).all() + + +def test_two_to_four_users_use_the_sixteen_row_trace_with_three_drafts(): + model = serving_model(8, 5, (1, 4)) + seen = [] + model.generator = stub_generator(seen, lambda rows: list(range(rows))) + block = torch.arange(48).reshape(8, 6) + positions = torch.full((8, 6), -1) + positions[:3, :4] = torch.arange(20, 24) + out = model._verify(block, positions, model.cache, [torch.ones(8, 4, dtype=torch.int32)], None) + state = seen[-1]["state"] + assert state is model.verify_caches[4] and state.batch == 16 + # four users x four columns: row j * 4 + u answers column j of user u; no column beyond the three drafts + assert out.argmax_ids[:4, :4].tolist() == [[0, 4, 8, 12], [1, 5, 9, 13], [2, 6, 10, 14], [3, 7, 11, 15]] + assert (out.argmax_ids[:, 4:] == -1).all() and (out.argmax_ids[4:] == -1).all() + + +def test_a_draftless_block_above_the_verify_traces_runs_as_a_plain_decode(): + model = serving_model(8, 5, (1, 4)) + seen = [] + model.generator = stub_generator(seen, lambda rows: [1000 + row for row in range(32)]) + live = 5 + block = torch.zeros(8, 6, dtype=torch.long) + block[:, 0] = torch.arange(8) + 50 + positions = torch.full((8, 6), -1) + positions[:live, 0] = 30 + valid = torch.zeros(8, dtype=torch.int32) + out = model._verify(block, positions, model.cache, [torch.ones(8, 4, dtype=torch.int32)], None, valid) + # five live requests: the 8-row plain trace (the full one), column 0 only + assert seen[-1]["state"] is model.cache + assert seen[-1]["tokens"] == list(range(50, 58)) and seen[-1]["positions"] == [30] * 5 + [-1] * 3 + assert out.argmax_ids[:, 0].tolist() == [1000 + row for row in range(8)] + assert (out.argmax_ids[:, 1:] == -1).all() + + +@pytest.mark.parametrize("valid", [None, torch.tensor([0, 0, 2, 0, 0, 0, 0, 0])]) +def test_a_block_with_drafts_above_the_verify_traces_is_refused(valid, expect_error): + model = serving_model(8, 5, (1, 4)) + model.generator = stub_generator([], lambda rows: list(range(32))) + positions = torch.full((8, 6), -1) + positions[:5, 0] = 30 + with expect_error(ValueError, "No verify trace covers 5"): + model._verify(torch.zeros(8, 6, dtype=torch.long), positions, model.cache, [None], None, valid) + + +def test_plain_ids_pick_the_trace_by_live_rows_and_pad_inactive_rows(): + model = serving_model(8, 5, (1, 4)) + seen = [] + model.generator = stub_generator(seen, lambda rows: list(range(32))) + ids = model._decode_plain_ids( + torch.tensor([7, 8, 9, 0, 0, 0, 0, 0]), + torch.tensor([3, 3, 3, -1, -1, -1, -1, -1]), + model.cache, + [torch.ones(8, 4, dtype=torch.int32)], + None, + ) + assert seen[-1]["state"] is model.sub_caches[4] and seen[-1]["tokens"] == [7, 8, 9, 0] + assert ids.tolist() == list(range(8)) + + +@pytest.mark.parametrize("remap", [None, [1, 0, 2, 3, 4, 5, 6, 7]]) +@pytest.mark.parametrize("plain_block", [False, True]) +def test_speculative_steps_release_the_prefill_tail_of_requests_entering_decode(remap, plain_block): + model = serving_model(8, 5, (1, 4)) + model.generator = stub_generator([], lambda rows: list(range(32))) + model.prefill_slot_keys = {0: 11, 1: 12} + model.cache.prefill_history = {11: object(), 12: object()} + block = torch.zeros(8, 6, dtype=torch.long) + positions = torch.full((8, 6), -1) + live = 5 if plain_block else 1 + positions[:live, 0] = 30 + valid = torch.zeros(8, dtype=torch.int32) if plain_block else None + model.decode_forward( + block, + positions, + None, + model.cache, + page_tables_per_layer=[torch.ones(8, 4, dtype=torch.int32)], + spec_mode="argmax_ids", + num_valid_drafts=valid, + slot_remap=None if remap is None else torch.tensor(remap), + ) + if plain_block: + # rows 0-4 decode: both slots with a private tail are released + assert model.prefill_slot_keys == {} and model.cache.prefill_history == {} + else: + # only row 0 decodes: its slot (0, or 1 after the remap) is released, the other slot's prefill continues + kept = 1 if remap is None else 0 + assert model.prefill_slot_keys == {kept: 11 + kept} and set(model.cache.prefill_history) == {11 + kept} + + +def test_warm_up_captures_every_decode_and_verify_trace(monkeypatch): + model = serving_model(8, 5, (1, 4)) + captured = [] + model.generator = SimpleNamespace( + _tables=lambda state, tables: None, + configure_sampling=lambda **kw: None, + prepare_entries=lambda states: captured.extend(states) or [{"tokens": None}] * len(states), + ) + model.mesh = None + monkeypatch.setattr( + ttnn, + "get_memory_view", + lambda *a: SimpleNamespace(total_bytes_allocated_per_bank=0, total_bytes_free_per_bank=1000 << 20), + ) + model.warmup_model_decode(model.cache, True) + expected = [model.cache, *model.sub_caches.values(), *model.verify_caches.values()] + assert len(captured) == len(expected) == 1 + 2 + 2 and all(any(c is e for c in captured) for e in expected) + + +def test_warm_up_stops_when_the_traces_leave_too_little_dram(monkeypatch, expect_error): + model = serving_model(8, 5, (1, 4)) + model.generator = SimpleNamespace( + _tables=lambda state, tables: None, + configure_sampling=lambda **kw: None, + prepare_entries=lambda states: [{"tokens": None}] * len(states), + ) + model.mesh = None + monkeypatch.setattr( + ttnn, + "get_memory_view", + lambda *a: SimpleNamespace(total_bytes_allocated_per_bank=0, total_bytes_free_per_bank=100 << 20), + ) + with expect_error(RuntimeError, "free after start-up"): + model.warmup_model_decode(model.cache, True) + + +@pytest.mark.parametrize("context", [100096, 1152, 129 * 128]) +def test_serving_context_must_be_a_multiple_of_1024(context, expect_error): + with expect_error(ValueError, "multiple of 1024"): + Gemma4ForCausalLM.initialize_vllm_model(SimpleNamespace(), None, 32, max_seq_len=context) + + +@pytest.mark.parametrize("context", [262144, 131072, 65536, 8192]) +def test_supported_serving_contexts_pass_the_page_table_check(context, expect_error): + config = SimpleNamespace(text_config=SimpleNamespace(hidden_size=1, num_hidden_layers=1)) + with expect_error(ValueError, "Expected the Gemma4 31B checkpoint"): # past the context checks + Gemma4ForCausalLM.initialize_vllm_model(config, None, 32, max_seq_len=context) diff --git a/models/demos/gemma4_31b_qb2/tests/test_weight_format.py b/models/demos/gemma4_31b_qb2/tests/test_weight_format.py new file mode 100644 index 000000000000..fe7eef32f6a3 --- /dev/null +++ b/models/demos/gemma4_31b_qb2/tests/test_weight_format.py @@ -0,0 +1,12 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: 2026 Tenstorrent USA, Inc. +"""The weight format is a fixed property of the model: every projection and the LM head are BF8, with no switch.""" + +import ttnn +from models.demos.gemma4_31b_qb2.tt import decoder, model + + +def test_every_projection_and_the_head_are_bf8(): + assert set(decoder.WEIGHT_DTYPES) == {"qkv", "o", "gate_up", "down"} + assert all(dtype == ttnn.bfloat8_b for dtype in decoder.WEIGHT_DTYPES.values()) + assert model.HEAD_DTYPE == ttnn.bfloat8_b diff --git a/models/demos/gemma4_31b_qb2/tt/decode_rows.py b/models/demos/gemma4_31b_qb2/tt/decode_rows.py new file mode 100644 index 000000000000..07cc154fef36 --- /dev/null +++ b/models/demos/gemma4_31b_qb2/tt/decode_rows.py @@ -0,0 +1,42 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: 2026 Tenstorrent USA, Inc. +"""Row counts of the decode traces and the choice of one for a step. + +A decode trace has a fixed number of rows. Rows without a request still cost device time (the full-attention SDPA of +one live row is 8-10x slower in a 32-row trace than in a 1-row trace at 28K context), so a server keeps traces +for a few row counts and each step runs the smallest one that covers its live rows. The vLLM batch is front-packed +(live requests occupy rows 0..n-1), so the live row count is the index of the last live row plus one. +""" + +ROW_SIZES = (1, 4, 8, 16, 32) + + +def decode_row_sizes(slots): + """Row counts of the extra decode traces for a server with `slots` slots (the full trace is always present).""" + return tuple(n for n in ROW_SIZES if n < slots) + + +def rows_covering(live_rows, sizes, full_rows): + """The smallest trace of `sizes` (or the full one) with at least `live_rows` rows.""" + return next((n for n in sizes if n >= live_rows), full_rows) + + +# Speculation by active requests: (max live requests, draft length cap). One request drafts up to MAX_K tokens; up to four +# requests draft 3 each (measured on recorded agent trajectories: K=5 pays for one user, K=3 for four with +# NUMBA_NUM_THREADS=1); beyond four requests a step is a plain decode. Verify traces have users x (1 + K) rows. +VERIFY_K_BY_USERS = ((1, 5), (4, 3)) + + +def verify_shapes(slots, requested_k): + """(users, draft length) of each verify trace of a server with `slots` slots and draft length `requested_k`.""" + shapes = [] + for users, k in VERIFY_K_BY_USERS: + users, k = min(users, slots), min(k, requested_k) + if k >= 1 and users * (1 + k) <= 32 and (not shapes or users > shapes[-1][0]): + shapes.append((users, k)) + return tuple(shapes) + + +def verify_users(live_rows, shapes): + """The users of the smallest verify trace that covers `live_rows`, or None when no verify trace does.""" + return next((users for users, _ in shapes if live_rows <= users), None) diff --git a/models/demos/gemma4_31b_qb2/tt/decoder.py b/models/demos/gemma4_31b_qb2/tt/decoder.py index e9ae70be6fef..e9611ca9ab02 100644 --- a/models/demos/gemma4_31b_qb2/tt/decoder.py +++ b/models/demos/gemma4_31b_qb2/tt/decoder.py @@ -15,6 +15,8 @@ arbitrary rollback into evicted tokens. Page-ID changes require live KV migration. """ +import torch + import ttnn from models.demos.gemma4_31b_qb2.tt.sdpa_l1 import BLACKHOLE_L1_BYTES, full_sdpa_fp32_l1_end from models.demos.gpt_oss.tt.ccl import CCLManager @@ -31,13 +33,20 @@ def _init_subdevice(self): self.ccl_sub_device_id = ttnn.SubDeviceId(0) -# Input cores, readers per DRAM bank, and K tiles per decode matmul block. +# Input cores, readers per DRAM bank, and K tiles per decode matmul block. The K block fixes the accumulation +# order; input cores (a K block may span several input shards) and readers per bank only move data, so they are +# tuned per matmul for speed without changing results. DECODE_GEOMETRY = { - "qkv": (8, 2, 21), + "qkv": (56, 2, 21), "gate_up": (56, 2, 3), - "o": (8, 2, 8), - "down": (8, 2, 21), + "o": (8, 1, 8), + "down": (56, 2, 21), } +# Output storage cores per decode matmul (sets per_core_N), independent of the input cores above, so the input +# multicast geometry can change without changing the output shards the following ops consume. +DECODE_OUTPUT_CORES = {"qkv": 8, "gate_up": 56, "o": 8, "down": 8} +# Storage format of each projection matrix (the tied LM head is in model.py). +WEIGHT_DTYPES = {"qkv": ttnn.bfloat8_b, "o": ttnn.bfloat8_b, "gate_up": ttnn.bfloat8_b, "down": ttnn.bfloat8_b} class Decoder: @@ -141,6 +150,7 @@ def from_state_dict(cls, state_dict, *, hf_config, layer_idx, mesh_device, share if q <= (128 if self.sliding else 256) } self.pref_sdpa_selections = {} + self.pref_weights = {} # the interleaved prefill weights shared by the chunks of one prefill_forward call prefix = f"model.language_model.layers.{layer_idx}." sd = {k[len(prefix) :]: v for k, v in state_dict.items() if k.startswith(prefix)} if not sd: @@ -184,7 +194,9 @@ def tensor(w, dtype=ttnn.bfloat16, dim=None, decode_only=None): def linear(name): return tensor( - sd[name + ".weight"].T, ttnn.bfloat4_b, dim=0 if name in ("self_attn.o_proj", "mlp.down_proj") else 1 + sd[name + ".weight"].T, + WEIGHT_DTYPES[{"self_attn.o_proj": "o", "mlp.down_proj": "down"}.get(name, "gate_up")], + dim=0 if name in ("self_attn.o_proj", "mlp.down_proj") else 1, ) q, k = (sd["self_attn.q_proj.weight"], sd["self_attn.k_proj.weight"]) @@ -197,17 +209,17 @@ def packed(*weights): self.qkv = tensor( packed(q, k, v), - ttnn.bfloat4_b, + WEIGHT_DTYPES["qkv"], dim=1, decode_only="qkv" if not self.sliding else None, ) if not self.sliding: - self.qk = tensor(packed(q, k), ttnn.bfloat4_b, dim=1) + self.qk = tensor(packed(q, k), WEIGHT_DTYPES["qkv"], dim=1) self.o = linear("self_attn.o_proj") self.gate, self.up, self.down = (linear("mlp." + name + "_proj") for name in ("gate", "up", "down")) self.gate_up = tensor( packed(sd["mlp.gate_proj.weight"], sd["mlp.up_proj.weight"]), - ttnn.bfloat4_b, + WEIGHT_DTYPES["gate_up"], dim=1, decode_only="gate_up", ) @@ -293,13 +305,11 @@ def packed(*weights): if weight.memory_config().is_sharded(): assert weight.memory_config() == mem assert weight.buffer_unique_id() == self.decode_weights[name].buffer_unique_id() - # Separate prefill gate/up weights are already resident. Decode - # owns its DRAM-sharded packed copy; retain no unused duplicate. - del self.gate_up - if not self.sliding: - # Tied full-attention prefill consumes qk. Decode already owns - # its separate DRAM-sharded qkv; this setup copy has no reader. - del self.qkv + # Prefill uses the decode copies too: each matrix is rebuilt as an interleaved temporary when a prefill + # chunk needs it (see _prefill_weight), so a layer keeps one copy of its weights, not two. + for name in ("gate_up", "gate", "up", "o", "down", "qkv", "qk"): + if hasattr(self, name): + delattr(self, name) self.layer_scalar = sd["layer_scalar"].item() table_key = ("rope", self.kind) if table_key not in shared_setup: @@ -330,6 +340,14 @@ def packed(*weights): k_chunk_size=128, exp_approx_mode=False, ) + # Sliding decode at 32 rows: the full grid gives each user more cores for its heads. Same results as the + # 8x4 grid at 32 rows (not at smaller batches, where the K range is split differently across cores). + self.decode_sdpa_sliding32 = ttnn.SDPAProgramConfig( + compute_with_storage_grid_size=ttnn.CoreCoord(11, 10), + q_chunk_size=0, + k_chunk_size=128, + exp_approx_mode=False, + ) self.decode_sdpa_medium = ttnn.SDPAProgramConfig( compute_with_storage_grid_size=ttnn.CoreCoord(8, 4), q_chunk_size=0, k_chunk_size=64, exp_approx_mode=False ) @@ -496,10 +514,34 @@ def _reduce(self, x, residual_width=None): ) return ttnn.typecast(y, ttnn.bfloat16) if y.dtype != ttnn.bfloat16 else y - def _pref_linear(self, x, name, activation=None): + def _prefill_weight(self, name): + """Interleaved prefill weight(s) rebuilt from the decode copy; same values, so prefill results are unchanged. + + Returns a tensor, or for "gate_up" the (gate, up) pair. The caller deallocates them after use. + """ + if name in ("qkv", "o", "down"): + return ttnn.to_memory_config(self.decode_weights[name], ttnn.DRAM_MEMORY_CONFIG) + # Slice the parts straight out of the sharded decode weight: copying the whole fused weight first and slicing the copy + # reads and writes the bytes twice and holds a full-size extra tensor while the parts are made. + fused = self.decode_weights["qkv" if name == "qk" else "gate_up"] + shape = list(fused.shape) + zeros = [0] * len(shape) + n = shape[-1] + if name == "qk": + # Tied K=V: the decode copy packs [q | k | v] per rank, prefill takes [q | k]. + width = n - self.kv_heads * self.head_dim + return ttnn.slice(fused, zeros, shape[:-1] + [width], memory_config=ttnn.DRAM_MEMORY_CONFIG) + half = n // 2 + gate = ttnn.slice(fused, zeros, shape[:-1] + [half], memory_config=ttnn.DRAM_MEMORY_CONFIG) + up = ttnn.slice(fused, zeros[:-1] + [half], shape[:-1] + [n], memory_config=ttnn.DRAM_MEMORY_CONFIG) + return gate, up + + def _pref_linear(self, x, name, activation=None, weight=None): import math - weight = getattr(self, name) + shared = weight is None and name in self.pref_weights + owned = weight is None and not shared + weight = self.pref_weights[name] if shared else self._prefill_weight(name) if owned else weight grid = (8, 8) if self.sliding and name in ("qkv", "qk") else (11, 8) if self.sliding else (11, 10) wide_qkv = ( self.sliding @@ -525,6 +567,10 @@ def _pref_linear(self, x, name, activation=None): cap = 32 if name == "o" else 28 if wide_qkv: cap = 42 + if weight.dtype != ttnn.bfloat4_b and x.shape[2] > 4096: + # BF8 weight tiles are twice the size of BF4: a long chunk's circular buffers (per-core M is large) + # would overflow L1 at the larger K block. A smaller K block only changes the accumulation split. + cap = min(cap, 4) kt = weight.shape[-2] // 32 block = max(i for i in range(1, min(kt, cap) + 1) if kt % i == 0) config = ttnn.MatmulMultiCoreReuseMultiCastProgramConfig( @@ -538,7 +584,7 @@ def _pref_linear(self, x, name, activation=None): fuse_batch=False, fused_activation=ttnn.UnaryWithParam(ttnn.UnaryOpType.GELU_TANH) if activation else None, ) - return ttnn.linear( + out = ttnn.linear( x, weight, program_config=config, @@ -546,18 +592,22 @@ def _pref_linear(self, x, name, activation=None): dtype=ttnn.bfloat16, memory_config=ttnn.DRAM_MEMORY_CONFIG, ) + if owned: + ttnn.deallocate(weight) + return out def _dlinear(self, x, name): weight = self.decode_weights[name] kdim, ndim = weight.shape[-2], weight.shape[-1] cores, readers, k_block = DECODE_GEOMETRY[name] + out_cores = DECODE_OUTPUT_CORES[name] row = name in ("o", "down") staged = row and self.sliding x = ttnn.to_memory_config(x, self._mem(kdim, cores)) config = ttnn.MatmulMultiCoreReuseMultiCastDRAMShardedProgramConfig( in0_block_w=k_block, per_core_M=1, - per_core_N=3 if staged else (ndim + cores * 32 - 1) // (cores * 32), + per_core_N=3 if staged else (ndim + out_cores * 32 - 1) // (out_cores * 32), num_workers_per_dram_bank=readers, ) y = ttnn.linear( @@ -643,8 +693,13 @@ def _input_norm(self, x, name, decode): def _prefill_mlp(self, x): n = self._input_norm(x, "pre_feedforward_layernorm", False) - gate = self._pref_linear(n, "gate", "gelu_tanh") - up = self._pref_linear(n, "up") + shared = "gate_up" in self.pref_weights + gate_w, up_w = self.pref_weights["gate_up"] if shared else self._prefill_weight("gate_up") + gate = self._pref_linear(n, "gate", "gelu_tanh", weight=gate_w) + up = self._pref_linear(n, "up", weight=up_w) + if not shared: + ttnn.deallocate(gate_w) + ttnn.deallocate(up_w) del n product = ttnn.multiply(gate, up) del gate, up @@ -681,6 +736,20 @@ def _decode_finish(self, residual, attention): activations=[ttnn.UnaryWithParam(ttnn.UnaryOpType.MUL_UNARY_SFPU, self.layer_scalar)], ) + @staticmethod + def _short_suffix_q_chunk(rows): + """Q chunk for a short chunk of query rows against a long cached prefix, or None for the default choice. + + The chunked full-attention SDPA runs one unit of work per (head, Q chunk) pair on 110 cores, and pairs of Q chunks + are scheduled together when their count is even: 512 rows in two chunks of 256 (or four of 128) keep about a + sixth of the cores busy while each unit streams the whole cached K/V. A Q chunk of 64 gives more units. Measured on a + full layer at 8K-48K cached tokens: 256/512/768/1024 rows take 20-40% less SDPA time (11.2 -> 6.9 ms at 512 rows + after 24K), a count of odd 128-row chunks (384/640/896 rows) is already best at 128, and above 1024 rows the extra K/V + streaming of a smaller chunk costs more than it gains. The result is bit-identical: K is still consumed in chunks of + 128 per query row. + """ + return 64 if rows <= 1024 and rows % 256 == 0 else None + def _prefill_sdpa_config(self, q, cache, page_table, *, start=0): k_chunk = 128 if not self.sliding and start % k_chunk: @@ -690,6 +759,9 @@ def _prefill_sdpa_config(self, q, cache, page_table, *, start=0): raise ValueError(f"No SDPA Q chunk aligns with rows {q.shape[2]} and start {start}") if self.sliding or not self.pref_compute.fp32_dest_acc_en: return self.pref_sdpa_by_q[eligible[0]] + preferred = self._short_suffix_q_chunk(q.shape[2]) + if preferred in eligible: + eligible = [preferred] + [c for c in eligible if c != preferred] tile_bytes = {ttnn.bfloat4_b: 576, ttnn.bfloat8_b: 1088, ttnn.bfloat16: 2048, ttnn.float32: 4096} grid = [11, 10] page_bytes = page_table.buffer_aligned_page_size() @@ -706,10 +778,11 @@ def _prefill_sdpa_config(self, q, cache, page_table, *, start=0): page_table_bytes=page_bytes, ) if end <= BLACKHOLE_L1_BYTES: - key = (q.shape[2], start, str(q.dtype), str(cache.dtype), page_bytes) + # Keyed by the chosen chunk, not the start: the start only decides which Q chunks are eligible, and a key per + # start would grow without bound over a serving run. + key = (q.shape[2], chunk, str(q.dtype), str(cache.dtype), page_bytes) self.pref_sdpa_selections[key] = dict( rows=q.shape[2], - start=start, q_dtype=str(q.dtype), kv_dtype=str(cache.dtype), page_table_bytes=page_bytes, @@ -723,14 +796,39 @@ def _prefill_sdpa_config(self, q, cache, page_table, *, start=0): def _concat_heads(self, attn): return ttnn.experimental.nlp_concat_heads(attn, memory_config=ttnn.DRAM_MEMORY_CONFIG) - def _decode_rope(self, x, cos, sin): - # The native single-user RoPE path exposes its 32 padded head rows. - # SDPA derives the GQA ratio from logical Q heads: restore local head - # ownership before attention (8 Q heads, not 32 padding rows). - result = self._apply_decode_rope(x, cos, sin) - return ttnn.reshape(result, ttnn.Shape(list(x.shape)), ttnn.Shape(list(x.padded_shape))) + def decode_rope_tables(self, rope_positions, batch): + """Per-step cos/sin rows for decode RoPE, shared by every layer of this kind. - def _prefill_qkv(self, x, start): + Returns (cos, sin, memory): [1,B,1,D] tables height-sharded one user per core (rows padded to a tile) + and the matching height-sharded memory for [1,B,heads,D] inputs. External fixed-slot schedulers may + use -1 positions to suppress cache/SDPA rows; their separate nonnegative RoPE indices avoid an + out-of-bounds lookup. + """ + idx = ttnn.reshape(ttnn.typecast(rope_positions, ttnn.uint32), (1, batch)) + grid = self.device.compute_with_storage_grid_size() + memory = ttnn.create_sharded_memory_config( + (32, self.head_dim), + ttnn.num_cores_to_corerangeset(batch, grid, True), + ttnn.ShardStrategy.HEIGHT, + ttnn.ShardOrientation.ROW_MAJOR, + use_height_and_width_as_shard_shape=True, + ) + tables = [] + for table in (self.decode_cos, self.decode_sin): + rows = ttnn.embedding(idx, table, layout=ttnn.ROW_MAJOR_LAYOUT) + rows = ttnn.reshape(rows, (1, batch, 1, self.head_dim)) + tables.append(ttnn.to_layout(rows, ttnn.TILE_LAYOUT, memory_config=memory)) + return tables[0], tables[1], memory + + def _decode_rope(self, x, rope, memory_config=None): + # HF rotate-half RoPE in decode mode: one user per core, cos/sin rows broadcast over the user's heads. + # Same per-element arithmetic as the transposed rotary_embedding path (rotate, *sin, *cos, add). + cos, sin, memory = rope + return ttnn.experimental.rotary_embedding_hf( + ttnn.to_memory_config(x, memory), cos, sin, is_decode_mode=True, memory_config=memory_config or memory + ) + + def _prefill_qkv(self, x, start, rope=None): tied = not self.sliding fused = self._pref_linear(self._input_norm(x, "input_layernorm", False), "qk" if tied else "qkv") q, k, v = ttnn.experimental.nlp_create_qkv_heads( @@ -748,7 +846,9 @@ def _prefill_qkv(self, x, start): del k # Only a start-zero, whole-width prefix can alias the table buffer. rope_shape = ttnn.Shape([1, 1, x.shape[2], self.head_dim]) - if start == 0: + if rope is not None: + cos, sin = rope + elif start == 0: cos = ttnn.reshape(self.cos, rope_shape, rope_shape) sin = ttnn.reshape(self.sin, rope_shape, rope_shape) else: @@ -765,10 +865,85 @@ def _prefill_qkv(self, x, start): v = self._norm(v) return (q, k, v) - def prefill_forward( - self, x, *, page_table, kv_cache, consume_input=False, start_pos=0, history=None, return_history=False + def prefill_chunk_inputs(self, seq, start_pos, host_row=None): + """Per-chunk inputs that depend on the absolute position, as device tensors (runtime values, not program attributes). + + A Python slice of the rope tables or the page table, or a Python start index, is part of the program hash, so each + new (length, start) compiled new programs. Here the rope rows are gathered by position ids, the cache-fill page ids + are uploaded as a row, and the chunk start is a one-element tensor; the programs then depend on the chunk length only. + """ + chunks = [] + for start in range(0, seq, self.PREFILL_CHUNK): + length = min(start + self.PREFILL_CHUNK, seq) - start + absolute = start_pos + start + if host_row is None: + ids = ttnn.from_torch( + torch.arange(absolute, absolute + length, dtype=torch.int32).reshape(1, length), + dtype=ttnn.uint32, + layout=ttnn.ROW_MAJOR_LAYOUT, + device=self.device, + mesh_mapper=ttnn.ReplicateTensorToMesh(self.device), + ) + shape = ttnn.Shape([1, 1, length, self.head_dim]) + rope = tuple( + ttnn.to_layout(ttnn.reshape(ttnn.embedding(ids, table), shape), ttnn.TILE_LAYOUT) + for table in (self.decode_cos, self.decode_sin) + ) + chunks.append(dict(rope=rope)) + continue + first = absolute // self.PAGE_SIZE + fill = ttnn.from_torch( + host_row[:, first : first + length // self.PAGE_SIZE].to(torch.int32), + dtype=ttnn.int32, + layout=ttnn.ROW_MAJOR_LAYOUT, + device=self.device, + mesh_mapper=ttnn.ReplicateTensorToMesh(self.device), + ) + chunk_start = ttnn.from_torch( + torch.tensor([absolute], dtype=torch.int32), + dtype=ttnn.int32, + layout=ttnn.ROW_MAJOR_LAYOUT, + device=self.device, + mesh_mapper=ttnn.ReplicateTensorToMesh(self.device), + ) + chunks.append(dict(fill=fill, start=chunk_start)) + return chunks + + def prefill_forward(self, x, **kwargs): + """Prefill a page-aligned span (see _prefill_chunks). A span of several chunks rebuilds the layer's interleaved + prefill weights once and shares them between its chunks (they are the same values for every chunk).""" + self.pref_weights = self._rebuild_weights_once(x.shape[0] * -(-x.shape[2] // self.PREFILL_CHUNK)) + try: + return self._prefill_chunks(x, **kwargs) + finally: + for weight in self.pref_weights.values(): + for part in weight if isinstance(weight, tuple) else (weight,): + ttnn.deallocate(part) + self.pref_weights = {} + + def _rebuild_weights_once(self, chunks): + if chunks < 2: + return {} + return {name: self._prefill_weight(name) for name in ("qkv" if self.sliding else "qk", "o", "gate_up", "down")} + + def _prefill_chunks( + self, + x, + *, + page_table, + kv_cache, + consume_input=False, + start_pos=0, + history=None, + return_history=False, + chunk_inputs=None, + padded_tails=False, ): - """Prefill a page-aligned span, optionally retaining sliding history for a continuation.""" + """Prefill a page-aligned span, optionally retaining sliding history for a continuation. + + chunk_inputs (prefill_chunk_inputs) makes the position-dependent ops runtime-valued; padded_tails keeps the + returned history to whole pages (the rows after a logical end are overwritten by the continuation's rewind). + """ if start_pos < 0 or start_pos % self.PAGE_SIZE: raise ValueError("Prefill start must be a nonnegative cache-page boundary") if start_pos and self.sliding and history is None: @@ -818,12 +993,16 @@ def prefill_forward( if pad: inp = ttnn.pad(inp, [(0, 0), (0, 0), (0, pad), (0, 0)], value=0.0) absolute_start = start_pos + start - q, k, v = self._prefill_qkv(inp, absolute_start) + prepared = None if chunk_inputs is None else chunk_inputs[start // self.PREFILL_CHUNK] + q, k, v = self._prefill_qkv(inp, absolute_start, None if prepared is None else prepared["rope"]) if not self.sliding: q = ttnn.typecast(q, ttnn.bfloat8_b) sdpa_config = self._prefill_sdpa_config(q, kv_cache[0], pt, start=absolute_start) length = inp.shape[2] - fill_pt = pt[:, absolute_start // self.PAGE_SIZE : (absolute_start + length) // self.PAGE_SIZE] + if prepared is None: + fill_pt = pt[:, absolute_start // self.PAGE_SIZE : (absolute_start + length) // self.PAGE_SIZE] + else: + fill_pt = prepared["fill"] ttnn.experimental.paged_fill_cache( kv_cache[0], ttnn.typecast(k, kv_cache[0].dtype), fill_pt, batch_idx=0 ) @@ -862,12 +1041,17 @@ def prefill_forward( compute_kernel_config=self.pref_compute, ) else: + start_arg = ( + {"chunk_start_idx": absolute_start} + if prepared is None + else {"chunk_start_idx_tensor": prepared["start"]} + ) attn = ttnn.transformer.chunked_scaled_dot_product_attention( q, kv_cache[0], kv_cache[1], pt, - chunk_start_idx=absolute_start, + **start_arg, scale=1.0, program_config=sdpa_config, compute_kernel_config=self.pref_compute, @@ -878,7 +1062,7 @@ def prefill_forward( # can replay its partial page without losing the prior window. tails = [] for index, value in enumerate((k, v)): - live = value[:, :, : end - start, :] + live = value if padded_tails else value[:, :, : end - start, :] joined = ttnn.concat([previous_kv[index], live], dim=2) if previous_kv is not None else live tail_start = max(0, joined.shape[2] - (self.window + self.PAGE_SIZE)) tails.append(joined[:, :, tail_start:, :]) @@ -961,8 +1145,22 @@ def prefill_forward( result = ttnn.concat(users, dim=0) if batch > 1 else users[0] return (result, next_history) if return_history else result - def decode_forward(self, x, *, positions, page_table, kv_cache, rope_positions=None, cyclic_cache=True): - """One token per user; all runtime data inputs and outputs are device tensors.""" + def decode_forward( + self, + x, + *, + positions, + page_table, + kv_cache, + rope_positions=None, + cyclic_cache=True, + rope=None, + update_positions=None, + ): + """One token per user; all runtime data inputs and outputs are device tensors. + + rope: optional (cos, sin, memory) from decode_rope_tables, looked up once per step for all layers of a kind. + """ batch = x.shape[2] assert x.shape[0] == x.shape[1] == 1 and x.shape[3] == self.residual_width assert 1 <= batch <= 32 and page_table.shape[0] == batch and (positions.shape[0] == batch) @@ -979,27 +1177,14 @@ def decode_forward(self, x, *, positions, page_table, kv_cache, rope_positions=N head_mem = ttnn.L1_MEMORY_CONFIG if batch != 1: q, k, v = (ttnn.to_memory_config(t, head_mem) for t in (q, k, v)) - # External fixed-slot schedulers may use -1 to suppress cache/SDPA rows. - # Their separate nonnegative RoPE indices avoid an out-of-bounds lookup. - idx = ttnn.reshape( - ttnn.typecast(positions if rope_positions is None else rope_positions, ttnn.uint32), (1, batch) - ) - # A full32-index row selects the embedding kernel with native TILE - # output, avoiding separate small tilize operations. - idx = ttnn.pad(idx, [(0, 0), (0, 32 - batch)], value=0) - cos = ttnn.embedding(idx, self.decode_cos, layout=ttnn.TILE_LAYOUT) - sin = ttnn.embedding(idx, self.decode_sin, layout=ttnn.TILE_LAYOUT) - # Explicit padded shapes crop logical rows without a device copy. - logical = ttnn.Shape([1, 1, batch, self.head_dim]) - padded = ttnn.Shape([1, 1, 32, self.head_dim]) - cos = ttnn.reshape(cos, logical, padded) - sin = ttnn.reshape(sin, logical, padded) - q = self._decode_rope(self._norm(q, "self_attn.q_norm"), cos, sin) + if rope is None: + rope = self.decode_rope_tables(positions if rope_positions is None else rope_positions, batch) + q = self._decode_rope(self._norm(q, "self_attn.q_norm"), rope) if not self.sliding: v = self._norm(v) - k = self._decode_rope(ttnn.multiply(v, self.norms["self_attn.k_norm"]), cos, sin) + k = self._decode_rope(ttnn.multiply(v, self.norms["self_attn.k_norm"]), rope, cache_mem) else: - k = self._decode_rope(self._norm(k, "self_attn.k_norm"), cos, sin) + k = self._decode_rope(self._norm(k, "self_attn.k_norm"), rope, cache_mem) v = self._norm(v) k = ttnn.to_memory_config(k, cache_mem) # Fused cache update requires disjoint K/V cores. K occupies rows0-3 @@ -1017,24 +1202,28 @@ def decode_forward(self, x, *, positions, page_table, kv_cache, rope_positions=N v = ttnn.to_memory_config(v, vmem) bounded = cyclic_cache and self.sliding and page_table.padded_shape[1] > kv_cache[0].shape[0] modulo = {"cache_position_modulo": self.SLIDING_WINDOW_PAGES * self.PAGE_SIZE} if bounded else {} - if bounded: - # Fused update still enforces logical pages <= physical pages. - # Native separate updates support a bounded pool explicitly. The - # documented cyclic table repeats its first nine private pages. - ttnn.experimental.paged_update_cache( - kv_cache[0], k, update_idxs_tensor=positions, page_table=page_table, **modulo - ) - ttnn.experimental.paged_update_cache( - kv_cache[1], v, update_idxs_tensor=positions, page_table=page_table, **modulo - ) - else: - ttnn.experimental.paged_fused_update_cache( - kv_cache[0], k, kv_cache[1], v, update_idxs_tensor=positions, page_table=page_table - ) + # Rows that write into the same cache tile (a speculative block's columns of one user) must be updated by + # separate calls: concurrent writers to one tile lose updates. update_positions holds one index tensor per + # call, with -1 for the rows a call skips. + for update_idxs in update_positions or [positions]: + if bounded: + # Fused update still enforces logical pages <= physical pages. + # Native separate updates support a bounded pool explicitly. The + # documented cyclic table repeats its first nine private pages. + ttnn.experimental.paged_update_cache( + kv_cache[0], k, update_idxs_tensor=update_idxs, page_table=page_table, **modulo + ) + ttnn.experimental.paged_update_cache( + kv_cache[1], v, update_idxs_tensor=update_idxs, page_table=page_table, **modulo + ) + else: + ttnn.experimental.paged_fused_update_cache( + kv_cache[0], k, kv_cache[1], v, update_idxs_tensor=update_idxs, page_table=page_table + ) ttnn.deallocate(k) ttnn.deallocate(v) q = ttnn.to_memory_config(q, ttnn.DRAM_MEMORY_CONFIG) - sdpa_config = self.decode_sdpa_config + sdpa_config = self.decode_sdpa_sliding32 if self.sliding and batch == 32 else self.decode_sdpa_config if not self.sliding: horizon = page_table.shape[1] * self.PAGE_SIZE # The maximum-capacity sweep slightly favors K128; preserve K64 @@ -1171,28 +1360,3 @@ def _norm(self, x, name=None): compute_kernel_config=self.compute, memory_config=ttnn.L1_MEMORY_CONFIG if small_l1 else ttnn.DRAM_MEMORY_CONFIG, ) - - def _apply_decode_rope(self, x, cos, sin): - batch = x.shape[1] - cos = ttnn.reshape(cos, (1, 1, batch, self.head_dim)) - sin = ttnn.reshape(sin, (1, 1, batch, self.head_dim)) - if batch == 1: - return ttnn.experimental.rotary_embedding( - x, - cos, - sin, - token_index=0, - memory_config=(ttnn.L1_HEIGHT_SHARDED_MEMORY_CONFIG if x.shape[2] != 32 else (ttnn.DRAM_MEMORY_CONFIG)), - ) - # Interleaved decode Q must reside in DRAM for SDPA. Write the - # required transpose there directly; K can remain in L1 until update. - result = ttnn.transpose( - ttnn.experimental.rotary_embedding(ttnn.transpose(x, 1, 2), cos, sin), - 1, - 2, - memory_config=(ttnn.DRAM_MEMORY_CONFIG if (x.shape[2] == 32) else x.memory_config()), - ) - result = ttnn.reshape( - result, ttnn.Shape([1, batch, x.shape[2], self.head_dim]), ttnn.Shape([1, batch, 32, self.head_dim]) - ) - return result diff --git a/models/demos/gemma4_31b_qb2/tt/generator.py b/models/demos/gemma4_31b_qb2/tt/generator.py index f2058eca3a93..8243223ef333 100644 --- a/models/demos/gemma4_31b_qb2/tt/generator.py +++ b/models/demos/gemma4_31b_qb2/tt/generator.py @@ -36,6 +36,8 @@ class CacheState: # Standalone mode retains the stricter private cyclic-cache contract. vllm_owned: bool = False prefill_history: dict = field(default_factory=dict) + # Draft columns of a speculative decode (see Decoder.decode_forward): rows = users x columns, one cache update per column. + columns: int = 1 class Gemma4Generator: @@ -49,7 +51,9 @@ def __init__(self, model, *, max_seq_len=2048): self.initial_capacity = max_seq_len self.states = {} self.traces = {} - self.prefill_shapes = set() + # Persistent inputs of each decode entry, kept when the traces are retired: only the traces are recaptured, one at a + # time, and capturing one trace allocates no persistent tensor that could land on another trace's memory. + self.entries = {} self.counters = { k: 0 for k in ( @@ -327,6 +331,42 @@ def _read_tokens(self, token, batch): self.counters["token_readbacks"] += 1 return ttnn.to_torch(ttnn.get_device_tensors(token)[0]).reshape(-1)[:batch].long() + def _restore_prefill_history(self, state, page_tables, slot, start): + """Gather the preceding sliding window from vLLM's retained KV pages. + + The scheduler owns cache identity and eviction. Only pages in this + request's current table are read; null pages are never valid history. + Decode and cached continuation consume the same stored BF8 values. + """ + histories = [] + bounds = {} + for i, layer in enumerate(self.model.layers): + if not layer.sliding: + histories.append(None) + continue + first = max(0, start - layer.window) // Decoder.PAGE_SIZE + pages = page_tables[i][slot, first : start // Decoder.PAGE_SIZE].tolist() + if not pages or any(p <= 0 or p >= state.kv[i][0].shape[0] for p in pages): + raise ValueError(f"Missing sliding prefix pages for layer {i} at position {start}: {pages}") + pair = [] + for cache in state.kv[i]: + # The page id is a runtime value of the slice (device tensors): a Python slice cache[p : p + 1] puts p + # in the program hash, so every request with new block ids compiled new programs and retired the traces. + shape = tuple(cache.shape) + parts = [] + for p in pages: + if (p, 0) not in bounds: + bounds[(p, 0)] = self._upload(torch.tensor([p, 0, 0, 0]), ttnn.uint32) + if (p, shape) not in bounds: + bounds[(p, shape)] = self._upload(torch.tensor([p + 1, *shape[1:]]), ttnn.uint32) + parts.append( + ttnn.slice(cache, bounds[(p, 0)], bounds[(p, shape)], slice_dim=0, num_devices=shape[0]) + ) + tail = parts[0] if len(parts) == 1 else ttnn.concat(parts, dim=2) + pair.append(ttnn.typecast(tail, ttnn.bfloat16)) + histories.append(tuple(pair)) + return histories + def prefill_forward( self, tokens, @@ -360,11 +400,7 @@ def prefill_forward( starts = [0] * len(prompt_lens) if start_positions is None else list(start_positions) if len(starts) != len(prompt_lens) or any(s < 0 or s >= n for s, n in zip(starts, prompt_lens)): raise ValueError("Prefill must advance each request from its current position") - shape_key = (state.batch, state.capacity, tuple(prompt_lens), tuple(starts), bool(return_all_logits)) - if shape_key not in self.prefill_shapes: - # A new prefill signature may create cached protocol semaphores or - # norm buffers. Retire old traces before those persistent allocations. - self._retire_traces() + programs = self.mesh.num_program_cache_entries() self._tables(state, page_table) outputs = [] for row, (slot, length) in enumerate(zip(slots, prompt_lens)): @@ -379,10 +415,22 @@ def prefill_forward( if start == 0: histories = [None] * len(self.model.layers) else: - previous = state.prefill_history[key] + previous = state.prefill_history.get(key) + if previous is None: + if not state.vllm_owned or start != aligned_start: + raise ValueError("Cached prefill must start at a whole page in the serving cache") + previous = { + "end": start, + "layers": self._restore_prefill_history(state, page_table, slot, start), + } if previous["end"] != start: - raise ValueError("Prefill history does not match the scheduler continuation") - rewind = start - aligned_start + raise ValueError( + "Prefill history does not match the scheduler continuation: " + f"key={key}, saved_end={previous['end']}, start={start}" + ) + # The saved tails end at tail_end: the logical end, or the page-padded end when the chunk ran + # with runtime-valued inputs (rows after the logical end are replaced by this continuation). + rewind = previous.get("tail_end", start) - aligned_start histories = [] for tail in previous["layers"]: if tail is None: @@ -394,10 +442,14 @@ def prefill_forward( continue begin = max(0, valid - 1024) histories.append(tuple(t[:, :, begin:valid, :] for t in tail)) - ids = torch.zeros(1, (chunk_length + 31) // 32 * 32, dtype=torch.int32) + # Runtime-valued chunk inputs (see Decoder.prefill_chunk_inputs) need whole cache pages; the programs then + # depend on the padded length only, not on the logical length or the start position. + runtime_inputs = not return_all_logits + padded = (chunk_length + Decoder.PAGE_SIZE - 1) // Decoder.PAGE_SIZE * Decoder.PAGE_SIZE + ids = torch.zeros(1, padded if runtime_inputs else (chunk_length + 31) // 32 * 32, dtype=torch.int32) ids[0, :chunk_length] = tokens[row, aligned_start:length] tt_ids = self._upload(ids, ttnn.uint32) - tables, uploaded_rows = {}, {} + tables, uploaded_rows, host_rows, rows_by_kind = {}, {}, {}, {} for kind, table in page_table.items(): sentinel = state.vllm_owned and self.model.layers[kind].kind == "sliding_attention" key = (id(table), sentinel) @@ -409,26 +461,43 @@ def prefill_forward( # persistent decode tables retain independent storage. row_table = row_table.masked_fill(row_table == 0, -1) uploaded_rows[key] = self._upload(row_table) + host_rows[key] = row_table self.counters["prefill_table_uploads"] += 1 tables[kind] = uploaded_rows[key] + rows_by_kind[kind] = host_rows[key] out = self.model.prefill_device( tt_ids, - sequence_length=chunk_length, + sequence_length=padded if runtime_inputs else chunk_length, page_tables=tables, kv_cache=state.kv, all_logits=return_all_logits, start_pos=aligned_start, histories=histories, + host_rows=rows_by_kind if runtime_inputs else None, + valid_length=chunk_length if runtime_inputs else None, ) if history_keys is not None: - state.prefill_history[history_keys[row]] = {"end": length, "layers": histories} + state.prefill_history[history_keys[row]] = { + "end": length, + "tail_end": aligned_start + padded if runtime_inputs else length, + "layers": histories, + } outputs.append( out if return_device else self._host_logits(out).reshape(1, -1, self.model.config.vocab_size) ) # Device-return outputs retain their own owners in outputs. Host # mode has consumed out; do not carry it into the next prefill. - del out, tt_ids, tables, uploaded_rows - self.prefill_shapes.add(shape_key) + del out, tt_ids, tables, uploaded_rows, host_rows, rows_by_kind + compiled = self.mesh.num_program_cache_entries() - programs + if compiled: + logger.debug( + "Prefill {} from {} compiled {} programs: decode traces retired", prompt_lens, starts, compiled + ) + # A prefill that compiled new programs created persistent device state (programs, semaphores, buffers) + # that a captured decode trace may overlap, so the traces are rebuilt. A signature the program cache + # already holds creates none: warm_prefill_shapes compiles every length class before serving, and + # the decode trace then survives the prefills of an agent loop. + self._retire_traces(keep_entries=True) if return_device: return outputs width = max(prompt_lens) if return_all_logits else 1 @@ -445,21 +514,37 @@ def _model_step(self, entry, state): kv_cache=state.kv, batch=state.batch, rope_positions=entry["rope_positions"], + update_positions=entry["update_positions"], ) ttnn.copy(out, entry["logits"]) + # Only the first update column advances with the step; the other draft columns are reloaded by every verify step before + # they are read, so they need no increment here. ttnn.plus_one(entry["positions"], skip_negative_entries=True) + if entry["update_positions"][0] is not entry["positions"]: + ttnn.plus_one(entry["update_positions"][0], skip_negative_entries=True) ttnn.plus_one(entry["rope_positions"]) def sample_prefill_outputs(self, outputs, state, sampling_params): """Consume compact prefill rows with the canonical sampling trace.""" batch = len(outputs) self.configure_sampling(batch=batch, **sampling_params) - merged = outputs[0] if batch == 1 else ttnn.concat(outputs, dim=2) - if batch > 1 and batch < 32: - merged = ttnn.pad(merged, [(0, 0), (0, 0), (0, 32 - batch), (0, 0)], value=0.0) + # The concat/pad programs of a new batch size compile here, with decode traces live, outside prefill_forward's + # count: they are counted too (warm_prefill_shapes compiles every batch size, so this normally adds none). + programs = self.mesh.num_program_cache_entries() + merged = self._merge_prefill_rows(outputs) + compiled = self.mesh.num_program_cache_entries() - programs entry = self._entry(state) + programs = self.mesh.num_program_cache_entries() shape = ttnn.Shape([1, 1, 32, self.model.config.vocab_size // 4]) ttnn.copy(ttnn.reshape(merged, shape, shape), entry["logits"]) + compiled += self.mesh.num_program_cache_entries() - programs + if compiled: + # Programs compiled while decode traces were live: no captured trace, the sampling trace included, may run before + # it is rebuilt. Rebuilding the entry overwrites its logits, so the merged rows are copied again. + logger.info("Sampling a batch of {} compiled {} programs: decode traces retired", batch, compiled) + self._retire_traces(keep_entries=True) + entry = self._entry(state) + ttnn.copy(ttnn.reshape(merged, shape, shape), entry["logits"]) # CQ0 consumes the copy before replay. Release temporary prefill owners # so a hot trace never replays with unrelated live postcapture buffers. outputs.clear() @@ -468,11 +553,92 @@ def sample_prefill_outputs(self, outputs, state, sampling_params): self.counters["sampling_replays"] += 1 return self._read_tokens(entry["tokens"], batch).reshape(-1, 1) + def _merge_prefill_rows(self, outputs): + """One 32-row logits tile from the compact rows of a prefill batch.""" + batch = len(outputs) + merged = outputs[0] if batch == 1 else ttnn.concat(outputs, dim=2) + if 1 < batch < 32: + merged = ttnn.pad(merged, [(0, 0), (0, 0), (0, 32 - batch), (0, 0)], value=0.0) + return merged + def _sample_step(self, entry): self.sampler(entry["logits"], tt_out_tok=entry["tokens"]) # Seed state advances on device, including when an explicit seed was set. ttnn.plus_one(self.sampler.seeds_tt_tensor) + def warm_prefill_shapes(self, state, page_table, max_tokens): + """Compile the prefill programs of every length class before serving. + + Prefill runs on whole cache pages (runtime-valued inputs, see Decoder.prefill_chunk_inputs), so programs are keyed + by the page-padded chunk length only. Warmed here: every padded length from position 0, the logical lengths 1..32 + and 129..160 (the selectable last-token rows at both widths), for every padded length a continuation after a full + 1024-token sliding window (from an aligned and from an unaligned previous end), and in the serving cache the + prefix-cache history restore for every page count. A continuation at any other start or logical length then + compiles nothing. Not covered, so each still compiles once and retires the decode traces: a continuation after + fewer than 1024 sliding tokens (a history of 128..896 rows: a chunked prompt whose first chunk was short, or a + prefix hit at a short aligned start), and a continuation chunk of more than max_tokens - 1152 padded rows (7041..8320 + rows for the 8192-token limit, since an unaligned start adds up to a page). Uses blocks 1.. of the (still empty) pool; + nothing there is read afterwards. + """ + page = Decoder.PAGE_SIZE + window = 1024 + page + lengths = list(range(page, max_tokens + 1, page)) + for length in lengths: + self.prefill_forward( + torch.ones(1, length, dtype=torch.int32), + page_table=page_table, + kv_cache=state, + prompt_lens=[length], + slots=[0], + ) + # The 32 last-token rows, at both residual widths (prompts of at most 128 tokens run at width 5376, longer at 1344). + for length in (*range(1, 33), *range(page + 1, page + 33)): + self.prefill_forward( + torch.ones(1, length, dtype=torch.int32), + page_table=page_table, + kv_cache=state, + prompt_lens=[length], + slots=[0], + ) + key = -1 + for chunk in range(page, max_tokens - window + 1, page): + for previous_end in (window, window + 1): + tokens = torch.ones(1, window + chunk, dtype=torch.int32) + self.prefill_forward( + tokens, + page_table=page_table, + kv_cache=state, + prompt_lens=[previous_end], + slots=[0], + history_keys=[key], + ) + self.prefill_forward( + tokens, + page_table=page_table, + kv_cache=state, + prompt_lens=[window + chunk], + slots=[0], + start_positions=[previous_end], + history_keys=[key], + ) + state.prefill_history.pop(key, None) + if getattr(state, "vllm_owned", False): + # A request whose first chunk starts after cached pages restores its sliding history from the pages (slice, concat and + # typecast programs per page count); the first prefix hit after start-up would compile them and retire the traces. + for start in range(page, 1024 + page, page): + self._restore_prefill_history(state, page_table, 0, start) + # Every prefill batch size of sample_prefill_outputs (the row merge compiles per size). + for batch in range(2, 33): + rows = [ + self.model.tensor(torch.zeros(1, 1, 1, self.model.config.vocab_size), dim=3, dtype=ttnn.bfloat16) + for _ in range(batch) + ] + self._merge_prefill_rows(rows) + del rows + state.prefill_history.pop(key, None) + self._retire_traces(keep_entries=True) + logger.info("Warmed {} prefill length classes up to {} tokens", len(lengths), max_tokens) + def _entry(self, state): key = id(state) if key in self.traces: @@ -484,7 +650,33 @@ def _entry(self, state): "Captured cache/table tensors were replaced; retire traces before replacing backing storage" ) return self.traces[key] - self._retire_traces() + entry = self.entries.get(key) + if entry is None: + # Persistent tensors must not be allocated while a trace is captured: they could land on its intermediates. + self._retire_traces(keep_entries=True) + entry = self.entries[key] = self._new_entry(state) + self._warm_entry(entry, state) + self._capture_entry(entry, state) + self.traces[key] = entry + return entry + + def prepare_entries(self, states): + """Capture one decode trace per cache state; the states share the KV pool and differ in rows. + + Every entry's persistent inputs are allocated before any trace is captured, so no persistent tensor can land + on memory that a captured trace reuses for intermediates. + """ + self._retire_traces(keep_entries=True) + entries = [self.entries.get(id(state)) or self._new_entry(state) for state in states] + self.entries.update({id(state): entry for state, entry in zip(states, entries)}) + for entry, state in zip(entries, states): + self._warm_entry(entry, state) + for entry, state in zip(entries, states): + self._capture_entry(entry, state) + self.traces[id(state)] = entry + return entries + + def _new_entry(self, state): entry = { "tokens": self._upload( torch.zeros(1, 1, 1, 32, dtype=torch.int32), @@ -504,6 +696,16 @@ def _entry(self, state): "table_ids": {k: id(v) for k, v in state.device_tables.items()}, "remaining_steps": 0, } + # One cache-update index tensor per draft column (see Decoder.decode_forward); a single column updates with the + # positions themselves. + entry["update_positions"] = ( + [entry["positions"]] + if state.columns == 1 + else [self._upload(torch.zeros(state.batch, dtype=torch.int32)) for _ in range(state.columns)] + ) + return entry + + def _warm_entry(self, entry, state): for kind, table in state.device_tables.items(): scratch_table = ( torch.tensor(state.scratch_pages[kind])[:, None] @@ -528,6 +730,9 @@ def _entry(self, state): self._refresh(torch.zeros(state.batch, dtype=torch.int32), entry["positions"], "position_refreshes") self._refresh(torch.zeros(state.batch, dtype=torch.int32), entry["rope_positions"], "rope_refreshes") self._refresh(torch.zeros(1, 1, 1, 32, dtype=torch.int32), entry["tokens"], "token_refreshes") + self._refresh_update_positions(entry, torch.zeros(state.batch, dtype=torch.int32)) + + def _capture_entry(self, entry, state): logger.info("Capturing model trace") with corruptible_allocation_scope(self.mesh): entry["model_trace"] = ttnn.begin_trace_capture(self.mesh, cq_id=0) @@ -538,8 +743,6 @@ def _entry(self, state): entry["sampling_trace"] = ttnn.begin_trace_capture(self.mesh, cq_id=0) self._sample_step(entry) ttnn.end_trace_capture(self.mesh, entry["sampling_trace"], cq_id=0) - self.traces[key] = entry - return entry def replay(self, entry, *, sample=True): """Continue a prepared fixed-slot request without rebuilding device inputs.""" @@ -554,6 +757,14 @@ def replay(self, entry, *, sample=True): ttnn.execute_trace(self.mesh, entry["sampling_trace"], cq_id=0, blocking=False) self.counters["sampling_replays"] += 1 + def _refresh_update_positions(self, entry, positions): + """Column c of the decode rows (row // slots) is updated by call c, which skips every other row.""" + columns = len(entry["update_positions"]) + if columns > 1: + column = torch.arange(len(positions)) // (len(positions) // columns) + for c, tensor in enumerate(entry["update_positions"]): + self._refresh(torch.where(column == c, positions, -1).int(), tensor, "position_refreshes") + def prepare_decode( self, tokens, @@ -589,6 +800,7 @@ def prepare_decode( self._refresh(ids, entry["tokens"], "token_refreshes") self._refresh(start_pos.int(), entry["positions"], "position_refreshes") self._refresh(start_pos.int().clamp_min(0), entry["rope_positions"], "rope_refreshes") + self._refresh_update_positions(entry, start_pos.int()) active = start_pos[start_pos >= 0] entry["remaining_steps"] = state.capacity - int(active.max()) if active.numel() else state.capacity return entry @@ -723,7 +935,7 @@ def reset(self): ) self.configure_sampling() - def _retire_traces(self): + def _retire_traces(self, *, keep_entries=False): if self.traces: ttnn.synchronize_device(self.mesh) self.counters["synchronizations"] += 1 @@ -731,6 +943,8 @@ def _retire_traces(self): for name in ("model_trace", "sampling_trace"): ttnn.release_trace(self.mesh, entry[name]) self.traces.clear() + if not keep_entries: + self.entries.clear() def close(self): self._retire_traces() diff --git a/models/demos/gemma4_31b_qb2/tt/generator_vllm.py b/models/demos/gemma4_31b_qb2/tt/generator_vllm.py index 943e2269be39..f80bbdc7e81b 100644 --- a/models/demos/gemma4_31b_qb2/tt/generator_vllm.py +++ b/models/demos/gemma4_31b_qb2/tt/generator_vllm.py @@ -2,9 +2,13 @@ # SPDX-FileCopyrightText: 2026 Tenstorrent USA, Inc. """Translate vLLM scheduler updates into the Gemma4 TP4 generator's inputs.""" +import time + import torch +from loguru import logger import ttnn +from models.demos.gemma4_31b_qb2.tt.decode_rows import decode_row_sizes, rows_covering, verify_shapes, verify_users from models.demos.gemma4_31b_qb2.tt.decoder import Decoder from models.demos.gemma4_31b_qb2.tt.generator import CacheState, build_generator @@ -13,11 +17,13 @@ class Gemma4ForCausalLM: _router = ttnn.FabricRouterConfig() _router.max_packet_payload_size_bytes = 8192 model_capabilities = { - "supports_prefix_caching": False, + "supports_prefix_caching": True, "supports_async_decode": True, "supports_sample_on_device": True, "supports_device_penalties": False, "supports_chunked_prefill": True, + "supports_spec_decode": True, + "spec_ids_are_sampled": True, "max_device_top_k": 32, "fabric_config": { "config": ttnn.FabricConfig.FABRIC_1D_RING, @@ -26,6 +32,26 @@ class Gemma4ForCausalLM: } decode_input_update_contract = 1 MAX_PREFILL_TOKENS = 8192 + # DRAM budget of the KV pool (QuietBox 2, per DRAM bank). The pool takes what start-up and the run time leave free: + # bank size (allocator-reported) - reserve - everything that is not KV at the end of start-up, per row of the wide + # backing. The reserve covers the run-time working set measured on the device (prefill activations, fragmentation: + # at least 545 MiB beyond start-up; a +2000-row pool with 650 MiB free died of an out-of-memory error on its first + # 24K prompt, +1200 rows with 915 MiB free ran every KV-pressure test) plus the 267 MiB safe minimum of the old stack. + DRAM_BANK_BYTES = 4_205_232_512 + DRAM_RESERVE_BYTES = 910 << 20 + # Weights, persistent buffers, traces and prefill programs at 32 slots, without KV: allocated per bank at the end of + # start-up (2699 MiB speculative profile, 2696 MiB plain, 4200 rows) minus the KV rows. The same constant results + # from a pool 1200 rows larger (3095 MiB), so it does not depend on the pool. + NON_KV_BYTES = 1305 << 20 + # One row of the wide KV backing per DRAM bank: 10 full-attention positions x K and V x [4, 128, 256] BF8 (17/16 bytes + # per value) spread over 8 banks. + KV_POSITIONS = 10 + KV_ROW_BYTES = KV_POSITIONS * 2 * 4 * 128 * 256 * 17 // 16 // 8 + # Free DRAM per bank that must remain after the decode traces exist; below it the first long prompts would run out + # of memory, so start-up stops instead. + MIN_FREE_AFTER_STARTUP_BYTES = 700 << 20 + # Draft length the launch settled on in spec_plan; 0 without speculation. + spec_k = 0 def __init__(self, generator, max_batch_size, max_seq_len, *, vllm_config=None): # vLLM inspects this keyword before the TT loader supplies a live mesh. @@ -35,9 +61,22 @@ def __init__(self, generator, max_batch_size, max_seq_len, *, vllm_config=None): self.mesh = generator.mesh self.max_batch_size = max_batch_size self.max_seq_len = max_seq_len + self.rows = max_batch_size + # Full-attention layers keep their KV in 256-token scheduler blocks packed into the shared wide rows (see + # allocate_kv_cache_per_layer and _tables). + self.full_layers = frozenset(i for i, layer in enumerate(generator.model.layers) if not layer.sliding) self.entry = None self.cache = None + # Decode traces of fewer rows over the same KV pool; a step runs the smallest one that covers its live rows. + self.row_sizes = decode_row_sizes(max_batch_size) + # Speculative verify traces: draft columns ride in extra decode rows, column j of user u is row j * users + u. + # One trace per (users, draft length) of decode_rows.VERIFY_K_BY_USERS; a step with more live requests is a plain decode. + self.verify_shapes = verify_shapes(max_batch_size, self.spec_k) if self.spec_k else () + self.verify_caches = {} + self.sub_caches = {} + self.cur_state = None self.prefill_slot_keys = {} + self.next_prefill_key = 0 def embed_input_ids(self, input_ids): raise RuntimeError("Use the TT prefill_forward/decode_forward interface") @@ -56,6 +95,12 @@ def initialize_vllm_model( raise ValueError("Gemma4 QB2 requires TP4, DP1, and batch 1..32") if not 128 <= max_seq_len <= 262144 or max_seq_len % 128: raise ValueError("Serving context must be 128..262144, aligned to 128-token pages") + if max_seq_len % 1024: + # Chunked prefill attention reads its start from a tensor (flexible chunked SDPA), which needs the page-table row + # (max_seq_len / 128 int32 entries) to be a multiple of 32 bytes: a multiple of 8 pages. + raise ValueError( + "Serving context must be a multiple of 1024 tokens (a page-table row of whole 32-byte groups)" + ) if hf_config.text_config.hidden_size != 5376 or hf_config.text_config.num_hidden_layers != 60: raise ValueError("Expected the Gemma4 31B checkpoint") torch.set_num_threads(8) @@ -63,76 +108,211 @@ def initialize_vllm_model( @classmethod def get_kv_cache_spec(cls, vllm_config): - from vllm.v1.kv_cache_interface import FullAttentionSpec, SlidingWindowSpec + from vllm.v1.kv_cache_interface import FullAttentionSpec + from vllm_tt_plugin.kv_cache_anchor import AnchoredSlidingWindowSpec - if vllm_config.cache_config.enable_prefix_caching: - raise ValueError("Gemma4 QB2 does not implement prefix caching") if vllm_config.max_in_flight_tokens > 2 * cls.MAX_PREFILL_TOKENS: raise ValueError("Gemma4 QB2 supports at most 8192 batched tokens with async chunked prefill") config = vllm_config.model_config.hf_text_config block = vllm_config.cache_config.block_size if block != 128: raise ValueError("Gemma4 QB2 uses 128-token cache pages") - # Equal accounting page sizes preserve the logical block size across - # hybrid groups. Different native geometries still use separate pools. + # Equal accounting page sizes: a sliding block is 128 tokens of 16 x 256 values and a full-attention block is 256 + # tokens of 4 x 512 values, the same 2 MiB (one wide row). The scheduler gives the full group 256-token blocks and + # the sliding groups 128-token blocks over one global ID space, so a full block owns one whole wide row where it + # used to leave half of a separate tensor idle; the hash block stays 128 tokens (cache_config.block_size). # The runner divides these global head counts by the TP degree. page_bytes = 2 * block * config.num_key_value_heads * config.head_dim * 2 + full_block = 2 * block + if 2 * full_block * config.num_global_key_value_heads * config.global_head_dim * 2 != page_bytes: + raise ValueError("A 256-token full-attention block must have the page size of a sliding block") result = {} for i, kind in enumerate(config.layer_types): sliding = kind == "sliding_attention" common = dict( - block_size=block, + block_size=block if sliding else full_block, num_kv_heads=config.num_key_value_heads if sliding else config.num_global_key_value_heads, head_size=config.head_dim if sliding else config.global_head_dim, dtype=torch.bfloat16, page_size_padded=page_bytes, ) result[f"model.layers.{i}.self_attn"] = ( - SlidingWindowSpec(**common, sliding_window=config.sliding_window) + AnchoredSlidingWindowSpec(**common, sliding_window=config.sliding_window) if sliding else FullAttentionSpec(**common) ) return result @classmethod - def get_max_tokens_all_users(cls, *, max_num_seqs, **kwargs): + def spec_plan(cls, vllm_config, max_num_seqs, requested_k): + """Speculation follows the number of live requests (decode_rows.VERIFY_K_BY_USERS); beyond that a step is plain. + + The draft cap above the last verify trace is 0, which needs the plugin's zero cap (k_by_rows entries may be 0). + """ + from vllm_tt_plugin.spec_decode import DRAFTER_STATE_INTERNAL, SpecPlan, SpecReject + + shapes = verify_shapes(max_num_seqs, requested_k) + if not shapes: + return SpecReject(reason=f"no verify trace fits {max_num_seqs} slots and K={requested_k}", supported_k=()) + cls.spec_k = shapes[0][1] + return SpecPlan( + effective_k=cls.spec_k, + lanes_per_request=1 + cls.spec_k, + extra_bytes_per_seq=0, + extra_bytes_per_token=0, + accept_modes=("argmax_ids",), + drafter_state=DRAFTER_STATE_INTERNAL, + supports_narrow_decode=True, + k_by_rows=tuple((users, k) for users, k in shapes) + + (((max_num_seqs, 0),) if max_num_seqs > shapes[-1][0] else ()), + ) + + @classmethod + def get_max_tokens_all_users(cls, *, max_num_seqs, max_model_len=262144, **kwargs): # The shared pool uses one full-attention group and five sliding groups. # Each window can straddle nine pages. Reserve both async batches of # in-flight prefill tokens globally, in addition to the per-request tails. + # The pool is one request of --max-model-len plus those windows, so a shorter context frees DRAM. + # vLLM checks that one request of the full context fits, and that request also keeps the eight pages of its + # prompt's cache-hit window (AnchoredSlidingWindowSpec): reserve them once. Many requests holding their + # windows at once wait for blocks in the scheduler instead of growing the pool. # The plugin separately adds one output page per request. - tails = Decoder.SLIDING_WINDOW_PAGES * Decoder.PAGE_SIZE * max_num_seqs - return 262144 + 5 * (tails + 2 * cls.MAX_PREFILL_TOKENS) + # An ID is one wide row: 128 sliding tokens or 256 full-attention tokens (allocate_kv_cache_per_layer), so the + # same number of IDs holds twice the full-attention tokens; the first term stays max_model_len / 128 IDs. + tails = ( + Decoder.SLIDING_WINDOW_PAGES * Decoder.PAGE_SIZE * max_num_seqs + + (Decoder.SLIDING_WINDOW_PAGES - 1) * Decoder.PAGE_SIZE + ) + needed = max_model_len + 5 * (tails + 2 * cls.MAX_PREFILL_TOKENS) + # The rest of the DRAM budget becomes cache: more full-attention history stays resident (the plugin adds one + # output page per request on top of this count). + budget_rows = (cls.DRAM_BANK_BYTES - cls.DRAM_RESERVE_BYTES - cls.NON_KV_BYTES) // cls.KV_ROW_BYTES + return max(needed, (budget_rows - max_num_seqs) * Decoder.PAGE_SIZE) + + @classmethod + def check_free_dram(cls, free_bytes_per_bank): + """Start-up stops when the traces left too little DRAM for the run-time working set.""" + if free_bytes_per_bank < cls.MIN_FREE_AFTER_STARTUP_BYTES: + raise RuntimeError( + f"Only {free_bytes_per_bank >> 20} MiB of DRAM per bank is free after start-up " + f"(at least {cls.MIN_FREE_AFTER_STARTUP_BYTES >> 20} MiB are needed for prefill); " + "lower --max-num-seqs or --max-model-len" + ) + + @staticmethod + def _full_view(wide): + """The full-attention view of a wide sliding-geometry backing: [P, 4, 128, 256] -> [2P, 1, 128, 512], zero-copy.""" + views = [] + for root in wide: + shape = tuple(int(x) for x in root.shape) + if ( + len(shape) != 4 + or shape[1:] != (4, 128, 256) + or root.dtype != ttnn.bfloat8_b + or root.layout != ttnn.TILE_LAYOUT + ): + raise ValueError(f"Unsupported wide KV backing: {shape} {root.dtype} {root.layout}") + view = ttnn.experimental.view(root, [2 * shape[0], 1, 128, 512]) + if ( + view.buffer_unique_id() != root.buffer_unique_id() + or view.buffer_address() != root.buffer_address() + or view.buffer_aligned_page_size() != root.buffer_aligned_page_size() + ): + raise RuntimeError("The full-attention view must alias the wide backing") + views.append(view) + return tuple(views) def allocate_kv_cache_per_layer(self, per_layer_specs): + """One wide backing per pool position; its full-attention layer reads it as 2P half-pages. + + A pool position holds one full-attention layer and its sliding layers. The scheduler hands every block ID to one + group at a time, so they can share memory: the backing has the sliding layers' geometry [P, 4, 128, 256] (P = IDs + + scratch rows) and the full layer sees it as [2P, 1, 128, 512] through a zero-copy view. A full-attention + block (256 tokens) owns the whole wide row b, native half-pages 2b and 2b + 1 (_tables); null block 0 stays unused. + """ pools, kv, scratch = {}, [], {} - for i, (layer, (shape, dtype, pool_id)) in enumerate(zip(self.generator.model.layers, per_layer_specs)): - if tuple(shape[1:]) != (layer.kv_heads, 128, layer.head_dim) or dtype != torch.bfloat16: - raise ValueError(f"Unexpected allocator geometry or accounting dtype: {shape}, {dtype}") - key = (pool_id, tuple(shape[1:])) - if key not in pools: - pools[key] = layer.allocate_cache(physical_pages=shape[0] + self.max_batch_size) - kv.append(pools[key]) - scratch[i] = list(range(shape[0], shape[0] + self.max_batch_size)) - if len(kv) != len(self.generator.model.layers): + scratch_rows = max([self.rows] + [users * (1 + k) for users, k in self.verify_shapes]) + layers = self.generator.model.layers + if len(per_layer_specs) != len(layers): raise ValueError("The allocator must supply one spec per layer") - tables = { - i: torch.zeros(self.max_batch_size, self.max_seq_len // 128, dtype=torch.int32) for i in range(len(kv)) - } + members = {} + for i, (layer, (shape, dtype, pool_id)) in enumerate(zip(layers, per_layer_specs)): + expected = (layer.kv_heads, 128 if layer.sliding else 256, layer.head_dim) + if tuple(shape[1:]) != expected or dtype != torch.bfloat16 or (i in self.full_layers) == layer.sliding: + raise ValueError(f"Unexpected allocator geometry or accounting dtype: {shape}, {dtype}") + members.setdefault(pool_id, []).append((i, layer, int(shape[0]))) + for pool_id, group in members.items(): + blocks = {n for _, _, n in group} + sliding = [layer for _, layer, _ in group if layer.sliding] + if len(blocks) != 1 or not sliding or sum(not layer.sliding for _, layer, _ in group) != 1: + raise ValueError( + "A pool position needs one full-attention layer and sliding layers with equal block counts" + ) + n = blocks.pop() + if n <= 1: + raise ValueError("A pool must contain the null block and live blocks") + wide = sliding[0].allocate_cache(physical_pages=n + scratch_rows) + pools[pool_id] = (wide, self._full_view(wide)) + for i, (layer, (shape, dtype, pool_id)) in enumerate(zip(layers, per_layer_specs)): + wide, view = pools[pool_id] + kv.append(wide if layer.sliding else view) + first = shape[0] if layer.sliding else 2 * shape[0] + scratch[i] = list( + range(first, first + (1 if layer.sliding else 2) * scratch_rows, 1 if layer.sliding else 2) + ) + tables = {i: torch.zeros(self.rows, self.max_seq_len // 128, dtype=torch.int32) for i in range(len(kv))} self.cache = CacheState( - kv, tables, self.max_batch_size, self.max_seq_len, scratch_pages=scratch, vllm_owned=True + kv, + tables, + self.rows, + self.max_seq_len, + scratch_pages={i: scratch[i][: self.rows] for i in scratch}, + vllm_owned=True, ) + self.sub_caches = { + n: CacheState( + kv, + {i: torch.zeros(n, self.max_seq_len // 128, dtype=torch.int32) for i in range(len(kv))}, + n, + self.max_seq_len, + scratch_pages={i: scratch[i][:n] for i in scratch}, + vllm_owned=True, + ) + for n in self.row_sizes + } + self.verify_caches = { + users: CacheState( + kv, + {i: torch.zeros(users * (1 + k), self.max_seq_len // 128, dtype=torch.int32) for i in range(len(kv))}, + users * (1 + k), + self.max_seq_len, + scratch_pages={i: scratch[i][: users * (1 + k)] for i in scratch}, + vllm_owned=True, + columns=1 + k, + ) + for users, k in self.verify_shapes + } return self.cache def _tables(self, cache, page_tables_per_layer, *, slots=None): - if cache is not self.cache: + if cache is not self.cache and all( + cache is not sub for sub in (*self.sub_caches.values(), *self.verify_caches.values()) + ): raise ValueError("Serving must use the cache allocated by vLLM") if page_tables_per_layer is None or len(page_tables_per_layer) != len(cache.kv): raise ValueError("Hybrid attention requires a page table for every layer") result, translated = {}, {} for i, source in enumerate(page_tables_per_layer): # Share immutable host rows only within this update: vLLM mutates - # its source tensors in place between scheduler steps. - key = id(source) + # its source tensors in place between scheduler steps. Layers of one group may be separate views of + # one table. + key = ( + source.data_ptr(), + source.storage_offset(), + tuple(source.shape), + source.stride(), + i in self.full_layers, + ) if key not in translated: dest = torch.zeros_like(cache.page_tables[i]) rows, cols = source.shape @@ -143,16 +323,40 @@ def _tables(self, cache, page_tables_per_layer, *, slots=None): raise ValueError("Page table has fewer rows than prefill requests") # Prefill rows are compact, then padded to the decode batch. # Scatter only the live prefix into its assigned state slots. - dest[indices, :cols] = source[: len(indices)].to(torch.int32).clamp_min(0) + live = source[: len(indices)].to(torch.int32).clamp_min(0) + if i in self.full_layers: + live = self._half_pages(live, dest.shape[1]) + dest[indices, : live.shape[1]] = live + if slots is None: + for column in range(1, cache.columns): + dest[column * rows : (column + 1) * rows] = dest[:rows] translated[key] = dest result[i] = translated[key] return result - def _sampling(self, params, batch, positions, *, reset_seed=True): + @staticmethod + def _half_pages(blocks, width): + """Full-attention scheduler blocks (256 tokens) as native 128-token half-pages: block b is pages 2b and 2b + 1. + + The null block 0 maps to page 0 in both halves. The scheduler's table is as wide as for 128-token blocks; the + entries past width // 2 columns belong to blocks that cannot exist and must be empty. + """ + columns = width // 2 + if blocks[:, columns:].any(): + raise ValueError("A full-attention block table has live entries past its 256-token capacity") + blocks = blocks[:, :columns] + pages = torch.stack((2 * blocks, 2 * blocks + 1), dim=2) + return torch.where(blocks[:, :, None] > 0, pages, torch.zeros_like(pages)).reshape(blocks.shape[0], -1) + + def _sampling(self, params, positions, *, batch=None, reset_seed=True): + """Per-row sampling state for `positions`, one per row; a slot's params repeat in its draft columns.""" + batch = self.max_batch_size if batch is None else batch + def rows(name): value = getattr(params, name) value = value.tolist() if isinstance(value, torch.Tensor) else value - return list(value)[:batch] if isinstance(value, (list, tuple)) else [value] * batch + slots = list(value)[:batch] if isinstance(value, (list, tuple)) else [value] * batch + return slots * (len(positions) // batch) for name, default in ( ("presence_penalty", 0), @@ -162,7 +366,7 @@ def rows(name): ): if any(x != default for x in rows(name)): raise ValueError(f"The shared host sampler must handle {name}") - seeds = [0] * batch + seeds = [0] * len(positions) if reset_seed: modulus = 2**31 - self.generator.max_seq_len seeds = [ @@ -192,11 +396,14 @@ def prefill_forward( slots = list(range(len(lengths))) if empty_slots is None else list(empty_slots) tables = self._tables(kv_cache, page_tables_per_layer, slots=slots) self.entry = None - # Full-attention block zero identifies a live request across row moves. - full_layer = next(i for i, layer in enumerate(self.generator.model.layers) if not layer.sliding) - keys = [int(tables[full_layer][slot, 0]) for slot in slots] - self.prefill_slot_keys = {s: k for s, k in self.prefill_slot_keys.items() if s not in slots and k not in keys} - self.prefill_slot_keys.update(zip(slots, keys)) + # Prefix pages can be shared by several live requests. History belongs + # to a request's stable state slot, never to a shared physical page. + keys = [] + for slot in slots: + if slot not in self.prefill_slot_keys: + self.prefill_slot_keys[slot] = self.next_prefill_key + self.next_prefill_key += 1 + keys.append(self.prefill_slot_keys[slot]) outputs = self.generator.prefill_forward( tokens, page_table=tables, @@ -209,7 +416,7 @@ def prefill_forward( ) if sampling_params is None: return outputs - params = self._sampling(sampling_params, len(lengths), [n - 1 for n in lengths]) + params = self._sampling(sampling_params, [n - 1 for n in lengths], batch=len(lengths)) return self.generator.sample_prefill_outputs(outputs, kv_cache, params) def decode_forward( @@ -227,6 +434,9 @@ def decode_forward( reload_page_table=False, reload_sampling_params=True, reset_sampling_state=True, + slot_remap=None, + spec_mode=None, + num_valid_drafts=None, **kwargs, ): if "reset_batch" in kwargs: @@ -237,29 +447,136 @@ def decode_forward( raise ValueError("Resetting sampling state requires current tokens and positions") if kv_cache is not self.cache: raise ValueError("Serving must use the cache allocated by vLLM") + if spec_mode is not None: + return self._verify( + tokens, start_pos, kv_cache, page_tables_per_layer, sampling_params, num_valid_drafts, slot_remap + ) sample = sampling_params is not None + slot_positions = start_pos + tokens = self._pad_rows(tokens.reshape(-1, 1), 0) + start_pos = self._pad_rows(start_pos.reshape(-1), -1) + # The batch is front-packed, so the live rows are 0..n-1 and the smallest trace covering row n-1 runs the step. + # A change of n is a layout change, which always reloads the inputs, so a trace switch needs no state moved. + live = int((start_pos >= 0).nonzero().max()) + 1 if bool((start_pos >= 0).any()) else 1 + rows = rows_covering(live, self.row_sizes, self.rows) + state = self.sub_caches.get(rows, kv_cache) if reload_inputs: - tables = self._tables(kv_cache, page_tables_per_layer) - full_layer = next(i for i, layer in enumerate(self.generator.model.layers) if not layer.sliding) - done = {int(tables[full_layer][row, 0]) for row, position in enumerate(start_pos) if int(position) >= 0} - for key in done: - kv_cache.prefill_history.pop(key, None) - self.prefill_slot_keys = {slot: key for slot, key in self.prefill_slot_keys.items() if key not in done} - self.entry = self.generator.prepare_decode(tokens, start_pos, page_table=tables, kv_cache=kv_cache) - elif self.entry is None or self.generator.traces.get(id(kv_cache)) is not self.entry: + tables = self._tables(state, [table[:rows] for table in page_tables_per_layer]) + # A row entering decode no longer needs its private prefill tail. + # The plugin notifies slot moves after submission succeeds, so + # histories still use source slots when a decode row is gathered. + self._release_entering_decode(slot_positions, slot_remap) + self.entry = self.generator.prepare_decode( + tokens[:rows], start_pos[:rows], page_table=tables, kv_cache=state + ) + self.cur_state = state + elif ( + self.entry is None or state is not self.cur_state or self.generator.traces.get(id(state)) is not self.entry + ): + # Steps that keep the loaded inputs rely on the plugin reloading them after anything that changes the + # trace: a prefill (note_prefill_submitted; one that compiles programs retires every trace) and any + # change of the live rows (a different covering trace). This check turns a broken contract into an error. raise ValueError("A retired decode trace requires current scheduler inputs") elif reload_page_table: - tables = self._tables(kv_cache, page_tables_per_layer) - self.generator._tables(kv_cache, tables) + tables = self._tables(state, [table[:rows] for table in page_tables_per_layer]) + self.generator._tables(state, tables) if sample and (reload_sampling_params or reset_sampling_state): - params = self._sampling(sampling_params, self.max_batch_size, start_pos, reset_seed=reset_sampling_state) - self.generator.configure_sampling(batch=self.max_batch_size, **params) + params = self._sampling(sampling_params, start_pos, reset_seed=reset_sampling_state) + self.generator.configure_sampling(batch=self.rows, **params) self.generator.replay(self.entry, sample=sample) if not sample: return self.generator._host_logits(self.entry["logits"])[0, 0, : self.max_batch_size].unsqueeze(1) output = self.entry["tokens"] return self.process_decode_output_host(output, is_tokens=True) if read_from_device else output + def _release_entering_decode(self, positions, slot_remap): + """A row entering decode no longer needs its private prefill tail (every decode step reloads, so this is idempotent).""" + for row, position in enumerate(positions): + if int(position) >= 0: + self.release_request(row if slot_remap is None else int(slot_remap[row])) + + def _pad_rows(self, values, fill): + """Extend a [slots, ...] decode input to the trace's rows; the draft-column rows stay inactive.""" + padded = torch.full((self.rows, *values.shape[1:]), fill, dtype=values.dtype) + padded[: values.shape[0]] = values + return padded + + def _verify( + self, + tokens, + start_pos, + kv_cache, + page_tables_per_layer, + sampling_params, + num_valid_drafts=None, + slot_remap=None, + ): + """Verify a [slots, 1 + K] candidate block as one decode step: column j of user u is row j * users + u. + + The block holds the live requests in its first rows; the trace of decode_rows.VERIFY_K_BY_USERS that covers + them verifies the columns up to its draft length (the plugin caps a request's drafts to the same table). Every + row writes its K/V at its own position before attention reads, so a draft attends to the drafts before it + exactly as sequential decode steps would. Each row samples with its slot's parameters and seed + position, the + draw a plain decode at that position makes, so accepted tokens are the plain decode's tokens. The plugin caps + the drafts of a step with more live requests than the table covers at 0; such a block carries no draft. + """ + from vllm_tt_plugin.spec_decode import ACCEPT_MODE_ARGMAX_IDS, VerifyOutput + + slots, width = tokens.shape + if slots != self.max_batch_size or width != 1 + self.spec_k: + raise ValueError(f"Expected a [{self.max_batch_size}, {1 + self.spec_k}] candidate block") + live = int((start_pos[:, 0] >= 0).nonzero().max()) + 1 if bool((start_pos[:, 0] >= 0).any()) else 1 + ids = torch.full((slots, width), -1, dtype=torch.int32) + self._release_entering_decode(start_pos[:, 0], slot_remap) + users = verify_users(live, self.verify_shapes) + if users is None: + # More live requests than any verify trace covers: the plan's draft cap is 0, so no row carries a draft. The + # plugin still sends a verify block when a row committed several tokens on its last step (a model with + # candidate state resolves it then); this model has none, so the step is a plain decode. + if num_valid_drafts is None or bool((num_valid_drafts > 0).any()): + raise ValueError(f"No verify trace covers {live} live requests and the block carries drafts") + ids[:, 0] = self._decode_plain_ids( + tokens[:, 0], start_pos[:, 0], kv_cache, page_tables_per_layer, sampling_params + ) + return VerifyOutput(spec_mode=ACCEPT_MODE_ARGMAX_IDS, argmax_ids=ids) + state = self.verify_caches[users] + columns = state.columns + block_tokens, block_positions = tokens[:users, :columns], start_pos[:users, :columns] + positions = block_positions.t().reshape(-1) + flat_tokens = torch.where(positions >= 0, block_tokens.t().reshape(-1), 0).reshape(-1, 1) + tables = self._tables(state, [table[:users] for table in page_tables_per_layer]) + self.entry = self.generator.prepare_decode(flat_tokens, positions, page_table=tables, kv_cache=state) + self.cur_state = state + if sampling_params is None: + self.generator.configure_sampling(batch=state.batch, temperature=0) + else: + self.generator.configure_sampling( + batch=state.batch, **self._sampling(sampling_params, positions, batch=users) + ) + self.generator.replay(self.entry) + out = self.read_decode_output(self.entry["tokens"]) + out = ttnn.to_torch(out).reshape(-1)[: state.batch].to(torch.int32) + ids[:users, :columns] = out.reshape(columns, users).t() + return VerifyOutput(spec_mode=ACCEPT_MODE_ARGMAX_IDS, argmax_ids=ids) + + def _decode_plain_ids(self, tokens, start_pos, kv_cache, page_tables_per_layer, sampling_params): + """One plain decode step from host inputs (the column-0 tokens of a draftless block); returns each slot's next token.""" + live = int((start_pos >= 0).nonzero().max()) + 1 if bool((start_pos >= 0).any()) else 1 + rows = rows_covering(live, self.row_sizes, self.rows) + state = self.sub_caches.get(rows, kv_cache) + tokens = self._pad_rows(tokens.reshape(-1, 1), 0)[:rows] + start_pos = self._pad_rows(start_pos.reshape(-1), -1) + tables = self._tables(state, [table[:rows] for table in page_tables_per_layer]) + self.entry = self.generator.prepare_decode(tokens, start_pos[:rows], page_table=tables, kv_cache=state) + self.cur_state = state + if sampling_params is None: + self.generator.configure_sampling(batch=self.rows, temperature=0) + else: + self.generator.configure_sampling(batch=self.rows, **self._sampling(sampling_params, start_pos)) + self.generator.replay(self.entry) + out = self.read_decode_output(self.entry["tokens"]) + return ttnn.to_torch(out).reshape(-1)[: self.max_batch_size].to(torch.int32) + def read_decode_output(self, output, async_read=False): if isinstance(output, torch.Tensor): return (output, []) if async_read else output @@ -276,13 +593,45 @@ def process_decode_output_host(self, output, is_tokens=True): return ttnn.to_torch(output).reshape(-1)[: self.max_batch_size].long().reshape(-1, 1) def warmup_model_prefill(self, kv_cache, enable_trace, **kwargs): - # Each unseen logical prefill shape retires traces before compilation. - pass + # Every 32-token prompt length from position 0 is compiled here: a first-time length costs up to tens of + # seconds of program compilation in the middle of a request, and retires the decode trace. + # The plugin's two-phase warm-up calls this twice for the same cache; the second pass would repeat the whole + # lattice (minutes of prefill compute) for programs the cache already holds. + # The lattice never exceeds the cache: a --max-model-len below the prefill limit has fewer page-table columns and + # rejects longer prompts. + limit = min(self.MAX_PREFILL_TOKENS, kv_cache.capacity) + marker = (id(kv_cache), limit) + if getattr(self, "_prefill_warmed", None) == marker: + logger.info("Prefill length classes already warmed for this cache; skipping") + return + pages = limit // Decoder.PAGE_SIZE + tables = {i: torch.zeros_like(table) for i, table in kv_cache.page_tables.items()} + for table in tables.values(): + table[0, :pages] = torch.arange(1, pages + 1) + self.generator.warm_prefill_shapes(kv_cache, tables, limit) + self._prefill_warmed = marker def warmup_model_decode(self, kv_cache, enable_trace, **kwargs): - self.generator._tables(kv_cache, kv_cache.page_tables) - self.generator.configure_sampling(batch=self.max_batch_size, temperature=0) - self.entry = self.generator._entry(kv_cache) + # Every trace a step can pick exists before the first request: a missing one would be compiled and captured in + # the middle of a request, retiring the others. + states = [kv_cache, *self.sub_caches.values(), *self.verify_caches.values()] + for state in states: + self.generator._tables(state, state.page_tables) + self.generator.configure_sampling(batch=self.rows, temperature=0) + # The full trace first (the returned entry is the full batch's). + start = time.perf_counter() + self.entry = self.generator.prepare_entries(states)[0] + self.cur_state = kv_cache + view = ttnn.get_memory_view(self.mesh, ttnn.BufferType.DRAM) + used = view.total_bytes_allocated_per_bank + logger.info( + "Captured {} decode traces ({} rows) in {:.1f} s; DRAM allocated per bank {} MiB", + len(states), + [state.batch for state in states], + time.perf_counter() - start, + used >> 20, + ) + self.check_free_dram(view.total_bytes_free_per_bank) def note_state_slots_moved(self, moves): self.prefill_slot_keys = {moves.get(slot, slot): key for slot, key in self.prefill_slot_keys.items()} @@ -297,4 +646,5 @@ def release_persistent_capture(self): self.cache.prefill_history.clear() self.prefill_slot_keys.clear() self.generator.close() + self._prefill_warmed = None self.entry = None diff --git a/models/demos/gemma4_31b_qb2/tt/model.py b/models/demos/gemma4_31b_qb2/tt/model.py index 5bd8795283a8..ce90b1991e4f 100644 --- a/models/demos/gemma4_31b_qb2/tt/model.py +++ b/models/demos/gemma4_31b_qb2/tt/model.py @@ -21,6 +21,8 @@ HF_ID = "google/gemma-4-31B-it" HF_REVISION = "842da3794eaa0b77d5f08bae87a17459d91ff475" +# Storage format of the tied LM head (the decoder projections are in decoder.WEIGHT_DTYPES). +HEAD_DTYPE = ttnn.bfloat8_b def checkpoint_path(): @@ -97,7 +99,7 @@ def __init__(self, mesh_device, *, checkpoint=None, layer_indices=None): weight = reader.get("model.language_model.embed_tokens.weight") # The tied embedding has one physical layout for each consuming operation. self.embedding = self.tensor(weight, dim=1, layout=ttnn.ROW_MAJOR_LAYOUT) - self.head = self.tensor(weight.T.contiguous(), dim=1, dtype=ttnn.bfloat8_b) + self.head = self.tensor(weight.T.contiguous(), dim=1, dtype=HEAD_DTYPE) self.embedding_scale = torch.tensor(self.config.hidden_size**0.5, dtype=weight.dtype).item() norm = reader.get("model.language_model.norm.weight") self.norm = self.tensor(norm.reshape(1, 1, 1, -1)) @@ -194,12 +196,40 @@ def terminal(self, hidden): return logits def prefill_device( - self, tokens, *, sequence_length, page_tables, kv_cache, all_logits=False, start_pos=0, histories=None + self, + tokens, + *, + sequence_length, + page_tables, + kv_cache, + all_logits=False, + start_pos=0, + histories=None, + host_rows=None, + valid_length=None, ): + """sequence_length is the (page-padded) length of the chunk; valid_length (<= it) is its logical length. + + With host_rows (the host page-table row of each table key) the position-dependent ops take runtime values (see + Decoder.prefill_chunk_inputs) and the logits row is selected by a device-valued slice, so the programs depend only + on the padded chunk length. + """ if len(kv_cache) != len(self.layers): raise ValueError("Expected one cache pair per loaded layer") x = self.embed(tokens, sequence_length=sequence_length) + rope, fills = {}, {} for i, (layer, cache) in enumerate(zip(self.layers, kv_cache)): + inputs = None + if host_rows is not None: + key = i if i in page_tables else layer.kind + if layer.kind not in rope: + rope[layer.kind] = layer.prefill_chunk_inputs(sequence_length, start_pos) + # Layers of one KV group share their page-table row (one host tensor): upload the fill pages and the chunk start once + # per row, not once per layer. + row = id(host_rows[key]) + if row not in fills: + fills[row] = layer.prefill_chunk_inputs(sequence_length, start_pos, host_rows[key]) + inputs = [dict(f, rope=r["rope"]) for r, f in zip(rope[layer.kind], fills[row])] # This hidden stream is private to this invocation. Completed # chunks can reuse it once their input reads and KV fills finish. x = layer.prefill_forward( @@ -210,22 +240,53 @@ def prefill_device( start_pos=start_pos, history=histories[i] if histories is not None else None, return_history=histories is not None, + chunk_inputs=inputs, + padded_tails=host_rows is not None, ) if histories is not None: x, histories[i] = x if not all_logits: # An unaligned slice begin untilizes its entire input. Copy # the aligned final tile first, then select its logical row. - tile_start = (sequence_length - 1) // 32 * 32 - x = x[:, :, tile_start:sequence_length, :] - last_row = sequence_length - 1 - tile_start + valid = sequence_length if valid_length is None else valid_length + tile_start = (valid - 1) // 32 * 32 + last_row = valid - 1 - tile_start + if host_rows is None: + x = x[:, :, tile_start:sequence_length, :] + else: + # The 32-row group holding the last token, selected by a device-valued offset (one program for every + # logical length of this padded chunk length). + width = x.shape[3] + upload = lambda values: ttnn.from_torch( + torch.tensor(values, dtype=torch.int32), + dtype=ttnn.uint32, + layout=ttnn.ROW_MAJOR_LAYOUT, + device=self.mesh, + mesh_mapper=ttnn.ReplicateTensorToMesh(self.mesh), + ) + x = ttnn.slice( + x, + upload([0, 0, tile_start, 0]), + upload([1, 1, tile_start + 32, width]), + slice_dim=2, + num_devices=x.shape[2] // 32, + ) x = x[:, :, last_row : last_row + 1, :] return self.terminal(x) - def decode_device(self, tokens, positions, *, page_tables, kv_cache, batch, rope_positions=None): + def decode_device( + self, tokens, positions, *, page_tables, kv_cache, batch, rope_positions=None, update_positions=None + ): if len(kv_cache) != len(self.layers): raise ValueError("Expected one cache pair per loaded layer") x = self.embed(tokens, batch=batch) + # RoPE cos/sin rows depend only on the positions and the layer kind: look them up once per step. + rope = {} + for layer in self.layers: + if layer.kind not in rope: + rope[layer.kind] = layer.decode_rope_tables( + positions if rope_positions is None else rope_positions, batch + ) for i, (layer, cache) in enumerate(zip(self.layers, kv_cache)): x = layer.decode_forward( x, @@ -233,7 +294,9 @@ def decode_device(self, tokens, positions, *, page_tables, kv_cache, batch, rope page_table=page_tables[i] if i in page_tables else page_tables[layer.kind], kv_cache=cache, rope_positions=rope_positions, + rope=rope[layer.kind], cyclic_cache=i not in page_tables, + update_positions=update_positions, ) logits = self.terminal(x) # Common samplers own a 32-row tile; inactive padding stays internal.