Repository navigation
Gemma 4 31B on QuietBox 2: the serving implementation (prefix caching, BF8, speculation, packed KV pool) - #65783
Draft
mvasiljevicTT wants to merge 19 commits into
Draft
mvasiljevicTT wants to merge 19 commits into
mvasiljevicTT wants to merge 19 commits into
Conversation
Decode-step changes that keep every result the same (the K blocking that fixes matmul accumulation order is unchanged): - qkv and down DRAM-sharded decode matmuls read their input from 56 cores instead of 8; their output keeps the 8-core shards the next ops read (new DECODE_OUTPUT_CORES, independent of the input cores). - RoPE: cos/sin rows are looked up once per step per layer kind (decode_rope_tables) and applied with the fused rotary_embedding_hf decode op, instead of a typecast + pad + two embeddings in every layer and transpose + rotary_embedding + transpose. - Sliding-window decode SDPA at 32 rows runs on the full 11x10 grid. Tests: test_decoder (layers 0 and 5, prefill + live decode trace vs Hugging Face) passes; traced decode outputs and KV cache of layers 0 and 5 were bit-identical to the previous layer at 1, 8 and 32 rows in a one-off comparison script run on both builds; that comparison is not a committed test (the committed test_decoder compares with Hugging Face at PCC 0.995). Measured (QB2, vLLM 0.31 + vllm-tt-plugin main, main model BF4, 32-slot server, no prefix cache, greedy), on top of the DRAM-reader placement layer: decode step: 49.6 -> 43.0 ms at 32 rows, 43.6 -> 33.9 ms at 1 row 1 user ISL128/OSL1024: 33.9 -> 42.4 tok/s/user (+25%), TTFT 76 -> 77 ms 1 user, 28K-token prompt: 23.8 -> 29.0 (+22%) 4 / 8 / 32 users (ISL128/OSL256): 37.5 / 37.5 / 35.2 -> 45.4 / 45.2 / 40.7 tok/s/user (+21% / +21% / +16%); 32 users 883 -> 996 tok/s aggregate (+13%) Depends on the DRAM-reader placement layer for the 1-user numbers. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Agent loops resend the whole conversation every turn; with prefix caching vLLM reuses the completed history instead of prefilling it again. Sliding layers keep a BF16 K/V tail between chunks, and a cached start has no tail, so _restore_prefill_history gathers the preceding window from the request's own vLLM-owned (BF8) pages. Request history is keyed by a private state slot instead of full-attention block 0 (shared by requests that share a prefix) and released by the decode source slot when a row enters decode. Needs the vllm-tt-plugin state-slot fix (a continuing prefill keeps its device state slot when its batch row moves). Tests: test_chunked_prefill (cached continuation, sliding-layer PCC >= 0.995 vs Hugging Face at 128, 1024 and 1152 token boundaries) and the CPU tests in test_serving_inputs (state-slot keys, table ownership) pass. Measured (QB2, 32-slot server, BF4, replay of recorded SWE-bench agent trajectories, cache on = stable cache_salt, off = unique salt, same server): cached fraction of later turns 0.77-0.83. Wall time / mean TTFT of later turns / decode tok/s per user, on vs off: 1 agent 86 vs 100 s / 5.6 vs 8.2 s / 37.7 vs 37.7; 8 agents 139 vs 354 s (2.5x) / 4.4 vs 21.4 s / 24.5 vs 14.8; 32 agents 401 vs 604 s / 12.5 vs 14.7 s / 9.6 vs 5.6. A 28K-token prompt: cold TTFT 4.8 s, cached 0.9 s. Accuracy: a cached continuation is numerically close to, not bit-identical with, a cold prefill (the suffix is prefilled from a different chunk alignment). Greedy 128-token continuations of 8 recorded agent prompts: two cold prefills of the same prompt are identical (8/8); a cache hit diverges from a cold prefill on 7/8 prompts (first divergence at token 0-56), always where the top-2 logprob gap is <= 0.5 (near-ties). Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…dow spec Prefix caching needs the window that ends at a prompt's last hit boundary to survive a long reply; the stock sliding-window manager frees it first. The sliding groups now return AnchoredSlidingWindowSpec from vllm-tt-plugin (registered through vLLM's KVCacheSpecRegistry, so vLLM stays unmodified), which keeps that window until the request finishes. No model or kernel change; needs the plugin's kv_cache_anchor module. Tests: test_decoder (sliding and full layers) and test_chunked_prefill pass on this commit; the policy itself is covered by the plugin's tests/test_kv_cache_anchor.py (real vLLM 0.31 block pool and managers). Measured (QB2, 32-slot server, BF4, replay of recorded SWE-bench trajectories with this layer and the prefix-cache layer): see the prefix-cache layer; with both, the cached fraction of later turns is 0.77-0.83. The 1-slot server needed the admission cap to count the anchor pages (get_max_tokens_all_users reserves them once). Earlier experiment with the same policy as a vLLM-core patch: Terminal-Bench 89 input found in cache 55% -> 92%, prefill time -70%. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The qkv, o, gate/up and down projections are stored as BF8 instead of BF4 (decoder.WEIGHT_DTYPES; the tied LM head was already BF8, model.HEAD_DTYPE). There is no switch: the format is a constant of the model. Why BF8: accuracy on agent traffic. BF4 projections shift the next-token distribution enough to break long agent runs (malformed tool calls, repeated channels); earlier SWE-bench Verified runs solved 52-55% with BF4 and 57-61% with BF8. Cheaper mixes were measured against all-BF8 on 32 recorded agent prompts (47,493 positions) and a 17,228-token prompt, full 60 layers, QB2: BF8 attention/down/head + BF4 gate/up KL 1.15 (top-1 agreement 0.70); BF8 attention/gate-up/head + BF4 down KL 1.14 (0.71); all-BF4 KL 2.06 (0.59). None keeps the BF8 greedy continuation (14 of 16 prompts diverge within 20 tokens), so all four projections stay BF8. BF8 weight tiles are twice the size of BF4 tiles: a 6656- or 8192-row prefill chunk overflowed L1 in the matmul circular buffers (server crash on an 8936-token prompt). Chunks above 4096 rows use a K block of at most 4 for non-BF4 weights; only the accumulation split changes. Tests: test_decoder (layers 0 and 5, prefill + live decode trace vs Hugging Face, PCC >= 0.995) now runs with every projection in BF4 and in BF8. Chunk sizes 6656, 8192 and 9216 rows run (6656 failed before). Measured (QB2, layer-level and full model, plain decode, 1 row): 50.3 -> 36.4 tok/s at 128 tokens of context (45.2 -> 33.6 at 28K); 32-row step 23.7 -> 31.3 ms; +467 MB weight DRAM per bank. On its own this layer does not leave room for the 32-slot KV pool at 262144 tokens: the next layers (one weight copy, pool budget) make that fit. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Prefill and decode each kept their own copy of every projection matrix (qkv, o, gate/up, down). Prefill now rebuilds its interleaved weights from the decode DRAM-sharded copy when a chunk needs them (layout move and slices only, same values) and frees them after the matmul. A layer keeps one copy of its weights instead of two. Why: with BF8 weights (previous layer) 32 slots with the full 262144-token context did not fit: the KV pool ran out of DRAM at start-up, and at --max-model-len 131072 a 32-agent replay ran out at run time. This frees about 0.46 GB per DRAM bank. Tests: test_decoder (PCC vs Hugging Face, BF4 and BF8) and test_chunked_prefill pass on this commit. Measured (QB2): prefill logits and KV cache of layers 0 and 5 are bit-identical to the two-copy code at 1024, 2048 and 4096 tokens. Cost: warm prefill +2 ms per layer per chunk (24.3 -> 28.6 ms for 2 layers at 4096 rows). Server (32 slots, max-model-len 262144, BF8): starts, pool 397,044 tokens, 32-agent replay completes (wall 208 s, cached fraction 0.81, mean TTFT 3.9 s); 32-user forced decode 27.7 tok/s/user. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
get_max_tokens_all_users reserved a full 262144-token request whatever context the profile serves. It now reserves max_model_len plus the sliding windows, so a profile with a shorter context frees DRAM for weights and prefill temporaries. The default (262144) is unchanged. Tests: CPU test in test_serving_inputs (pool size for 262144 and for a shorter context). Measured (BF8, 32 slots): --max-model-len 262144 gives a 397,044-token pool; 131072: 238,012; 65536: 141,138. Shorter contexts cost cache capacity: at 65536 the 32-agent replay cached fraction fell from 0.81 to 0.32. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…de traces only when programs were compiled A prefill of a prompt length the program cache has not seen compiles new programs (up to tens of seconds for the first of a tile-padded shape) and, as before, retires the decode traces because new programs may leave persistent device state a captured trace overlaps. Previously every new prefill signature retired the traces before running. Now a prefill retires them only if the prefill-and-sample call grew the program cache (the row merge of sample_prefill_outputs, whose concat/pad programs compile per batch size, used to compile outside the count; it is counted now and every batch size 2..32 is compiled at start-up), and warmup_model_prefill compiles every 32-token prompt length from position 0 up to the prefill limit. The decode trace survives prefills whose signature the program cache already holds. It does not survive a never-seen signature: continuation chunks (start > 0) and every new (length, start) still compile 12-47 programs (0.1-0.8 s) and retire the traces; the last layer of this stack (continuation prefill without per-signature programs) removes that. The plugin's two-phase warm-up called warmup_model_prefill twice for the same cache and repeated the whole lattice (3 min 37 s of the server log); the second call is skipped. The lattice is limited to the cache capacity (a --max-model-len below 8192 has fewer page-table columns and rejects longer prompts). A program compiled in the sampling step retires the traces before the sampling trace runs (the entry is rebuilt and the merged rows copied again), not after. Tests: tests/test_prefill_warmup.py (layers 0 and 5): warm-up leaves no new compilation for warmed lengths, the trace survives them, and greedy decode is identical with the kept trace, after other prefills, and after a forced trace retire. CPU tests: warm-up call list including every prefill batch size, serving warm-up limited to the capacity, the second warm-up of one cache skipped, program counting in sample_prefill_outputs (no growth: no retire; growth: retire before the sampling trace executes). Measured (QB2): first prefill of a new length 39.8 s cold; after warm-up of 2048 tokens (141 s for 2 layers) lengths 320/1120/1920 take 6-11 ms of host-side prefill time; an off-lattice length (1000) takes 0.67 s. On the full model the 256 lengths up to 8192 add about 7.5 minutes to start-up. Lattice choice (40 random lengths 130-2040, cold kernel cache, 2 layers, first-use TTFT): every 32 tokens: median 0.95 s, worst 1.5 s, none above 3 s; every 128 tokens: median 1.23 s, one 6.7 s stall in 39 (a stall also retires every active user's decode trace); no warm-up: stalls of 3-9 s on 28% of lengths. A length off the lattice can still compile a few programs (about 1 s typically), because programs depend on the exact length. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…ompiling a program per id A cache-hit continuation restores the preceding sliding window from the request's KV pages with one slice per physical page, cache[p : p + 1]. The page id is part of the slice's program hash, so every request that received new block ids from vLLM compiled new slice programs, and a prefill that compiles programs retires every decode trace (new program buffers allocated while traces exist are unsafe for them: the trace allocation tracker reports them). Almost every call of an agent loop retired the traces and paid a recapture. The page id is now a runtime value: ttnn.slice with device start and end tensors (slice_dim=0, num_devices=pages) reads any page with one program per cache shape. The start and end tensors are uploaded once per page id and reused by all sliding layers and both caches. Tests: tests/test_history_restore.py: the restored history is bit-identical to the previous Python-slice implementation, and restores with other block ids add no programs. Measured (QuietBox 2, BF8, 1-slot server, replay of recorded SWE-bench trajectories, warm pass then identical measured pass): programs added per continuation prefill in the measured pass 40, 40, 15, 20, 40, 15, 40 -> 0 on all 8 prefills; trace retires 7 of 8 -> 0 (captures in the run: 10, all at start-up); later-turn TTFT mean 0.66 s -> 0.37 s (p90 0.68 -> 0.40 s). A first-time (length, start) signature still compiles 14-25 programs once (was 30-63) and retires the traces. Accuracy: bit-identical (see Tests). Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…rows Also (the capture machinery of this layer): Gemma4Generator._entry is split into _new_entry / _warm_entry / _capture_entry and prepare_entries(states) captures one decode trace per cache state; all persistent inputs are allocated before any trace is captured, so no persistent tensor lands on memory a captured trace reuses for intermediates. What: a server keeps decode traces of 1, 4, 8, 16 and 32 rows (those below its slot count, plus the full one) over the same KV pool. Each decode step runs the smallest trace with at least last-live-row + 1 rows. tt/decode_rows.py holds the sizes and the choice. The vLLM batch is front-packed (live requests occupy rows 0..n-1), and any change of the live rows is a layout change that already reloads tokens, positions, page tables and sampler seeds, so a trace switch moves no state. Why: rows without a request still cost device time, and the full-attention SDPA of one live row is 8-10x slower in a 32-row trace than in a 1-row trace at long context. One user on a 32-slot server decodes at the speed of a 1-slot server instead of 11-62% slower than it (device wait 21.7 / 26.4 / 34.5 / 68.6 ms at 128 / 4K / 28K / 130K context with 32 rows against 19.5 / 20.8 / 21.5 / 24.7 ms with 1 row). Traces specialised by context bucket (page-table horizon 4K / 32K / 262144) gave no gain: the step changes by <= 0.5% at 1-4 rows, so only row count is used. Tests: host tests for the choice (test_decode_rows_host.py, 21 cases); layer-level test of 1/4/8/16-row decode against the 32-row decode on a sliding and a full layer, PCC >= 0.998 (test_decode_rows_layer.py: worst 0.99873 and 0.99987). Server runs on this branch, BF8, 32 slots, 262144 tokens: one-user decode tok/s/user at 128 / 4K / 28K: fixed 32-row 33.82 / 29.40 / 23.76, this layer 36.82 / 35.27 / 34.41 (+8.9% / +20.0% / +44.8%), equal to the 1-slot server. 8-user replay of recorded agent trajectories (mixed arrivals and departures, trace switches): 21.2 tok/s/user against 21.6 for the fixed server (within noise). Start-up: capturing the 5 traces takes 24.5 s the first time and 1.4 s with kernels cached (+70 s to a healthy server); DRAM allocated per bank 3407 MiB of a 4010 MiB bank (about 600 MiB left); the 512 MiB trace region holds the five traces (its usage is not readable from Python). Faithfulness and accuracy: a lone request on this server gives bit-identical greedy and seeded outputs (8 real prompts x 128 tokens each) to a 1-slot server, which runs the same 1-row trace. Outputs are not bit-identical across row counts (the decode SDPA and the sharded memory configs depend on the row count), so they can differ slightly with the number of active users, exactly as they already do between the 1-slot and 32-slot profiles (greedy identical on 1 of 8 prompts, seeded on 0 of 8 between those two profiles today). On the teacher-forced decode path over 32 recorded agent prompts (6,630 positions, last 256 tokens of each), KL(32-row || N-row) over the top 64 tokens is 0.70 / 0.62 / 0.63 for 1 / 4 / 8 rows (0.19 / 0.18 / 0.18 on model-turn positions), top-1 agreement 0.83-0.84, and the mean NLL of the true tokens moves by <= 0.03 (4.405 for 32 rows; 1 row is the lowest, 4.391). Every size is equally far from the prefill answer for the same positions (KL 0.61-0.62, top-1 0.81), so the row count adds noise of the size of the decode-vs-prefill difference, not an accuracy loss; the BF4 weight mixes lose 0.14-0.97 NLL. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
… log the prefill that retires them What: a decode entry's persistent inputs (tokens, positions, logits, update positions) are kept in Gemma4Generator.entries when a prefill retires the traces. The next decode recaptures only the trace it needs, and capturing one trace no longer retires the others. Why: with several decode traces, retiring on every capture made a server that switches between traces capture on every step after a retire (903 captures in one 8-call replay of the speculative server: a capture per verify/plain switch). Once the entry's persistent inputs exist a capture allocates only transient tensors, so another trace stays valid; a new entry and a new prefill signature still retire all traces first, as before. Tests: server runs (see the previous commit): 8-user replay 21.2 tok/s/user against 19.9-20.3 before this change. Also: a prefill that compiles programs and retires the traces logs its prompt lengths, start positions and the number of new program-cache entries (at debug level: a per-request diagnostic of the serving path); in a replay of recorded agent trajectories every continuation prefill added 15-63 programs, which is why each call retired the traces. Review fixes folded in: the layer test passes rope_positions clamped at 0 for its inactive rows (the rope lookup casts the index to uint32), and the retire line is at debug. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
… matmul The decode matmul of the attention output projection (o) read its weights with two workers per DRAM bank. A sweep of the geometry (8-64 input cores, 1-4 readers per bank, K block unchanged) on layers 0 and 5 found one reader per bank fastest for both layer kinds. The K block fixes the accumulation order and the input cores and readers only move data, so the results do not change. qkv, down and gate/up already run in their best geometry (sweeps: no gain). Tests: tests/test_decode_geometry.py (sliding and full layer): the o projection output with one and with two readers per bank is bit-identical; test_decoder passes. Measured (QuietBox 2, BF8, op-level real-time profiler, 1 row): o matmul 35.4 -> 28.4 us (sliding layer) and 67.0 -> 50.9 us (full layer). Full model, traced decode: 1 row at 128 tokens of context 36.44 -> 37.11 tok/s (+1.8%), at 28K 33.66 -> 34.25 (+1.7%); 32-row step 31.28 -> 30.79 ms at 128 tokens, 43.34 -> 42.85 ms at 4K. Accuracy: bit-identical (see Tests). Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…t trace for the live requests What: the model accepts the vLLM plugin's host n-gram drafts and verifies them as a multi-column decode step. Column j of user u is row j * users + u; every row writes its K/V at its own position before attention reads and samples with its slot's parameters and seed + position, so the accepted tokens are the tokens a plain decode produces. Rows that write into one cache tile (a user's draft columns) are updated by separate paged_update_cache calls (concurrent writers to one tile lose updates); draft columns are a property of the cache state (CacheState.columns), so plain traces pay no extra cache updates. Page-table translation is de-duplicated by storage address. Speculation follows the number of live requests (VERIFY_K_BY_USERS in tt/decode_rows.py): one request drafts K=5 (a 6-row verify trace), up to four draft K=3 (a 16-row trace), more run plain decode; spec_plan publishes this as the plugin's draft cap per live rows (k_by_rows, a cap of 0 above four requests, which needs the plugin cap-zero change). All plain and verify traces are captured at start-up. A verify block with no draft (the plugin sends one after a multi-token commit) runs as a plain decode; a block with drafts that no verify trace covers is an error. Rows entering decode release their private sliding-history tails in verify steps as in plain steps. Whether speculation is on is a profile setting (speculative_config in the TTI entry), not a model flag. Why: speculation pays for one user (+40% on a replay of recorded agent trajectories, K=5) and four (+22%, K=3, NUMBA_NUM_THREADS=1) and loses beyond that (0.98x at 8, 0.84-0.87 at 8-16 slots with the old fixed trace), so it follows the active requests instead of the slot count. Tests: host tests with a stubbed generator (spec_plan and its class attribute, verify layout and answers per slot, fewer users than slots, draftless block above the verify traces, drafts there refused, plain-decode trace choice, release of tails, warm-up captures every trace); tests/test_decode_columns.py (sliding and full layer): draft columns equal the same decode steps run one by one at the same row count. Whole host suite of the layer. Measured (QB2, BF8, 32 slots, K=5, sync, NUMBA_NUM_THREADS=1, replay of recorded agent trajectories, tok/s/user): 1 / 4 / 8 agents 49.3 / 37.0 / 24.3 against 35.3 / 30.3 / 24.8 for the plain adaptive server (1.40x / 1.22x / 0.98x); one user, random-token prompt 178 / 166 / 155 at 128 / 4K / 28K context. Exactness: speculation vs plain with every step in the same 6-row layout, 8 prompts x 128 tokens: greedy 8/8 and seeded 8/8 identical. Against the 1-row plain server the streams differ like any two row counts (the row-count noise stated in the adaptive-rows layer). Limits: synchronous scheduling only (vLLM refuses async scheduling with n-gram), so many agents lose a little against the async plain server (replay on the final stack, tok/s/user: 8 agents 25.2 vs 25.9, 32 agents 12.3 vs 12.8, i.e. -3..-4%; 1 and 4 agents +34% / +20%): serve speculation for few users only; NUMBA_NUM_THREADS=1 is required. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
What: every op of a prefill chunk whose program hash held the position or the length takes a runtime value, and the prompt is padded to whole cache pages, so the programs depend only on the padded chunk length: - rope cos/sin rows are gathered by position ids (ttnn.embedding on the table, tile layout) instead of sliced by [start:start+n]; - the cache-fill page ids are uploaded as a row (the host knows the page table) instead of a Python slice of the device table; layers that share a page-table row share the upload; - chunked SDPA reads its start from a one-element tensor (chunk_start_idx_tensor) instead of the Python chunk_start_idx; - the 32-row group of the last token is a tensor-argument slice (device-valued offset); the row inside that group is still a Python-valued slice (32 values, all compiled at start-up); - the prompt is padded to 128 on the host (no per-length pad), the saved sliding tails end at the padded end and the continuation rewinds to its aligned start (history tail_end). A prefill that returns all logits (return_all_logits: tests and evaluation) still runs the per-length path. warm_prefill_shapes now warms every 128-token padded length, the 32 last-token rows at both residual widths, for every padded length a continuation after a full sliding window (aligned and unaligned previous end), and in the serving cache the prefix-cache history restore for every page count (the first prefix hit after start-up used to compile slice/concat/typecast programs once and retire the traces). The flexible chunked SDPA needs the page-table row (max_seq_len / 128 int32 entries) to be a multiple of 32 bytes, so the serving context must be a multiple of 1024 tokens (checked in initialize_vllm_model; 262144 and 131072 qualify). The per-start pref_sdpa_selections record is keyed by the chosen chunk instead of the start (it grew without bound). Why: a (length, start) never seen before compiled 12-47 programs (measured, 2 layers) and retired the decode traces, costing about 0.76 s per call at one slot (0.35-0.5 s per trace recapture, three traces on the speculative adaptive server). Live traffic produces new signatures constantly: this was the largest remaining cost of an agent loop. Tests (layers 0 and 5 on the device): tests/test_continuation_programs.py: after warm-up, never-seen (length, start) pairs (unaligned lengths, page-aligned starts, one and several chunks) add 0 programs; at layer level (sliding and full, three continuations each) the chunk with runtime-valued inputs padded to whole pages gives the same layer output as the per-length chunk, bit for bit; and chunk chains with unaligned starts (1000-2100-3000, 300-1500-2560-3500, 2600-3500) give last-row logits with PCC >= 0.9999 against the all-logits path and equal KV rows of the last window in every layer (the logits differ only by the head's matmul shape). CPU: warm-up call list, the context check (test_serving_inputs.py, test_prefill_warmup.py). Probe evidence (scripts in the PR draft, not committed): l13_probe.py runs one sequence of prefills (aligned and unaligned starts, one and several chunks) after warm-up and records the programs each added and the last-token logits; l13_compare_logits.py compares the logits of two builds with torch.equal. Against the previous layer: 24 novel signatures at 12 starts add 0 programs (previous layer: 12-47 per first call of a start, 2-20 per second call); the 38 prefills both builds ran have bit-identical logits (one-row head). The same probe was re-run on the final tree of this stack: 24 novel signatures add 0 programs, the 38 prefills shared with the previous run are bit-identical to the previous layer's logits (l13_compare_logits.py: compared 38, differing none), and prefill batch sizes 2, 3, 5 and 8 through sample_prefill_outputs add 0 programs; the device tests of the changed files pass (28). Limits: a continuation after fewer than 1024 sliding tokens (a history of 128..896 rows: a chunked prompt whose first chunk was short, or a prefix hit at a short aligned start) and a continuation chunk of more than max_tokens - 1152 padded rows (7041..8320 rows at the 8192-token limit; an unaligned start adds up to a page) are other signatures: each compiles once and retires the traces. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…r a shared wide backing
What: a pool position holds one full-attention layer and five sliding layers. They share one BF8 backing of wide rows with the sliding
geometry [P, 4, 128, 256] (P = block IDs + scratch rows); the full-attention layer reads it as [2P, 1, 128, 512] through a zero-copy
ttnn.experimental.view. The scheduler gives the full-attention group 256-token blocks (the same 2 MiB page as a 128-token sliding
block) and the sliding groups 128-token blocks over one global ID space; every ID is owned by one group at a time (vLLM's block pool),
so a full block owns a whole wide row = native half-pages 2b and 2b + 1 (_tables expands the full-attention tables; the null block stays
0). Kernels are unchanged. The pool IDs are unchanged (get_max_tokens_all_users): each row now holds 256 full-attention tokens instead
of 128, and the separate full-attention tensors (10 pools x n x 65,536 x 1.0625 B x 2 = about 725 MB per DRAM bank at n = 4168) are gone.
Prefix hits of the full-attention group are 256-aligned (at most 255 recomputed tokens instead of 127). No flag: the layout is fixed.
Why: with 32 agents at 25-100K tokens the pool (~410K tokens left for full histories after the sliding windows of 21 running requests) is
exhausted: prefix-cache hit rate fell from 60% to 14% and decode to 3 tok/s/user in the Harbor-32 smoke. The original effort measured this
mechanism ("pack2") as the fix: 825K full-attention tokens in the backing that held 459K, SWE100 at Harbor 32 from 26% to 95.5% cache hits.
Tests: CPU (tests/test_kv_packing.py): the full-attention spec is 256 tokens with the page size of a sliding block, half-page expansion and
the null block, narrow and padded tables, draft-column replication, allocation of one wide backing per pool position with the view and the
scratch rows (stubbed device), geometry and pool-composition checks. tests/test_kv_packing_simulation.py: the real vLLM KV cache manager
with the anchored sliding spec and the model's real _tables, 5 sliding groups + 1 full group, every attention read checked against a shadow
model of the wide rows under pool pressure, preemption, prefix hits, eviction and reuse; a control with a wrong translation is detected.
Device probe (l14_view_probe.py, in the PR draft): the view of [P,4,128,256] bf8 TILE as [2P,1,128,512] is accepted and zero-copy;
fills of 256-token blocks, sliding writes into other rows, decode SDPA over 1 and 3 blocks and chunked-prefill SDPA through the view are
bit-identical to the unpacked layout (21 of 21 checks).
Device tests on the rebuilt tree: the whole gemma4_31b_qb2 tests directory with this layer (43 passed); spec-profile server: the 8 seeded
and 8 greedy panel outputs are identical to the unpacked layer-13 stack, 1-agent replay TTFT 0.37 s. KV pressure (plain 32-slot server,
24K-token unique contexts, +600 tokens/turn, 4 turns; later-turn prefix hit / mean TTFT, unpacked -> packed): 16 agents 0% / 119.8 s ->
97.2% / 9.5 s, 24 agents 0% / 110.6 s -> 97.2% / 17.5 s, 32 agents 0% -> 0% (787K tokens of context exceed the ~698K usable
full-attention tokens). vLLM KV size 397,044 -> 629,505 tokens (1.51x -> 2.40x), the thrash cliff moves from about 12-13 to about 28 agents of
24K contexts. Client concurrency up to 16 agents is the supported setting.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…of short suffixes (exact) What: for 256, 512, 768 and 1024 query rows the full-attention prefill SDPA uses a Q chunk of 64 instead of the largest chunk that fits L1 (128). The kernel runs one unit per (head, Q chunk) on 110 cores and schedules pairs of chunks together when their count is even, so a short chunk of rows against a long cached prefix keeps few cores busy while each unit streams the whole cached K/V. Other row counts keep the default (an odd count of 128-row chunks is already best; above 1024 rows a smaller chunk streams K/V too often). The result is bit-identical (K is still consumed in chunks of 128 per query row); no flag. Why: after a prefix-cache hit an agent turn prefills a few hundred new tokens against 10-100K cached tokens, and there the attention dominates (65% of the device time of a 512-token suffix after 24K on a sliding+full layer pair). Tests: CPU: the selection rule per row count, sliding layers and the start do not matter (test_prefill_sdpa_chunks.py). Device: outputs of the rule and the default choice are bit-equal for 256-2048 rows after 8K cached tokens (probes at 8K, 24K and 48K starts); op probe, 512 rows at 24K: SDPA 11.23 -> 6.89 ms (-39%), device total 17.2 -> 12.9 ms (-25%); 256 rows at 48K: 21.4 -> 13.1 ms. End-to-end TTFT of a short suffix after a cached prefix (ms, 8K / 24K / 48K context): 256 tokens 211 / 260 / 333 -> 212 / 264 / 334 (unchanged: other ops dominate), 512 tokens 241 / 328 / 457 -> 228 / 288 / 373, 768 tokens 261 / 345 / 470 -> 250 / 311 / 400, 1024 tokens 287 / 373 / 505 -> 283 / 360 / 471, 1536 tokens unchanged. Device tests 32 passed; server pass: panels identical, replay 49.8 / 50.0 tok/s/user (50.1 / 50.0 before), TTFT 0.36 s. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…-up when too little DRAM is left What: get_max_tokens_all_users returns the DRAM budget in rows of the wide KV backing instead of only one request of --max-model-len plus the sliding windows: rows = (bank size - reserve - non-KV DRAM) / row bytes, minus the one output page per request the plugin adds. Per DRAM bank: bank 4,205,232,512 B (allocator-reported), reserve 910 MiB, non-KV 1305 MiB (weights, buffers, traces and prefill programs at 32 slots at the end of start-up, calibrated: the same constant from pools of 4200 and 5400 rows), row 348,160 B (10 full-attention positions x K and V x [4, 128, 256] BF8 over 8 banks). At 32 slots that is 5407 rows = 688,000 tokens (533,504 before; the packed pool, layer 14). A request longer than the budget still gets its pool, and shorter --max-model-len values use the same budget. A new check after the decode traces exist stops start-up with a clear error when less than 700 MiB per bank is free (the run-time working set of long prompts is at least 545 MiB), instead of a request-time out-of-memory error. No flag. Why: layer 14 freed 702 MiB per bank, and the thrash cliff of the prefix cache sits where the working set of the agents exceeds the pool (32 agents x 24K = 787K tokens against about 698K usable). The reserve is what the device showed: a pool of +2000 rows (650 MiB free after start-up) started, then died with an out-of-memory error on its first 24K prompt (110 MB free, largest block 3.65 MB); +1200 rows (915 MiB free) passed every test. Tests: CPU (test_serving_inputs.py): the budget arithmetic and its calibration against the two measured start-ups (4200 rows: 2699 MiB speculative, 5400 rows: 3098 MiB), the free-DRAM reserve left after start-up, a context beyond the budget, the start-up check with a stub memory view (stops below 700 MiB, passes above); 99 CPU tests pass. Device (stack with layers 14 and 15 and the eviction plugin, pool +1200 rows = this rule within 1%; later-turn prefix hit / mean TTFT, 24K-token contexts, 4 turns): 32 agents synchronous 0% / 158 s -> 97.2% / 26 s, 32 agents staggered 78% / 22 s -> 97.1% / 4.1 s, 40 agents staggered 96.5% / 6.2 s; 40 agents synchronous stays at 0.8% (960K tokens of context against about 913K usable). KV size reported by vLLM 629,505 -> 809,363 tokens. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…rom the sharded decode weight What: Decoder._prefill_weight rebuilt the gate and up weights (and, for full-attention layers, the [q | k] weight) by copying the whole fused DRAM-sharded decode weight to interleaved DRAM and then slicing the copy: the bytes were read and written twice and a full-size extra tensor was alive while the parts were made. The parts are now sliced directly from the decode weight into interleaved DRAM (ttnn.slice with an interleaved output memory config). Same values, so prefill results are unchanged; no flag. Why: the prefill weight rebuild runs once per layer per prompt chunk and is 12-18% of the device time of a cold prefill and about 12% of a 512-token suffix after a cached prefix (profile: 40 copies of 156 us on average in an 8K prefill of two layers). Tests: device (test_prefill_weight_slice.py): the sliced gate/up of a sliding and a full-attention layer and the sliced [q | k] equal the copy-then-slice rebuild bit for bit. Op timing per rebuild on the device: sliding gate_up 0.645 -> 0.485 ms, full gate_up 0.634 -> 0.485 ms, full [q | k] 0.315 -> 0.164 ms. Device tests of the layer: 40 passed (the new bit-equality cases, decoder, continuation programs, chunked prefill). End-to-end TTFT of a short suffix after a cached prefix on the packed 32-slot server (ms, 8K / 24K / 48K context, before -> after): 256 tokens 233 / 292 / 377 -> 225 / 279 / 364, 512 tokens 233 / 291 / 373 -> 224 / 279 / 363, 768 tokens 256 / 314 / 403 -> 245 / 302 / 393, 1024 tokens 291 / 366 / 478 -> 283 / 350 / 460, 1536 tokens 439 / 527 / 655 -> 422 / 503 / 633 (3-5% lower everywhere). Speculative server: the 8 seeded and 8 greedy panel outputs are identical to the stack without this layer, 1-agent replay 52.5 / 52.6 tok/s/user (52.6 / 52.5), TTFT 0.35 s.
…er and span, not once per chunk What: prefill_forward walks a span in chunks (1024 rows for sliding layers, 6656 for full-attention layers) and every chunk rebuilt the layer's interleaved weights from the DRAM-sharded decode copy (qkv or qk, o, gate/up, down: 4 copies and up to 3 slices per chunk). A span of several chunks now rebuilds them once, shares them between the chunks and frees them when the span is done (the per-layer transient is the layer's weights, about 16 MB per DRAM bank, instead of one weight at a time). The values are the same, so results are unchanged; a single chunk behaves as before; no flag. Why: the rebuild is 12-18% of the device time of a cold prefill: in the profile of an 8K prompt the 40 copies (6.3 ms) and the weight slices belong to 8 sliding and 2 full chunks per layer pair, i.e. 32 of the 40 copies are repeats within a layer. Long cold prompts (a 24-100K first agent turn, 50 sliding layers x 24-100 chunks) pay it for every chunk. Tests: device (test_prefill_weights_once.py): a sliding layer over 3 and 7 chunks and a full layer over 2 chunks give outputs bit-equal to the per-chunk rebuild, the number of rebuilds is 4 for the span instead of 4 x chunks, and a single chunk rebuilds 4 times as before. Device tests: 50 passed (the new test, the weight-slice test, decoder, continuation programs, chunked prefill, prefill warm-up). Cold prefill TTFT of a fresh random prompt on the packed 32-slot server (ms, mean of 3, before -> after): 4K 786 -> 672 (-14.5%), 8K 1586 -> 1306 (-17.7%), 24K 5255 -> 4440 (-15.5%), 48K 11932 -> 10331 (-13.4%). Speculative server: the 8 seeded and 8 greedy panel outputs are identical to the stack without this layer, 1-agent replay 52.0 / 52.2 tok/s/user (52.5 / 52.6 before: decode is untouched, within run-to-run noise), TTFT 0.33 s (0.35), DRAM allocated per bank 3099 MiB (unchanged).
What: the example server command uses --enable-prefix-caching (it showed --no-enable-prefix-caching), and the README gives the two profiles of tt-inference-server: the many-agent profile (async scheduling, no speculation) and the few-user profile (n-gram speculation K = 5, --no-async-scheduling, NUMBA_NUM_THREADS=1). The text on prefix caching no longer calls it optional, and the measured-serving table is described as an earlier measurement without prefix caching, speculation and KV packing. Why: the profiles that are validated and served have prefix caching on; the README still described the first measurement setup. Tests: documentation only; the commands match the two profiles in tt-inference-server (workflows/model_specs/dev/llm.yaml) and the server passes run in this stack. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This was referenced Nov 16, 2026
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What / why
The dedicated
gemma4_31b_qb2demo served one request well but not an agent workload: 32 agents with 25-100K-token contexts re-send their history every turn, so the server must keep a prefix cache that survives long replies, hold all 32 contexts in DRAM, decode with the smallest trace that covers the live requests, and never compile a program in the middle of a request. This PR is the demo-side work for that, as a stack of exact layers (each commit is one layer and says what / why / tests / numbers in its message). Everything is undermodels/demos/gemma4_31b_qb2; no flags or environment switches were added to the production code.Layers (one commit each, in order)
--max-model-lenno longer shrinks the poolTests
Results (final stack, QuietBox 2, one box; speculative profile =
gemma4_31b_qb2, plain =gemma4_31b_qb2_c32in tt-inference-server)Evaluation of the final stack (through tt-inference-server, QuietBox 2, local; source: spec-investigation/smoke/TTI_ACCEPTANCE_RESULT.md)
Stack: this branch (identical gemma tree to the tested M19 head) + the combined plugin + the two TTI profiles; plain
gemma4_31b_qb2_c32profile, Harbor concurrency 16, mini-swe-agent, temperature 1.0 / top-p 0.95 / top-k 20, thinking on, max_tokens 32768.Main with BF8 weights does not start at 32 slots / 262144 (DRAM out of memory at start-up), which the one-weight-copy layer solves. Costs of this stack: cold prefill 8-10% slower than main at 8K-24K (BF8, weight rebuild), batch-1 TTFT 135 vs 78 ms, start-up longer. Accuracy is not part of this A/B (see the SWE-bench / Terminal-Bench results above).
Depends on
mvasiljevic/gemma31b-pr-ttnn-placement): the DRAM-sharded matmul placement fix; the demo runs without it but QuietBox 2 decode is ~12% slower.Caveats
NUMBA_NUM_THREADS=1and synchronous scheduling.Pending (will be added here): thinking-off SWE shard, SPEED-Bench.
🤖 Generated with Claude Code