Skip to content

Gemma 4 31B on QuietBox 2: the serving implementation (prefix caching, BF8, speculation, packed KV pool) - #65783

Draft
mvasiljevicTT wants to merge 19 commits into
mainfrom
mvasiljevic/gemma31b-pr-demo
Draft

mvasiljevicTT wants to merge 19 commits into
mainfrom
mvasiljevic/gemma31b-pr-demo

Conversation

@mvasiljevicTT

@mvasiljevicTT mvasiljevicTT commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

What / why

The dedicated gemma4_31b_qb2 demo served one request well but not an agent workload: 32 agents with 25-100K-token contexts re-send their history every turn, so the server must keep a prefix cache that survives long replies, hold all 32 contexts in DRAM, decode with the smallest trace that covers the live requests, and never compile a program in the middle of a request. This PR is the demo-side work for that, as a stack of exact layers (each commit is one layer and says what / why / tests / numbers in its message). Everything is under models/demos/gemma4_31b_qb2; no flags or environment switches were added to the production code.

Layers (one commit each, in order)

# commit what
M1 faster decode step with the same outputs model-only decode ops (56-core qkv/down inputs, one rope lookup per step); bit-identical
M2 prefix caching with restored sliding history a cached request rebuilds each sliding layer's window from vLLM-owned BF8 pages
M3 sliding groups use the plugin's anchored spec the prompt's cache-hit window stays until the request ends (needs the plugin PR)
M4 BF8 projection weights as the fixed weight format accuracy first: BF4 projections break long agent runs; decode 50.3 -> 36.4 tok/s at 1 row
M5 keep one copy of the projection weights prefill rebuilds its weights from the decode copy; makes 32 slots at 262144 fit with BF8
M6 the KV pool follows --max-model-len pool sizing from the served context
M7 compile every prompt length at start-up warm-up lattice; decode traces retire only when programs were compiled
M9 read cached sliding history by page id tensor-argument slice: no program per page id
M10 decode with the smallest trace that covers the live rows adaptive rows over one KV pool (two commits)
M11 one DRAM reader per bank for the o projection one constant, bit-identical
M12 n-gram speculative decoding verified in the smallest trace for the live requests (needs the plugin cap-zero PR for more than four live requests)
M13 continuation prefill without per-signature programs runtime-valued rope/fill/start inputs: a new (length, start) compiles nothing
M14 pack the full-attention KV into 256-token blocks one wide backing per pool position, zero-copy view: 2x full-attention tokens per row, -702 MiB/bank
M15 smaller Q chunk for short suffixes exact; 512-token suffix after 24K/48K: SDPA -39%, TTFT 328 -> 288 / 457 -> 373 ms
M16 size the KV pool from the DRAM budget rows = (bank - 910 MiB - non-KV)/row bytes, 688,000 tokens at 32 slots; start-up stops below 700 MiB free; a shorter --max-model-len no longer shrinks the pool
M17 slice gate/up and q/k straight from the sharded decode weight exact; short-suffix TTFT 3-5% lower
M18 rebuild the prefill weights once per layer and span exact; cold prefill TTFT 13-18% lower (8K 1586 -> 1306 ms)
M19 README: serving profiles with prefix caching docs

Tests

  • CPU (no device): 99 tests on this branch (decode rows, serving inputs, weight format, prefill warm-up, KV packing incl. a real-vLLM-KV-manager simulation).
  • Device (QuietBox 2), on the tested stack that this branch's gemma tree equals byte for byte: per-layer device tests (decoder bit-for-bit against the HF reference at layer level, chunked prefill, history restore, continuation programs, prefill chunk rules, weight slice, weights-once; 62 passed on the v7 tree incl. the ttnn placement test), spec server panels (8 seeded + 8 greedy prompts) identical across the layers after M12, replay of recorded SWE-bench trajectories.

Results (final stack, QuietBox 2, one box; speculative profile = gemma4_31b_qb2, plain = gemma4_31b_qb2_c32 in tt-inference-server)

measurement result
1-agent replay of recorded agent turns, speculative profile 52.3 tok/s/user, TTFT 0.33 s, 75.6% of later-turn input cached
start-up (32 slots, 262144 context) 380-410 s on an idle host (a busy host stretches trace capture)
KV pool 810,413 tokens reported by vLLM; 911-913 MiB DRAM free per bank after start-up
32 agents x 24K-token contexts, later-turn prefix hits 97.2% synchronous (mean TTFT 22 s), 97.1% staggered (3.3 s); previous pool 0% / 78%
40 agents x 24K, staggered 95.3% (synchronous 40 collapses: 960K tokens of context vs ~913K usable)
cold prefill TTFT 4K / 8K / 24K / 48K 672 / 1306 / 4440 / 10331 ms (before M17/M18: 786 / 1586 / 5255 / 11932)
  • Re-validated on the current main (ce962dd) after the C++ rebuild with PR1 applied: placement op test + decoder, weights-once and continuation device tests 33 passed; speculative server start-up 591 s (host shared), panels 8 seeded + 8 greedy identical to the previous validated stack, 1-agent replay 52.4 / 52.4 tok/s/user, TTFT 0.33 s, DRAM 3099 MiB per bank, no out-of-memory.

Evaluation of the final stack (through tt-inference-server, QuietBox 2, local; source: spec-investigation/smoke/TTI_ACCEPTANCE_RESULT.md)

Stack: this branch (identical gemma tree to the tested M19 head) + the combined plugin + the two TTI profiles; plain gemma4_31b_qb2_c32 profile, Harbor concurrency 16, mini-swe-agent, temperature 1.0 / top-p 0.95 / top-k 20, thinking on, max_tokens 32768.

benchmark result
SWE-bench Verified 500 (5 shards of 100) 298/500 = 59.6% (Wilson 95% CI 55.2-63.8) after retrying the 6 Docker-compose environment failures once (4 of the 6 solved); first pass 294/500 = 58.8% (CI 54.4-63.0), 7.2 h of Harbor wall (429 min summed, plus ~6 min start-up per shard); earlier BF8 SWE500 range 57-61%, previous reference runs 299/500 and 306/500
Terminal-Bench 89 37/89 = 41.6% (95% CI 31.9-52.0), wall 200.6 min, 1 agent timeout
server during SWE500 TTFT p50 0.43-0.45 s, p95 0.97-1.42 s; per-user decode 13.8-15.7 tok/s (median at 14+ running); KV peak 51-74%; prefix hit rate 94.7-96.2%; max waiting 4; repetition guard fired 18 times
server during Terminal-Bench 89 TTFT p50/p95 0.52/2.01 s; per-user decode 20.6 tok/s (median at 14+ running); KV peak 65%; prefix hit rate 93.3%; guard fired 2 times
32 users x ~31K contexts x 3 turns (plain profile) 32 users x ~31K-token contexts x 3 turns on the plain profile: 32/32 completed in 595 s, no errors, no out-of-memory, no preemption; KV peak 82%, up to 31 waiting; prefix hit rate 0% (the pool is oversubscribed: ~24 contexts of 33K is the cache-friendly limit, beyond it every turn re-prefills); turns 253 / 163 / 148 s
other end-to-end runs spec profile, 1 agent, 10 tasks: 4.15 min/trial, 47.8 tok/s; plain 16 agents, 20 tasks: 10.2 min/trial, 19.1 tok/s, KV peak 41%, hit 95.2%; regression A/B vs the earlier stack on 100 SWE tasks: 58 vs 54 and 51 vs 51 solved (within noise, McNemar p 0.45 / 1.0)
Deviations from the TTI evaluation config (stated, not hidden): agent timeout 7200 s (the config allows 6 h; a looping trial would hold a slot for hours), thinking on and 32K output (the config uses 8192 and no thinking), the SWE agent runs with the TTI-default shell (no testbed-first PATH), Harbor concurrency 16 (a local limit: Docker address pools and KV capacity; the reference runs used 32-40 agents, which is the main reason this run took 7.2 h). The 6 Docker-compose environment errors (agent never ran) were retried once: 4 of the 6 solved (298/500); 1 agent exit 137 stays unsolved.
CI is not available for this effort, so these are local numbers (CI runs ran 15-24% faster than local on the same build in earlier comparisons); no result here comes from the CI benchmark workflows.
Baseline A/B (same conditions, unmodified main demo as shipped vs this stack, QuietBox 2, one box; full table and sources in TOTAL_IMPROVEMENT.md):
metric main as shipped (BF4, no prefix cache)
--- ---
batch 1 ISL128/OSL128, tok/s/user / TTFT p50 33.88 / 78 ms
batch 1 ISL128/OSL1024 30.37 / 74 ms
1 user, 28K prompt 22.04 tok/s/user
4 / 8 / 32 users, tok/s/user 33.3 / 33.3 / 31.4
replay of recorded agent turns, 1 / 8 / 32 agents: wall 251 / 668 / 1682 s
later-turn TTFT / prefix hit (replay, 8 agents) 32.3 s / 0%
32 agents x 24K, 4 turns: later-turn hit / wall main with the prefix-cache flag: 0% / 932 s
cold prefill TTFT 8K / 24K / 48K 1191 / 4081 / 14610 ms
KV pool at 32 slots, 262144 399,019 tokens
start-up 180 s
Known trade-off (cold TTFT at short prompts): batch-1 TTFT at ISL128 is 135 ms here against 75 ms on main, and 32 simultaneous cold 128-token prompts take 3.5 s against 1.7 s. A layer-level probe attributes about 50 ms of the gap to the interleaved-weight rebuild that every prefill call performs (the price of keeping one weight copy so that BF8 fits at 32 slots / 262144; 0.84 ms per layer against 1.13 ms of compute at 128 tokens). Keeping the interleaved weights resident would remove it (estimated TTFT ~85 ms) but costs 0.46 GB per DRAM bank, about 20% of the KV pool; overlapping the rebuild with the previous layer's compute turned out not to be possible in this tree (no data-movement op takes a command-queue id, the served mesh has one queue, and both would use the whole compute grid), the rebuild already runs at about three quarters of DRAM bandwidth, and sharing it between requests needs a restructuring of the per-request prefill loop; none of this was built. Details in TOTAL_IMPROVEMENT.md sections 5 and 8.

Main with BF8 weights does not start at 32 slots / 262144 (DRAM out of memory at start-up), which the one-weight-copy layer solves. Costs of this stack: cold prefill 8-10% slower than main at 8K-24K (BF8, weight rebuild), batch-1 TTFT 135 vs 78 ms, start-up longer. Accuracy is not part of this A/B (see the SWE-bench / Terminal-Bench results above).

Depends on

Caveats

  • BF8 weights are the default format (M4): decode is ~27% slower than BF4 at one row, +467 MB weight DRAM per bank; mixed formats were measured and rejected.
  • A prefix-cache hit is not bit-identical to a cold prefill (prefix K/V are BF8-rounded); greedy output may differ.
  • Outputs depend slightly on the number of live users (trace row counts change reduction shapes); speculation (M12) gains only for few users (real text accepts 1.3-1.7 tokens per step) and needs NUMBA_NUM_THREADS=1 and synchronous scheduling.
  • Client concurrency up to 16 agents is the supported setting; 32 agents with 24K+ contexts need M14+M16 and still collapse for synchronous cyclic access above ~38 agents; Terminal-Bench with n_concurrent 24 would thrash long contexts.
  • Start-up is ~7-10 minutes (warm-up lattice and traces); TTI health timeout 3600 s.
  • A rare traceback-underline (caret) loop is model behaviour (7/435 replies on one xarray task); the plugin PR stops it.
  • Evidence is local (QuietBox 2, one box); CI numbers run 15-24% higher; CI is not available (local runs only); SWE-bench Verified 500 and Terminal-Bench 89 results of the final stack are in the evaluation section.

Pending (will be added here): thinking-off SWE shard, SPEED-Bench.

🤖 Generated with Claude Code

mvasiljevicTT and others added 19 commits November 15, 2026 18:22
Decode-step changes that keep every result the same (the K blocking that fixes matmul accumulation order is
unchanged):
- qkv and down DRAM-sharded decode matmuls read their input from 56 cores instead of 8; their output keeps the
  8-core shards the next ops read (new DECODE_OUTPUT_CORES, independent of the input cores).
- RoPE: cos/sin rows are looked up once per step per layer kind (decode_rope_tables) and applied with the fused
  rotary_embedding_hf decode op, instead of a typecast + pad + two embeddings in every layer and transpose +
  rotary_embedding + transpose.
- Sliding-window decode SDPA at 32 rows runs on the full 11x10 grid.

Tests: test_decoder (layers 0 and 5, prefill + live decode trace vs Hugging Face) passes; traced decode outputs and
KV cache of layers 0 and 5 were bit-identical to the previous layer at 1, 8 and 32 rows in a one-off comparison script run on both builds; that comparison is not a committed test (the committed test_decoder compares with Hugging Face at PCC 0.995).

Measured (QB2, vLLM 0.31 + vllm-tt-plugin main, main model BF4, 32-slot server, no prefix cache, greedy), on top
of the DRAM-reader placement layer:
  decode step: 49.6 -> 43.0 ms at 32 rows, 43.6 -> 33.9 ms at 1 row
  1 user ISL128/OSL1024: 33.9 -> 42.4 tok/s/user (+25%), TTFT 76 -> 77 ms
  1 user, 28K-token prompt: 23.8 -> 29.0 (+22%)
  4 / 8 / 32 users (ISL128/OSL256): 37.5 / 37.5 / 35.2 -> 45.4 / 45.2 / 40.7 tok/s/user (+21% / +21% / +16%);
  32 users 883 -> 996 tok/s aggregate (+13%)
Depends on the DRAM-reader placement layer for the 1-user numbers.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Agent loops resend the whole conversation every turn; with prefix caching vLLM reuses the completed history
instead of prefilling it again. Sliding layers keep a BF16 K/V tail between chunks, and a cached start has no tail,
so _restore_prefill_history gathers the preceding window from the request's own vLLM-owned (BF8) pages. Request
history is keyed by a private state slot instead of full-attention block 0 (shared by requests that share a
prefix) and released by the decode source slot when a row enters decode. Needs the vllm-tt-plugin state-slot fix
(a continuing prefill keeps its device state slot when its batch row moves).

Tests: test_chunked_prefill (cached continuation, sliding-layer PCC >= 0.995 vs Hugging Face at 128, 1024 and
1152 token boundaries) and the CPU tests in test_serving_inputs (state-slot keys, table ownership) pass.

Measured (QB2, 32-slot server, BF4, replay of recorded SWE-bench agent trajectories, cache on = stable cache_salt,
off = unique salt, same server): cached fraction of later turns 0.77-0.83. Wall time / mean TTFT of later turns /
decode tok/s per user, on vs off: 1 agent 86 vs 100 s / 5.6 vs 8.2 s / 37.7 vs 37.7; 8 agents 139 vs 354 s
(2.5x) / 4.4 vs 21.4 s / 24.5 vs 14.8; 32 agents 401 vs 604 s / 12.5 vs 14.7 s / 9.6 vs 5.6. A 28K-token prompt:
cold TTFT 4.8 s, cached 0.9 s.

Accuracy: a cached continuation is numerically close to, not bit-identical with, a cold prefill (the suffix is
prefilled from a different chunk alignment). Greedy 128-token continuations of 8 recorded agent prompts: two cold
prefills of the same prompt are identical (8/8); a cache hit diverges from a cold prefill on 7/8 prompts (first
divergence at token 0-56), always where the top-2 logprob gap is <= 0.5 (near-ties).

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…dow spec

Prefix caching needs the window that ends at a prompt's last hit boundary to survive a long reply; the stock
sliding-window manager frees it first. The sliding groups now return AnchoredSlidingWindowSpec from
vllm-tt-plugin (registered through vLLM's KVCacheSpecRegistry, so vLLM stays unmodified), which keeps that window
until the request finishes. No model or kernel change; needs the plugin's kv_cache_anchor module.

Tests: test_decoder (sliding and full layers) and test_chunked_prefill pass on this commit; the policy itself is
covered by the plugin's tests/test_kv_cache_anchor.py (real vLLM 0.31 block pool and managers).

Measured (QB2, 32-slot server, BF4, replay of recorded SWE-bench trajectories with this layer and the prefix-cache
layer): see the prefix-cache layer; with both, the cached fraction of later turns is 0.77-0.83. The 1-slot server
needed the admission cap to count the anchor pages (get_max_tokens_all_users reserves them once).
Earlier experiment with the same policy as a vLLM-core patch: Terminal-Bench 89 input found in cache 55% -> 92%,
prefill time -70%.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The qkv, o, gate/up and down projections are stored as BF8 instead of BF4 (decoder.WEIGHT_DTYPES; the tied LM head
was already BF8, model.HEAD_DTYPE). There is no switch: the format is a constant of the model.

Why BF8: accuracy on agent traffic. BF4 projections shift the next-token distribution enough to break long agent
runs (malformed tool calls, repeated channels); earlier SWE-bench Verified runs solved 52-55% with BF4 and 57-61%
with BF8. Cheaper mixes were measured against all-BF8 on 32 recorded agent prompts (47,493 positions) and a
17,228-token prompt, full 60 layers, QB2: BF8 attention/down/head + BF4 gate/up KL 1.15 (top-1 agreement 0.70);
BF8 attention/gate-up/head + BF4 down KL 1.14 (0.71); all-BF4 KL 2.06 (0.59). None keeps the BF8 greedy
continuation (14 of 16 prompts diverge within 20 tokens), so all four projections stay BF8.

BF8 weight tiles are twice the size of BF4 tiles: a 6656- or 8192-row prefill chunk overflowed L1 in the matmul
circular buffers (server crash on an 8936-token prompt). Chunks above 4096 rows use a K block of at most 4 for
non-BF4 weights; only the accumulation split changes.

Tests: test_decoder (layers 0 and 5, prefill + live decode trace vs Hugging Face, PCC >= 0.995) now runs with every
projection in BF4 and in BF8. Chunk sizes 6656, 8192 and 9216 rows run (6656 failed before).

Measured (QB2, layer-level and full model, plain decode, 1 row): 50.3 -> 36.4 tok/s at 128 tokens of context
(45.2 -> 33.6 at 28K); 32-row step 23.7 -> 31.3 ms; +467 MB weight DRAM per bank. On its own this layer does not
leave room for the 32-slot KV pool at 262144 tokens: the next layers (one weight copy, pool budget) make that fit.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Prefill and decode each kept their own copy of every projection matrix (qkv, o, gate/up, down). Prefill now
rebuilds its interleaved weights from the decode DRAM-sharded copy when a chunk needs them (layout move and slices
only, same values) and frees them after the matmul. A layer keeps one copy of its weights instead of two.

Why: with BF8 weights (previous layer) 32 slots with the full 262144-token context did not fit: the KV pool ran
out of DRAM at start-up, and at --max-model-len 131072 a 32-agent replay ran out at run time. This frees about
0.46 GB per DRAM bank.

Tests: test_decoder (PCC vs Hugging Face, BF4 and BF8) and test_chunked_prefill pass on this commit.

Measured (QB2): prefill logits and KV cache of layers 0 and 5 are bit-identical to the two-copy code at 1024, 2048
and 4096 tokens. Cost: warm prefill +2 ms per layer per chunk (24.3 -> 28.6 ms for 2 layers at 4096 rows). Server
(32 slots, max-model-len 262144, BF8): starts, pool 397,044 tokens, 32-agent replay completes (wall 208 s, cached
fraction 0.81, mean TTFT 3.9 s); 32-user forced decode 27.7 tok/s/user.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
get_max_tokens_all_users reserved a full 262144-token request whatever context the profile serves. It now reserves
max_model_len plus the sliding windows, so a profile with a shorter context frees DRAM for weights and prefill
temporaries. The default (262144) is unchanged.

Tests: CPU test in test_serving_inputs (pool size for 262144 and for a shorter context).

Measured (BF8, 32 slots): --max-model-len 262144 gives a 397,044-token pool; 131072: 238,012; 65536: 141,138.
Shorter contexts cost cache capacity: at 65536 the 32-agent replay cached fraction fell from 0.81 to 0.32.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…de traces only when programs were compiled

A prefill of a prompt length the program cache has not seen compiles new programs (up to tens of seconds for the
first of a tile-padded shape) and, as before, retires the decode traces because new programs may leave persistent
device state a captured trace overlaps. Previously every new prefill signature retired the traces before running.

Now a prefill retires them only if the prefill-and-sample call grew the program cache (the row merge of sample_prefill_outputs, whose concat/pad programs compile per batch size, used to compile outside the count; it is counted now and every batch size 2..32 is compiled at start-up), and warmup_model_prefill compiles every 32-token prompt length from position 0 up to the prefill limit. The decode trace survives prefills whose signature the program cache already holds. It does not survive a never-seen signature: continuation chunks (start > 0) and every new (length, start) still compile 12-47 programs (0.1-0.8 s) and retire the traces; the last layer of this stack (continuation prefill without per-signature programs) removes that. The plugin's two-phase warm-up called warmup_model_prefill twice for the same cache and repeated the whole lattice (3 min 37 s of the server log); the second call is skipped. The lattice is limited to the cache capacity (a --max-model-len below 8192 has fewer page-table columns and rejects longer prompts). A program compiled in the sampling step retires the traces before the sampling trace runs (the entry is rebuilt and the merged rows copied again), not after.

Tests: tests/test_prefill_warmup.py (layers 0 and 5): warm-up leaves no new compilation for warmed lengths, the
trace survives them, and greedy decode is identical with the kept trace, after other prefills, and after a forced
trace retire. CPU tests: warm-up call list including every prefill batch size, serving warm-up limited to the capacity, the second warm-up of one cache skipped, program counting in sample_prefill_outputs (no growth: no retire; growth: retire before the sampling trace executes).

Measured (QB2): first prefill of a new length 39.8 s cold; after warm-up of 2048 tokens (141 s for 2 layers)
lengths 320/1120/1920 take 6-11 ms of host-side prefill time; an off-lattice length (1000) takes 0.67 s. On the
full model the 256 lengths up to 8192 add about 7.5 minutes to start-up.

Lattice choice (40 random lengths 130-2040, cold kernel cache, 2 layers, first-use TTFT): every 32 tokens: median 0.95 s,
worst 1.5 s, none above 3 s; every 128 tokens: median 1.23 s, one 6.7 s stall in 39 (a stall also retires every active
user's decode trace); no warm-up: stalls of 3-9 s on 28% of lengths. A length off the lattice can still compile a few
programs (about 1 s typically), because programs depend on the exact length.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…ompiling a program per id

A cache-hit continuation restores the preceding sliding window from the request's KV pages with one slice per
physical page, cache[p : p + 1]. The page id is part of the slice's program hash, so every request that received new
block ids from vLLM compiled new slice programs, and a prefill that compiles programs retires every decode trace (new
program buffers allocated while traces exist are unsafe for them: the trace allocation tracker reports them). Almost
every call of an agent loop retired the traces and paid a recapture.

The page id is now a runtime value: ttnn.slice with device start and end tensors (slice_dim=0, num_devices=pages)
reads any page with one program per cache shape. The start and end tensors are uploaded once per page id and reused
by all sliding layers and both caches.

Tests: tests/test_history_restore.py: the restored history is bit-identical to the previous Python-slice
implementation, and restores with other block ids add no programs.

Measured (QuietBox 2, BF8, 1-slot server, replay of recorded SWE-bench trajectories, warm pass then identical measured
pass): programs added per continuation prefill in the measured pass 40, 40, 15, 20, 40, 15, 40 -> 0 on all 8
prefills; trace retires 7 of 8 -> 0 (captures in the run: 10, all at start-up); later-turn TTFT mean 0.66 s -> 0.37 s
(p90 0.68 -> 0.40 s). A first-time (length, start) signature still compiles 14-25 programs once (was 30-63) and
retires the traces.

Accuracy: bit-identical (see Tests).

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…rows

Also (the capture machinery of this layer): Gemma4Generator._entry is split into _new_entry / _warm_entry / _capture_entry and
prepare_entries(states) captures one decode trace per cache state; all persistent inputs are allocated before any trace is
captured, so no persistent tensor lands on memory a captured trace reuses for intermediates.

What: a server keeps decode traces of 1, 4, 8, 16 and 32 rows (those below its slot count, plus the full one) over the
same KV pool. Each decode step runs the smallest trace with at least last-live-row + 1 rows. tt/decode_rows.py holds
the sizes and the choice. The vLLM batch is front-packed (live requests occupy rows 0..n-1), and any change of the
live rows is a layout change that already reloads tokens, positions, page tables and sampler seeds, so a trace switch
moves no state.

Why: rows without a request still cost device time, and the full-attention SDPA of one live row is 8-10x slower in a
32-row trace than in a 1-row trace at long context. One user on a 32-slot server decodes at the speed of a 1-slot
server instead of 11-62% slower than it (device wait 21.7 / 26.4 / 34.5 / 68.6 ms at 128 / 4K / 28K / 130K context
with 32 rows against 19.5 / 20.8 / 21.5 / 24.7 ms with 1 row). Traces specialised by context bucket (page-table
horizon 4K / 32K / 262144) gave no gain: the step changes by <= 0.5% at 1-4 rows, so only row count is used.

Tests: host tests for the choice (test_decode_rows_host.py, 21 cases); layer-level test of 1/4/8/16-row decode
against the 32-row decode on a sliding and a full layer, PCC >= 0.998 (test_decode_rows_layer.py: worst 0.99873 and
0.99987). Server runs on this branch, BF8, 32 slots, 262144 tokens:
  one-user decode tok/s/user at 128 / 4K / 28K: fixed 32-row 33.82 / 29.40 / 23.76, this layer 36.82 / 35.27 / 34.41
  (+8.9% / +20.0% / +44.8%), equal to the 1-slot server.
  8-user replay of recorded agent trajectories (mixed arrivals and departures, trace switches): 21.2 tok/s/user
  against 21.6 for the fixed server (within noise).
  Start-up: capturing the 5 traces takes 24.5 s the first time and 1.4 s with kernels cached (+70 s to a healthy
  server); DRAM allocated per bank 3407 MiB of a 4010 MiB bank (about 600 MiB left); the 512 MiB trace region holds
  the five traces (its usage is not readable from Python).

Faithfulness and accuracy: a lone request on this server gives bit-identical greedy and seeded outputs (8 real
prompts x 128 tokens each) to a 1-slot server, which runs the same 1-row trace. Outputs are not bit-identical across
row counts (the decode SDPA and the sharded memory configs depend on the row count), so they can differ slightly with
the number of active users, exactly as they already do between the 1-slot and 32-slot profiles (greedy identical on
1 of 8 prompts, seeded on 0 of 8 between those two profiles today). On the teacher-forced decode path over 32 recorded
agent prompts (6,630 positions, last 256 tokens of each), KL(32-row || N-row) over the top 64 tokens is 0.70 / 0.62 /
0.63 for 1 / 4 / 8 rows (0.19 / 0.18 / 0.18 on model-turn positions), top-1 agreement 0.83-0.84, and the mean NLL of the
true tokens moves by <= 0.03 (4.405 for 32 rows; 1 row is the lowest, 4.391). Every size is equally far from the prefill
answer for the same positions (KL 0.61-0.62, top-1 0.81), so the row count adds noise of the size of the decode-vs-prefill
difference, not an accuracy loss; the BF4 weight mixes lose 0.14-0.97 NLL.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
… log the prefill that retires them

What: a decode entry's persistent inputs (tokens, positions, logits, update positions) are kept in Gemma4Generator.entries
when a prefill retires the traces. The next decode recaptures only the trace it needs, and capturing one trace no
longer retires the others.

Why: with several decode traces, retiring on every capture made a server that switches between traces capture on every
step after a retire (903 captures in one 8-call replay of the speculative server: a capture per verify/plain switch).
Once the entry's persistent inputs exist a capture allocates only transient tensors, so another trace stays valid; a new
entry and a new prefill signature still retire all traces first, as before.

Tests: server runs (see the previous commit): 8-user replay 21.2 tok/s/user against 19.9-20.3 before this change.

Also: a prefill that compiles programs and retires the traces logs its prompt lengths, start positions and the number of new
program-cache entries (at debug level: a per-request diagnostic of the serving path); in a replay of recorded agent
trajectories every continuation prefill added 15-63 programs, which is why each call retired the traces.

Review fixes folded in: the layer test passes rope_positions clamped at 0 for its inactive rows (the rope lookup casts the index
to uint32), and the retire line is at debug.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
… matmul

The decode matmul of the attention output projection (o) read its weights with two workers per DRAM bank. A sweep of
the geometry (8-64 input cores, 1-4 readers per bank, K block unchanged) on layers 0 and 5 found one reader per bank
fastest for both layer kinds. The K block fixes the accumulation order and the input cores and readers only move
data, so the results do not change. qkv, down and gate/up already run in their best geometry (sweeps: no gain).

Tests: tests/test_decode_geometry.py (sliding and full layer): the o projection output with one and with two readers
per bank is bit-identical; test_decoder passes.

Measured (QuietBox 2, BF8, op-level real-time profiler, 1 row): o matmul 35.4 -> 28.4 us (sliding layer) and
67.0 -> 50.9 us (full layer). Full model, traced decode: 1 row at 128 tokens of context 36.44 -> 37.11 tok/s (+1.8%),
at 28K 33.66 -> 34.25 (+1.7%); 32-row step 31.28 -> 30.79 ms at 128 tokens, 43.34 -> 42.85 ms at 4K.

Accuracy: bit-identical (see Tests).

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…t trace for the live requests

What: the model accepts the vLLM plugin's host n-gram drafts and verifies them as a multi-column decode step. Column j of
user u is row j * users + u; every row writes its K/V at its own position before attention reads and samples with its
slot's parameters and seed + position, so the accepted tokens are the tokens a plain decode produces. Rows that write into
one cache tile (a user's draft columns) are updated by separate paged_update_cache calls (concurrent writers to one tile
lose updates); draft columns are a property of the cache state (CacheState.columns), so plain traces pay no extra cache
updates. Page-table translation is de-duplicated by storage address.
Speculation follows the number of live requests (VERIFY_K_BY_USERS in tt/decode_rows.py): one request drafts K=5 (a 6-row
verify trace), up to four draft K=3 (a 16-row trace), more run plain decode; spec_plan publishes this as the plugin's draft
cap per live rows (k_by_rows, a cap of 0 above four requests, which needs the plugin cap-zero change). All plain and verify
traces are captured at start-up. A verify block with no draft (the plugin sends one after a multi-token commit) runs as a
plain decode; a block with drafts that no verify trace covers is an error. Rows entering decode release their private
sliding-history tails in verify steps as in plain steps. Whether speculation is on is a profile setting
(speculative_config in the TTI entry), not a model flag.

Why: speculation pays for one user (+40% on a replay of recorded agent trajectories, K=5) and four (+22%, K=3,
NUMBA_NUM_THREADS=1) and loses beyond that (0.98x at 8, 0.84-0.87 at 8-16 slots with the old fixed trace), so it follows
the active requests instead of the slot count.

Tests: host tests with a stubbed generator (spec_plan and its class attribute, verify layout and answers per slot, fewer
users than slots, draftless block above the verify traces, drafts there refused, plain-decode trace choice, release of
tails, warm-up captures every trace); tests/test_decode_columns.py (sliding and full layer): draft columns equal the same
decode steps run one by one at the same row count. Whole host suite of the layer.

Measured (QB2, BF8, 32 slots, K=5, sync, NUMBA_NUM_THREADS=1, replay of recorded agent trajectories, tok/s/user): 1 / 4 / 8
agents 49.3 / 37.0 / 24.3 against 35.3 / 30.3 / 24.8 for the plain adaptive server (1.40x / 1.22x / 0.98x); one user,
random-token prompt 178 / 166 / 155 at 128 / 4K / 28K context. Exactness: speculation vs plain with every step in the same
6-row layout, 8 prompts x 128 tokens: greedy 8/8 and seeded 8/8 identical. Against the 1-row plain server the streams differ
like any two row counts (the row-count noise stated in the adaptive-rows layer).
Limits: synchronous scheduling only (vLLM refuses async scheduling with n-gram), so many agents lose a little against the async
plain server (replay on the final stack, tok/s/user: 8 agents 25.2 vs 25.9, 32 agents 12.3 vs 12.8, i.e. -3..-4%; 1 and 4 agents
+34% / +20%): serve speculation for few users only; NUMBA_NUM_THREADS=1 is required.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
What: every op of a prefill chunk whose program hash held the position or the length takes a runtime value, and the prompt is
padded to whole cache pages, so the programs depend only on the padded chunk length:
- rope cos/sin rows are gathered by position ids (ttnn.embedding on the table, tile layout) instead of sliced by [start:start+n];
- the cache-fill page ids are uploaded as a row (the host knows the page table) instead of a Python slice of the device table;
  layers that share a page-table row share the upload;
- chunked SDPA reads its start from a one-element tensor (chunk_start_idx_tensor) instead of the Python chunk_start_idx;
- the 32-row group of the last token is a tensor-argument slice (device-valued offset); the row inside that group is still a
  Python-valued slice (32 values, all compiled at start-up);
- the prompt is padded to 128 on the host (no per-length pad), the saved sliding tails end at the padded end and the
  continuation rewinds to its aligned start (history tail_end).
A prefill that returns all logits (return_all_logits: tests and evaluation) still runs the per-length path.
warm_prefill_shapes now warms every 128-token padded length, the 32 last-token rows at both residual widths, for every padded
length a continuation after a full sliding window (aligned and unaligned previous end), and in the serving cache the
prefix-cache history restore for every page count (the first prefix hit after start-up used to compile slice/concat/typecast
programs once and retire the traces).
The flexible chunked SDPA needs the page-table row (max_seq_len / 128 int32 entries) to be a multiple of 32 bytes, so the serving
context must be a multiple of 1024 tokens (checked in initialize_vllm_model; 262144 and 131072 qualify). The per-start
pref_sdpa_selections record is keyed by the chosen chunk instead of the start (it grew without bound).

Why: a (length, start) never seen before compiled 12-47 programs (measured, 2 layers) and retired the decode traces, costing
about 0.76 s per call at one slot (0.35-0.5 s per trace recapture, three traces on the speculative adaptive server). Live
traffic produces new signatures constantly: this was the largest remaining cost of an agent loop.

Tests (layers 0 and 5 on the device): tests/test_continuation_programs.py: after warm-up, never-seen (length, start) pairs (unaligned
lengths, page-aligned starts, one and several chunks) add 0 programs; at layer level (sliding and full, three continuations each)
the chunk with runtime-valued inputs padded to whole pages gives the same layer output as the per-length chunk, bit for bit; and
chunk chains with unaligned starts (1000-2100-3000, 300-1500-2560-3500, 2600-3500) give last-row logits with PCC >= 0.9999 against the
all-logits path and equal KV rows of the last window in every layer (the logits differ only by the head's matmul shape). CPU:
warm-up call list, the context check (test_serving_inputs.py, test_prefill_warmup.py).
Probe evidence (scripts in the PR draft, not committed): l13_probe.py runs one sequence of prefills (aligned and unaligned
starts, one and several chunks) after warm-up and records the programs each added and the last-token logits; l13_compare_logits.py
compares the logits of two builds with torch.equal. Against the previous layer: 24 novel signatures at 12 starts add 0 programs
(previous layer: 12-47 per first call of a start, 2-20 per second call); the 38 prefills both builds ran have bit-identical logits
(one-row head). The same probe was re-run on the final tree of this stack: 24 novel signatures add 0 programs, the 38 prefills shared with the previous run are bit-identical to the previous layer's logits (l13_compare_logits.py: compared 38, differing none), and prefill batch sizes 2, 3, 5 and 8 through sample_prefill_outputs add 0 programs; the device tests of the changed files pass (28).
Limits: a continuation after fewer than 1024 sliding tokens (a history of 128..896 rows: a chunked prompt whose first chunk was
short, or a prefix hit at a short aligned start) and a continuation chunk of more than max_tokens - 1152 padded rows (7041..8320 rows
at the 8192-token limit; an unaligned start adds up to a page) are other signatures: each compiles once and retires the traces.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…r a shared wide backing

What: a pool position holds one full-attention layer and five sliding layers. They share one BF8 backing of wide rows with the sliding
geometry [P, 4, 128, 256] (P = block IDs + scratch rows); the full-attention layer reads it as [2P, 1, 128, 512] through a zero-copy
ttnn.experimental.view. The scheduler gives the full-attention group 256-token blocks (the same 2 MiB page as a 128-token sliding
block) and the sliding groups 128-token blocks over one global ID space; every ID is owned by one group at a time (vLLM's block pool),
so a full block owns a whole wide row = native half-pages 2b and 2b + 1 (_tables expands the full-attention tables; the null block stays
0). Kernels are unchanged. The pool IDs are unchanged (get_max_tokens_all_users): each row now holds 256 full-attention tokens instead
of 128, and the separate full-attention tensors (10 pools x n x 65,536 x 1.0625 B x 2 = about 725 MB per DRAM bank at n = 4168) are gone.
Prefix hits of the full-attention group are 256-aligned (at most 255 recomputed tokens instead of 127). No flag: the layout is fixed.

Why: with 32 agents at 25-100K tokens the pool (~410K tokens left for full histories after the sliding windows of 21 running requests) is
exhausted: prefix-cache hit rate fell from 60% to 14% and decode to 3 tok/s/user in the Harbor-32 smoke. The original effort measured this
mechanism ("pack2") as the fix: 825K full-attention tokens in the backing that held 459K, SWE100 at Harbor 32 from 26% to 95.5% cache hits.

Tests: CPU (tests/test_kv_packing.py): the full-attention spec is 256 tokens with the page size of a sliding block, half-page expansion and
the null block, narrow and padded tables, draft-column replication, allocation of one wide backing per pool position with the view and the
scratch rows (stubbed device), geometry and pool-composition checks. tests/test_kv_packing_simulation.py: the real vLLM KV cache manager
with the anchored sliding spec and the model's real _tables, 5 sliding groups + 1 full group, every attention read checked against a shadow
model of the wide rows under pool pressure, preemption, prefix hits, eviction and reuse; a control with a wrong translation is detected.
Device probe (l14_view_probe.py, in the PR draft): the view of [P,4,128,256] bf8 TILE as [2P,1,128,512] is accepted and zero-copy;
fills of 256-token blocks, sliding writes into other rows, decode SDPA over 1 and 3 blocks and chunked-prefill SDPA through the view are
bit-identical to the unpacked layout (21 of 21 checks).
Device tests on the rebuilt tree: the whole gemma4_31b_qb2 tests directory with this layer (43 passed); spec-profile server: the 8 seeded
and 8 greedy panel outputs are identical to the unpacked layer-13 stack, 1-agent replay TTFT 0.37 s. KV pressure (plain 32-slot server,
24K-token unique contexts, +600 tokens/turn, 4 turns; later-turn prefix hit / mean TTFT, unpacked -> packed): 16 agents 0% / 119.8 s ->
97.2% / 9.5 s, 24 agents 0% / 110.6 s -> 97.2% / 17.5 s, 32 agents 0% -> 0% (787K tokens of context exceed the ~698K usable
full-attention tokens). vLLM KV size 397,044 -> 629,505 tokens (1.51x -> 2.40x), the thrash cliff moves from about 12-13 to about 28 agents of
24K contexts. Client concurrency up to 16 agents is the supported setting.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…of short suffixes (exact)

What: for 256, 512, 768 and 1024 query rows the full-attention prefill SDPA uses a Q chunk of 64 instead of the largest chunk that fits L1 (128).
The kernel runs one unit per (head, Q chunk) on 110 cores and schedules pairs of chunks together when their count is even, so a short
chunk of rows against a long cached prefix keeps few cores busy while each unit streams the whole cached K/V. Other row counts keep
the default (an odd count of 128-row chunks is already best; above 1024 rows a smaller chunk streams K/V too often). The result is
bit-identical (K is still consumed in chunks of 128 per query row); no flag.

Why: after a prefix-cache hit an agent turn prefills a few hundred new tokens against 10-100K cached tokens, and there the attention
dominates (65% of the device time of a 512-token suffix after 24K on a sliding+full layer pair).

Tests: CPU: the selection rule per row count, sliding layers and the start do not matter (test_prefill_sdpa_chunks.py). Device: outputs of the rule and the default choice are bit-equal for 256-2048 rows after 8K cached tokens (probes at 8K, 24K and 48K starts);
op probe, 512 rows at 24K: SDPA 11.23 -> 6.89 ms (-39%), device total 17.2 -> 12.9 ms (-25%); 256 rows at 48K: 21.4 -> 13.1 ms.
End-to-end TTFT of a short suffix after a cached prefix (ms, 8K / 24K / 48K context): 256 tokens 211 / 260 / 333 -> 212 / 264 / 334
(unchanged: other ops dominate), 512 tokens 241 / 328 / 457 -> 228 / 288 / 373, 768 tokens 261 / 345 / 470 -> 250 / 311 / 400, 1024 tokens
287 / 373 / 505 -> 283 / 360 / 471, 1536 tokens unchanged. Device tests 32 passed; server pass: panels identical, replay 49.8 / 50.0 tok/s/user
(50.1 / 50.0 before), TTFT 0.36 s.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…-up when too little DRAM is left

What: get_max_tokens_all_users returns the DRAM budget in rows of the wide KV backing instead of only one request of
--max-model-len plus the sliding windows: rows = (bank size - reserve - non-KV DRAM) / row bytes, minus the one output page per
request the plugin adds. Per DRAM bank: bank 4,205,232,512 B (allocator-reported), reserve 910 MiB, non-KV 1305 MiB (weights, buffers,
traces and prefill programs at 32 slots at the end of start-up, calibrated: the same constant from pools of 4200 and 5400 rows), row
348,160 B (10 full-attention positions x K and V x [4, 128, 256] BF8 over 8 banks). At 32 slots that is 5407 rows = 688,000 tokens
(533,504 before; the packed pool, layer 14). A request longer than the budget still gets its pool, and shorter --max-model-len values
use the same budget. A new check after the decode traces exist stops start-up with a clear error when less than 700 MiB per bank is
free (the run-time working set of long prompts is at least 545 MiB), instead of a request-time out-of-memory error. No flag.

Why: layer 14 freed 702 MiB per bank, and the thrash cliff of the prefix cache sits where the working set of the agents exceeds the pool
(32 agents x 24K = 787K tokens against about 698K usable). The reserve is what the device showed: a pool of +2000 rows (650 MiB free after
start-up) started, then died with an out-of-memory error on its first 24K prompt (110 MB free, largest block 3.65 MB); +1200 rows (915 MiB
free) passed every test.

Tests: CPU (test_serving_inputs.py): the budget arithmetic and its calibration against the two measured start-ups (4200 rows: 2699 MiB
speculative, 5400 rows: 3098 MiB), the free-DRAM reserve left after start-up, a context beyond the budget, the start-up check with a stub
memory view (stops below 700 MiB, passes above); 99 CPU tests pass. Device (stack with layers 14 and 15 and the eviction plugin, pool +1200 rows =
this rule within 1%; later-turn prefix hit / mean TTFT, 24K-token contexts, 4 turns): 32 agents synchronous 0% / 158 s -> 97.2% / 26 s,
32 agents staggered 78% / 22 s -> 97.1% / 4.1 s, 40 agents staggered 96.5% / 6.2 s; 40 agents synchronous stays at 0.8% (960K tokens of
context against about 913K usable). KV size reported by vLLM 629,505 -> 809,363 tokens.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…rom the sharded decode weight

What: Decoder._prefill_weight rebuilt the gate and up weights (and, for full-attention layers, the [q | k] weight) by copying the whole fused
DRAM-sharded decode weight to interleaved DRAM and then slicing the copy: the bytes were read and written twice and a full-size extra
tensor was alive while the parts were made. The parts are now sliced directly from the decode weight into interleaved DRAM
(ttnn.slice with an interleaved output memory config). Same values, so prefill results are unchanged; no flag.

Why: the prefill weight rebuild runs once per layer per prompt chunk and is 12-18% of the device time of a cold prefill and about 12% of a
512-token suffix after a cached prefix (profile: 40 copies of 156 us on average in an 8K prefill of two layers).

Tests: device (test_prefill_weight_slice.py): the sliced gate/up of a sliding and a full-attention layer and the sliced [q | k] equal the
copy-then-slice rebuild bit for bit. Op timing per rebuild on the device: sliding gate_up 0.645 -> 0.485 ms, full gate_up 0.634 -> 0.485 ms,
full [q | k] 0.315 -> 0.164 ms. Device tests of the layer: 40 passed (the new bit-equality cases, decoder, continuation programs, chunked prefill).
End-to-end TTFT of a short suffix after a cached prefix on the packed 32-slot server (ms, 8K / 24K / 48K context, before -> after): 256 tokens
233 / 292 / 377 -> 225 / 279 / 364, 512 tokens 233 / 291 / 373 -> 224 / 279 / 363, 768 tokens 256 / 314 / 403 -> 245 / 302 / 393,
1024 tokens 291 / 366 / 478 -> 283 / 350 / 460, 1536 tokens 439 / 527 / 655 -> 422 / 503 / 633 (3-5% lower everywhere). Speculative server:
the 8 seeded and 8 greedy panel outputs are identical to the stack without this layer, 1-agent replay 52.5 / 52.6 tok/s/user (52.6 / 52.5), TTFT 0.35 s.
…er and span, not once per chunk

What: prefill_forward walks a span in chunks (1024 rows for sliding layers, 6656 for full-attention layers) and every chunk rebuilt the
layer's interleaved weights from the DRAM-sharded decode copy (qkv or qk, o, gate/up, down: 4 copies and up to 3 slices per chunk). A span
of several chunks now rebuilds them once, shares them between the chunks and frees them when the span is done (the per-layer transient is
the layer's weights, about 16 MB per DRAM bank, instead of one weight at a time). The values are the same, so results are unchanged; a
single chunk behaves as before; no flag.

Why: the rebuild is 12-18% of the device time of a cold prefill: in the profile of an 8K prompt the 40 copies (6.3 ms) and the weight slices
belong to 8 sliding and 2 full chunks per layer pair, i.e. 32 of the 40 copies are repeats within a layer. Long cold prompts (a 24-100K first
agent turn, 50 sliding layers x 24-100 chunks) pay it for every chunk.

Tests: device (test_prefill_weights_once.py): a sliding layer over 3 and 7 chunks and a full layer over 2 chunks give outputs bit-equal to the
per-chunk rebuild, the number of rebuilds is 4 for the span instead of 4 x chunks, and a single chunk rebuilds 4 times as before.
Device tests: 50 passed (the new test, the weight-slice test, decoder, continuation programs, chunked prefill, prefill warm-up). Cold prefill TTFT of a
fresh random prompt on the packed 32-slot server (ms, mean of 3, before -> after): 4K 786 -> 672 (-14.5%), 8K 1586 -> 1306 (-17.7%), 24K 5255 -> 4440
(-15.5%), 48K 11932 -> 10331 (-13.4%). Speculative server: the 8 seeded and 8 greedy panel outputs are identical to the stack without this layer,
1-agent replay 52.0 / 52.2 tok/s/user (52.5 / 52.6 before: decode is untouched, within run-to-run noise), TTFT 0.33 s (0.35), DRAM allocated per bank
3099 MiB (unchanged).
What: the example server command uses --enable-prefix-caching (it showed --no-enable-prefix-caching), and the README gives the two profiles of
tt-inference-server: the many-agent profile (async scheduling, no speculation) and the few-user profile (n-gram speculation K = 5,
--no-async-scheduling, NUMBA_NUM_THREADS=1). The text on prefix caching no longer calls it optional, and the measured-serving table is
described as an earlier measurement without prefix caching, speculation and KV packing.

Why: the profiles that are validated and served have prefix caching on; the README still described the first measurement setup.

Tests: documentation only; the commands match the two profiles in tt-inference-server (workflows/model_specs/dev/llm.yaml) and the
server passes run in this stack.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

unknown Classification failed or was unavailable - needs manual review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant