Skip to content

Gemma 4 31B QB2: two profiles of one 32-slot server, with and without n-gram speculation - #5577

Draft
mvasiljevicTT wants to merge 1 commit into
mainfrom
mvasiljevic/gemma31b-pr-tti
Draft

mvasiljevicTT wants to merge 1 commit into
mainfrom
mvasiljevic/gemma31b-pr-tti

Conversation

@mvasiljevicTT

@mvasiljevicTT mvasiljevicTT commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

What / why

The dedicated gemma4_31b_qb2 implementation supports prefix caching, adaptive decode rows and n-gram speculative decoding. The dev catalog gets two profiles of the same server (32 decode slots, 262144 context, prefix caching, shared YAML anchors) that differ only in speculation and scheduling:

  • gemma4_31b_qb2 (few users): n-gram speculation chosen by the number of live requests (K = 5 for one, K = 3 up to four, none beyond: the plugin plan caps larger batches at zero drafts), synchronous scheduling (vLLM refuses async scheduling with the n-gram method), NUMBA_NUM_THREADS=1.
  • gemma4_31b_qb2_c32 (many agents): speculation off, async scheduling.
    The server decodes with the smallest trace that covers the live rows, so a lone user runs at the one-slot speed on either profile. Files: workflows/model_specs/dev/llm.yaml, workflows/model_spec.py (8 lines), tests/workflows/test_gemma4_31b_qb2_profiles.py (profile tests).

Tests and results

Profile tests for both entries (flags, env, health timeout 3600 s); the server runs of both profiles on the full stack: panels 8+8 identical, 1-agent replay 52.3 tok/s/user (speculative), 32 agents x 24K later-turn hits 97% (plain profile), start-up 380-410 s on an idle host.

Evaluation of the two profiles (through this TTI entry, local; source: spec-investigation/smoke/TTI_ACCEPTANCE_RESULT.md)

Launched through these profiles: plain gemma4_31b_qb2_c32, Harbor 16: SWE-bench Verified 500 298/500 = 59.6% (CI 55.2-63.8 after retrying the 6 environment failures once; first pass 294/500 = 58.8%; 7.2 h wall), Terminal-Bench 89 37/89 = 41.6% (CI 31.9-52.0, 200.6 min); speculative gemma4_31b_qb2, 1 agent, 10 tasks: 4.15 min/trial, 47.8 tok/s/user; start-up through TTI 366-431 s warm (1501 s with an empty kernel cache). Deviations from the eval config: agent timeout 7200 s, thinking on and 32K output, TTI-default shell, Harbor 16 (local). CI is not available, so these are local numbers. Baseline A/B (unmodified main vs this stack, same conditions): done, results in the Gemma demo PR description (replay wall at 8 / 32 agents 668 -> 112 s and 1682 -> 165 s; batch-1 decode +11% to +22% plain).

Depends on

tt-metal demo PR tenstorrent/tt-metal#65783 (M12 for speculation) and the vllm-tt-plugin PR tenstorrent/vllm-tt-plugin#308 (cap-zero P3 for more than four live requests).

Caveats

  • Speculation pays only for few users; the plain profile is the default for many agents. Client concurrency up to 16 agents is the supported setting.
  • Start-up takes ~7-10 minutes; the health timeout is 3600 s.

Pending (will be added here): thinking-off SWE shard, SPEED-Bench.

🤖 Generated with Claude Code

… n-gram speculation

Adds the gemma4_31b_qb2_c32 impl and a second dev profile for the same implementation:
gemma4_31b_qb2 (n-gram speculation, synchronous scheduling, up to 4 agents) and
gemma4_31b_qb2_c32 (no speculation, async scheduling, 8 and more agents). Both serve 32 slots, 262144
context, prefix caching and adaptive decode rows.

The profile notes carry the end-to-end SWE-bench agent results (1 agent: +37% tok/s/user with speculation;
16 agents: plain +13%; 20 agents: 12-13 tok/s/user, KV peak 68%, hit rate 95%, no thrash), the crossover
at about 8 agents and the plugin changes both profiles need. The profile test also pins block size, trace
region and l1_small, the settings the model's KV pool rule was calibrated with.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant