Repository navigation
Gemma 4 31B QB2: two profiles of one 32-slot server, with and without n-gram speculation - #5577
Draft
mvasiljevicTT wants to merge 1 commit into
Draft
mvasiljevicTT wants to merge 1 commit into
mvasiljevicTT wants to merge 1 commit into
Conversation
… n-gram speculation Adds the gemma4_31b_qb2_c32 impl and a second dev profile for the same implementation: gemma4_31b_qb2 (n-gram speculation, synchronous scheduling, up to 4 agents) and gemma4_31b_qb2_c32 (no speculation, async scheduling, 8 and more agents). Both serve 32 slots, 262144 context, prefix caching and adaptive decode rows. The profile notes carry the end-to-end SWE-bench agent results (1 agent: +37% tok/s/user with speculation; 16 agents: plain +13%; 20 agents: 12-13 tok/s/user, KV peak 68%, hit rate 95%, no thrash), the crossover at about 8 agents and the plugin changes both profiles need. The profile test also pins block size, trace region and l1_small, the settings the model's KV pool rule was calibrated with. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This was referenced Nov 16, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What / why
The dedicated
gemma4_31b_qb2implementation supports prefix caching, adaptive decode rows and n-gram speculative decoding. The dev catalog gets two profiles of the same server (32 decode slots, 262144 context, prefix caching, shared YAML anchors) that differ only in speculation and scheduling:gemma4_31b_qb2(few users): n-gram speculation chosen by the number of live requests (K = 5 for one, K = 3 up to four, none beyond: the plugin plan caps larger batches at zero drafts), synchronous scheduling (vLLM refuses async scheduling with the n-gram method),NUMBA_NUM_THREADS=1.gemma4_31b_qb2_c32(many agents): speculation off, async scheduling.The server decodes with the smallest trace that covers the live rows, so a lone user runs at the one-slot speed on either profile. Files:
workflows/model_specs/dev/llm.yaml,workflows/model_spec.py(8 lines),tests/workflows/test_gemma4_31b_qb2_profiles.py(profile tests).Tests and results
Profile tests for both entries (flags, env, health timeout 3600 s); the server runs of both profiles on the full stack: panels 8+8 identical, 1-agent replay 52.3 tok/s/user (speculative), 32 agents x 24K later-turn hits 97% (plain profile), start-up 380-410 s on an idle host.
Evaluation of the two profiles (through this TTI entry, local; source: spec-investigation/smoke/TTI_ACCEPTANCE_RESULT.md)
Launched through these profiles: plain
gemma4_31b_qb2_c32, Harbor 16: SWE-bench Verified 500 298/500 = 59.6% (CI 55.2-63.8 after retrying the 6 environment failures once; first pass 294/500 = 58.8%; 7.2 h wall), Terminal-Bench 89 37/89 = 41.6% (CI 31.9-52.0, 200.6 min); speculativegemma4_31b_qb2, 1 agent, 10 tasks: 4.15 min/trial, 47.8 tok/s/user; start-up through TTI 366-431 s warm (1501 s with an empty kernel cache). Deviations from the eval config: agent timeout 7200 s, thinking on and 32K output, TTI-default shell, Harbor 16 (local). CI is not available, so these are local numbers. Baseline A/B (unmodified main vs this stack, same conditions): done, results in the Gemma demo PR description (replay wall at 8 / 32 agents 668 -> 112 s and 1682 -> 165 s; batch-1 decode +11% to +22% plain).Depends on
tt-metal demo PR tenstorrent/tt-metal#65783 (M12 for speculation) and the vllm-tt-plugin PR tenstorrent/vllm-tt-plugin#308 (cap-zero P3 for more than four live requests).
Caveats
Pending (will be added here): thinking-off SWE shard, SPEED-Bench.
🤖 Generated with Claude Code