Skip to content

feat(gemma4): g4poc Gemma-4-26B-A4B FP8 serving on one RTX 5090 (ship) - #7

Open
jumanzii wants to merge 10 commits into
jumanzii/g4poc-ship-basefrom
jumanzii/g4poc-ship
Open

jumanzii wants to merge 10 commits into
jumanzii/g4poc-ship-basefrom
jumanzii/g4poc-ship

Conversation

@jumanzii

@jumanzii jumanzii commented Oct 6, 2026 •

Copy link
Copy Markdown

Description

TL;DR

This packages the g4poc study's adopted serving changes for google/gemma-4-26B-A4B-it on one RTX 5090 into one
reviewable branch. The study measured the official checkpoint converted offline to FP8 (compressed-tensors, per-channel
weights, dynamic per-token activations), serving non-streaming multi-turn role-play. Internal PR for team review,
not for upstream.

  • Base: upstream sgl-project a9871012ac (2026-09-21), the commit the gemma4nv/g4poc lineage forked from. Nine
    lever commits and a deployment guide sit on top.
  • Defaults: every lever is an opt-in switch, default off, except two config files that apply only on an RTX 5090.
  • The measured tree: the levers were measured on jumanzii/g4poc-r2-k2 (cfc12c0). This branch drops the code
    no shipped config runs and reproduces that tree's outputs token for token (verification below).
traffic, one RTX 5090 operating point result $/1M output @ $0.70/GPU-h
requests always in flight 28 in flight 30-min soak: p90 7.33 s, p99 8.95 s, 1,127 tok/s, 0 failed 0.173 (round 1 0.204, FP8 base 0.343)
6 s p90 SLO 16 in flight p90 5.05 s, 941 tok/s 0.207 (round 1 0.300)
chat, >= 30 s mean think ~88 sessions per GPU 10 s p90 SLO ~0.36 (round 1 ~70 sessions, 0.45-0.47)

Guide: G4POC-DEPLOY.md. It has
the FP8 checkpoint preparation, the full launch command for each traffic type, every switch and kill switch, the
results and quality tables, deployment caveats, hardware dependence and how to reproduce.

Levers (one commit each, tests in the same commit)

# commit lever switch (default) shipped in
1 02a86e4 K1: Gemma-4 decode-glue fusion (q/k/v norm + RoPE + FP8 KV store, norm pairs) SGLANG_OPT_GEMMA4_FUSED_GLUE=2 (0) both configs
2 9b68829 L8: one FP8 table for the tied embedding and LM head, with its Triton FP8 vocab-head kernel SGLANG_OPT_GEMMA4_FP8_VOCAB_TABLE=1 (off) both
3 aa13950 C2-A: SM120 FP8-KV Triton extend tiles SGLANG_OPT_TRITON_EXTEND_SM120_FP8_KV_TILES=1 (off) both
4 99a6c5a C1: tuned fused_moe config, RTX 5090, E=128 N=704 FP8 per-channel, in the default config dir config file, on for an RTX 5090 both
5 6204224 HiCache fix 1: pin the load-back window during SWA prefill admission (scheduler crash at 28 in flight) SGLANG_OPT_HICACHE_PIN_LOAD_BACK_WINDOW=1 (off) in-flight
6 2f61d4b HiCache fix 2: fence write-through copies behind the forward stream (stale load-back) SGLANG_OPT_HICACHE_FENCE_WRITE_THROUGH=1 (off) in-flight
7 1a85687 c1: RTX 5090 tiles for the six dense FP8 shapes at decode M 1-48, on upstream's tuned-tile route (sgl-project#34331) config files, on for an RTX 5090; kill SGLANG_ENABLE_FP8_GEMM_CONFIG_TUNE=0 both
8 e052e05 c2 (optional): SWA window translate without a host sync in the Triton decode replay SGLANG_OPT_SWA_DECODE_NO_HOST_SYNC=1 (off) neither
9 e652f30 SWA margin (optional): a chat's next turn can reuse its prompt under Gemma-4's template SGLANG_OPT_SWA_PREFILL_WINDOW_MARGIN=128 (0) neither
10 8cf659b G4POC-DEPLOY.md
  • Measured gains: each commit message has the measured gain, the default and the switch.
  • Defaults test: test/registered/unit/test_opt_in_switch_defaults.py pins every switch's default (off, margin 0)
    and documents the one on-by-default route.
  • K1 is not in this base. It came into the study's trees from earlier work, so here it is commit 1.

Not shipped

  • HiCache retraction-backup fence (jumanzii/hicache-retraction-fence, 7d0ba71). It fenced retraction_backup
    in PD-disaggregated decode. Later analysis found that path is not racy: the backup copies seqlen - 1 tokens, all
    written by forwards whose results the scheduler has already processed. The race is on the restore side, which
    [Fix] PD: retraction restores racing the in-flight decode forward sgl-project/sglang#42702 fixes. The fence also rewired scheduler.py outside its switch, and no shipped config runs
    PD disaggregation.
  • L5 (SGLANG_OPT_SWA_RELEASE_SLID_WINDOW, rejected) and its NaN-poison debug switch.
  • gemma4nv's NVFP4 and speculative code: MTP FP8 head, split-KV verify on CUDA, spec replay, small-M BF16 /
    weight-only GEMM routes, the multimodal FP8 LM-head copy.
  • Every experiments/ directory. They stay on jumanzii/g4poc.

Difference from the measured tree

git diff cfc12c0bac jumanzii/g4poc-ship -- python/:

  • Removed: the code listed above. All of it sat behind switches the shipped configs leave off.
  • Added: the margin (default 0).
  • Moved: the C1 MoE JSON, from SGLANG_MOE_CONFIG_DIR into python/sglang/srt/layers/moe/moe_runner/triton_utils/configs/triton_3_7_1/. Same sha256 (b3bcce12…9576).
  • Trimmed: triton_small_m_bf16_gemm.py, to L8's FP8 vocab-head path. The kernel source is byte-identical.
  • Cosmetic: one isort reorder in gemma4_causal.py and two comment edits.

Verification

All on build-server-3's RTX 5090, through the g4poc gate, under the host's unit lock. Refs and the unit script are on
jumanzii/g4poc-ship-verify (experiments/g4poc/ship/).

Ship head e652f30b1d (python/ tree bd956697e2); the doc commit on top changes only the guide. Records:
experiments/g4poc/ship/VERIFY.md and ship/runs/ on jumanzii/g4poc-ship-verify.

check result
unit tests, per commit (each commit's own tests at that commit, GPU included) and at the head all pass; head 1,375 passed, 0 failed
greedy identity vs cfc12c0, in-flight config (concurrency 1, 4 role-play sessions x 3 turns) 12/12 token-identical, identical cached_tokens
greedy identity vs cfc12c0, chat config 12/12
HiCache multi-turn exactness (16K device pool, every later turn a host load-back, vs the same flags without HiCache) 12/12; both arms also 12/12 against round 2's cfc12c0 runs of the same pair
upstream equivalence: every switch unset, dense tile route killed, MoE config dir emptied, vs upstream a987101 12/12
28 in flight, A-B-B-A vs cfc12c0 p90 7.43 / 7.36 s vs 7.34 / 7.33 s (gain 0.992); 1,125 / 1,120 vs 1,134 / 1,128 tok/s (0.992); p99 gain 1.023; 0 failed; drift <= 0.5%. Round 2's soak: 7.33 s, 1,127 tok/s
chat smoke, pthink30 at 72 63.3 live sessions, p90 5.97 s, 432 tok/s, 0 failed, 0 retracted (round 2 on the same plan: 6.06 s, 432)
GPU memory at 28 in flight plateau ~31,350 MiB, under the 31,642 MiB rule. Two samples hit the host's periodic canary; two consecutive samples read 32,097 MiB (+743 MiB for 2 s) with the process not recorded (see the guide, section 9)
server starts both configs start. The HiCache host-memory check failed 4 of 14 starts in a 28 GB scope (1 on this branch, 3 on cfc12c0); all passed on retry or a rerun pair

The code paths the shipped configs run were already identical to cfc12c0 by construction. The identity runs confirm
that end to end, so the round-2 quality results (GSM8K, tool-JSON, role-play, KL) carry over unchanged.

Known limits / open items

Related Issues/PRs

Checklist

  • Branch name follows Branch Guideline (jumanzii/g4poc-ship, as requested for this study)
  • Commit messages follow Commit Message Guideline
  • Checked Pull Request Guideline

🤖 Generated with Claude Code


CI States

Latest PR Test (Base): ❌ Run #37450538972
Latest PR Test (Extra): ❌ Run #37450538485
Latest PR Test (AMD ROCm 10): ❌ Run #37450538867

jumanzii and others added 10 commits October 6, 2026 18:16
…_FUSED_GLUE

Each Gemma4 decoder layer launched 13 small glue kernels per decode step:
q/k/v RMSNorm, RoPE, four ATen elementwise ops for the FP8 KV-cache
quantize in MHATokenToKVPool.set_kv_buffer, store_kvcache, and seven
norm launches. The new switch fuses them in levels (Gemma4FusedGlue):

- QKV_ROPE_KV (1): one Triton kernel normalizes q/k/v, applies RoPE and
  stores K and V as E4M3 at the layer's cache rows, sliding layers at the
  Triton backend's swa_out_cache_loc. It reproduces the unfused rounding
  steps (bf16 stores, the bf16-cast scale in div_, nvcc's FMA forms in
  rope.cuh), so activations and every KV-cache byte match bit for bit.
  Pools other than the static NHD FP8 pool, or other backends, keep the
  unfused path.
- ALL (2): also the post-attention norm pair (RMSNorm + FusedAddRMSNorm),
  the router / pre-FF-2 norm pair over one input, and the next layer's
  input norm (or the final norm) folded into the dual-norm epilogue.
  These reorder the norm sums against the FlashInfer kernels.

At level 2 a Gemma-4-26B-A4B decode step runs 791 kernels instead of
1,061. Measured on one RTX 5090 serving the compressed-tensors FP8
per-channel checkpoint (same-host A-B-B-A, g4poc study, COMPUTE.md
section 7 on fractalyze's jumanzii/g4poc): E2E p90 -4.4% and output
tok/s +3.9% at 28 requests in flight, -2.2% / +2.5% at 12, and chat p90
-8.1% at a 30 s mean think time. KL, GSM8K, tool-JSON, role-play and
HiCache multi-turn exactness (12/12) passed.

Default: off (Gemma4FusedGlue.OFF). Switch: SGLANG_OPT_GEMMA4_FUSED_GLUE=2,
the measured level (1 for the bit-exact KV store alone).

The tree's own RoPE + KV-write fusion (can_fuse = False in
Gemma4Attention.forward, from sgl-project#23280) stays disabled; its CUDA helper
only supports a bf16, non-hybrid pool without KV scales, none of which
holds for this checkpoint.

Tests: the fused kernels byte for byte against the unfused path
(test_gemma4_fused_qkv_rope_kv.py, test_gemma4_fused_norm_pairs.py);
every byte of both sub-pools of the hybrid SWA pool a served Gemma-4
uses, against the backend's own store (test_gemma4_fused_glue_hybrid_pool.py);
the switch's default (test_opt_in_switch_defaults.py).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…head

SGLANG_OPT_GEMMA4_FP8_VOCAB_TABLE replaces the target's tied BF16 table
(262144 x 2816, 1.375 GiB) after load with one E4M3 table with per-row
fp32 scales (0.69 GiB). The lookup gathers FP8 rows and dequantizes them;
the head runs a Triton small-M FP8 vocab-head kernel in row chunks of its
max M (48), so every batch width reads FP8. The BF16 table is dropped
before the KV pool is sized, so the freed memory goes to the pool. It
changes numerics on both the input and the output side.

The kernel (triton_small_m_bf16_gemm.py) streams each weight row block
once, upcasting the FP8 tile to bf16 for the MMA with fp32 accumulation;
its one tuned tile is the target head's shape on the RTX 5090 (443-525 us
against cuBLAS BF16's 880-925 us over M 4-48). The switch refuses an
untied head or a shape without a tuned tile.

Measured on one RTX 5090 with the compressed-tensors FP8 per-channel
checkpoint (g4poc study, memory/REPORT.md on fractalyze's jumanzii/g4poc):
weights 24.43 -> 23.74 GiB, +4 concurrent 5K-token sessions in the burst
probe (29 -> 33); GSM8K 97.0 vs 97.0, tool-JSON 40/40, role-play NLL
+0.001 nats/token, language adherence 67/80 vs 67/80.

Default: off. Switch: SGLANG_OPT_GEMMA4_FP8_VOCAB_TABLE=1.

Tests: test_gemma4_fp8_vocab_table.py (head and lookup within E4M3's
rounding bound of the BF16 table across the 48-row chunk edge, the swap
drops the BF16 table and refuses an untied head or untuned shape, the
loader's post-load pass leaves the table alone, LogitsProcessor takes the
hook); the default in test_opt_in_switch_defaults.py.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
On SM120 with an FP8 KV cache, Triton is the only prefill backend that
serves Gemma-4, and its extend tiles are sized for BF16 K/V: head_dim 512
uses 32x32 with 8 warps, head_dim 256 64x64 with 8 warps, which runs at
255 registers per thread. SGLANG_OPT_TRITON_EXTEND_SM120_FP8_KV_TILES
selects a per-head-dim table of (BLOCK_M, BLOCK_N, BLOCK_N_PREFIX,
num_warps, num_stages) for that case; everything else keeps today's tiles.

A sweep on an RTX 5090 over Gemma-4-26B-A4B prefill shapes (108-162
configs per head dim against the switch-off default) picks one stage for
both head dims: head_dim 512 keeps 32x32 but widens the prefix tile to 64
(2.3-3.8x faster per shape), head_dim 256 drops to 32x32 / 4 warps
(1.7-2.3x). The KV tile width is part of the softmax reduction order, so
outputs differ from the default by at most 0.031 (512) and 0.016 (256) on
those shapes; against fp32 attention the error equals the default's.

Measured end to end (g4poc study, COMPUTE.md section 2 on fractalyze's
jumanzii/g4poc; 4 ABBA pairs at 12 requests in flight): E2E p90 -10.3%
(gain 1.115), output tok/s +10.6%; long role-play KL 0.032 against a
0.041 limit.

Default: off. Switch: SGLANG_OPT_TRITON_EXTEND_SM120_FP8_KV_TILES=1.

Tests: test_triton_extend_sm120_fp8_kv_tiles.py (the table applies only
on sm120, an FP8 KV cache, a tuned head dim, Lq == Lv and the switch on);
the default in test_opt_in_switch_defaults.py.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…5090

Gemma-4-26B-A4B (128 experts, moe_intermediate_size 704) served from a
compressed-tensors FP8 per-channel checkpoint had no tuned Triton
fused_moe config for the RTX 5090, so every MoE layer ran the default
config. This adds one, tuned on an RTX 5090 with SGLang's fused_moe tuner
(decode M 1-32 and prefill M 256-4096, 648 configs per part) under
Triton 3.7.1. The kernel runs 31 -> 21 us at M 1, 2-5% faster at M
4-512 and 6-8% at M 1536-4096, 3% slower at M 1024.

Measured end to end (g4poc study, COMPUTE.md section 2 on fractalyze's
jumanzii/g4poc; 4 ABBA pairs at 12 requests in flight): E2E p90 -1.2%
(gain 1.012), output tok/s +1.1%. The study loaded it through
SGLANG_MOE_CONFIG_DIR; here it sits in the default config directory, so
it applies with no env on an RTX 5090 and nowhere else. There is no
separate down-projection file, so the down GEMM reuses this config as it
did in the study. To run the default config instead, point
SGLANG_MOE_CONFIG_DIR at a directory with an empty configs/ subdirectory.

Test: test_fused_moe_triton_config.py finds the file by default for this
device, dtype and quantization, for both the up and the down GEMM.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
With HiCache on a sliding-window model, a request whose prefix sits partly on
host is checked against the SWA budget while only its device-matched
last_node is pinned. Once admitted, init_load_back locks the window anchored
at best_match_node. That window can cover device-resident SWA past last_node,
which the check counted as evictable and nothing charged
(swa_host_hit_length covers only the host-only part). The allocation then
comes up short and alloc_token_slots raises out of memory, killing the
scheduler.

Seen on Gemma-4-26B-A4B (window 1024) on one RTX 5090 at 28 in-flight
multi-turn sessions: a 2071-token chunk plus a 247-token load-back request
(1019 SWA tokens to load) passed the check with rem_swa 1567 for 1442 needed;
the load-back then locked 350 device SWA tokens (two nodes above the loaded
segment), leaving 2270 free + evictable for 2318 tokens.

SGLANG_OPT_HICACHE_PIN_LOAD_BACK_WINDOW also pins best_match_node for the
admission check, so that SWA leaves the evictable budget first and an
over-committing candidate waits instead. The pin is the same scoped lock as
the last_node pin and is released on every exit path. The full-attention
side needs no change: its charge already bills every token past the device
match as a new allocation. With it the same server ran 20-32 in flight with
no failures (g4poc study, hicache/REPORT.md on fractalyze's jumanzii/g4poc),
where stock HiCache crashed at 28.

Default: off. Switch: SGLANG_OPT_HICACHE_PIN_LOAD_BACK_WINDOW=1, needed with
--enable-hierarchical-cache on a hybrid-SWA model.

Tests: TestHiCacheLoadBackSWAWindowPin in test_prefill_adder.py reproduces
the trace's numbers (the default admission over-commits the SWA pool, the
pin refuses it, and the pin is released after admission and finish); the
default in test_opt_in_switch_defaults.py.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Under the overlap scheduler, cache_finished_req runs while the next forward,
which writes the KV of the request's last output token, is still queued on
the model runner's forward stream. start_writing ordered the D2H copy only
after an event on the scheduler's current stream, so the copy could read
that KV half-written, and a later load-back restored it stale. Load-back
already waits on the forward stream (load_fence_stream); write-through did
not.

Seen on Gemma-4-26B-A4B with HiCache (write-through, kernel io,
page_first): a byte-level device/host comparison at each write ack found
one stale token per finished turn (full layer 0 and SWA layers 5-24).
Multi-turn greedy outputs at concurrency 1, with every later turn loaded
back from host, matched a device-only control in only 5 of 12 turns, the
upstream base commit included; with the fence, 12 of 12 (g4poc study,
hicache/REPORT.md on fractalyze's jumanzii/g4poc). Throughput cost -1.0
to -1.8% output tok/s at 20-32 in flight.

SGLANG_OPT_HICACHE_FENCE_WRITE_THROUGH makes the D2H stream wait on the
forward stream before each write submit. wait_stream covers only the work
queued so far, so later forwards still overlap with the copy.

Default: off. Switch: SGLANG_OPT_HICACHE_FENCE_WRITE_THROUGH=1, needed with
--enable-hierarchical-cache --hicache-write-policy write_through under the
overlap scheduler.

Tests: test_hicache_write_fence.py (the fenced copy is ordered after the
queued forwards, the unfenced one is not); the default in
test_opt_in_switch_defaults.py.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… RTX 5090

At decode batch sizes on SM120, apply_fp8_linear sends Gemma-4-26B-A4B's
per-token x per-channel FP8 dense linears (qkv, o, gate_up, down) to the
CUTLASS scaled mm, which launches 22-64 CTAs and streams the weights at
0.2-0.6 TB/s: 4.87 ms of every decode step. With the tree's tuned-config
route (sgl-project#34331: get_w8a8_channelwise_fp8_config in apply_fp8_linear),
triton_scaled_mm at the measured tiles takes 1.40-1.46 ms for the same
120 calls at M 1-48, activation quantization included (CUDA-graph
microbench, 30 rotating weight copies per shape).

The six config files give each decode M (1-48) its fastest tile and leave
M >= 64 on CUTLASS (null entries), so prefill chunks keep their kernel.
Numerics stay W8A8 (same quantized inputs, fp32 accumulation); only the
summation order changes.

Measured end to end on the stack with the decode-glue fusion (g4poc study,
COMPUTE.md section 8 on fractalyze's jumanzii/g4poc; same-host A-B-B-A):
at 28 requests in flight E2E p90 8.29 -> 7.36 s (-11.2%) and output tok/s
983 -> 1,122 (+14.1%); at 12, 5.29 -> 4.35 s (-17.8%) and +23%. KL,
GSM8K, tool-JSON, role-play and multi-turn exactness passed.

Default: on, like the route itself, but only on an RTX 5090 for these six
(N, K) shapes. Kill switch: SGLANG_ENABLE_FP8_GEMM_CONFIG_TUNE=0.

Tests: test_fp8_channelwise_rtx5090_configs.py (decode M gets a tile and
prefill keeps CUTLASS on the 5090, other devices are untouched, and on an
SM120 GPU every tuned tile matches CUTLASS to 1e-2 relative error);
test_opt_in_switch_defaults.py pins the route's default as the one switch
that is on.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…riton decode

update_sliding_window_buffer maps the CUDA-graph window-KV ids of a static
SWA pool to SWA ids by slicing the buffer with the GPU scalar
window_kv_indptr[-1]. Each slice reads that scalar on the host, so every
decode replay waits for the previous forward to finish before it launches
the next graph, and the GPU idles through the remaining prep and the graph
launch.

SGLANG_OPT_SWA_DECODE_NO_HOST_SYNC bounds the translate on the host by
bs * sliding_window_size, which no request's window exceeds, and confines it
to the filled ids with a device-side mask. The ids it writes match the
synced path; the buffer's stale tail is left unchanged.

Measured on Gemma-4-26B-A4B on an RTX 5090 (g4poc study, COMPUTE.md
section 8 on fractalyze's jumanzii/g4poc): the pre-graph gap falls from
0.47 to 0.06 ms per decode step, but the untraced step only 0.18-0.29 ms;
E2E p90 gain 1.012 at 28 in flight and 1.006 at 12, output tok/s +0.4% /
+0.6%. Outputs are exact (greedy 8/8, multi-turn 12/12). At or under the
study's 1% bar, so it ships opt-in.

Default: off. Switch: SGLANG_OPT_SWA_DECODE_NO_HOST_SYNC=1.

Tests: test_triton_swa_window_no_host_sync.py (same ids as the synced path,
stale tail untouched, no tensor read on the host); the default in
test_opt_in_switch_defaults.py.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A tree insert frees the request's SWA slots out of the sliding window first,
so the inserted prompt keeps exactly window + 1 live SWA tokens behind its
end. A chat template that renders a past assistant turn differently from
the generation prompt it was decoded under ends the next turn's shared
prefix a few tokens before that end: Gemma-4 renders `<|turn>model\n<reply>`
but prompts with `<|turn>model\n<|channel>thought\n<channel|>`, so the next
turn matches 4 tokens short. Behind that match point the live SWA run is
window + 1 - 4 tokens, below the window the match validator requires, and
the whole prefix is refused: the second turn of every chat re-prefills its
history although the tree still holds all of it.

SGLANG_OPT_SWA_PREFILL_WINDOW_MARGIN keeps that many extra SWA tokens live
behind an insert's end, and widens the lock cap on a fresh in-window leaf by
the same amount so the margin stays in the node the window-bounded LRU
refresh keeps. A prompt whose prefill insert stopped at an SWA branch point
stays request-owned until the finish insert, so swa_retain_floor also
returns prompt_len - window - margin to keep decode from freeing that span.

Measured on Gemma-4-26B-A4B on an RTX 5090 (g4poc study, COMPUTE.md
section 3 on fractalyze's jumanzii/g4poc), margin 128: on the HiCache
config with default chunking, +7.1% output tok/s and -7.3% p90 at 28 in
flight (prefix hit 0.70 -> 0.80); on the shipped in-flight config (chunk
2048 + lpm), p90 gain 1.014 and +1.3% tok/s, but retractions rose ~3x
(30-min soak: 2.67% vs 0.69%), so it is not part of the shipped configs.
Multi-turn exactness 12/12.

Known limitation: the retain floor here applies to every decoding request,
not only branch-inserted ones, so each running request holds window +
margin SWA plus its reply's; that is the likely source of the retraction
rise. The upstream version of this fix (sgl-project#42703) gives a
prompt already in the tree no floor.

Default: 0 (one window, unchanged behavior). Switch:
SGLANG_OPT_SWA_PREFILL_WINDOW_MARGIN=<tokens>, e.g. 128 (Gemma-4's
templates cut 4).

Tests: TestSWAPrefillWindowMargin in test_unified_radix_cache_unittest.py
(without the margin the next turn's short match is refused; with it the
prefix is kept, including through decode after a branch insert); the
default in test_opt_in_switch_defaults.py.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The guide stands alone for someone holding only this branch. It covers:
- the FP8 checkpoint preparation;
- the full launch command for in-flight traffic (28 in flight, $0.173 per
  1M output tokens at $0.70/GPU-hour), the 6 s SLO point (16 in flight,
  $0.207) and chat with >= 30 s think time (~88 sessions per GPU, ~$0.36);
- one line per lever with its switch, default and measured effect, and
  what was not shipped and why;
- round 1 -> round 2 results and quality;
- deployment caveats and hardware dependence;
- how to reproduce, and the checks this branch passed against the measured
  tree cfc12c0.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant