Repository navigation
Conversation
…_FUSED_GLUE Each Gemma4 decoder layer launched 13 small glue kernels per decode step: q/k/v RMSNorm, RoPE, four ATen elementwise ops for the FP8 KV-cache quantize in MHATokenToKVPool.set_kv_buffer, store_kvcache, and seven norm launches. The new switch fuses them in levels (Gemma4FusedGlue): - QKV_ROPE_KV (1): one Triton kernel normalizes q/k/v, applies RoPE and stores K and V as E4M3 at the layer's cache rows, sliding layers at the Triton backend's swa_out_cache_loc. It reproduces the unfused rounding steps (bf16 stores, the bf16-cast scale in div_, nvcc's FMA forms in rope.cuh), so activations and every KV-cache byte match bit for bit. Pools other than the static NHD FP8 pool, or other backends, keep the unfused path. - ALL (2): also the post-attention norm pair (RMSNorm + FusedAddRMSNorm), the router / pre-FF-2 norm pair over one input, and the next layer's input norm (or the final norm) folded into the dual-norm epilogue. These reorder the norm sums against the FlashInfer kernels. At level 2 a Gemma-4-26B-A4B decode step runs 791 kernels instead of 1,061. Measured on one RTX 5090 serving the compressed-tensors FP8 per-channel checkpoint (same-host A-B-B-A, g4poc study, COMPUTE.md section 7 on fractalyze's jumanzii/g4poc): E2E p90 -4.4% and output tok/s +3.9% at 28 requests in flight, -2.2% / +2.5% at 12, and chat p90 -8.1% at a 30 s mean think time. KL, GSM8K, tool-JSON, role-play and HiCache multi-turn exactness (12/12) passed. Default: off (Gemma4FusedGlue.OFF). Switch: SGLANG_OPT_GEMMA4_FUSED_GLUE=2, the measured level (1 for the bit-exact KV store alone). The tree's own RoPE + KV-write fusion (can_fuse = False in Gemma4Attention.forward, from sgl-project#23280) stays disabled; its CUDA helper only supports a bf16, non-hybrid pool without KV scales, none of which holds for this checkpoint. Tests: the fused kernels byte for byte against the unfused path (test_gemma4_fused_qkv_rope_kv.py, test_gemma4_fused_norm_pairs.py); every byte of both sub-pools of the hybrid SWA pool a served Gemma-4 uses, against the backend's own store (test_gemma4_fused_glue_hybrid_pool.py); the switch's default (test_opt_in_switch_defaults.py). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…head SGLANG_OPT_GEMMA4_FP8_VOCAB_TABLE replaces the target's tied BF16 table (262144 x 2816, 1.375 GiB) after load with one E4M3 table with per-row fp32 scales (0.69 GiB). The lookup gathers FP8 rows and dequantizes them; the head runs a Triton small-M FP8 vocab-head kernel in row chunks of its max M (48), so every batch width reads FP8. The BF16 table is dropped before the KV pool is sized, so the freed memory goes to the pool. It changes numerics on both the input and the output side. The kernel (triton_small_m_bf16_gemm.py) streams each weight row block once, upcasting the FP8 tile to bf16 for the MMA with fp32 accumulation; its one tuned tile is the target head's shape on the RTX 5090 (443-525 us against cuBLAS BF16's 880-925 us over M 4-48). The switch refuses an untied head or a shape without a tuned tile. Measured on one RTX 5090 with the compressed-tensors FP8 per-channel checkpoint (g4poc study, memory/REPORT.md on fractalyze's jumanzii/g4poc): weights 24.43 -> 23.74 GiB, +4 concurrent 5K-token sessions in the burst probe (29 -> 33); GSM8K 97.0 vs 97.0, tool-JSON 40/40, role-play NLL +0.001 nats/token, language adherence 67/80 vs 67/80. Default: off. Switch: SGLANG_OPT_GEMMA4_FP8_VOCAB_TABLE=1. Tests: test_gemma4_fp8_vocab_table.py (head and lookup within E4M3's rounding bound of the BF16 table across the 48-row chunk edge, the swap drops the BF16 table and refuses an untied head or untuned shape, the loader's post-load pass leaves the table alone, LogitsProcessor takes the hook); the default in test_opt_in_switch_defaults.py. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
On SM120 with an FP8 KV cache, Triton is the only prefill backend that serves Gemma-4, and its extend tiles are sized for BF16 K/V: head_dim 512 uses 32x32 with 8 warps, head_dim 256 64x64 with 8 warps, which runs at 255 registers per thread. SGLANG_OPT_TRITON_EXTEND_SM120_FP8_KV_TILES selects a per-head-dim table of (BLOCK_M, BLOCK_N, BLOCK_N_PREFIX, num_warps, num_stages) for that case; everything else keeps today's tiles. A sweep on an RTX 5090 over Gemma-4-26B-A4B prefill shapes (108-162 configs per head dim against the switch-off default) picks one stage for both head dims: head_dim 512 keeps 32x32 but widens the prefix tile to 64 (2.3-3.8x faster per shape), head_dim 256 drops to 32x32 / 4 warps (1.7-2.3x). The KV tile width is part of the softmax reduction order, so outputs differ from the default by at most 0.031 (512) and 0.016 (256) on those shapes; against fp32 attention the error equals the default's. Measured end to end (g4poc study, COMPUTE.md section 2 on fractalyze's jumanzii/g4poc; 4 ABBA pairs at 12 requests in flight): E2E p90 -10.3% (gain 1.115), output tok/s +10.6%; long role-play KL 0.032 against a 0.041 limit. Default: off. Switch: SGLANG_OPT_TRITON_EXTEND_SM120_FP8_KV_TILES=1. Tests: test_triton_extend_sm120_fp8_kv_tiles.py (the table applies only on sm120, an FP8 KV cache, a tuned head dim, Lq == Lv and the switch on); the default in test_opt_in_switch_defaults.py. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…5090 Gemma-4-26B-A4B (128 experts, moe_intermediate_size 704) served from a compressed-tensors FP8 per-channel checkpoint had no tuned Triton fused_moe config for the RTX 5090, so every MoE layer ran the default config. This adds one, tuned on an RTX 5090 with SGLang's fused_moe tuner (decode M 1-32 and prefill M 256-4096, 648 configs per part) under Triton 3.7.1. The kernel runs 31 -> 21 us at M 1, 2-5% faster at M 4-512 and 6-8% at M 1536-4096, 3% slower at M 1024. Measured end to end (g4poc study, COMPUTE.md section 2 on fractalyze's jumanzii/g4poc; 4 ABBA pairs at 12 requests in flight): E2E p90 -1.2% (gain 1.012), output tok/s +1.1%. The study loaded it through SGLANG_MOE_CONFIG_DIR; here it sits in the default config directory, so it applies with no env on an RTX 5090 and nowhere else. There is no separate down-projection file, so the down GEMM reuses this config as it did in the study. To run the default config instead, point SGLANG_MOE_CONFIG_DIR at a directory with an empty configs/ subdirectory. Test: test_fused_moe_triton_config.py finds the file by default for this device, dtype and quantization, for both the up and the down GEMM. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
With HiCache on a sliding-window model, a request whose prefix sits partly on host is checked against the SWA budget while only its device-matched last_node is pinned. Once admitted, init_load_back locks the window anchored at best_match_node. That window can cover device-resident SWA past last_node, which the check counted as evictable and nothing charged (swa_host_hit_length covers only the host-only part). The allocation then comes up short and alloc_token_slots raises out of memory, killing the scheduler. Seen on Gemma-4-26B-A4B (window 1024) on one RTX 5090 at 28 in-flight multi-turn sessions: a 2071-token chunk plus a 247-token load-back request (1019 SWA tokens to load) passed the check with rem_swa 1567 for 1442 needed; the load-back then locked 350 device SWA tokens (two nodes above the loaded segment), leaving 2270 free + evictable for 2318 tokens. SGLANG_OPT_HICACHE_PIN_LOAD_BACK_WINDOW also pins best_match_node for the admission check, so that SWA leaves the evictable budget first and an over-committing candidate waits instead. The pin is the same scoped lock as the last_node pin and is released on every exit path. The full-attention side needs no change: its charge already bills every token past the device match as a new allocation. With it the same server ran 20-32 in flight with no failures (g4poc study, hicache/REPORT.md on fractalyze's jumanzii/g4poc), where stock HiCache crashed at 28. Default: off. Switch: SGLANG_OPT_HICACHE_PIN_LOAD_BACK_WINDOW=1, needed with --enable-hierarchical-cache on a hybrid-SWA model. Tests: TestHiCacheLoadBackSWAWindowPin in test_prefill_adder.py reproduces the trace's numbers (the default admission over-commits the SWA pool, the pin refuses it, and the pin is released after admission and finish); the default in test_opt_in_switch_defaults.py. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Under the overlap scheduler, cache_finished_req runs while the next forward, which writes the KV of the request's last output token, is still queued on the model runner's forward stream. start_writing ordered the D2H copy only after an event on the scheduler's current stream, so the copy could read that KV half-written, and a later load-back restored it stale. Load-back already waits on the forward stream (load_fence_stream); write-through did not. Seen on Gemma-4-26B-A4B with HiCache (write-through, kernel io, page_first): a byte-level device/host comparison at each write ack found one stale token per finished turn (full layer 0 and SWA layers 5-24). Multi-turn greedy outputs at concurrency 1, with every later turn loaded back from host, matched a device-only control in only 5 of 12 turns, the upstream base commit included; with the fence, 12 of 12 (g4poc study, hicache/REPORT.md on fractalyze's jumanzii/g4poc). Throughput cost -1.0 to -1.8% output tok/s at 20-32 in flight. SGLANG_OPT_HICACHE_FENCE_WRITE_THROUGH makes the D2H stream wait on the forward stream before each write submit. wait_stream covers only the work queued so far, so later forwards still overlap with the copy. Default: off. Switch: SGLANG_OPT_HICACHE_FENCE_WRITE_THROUGH=1, needed with --enable-hierarchical-cache --hicache-write-policy write_through under the overlap scheduler. Tests: test_hicache_write_fence.py (the fenced copy is ordered after the queued forwards, the unfenced one is not); the default in test_opt_in_switch_defaults.py. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… RTX 5090 At decode batch sizes on SM120, apply_fp8_linear sends Gemma-4-26B-A4B's per-token x per-channel FP8 dense linears (qkv, o, gate_up, down) to the CUTLASS scaled mm, which launches 22-64 CTAs and streams the weights at 0.2-0.6 TB/s: 4.87 ms of every decode step. With the tree's tuned-config route (sgl-project#34331: get_w8a8_channelwise_fp8_config in apply_fp8_linear), triton_scaled_mm at the measured tiles takes 1.40-1.46 ms for the same 120 calls at M 1-48, activation quantization included (CUDA-graph microbench, 30 rotating weight copies per shape). The six config files give each decode M (1-48) its fastest tile and leave M >= 64 on CUTLASS (null entries), so prefill chunks keep their kernel. Numerics stay W8A8 (same quantized inputs, fp32 accumulation); only the summation order changes. Measured end to end on the stack with the decode-glue fusion (g4poc study, COMPUTE.md section 8 on fractalyze's jumanzii/g4poc; same-host A-B-B-A): at 28 requests in flight E2E p90 8.29 -> 7.36 s (-11.2%) and output tok/s 983 -> 1,122 (+14.1%); at 12, 5.29 -> 4.35 s (-17.8%) and +23%. KL, GSM8K, tool-JSON, role-play and multi-turn exactness passed. Default: on, like the route itself, but only on an RTX 5090 for these six (N, K) shapes. Kill switch: SGLANG_ENABLE_FP8_GEMM_CONFIG_TUNE=0. Tests: test_fp8_channelwise_rtx5090_configs.py (decode M gets a tile and prefill keeps CUTLASS on the 5090, other devices are untouched, and on an SM120 GPU every tuned tile matches CUTLASS to 1e-2 relative error); test_opt_in_switch_defaults.py pins the route's default as the one switch that is on. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…riton decode update_sliding_window_buffer maps the CUDA-graph window-KV ids of a static SWA pool to SWA ids by slicing the buffer with the GPU scalar window_kv_indptr[-1]. Each slice reads that scalar on the host, so every decode replay waits for the previous forward to finish before it launches the next graph, and the GPU idles through the remaining prep and the graph launch. SGLANG_OPT_SWA_DECODE_NO_HOST_SYNC bounds the translate on the host by bs * sliding_window_size, which no request's window exceeds, and confines it to the filled ids with a device-side mask. The ids it writes match the synced path; the buffer's stale tail is left unchanged. Measured on Gemma-4-26B-A4B on an RTX 5090 (g4poc study, COMPUTE.md section 8 on fractalyze's jumanzii/g4poc): the pre-graph gap falls from 0.47 to 0.06 ms per decode step, but the untraced step only 0.18-0.29 ms; E2E p90 gain 1.012 at 28 in flight and 1.006 at 12, output tok/s +0.4% / +0.6%. Outputs are exact (greedy 8/8, multi-turn 12/12). At or under the study's 1% bar, so it ships opt-in. Default: off. Switch: SGLANG_OPT_SWA_DECODE_NO_HOST_SYNC=1. Tests: test_triton_swa_window_no_host_sync.py (same ids as the synced path, stale tail untouched, no tensor read on the host); the default in test_opt_in_switch_defaults.py. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A tree insert frees the request's SWA slots out of the sliding window first, so the inserted prompt keeps exactly window + 1 live SWA tokens behind its end. A chat template that renders a past assistant turn differently from the generation prompt it was decoded under ends the next turn's shared prefix a few tokens before that end: Gemma-4 renders `<|turn>model\n<reply>` but prompts with `<|turn>model\n<|channel>thought\n<channel|>`, so the next turn matches 4 tokens short. Behind that match point the live SWA run is window + 1 - 4 tokens, below the window the match validator requires, and the whole prefix is refused: the second turn of every chat re-prefills its history although the tree still holds all of it. SGLANG_OPT_SWA_PREFILL_WINDOW_MARGIN keeps that many extra SWA tokens live behind an insert's end, and widens the lock cap on a fresh in-window leaf by the same amount so the margin stays in the node the window-bounded LRU refresh keeps. A prompt whose prefill insert stopped at an SWA branch point stays request-owned until the finish insert, so swa_retain_floor also returns prompt_len - window - margin to keep decode from freeing that span. Measured on Gemma-4-26B-A4B on an RTX 5090 (g4poc study, COMPUTE.md section 3 on fractalyze's jumanzii/g4poc), margin 128: on the HiCache config with default chunking, +7.1% output tok/s and -7.3% p90 at 28 in flight (prefix hit 0.70 -> 0.80); on the shipped in-flight config (chunk 2048 + lpm), p90 gain 1.014 and +1.3% tok/s, but retractions rose ~3x (30-min soak: 2.67% vs 0.69%), so it is not part of the shipped configs. Multi-turn exactness 12/12. Known limitation: the retain floor here applies to every decoding request, not only branch-inserted ones, so each running request holds window + margin SWA plus its reply's; that is the likely source of the retraction rise. The upstream version of this fix (sgl-project#42703) gives a prompt already in the tree no floor. Default: 0 (one window, unchanged behavior). Switch: SGLANG_OPT_SWA_PREFILL_WINDOW_MARGIN=<tokens>, e.g. 128 (Gemma-4's templates cut 4). Tests: TestSWAPrefillWindowMargin in test_unified_radix_cache_unittest.py (without the margin the next turn's short match is refused; with it the prefix is kept, including through decode after a branch insert); the default in test_opt_in_switch_defaults.py. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The guide stands alone for someone holding only this branch. It covers: - the FP8 checkpoint preparation; - the full launch command for in-flight traffic (28 in flight, $0.173 per 1M output tokens at $0.70/GPU-hour), the 6 s SLO point (16 in flight, $0.207) and chat with >= 30 s think time (~88 sessions per GPU, ~$0.36); - one line per lever with its switch, default and measured effect, and what was not shipped and why; - round 1 -> round 2 results and quality; - deployment caveats and hardware dependence; - how to reproduce, and the checks this branch passed against the measured tree cfc12c0. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
TL;DR
This packages the g4poc study's adopted serving changes for
google/gemma-4-26B-A4B-iton one RTX 5090 into onereviewable branch. The study measured the official checkpoint converted offline to FP8 (compressed-tensors, per-channel
weights, dynamic per-token activations), serving non-streaming multi-turn role-play. Internal PR for team review,
not for upstream.
a9871012ac(2026-09-21), the commit the gemma4nv/g4poc lineage forked from. Ninelever commits and a deployment guide sit on top.
jumanzii/g4poc-r2-k2(cfc12c0). This branch drops the codeno shipped config runs and reproduces that tree's outputs token for token (verification below).
Guide:
G4POC-DEPLOY.md. It hasthe FP8 checkpoint preparation, the full launch command for each traffic type, every switch and kill switch, the
results and quality tables, deployment caveats, hardware dependence and how to reproduce.
Levers (one commit each, tests in the same commit)
SGLANG_OPT_GEMMA4_FUSED_GLUE=2(0)SGLANG_OPT_GEMMA4_FP8_VOCAB_TABLE=1(off)SGLANG_OPT_TRITON_EXTEND_SM120_FP8_KV_TILES=1(off)SGLANG_OPT_HICACHE_PIN_LOAD_BACK_WINDOW=1(off)SGLANG_OPT_HICACHE_FENCE_WRITE_THROUGH=1(off)SGLANG_ENABLE_FP8_GEMM_CONFIG_TUNE=0SGLANG_OPT_SWA_DECODE_NO_HOST_SYNC=1(off)SGLANG_OPT_SWA_PREFILL_WINDOW_MARGIN=128(0)G4POC-DEPLOY.mdtest/registered/unit/test_opt_in_switch_defaults.pypins every switch's default (off, margin 0)and documents the one on-by-default route.
Not shipped
jumanzii/hicache-retraction-fence, 7d0ba71). It fencedretraction_backupin PD-disaggregated decode. Later analysis found that path is not racy: the backup copies
seqlen - 1tokens, allwritten by forwards whose results the scheduler has already processed. The race is on the restore side, which
[Fix] PD: retraction restores racing the in-flight decode forward sgl-project/sglang#42702 fixes. The fence also rewired
scheduler.pyoutside its switch, and no shipped config runsPD disaggregation.
SGLANG_OPT_SWA_RELEASE_SLID_WINDOW, rejected) and its NaN-poison debug switch.weight-only GEMM routes, the multimodal FP8 LM-head copy.
experiments/directory. They stay onjumanzii/g4poc.Difference from the measured tree
git diff cfc12c0bac jumanzii/g4poc-ship -- python/:SGLANG_MOE_CONFIG_DIRintopython/sglang/srt/layers/moe/moe_runner/triton_utils/configs/triton_3_7_1/. Same sha256 (b3bcce12…9576).triton_small_m_bf16_gemm.py, to L8's FP8 vocab-head path. The kernel source is byte-identical.gemma4_causal.pyand two comment edits.Verification
All on build-server-3's RTX 5090, through the g4poc gate, under the host's unit lock. Refs and the unit script are on
jumanzii/g4poc-ship-verify(experiments/g4poc/ship/).Ship head
e652f30b1d(python/treebd956697e2); the doc commit on top changes only the guide. Records:experiments/g4poc/ship/VERIFY.mdandship/runs/onjumanzii/g4poc-ship-verify.cached_tokensThe code paths the shipped configs run were already identical to cfc12c0 by construction. The identity runs confirm
that end to end, so the round-2 quality results (GSM8K, tool-JSON, role-play, KL) carry over unchanged.
Known limits / open items
host pool) are tuned for a 32 GB card. Other GPUs and TP2 need re-tuning; under TP2, L8 refuses to load. See the
guide, section 7.
the weights and retry.
exact.
retraction rise. Upstream [Radix Cache] Optionally keep an SWA margin beyond the window in cached prompts sgl-project/sglang#42703 narrows it.
restore fence), [Radix Cache] Optionally keep an SWA margin beyond the window in cached prompts sgl-project/sglang#42703 (margin).
Related Issues/PRs
jumanzii/g4poc(experiments/g4poc/COMPUTE.md,memory/REPORT.md,hicache/REPORT.md)jumanzii/g4poc-r2-k2(cfc12c0)Checklist
jumanzii/g4poc-ship, as requested for this study)🤖 Generated with Claude Code
CI States
Latest PR Test (Base): ❌ Run #37450538972
Latest PR Test (Extra): ❌ Run #37450538485
Latest PR Test (AMD ROCm 10): ❌ Run #37450538867