Repository navigation
[Perf] [NVIDIA] mhc: Use BLOCK_K=32 for the hc_mix kernel on SM120 - #45264
Conversation
|
Ran this on an RTX PRO 5000 72GB (SM120, CUDA 13.0, sglang 0.5.22's copy of this module) at kernel level before it lands. Setup: Three things came out differently than the body describes. 1. The M <= 1024 rows of the speedup table depend on L2 state. This part has 96 MiB of L2, and the input at M = 256 ... 1024 is 10.5 ... 41.9 MB of bf16, so that half of the table is cache-resident. Timing the same tensors twice, once with a 256 MB flush before each call and once without (median of 15, us):
M >= 4096 agrees with you in absolute value and in ratio, warm or cold. Below that the speedup is a function of whether the input is resident, so the table would read differently depending on harness state. Two follow-ups:
2. BLOCK_K = 32 makes the output depend on BLOCK_M, which is the assumption the contract rests on. The comment above these constants requires K slicing / BLOCK_K / dot precision to be independent of M because "a row must produce the same bits alone and in any batch routed to this backend", and
(the five numbers are seeds 7/11/13/17/19; on an m = 8 batch the same comparison gives 1, 1, 0, 1, 0 rows differing out of 8). At BLOCK_K = 64 all three heights are bitwise identical; at BLOCK_K = 32 BM = 16 still matches BM = 32 but BM = 8 does not, at ~1 ULP. Since Small and data-dependent, but it does contradict the stated contract and nothing in CI would notice. Minimal reproducer: import torch, sglang.kernels.ops.layernorm.mhc as MH
from sglang.kernels.ops.layernorm.mhc import hc_mix_stats
K, MIX = 20480, 24
torch.manual_seed(7)
w = torch.randn(MIX, K, device="cuda", dtype=torch.float32) / K ** 0.5
x = torch.randn(4096, K, device="cuda", dtype=torch.bfloat16)
for bk in (64, 32):
MH._HC_MIX_BLOCK_K = bk
MH._HC_MIX_NUM_STAGES = 1
MH._HC_MIX_BLOCK_M = 32; a = hc_mix_stats(x, w, 1e-5)
MH._HC_MIX_BLOCK_M = 8; b = hc_mix_stats(x, w, 1e-5)
print(bk, int((a != b).any(dim=-1).sum())) # 0 at 64, ~230 at 32Unless the 8-row tile is load-bearing for the small-M case, pinning BLOCK_M to >= 16 on the SM120 arm looks like the cheaper way to keep the property. Either way a fixed-input test that asserts BM 8 / 16 / 32 are bitwise equal would be worth adding — 3. Two smaller points about the shape of the patch.
On the SM121/GB10 exclusion: fine as-is, and the reason is in the body. One thing to keep in mind for whoever touches this gate next: the imported We measured at kernel level only; we did not re-run the server, so we can't speak to the 32K-prefill +2.5% or the combined-branch table. At that margin it would be good to see the prefill pair repeated over a couple more boots — we have not characterized this card's boot-to-boot prefill spread and one pair doesn't discriminate. Happy to re-run with a different flush, shape or timing protocol if it helps settle 1 and 2. |
|
Follow-up on the same card, because the constant this patch moves is shared by two kernels and the second one goes the other way. Same setup as before (RTX PRO 5000 72GB, SM120, CUDA 13.0, the 0.5.22 copy of Partial path,
This reproduces your table on the path SM120 actually runs, including the caveats from my earlier comment: M >= 4096 matches you in value and ratio, and the M = 1 cell is a wash rather than a win. Compensated bf16x3 path, slice count fixed at
Sign reversed, at every height and in both cache states: BLOCK_K = 32 costs this kernel 1.43x to 1.57x. The kernel's grid is Why this matters for the patch as written: Suggested shape: keep the SM120 choice, but scope it to the kernel it was measured on, e.g. a separate Two things that did check out, both in your favour: Batch invariance holds on both paths at both widths, over the row counts each path can actually be launched with. 4 seeds against a 4096-row reference built from the same rows: the partial path at M = 1, 8, 32, 1024 and 8192, the bf16x3 path at M = 8192 (its wrapper asserts BLOCK_K does not cost accuracy on the compensated path. Against an fp32 mirror of Not blocking, and the partial-path gain is real; it is the granularity of the constant I would change before this lands. Happy to post the harness if useful. |
BBuf
left a comment
There was a problem hiding this comment.
[BBuf Review Agent] LGTM. Let's wait for the related CI.
|
/tag-and-rerun-ci |
|
/rerun-failed-ci |
1 similar comment
|
/rerun-failed-ci |
|
[BBuf Review Agent] @moria97 these still fail after a rerun, and I can't tell from the logs whether it's this PR. Could you take a look?
|
|
/rerun-failed-ci |
|
[BBuf Review Agent] Could you rebase this onto the latest main? It conflicts with main now. |
Measured on RTX PRO 5000 (SM120), DeepSeek-V4.1-Flash shapes: 1.42x at M=1024, 1.5x at M>=4096, never slower; rel diff ~3e-7. The value stays a per-platform constant, so the batch-invariance contract is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
9eeeda4 to
ac16bd5
Compare
|
/tag-and-rerun-ci |
2 similar comments
|
/tag-and-rerun-ci |
|
/tag-and-rerun-ci |
Select BLOCK_K lazily per input device and retain BLOCK_K=64 for non-fused and compensated bf16x3 projections. Use at least 16 rows with BK32 to preserve batch invariance. Add SM120 regression coverage and a reproducible cold/warm kernel benchmark.
Document controlled kernel speedups, small-batch costs, preserved bf16x3 performance and one-pair serving observations with the CPU huge-page limitation.
This comment has been minimized.
This comment has been minimized.
|
/rerun-failed-ci |
|
/tag-and-rerun-ci |
Remove the full benchmark and performance report from the PR; retain them as local validation artifacts. Inline row-tile selection, restore the unchanged non-fused call, and combine overlapping regression checks while preserving accuracy, batch invariance, lazy dispatch and bf16x3 isolation coverage.
|
/tag-and-rerun-ci |
2 similar comments
|
/tag-and-rerun-ci |
|
/tag-and-rerun-ci |
|
/rerun-failed-ci |
|
[BBuf Review Agent] Could you merge the latest main into this branch? CI's maintenance gate won't run the tests until it does. |
|
@BBuf The extra checks are still failing. Is there anything else I need to do, or is this PR ready to merge |
|
@BBuf Sure. The PR gate is still in its 120-minute cooldown period. I’ll merge the latest main into my branch once the cooldown is over |
|
/rerun-failed-ci |
1 similar comment
|
/rerun-failed-ci |
|
[BBuf Review Agent] @moria97 these still fail after a rerun, and I can't tell from the logs whether it's this PR. Could you take a look?
|
@BBuf I investigated the failure and it appears unrelated to the mHC changes in this PR. Upstream PR #45859 added architectures as a required keyword argument to ModelConfig, but test_rust_server_crash_dump.py does not pass it when constructing the config, causing a TypeError before model execution or the mHC kernel is reached. I compared this PR with the corresponding main branch: both the Rust interface and the test are identical. The only net changes in this PR are the mHC implementation and its regression test. It looks like the test needs to be updated on main. |
|
/rerun-failed-ci |
|
Verified this on 8x RTX PRO 5000 72GB (sm_120), same card family as the table above. Two of the Method: the three Triton kernels in
Repeat-to-repeat ratios for the two headline cells: M=1 1.222 / 1.222 / 1.219, M=256 0.749 / 0.749 / Two things worth separating, because they are both part of the change and both cost time at M=1: The absolute numbers suggest the difference is the timing instrument rather than hardware: your M=1 Would you mind re-checking the M=1 and M=8 rows with the launches captured in a CUDA graph (or just |
|
Thanks @xulunfan for the graph-mode recheck. On 2026-11-14 I reran this on two RTX PRO 5000 72GB cards and confirmed the small-M regression. The original M=1 row (23.2 -> 22.5 us, 1.03x faster) should be superseded; it is not evidence that the final implementation improves decode-size kernels. I verified that the three relevant Triton kernel bodies are byte-identical in v0.5.22, the merged PR and main at 1be399b. I extracted those functions verbatim to avoid unrelated SGLang dependencies on the new host, retained the actual production wrapper, and changed only the BK/BM launcher choices for the control arms. Setup: K=20480, MIX=24, hc_mult=4, bf16 activations, fp32 weights, tf32x3, four warps, one stage, 20 Sinkhorn iterations, rms_eps=1e-20, hc_eps=1e-6. CUDA graph: 20 calls, median of seven replays, three independently captured repeats per cell, randomized policy order, GPU timestamp events inside the graph. Warm inputs/weights reused; automatic clocks, no serving engine on the measured cards. Runtime: Torch 2.10.0a0 nv25.11 / CUDA 13.0, Triton 3.8.0 with its default PTXAS 13.3.33. Primary card, Gaussian inputs/weights (ratio >1 means slower):
This agrees with your direction and closely matches the absolute timings. M=1 is about 19.2% slower here versus your reported roughly 22%; M=8 and M=32 are about 16.5% and 7.3% slower. The second card gives M=1 8.21->9.75 us and M=8 9.78->11.42 us. With the checkpoint's actual layer-0 hc_attn_fn, scale and base, the primary card gives M=1 8.17->9.73 us and M=8 9.88->11.51 us. M=1 ablation on the primary card: BK64/original BM=8.21 us; BK32 alone=9.36 us; BM>=16 alone=8.69 us; combined=9.78 us. At M>=32 the BM floor no longer changes the original row-tile choice. The BM floor remains necessary for the bitwise batch-invariance contract: regrouping all 4096 rows into batches of eight gives 0/0/0 differing rows with production, but 149/75/225 with BK32 and the original BM8 policy (Gaussian seed 7); real checkpoint parameters also fail without the floor. BLOCK_K remains independent of M. One correction to the register-pressure explanation: the BK64 M=1/BM8 kernel uses 154 registers/thread and has no local allocation, so the large-M spill argument does not apply there. At M=4096/BM32, BK64 does hit 255 registers with 352 bytes of local allocation per thread; BK32 uses 166 registers without that allocation. Thus large-M gains and small-M losses are consistent with the resource measurements. The checkpoint has 40 main layers, so at 80 HC sites the warm M=1 delta corresponds to roughly 126 us of summed kernel time per ordinary forward. This is not a measured TPOT regression: V4.1 supports overlapping stats on a side stream, and no full-model A/B was rerun here. The original blanket "decode unchanged" statement is not justified by this kernel test and should be qualified accordingly. Thanks again for identifying the timing issue and asking that the small-M tradeoff be recorded. |
Motivation
Tune the SM120 fused mHC stats/Sinkhorn projection for larger M while preserving bitwise batch invariance. This is a workload-dependent tradeoff: measured M=1/8/32 kernels regress, while the tested M>=64 cases improve.
Modifications
Kernel validation and timing
Rechecked on 2026-11-14 on RTX PRO 5000 72GB SM120. Main at 1be399b has byte-identical relevant Triton kernel bodies to v0.5.22 and the merged PR. K=20480, MIX=24, hc_mult=4, bf16 activations, fp32 weights, tf32x3, four warps, one stage, 20 Sinkhorn iterations, rms_eps=1e-20, hc_eps=1e-6. Runtime: Torch 2.10.0a0 nv25.11 / CUDA 13.0, Triton 3.8.0 / default PTXAS 13.3.33.
Warm input/weight reuse; 20 wrapper calls per CUDA graph, median of 7 replays, 3 independent captures per cell, shuffled policy order, internal GPU timestamp events. Values are median per-call GPU times; ratio>1 means slower. This table supersedes the original M=1 speedup claim.
The small-M regression was confirmed on a second card and with real checkpoint HC parameters. Cache state matters: with a 256 MiB flush before each timed call, M=256 is 60.01->55.09 us rather than the warm values above. Flush time is excluded.
The final BM floor preserves exact per-row results when regrouping 4096 rows into batches of 8. BK32 with the old BM8 policy fails that check. The FP32 baseline and BK32 results remain numerically close; changing the reduction tile is not expected to be bitwise identical to BK64.
End-to-end scope
These are kernel measurements, not an isolated end-to-end serving result. The prior single serving A/B pair had different CPU Engram huge-page coverage between boots and cannot establish the tuning's independent throughput contribution or prove decode unchanged. No new full-model benchmark was run for this correction. The roughly126 us warm summed-kernel budget at 80 HC sites is an estimate, not a TPOT measurement; stats-stream overlap may hide some of it.
Checklist
CI States
Latest PR Test (Base): ✅ Run #42777996150
Latest PR Test (Extra): ❌ Run #42777995973
Latest PR Test (AMD ROCm 10): ⏳ Run #42777996234
Tip
Base CI only covers the core cases, and its workload may not exercise your change. Use
/rerun-testto run the tests your change touches, or copy this prompt for an agent: