Skip to content

[Perf] [NVIDIA] mhc: Use BLOCK_K=32 for the hc_mix kernel on SM120 - #45264

Merged
BBuf merged 6 commits into
sgl-project:mainfrom
moria97:sm120-mhc-block-k
Nov 10, 2026
Merged

BBuf merged 6 commits into
sgl-project:mainfrom
moria97:sm120-mhc-block-k

Conversation

@moria97

@moria97 moria97 commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

Motivation

Tune the SM120 fused mHC stats/Sinkhorn projection for larger M while preserving bitwise batch invariance. This is a workload-dependent tradeoff: measured M=1/8/32 kernels regress, while the tested M>=64 cases improve.

Modifications

  • Lazily select BLOCK_K=32 by input device for exact SM120 in hc_mix_stats_sinkhorn. Other architectures retain64; SM121 remains excluded.
  • Use BLOCK_M>=16 with BK32 to preserve batch invariance.
  • Keep non-fused stats and compensated bf16x3 at BK64.
  • Add focused regression coverage.

Kernel validation and timing

Rechecked on 2026-11-14 on RTX PRO 5000 72GB SM120. Main at 1be399b has byte-identical relevant Triton kernel bodies to v0.5.22 and the merged PR. K=20480, MIX=24, hc_mult=4, bf16 activations, fp32 weights, tf32x3, four warps, one stage, 20 Sinkhorn iterations, rms_eps=1e-20, hc_eps=1e-6. Runtime: Torch 2.10.0a0 nv25.11 / CUDA 13.0, Triton 3.8.0 / default PTXAS 13.3.33.

Warm input/weight reuse; 20 wrapper calls per CUDA graph, median of 7 replays, 3 independent captures per cell, shuffled policy order, internal GPU timestamp events. Values are median per-call GPU times; ratio>1 means slower. This table supersedes the original M=1 speedup claim.

M BK64 baseline (us) BK32 + BM>=16 (us) Updated / baseline
1 8.21 9.78 1.192
8 9.88 11.51 1.165
32 11.41 12.25 1.073
64 16.23 12.87 0.793
128 21.57 18.98 0.880
256 36.53 27.29 0.747
1024 125.90 88.63 0.704
4096 712.38 469.86 0.660
16384 2914.61 1858.61 0.638

The small-M regression was confirmed on a second card and with real checkpoint HC parameters. Cache state matters: with a 256 MiB flush before each timed call, M=256 is 60.01->55.09 us rather than the warm values above. Flush time is excluded.

The final BM floor preserves exact per-row results when regrouping 4096 rows into batches of 8. BK32 with the old BM8 policy fails that check. The FP32 baseline and BK32 results remain numerically close; changing the reduction tile is not expected to be bitwise identical to BK64.

End-to-end scope

These are kernel measurements, not an isolated end-to-end serving result. The prior single serving A/B pair had different CPU Engram huge-page coverage between boots and cannot establish the tuning's independent throughput contribution or prove decode unchanged. No new full-model benchmark was run for this correction. The roughly126 us warm summed-kernel budget at 80 HC sites is an estimate, not a TPOT measurement; stats-stream overlap may hide some of it.

Checklist

  • Scoped implementation and targeted regression coverage
  • CUDA-graph timing with explicit small-M tradeoff
  • Original M=1 and blanket decode-unchanged claims superseded by the graph measurements above

CI States

Latest PR Test (Base): ✅ Run #42777996150
Latest PR Test (Extra): ❌ Run #42777995973
Latest PR Test (AMD ROCm 10): ⏳ Run #42777996234

Tip

Base CI only covers the core cases, and its workload may not exercise your change. Use /rerun-test to run the tests your change touches, or copy this prompt for an agent:

Find the registered tests this PR affects and run them with /rerun-test, following https://github.com/sgl-project/sglang/blob/main/.agents/skills/rerun-test/SKILL.md

@moria97 moria97 changed the title [DSv4] mHC: use BLOCK_K=32 for the hc_mix kernel on SM120 [Perf] [NVIDIA] mhc: Use BLOCK_K=32 for the hc_mix kernel on SM120 Oct 20, 2026
@moria97

moria97 commented Oct 20, 2026

Copy link
Copy Markdown
Contributor Author

@BBuf @HaiShaw Could you take a look and /tag-and-rerun-ci when you get a chance? One-line SM120-only change: mHC hc_mix BLOCK_K=32 (1.4-1.55x kernel speedup at M>=1024, still a per-platform constant so batch invariance holds). Thanks!

@xulunfan

xulunfan commented Nov 3, 2026

Copy link
Copy Markdown
Contributor

Ran this on an RTX PRO 5000 72GB (SM120, CUDA 13.0, sglang 0.5.22's copy of this module) at kernel level before it lands. Setup: hc_mix_stats_sinkhorn directly — that is the branch SM120 reaches, since is_blackwell puts it in use_fused_stats while the tf32/bf16 projection gate requires sm90/sm100 and leaves parts/bf16_parts None — with K = 20480 (hc_mult 4 x hidden_size 5120, the DeepSeek-V4.1-Flash shape), MIX = 24, sinkhorn_iters = 20, and the file's other constants untouched (num_warps 4, num_stages 1, tf32x3, BLOCK_M 8/16/32). BLOCK_K and num_stages are re-assigned as module globals; both are read per launch, so one process covers both arms. Accuracy reproduces: rel diff vs BLOCK_K=64 is 1.8e-7 ... 3.9e-7 across pre/post/comb and M = 1 ... 16384, matching your ~3e-7, and each (M, BLOCK_K) output is bitwise stable across repeat calls in the same process.

Three things came out differently than the body describes.

1. The M <= 1024 rows of the speedup table depend on L2 state.

This part has 96 MiB of L2, and the input at M = 256 ... 1024 is 10.5 ... 41.9 MB of bf16, so that half of the table is cache-resident. Timing the same tensors twice, once with a 256 MB flush before each call and once without (median of 15, us):

M cold BK=64 cold BK=32 cold speedup warm BK=64 warm BK=32 warm speedup your table
1 38.2 36.5 1.05x 34.1 33.3 1.02x 1.03x
256 78.7 74.4 1.06x 55.6 46.3 1.20x 1.38x
1024 190.3 170.4 1.12x 147.5 112.0 1.32x 1.42x
4096 729.7 509.3 1.43x 714.4 483.9 1.48x 1.51x
16384 2804.3 1844.0 1.52x 2883.1 1868.4 1.54x 1.55x

M >= 4096 agrees with you in absolute value and in ratio, warm or cold. Below that the speedup is a function of whether the input is resident, so the table would read differently depending on harness state. Two follow-ups:

  • Could you post the timing harness (or say which entry point it calls — hc_mix_stats vs hc_mix_stats_sinkhorn — the exact K/MIX, and whether L2 is flushed)? Our warm BK=64 baseline is still 1.2-1.5x above your M = 1024/256 rows, so we're not sure we are timing the same thing, and the "1.4-1.55x at M >= 1024" line in the code comment is the part that will be read as a durable claim.
  • "never slower" is the piece we would soften. At M = 1 the sign is not stable: across two independent sweeps the cold medians went 35.4/36.0 us (BK=32 ~2% behind) and 38.2/36.5 us (BK=32 5% ahead), while the min-of-15 in the first sweep was 34.2/33.0. We'd call that cell indistinguishable rather than a win.

num_stages = 2 is worse at every M for both widths (M = 1024: 475.8 / 421.2 us vs 189.2 / 169.3), so there is no stage setting that changes this picture.

2. BLOCK_K = 32 makes the output depend on BLOCK_M, which is the assumption the contract rests on.

The comment above these constants requires K slicing / BLOCK_K / dot precision to be independent of M because "a row must produce the same bits alone and in any batch routed to this backend", and _block_m_for is allowed to vary with M specifically because the row-tile heights preserve each row's reduction order. Holding the input tensor fixed and changing only _HC_MIX_BLOCK_M — which is what two different batch sizes produce in production — hc_mix_stats gives:

BLOCK_K BM=16 vs BM=32 BM=8 vs BM=32, rows differing out of 4096 max rel
64 0 0 0
32 0 230, 211, 221, 241, 240 ~1.0e-7

(the five numbers are seeds 7/11/13/17/19; on an m = 8 batch the same comparison gives 1, 1, 0, 1, 0 rows differing out of 8). At BLOCK_K = 64 all three heights are bitwise identical; at BLOCK_K = 32 BM = 16 still matches BM = 32 but BM = 8 does not, at ~1 ULP. Since _block_m_for returns 8 for m <= 8, that height is what decode with batch <= 8 and short prefills use, so with this patch a row's bits start depending on the size of the batch it was routed in. In the fused hc_mix_stats_sinkhorn that SM120 actually runs the same asymmetry appears but is much rarer — for seed 7 we saw 1 of 4096 rows in pre/post and 4 in comb, for other seeds zero; every BLOCK_K = 64 cell was exact, including the alone-vs-in-batch comparison.

Small and data-dependent, but it does contradict the stated contract and nothing in CI would notice. Minimal reproducer:

import torch, sglang.kernels.ops.layernorm.mhc as MH
from sglang.kernels.ops.layernorm.mhc import hc_mix_stats
K, MIX = 20480, 24
torch.manual_seed(7)
w = torch.randn(MIX, K, device="cuda", dtype=torch.float32) / K ** 0.5
x = torch.randn(4096, K, device="cuda", dtype=torch.bfloat16)
for bk in (64, 32):
    MH._HC_MIX_BLOCK_K = bk
    MH._HC_MIX_NUM_STAGES = 1
    MH._HC_MIX_BLOCK_M = 32; a = hc_mix_stats(x, w, 1e-5)
    MH._HC_MIX_BLOCK_M = 8;  b = hc_mix_stats(x, w, 1e-5)
    print(bk, int((a != b).any(dim=-1).sum()))      # 0 at 64, ~230 at 32

Unless the 8-row tile is load-bearing for the small-M case, pinning BLOCK_M to >= 16 on the SM120 arm looks like the cheaper way to keep the property. Either way a fixed-input test that asserts BM 8 / 16 / 32 are bitwise equal would be worth adding — test/registered/kernels/ops/layernorm/test_mhc_kernels.py has no BLOCK_K or invariance coverage today, which is why this is on you to notice rather than on CI.

3. Two smaller points about the shape of the patch.

  • _num_slices_for derives the split-K count from k // _HC_MIX_BLOCK_K, so BLOCK_K is not purely a tile width. For K = 20480 the halving gives 320 -> 640 blocks and num_slices stays 80 in both (4 -> 8 dots per slice), so on this model it is a pure width change and that is probably why the numbers look clean; but for a K where the block count stops being divisible by the leading slice choice after halving, the same one-line patch also changes the split count, which is a different arithmetic change under the same "per-platform constant" justification. Worth one line in the comment so a future reader does not treat BLOCK_K as tile-width-only. (hc_mix_stats_sinkhorn_bf16x3 passes the constant to an assert on k % (16 * BLOCK_K), which 20480 satisfies at both widths.)
  • _HC_MIX_BLOCK_K = 32 if is_sm120() else 64 is evaluated at module import, and is_sm120() calls torch.cuda.get_device_capability(). We measured this on the SM120 box: importing sglang.kernels.ops.layernorm.mhc leaves torch.cuda.is_initialized() False, and calling is_sm120() flips it to True. mhc.py is reached at module scope by python/sglang/srt/layers/layer_boundary/residual/mhc.py:21, so this patch adds CUDA context creation to whatever imports mhc.py; we did not trace whether that includes the scheduler's parent process before it forks. The repo guards the same hazard deliberately in get_device_sm() (srt/utils/common.py), which routes through NVML while torch.cuda is uninitialized with the note that initializing torch.cuda there "would create a context and poison fork() startup". Reading the constant lazily inside the wrapper functions — which already read the global per launch — would keep the tuning and avoid the import-time context.

On the SM121/GB10 exclusion: fine as-is, and the reason is in the body. One thing to keep in mind for whoever touches this gate next: the imported is_sm120() is exact (12, 0), whereas get_platform().is_sm120 is aliased to is_sm120_supported and matches on the major, so it includes SM121 — rewriting this as the platform property would silently widen the scope past what you measured.

We measured at kernel level only; we did not re-run the server, so we can't speak to the 32K-prefill +2.5% or the combined-branch table. At that margin it would be good to see the prefill pair repeated over a couple more boots — we have not characterized this card's boot-to-boot prefill spread and one pair doesn't discriminate.

Happy to re-run with a different flush, shape or timing protocol if it helps settle 1 and 2.

@xulunfan

xulunfan commented Nov 3, 2026

Copy link
Copy Markdown
Contributor

Follow-up on the same card, because the constant this patch moves is shared by two kernels and the second one goes the other way.

Same setup as before (RTX PRO 5000 72GB, SM120, CUDA 13.0, the 0.5.22 copy of mhc.py, K = 20480, MIX = 24, num_stages 1, num_warps 4), with rms_eps = 1e-20, hc_eps = 1e-6 and sinkhorn_iters = 20 taken from a DeepSeek-V4.1-Flash config rather than assumed. I ran _HC_MIX_BLOCK_K = 64 and 32 against both consumers of it — hc_mix_stats_sinkhorn (the tf32x3 partial path, which is what SM120 reaches today) and hc_mix_stats_sinkhorn_bf16x3 (the compensated path). Median of 15, us, with and without a 256 MB L2 flush before each call; ratio is BK64/BK32, so > 1 means 32 is faster.

Partial path, num_slices = 80 at both widths:

M cold BK=64 cold BK=32 ratio warm BK=64 warm BK=32 ratio
1 35.3 35.4 1.00x 31.8 31.7 1.00x
256 77.9 73.0 1.07x 54.5 45.8 1.19x
1024 188.1 169.0 1.11x 146.3 111.3 1.32x
4096 730.3 503.7 1.45x 714.0 479.5 1.49x
8192 1415.7 947.1 1.49x 1402.9 928.0 1.51x
16384 2873.7 1883.7 1.53x 2911.8 1870.5 1.56x
65536 11833.0 7494.2 1.58x 11863.3 7485.1 1.58x

This reproduces your table on the path SM120 actually runs, including the caveats from my earlier comment: M >= 4096 matches you in value and ratio, and the M = 1 cell is a wash rather than a win.

Compensated bf16x3 path, slice count fixed at _HC_MIX_COMPENSATED_SLICES = 16 regardless of width:

M cold BK=64 cold BK=32 ratio warm BK=64 warm BK=32 ratio
4096 206.0 300.5 0.69x 191.4 273.6 0.70x
8192 383.4 558.6 0.69x 357.5 522.1 0.68x
16384 717.4 1061.7 0.68x 675.5 1016.7 0.66x
65536 2628.8 4086.5 0.64x 2574.7 4051.5 0.64x

Sign reversed, at every height and in both cache states: BLOCK_K = 32 costs this kernel 1.43x to 1.57x. The kernel's grid is (cdiv(m, BLOCK_M), 16) and _HC_MIX_BLOCK_K does not enter it, so halving the width does not add parallelism — it only doubles the inner loop from 1280 / 64 = 20 to 1280 / 32 = 40 iterations, each carrying three tl.dot calls plus the sq reduction. The partial path at least gets a different BLOCK_M out of _block_m_for, which may be why the two react oppositely; I have not confirmed that with ncu.

Why this matters for the patch as written: hc_mix_stats_sinkhorn_bf16x3 launches with BLOCK_K=_HC_MIX_BLOCK_K and its only guard is assert k % (slices * _HC_MIX_BLOCK_K) == 0, which 20480 % (16 * 32) == 0 passes, so it silently takes the slower tile. Nothing breaks today because the bf16x3 route is gated on is_sm100 or is_sm90 in deepseek_v4.py / deepseek_v4_mhc.py, so SM120 never reaches it. #47658 opens that gate for SM120, and the two changes then compose into the 300 us arm above instead of the 206 us one. Note the absolute numbers too: bf16x3 at BK = 64 is still 2.4x faster than this patch's BK = 32 partial path at M = 4096 (206.0 us against 503.7 us) and 2.9x faster at M = 65536, so the two knobs are not comparable in size and one constant cannot serve both.

Suggested shape: keep the SM120 choice, but scope it to the kernel it was measured on, e.g. a separate _HC_MIX_BF16X3_BLOCK_K = 64 next to _HC_MIX_BF16X3_BLOCK_M and pass it at that launch site, rather than moving the shared _HC_MIX_BLOCK_K. Then whichever of the two PRs merges second inherits neither the gain nor the loss by accident.

Two things that did check out, both in your favour:

Batch invariance holds on both paths at both widths, over the row counts each path can actually be launched with. 4 seeds against a 4096-row reference built from the same rows: the partial path at M = 1, 8, 32, 1024 and 8192, the bf16x3 path at M = 8192 (its wrapper asserts 4096 <= m off SM90, so it has no decode row counts to test) — 48 comparisons in all, and every one of them reports zero rows differing in pre, post or comb. To be precise about my earlier note on BLOCK_M sensitivity — that measurement was on hc_mix_stats, the non-fused projection, which SM120 does not call; the fused path SM120 runs is clean here.

BLOCK_K does not cost accuracy on the compensated path. Against an fp32 mirror of _hc_mix_reduce_sinkhorn_kernel (shared by both wrappers), bf16x3 gives max|d| 1.788e-06 / 4.590e-06 / 8.941e-07 and max_rel 1.03e-05 / 1.18e-05 / 8.01e-06 for pre / post / comb, and those figures are identical at 64 and at 32. The two widths are not bitwise equal on either path — at M = 4096 bf16x3 differs in 1285 / 1378 / 9428 entries with max|d| <= 2.4e-07, the partial path in 8914 / 9433 / 51133 with max|d| <= 8.3e-07 — consistent with the ~1e-7 you report.

Not blocking, and the partial-path gain is real; it is the granularity of the constant I would change before this lands. Happy to post the harness if useful.

@BBuf BBuf left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[BBuf Review Agent] LGTM. Let's wait for the related CI.

@BBuf

BBuf commented Nov 3, 2026

Copy link
Copy Markdown
Collaborator

/tag-and-rerun-ci

@github-actions github-actions Bot added the run-ci CI: run the baseline test suite on this PR label Nov 3, 2026
@BBuf

BBuf commented Nov 4, 2026

Copy link
Copy Markdown
Collaborator

/rerun-failed-ci

1 similar comment
@BBuf

BBuf commented Nov 5, 2026

Copy link
Copy Markdown
Collaborator

/rerun-failed-ci

@BBuf

BBuf commented Nov 5, 2026

Copy link
Copy Markdown
Collaborator

[BBuf Review Agent] @moria97 these still fail after a rerun, and I can't tell from the logs whether it's this PR. Could you take a look?

  • base-b-test-1-gpu-large (0): Post job cleanup." The supplied tail contains only cleanup activity and a Node.js deprecation warning; it includes no failing test, exception, or infrastructure error that identifies the cause.

@BBuf

BBuf commented Nov 5, 2026

Copy link
Copy Markdown
Collaborator

/rerun-failed-ci

@BBuf

BBuf commented Nov 6, 2026

Copy link
Copy Markdown
Collaborator

[BBuf Review Agent] Could you rebase this onto the latest main? It conflicts with main now.

Measured on RTX PRO 5000 (SM120), DeepSeek-V4.1-Flash shapes: 1.42x at
M=1024, 1.5x at M>=4096, never slower; rel diff ~3e-7. The value stays a
per-platform constant, so the batch-invariance contract is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@BBuf BBuf left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[BBuf Review Agent] LGTM.

@BBuf

BBuf commented Nov 9, 2026

Copy link
Copy Markdown
Collaborator

/tag-and-rerun-ci

2 similar comments
@BBuf

BBuf commented Nov 9, 2026

Copy link
Copy Markdown
Collaborator

/tag-and-rerun-ci

@BBuf

BBuf commented Nov 9, 2026

Copy link
Copy Markdown
Collaborator

/tag-and-rerun-ci

Select BLOCK_K lazily per input device and retain BLOCK_K=64 for non-fused and compensated bf16x3 projections. Use at least 16 rows with BK32 to preserve batch invariance. Add SM120 regression coverage and a reproducible cold/warm kernel benchmark.
Document controlled kernel speedups, small-batch costs, preserved bf16x3 performance and one-pair serving observations with the CPU huge-page limitation.
@github-actions github-actions Bot added the documentation Improvements or additions to documentation label Nov 9, 2026
@github-actions

This comment has been minimized.

@BBuf

BBuf commented Nov 9, 2026

Copy link
Copy Markdown
Collaborator

/rerun-failed-ci

@BBuf

BBuf commented Nov 9, 2026

Copy link
Copy Markdown
Collaborator

/tag-and-rerun-ci

Remove the full benchmark and performance report from the PR; retain them as local validation artifacts. Inline row-tile selection, restore the unchanged non-fused call, and combine overlapping regression checks while preserving accuracy, batch invariance, lazy dispatch and bf16x3 isolation coverage.

@BBuf BBuf left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[BBuf Review Agent] LGTM.

@BBuf

BBuf commented Nov 9, 2026

Copy link
Copy Markdown
Collaborator

/tag-and-rerun-ci

2 similar comments
@BBuf

BBuf commented Nov 9, 2026

Copy link
Copy Markdown
Collaborator

/tag-and-rerun-ci

@BBuf

BBuf commented Nov 9, 2026

Copy link
Copy Markdown
Collaborator

/tag-and-rerun-ci

@BBuf

BBuf commented Nov 9, 2026

Copy link
Copy Markdown
Collaborator

/rerun-failed-ci

@BBuf

BBuf commented Nov 9, 2026

Copy link
Copy Markdown
Collaborator

[BBuf Review Agent] Could you merge the latest main into this branch? CI's maintenance gate won't run the tests until it does.

@moria97

moria97 commented Nov 10, 2026

Copy link
Copy Markdown
Contributor Author

@BBuf The extra checks are still failing. Is there anything else I need to do, or is this PR ready to merge

@BBuf

BBuf commented Nov 10, 2026

Copy link
Copy Markdown
Collaborator

[BBuf Review Agent] @moria97 Could you merge the latest main first? f699d1c still branches from a1cb9e4, and I can't clear the Extra failures for merge from the status badges alone.

@moria97

moria97 commented Nov 10, 2026

Copy link
Copy Markdown
Contributor Author

@BBuf Sure. The PR gate is still in its 120-minute cooldown period. I’ll merge the latest main into my branch once the cooldown is over

@BBuf

BBuf commented Nov 10, 2026

Copy link
Copy Markdown
Collaborator

/rerun-failed-ci

1 similar comment
@BBuf

BBuf commented Nov 10, 2026

Copy link
Copy Markdown
Collaborator

/rerun-failed-ci

@BBuf

BBuf commented Nov 10, 2026

Copy link
Copy Markdown
Collaborator

[BBuf Review Agent] @moria97 these still fail after a rerun, and I can't tell from the logs whether it's this PR. Could you take a look?

  • base-b-test-1-gpu-small (7): {"file": "test/registered/rust/test_rust_server_crash_dump.py", "passed": false, "elapsed": 13}" identifies the failed test file, but the tail contains no exception, assertion, or stack trace establis

@moria97

moria97 commented Nov 10, 2026

Copy link
Copy Markdown
Contributor Author

[BBuf Review Agent] @moria97 these still fail after a rerun, and I can't tell from the logs whether it's this PR. Could you take a look?

  • base-b-test-1-gpu-small (7): {"file": "test/registered/rust/test_rust_server_crash_dump.py", "passed": false, "elapsed": 13}" identifies the failed test file, but the tail contains no exception, assertion, or stack trace establis

@BBuf I investigated the failure and it appears unrelated to the mHC changes in this PR. Upstream PR #45859 added architectures as a required keyword argument to ModelConfig, but test_rust_server_crash_dump.py does not pass it when constructing the config, causing a TypeError before model execution or the mHC kernel is reached.

I compared this PR with the corresponding main branch: both the Rust interface and the test are identical. The only net changes in this PR are the mHC implementation and its regression test. It looks like the test needs to be updated on main.

@BBuf

BBuf commented Nov 10, 2026

Copy link
Copy Markdown
Collaborator

/rerun-failed-ci

@BBuf
BBuf merged commit d684cc9 into sgl-project:main Nov 10, 2026
158 of 170 checks passed
@xulunfan

Copy link
Copy Markdown
Contributor

Verified this on 8x RTX PRO 5000 72GB (sm_120), same card family as the table above. Two of the
rows reproduce exactly, the M=1 row does not, and the sign flips.

Method: the three Triton kernels in hc_mix_stats_sinkhorn are byte-identical between v0.5.22 and
current main, so I built both configs in one process by rewriting only the launcher arguments
(BLOCK_K, and the block_m = max(16, _block_m_for(m)) floor) on top of the identical kernels.
CUDA-graph timing, 20 launches per graph replay, median of 7 replays, three independent repeats per
cell, fp32 hc_fn of the checkpoint geometry (24, 20480). µs:

M BLOCK_K=64 (pre-PR) BLOCK_K=32, BLOCK_M floor 16 (this PR) this PR / pre-PR
1 8.00 9.72 1.222 (slower)
8 9.82 11.44 1.166 (slower)
32 11.33 12.19 1.076 (slower)
64 16.14 12.83 0.795 (faster)
128 21.44 18.83 0.878 (faster)
256 36.41 27.22 0.748 (faster)
1024 125.28 89.04 0.711 (faster)

Repeat-to-repeat ratios for the two headline cells: M=1 1.222 / 1.222 / 1.219, M=256 0.749 / 0.749 /
0.752. Your M=256 (37.4 -> 27.2), M=1024 (125.6 -> 88.3) cells match mine to about 2%, so the two
measurements agree above M=64. Below that they disagree in direction: you report M=1 getting 1.03x
faster (23.2 -> 22.5), I measure 1.22x slower (8.00 -> 9.72).

Two things worth separating, because they are both part of the change and both cost time at M=1:
BLOCK_K=32 alone is 8.00 -> 9.03 (+12.9%), and the block_m >= 16 floor alone is 8.00 -> 8.47
(+5.9%). Above M=64 the floor is free (16.14 -> 16.15, 36.41 -> 36.44) and BLOCK_K=32 is the whole
gain.

The absolute numbers suggest the difference is the timing instrument rather than hardware: your M=1
row is 23.2us where the graph-mode pair costs 8.0us, so roughly 14us of per-launch overhead is
inside your measurement, which is about 1.8x the effect being measured. Since hc_mix runs 80 times
per forward step, +1.7us at M=1 is ~+140us of GPU time per decode step, and decode is the case where
this path is taken at M=1.

Would you mind re-checking the M=1 and M=8 rows with the launches captured in a CUDA graph (or just
with torch.profiler kernel durations rather than wall time)? If the regression is real, I am not
suggesting a shape-dependent constant, since keeping BLOCK_K independent of M is exactly what the
batch-invariance comment in the kernel asks for — I only wanted the measurement on record. Happy to
re-run anything here with a different operand distribution or with the real hc_attn_fn if that
helps.

@moria97

moria97 commented Nov 16, 2026

Copy link
Copy Markdown
Contributor Author

Thanks @xulunfan for the graph-mode recheck. On 2026-11-14 I reran this on two RTX PRO 5000 72GB cards and confirmed the small-M regression. The original M=1 row (23.2 -> 22.5 us, 1.03x faster) should be superseded; it is not evidence that the final implementation improves decode-size kernels.

I verified that the three relevant Triton kernel bodies are byte-identical in v0.5.22, the merged PR and main at 1be399b. I extracted those functions verbatim to avoid unrelated SGLang dependencies on the new host, retained the actual production wrapper, and changed only the BK/BM launcher choices for the control arms. Setup: K=20480, MIX=24, hc_mult=4, bf16 activations, fp32 weights, tf32x3, four warps, one stage, 20 Sinkhorn iterations, rms_eps=1e-20, hc_eps=1e-6. CUDA graph: 20 calls, median of seven replays, three independently captured repeats per cell, randomized policy order, GPU timestamp events inside the graph. Warm inputs/weights reused; automatic clocks, no serving engine on the measured cards. Runtime: Torch 2.10.0a0 nv25.11 / CUDA 13.0, Triton 3.8.0 with its default PTXAS 13.3.33.

Primary card, Gaussian inputs/weights (ratio >1 means slower):

M BK64 baseline (us) BK32 + BM>=16 (us) Updated / baseline
1 8.21 9.78 1.192
8 9.88 11.51 1.165
32 11.41 12.25 1.073
64 16.23 12.87 0.793
128 21.57 18.98 0.880
256 36.53 27.29 0.747
1024 125.90 88.63 0.704

This agrees with your direction and closely matches the absolute timings. M=1 is about 19.2% slower here versus your reported roughly 22%; M=8 and M=32 are about 16.5% and 7.3% slower. The second card gives M=1 8.21->9.75 us and M=8 9.78->11.42 us. With the checkpoint's actual layer-0 hc_attn_fn, scale and base, the primary card gives M=1 8.17->9.73 us and M=8 9.88->11.51 us.

M=1 ablation on the primary card: BK64/original BM=8.21 us; BK32 alone=9.36 us; BM>=16 alone=8.69 us; combined=9.78 us. At M>=32 the BM floor no longer changes the original row-tile choice.

The BM floor remains necessary for the bitwise batch-invariance contract: regrouping all 4096 rows into batches of eight gives 0/0/0 differing rows with production, but 149/75/225 with BK32 and the original BM8 policy (Gaussian seed 7); real checkpoint parameters also fail without the floor. BLOCK_K remains independent of M.

One correction to the register-pressure explanation: the BK64 M=1/BM8 kernel uses 154 registers/thread and has no local allocation, so the large-M spill argument does not apply there. At M=4096/BM32, BK64 does hit 255 registers with 352 bytes of local allocation per thread; BK32 uses 166 registers without that allocation. Thus large-M gains and small-M losses are consistent with the resource measurements.

The checkpoint has 40 main layers, so at 80 HC sites the warm M=1 delta corresponds to roughly 126 us of summed kernel time per ordinary forward. This is not a measured TPOT regression: V4.1 supports overlapping stats on a side stream, and no full-model A/B was rerun here. The original blanket "decode unchanged" statement is not justified by this kernel test and should be qualified accordingly.

Thanks again for identifying the timing issue and asking that the small-M tradeoff be recorded.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation jit-kernel run-ci CI: run the baseline test suite on this PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants