perf: FA kernel optimizations + auto-asymmetric KV + warp shuffle WHT#36

Closed

signalnine wants to merge 136 commits into

TheTom:feature/turboquant-kv-cachefrom

signalnine:feature/turboquant-kv-cache

Commits on Mar 26, 2026

Commits on Mar 27, 2026

perf: parallel k_set_rows_turbo3 + optimise KQ/V dequant — +31% decode (71→94 t/s, 98.5% of F16)

signalnine
and
claude
committed
experiment: batched byte extraction + explicit bit field pre-extract

committed
experiment: profiling modes for turbo3 decode bottleneck isolation

TheTom
and
claude
committed
fix: turbo4 SET_ROWS corruption, tail-block truncation, constant coupling (Issue #29)

seanrasch
and
claude
committed
experiment: 4-entry magnitude LUT + branchless sign (XOR trick)
TheTom
committed
experiment: force non-vec FA path for turbo3 (nl=2 vs nl=8)

TheTom
and
claude
committed
fix: stack overflow in turbo4 CPU init — 64KB array on worker thread stack

seanrasch
and
claude
committed
experiment: zero-LUT select chain — 2-level ternary, no constant memory
TheTom
committed
feat: auto-detect hardware, use 4-mag LUT on pre-M5 (+38-45% decode)

committed
experiment: 2-pair half2 LUT — only 2 constant addresses per lookup
TheTom
committed
experiment: deferred norm multiply (batch float4 * norm at end)
TheTom
committed
revert to proven 4-mag + per-element norm (deferred norm was slower)

TheTom
and
claude
committed
experiment: named-register centroid×norm — 4 constant reads upfront, zero divergence, ternary select
TheTom
committed
revert to 4-mag LUT (proven best), document all findings
TheTom
committed
experiment: inline block processing — bypass template dequant in FA inner loop
TheTom
committed
experiment: inline block WORSE on M2 (-10-15%), reverted to 4-mag

TheTom
and
claude
committed
experiment: FULLY BRANCHLESS FMA decode — zero ternary, zero memory, zero branch
TheTom
committed
final: 12 approaches tested, 4-mag LUT is the hardware limit

TheTom
and
claude
committed
experiment: SIMD SHUFFLE magnitude select — cross-lane LUT replacement
TheTom
committed
experiment: simd_shuffle 14.7 at 8K — close to 4-mag (15.1) but not better

TheTom
and
claude
committed
experiment: fused block dot — per-centroid Q accumulation, 4 constant reads total
TheTom
committed
experiment: fused block dot 8.1 at 8K — worst result, 64 comparisons kill it

TheTom
and
claude
committed
experiment: 4-mag helps M5 at 16K (+2.4%) but hurts at 32K (-7.3%)

TheTom
and
claude
committed
experiment: M5 LUT cost grows to 34% at 32K context

TheTom
and
claude
committed
feat: sparse V dequant — +12% decode at 32K, zero quality loss

TheTom
and
claude
committed
feat: sparse V dequant — +22% decode at 32K on M5, auto-enabled

TheTom
and
claude
committed
Merge remote-tracking branch 'upstream/feature/turboquant-kv-cache' into feature/turboquant-kv-cache
seanrasch
committed
fix: VEC flash-attn Q/K stride mismatch in vec_dot_fattn_vec_KQ_turbo3_0

signalnine
and
claude
committed
Merge pull request #4 from seanrasch/feature/turboquant-kv-cache
TheTom
authored
Revert "Merge pull request #4 from seanrasch/feature/turboquant-kv-cache"
TheTom
committed

Commits on Mar 28, 2026

Commits on Mar 29, 2026

Commits on Mar 31, 2026

Commits on Apr 1, 2026