Trellis quants with CPU inference#441

Merged

ikawrakow merged 73 commits into

ikawrakow:mainfrom

andrewkchan:andrewkchan/try_trellis

May 23, 2025

Commits on Nov 21, 2024

WIP
Iwan Kawrakow
committed
WIP
Iwan Kawrakow
committed
WIP
Iwan Kawrakow
committed
Testing Trellis quantization
Iwan Kawrakow
committed
Testing Trellis quantization: 4-bit quantized block scales
Iwan Kawrakow
committed
Testing Trellis quantization: playing with scales and generators
Iwan Kawrakow
committed
iq2_kt: quantize / dequantize
Iwan Kawrakow
committed
iq2_kt: CUDA dequantize
Iwan Kawrakow
committed
WIP
Iwan Kawrakow
committed
WIP
Iwan Kawrakow
committed
WIP - try larger blocks
Iwan Kawrakow
committed
iq2_kt - this is better
Iwan Kawrakow
committed
iq2_kt - even better
Iwan Kawrakow
committed
iq2_kt: CUDA dot product
Iwan Kawrakow
committed
iq2_kt: very slightly faster CUDA dot product
Iwan Kawrakow
committed
iq2_kt: f16 CUDA dot product
Iwan Kawrakow
committed
iq2_kt: faster f16 CUDA dot product
Iwan Kawrakow
committed
iq2_kt: faster f16 CUDA dot product
Iwan Kawrakow
committed
Minor
Iwan Kawrakow
committed
Adding iq3_kt
Iwan Kawrakow
committed
Forgotten change
Iwan Kawrakow
committed
WIP
Iwan Kawrakow
committed
WIP
Iwan Kawrakow
committed
iq3_kt WIP: slowly improving
Iwan Kawrakow
committed
WIP
Iwan Kawrakow
committed
iq3_kt WIP: slowly improving
Iwan Kawrakow
committed
iq3_kt WIP: slowly improving
Iwan Kawrakow
committed
iq3_kt WIP: speed up quantization
Iwan Kawrakow
committed
iq3_kt speed up quantization
Iwan Kawrakow
committed
iq3_kt: CUDA dot product
Iwan Kawrakow
committed
iq2_kt: SOTA
Iwan Kawrakow
committed
iq2_kt: SOTA
Iwan Kawrakow
committed
Adding iq4_kt - not competitive at this point
Iwan Kawrakow
committed
WIP
Iwan Kawrakow
committed
WIP
Iwan Kawrakow
committed
iq4_kt: CUDA dot product
Iwan Kawrakow
committed
iq4_kt: minor tweaks
Iwan Kawrakow
committed
iq2_kt: SOTA
Iwan Kawrakow
committed
iq2_kt: SOTA
Iwan Kawrakow
committed
iq3_kt: small improvements and faster quantization
Iwan Kawrakow
committed
iq2_kt: SOTA
Iwan Kawrakow
committed
iq3_kt: small progress
Iwan Kawrakow
committed
WIP
Iwan Kawrakow
committed
iq4_kt: go to 4.0 bpw
Iwan Kawrakow
committed
iq4_kt: very slightly better
Iwan Kawrakow
committed
iq4_kt: failed attemt to adjust CUDA dot product
Iwan Kawrakow
committed
DRY
Iwan Kawrakow
committed
DRY
Iwan Kawrakow
committed
iq4_kt: CUDA dot product works
Iwan Kawrakow
committed
DRY
Iwan Kawrakow
committed
Report actual bpw
Iwan Kawrakow
committed
Minor tweaks
Iwan Kawrakow
committed
Checkpoint
Iwan Kawrakow
committed

Commits on Jan 24, 2025

Merge remote-tracking branch 'origin/main' into ik/try_trellis
Iwan Kawrakow
committed

Commits on Feb 9, 2025

Merge remote-tracking branch 'origin/main' into ik/try_trellis
Iwan Kawrakow
committed

Commits on May 19, 2025

Commits on May 22, 2025

Some performance tweaks

Iwan Kawrakow
authored and
andrewkchan
committed
Slighty faster iq2_kt

Iwan Kawrakow
authored and
andrewkchan
committed
port Trellis struct to iq3_kt, iq4_kt
andrewkchan
committed
Merge remote-tracking branch 'origin/main' into andrewkchan/try_trellis
andrewkchan
committed
oops untracked files
andrewkchan
committed