Skip to content

Adding IQ5_KS - 5.25 bpw quants#422

Merged
ikawrakow merged 10 commits into
mainfrom
ik/iq5_ks
May 15, 2025
Merged

Adding IQ5_KS - 5.25 bpw quants#422
ikawrakow merged 10 commits into
mainfrom
ik/iq5_ks

Conversation

@ikawrakow

Copy link
Copy Markdown
Owner

For motivation, see the CUDA performance graphs in #417 and #418.

Implementation for AVX2, Zen4, ARM_NEON, CUDA, Metal.

The AVX2 implementation suffers from int16_t overflow, and so do the IQ4_K, IQ5_K, IQ6_K and IQ4_KS, so I will have to fix all of these in a follow up PR.

I also want to add interleaved variant IQ5_KS_R4 before giving more performance and accuracy details.

@ubergarm

ubergarm commented May 18, 2025

Copy link
Copy Markdown
Contributor

Just did some testing of a mixed IQ5_KS / IQ4_KS quant of Qwen3-14B dense showing some Perplexity and Speed comparisons for full CUDA offload in this new quant cookers guide (just scroll to bottom, can't link anchors in gh discussions...)

Thanks for adding, the quality looks really good for the size!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants