KV quantization was recently enhanced upstream by adding rotation similar to TurboQuant, and is enabled by default.
It may be helpful to add the LLAMA_ATTN_ROT_DISABLE variable to xllamacpp so that classic KV attention can be enabled for troubleshooting purposes.
KV quantization was recently enhanced upstream by adding rotation similar to TurboQuant, and is enabled by default.
It may be helpful to add the
LLAMA_ATTN_ROT_DISABLEvariable to xllamacpp so that classic KV attention can be enabled for troubleshooting purposes.