running the QTIP inference code

[tsengalb99]

@turboderp I noticed you guys had some trouble running our inference code. The main blocker is usually installing fast-hadamard-transform due to a ninja issue. The way I set up the environment is:

1. Create a new conda env
2. Install torch and ninja=1.11.1 (not 1.11.4 or whatever the latest is)
3. Clone fast-hadamard-transform, cd into it, and run python setup.py install (alternatively, pip install -e . also works)
4. Install the qtip kernels. Cd into the qtip-kernels folder and run python setup.py install
5. Install everything else in requirements.txt

To generate text with our kernels, run interactive_gen.py with the commands in the repo. To evaluate perplexity and zeroshot performance, use the scripts in the eval folder.

Also, feel free to reach out to me (albert@cs.cornell.edu) if you have any questions about QTIP. We're excited to see our stuff adopted in bigger projects!

[Vhallo]

Fine work with QTIP & YAQA! Of course, would be great to see the YAQA in EXL too, but I imagine the compute requirements might stand in the way.

[turboderp]

I'm also very wary of the compute requirement, but I did want to try this out so I gave it another go. Eventually, I was able to get it running alongside the other libraries I need. Had to jump through a lot of hoops, and apparently fast-hadamard-transform really doesn't like my 5090, but I ended up not needing it for the tests anyway.

Apparently the hybrid codebook does quite a bit better than the procedural ones at 2 bpw. Either that, or the blockwise regularization is at fault, though in a way that only manifests at 2 bpw for some reason. Might be worth revisiting both of those.

For YAQA there's only Llama-3.1-70B-Instruct to compare to, but the improvement is clear up to 3 bpw:

Wish there were some more models to try out to see how well this holds across architectures and training recipes.

[tsengalb99]

Both of these results seem a bit odd to me. HYB should do worse than 3INST in general and I would have expected a larger KL gap in your testing, even at 4 bits. Is there documentation on where EXL3 differs from QTIP?

[turboderp]

Sadly it's not really documented anywhere outside of the source code, and I'm also constantly tweaking it. I don't really have the bandwidth to thoroughly document everything I'm doing.

At its core, though, it's the same method as QTIP, or at least an attempt at an implementation of it. It's not inconceivable that something has been lost in translation, that there's some bad math somewhere, or maybe I'm rounding an intermediate value to float16 somewhere it really needs to be float32, or something. But outside of those possiblities, the main differences between EXL3 and QTIP/HYB are:

1. Rather than a sample of RedPajama, The calibration dataset is a mix of English and multilingual data, and about 15% are random tokens to help discourage overfitting.

2. Instead of rotating whole rows and columns, EXL3 rotates groups of 128 weights on either axis. This helps with kernel fusion, and more importantly it creates quantized tensors that can be split at load time to a granularity of 128 rows or columns, which lines up well with the head dimension of most models. So there's no need to requantize the same model for different tensor-parallel setups.

   After this rotation, you end up with blocks of 128x128 weights that are individually close to being Gaussian, though obviously not to as high a degree as with whole-tensor regularization. They also have a greater number of outliers getting clamped during quantization. More alarmingly, these groups currently are not individually scaled, so there's an assumption that the RMS is roughly even between blocks, which is only like 95% sound.

3. As for the other 5% (and to address a few especially difficult models), EXL3 also scales the input and output channels of each tensor individually, baking those scales into the su/sv tensors. This isn't foolproof, of course, and it was a bit of a kludge because I didn't want to revisit the GEMM kernel at the time to add block scales. But I'm working on those now, and then most likely the channel scales won't be needed. Early results suggest a somewhat smaller gap at 2 bpw this way.

4. Before doing the final encoding on each tensor, the quantizer runs some test passes (without LDLQ) on a sample of 16x16 tiles along a wrapped diagonal to determine a fudge factor for the whole tensor. At higher bitrates (>2), at least with the implementation as it is, quantization definitely benefits from scaling down the input slightly below the RMS of the codebook. I haven't had time to fully explore this yet, but it makes sense that the penalty for doing so is lower at higher bitrates, while the penalty for outliers wholly outside the range of the codebook is constant, so there would be a happy medium somewhere for any given bitrate.

At any rate, I for sure haven't fully explored all of this. I felt that outperforming AQLM was a decent point at which to set quantization aside for a while and start focusing on other aspects of the library. But obviously I'm missing out on some of the potential of QTIP here.

It would be nice to have some models besides Llama to compare against. Are there by chance any other models you've already quantized while developing QTIP that are fit to release, just to have some more points of comparison? Both Mistral and Qwen2 should be compatible but have different training recipes.