VRAM usage vs. EXL2

[cmoncure]

I'm working with Mistral Large and discovering that the VRAM footprint of the models loaded with EXL3 seems to be much larger than with EXL2.

For instance, the 2.5 bit quantization of EXL2 would fit in one 48 GB card.
The 5.0 bit quantization with EXL2 fits handily in 2x48 GB cards (~41 GB each).

However, the 2.5 bit quantization with EXL3 uses just over 50 GB and loading spills over onto my second GPU. The 5.0 bit quantization fails to load with MemoryError. Is this expected or have I done something wrong? If I have to down-size to a lower quant to fit in the same memory capacity, it somewhat cancels out the benefits of using the EXL3.

[Originalimoc]

Same cache_type and cache_size?

[turboderp]

How are you loading it? With Tabby?

[Ph0rk0z]

Set PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True, that lowered memory usage by an obscene amount and things fit again.

[Originalimoc]

Set PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True, that lowered memory usage by an obscene amount and things fit again.

I tried this once and it segfaulted

[Ph0rk0z]

As of torch 2.7 and dev compile of both exl2 and exl3 it hasn't done so for me. On previous torch with exl2 it never had any benefit. I got a tiny bit of break from enabling cudamalloc and that's it. At least with linux, no idea what happens on windows.

[Downtown-Case]

It's similar for me as well, but I don't know about exactly the same.

EXl3 should still trounce a slightly larger exl2, though, especially at 2.5bpw.