GPU split isn't followed here?

[Ph0rk0z]

I'm using tabbyAPI and setting a list of splits but it doesn't seem to respect them anymore. Max vram allocated is about 23700 and anything bigger than that is ignored. Any granular settings like 22.1 simply load whatever they feel like, stopping early or stopping late.

As a result, I can't get over 8k context on qwen 235b but have one gpu with 95% and the main gpu with only 93%. 2 of the GPUs load to 97% and it doesn't really grow during inference.

On llama.cpp derivatives, I know I can allocate at least 24050 MiB and with exl2 there was at least some control by incrementing the decimal. I've seen even 98% before.

[Ph0rk0z]

k, I dived into the code and what it's doing.

The allocator takes some baseline + usage and converts it into a fraction of total GPU memory as reported by pytorch. 23.59 gets you to the maximum. It's basically gpu_split * 1024/free mem, which is why it doesn't match what I requested.

Torch for some reason only allows 23798, maybe due to overhead? In either case, the fractions were 1.0 and it still wasn't using the whole thing and going OOM early.

Searching around I found PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True

After setting this, suddenly there is a whole lot of free memory, one GPU is in the 70s even with 32k context. Dunno what it does to cuda malloc, but for now it seems to have solved the issue. Model load no longer fails. Don't remember having to do this previously or getting anything back from it.

[turboderp]

EXL3 doesn't statically allocate all the memory it needs for inference. While loading it runs a worst-case forward pass to know when to split layers so as to make sure inference will subsequently work, but it doesn't hold on to that memory after the model is loaded. Among other things, this allows for a vision and text model to share the same dynamic memory since they won't be running concurrently anyway. It may leave memory seemingly unused when you're forwarding one token at a time, but since that memory is accounted for while loading it prevents OoM errors when you later want to process a prompt in 2048-token chunks. If that's not a situation you care about you can reduce the max chunk size and reserve less memory, but then you'll also have slower prompt processing as a tradeoff.

It's not always possible to reach 100% utilization on each GPU regardless, though, since the model is split into whole layers. Each layer on Qwen3-235B is about 2.5 billion params, so even at 2 bpw that's a 600 MB chunk of weights (plus 1/94 of the KV cache) that resides either on CUDA:0 or CUDA:1, making it difficult to achieve a split that hits exactly 99.9% VRAM usage. But that's not necessarily something you'd want to aim for anyway, any more than you'd want to use 99.9% of your system RAM or your disk space at all times. A little bit of headroom is a good thing.

expandable_segments changes the behavior of Torch's caching allocator, possibly leading to less fragmentation, but in other cases potentially hurting performance. It's something you can try but not really something I can test and document. More along the lines of "maybe try a different driver version and see what happens." 🤷

Main point though is that EXL3 uses dynamic allocations in some places where EXL2 uses static allocations. Just because usage is sitting at 93% after loading doesn't mean it won't jump to 99% momentarily if you feed it a long prompt.

[Ph0rk0z]

I'm going to try long prompts on it and see what happens (maybe even 4096 batch). Speed actually increased by .20 t/s. I thought that tabby attempts to test full context length when loading but could be wrong.

Smells like we're bitten by pytorch here again. You use the native functions for memory, something like llama.cpp uses C. Currently on EXL2 PT prevents TP for image models despite it working flawlessly after I patched it (at least single user non-batched).

Feels like documented ampere slowness got "fixed" too. I get 15t/s which looks normal on a 90gb model pipeline parallel. Compared to ik_llama, IQ4 gets 18ts there but only 200PP. EXL3 prompt processing was double and outputs between 3.0bpw and IQ4 are about the same.

edit: 23k token prompt passed. Even at 4096.