Command A support

[MRGRD56]

It would be nice to have support for CohereLabs/c4ai-command-a-03-2025

I see that Command R+ is already available, but I haven’t been able to find any EXL3-quantized version of Command A

[Ph0rk0z]

I don't know if anyone made quants but the cohere2 architecture is supported: https://github.com/turboderp-org/exllamav3/blob/master/exllamav3/models/cohere2.py

[Downtown-Case]

@MRGRD56 Do you want a specific bpw? I can run one overnight.

Quantizing locally is quite vram light FYI.

[Ph0rk0z]

In my case, I'm gate kept by the file sizes.

[MRGRD56]

@Downtown-Case

Quantizing locally is quite vram light FYI.

Oh, really? Hmm, do you know how much would actually be needed to quantize it? As for me, I have an RTX 5090 (32GB, @ PCIe 5.0 x8) and an RTX 4070 Ti Super (16GB, @ PCIe 4.0 x8) so 32+16 = 48 GB VRAM in total. And also 64 GB of DDR5 RAM, and my CPU is a 7950X3D. I thought that might not be enough, or it would be too slow... Do you think I'd actually be able to quantize such a big model with this setup?

Do you want a specific bpw? I can run one overnight.

I'd actually appreciate that if you're up for it :)
I was thinking something like 3.12 bpw, so I could fit it into 48 GB with at least 16K context using 8-bit KV cache. I wanted to see if it performs better than IQ3_XXS in llama.cpp. Based on the charts, it should be better.

[Downtown-Case]

Generally it takes less vram to quantize exl3s than to run inference. I could do Command A on my single 3090 (when there's not a chance I could run it), but your 5090 alone is more than enough.

Might take a bit, but probably not more than a night?

I assumed the issue (as it is for some) was bandwidth limitations, eg being basically unable to download a 250GB model, but if you can download the fp16 you can quanitze it for sure.

Yeah, 3.12bpw is right in the sweetspot for exl3 and where llama.cpp tends to crash. In exl3, that tends to be 4bpw for attention and 3bpw for the mlp layers.

[MRGRD56]

Oh that's interesting. Thanks for the info! If I quantized it myself, I wonder if my 2nd GPU could help somehow, or on the contrary, it would slow down the process...
But still, that's cool that you can quantize 111B with just 24 GB VRAM, I thought you'd have to fit it all into your VRAM or at least RAM.

Well 250GB is a lot, yeah, especially for me cause I'm used to just downloading quants without having to download hundreds of gigs and waiting for hours until it's quantized 😅
I could try quantizing it myself but it'd take a while and I'm not even sure if it'll actually work..

By the way, do you know, is it possible to resume the quantization process if it was interrupted?

For example, I quantized part of the model, turned off the PC, turned it on the next day, and continued the quantization from the same place? Is this possible? Or should the quantization be done entirely in one go? And you have to start over after a single Ctrl+C?

[Downtown-Case]

For example, I quantized part of the model, turned off the PC, turned it on the next day, and continued the quantization from the same place? Is this possible?

Yes!

So the way it works (from my layman's perspective) is the model is quantized one layer at a time ( Command A, for instance, has 59 layers), and it checkpoints to disk after every layer. Run the same command again, and it'll resume from the last checkpoint.

If I quantized it myself, I wonder if my 2nd GPU could help somehow, or on the contrary, it would slow down the process...

Not sure, but I'd guess the 5090 alone is best.

I could try quantizing it myself but it'd take a while and I'm not even sure if it'll actually work..

You should try it! It's easy, you can't mess it up and you can quantize to the exact bpw you want; read about it here: https://github.com/turboderp-org/exllamav3/blob/master/doc/convert.md

[MRGRD56]

Okay, thanks again! Doesn't sound that bad, not as bad as I thought. Then I'll try if I find the time for that 🙂 If I make a working quant, I'll upload it to huggingface I guess

[Downtown-Case]

If I make a working quant, I'll upload it to huggingface I guess

Please do!

Not trying to deny your request (I can still grab Command A if you can't get it to work), but it's a good thing to try.

And one more thing: there are many ways to download HF models, but this one's really fast/turnkey, and hash checks all the files to make sure they aren't corrupted: https://github.com/bodaay/HuggingFaceModelDownloader

Or the official CLI tool: https://huggingface.co/docs/hub/models-downloading#using-the-hugging-face-client-library

I mention this because plain git LFS has corrupted a raw weights download for me.

[mindworksmanagement]

Thanks for the insights — the layer-level checkpointing and resumability clears up a lot. That was my main hesitation.

I’ll be testing Command A quant at 3.12bpw targeting 16K context with 8-bit KV cache, aiming for stability beyond the llama.cpp IQ3_XXS threshold.

Setup:
• CPU: Ryzen 9 9950X
• RAM: 192GB DDR5
• GPUs: RTX 5090 (32GB, PCIe 5.0) + 4070 Ti Super (16GB, PCIe 4.0)

With this config, I’ll start on the 5090 solo to avoid multi-GPU bottlenecks unless I find a good split load strategy. Might explore dual-offload later via a custom pipe or exllama:v2-style extension.

Will report back with performance metrics and upload the .exl3 if it completes cleanly.

Also appreciate the heads-up on bodaay/HuggingFaceModelDownloader — I’ve had LFS corrupt large shards before too, so this’ll be my go-to fetcher for raw weights.

[MRGRD56]

I tried quantizing it myself.

I used the exllamav3 built into tabbyAPI so I didn't have to install it again.
I only had to create a script exl3_convert.py by copying convert.py from the exllamav3 repository.
And also copy the standard_cal_data into tabbyAPI\venv\Lib\site-packages\exllamav3\conversion\standard_cal_data

I used this command for the quantization:

python .\exl3_convert.py -i C:\Users\SU\LLM\CohereLabs\c4ai-command-a-03-2025 -o C:\Users\SU\LLM\CohereLabs\c4ai-command-a-03-2025-exl3-3.12bpw -w C:\Users\SU\LLM\CohereLabs\c4ai-command-a-03-2025-exl3-3.12bpw-TMP -b 3.12 -d 0,1 -dr 32,16

I also tried quantizing just one layer at first, to compare the speeds. Here's what I got:

With just the RTX 5090 it said the Estimated remaining time was 2h 55m
With both RTX 5090 and RTX 4070 Ti Super the estimated time was less: 2h 17m

So I can assume that the 2nd GPU could actually speed it up a bit, so I decided to use 2 GPUs for quantizing.

As for VRAM used:
RTX 5090 - 26-27.5 GB was used in total (with also 2 monitors plugged in)
RTX 4070 Ti Super - 1-1.5 GB was used (with no monitors)

Also the software itself was using up to 32 GB of the system RAM or so, but most of the time it was 21-25 GB.

For some reason it barely used the VRAM of the 2nd card, even though both cards were under high loads (95-100% usage)

So, I ran it. It quantized every layer, and then...
It finished like this, with an error:

 -- Estimated remaining time: 13 minutes
 -- Saving checkpoint
 -- Loading unquantized module: model.layers.61
 -- Captured: model.layers.61
 -- Quantized: model.layers.61.self_attn.q_proj                 bpw:  3.00  proxy_err: 0.003413  o  g_sc: 0.945322  [9.44 s]
 -- Quantized: model.layers.61.self_attn.k_proj                 bpw:  4.00  proxy_err: 0.000587  o  g_sc: 0.901417  [2.41 s]
 -- Quantized: model.layers.61.self_attn.v_proj                 bpw:  5.00  proxy_err: 0.000467  o  g_sc: 0.850602  [2.38 s]
 -- Quantized: model.layers.61.self_attn.o_proj                 bpw:  4.00  proxy_err: 0.000303  o  g_sc: 0.897962  [8.59 s]
 -- Quantized: model.layers.61.mlp.up_proj                      bpw:  3.00  proxy_err: 0.003448  o  g_sc: 0.945322  [25.20 s]
 -- Quantized: model.layers.61.mlp.gate_proj                    bpw:  3.00  proxy_err: 0.003318  o  g_sc: 0.939732  [25.31 s]
 -- Quantized: model.layers.61.mlp.down_proj                    bpw:  3.00  proxy_err: 0.001160  o  g_sc: 0.945322  [32.89 s]
 -- Quantized: model.layers.61                                  bpw:  3.11  rfn: 0.027311  cos: 0.000471  sqnr: 31.587912  [130.11 s]
 -- Estimated remaining time: 11 minutes
 -- Saving checkpoint
 -- Loading unquantized module: model.layers.62
 -- Captured: model.layers.62
 !! Warning: block.parallel.down state has 38,057 inf values and 0 NaN values (out of 7,549,747,200)
 -- Quantized: model.layers.62.self_attn.q_proj                 bpw:  3.00  proxy_err: 0.000496  o  g_sc: 0.901417  [8.85 s]
 -- Quantized: model.layers.62.self_attn.k_proj                 bpw:  4.00  proxy_err: 0.000196  o  g_sc: 0.830377  [2.39 s]
 -- Quantized: model.layers.62.self_attn.v_proj                 bpw:  5.00  proxy_err: 0.000076  o  g_sc: 0.824786  [2.35 s]
 -- Quantized: model.layers.62.self_attn.o_proj                 bpw:  4.00  proxy_err: 0.000002  o  g_sc: 0.845012  [8.15 s]
 -- Quantized: model.layers.62.mlp.up_proj                      bpw:  3.00  proxy_err: 0.000308  o  g_sc: 0.925097  [25.21 s]
 -- Quantized: model.layers.62.mlp.gate_proj                    bpw:  3.00  proxy_err: 0.000382  o  g_sc: 0.916052  [25.34 s]
Traceback (most recent call last):
  File "C:\Users\SU\tabbyAPI\exl3_convert.py", line 11, in <module>
    main(_in_args, _job_state)
  File "C:\Users\SU\tabbyAPI\venv\Lib\site-packages\torch\utils\_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\SU\tabbyAPI\venv\Lib\site-packages\exllamav3\conversion\convert_model.py", line 418, in main
    proxy_err = linear.convert_exl3(
                ^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\SU\tabbyAPI\venv\Lib\site-packages\exllamav3\modules\linear.py", line 237, in convert_exl3
    weight_q, proxy_err, out_tensors = quantize_exl3(
                                       ^^^^^^^^^^^^^^
  File "C:\Users\SU\tabbyAPI\venv\Lib\site-packages\exllamav3\modules\quant\exl3_lib\quantize.py", line 781, in quantize_exl3
    H, L, su, H_diag = finalize_capture_H(H_data, quant_args, verbose)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\SU\tabbyAPI\venv\Lib\site-packages\exllamav3\modules\quant\exl3_lib\quantize.py", line 480, in finalize_capture_H
    L, H = block_ldl(H, 16, verbose)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\SU\tabbyAPI\venv\Lib\site-packages\exllamav3\modules\quant\exl3_lib\quantize.py", line 287, in block_ldl
    raise e
  File "C:\Users\SU\tabbyAPI\venv\Lib\site-packages\exllamav3\modules\quant\exl3_lib\quantize.py", line 274, in block_ldl
    L = torch.linalg.cholesky(H)
        ^^^^^^^^^^^^^^^^^^^^^^^^
torch._C._LinAlgError: linalg.cholesky: The factorization could not be completed because the input is not positive-definite (the leading minor of order 1 is not positive-definite).
(venv) PS C:\Users\SU\tabbyAPI>

Running it again with --resume didn't help: it started quantizing the layer 62 and ended up failing with the same error.

(venv) PS C:\Users\SU\tabbyAPI> python .\exl3_convert.py -i C:\Users\SU\LLM\CohereLabs\c4ai-command-a-03-2025 -o C:\Users\SU\LLM\CohereLabs\c4ai-command-a-03-2025-exl3-3.12bpw -w C:\Users\SU\LLM\CohereLabs\c4ai-command-a-03-2025-exl3-3.12bpw-TMP -b 3.12 -d 0,1 -dr 32,16 --resume
 -- Resuming existing job
    Input directory: C:\Users\SU\LLM\CohereLabs\c4ai-command-a-03-2025
    Output directory: C:\Users\SU\LLM\CohereLabs\c4ai-command-a-03-2025-exl3-3.12bpw
    Working directory: C:\Users\SU\LLM\CohereLabs\c4ai-command-a-03-2025-exl3-3.12bpw-TMP
    Calibration size: 100 rows, 2048 columns
    Target bitrate: 3.12 (decoder), 6 (head)
    Output scales: auto
 -- Loaded model config
    Architecture: Cohere2ForCausalLM
 -- Created model instance:
     - Cohere2Model
         - Embedding
         - [64x] ParallelDecoderBlock
             - LayerNorm
             - Attention
                 - [4x] Linear
             - GatedMLP
                 - [3x] Linear
         - LayerNorm
         - Linear
 -- Loaded tokenizer
    Vocab size: 255033
 -- Resuming at: model.layers.62
 -- Loading unquantized module: model.layers.62
 -- Captured: model.layers.62
 !! Warning: block.parallel.down state has 38,057 inf values and 0 NaN values (out of 7,549,747,200)
 -- Quantized: model.layers.62.self_attn.q_proj                 bpw:  3.00  proxy_err: 0.000496  o  g_sc: 0.901417  [8.81 s]
 -- Quantized: model.layers.62.self_attn.k_proj                 bpw:  4.00  proxy_err: 0.000196  o  g_sc: 0.830377  [2.37 s]
 -- Quantized: model.layers.62.self_attn.v_proj                 bpw:  5.00  proxy_err: 0.000076  o  g_sc: 0.824786  [2.32 s]
 -- Quantized: model.layers.62.self_attn.o_proj                 bpw:  4.00  proxy_err: 0.000002  o  g_sc: 0.845012  [8.06 s]
 -- Quantized: model.layers.62.mlp.up_proj                      bpw:  3.00  proxy_err: 0.000308  o  g_sc: 0.925097  [24.98 s]
 -- Quantized: model.layers.62.mlp.gate_proj                    bpw:  3.00  proxy_err: 0.000382  o  g_sc: 0.916052  [25.07 s]
Traceback (most recent call last):
  File "C:\Users\SU\tabbyAPI\exl3_convert.py", line 11, in <module>
    main(_in_args, _job_state)
  File "C:\Users\SU\tabbyAPI\venv\Lib\site-packages\torch\utils\_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\SU\tabbyAPI\venv\Lib\site-packages\exllamav3\conversion\convert_model.py", line 418, in main
    proxy_err = linear.convert_exl3(
                ^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\SU\tabbyAPI\venv\Lib\site-packages\exllamav3\modules\linear.py", line 237, in convert_exl3
    weight_q, proxy_err, out_tensors = quantize_exl3(
                                       ^^^^^^^^^^^^^^
  File "C:\Users\SU\tabbyAPI\venv\Lib\site-packages\exllamav3\modules\quant\exl3_lib\quantize.py", line 781, in quantize_exl3
    H, L, su, H_diag = finalize_capture_H(H_data, quant_args, verbose)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\SU\tabbyAPI\venv\Lib\site-packages\exllamav3\modules\quant\exl3_lib\quantize.py", line 480, in finalize_capture_H
    L, H = block_ldl(H, 16, verbose)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\SU\tabbyAPI\venv\Lib\site-packages\exllamav3\modules\quant\exl3_lib\quantize.py", line 287, in block_ldl
    raise e
  File "C:\Users\SU\tabbyAPI\venv\Lib\site-packages\exllamav3\modules\quant\exl3_lib\quantize.py", line 274, in block_ldl
    L = torch.linalg.cholesky(H)
        ^^^^^^^^^^^^^^^^^^^^^^^^
torch._C._LinAlgError: linalg.cholesky: The factorization could not be completed because the input is not positive-definite (the leading minor of order 1 is not positive-definite).
(venv) PS C:\Users\SU\tabbyAPI>

Any ideas on what could've gone wrong? How can it be fixed?

I downloaded the model this way if it matters:

huggingface-cli download CohereLabs/c4ai-command-a-03-2025 --local-dir ./CohereLabs/c4ai-command-a-03-2025

[Downtown-Case]

Not 100% sure huggingface cli hashchecks the files, so it could be corrupt, theoretically.

torch._C._LinAlgError: linalg.cholesky: The factorization could not be completed because the input is not positive-definite (the leading minor of order 1 is not positive-definite).

But this sounds like a bug! Something for turboderp to fix. I will run it tonight (if I can do it without OOMing) and see if the same thing happens.

[ja231c]

torch._C._LinAlgError: linalg.cholesky: The factorization could not be completed because the input is not positive-definite (the leading minor of order 1 is not positive-definite).

I get the exact same error trying to quantize https://huggingface.co/TheDrummer/Agatha-111B-v1 to 3.1 bpw.

[MRGRD56]

If anything, I was using the latest version of exllamav3:

PS C:\Users\SU\tabbyAPI> venv/Scripts/activate
(venv) PS C:\Users\SU\tabbyAPI> pip show exllamav3
Name: exllamav3
Version: 0.0.4+cu128.torch2.7.0
Summary:
Home-page: https://github.com/turboderp/exllamav3
Author: turboderp
Author-email:
License: MIT
Location: C:\Users\SU\tabbyAPI\venv\Lib\site-packages
Requires: flash-attn, ninja, numpy, rich, safetensors, tokenizers, torch, typing-extensions
Required-by:

[ja231c]

I tried merging @memmaptensor's changes mentioned in #34 - this made it complete quantizing layer 62... but now it crashes (with no error, the terminal window just closes) after it finishes quantizing the lm_head, no idea why.