dots.llm1 support

[louiehelm]

New architecture dots.llm1

Might somewhat easy to support if Qwen3 and Deepseek are working according to this:

It looks like it is a mixture of deepseek-3 MoE modules and qwen-3 attention modules:

https://github.com/huggingface/transformers/blob/ffe12627b4e84489d2ab91dd0ec00614855edc79/src/transformers/models/dots1/modular_dots1.py

and uses qwen-2 tokeniser:

https://huggingface.co/rednote-hilab/dots.llm1.inst/blob/main/tokenizer_config.json

So it's probably just a case of glueing all this together?

There's a working hf patch and llama.cpp patch in testing.

People are planning Deepseek distills and other tunes with this model since it has several base checkpoints released + more world knowledge than anything outside 5x+ it's size. At 143B, dots.llm1 tunes could plausibly become the premier EXL3 models for people with dual GPUs who can't run any Deepseek quant but want to run something better than Qwen3 32B.

And even if others don't, I will quantize and release EXL3 quants of this model if a working definition file gets created.

[turboderp]

I pushed support for dots.llm1 today. Had to include a kludge that specifically skips quantizing the attn out projection on the first layer, since it's apparently too irregular.

As far as I can tell values have been clamped to +/- 2.0 so I suspect something got out of hand in training and this is how they painted over it? In any case, the rest of the model seems fine.

Since it doesn't use routing groups, the only difference to Qwen3MoE is using sigmoid instead of softmax on the router logits, and then using a single dense layer. So supporting it wasn't hard, but it's still a very large sparse model. Quantization isn't fast so I've only really tested 4.0bpw for now. It's on HF here, and I'll queue up some more jobs to run overnight and upload tomorrow if all goes well.

The speed is okay, comparable to Qwen3-32B, but of course this only holds at bsz 1 because of the sparsity. Even with further optimization to reduce the MoE overhead, this still isn't going to be super fast by comparison, and prompt ingestion is always going to be slow (due to activating all 142B params), as is batched inference.

Now, there aren't a lot of benchmarks yet, but the ones in the paper don't show this outperforming Qwen3-32B by all that much, and it even loses in a few categories. So outside of CPU inference, I'm not sure exactly where you'd want to consider this model over Qwen3-32B, or maybe Qwen2.5-72B with a draft model.

But it works so there you go. (:

[louiehelm]

😂 Yes definitely MS Paint sad-face + arrow

Still great work! Happy to have new model type. Skipping first attn out layer isn't your kludge -- it's a response to their kludge.