Feature Request: Support for Qwen 3.5

Prerequisites

• I am running the latest code. Mention the version if possible as well.
• I carefully followed the README.md.
• I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
• I reviewed the Discussions, and have a new and useful enhancement to share.

Feature Description

I suggest the port of Qwen 3.5 support for day 1 availability.
This will seemingly have a vision model too.

Motivation

The model should be made available shortly, maybe next week, not out yet

Possible Implementation

huggingface/transformers#43830

Mainline implementation

ggml-org/llama.cpp#19435

[CISC]

You may want to curb your enthusiasm, it was not meant to be merged yet.

[ikawrakow]

Not to mention the fact that ik_llama.cpp first needs support for Qwen3-Next before even thinking about Qwen3.5.

@CISC I haven't looked at all at Qwen3.5, but someone was claiming that the architecture is similar to Qwen3-Next. If that were true, why on earth would a Qwen-3.5 PR require 1500 LoC?

[CISC]

Not to mention the fact that ik_llama.cpp first needs support for Qwen3-Next before even thinking about Qwen3.5.

@CISC I haven't looked at all at Qwen3.5, but someone was claiming that the architecture is similar to Qwen3-Next. If that were true, why on earth would a Qwen-3.5 PR require 1500 LoC?

Yes, that is correct, the main body of change was the merge of another PR refactoring delta net.

We have weights.

https://huggingface.co/Qwen/Qwen3.5-397B-A17B

Seems extremely strong based on benchmarks, and it's VL too.

[ubergarm]

I got an early start on it with a mainline imatrix and test quant: https://huggingface.co/ubergarm/Qwen3.5-397B-A17B-GGUF Q3_K 179.97 GiB (3.90 BPW) custom recipe.

Also a smaller smol-IQ2_XS 113.41 GiB (2.46 BPW) also with imatrix.

I've since circled back around and testing ik_llama.cpp with Qwen3-Coder-Next as qwen3next seems important before qwen35moe.

[buliaoyin]

I tested pr1288，found two problem:

1. not work with --batch-size 8192 --ubatch-size 8192, 4096 works fine, but cannot get the best pp perf
2. not work with --mmproj

The tg was faster than mainline version.

I tested pr1288，found two problem:

1. not work with --batch-size 8192 --ubatch-size 8192, 4096 works fine, but cannot get the best pp  perf

2. not work with --mmproj

The tg was faster than mainline version.

Same problem.

llama_init_from_model: KV self size = 2040.00 MiB, K (q8_0): 1020.00 MiB, V (q8_0): 1020.00 MiB llama_init_from_model: CUDA_Host output buffer size = 1.89 MiB ggml_new_object: not enough space in the context's memory pool (needed 52462080, available 52461712) ./Qwen3.5b: line 38: 475355 Segmentation fault (core dumped) "${args[@]}"

I can use around 7000 ub at best. Is this a VRAM problem? Nvtop shows a bunch of memory still free.. In any case, Any way to reduce it? The kv cache for 131000 tokens is only 2GB, it's a waste not being able to use it. the -ub here becomes a hard cap for usable context length.

[ikawrakow]

I can use around 7000 ub at best. Is this a VRAM problem?

No, it is not a VRAM issue. The delta net requires a huge number of compute nodes, and the number of compute nodes grows linearly with u-batch size. The problem comes from the obsession of llama.cpp developers to not allocate any memory during run time, everything needs to be pre-allocated. This has been inherited here. One of the things that needs to get pre-allocated is a ggml context, which holds the tensors required for the compute graph. There is a heuristics how much memory will be needed for this. The heuristics fails for Qwen3-Next/Qwen3.5 because of the delta net. I have already adjusted once to make sure it runs with u-batch size of 4096. It will require yet another adjustment to be able to get up to 8192 (or more).

But just out of curiosity: do you actually gain performance by going to 8192? The processing time is heavily dominated by the delta net, which seems to slow down somewhat with increasing u-batch size, so at some point there out to be diminishing returns from increasing u-batch, even for hybrid inference.

[ikawrakow]

2. not work with --mmproj

Well, yes. The text part has been implemented, not the vision part.

I can use around 7000 ub at best. Is this a VRAM problem?

No, it is not a VRAM issue. The delta net requires a huge number of compute nodes, and the number of compute nodes grows linearly with u-batch size. The problem comes from the obsession of llama.cpp developers to not allocate any memory during run time, everything needs to be pre-allocated. This has been inherited here. One of the things that needs to get pre-allocated is a ggml context, which holds the tensors required for the compute graph. There is a heuristics how much memory will be needed for this. The heuristics fails for Qwen3-Next/Qwen3.5 because of the delta net. I have already adjusted once to make sure it runs with u-batch size of 4096. It will require yet another adjustment to be able to get up to 8192 (or more).

But just out of curiosity: do you actually gain performance by going to 8192? The processing time is heavily dominated by the delta net, which seems to slow down somewhat with increasing u-batch size, so at some point there out to be diminishing returns from increasing u-batch, even for hybrid inference.

Well, I don't know how much I will gain from 8192, but going from 4096 to 6144 is about a 20% boost on my system.

(7800x3d 8 core, ddr5 6000mt 2 channels, 5090, pcie5)

If the additional memory needed for the ggml context to get to 8192 isn't too great I would argue it would be a good idea. That message said "needed 52462080, available 52461712" btw, is that bytes? That's how much I'm missing to get to -ub 8192?

[ikawrakow]

Yes, these are bytes. The main concern here would be CPU cache sizes, not running out of RAM.

But, OK. See PR #1296. If you have enough VRAM, you can now use u-batch sizes of 16k if you like.

Thanks!