feat: TQ4_1S weight compression (Metal only, needs CUDA port)

[TheTom]

Summary

• TQ3_1S (3-bit, 4.0 BPW) and TQ4_1S (4-bit, 5.0 BPW) weight quantization using WHT rotation + Lloyd-Max centroids
• V2.1 fused Metal kernel: zero threadgroup memory, cooperative SIMD rotation via simd_shuffle_xor, NR0=8
• Post-training quantization — no retraining, calibration data, or model modification required
• Quantize via llama-quantize --allow-requantize --tensor-type-file config.txt

Tested Models

Model	Config	Size Reduction	PPL Delta	Decode	NIAH
Qwen2.5-1.5B	Config I	-27%	+1.9%	96%	6/6
Qwen3.5-27B	Config I	-28%	+1.3%	99%	3/3
Qwen3.5-35B MoE	Config I	-37%	+1.4%	102%	—
Qwen2.5-72B	Config I	-38%	+3.9%	95%	3/3
Phi-4 14B	Config I	-36%	+1.0%	254%	3/3
Llama 3.1 70B	Premium	-29%	+5.8%	fast	3/3
Llama 3.1 70B	Hybrid	-42%	+16%	133%	3/3

Llama Note

Llama-family models show 6-8x higher per-layer error amplification with WHT-rotated FFN tensors. Use Hybrid (TQ4 attn + Q4_K FFN) or Premium (TQ4 attn + Q5_K/Q6_K FFN) configs. Both beat Q4_K_M in quality and speed at similar size. Full investigation in the paper.

What's needed before merge

• CUDA port of V2.1 kernel (calling @signalnine 👀)
• HIP/ROCm testing
• Regression tests on existing TurboQuant KV functionality
• Community validation on untested model families

Metal only

The quantization step (llama-quantize) works on any platform. The runtime dequant kernels are Metal-specific. Compressed GGUFs will not run correctly on CUDA/HIP until those backends are ported.

Paper: https://github.com/TheTom/turboquant_plus/blob/main/docs/papers/weight-compression-tq4.md
Getting started: https://github.com/TheTom/turboquant_plus/blob/main/docs/getting-started.md

🤖 Generated with Claude Code

[TheTom]

Regression Test Results — PR #45

Verified that the TQ4_1S weight compression PR does NOT break existing TurboQuant KV cache functionality or standard inference on non-compressed models.

Hardware: M5 Max (128GB) + Mac Mini M2 Pro (32GB)
Branch: pr/tq4-weight-compression (commit 6c3e503)

Speed — No Regressions

Model	Hardware	Config	pp512	tg128
Qwen2.5-1.5B Q8_0	M5 Max	q8_0/q8_0	10,787	198
Qwen2.5-1.5B Q8_0	M5 Max	q8_0/turbo4	10,460	141
Qwen2.5-1.5B Q8_0	M5 Max	q8_0/turbo3	10,468	138
Phi-4 14B Q8_0	M5 Max	q8_0/q8_0	1,052	33.7
Phi-4 14B Q8_0	M5 Max	q8_0/turbo4	1,051	30.9
Qwen3.5-27B Q8_0	M5 Max	q8_0/q8_0	408	17.6
Qwen3.5-27B Q8_0	M5 Max	q8_0/turbo4	497	17.1
Qwen3.5-27B Q8_0	M5 Max	q8_0/turbo3	487	17.0
Qwen3.5-35B MoE Q8_0	M5 Max	q8_0/q8_0	2,920	76.6
Qwen3.5-35B MoE Q8_0	M5 Max	q8_0/turbo4	2,878	69.6
Qwen2.5-7B Q4_K_M	M2 Pro	q8_0/q8_0	352	34.9
Qwen2.5-7B Q4_K_M	M2 Pro	q8_0/turbo4	351	30.8
Qwen2.5-7B Q4_K_M	M2 Pro	q8_0/turbo3	350	30.0
Qwen2.5-7B Q4_K_M	M2 Pro	turbo3/turbo3	346	26.4

All speeds normal or improved. No regressions.

PPL — No Regressions (full wikitext-2 runs)

Model	q8_0/q8_0	q8_0/turbo4	q8_0/turbo3
Qwen2.5-1.5B	10.31	10.45 (+1.4%)	10.55 (+2.4%)
Phi-4 14B	6.54	6.55 (+0.2%)	—
Qwen3.5-27B	6.87	6.89 (+0.3%)	—
Qwen3.5-35B MoE	6.53	6.56 (+0.5%)	—

All PPL values match known-good. MUL_MAT_ID (MoE path) verified working.

Verdict

ALL TESTS PASS. 5 models, 2 hardware platforms, 4 KV configs. The ggml-metal-ops.cpp restructuring (MUL_MAT + MUL_MAT_ID dispatch) did not break any existing functionality. Safe for review.

[TheTom]

Update: Rebased on upstream master + regression test

Branch force-pushed. Now rebased on latest ggml-org/llama.cpp master (106 upstream commits). 3 commits on top:

1. 187a2c1 feat: TQ3_1S + TQ4_1S weight quantization with V2.1 fused Metal kernels
2. 9aaa054 fix: add post-unrotate memory barrier for in-layer mixing safety
3. cb8bddc fix: disable upstream attn rotation by default (conflicts with TurboQuant)

Upstream conflict: activation rotation (commit 744c0c7)

Upstream added graph-level Hadamard rotation for KV cache quantization (llama : rotate activations for better quantization). This feature:

• Crashes on Phi-4 (graph hash table overflow from extra rotation nodes)
• Is redundant with our kernel-level WHT rotation (which is more efficient — no extra graph nodes)

Fix: disabled upstream rotation by default in our fork. Users can re-enable with LLAMA_ATTN_ROT_DISABLE=0. Our TurboQuant KV rotation is unaffected.

Regression test (M5 Max, rebased branch cb8bddc)

Test	What	Result	Status
Config I quantize + speed	Qwen 1.5B, 202 tg/s	Matches expected	✅
Config I PPL	Qwen 1.5B, 10.77 (8ch)	Within noise	✅
Turbo4 KV speed	Qwen 1.5B, 148 tg/s	Matches expected	✅
Turbo4 KV PPL	Qwen 1.5B, 10.74 (8ch)	Within noise	✅
Phi-4 + turbo4	31.5 tg/s	No crash	✅ Fixed
Large model turbo4	Qwen 27B, 17.4 tg/s	Matches expected	✅

All tests pass. No regressions. Phi-4 crash resolved.

[signalnine]

CUDA port available on our branch: signalnine/llama-cpp-turboquant feature/tq4-weight-cuda

What's implemented:

• CUDA dequant for TQ4_1S/TQ3_1S (convert path + cuBLAS MUL_MAT)
• Fused mul_mat_vec kernel with pre-rotated activations (warp shuffle WHT)
• mmvq exclusion for fused dispatch path
• llama-quantize registration for TQ4_1S/TQ3_1S types

Results (Qwen2.5-7B TQ4_1S, RTX 5090):

Metric	cuBLAS path	Fused kernel
Decode tg128	20 t/s	69 t/s
vs q8_0 (177 t/s)	11%	39%
PPL	8.82	8.82

The fused kernel pre-rotates the activation vector once per mul_mat via __shfl_xor_sync (5 butterfly stages, 32-element blocks), then the mmvq kernel just does centroid[idx] × scale with no per-block WHT. 13 kernel variants tested — the gap to q4_0 (275 t/s) is from dp4a integer intrinsics that TQ4_1S can't use (centroid lookup requires float). The gap to your Metal (85-99% of q8_0) is from Apple Silicon's cooperative SIMD efficiency.

Happy to iterate on this if you have ideas for closing the CUDA gap further.

[TheTom]

This is great work, thank you for turning this around so fast. PPL matching between cuBLAS and fused confirms correctness.

One question before we merge: can you confirm that uncompressed models (q8_0, q4_0, etc.) show no decode regression on this branch? i.e. the new code paths only activate for TQ4_1S/TQ3_1S and existing quant types run at the same speed as before the PR.

[TheTom]

CUDA kernel review — performance improvement opportunities

Nice work on the V8 pre-rotation approach. PPL matching confirms correctness. Here's what I see for closing the gap from 39% to 70-85% of q8_0:

High priority (biggest decode wins)

1. NR0 multi-row CTA with shared activation reuse — Currently each warp handles one row independently, re-reading vy_rot from global memory. On Metal, NR0=8 (8 rows sharing one activation tile) was the single biggest optimization (decode went from 70% to 99% of baseline). On CUDA the benefit is shared reuse of vy_rot across rows per CTA. Realistic target: 69 → 110-140 t/s.

2. Hot loop load dedup — d0/d1 are loaded per-lane but only have 2 unique values per block. Broadcast via __shfl_sync once per half-warp. Similarly, qs[lane/2] is loaded by both even and odd lanes — load once, broadcast to partner.

3. __restrict__ qualifiers + vectorized loads — The kernel pointers lack __restrict__ (compiler can't prove no aliasing). Adding it enables better instruction scheduling. Also consider float4 loads for the pre-rotated activation to hit 128-bit memory transactions.

Medium priority

4. Small-batch kernels (ne[1]=2..8) — Real serving hits batch > 1 frequently. A tuned small-batch path before falling to cuBLAS would help.

5. CTA shape sweep per architecture — NWARPS=8 was tested on 5090. Ampere/Ada may prefer different occupancy points. Worth parameterizing.

6. Fused prefill path (MMQ-style) — Currently falls to dequant→cuBLAS for ne[1] > 1. A tiled matmul with shared-memory activation rotation would be faster for medium batch.

Skip / low value

• dp4a / fp16 LUT — The pipeline is float activation × float centroid × float scale. Forcing int8 paths is awkward and unlikely to win vs a well-tuned float path.
• Scratch buffer cleanup — Lifecycle issue (static cudaMalloc leak), not a throughput lever. Fix eventually but not urgent.

Realistic ceiling

Per architecture with full tuning (NR0 + load dedup + vectorized + batch):

• Ampere: ~55-70% of q8_0
• Ada: ~60-75%
• Blackwell: ~70-85%

The 39% → 70-85% gap is primarily data reuse, not math precision. The pre-rotation design is correct — it just needs the activation tile shared across more rows per CTA.

[TheTom]

Full Regression Test — PR #45 (cb8bddc)

Hardware

• M5 Max 128GB (Apple Silicon)
• Mac Mini M2 Pro 32GB (Apple Silicon)

Quantize tool verification

Format	Quantize	Loads	Generates	Status
TQ4_1S Config I	1312 MiB (6.20 BPW)	✅	✅ 198 tg	✅
TQ3_1S attn-only	1730 MiB (8.17 BPW)	✅	✅	✅
Q4_K_M (standard)	1060 MiB (5.00 BPW)	✅	✅	✅

---

M5 Max — Uncompressed weights + TurboQuant KV

Qwen2.5-1.5B Q8_0

Config	pp512	tg128	PPL (full)
q8_0/q8_0	10,742	209	10.31
q8_0/turbo4	10,604	148	10.45
q8_0/turbo3	10,537	141	10.55

Phi-4 14B Q8_0 (crash fix verification)

Config	pp512	tg128	PPL (full)
q8_0/q8_0	1,083	34.0	6.54
q8_0/turbo4	1,088	31.1	6.55

No crash. Upstream attn_rot disabled by default (commit cb8bddc).

Qwen3.5-27B Q8_0

Config	pp512	tg128	PPL (full)
q8_0/q8_0	554	17.5	6.87
q8_0/turbo4	498	16.9	6.89
q8_0/turbo3	501	16.0	—

Qwen3.5-35B MoE Q8_0 (MUL_MAT_ID path)

Config	pp512	tg128	PPL (full)
q8_0/q8_0	2,837	77.0	6.53
q8_0/turbo4	2,826	77.8	6.56

---

M5 Max — TQ4_1S Weight Compression

Qwen2.5-1.5B Config I (1.28 GiB, 6.20 BPW)

Config	pp512	tg128	PPL (full)
Config I	7,610	198	10.53
Config I + turbo4 KV	7,394	142	—

---

Mac Mini M2 Pro — Qwen2.5-7B Q4_K_M

Config	pp512	tg128
q8_0/q8_0	316 ± 4.6	33.6 ± 0.2
q8_0/turbo4	348 ± 0.5	30.3 ± 0.3
q8_0/turbo3	346 ± 0.3	28.7 ± 0.4
turbo3/turbo3	338 ± 1.2	25.7 ± 0.3

---

Summary

• 5 models tested across 2 hardware platforms
• All PPL values match known-good (within measurement noise)
• All speed values normal — no regressions
• Phi-4 crash fixed (upstream attn_rot disabled)
• MUL_MAT_ID (MoE) verified
• Weight compression quantize + inference verified (TQ4_1S, TQ3_1S, Q4_K_M)
• TurboQuant KV configs verified (q8_0/turbo4, q8_0/turbo3, turbo3/turbo3)

All tests pass. PR is safe for review.

[Michael-Z-Freeman]

Here's my report :)

PR #45 Benchmark Report

GPU: AMD Radeon RX 9060 XT (ROCm, gfx1200), 16GB VRAM
Backend: HIP/ROCm
Build: e9c54d5 (8772)

Model under test:

• Source: qwen2.5-7b-instruct-q8_0.gguf
• Compressed (Config I): qwen2.5-7b-instruct-config-i.gguf
• Config I quantization result: 7.54 GiB -> 5.16 GiB

Bench settings used:

• -p 512 -n 128 -fa 1 -ngl 99 -b 256 -ub 128 -r 1

1) Q8_0 baseline

ggml_cuda_init: found 1 ROCm devices (Total VRAM: 16304 MiB):
  Device 0: AMD Radeon RX 9060 XT, gfx1200 (0x1200), VMM: no, Wave Size: 32, VRAM: 16304 MiB
| model                          |       size |     params | backend    | ngl | n_batch | n_ubatch | fa |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | ------: | -------: | -: | --------------: | -------------------: |
| qwen2 7B Q8_0                  |   7.54 GiB |     7.62 B | ROCm       |  99 |     256 |      128 |  1 |           pp512 |       1163.73 ± 0.00 |
| qwen2 7B Q8_0                  |   7.54 GiB |     7.62 B | ROCm       |  99 |     256 |      128 |  1 |           tg128 |         38.15 ± 0.00 |

build: e9c54d557 (8772)

2) Config I (TQ4_1S)

ggml_cuda_init: found 1 ROCm devices (Total VRAM: 16304 MiB):
  Device 0: AMD Radeon RX 9060 XT, gfx1200 (0x1200), VMM: no, Wave Size: 32, VRAM: 16304 MiB
| model                          |       size |     params | backend    | ngl | n_batch | n_ubatch | fa |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | ------: | -------: | -: | --------------: | -------------------: |
| qwen2 7B Q8_0                  |   5.16 GiB |     7.62 B | ROCm       |  99 |     256 |      128 |  1 |           pp512 |       1330.85 ± 0.00 |
| qwen2 7B Q8_0                  |   5.16 GiB |     7.62 B | ROCm       |  99 |     256 |      128 |  1 |           tg128 |         42.33 ± 0.00 |

build: e9c54d557 (8772)

3) Config I + TurboQuant KV (-ctk q8_0 -ctv turbo4)

ggml_cuda_init: found 1 ROCm devices (Total VRAM: 16304 MiB):
  Device 0: AMD Radeon RX 9060 XT, gfx1200 (0x1200), VMM: no, Wave Size: 32, VRAM: 16304 MiB
| model                          |       size |     params | backend    | ngl | n_batch | n_ubatch | type_k | type_v | fa |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | ------: | -------: | -----: | -----: | -: | --------------: | -------------------: |
| qwen2 7B Q8_0                  |   5.16 GiB |     7.62 B | ROCm       |  99 |     256 |      128 |   q8_0 | turbo4 |  1 |           pp512 |       1303.25 ± 0.00 |
| qwen2 7B Q8_0                  |   5.16 GiB |     7.62 B | ROCm       |  99 |     256 |      128 |   q8_0 | turbo4 |  1 |           tg128 |         41.57 ± 0.00 |

build: e9c54d557 (8772)

[TheTom]

Good catch @swfsql. I verified this on Qwen3.5-27B-Q8_0.

The layer pattern is a 1:3 interleave — only 16 of 64 layers are self-attention with split attn_q/attn_k/attn_v (blk.3, 7, 11, 15, ..., 63). The other 48 are Gated Delta Net layers with fused attn_qkv + SSM tensors.

With boundary=2, blk.0-1 are protected but those are delta net layers — no split attention tensors to protect. The first real self-attention layer (blk.3) is unprotected. The fused attn_qkv tensors are not in our config pattern, so they stay at q8_0 by default regardless.

I tested boundary=2 vs boundary=4 on M5 Max to measure the impact:

Config	PPL (8 chunks)	Delta vs Q8_0	Size
Q8_0 baseline	6.8879	—	27.3G
Config I boundary=2	7.0564	+2.44%	19.1G
Config I boundary=4	7.0349	+2.13%	19.6G

boundary=4 protects blk.3 (first real self-attention) and gives a 0.3% PPL improvement at the cost of 500 MiB. Real but small.

The delta net layers are accidentally safe — their fused attn_qkv tensors do not match the attn_q/attn_k/attn_v patterns in the config file, so they remain at q8_0 regardless of boundary setting.

For the default Config I recommendation, I am keeping boundary=2 since the practical difference is marginal. But this is useful context for anyone tuning Qwen 3.5 specifically — boundary=4 is a free 0.3% if you can spare the 500 MiB.

Re: compressing the fused attn_qkv tensors in delta net layers — that would require adding attn_qkv as a target in the config file. Worth exploring but would need separate testing since the delta net attention mechanism may have different sensitivity characteristics than standard self-attention.

[PeterArbeitsfrei]

I guess you're done here anyway, yet wanting to let you know that the compressing as described in Getting started seems to either miss something or isn't very clear, at least in my case I wasn't able to replicate the "Quick Test" as the output model (Qwen 3.5 35b) always turned out nearly the same size with config_i except for a few MBs.

Or do I need to change the config_i for the 35b model somewhat?

str: cannot properly format tensor name position_embd with suffix=weight bid=-1 xid=-1
str: cannot properly format tensor name token_types with suffix=weight bid=-1 xid=-1
[...]
llama_tensor_get_type: blk.15.attn_k.weight - applying manual override: q8_0 -> tq4_1s
str: cannot properly format tensor name position_embd with suffix=weight bid=-1 xid=-1
str: cannot properly format tensor name token_types with suffix=weight bid=-1 xid=-1
str: cannot properly format tensor name position_embd with suffix=weight bid=-1 xid=-1
str: cannot properly format tensor name token_types with suffix=weight bid=-1 xid=-1
llama_tensor_get_type: blk.15.attn_output.weight - applying manual override: q8_0 -> tq4_1s
str: cannot properly format tensor name position_embd with suffix=weight bid=-1 xid=-1
str: cannot properly format tensor name token_types with suffix=weight bid=-1 xid=-1
str: cannot properly format tensor name position_embd with suffix=weight bid=-1 xid=-1
str: cannot properly format tensor name token_types with suffix=weight bid=-1 xid=-1
llama_tensor_get_type: blk.15.attn_q.weight - applying manual override: q8_0 -> tq4_1s
str: cannot properly format tensor name position_embd with suffix=weight bid=-1 xid=-1
str: cannot properly format tensor name token_types with suffix=weight bid=-1 xid=-1
str: cannot properly format tensor name position_embd with suffix=weight bid=-1 xid=-1
str: cannot properly format tensor name token_types with suffix=weight bid=-1 xid=-1
llama_tensor_get_type: blk.15.attn_v.weight - applying manual override: q8_0 -> tq4_1s
str: cannot properly format tensor name position_embd with suffix=weight bid=-1 xid=-1
str: cannot properly format tensor name token_types with suffix=weight bid=-1 xid=-1
[...]
str: cannot properly format tensor name position_embd with suffix=weight bid=-1 xid=-1
str: cannot properly format tensor name token_types with suffix=weight bid=-1 xid=-1
[ 1/ 733] output.weight - [ 2048, 248320, 1, 1], type = q8_0, size = 515.312 MiB
[ 2/ 733] output_norm.weight - [ 2048, 1, 1, 1], type = f32, size = 0.008 MiB
[ 3/ 733] token_embd.weight - [ 2048, 248320, 1, 1], type = q8_0, size = 515.312 MiB
[ 4/ 733] blk.0.attn_gate.weight - [ 2048, 4096, 1, 1], type = q8_0, size = 8.500 MiB
[ 5/ 733] blk.0.attn_norm.weight - [ 2048, 1, 1, 1], type = f32, size = 0.008 MiB
[ 6/ 733] blk.0.attn_qkv.weight - [ 2048, 8192, 1, 1], type = q8_0, size = 17.000 MiB
[...]
[ 419/ 733] blk.22.ssm_a - [ 32, 1, 1, 1], type = f32, size = 0.000 MiB
[ 420/ 733] blk.22.ssm_alpha.weight - [ 2048, 32, 1, 1], type = q8_0, size = 0.066 MiB
[ 421/ 733] blk.22.ssm_beta.weight - [ 2048, 32, 1, 1], type = q8_0, size = 0.066 MiB
[ 422/ 733] blk.22.ssm_conv1d.weight - [ 4, 8192, 1, 1], type = f32, size = 0.125 MiB
[ 423/ 733] blk.22.ssm_dt.bias - [ 32, 1, 1, 1], type = f32, size = 0.000 MiB
[ 424/ 733] blk.22.ssm_norm.weight - [ 128, 1, 1, 1], type = f32, size = 0.000 MiB
[ 425/ 733] blk.22.ssm_out.weight - [ 4096, 2048, 1, 1], type = q8_0, size = 8.500 MiB
[ 426/ 733] blk.23.attn_k.weight - [ 2048, 512, 1, 1], type = q8_0, converting to tq4_1s .. size = 1.06 MiB -> 0.62 MiB
[ 427/ 733] blk.23.attn_k_norm.weight - [ 256, 1, 1, 1], type = f32, size = 0.001 MiB
[ 428/ 733] blk.23.attn_norm.weight - [ 2048, 1, 1, 1], type = f32, size = 0.008 MiB
[ 429/ 733] blk.23.attn_output.weight - [ 4096, 2048, 1, 1], type = q8_0, converting to tq4_1s .. size = 8.50 MiB -> 5.00 MiB
[ 430/ 733] blk.23.attn_q.weight - [ 2048, 8192, 1, 1], type = q8_0, converting to tq4_1s .. size = 17.00 MiB -> 10.00 MiB
[ 431/ 733] blk.23.attn_q_norm.weight - [ 256, 1, 1, 1], type = f32, size = 0.001 MiB
[ 432/ 733] blk.23.attn_v.weight - [ 2048, 512, 1, 1], type = q8_0, converting to tq4_1s .. size = 1.06 MiB -> 0.62 MiB
[...]
[ 729/ 733] blk.39.ffn_gate_inp_shexp.weight - [ 2048, 1, 1, 1], type = f32, size = 0.008 MiB
[ 730/ 733] blk.39.ffn_gate_shexp.weight - [ 2048, 512, 1, 1], type = q8_0, size = 1.062 MiB
[ 731/ 733] blk.39.ffn_up_exps.weight - [ 2048, 512, 256, 1], type = q8_0, size = 272.000 MiB
[ 732/ 733] blk.39.ffn_up_shexp.weight - [ 2048, 512, 1, 1], type = q8_0, size = 1.062 MiB
[ 733/ 733] blk.39.post_attention_norm.weight - [ 2048, 1, 1, 1], type = f32, size = 0.008 MiB
llama_model_quantize_impl: model size = 35183.10 MiB (8.52 BPW)
llama_model_quantize_impl: quant size = 35069.35 MiB (8.49 BPW)`

[TheTom]

thank you for the continued benchmarks!

[TheTom]

I guess you're done here anyway, yet wanting to let you know that the compressing as described in Getting started seems to either miss something or isn't very clear, at least in my case I wasn't able to replicate the "Quick Test" as the output model (Qwen 3.5 35b) always turned out nearly the same size with config_i except for a few MBs.

Or do I need to change the config_i for the 35b model somewhat?

str: cannot properly format tensor name position_embd with suffix=weight bid=-1 xid=-1 str: cannot properly format tensor name token_types with suffix=weight bid=-1 xid=-1 [...] llama_tensor_get_type: blk.15.attn_k.weight - applying manual override: q8_0 -> tq4_1s str: cannot properly format tensor name position_embd with suffix=weight bid=-1 xid=-1 str: cannot properly format tensor name token_types with suffix=weight bid=-1 xid=-1 str: cannot properly format tensor name position_embd with suffix=weight bid=-1 xid=-1 str: cannot properly format tensor name token_types with suffix=weight bid=-1 xid=-1 llama_tensor_get_type: blk.15.attn_output.weight - applying manual override: q8_0 -> tq4_1s str: cannot properly format tensor name position_embd with suffix=weight bid=-1 xid=-1 str: cannot properly format tensor name token_types with suffix=weight bid=-1 xid=-1 str: cannot properly format tensor name position_embd with suffix=weight bid=-1 xid=-1 str: cannot properly format tensor name token_types with suffix=weight bid=-1 xid=-1 llama_tensor_get_type: blk.15.attn_q.weight - applying manual override: q8_0 -> tq4_1s str: cannot properly format tensor name position_embd with suffix=weight bid=-1 xid=-1 str: cannot properly format tensor name token_types with suffix=weight bid=-1 xid=-1 str: cannot properly format tensor name position_embd with suffix=weight bid=-1 xid=-1 str: cannot properly format tensor name token_types with suffix=weight bid=-1 xid=-1 llama_tensor_get_type: blk.15.attn_v.weight - applying manual override: q8_0 -> tq4_1s str: cannot properly format tensor name position_embd with suffix=weight bid=-1 xid=-1 str: cannot properly format tensor name token_types with suffix=weight bid=-1 xid=-1 [...] str: cannot properly format tensor name position_embd with suffix=weight bid=-1 xid=-1 str: cannot properly format tensor name token_types with suffix=weight bid=-1 xid=-1 [ 1/ 733] output.weight - [ 2048, 248320, 1, 1], type = q8_0, size = 515.312 MiB [ 2/ 733] output_norm.weight - [ 2048, 1, 1, 1], type = f32, size = 0.008 MiB [ 3/ 733] token_embd.weight - [ 2048, 248320, 1, 1], type = q8_0, size = 515.312 MiB [ 4/ 733] blk.0.attn_gate.weight - [ 2048, 4096, 1, 1], type = q8_0, size = 8.500 MiB [ 5/ 733] blk.0.attn_norm.weight - [ 2048, 1, 1, 1], type = f32, size = 0.008 MiB [ 6/ 733] blk.0.attn_qkv.weight - [ 2048, 8192, 1, 1], type = q8_0, size = 17.000 MiB [...] [ 419/ 733] blk.22.ssm_a - [ 32, 1, 1, 1], type = f32, size = 0.000 MiB [ 420/ 733] blk.22.ssm_alpha.weight - [ 2048, 32, 1, 1], type = q8_0, size = 0.066 MiB [ 421/ 733] blk.22.ssm_beta.weight - [ 2048, 32, 1, 1], type = q8_0, size = 0.066 MiB [ 422/ 733] blk.22.ssm_conv1d.weight - [ 4, 8192, 1, 1], type = f32, size = 0.125 MiB [ 423/ 733] blk.22.ssm_dt.bias - [ 32, 1, 1, 1], type = f32, size = 0.000 MiB [ 424/ 733] blk.22.ssm_norm.weight - [ 128, 1, 1, 1], type = f32, size = 0.000 MiB [ 425/ 733] blk.22.ssm_out.weight - [ 4096, 2048, 1, 1], type = q8_0, size = 8.500 MiB [ 426/ 733] blk.23.attn_k.weight - [ 2048, 512, 1, 1], type = q8_0, converting to tq4_1s .. size = 1.06 MiB -> 0.62 MiB [ 427/ 733] blk.23.attn_k_norm.weight - [ 256, 1, 1, 1], type = f32, size = 0.001 MiB [ 428/ 733] blk.23.attn_norm.weight - [ 2048, 1, 1, 1], type = f32, size = 0.008 MiB [ 429/ 733] blk.23.attn_output.weight - [ 4096, 2048, 1, 1], type = q8_0, converting to tq4_1s .. size = 8.50 MiB -> 5.00 MiB [ 430/ 733] blk.23.attn_q.weight - [ 2048, 8192, 1, 1], type = q8_0, converting to tq4_1s .. size = 17.00 MiB -> 10.00 MiB [ 431/ 733] blk.23.attn_q_norm.weight - [ 256, 1, 1, 1], type = f32, size = 0.001 MiB [ 432/ 733] blk.23.attn_v.weight - [ 2048, 512, 1, 1], type = q8_0, converting to tq4_1s .. size = 1.06 MiB -> 0.62 MiB [...] [ 729/ 733] blk.39.ffn_gate_inp_shexp.weight - [ 2048, 1, 1, 1], type = f32, size = 0.008 MiB [ 730/ 733] blk.39.ffn_gate_shexp.weight - [ 2048, 512, 1, 1], type = q8_0, size = 1.062 MiB [ 731/ 733] blk.39.ffn_up_exps.weight - [ 2048, 512, 256, 1], type = q8_0, size = 272.000 MiB [ 732/ 733] blk.39.ffn_up_shexp.weight - [ 2048, 512, 1, 1], type = q8_0, size = 1.062 MiB [ 733/ 733] blk.39.post_attention_norm.weight - [ 2048, 1, 1, 1], type = f32, size = 0.008 MiB llama_model_quantize_impl: model size = 35183.10 MiB (8.52 BPW) llama_model_quantize_impl: quant size = 35069.35 MiB (8.49 BPW)`

Good catch. Qwen3.5-35B is a hybrid model — only 16 of 64 layers have split attn_q/attn_k/attn_v tensors (blk.3, 7, 11, 15, ..., 63). The other 48 are Gated Delta Net layers with fused attn_qkv tensors that don't match the Config I pattern, so they stay at q8_0.

You need to adjust n_layers to 64 for this model, but the real issue is that the config pattern only hits the self-attention layers. The fused attn_qkv and SSM tensors in delta net layers aren't targeted, so compression on this model will be smaller than expected (~30% vs ~37% on pure attention models like Qwen2.5-27B).

Compressing fused attn_qkv would need adding it as a target in the config file. Haven't validated that yet — delta net attention may have different sensitivity. i'll check on this more tomororw

[Titaniumtown]

I'm working on a vulkan port of this. PR soon