tts : add native Higgs Audio v3 support

[mirek190]

Overview

Adds a native llama.cpp-side Higgs Audio v3 TTS path around the Qwen3 backbone.

This includes:

• Higgs companion GGUF conversion/loading support
• native codebook sampling, RVQ, DAC, HuBERT/semantic/acoustic reference encoding pieces
• llama-tts/llama-higgs-tts/llama-higgs-reference-encode tooling
• /v1/audio/speech server wiring for Higgs speech generation
• focused Higgs unit/fixture tests and local documentation

Models for test
https://huggingface.co/mirek190/higgs_v3_gguf/tree/main

Original model
https://huggingface.co/bosonai/higgs-audio-v3-tts-4b

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure:

[ggml-gh-bot]

Hi @mirek190, thanks for your contribution!

Per our contribution guidelines, the automated PR checker found the following issue(s) that need your attention:

• Large PR: Large changes require prior discussion (e.g. an issue or RFC) and maintainers may not be able to review this PR as-is. Consider splitting it into smaller, focused PRs.

---

Please note that maintainers reserve the right to make final decisions on PRs. If you believe there is a mistake, please comment below.

[mirek190]

Models for test

https://huggingface.co/mirek190/higgs_v3_gguf/upload/main

That higgs v3 is fast.

• 21 seconds of audio from text ,

GPU RTX 3090 , Ryzen 7950

CPU - MODEL Q8 + CPU DAC -> 30 GB RAM ( around 43 seconds but even using 16 thread I have only CPU 20 % load)
Vulkan - MODEL Q8 + CPU DAC -> 8.5 GB VRAM ( around 6 seconds )
Vulkan - MODEL Q8 + Vulcan DAC -> 23 GB VRAM ( the fastest, around 4 seconds )
CUDA - MODEL Q8 + CPU DAC -> 8.9 GB VRAM ( around 6 seconds )

[ServeurpersoCom]

Honestly this is probably too big to land as-is (+ AI usage disclosure not respected), and there's already a lot of audio work in flight.
Did you study how this vocoder actually runs under the hood? It's a BigVGAN (see https://arxiv.org/abs/2206.04658), like almost everything that does audio, so this matters.

• Snake: adds a periodic inductive bias via sin²(αx), so the vocoder models and extrapolates audio harmonics where ReLU saturates.
• Deconv: the transposed conv upsamples frames to waveform; as mul_mat + col2im_1d it's a backend GEMM plus overlap-add fold instead of the naive scatter.

The Snake doesn't hit the graph fusion the way it's written (the ggml_repeat break the pattern), and the transposed conv should go through col2im_1d instead of conv_transpose_1d. Right now col2im_1d is Vulkan only, so on CUDA it'll fall back to CPU until the CUDA kernel lands (soon), but once you respect the Snake graph pattern matcher and move to col2im_1d it'll be way faster!

[mirek190]

I was working on it from 2 weeks
Of course AI was helping me (who is not using AI nowadays for coding? ) GPT with codex-cli was explaining me problems and allowing me to understand what I am doing but I was writing a code as well.

"The Snake doesn't hit the graph fusion the way it's written (the ggml_repeat break the pattern), and the transposed conv should go through col2im_1d instead of conv_transpose_1d. Right now col2im_1d is Vulkan only, so on CUDA it'll fall back to CPU until the CUDA kernel lands (soon), but once you respect the Snake graph pattern matcher and move to col2im_1d it'll be way faster!"

So
reference WAV -> HuBERT feature extractor -> HuBERT encoder -> semantic encoder -> DAC acoustic encoder -> FC -> RVQ encoder -> reference codes JSON

DAC ....

Replaced DAC transposed conv graph ggml_conv_transpose_1d from ggml_mul_mat to existing already ggml_col2im_1d for cuda

I changed that under higgs-dac-backend.h

now is

mul -> sin -> sqr -> mul -> add

Vulkan DAC uses native ggml_conv_transpose_1d
CUDA uses col2im_1d which was a bit more difficult as depends on ... col2im_1d

Tests passes.
Now is fully working fast DAC for cuda as well. So I have now:

• Device: Vulkan1 with DAC: Vulkan1
• Device: CUDA0 with DAC: CUDA0
• CPU remains DAC: CPU only

for vulkan , cuda

llama-server.exe -m higgs-qwen3-backbone-q8_0.gguf --mmproj higgs-audio-f16.gguf --host 127.0.0.1 --port 8096 --device Vulkan1 -ngl 99 -c 1024 --parallel 1 --no-webui

Continuation is here

#24780

[ServeurpersoCom]

#24417 col2im_1d cuda kernel just merged.
I'm going to PR Metal too. The ops must be impeccable and consistent across the backend.
You can try it and see how much faster it is
The main challenge is to make MTMD's APIs flexible enough to support its models more easily.

[mirek190]

#24417 col2im_1d cuda kernel just merged. I'm going to PR Metal too. The ops must be impeccable and consistent across the backend. You can try it and see how much faster it is The main challenge is to make MTMD's APIs flexible enough to support its models more easily.

YEP that kernel is fast

look

VULKAN
Higgs timing: prompt_decode=182.491 ms codebook=51.498 ms embed=1.431 ms step_decode=959.972 ms rvq=13.381 ms dac=194.013 ms stream_decode=0.000 ms steps=168 avg_step_decode=5.748 ms

CUDA
Higgs timing: prompt_decode=19.982 ms codebook=34.360 ms embed=1.395 ms step_decode=39.827 ms rvq=3.518 ms dac=206.493 ms stream_decode=0.000 ms steps=168 avg_step_decode=0.238 ms

[mirek190]

#24417 col2im_1d cuda kernel just merged. I'm going to PR Metal too. The ops must be impeccable and consistent across the backend. You can try it and see how much faster it is The main challenge is to make MTMD's APIs flexible enough to support its models more easily.

I actually working on OMTD ( output multi modal ) library handling audio as first and other modalities in the future as MTMD is a multimodal input library.
So llama-tts, llama-higgs-tts (my current implementation ), and llama-higgs-probe go into OMTD