Skip to content

feat(speech): add pocket-tts CPU text-to-speech pipeline - #2350

Open
sandyz1000 wants to merge 3 commits into
EricLBuehler:masterfrom
sandyz1000:master
Open

feat(speech): add pocket-tts CPU text-to-speech pipeline#2350
sandyz1000 wants to merge 3 commits into
EricLBuehler:masterfrom
sandyz1000:master

Conversation

@sandyz1000

Copy link
Copy Markdown

Summary

Adds pocket-tts as a second speech pipeline alongside Dia, over the same OpenAI /v1/audio/speech contract. It's a Kyutai-style Mimi codec + FlowLM transformer that runs fast on CPU, so one mistral.rs binary can serve an LLM plus a CPU-fast TTS (Dia is GPU-oriented).

Selectable via serve -m (auto-detected), serve speech, or from-config; voice defaults to alba and is settable via --voice / config / SDK. /v1/audio/speech now routes by the request's model, so pocket-tts (CPU) and Dia (GPU) can be loaded together and picked per request.

Attribution

Model + inference code under mistralrs-core/src/speech_models/pockettts/ is ported from the pocket-tts Rust crate: https://github.com/babybirdprd/pocket-tts — mostly near-verbatim, recompiled against this repo's candle 0.11 (upstream pins 0.9.1). The rest is the mistral.rs integration layer.

Source is MIT OR Apache-2.0, this repo is MIT — compatible. Attribution is also in the module header. Opening transparently so maintainers (and the pocket-tts author, if they wish) can review provenance.

Verification

  • cargo check clean (0 warnings) and cargo fmt --all --check pass.
  • End-to-end on Apple Metal and over HTTP: kyutai/pocket-tts-without-voice-cloning produces healthy 24 kHz mono speech.

@sandyz1000 sandyz1000 changed the title Title: feat(speech): add pocket-tts CPU text-to-speech pipeline feat(speech): add pocket-tts CPU text-to-speech pipeline Jul 21, 2026
@github-actions

github-actions Bot commented Jul 21, 2026

Copy link
Copy Markdown
Code Metrics Report
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
 Language              Files        Lines         Code     Comments       Blanks
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
 C Header                 23         4454         3116          790          548
 CSS                       3          282          252            6           24
 CUDA                    121        25439        20835         1707         2897
 Dockerfile                1           35           19            9            7
 HTML                      2           27           27            0            0
 JavaScript                3          578          563           12            3
 Jinja2                    7          694          656            5           33
 JSON                     27        17989        17986            0            3
 Makefile                  1           18           16            0            2
 MDX                      36         6285            0         4616         1669
 Metal Shading Lan|       37        14394        11386         1136         1872
 PowerShell                1          657          571           31           55
 Python                  151        12539        10418          480         1641
 Shell                     3         1071          852          115          104
 Plain Text               53        10687            0         9209         1478
 TOML                     28         1389         1209           39          141
 TypeScript               11         1649         1410           66          173
 YAML                      3           25           23            2            0
─────────────────────────────────────────────────────────────────────────────────
 Jupyter Notebooks         3          122           83           23           16
 |- Markdown               1           60           30           22            8
 |- Python                 1          122          113            1            8
 (Total)                              304          226           46           32
─────────────────────────────────────────────────────────────────────────────────
 Markdown                273        12055            0         8981         3074
 |- BASH                  24          300          221           47           32
 |- Dockerfile             2           14           12            0            2
 |- JSON                   6          289          289            0            0
 |- PowerShell             1            1            1            0            0
 |- Python               135         7349         6120          306          923
 |- Rust                  62         3840         2849          394          597
 |- TOML                   7           92           78            0           14
 (Total)                            23940         9570         9728         4642
─────────────────────────────────────────────────────────────────────────────────
 Rust                    701       331371       296672         5815        28884
 |- Markdown             418         9947          452         8304         1191
 (Total)                           341318       297124        14119        30075
─────────────────────────────────────────────────────────────────────────────────
 Svelte                   19         1969         1826           51           92
 |- CSS                    1            4            4            0            0
 |- JavaScript            19          921          767           25          129
 (Total)                             2894         2597           76          221
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
 Total                  1507       466668       378856        42192        45620
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant