XTTS v2 - please help out a noob

[andykaufseo]

I have a long list of questions, recently starting using coqui (xtts v2 cloned voice), i have some experience with LLMs and more experience Stable Diffusion, but i'm just getting started with TTS (and STT). Here are my questions:

1. i tested xtts v2 with a cloned voice, it sounds great - a bit slow tho but no biggie - question is, if this sounds so good, why is there a fine tuning setup for this? i mean, do i fine tune the base with the cloned voice - to sound even more like the cloned voice? if that's the case, if i clone another voice (while using the fine tuned model with the first cloned voice, what will happen)

2. as of today, which is the best model for cloning voices? xtts2 sounds great, i remember trying bark a few months ago when it got out and it was a mess, so was vits (code was messed up, couldn't get it to work and the demo was bad).

3. i see that with cloned voices, xtts2 is a bit slow (not a pain in the ass, but could be faster). i see some things that claim to do almost real time voice gen (https://github.com/KoljaB/RealtimeTTS) - is this possible with xtts2?

4. i remember Bark could insert things like laughter, music, etc - could i use that in combination with xtts2

[m000lie]

the finetuning setup is meant for you to further refined the base model when cloning your target voice. there are some instances where the base model doesn't perform to it's best on zero shot inference (unseen data).

as such, fine tuning is still necessary for a subset of use-cases.

[guptaprakhariitr]

Hi @andykaufseo, If you've found answers to these, please update them here. Would like to hear from your experience.

[MethanJess]

@andykaufseo

1. Fine-tuning a model can get your voice to sound more like the target voice, the 0-shot cloning is not perfect, I have noticed a massive difference between a finetuned model and a 0-shot one.
2. The best model i would definitely say is the one used on myvoice.speechfiy, it can clone even the most unique voices, and gives out very realistic voices that almost perfectly match the original audio, and can do very well even on a 1 minute sample, sadly it's locked behind a very hefty subscription, a free one is play.ht, but it is not as good, for open source ones, Metavoice and VoiceCraft are very good.
3. Yeah, that repository seems to be legit from just looking at it...
4. I could be wrong, but I believe that if you fine tune your own model and add laughter sounds in the sample then put in the transcript something like "[laughter]" then I think the model will be output a laughter sound each time you input "[laughter]", but not sure if there's a way currently to combine bark and coqui together...

[jash2129]

Does it have Telugu(Indian language) library.

[MethanJess]

@jash2129 not sure.... coqui supports 16 langues, but i doubt that Telugu is one of them...
You can try other open source projects like this one: https://github.com/JarodMica/ai-voice-cloning
I heard it can train on any language or something like that

[AjibolaPy]

I tried xtts base model with my own voice it got it quite close but the accent was very different. Will fine-tuning get my voice and accent correct, or close atleast?, And in finetuning do I need sample of only my voice or different voice with same accent

[SMK1311]

做到了吗，我也再微调使口音更加接近，发现效果很不好，可以给点意见吗

[rahafalhallay32]

short sample of the speaker's voice

[fkmaan-commits]

Can the generated voice quality increase by putting longer reference audio files?

[Deathgod999]

Hello everyone anyone pls give me s code how to use xtts v2 model 😁

[rahafalhallay32]

1. Download the Model Manually
   Go to:
   👉 https://huggingface.co/coqui/XTTS-v2

Click the “Files and versions” tab and download the following files:

model.pth
config.json
vocab.json
speakers_xtts.pth

Then create a folder named xtts_v2/ and put all the downloaded files inside it.

2. Create Your Script
   Create a Python file named xtts.py and use the Coqui TTS API to load the model and synthesize speech. Here's a minimal example:

from TTS.api import TTS

Load XTTS v2 from local directory

tts = TTS(model_path="xtts_v2/model.pth", config_path="xtts_v2/config.json", progress_bar=True)

Run synthesis

tts.tts_to_file(
text="......",
speaker_wav=["your_voice_sample.wav"],
language="....",
file_path="output.wav"
)
Make sure to replace "your_voice_sample.wav" with a real WAV file (short sample of the speaker's voice).

3. Set Up Your Environment
4. Run the Script

python xtts.py
You should get an output.wav file containing the synthesized speech.

[Hemilibeatriz]

Wow, this is so helpful, thank you so much @rahafalhallay32 !

[Deathgod999]

I always face some error, sometimes about the lang and sometimes about something else. Please can I get the update code for 2025 now?

[ClaudeRobbinCR]

Hi! Does coqui have a future?
I am looking for a reliable web ui!

[eginhard]

We'll continue maintaining the fork, yes, and that webui is already using it.

[ClaudeRobbinCR]

Yes. But will there be major releases?

Like v3 for example.

I want to helo but I'm only a student and my knowledge isn't deep enough.

If we were to work on v3 how would it be done?

1. We need to solve the acronym issue.

2. We need the a single language targeted model that NEVER stumbles in any pronunciation.

Can I do this as a solo dev?

[eginhard]

We're mainly maintaining the code, but sometimes new models are added, see the release notes for more details: https://github.com/idiap/coqui-ai-TTS/releases

If by v3 you mean a new version of the XTTS model, then no. We don't have any affiliation with the original Coqui company, so don't have access to the full model weights and code needed for training.

1. We need to solve the acronym issue.

This is independent of the TTS model. You could use external libraries to convert acronyms/numbers to text first.

2. We need the a single language targeted model that NEVER stumbles in any pronunciation.

Coqui includes many single-language models (e.g. Vits) that don't make pronunciation errors like XTTS, although the output speech quality won't be as good.

[ClaudeRobbinCR]

Oh that's really sad.
The community was onto something big here. XTTS sometimes hallucinates and sounds like a human person it's scary.

How much funding would you need to continue this work/obtain the weights you would need to build a new version of XTTS?

Can we build a community that can come together and pick up where the devs left?

Also I noticed that you volunteered to maintain this repo. Thank you so much for doing this!

XTTS helps me earn in my side gigs as a student. Thank you so much. 🙏

[SMK1311]

帮助下新手，为什么微调模型，我的 grad_norm: 0 (0.0) 是什么情况 是因为我的数据集太少了 只有100条

[Rajatavaa]

Please add support for Python 13 and later versions!

[fkmaan-commits]

Just install python 3.10 and use it.

[eginhard]

You can follow idiap#108 for updates, it's still blocked by dependencies, only 3.10-3.12 are supported in the fork (and 3.9-3.11 in this repo). Is there any specific reason you can only use 3.13?

[Rajatavaa]

Not specific but i have it as a primary interpreter installed though i have 3.11 which helps me to run and test the model,Thanks for the resource i'll go through it.
Meanwhile can you please specify that i have noticed there is no tagging system like [happy],[sad],[excited] like other models but there are still some emotional capabilities pre-built when i inference out something.So to make the model more emotionally capable what do i need to do?Do i have to train/finetune it on good emotional voice datasets and does it capture the emotional capabilities or fine-tuning is just for cloning?

[seanpowell75]

is python 3.12 definitely supported? if I run pip install tts on python 3.12.0 I get an error on version not supported

[seanpowell75]

Edit: with a bit more research, I see python 3.12 is only supported through coqui-tts fork, I will try that...