Skip to content

[Bug] HF transformers update breaks XTTS streaming #31

Closed

Description

@eginhard

Describe the bug

huggingface/transformers#30624 broke the XTTS streaming code (https://github.com/idiap/coqui-ai-TTS/blob/df088e99dfda8976c47235626d3afb7d7d70fea2/TTS/tts/layers/xtts/stream_generator.py) as reported in huggingface/transformers#31040. The streaming code needs to be updated accordingly.

To Reproduce

https://coqui-tts.readthedocs.io/en/latest/models/xtts.html#streaming-manually

Expected behavior

No response

Logs

Loading model...
DEBUG:fsspec.local:open file: ~/.local/share/tts/tts_models--multilingual--multi-dataset--xtts_v2/model.pth
Computing speaker latents...
Inference...
~/applications/miniconda3/envs/coqui-3.12/lib/python3.12/site-packages/TTS/tts/layers/xtts/stream_generator.py:138: UserWarning: You have modified the pretrained model configuration to control generation. This is a deprecated strategy to control generation and will be removed soon, in a future version. Please use a generation configuration file (see https://huggingface.co/docs/transformers/main_classes/text_generation)
  warnings.warn(
Traceback (most recent call last):
  File "~/projects/testing/xtts.py", line 33, in <module>
    for i, chunk in enumerate(chunks):
  File "~/applications/miniconda3/envs/coqui-3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 35, in generator_context
    response = gen.send(None)
               ^^^^^^^^^^^^^^
  File "~/applications/miniconda3/envs/coqui-3.12/lib/python3.12/site-packages/TTS/tts/models/xtts.py", line 657, in inference_stream
    gpt_generator = self.gpt.get_generator(
                    ^^^^^^^^^^^^^^^^^^^^^^^
  File "~/applications/miniconda3/envs/coqui-3.12/lib/python3.12/site-packages/TTS/tts/layers/xtts/gpt.py", line 602, in get_generator
    return self.gpt_inference.generate_stream(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/applications/miniconda3/envs/coqui-3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "~/applications/miniconda3/envs/coqui-3.12/lib/python3.12/site-packages/TTS/tts/layers/xtts/stream_generator.py", line 186, in generate
    model_kwargs["attention_mask"] = self._prepare_attention_mask_for_generation(
                                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/applications/miniconda3/envs/coqui-3.12/lib/python3.12/site-packages/transformers/generation/utils.py", line 473, in _prepare_attention_mask_for_generation
    torch.isin(elements=inputs, test_elements=pad_token_id).any()
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: isin() received an invalid combination of arguments - got (test_elements=int, elements=Tensor, ), but expected one of:
 * (Tensor elements, Tensor test_elements, *, bool assume_unique, bool invert, Tensor out)
 * (Number element, Tensor test_elements, *, bool assume_unique, bool invert, Tensor out)
 * (Tensor elements, Number test_element, *, bool assume_unique, bool invert, Tensor out)

Environment

{
    "CUDA": {
        "GPU": [],
        "available": false,
        "version": "12.1"
    },
    "Packages": {
        "PyTorch_debug": false,
        "PyTorch_version": "2.3.0+cu121",
        "TTS": "0.24.0",
        "numpy": "1.26.4"
    },
    "System": {
        "OS": "Linux",
        "architecture": [
            "64bit",
            "ELF"
        ],
        "processor": "x86_64",
        "python": "3.12.3",
        "version": "#35~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Tue May  7 09:00:52 UTC 2"
    }
}

Additional context

No response

Activity

  1. eginhard commented on May 29, 2024

    @eginhard
    MemberAuthor

    coqui-tts version 0.24.1 limits transformers to lower versions for a temporary fix until someone properly updates the streaming code.

  2. pseudotensor commented on May 29, 2024

    @pseudotensor
  3. eginhard commented on May 30, 2024

    @eginhard
    MemberAuthor
  4. self-assigned this
    on Jun 15, 2024
  5. added 3 commits that reference this issue on Jun 16, 2024
    47aa14a
    d501927
    4b6da4e
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions