Skip to content
Merged
Show file tree
Hide file tree
Changes from 130 commits
Commits
Show all changes
137 commits
Select commit Hold shift + click to select a range
277a96f
Add xcodec model
Apr 29, 2025
349feae
code formatting
Apr 29, 2025
e5f1da8
typo xcodec2 name
Deep-unlearning May 20, 2025
fc0907c
add xcodec2 in init file
Deep-unlearning May 20, 2025
ea0acbf
fix import
Deep-unlearning May 20, 2025
8542db7
fix weight_norm init
Deep-unlearning May 21, 2025
e98d981
remove unused import
Deep-unlearning May 21, 2025
d1cd3ac
add convert file
Deep-unlearning May 26, 2025
74fa506
add ModelOutput class
Deep-unlearning May 26, 2025
02f5c94
nit
Deep-unlearning May 26, 2025
dd0a17c
fix device issue
May 27, 2025
3786203
fix forward
May 27, 2025
93dbfad
nit
May 27, 2025
d4d8c6a
doc draft
May 27, 2025
c40912e
draft test
May 27, 2025
17eb48c
match tensor with the orignal implementation
Jun 3, 2025
3760438
Add doc file for xcodec2
Jun 4, 2025
a2faa55
finish model doc for xcodec2
Jun 4, 2025
31319fb
update doc
Deep-unlearning Jun 5, 2025
8d9f8df
working xcodec2
Deep-unlearning Jun 5, 2025
e5a1838
add test file for xcodec2
Deep-unlearning Jun 5, 2025
473f95a
nit
Deep-unlearning Jun 24, 2025
dd8aace
xcodec2 use EncodecFeatureExtractor
Deep-unlearning Jul 7, 2025
f6cf875
Merge branch 'main' into add-xcodec2
ebezzam Aug 26, 2025
244bdb6
Standardize with Xcodec.
ebezzam Aug 29, 2025
a84a69f
Merge branch 'main' into add-xcodec2
ebezzam Aug 29, 2025
9d743e8
Merge branch 'main' into add-xcodec2
ebezzam Aug 29, 2025
2e23505
Address some PR comments and standardize.
ebezzam Sep 2, 2025
0316080
Remove Sequential.
ebezzam Sep 3, 2025
fcbeab7
Remove weight norm from model definition.
ebezzam Sep 3, 2025
3c50dd2
Remove padding.
ebezzam Sep 3, 2025
bfe535b
Better use of modular and better init.
ebezzam Sep 3, 2025
f287f6a
Style and format checks.
ebezzam Sep 3, 2025
88cc8a7
Address some modeling tests.
ebezzam Sep 4, 2025
8eddd59
Better use of modular for Attention and cleanup.
ebezzam Sep 5, 2025
dda588b
Remove asserts, expose params in config.
ebezzam Sep 5, 2025
1218679
Clean up internal and better docstrings.
ebezzam Sep 5, 2025
5378c81
Fix example.
ebezzam Sep 5, 2025
1005594
Make clear how self.causal is used
ebezzam Sep 6, 2025
ee09e64
Add padding consistent to original.
ebezzam Sep 6, 2025
857562f
Clean up and matching integration test.
ebezzam Sep 8, 2025
a92c67c
Update docs and clean up.
ebezzam Sep 8, 2025
de3a7f8
Correct doc location among audio models.
ebezzam Sep 8, 2025
767208b
Remove flash tests.
ebezzam Sep 8, 2025
48a69c2
Skip flash attention tests.
ebezzam Sep 8, 2025
2dfb96e
Switch to processor.
ebezzam Sep 26, 2025
1cb9e89
New feature extractor.
ebezzam Oct 2, 2025
892d8e8
Update modular.
ebezzam Oct 2, 2025
edf0738
Add feature extractor.
ebezzam Oct 2, 2025
185afea
Clean feature extractor and expose some parameters.
ebezzam Oct 2, 2025
755aaec
Clean up feature extractor and add torch support (wip).
ebezzam Oct 2, 2025
2a2d037
Clean up seamless feature extractor.
ebezzam Oct 2, 2025
af636f9
Modular cleanup.
ebezzam Oct 2, 2025
fcb0ee5
Remove processor.
ebezzam Oct 2, 2025
e10b04a
Update docs.
ebezzam Oct 2, 2025
bc50545
Fix copy paths.
ebezzam Oct 3, 2025
5493319
Fix modeling tests for audio_spectrogram input.
ebezzam Oct 3, 2025
2908abd
Fix torch support with new spectrogram torch utility.
ebezzam Oct 3, 2025
cf9fd28
Repo consistency.
ebezzam Oct 3, 2025
d97b48c
Feature extraction tests.
ebezzam Oct 3, 2025
3c0a5c7
Merge branch 'main' into add-xcodec2
ebezzam Oct 3, 2025
4d78a95
Make style happy
ebezzam Oct 3, 2025
bd9f37f
Remove unprotected import.
ebezzam Oct 3, 2025
3d365e5
Another unprotected import.
ebezzam Oct 3, 2025
6b812a0
Remove more unprotected torches.
ebezzam Oct 3, 2025
2bfc30a
zero_mean_unit_var_norm needed for a test
ebezzam Oct 3, 2025
1911438
Modify Vocos component to be able to use modular later.
ebezzam Oct 3, 2025
dd3f45f
Update modular
ebezzam Oct 3, 2025
8cfce62
Make style happy.
ebezzam Oct 3, 2025
a76717a
Merge branch 'main' into add-xcodec2
ebezzam Feb 20, 2026
3650d87
Regenerate modular, update rope config.
ebezzam Feb 27, 2026
a4cca27
Update init weights
ebezzam Mar 13, 2026
e93c846
Add to init weights
ebezzam Mar 13, 2026
86ee458
FIxed _init_weights
ebezzam Mar 13, 2026
1a8b461
Integration passing!
ebezzam Mar 13, 2026
a2dbfdb
Cleanup
ebezzam Mar 13, 2026
4129a27
Better use of modular, and cleanup.
ebezzam Mar 18, 2026
ab62936
Merge branch 'main' into add-xcodec2
ebezzam Mar 18, 2026
4ffff44
Update config after mering with main, and other nits.
ebezzam Mar 18, 2026
b5ec79a
Merge branch 'main' into add-xcodec2
ebezzam Mar 18, 2026
642b56c
Address tests and other nits.
ebezzam Mar 18, 2026
1fbe78d
Nits, better wav2vec2 bert init
ebezzam Mar 18, 2026
8570ded
CLean up feature extraction and other nits
ebezzam Mar 19, 2026
15e9349
doc nits
ebezzam Mar 19, 2026
7bd81c4
Nit
ebezzam Mar 19, 2026
a97cc2c
Simplify feature extraction.
ebezzam Mar 19, 2026
29ce7fd
Torch compiile compatibility.
ebezzam Apr 13, 2026
02e92ac
Rename output entries, semantic encoder. Move weight norm utilities. …
ebezzam Apr 28, 2026
811aa2a
Add batch test and spectrogram padding for equivalence.
ebezzam Apr 28, 2026
672e1d2
Refactoring.
ebezzam Apr 30, 2026
d221fbf
Remove defensive trimming.
ebezzam Apr 30, 2026
c2ab39c
Merge branch 'main' into add-xcodec2
ebezzam Apr 30, 2026
4c6f001
Make style happy.
ebezzam Apr 30, 2026
6a01e49
Apply suggestion from @vasqu
ebezzam Jun 4, 2026
dad49b0
Fix feature extraction tests, remove integration since original doesn…
ebezzam Jun 4, 2026
72f19cf
Update date.
ebezzam Jun 4, 2026
49d3671
Merge branch 'add-xcodec2' of github.com:ebezzam/transformers into ad…
ebezzam Jun 4, 2026
7aa53eb
Address docs comments.
ebezzam Jun 4, 2026
63a86b8
Apply suggestion from @vasqu
ebezzam Jun 4, 2026
4755347
Apply suggestion from @vasqu
ebezzam Jun 4, 2026
17a4b2b
Address various comments.
ebezzam Jun 4, 2026
7fd8f9e
Merge branch 'add-xcodec2' of github.com:ebezzam/transformers into ad…
ebezzam Jun 4, 2026
f2ee118
Add import for RopeParameters
ebezzam Jun 4, 2026
0ba5a74
Address feature extractor comments.
ebezzam Jun 4, 2026
9e3a38d
Adding note on spectrogram computation.
ebezzam Jun 4, 2026
1132c46
Address smaller modeling comments.
ebezzam Jun 4, 2026
0473f59
Address various modeling comments.
ebezzam Jun 5, 2026
f5d2968
Merge branch 'main' into add-xcodec2
ebezzam Jun 5, 2026
fc2abde
Style fixes after merge.
ebezzam Jun 5, 2026
eacfd3b
better api for xcodec forward
ebezzam Jun 15, 2026
2ba74c2
Better module naming in Qwen2.5 Omni
ebezzam Jun 15, 2026
b88a748
Address various comments.
ebezzam Jun 15, 2026
c7e3247
Add comments, add audio utils, address nits.
ebezzam Jun 16, 2026
4126e85
Merge branch 'main' into add-xcodec2
ebezzam Jun 16, 2026
b658be7
Keep current audio input names, various nits.
ebezzam Jun 16, 2026
070af74
Add casting for cpu
ebezzam Jun 16, 2026
ffddbcc
Merge branch 'main' into add-xcodec2
ebezzam Jun 18, 2026
7a99db7
Move utils for testing audio.
ebezzam Jun 18, 2026
2a02a01
Merge branch 'add-xcodec2' of github.com:ebezzam/transformers into ad…
ebezzam Jun 18, 2026
4320269
Merge branch 'main' into add-xcodec2
ebezzam Jun 18, 2026
b8df2c7
Merge branch 'main' into add-xcodec2
ebezzam Jun 23, 2026
65f73f1
Update checkpoint name.
ebezzam Jun 23, 2026
e92c4c4
Merge branch 'add-xcodec2' of github.com:ebezzam/transformers into ad…
ebezzam Jun 23, 2026
f230672
Merge branch 'main' into add-xcodec2
ebezzam Jun 23, 2026
8c9b8d9
FIx for torch compile
ebezzam Jun 23, 2026
7d91a28
Merge branch 'add-xcodec2' of github.com:ebezzam/transformers into ad…
ebezzam Jun 23, 2026
dc3378d
Merge branch 'main' into add-xcodec2
ebezzam Jun 23, 2026
0142779
Set test config for offload to pass.
ebezzam Jun 23, 2026
512c681
Merge branch 'add-xcodec2' of github.com:ebezzam/transformers into ad…
ebezzam Jun 23, 2026
a31ed1c
Increase allowed num layers.
ebezzam Jun 24, 2026
78c3ed9
Config that passes cpu offload.
ebezzam Jun 24, 2026
6d9c51c
test config for disk/cpu offload
ebezzam Jun 24, 2026
a5c07db
test nits
ebezzam Jun 24, 2026
2fe0150
Merge branch 'main' into add-xcodec2
ebezzam Jun 24, 2026
12e6464
Merge branch 'add-xcodec2' of github.com:ebezzam/transformers into ad…
ebezzam Jun 25, 2026
975813f
update date
ebezzam Jun 25, 2026
6242646
Merge branch 'main' into add-xcodec2
ebezzam Jun 25, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions docs/source/en/_toctree.yml
Original file line number Diff line number Diff line change
Expand Up @@ -1165,6 +1165,8 @@
title: Whisper
- local: model_doc/xcodec
title: X-Codec
- local: model_doc/xcodec2
title: X-Codec2
- local: model_doc/xls_r
title: XLS-R
- local: model_doc/xlsr_wav2vec2
Expand Down
161 changes: 161 additions & 0 deletions docs/source/en/model_doc/xcodec2.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,161 @@
<!--Copyright 2026 The HuggingFace Team. All rights reserved.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
specific language governing permissions and limitations under the License.

⚠️ Note that this file is in Markdown but contain specific syntax for our doc-builder (similar to MDX) that may not be
rendered properly in your Markdown viewer.

-->
*This model was published in HF papers on 2025-02-06 and contributed to Hugging Face Transformers on 2026-06-23.*

# X-Codec2

<div class="flex flex-wrap space-x-1">
<img alt="SDPA" src="https://img.shields.io/badge/SDPA-DE3412?style=flat&logo=pytorch&logoColor=white">
</div>

## Overview

The X-Codec2 model was proposed in [Llasa: Scaling Train-Time and Inference-Time Compute for Llama-based Speech Synthesis](https://huggingface.co/papers/2502.04128).

X-Codec2 is a neural audio codec designed to improve speech synthesis and general audio generation for large language model (LLM) pipelines. It extends the original X-Codec by refining how semantic and acoustic information is integrated and tokenized, enabling efficient and high-fidelity audio representation.

About its architecture:
- **Unified Semantic-Acoustic Tokenization**: X-Codec2 fuses outputs from a semantic encoder (e.g., Wav2Vec2-BERT) and an acoustic encoder into a single embedding, capturing both high-level meaning (e.g., text content, emotion) and low-level audio details (e.g., timbre).
- **Single-Stage Feature Scalar Quantization (FSQ)**: Unlike the multi-layer residual VQ in most approaches (e.g., [DAC](./dac), [EnCodec](./encodec), [X-Codec](./xcodec), [Mimi](./mimi.md)), X-Codec2 uses a single-layer of Feature Scalar Quantization (FSQ) for stability and compatibility with causal, autoregressive LLMs.
- **Transformer-Friendly Design**: The 1D token structure of X-Codec2 naturally aligns with the autoregressive modeling in LLMs like LLaMA, improving training efficiency and downstream compatibility.

A model checkpoint is available at [HKUSTAudio/xcodec2-hf](https://huggingface.co/HKUSTAudio/xcodec2-hf).

This model was contributed by [Eric Bezzam](https://huggingface.co/bezzam) and [Steven Zheng](https://huggingface.co/Steveeeeeeen).
The original modeling code can be found [here](https://huggingface.co/HKUSTAudio/xcodec2/blob/main/modeling_xcodec2.py), while their training code is [here](https://github.com/zhenye234/X-Codec-2.0).

## Usage example

Here is a quick example of how to encode and decode an audio using this model:

```python
from datasets import Audio, load_dataset
from transformers import AutoFeatureExtractor, AutoModel

model_id = "HKUSTAudio/xcodec2-hf"
model = AutoModel.from_pretrained(model_id, device_map="auto")
feature_extractor = AutoFeatureExtractor.from_pretrained(model_id)

dataset = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
dataset = dataset.cast_column("audio", Audio(sampling_rate=feature_extractor.sampling_rate))
audio = dataset[0]["audio"]["array"]
inputs = feature_extractor(audio=audio, sampling_rate=feature_extractor.sampling_rate, return_tensors="pt").to(
model.device, model.dtype
)
print("Input waveform shape:", inputs["input_values"].shape)
# Input waveform shape: torch.Size([1, 1, 93760])

# encoder and decoder
audio_codes = model.encode(**inputs).audio_codes
print("Audio codes shape:", audio_codes.shape)
# Audio codes shape: torch.Size([1, 1, 293])
audio_values = model.decode(audio_codes).audio_values
print("Audio values shape:", audio_values.shape)
# Audio values shape: torch.Size([1, 1, 93760])

# Equivalently, you can do encoding and decoding in one step
model_output = model(**inputs)
audio_codes = model_output.audio_codes
audio_values = model_output.audio_values
```

### Batch processing

This implementation also supports batched input, unlike the original [release](https://huggingface.co/HKUSTAudio/xcodec2)!
Comment thread
vasqu marked this conversation as resolved.
Resolved

```python
from datasets import Audio, load_dataset
from transformers import AutoFeatureExtractor, AutoModel

batch_size = 2
model_id = "HKUSTAudio/xcodec2-hf"
model = AutoModel.from_pretrained(model_id, device_map="auto")
feature_extractor = AutoFeatureExtractor.from_pretrained(model_id)

dataset = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
dataset = dataset.cast_column("audio", Audio(sampling_rate=feature_extractor.sampling_rate))
audios = [dataset[i]["audio"]["array"] for i in range(batch_size)]
inputs = feature_extractor(audio=audios, sampling_rate=feature_extractor.sampling_rate, return_tensors="pt").to(
model.device, model.dtype
)
print("Input waveform shape:", inputs["input_values"].shape)
# Input waveform shape: torch.Size([2, 1, 93760])

# encoder and decoder
encoder_output = model.encode(**inputs)
audio_codes = encoder_output.audio_codes
print("Audio codes shape:", audio_codes.shape)
# Audio codes shape: torch.Size([2, 1, 293])
audio_values = model.decode(audio_codes).audio_values
print("Audio values shape:", audio_values.shape)
# Audio values shape: torch.Size([2, 1, 93760])

# Equivalently, you can do encoding and decoding in one step
model_output = model(**inputs)
audio_codes = model_output.audio_codes
audio_values = model_output.audio_values
```

### Speed-up with `torch.compile`
Comment thread
vasqu marked this conversation as resolved.
Resolved

You can speed up inference with [`torch.compile`](https://pytorch.org/docs/stable/generated/torch.compile.html). The first few calls will be slower due to compilation overhead, but subsequent calls will be faster.

On an A100, we observed a speed-up of ~1.35 for a batch size of 4 ([script](https://gist.github.com/ebezzam/3b79481b5d48d8e35c4ecc582aee0cb3#file-benchmark_torch_compile-py)).

```python
import torch
from datasets import Audio, load_dataset
from transformers import AutoFeatureExtractor, AutoModel

batch_size = 4
model_id = "HKUSTAudio/xcodec2-hf"
model = AutoModel.from_pretrained(model_id, device_map="auto")
feature_extractor = AutoFeatureExtractor.from_pretrained(model_id)

dataset = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
dataset = dataset.cast_column("audio", Audio(sampling_rate=feature_extractor.sampling_rate))
audios = [dataset[i]["audio"]["array"] for i in range(batch_size)]
inputs = feature_extractor(
audio=audios, sampling_rate=feature_extractor.sampling_rate, padding=True, return_tensors="pt"
).to(model.device, model.dtype)

compiled_model = torch.compile(model, fullgraph=True)

# Warmup (includes compilation on first call)
for _ in range(10):
with torch.inference_mode():
_ = compiled_model(**inputs)

with torch.inference_mode():
output = compiled_model(**inputs)
print("Audio values shape:", output.audio_values.shape)
```

## Xcodec2Config

[[autodoc]] Xcodec2Config

## Xcodec2FeatureExtractor

[[autodoc]] Xcodec2FeatureExtractor
- __call__

## Xcodec2Model

[[autodoc]] Xcodec2Model
- decode
- encode
- forward
1 change: 1 addition & 0 deletions src/transformers/models/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -484,6 +484,7 @@
from .whisper import *
from .x_clip import *
from .xcodec import *
from .xcodec2 import *
from .xglm import *
from .xlm import *
from .xlm_roberta import *
Expand Down
2 changes: 2 additions & 0 deletions src/transformers/models/auto/auto_mappings.py
Original file line number Diff line number Diff line change
Expand Up @@ -666,6 +666,7 @@
("xclip_text_model", "XCLIPTextConfig"),
("xclip_vision_model", "XCLIPVisionConfig"),
("xcodec", "XcodecConfig"),
("xcodec2", "Xcodec2Config"),
("xglm", "XGLMConfig"),
("xlm", "XLMConfig"),
("xlm-roberta", "XLMRobertaConfig"),
Expand Down Expand Up @@ -968,6 +969,7 @@
("voxtral_realtime", "VoxtralRealtimeFeatureExtractor"),
("wav2vec2", "Wav2Vec2FeatureExtractor"),
("whisper", "WhisperFeatureExtractor"),
("xcodec2", "Xcodec2FeatureExtractor"),
]
)

Expand Down
2 changes: 2 additions & 0 deletions src/transformers/models/auto/modeling_auto.py
Comment thread
ebezzam marked this conversation as resolved.
Resolved
Original file line number Diff line number Diff line change
Expand Up @@ -530,6 +530,7 @@ class _BaseModelWithGenerate(PreTrainedModel, GenerationMixin):
("whisper", "WhisperModel"),
("xclip", "XCLIPModel"),
("xcodec", "XcodecModel"),
("xcodec2", "Xcodec2Model"),
("xglm", "XGLMModel"),
("xlm", "XLMModel"),
("xlm-roberta", "XLMRobertaModel"),
Expand Down Expand Up @@ -1937,6 +1938,7 @@ class _BaseModelWithGenerate(PreTrainedModel, GenerationMixin):
("dac", "DacModel"),
("higgs_audio_v2_tokenizer", "HiggsAudioV2TokenizerModel"),
("vibevoice_acoustic_tokenizer", "VibeVoiceAcousticTokenizerModel"),
("xcodec2", "Xcodec2Model"),
Comment thread
ebezzam marked this conversation as resolved.
Resolved
]
)

Expand Down
5 changes: 2 additions & 3 deletions src/transformers/models/dac/modeling_dac.py
Original file line number Diff line number Diff line change
Expand Up @@ -445,18 +445,17 @@ class DacEncoder(nn.Module):
def __init__(self, config: DacConfig):
super().__init__()

strides = config.downsampling_ratios
# Create first convolution
self.conv1 = nn.Conv1d(1, config.encoder_hidden_size, kernel_size=7, padding=3)

self.block = []
# Create EncoderBlocks that double channels as they downsample by `stride`
for stride_index, stride in enumerate(strides):
for stride_index, stride in enumerate(config.downsampling_ratios):
stride_index = stride_index + 1
self.block += [DacEncoderBlock(config, stride=stride, stride_index=stride_index)]

self.block = nn.ModuleList(self.block)
d_model = config.encoder_hidden_size * 2**stride_index
d_model = config.encoder_hidden_size * 2 ** len(config.downsampling_ratios)
self.snake1 = Snake1d(d_model)
self.conv2 = nn.Conv1d(d_model, config.hidden_size, kernel_size=3, padding=1)

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,8 @@
from ... import initialization as init
from ...audio_utils import conv1d_output_length
from ...modeling_utils import PreTrainedAudioTokenizerBase
from ...utils import ModelOutput, auto_docstring
from ...processing_utils import Unpack
from ...utils import ModelOutput, TransformersKwargs, auto_docstring, can_return_tuple
from ...utils.import_utils import requires
from ..auto import AutoModel
from .configuration_higgs_audio_v2_tokenizer import HiggsAudioV2TokenizerConfig
Expand Down Expand Up @@ -589,12 +590,13 @@ def decode(
return HiggsAudioV2TokenizerDecoderOutput(audio_values)

@auto_docstring
@can_return_tuple
def forward(
self,
input_values: torch.Tensor,
audio_codes: torch.Tensor | None = None,
bandwidth: float | None = None,
return_dict: bool | None = None,
**kwargs: Unpack[TransformersKwargs],
) -> tuple[torch.Tensor, torch.Tensor] | HiggsAudioV2TokenizerOutput:
r"""
input_values (`torch.FloatTensor` of shape `(batch_size, channels, num_samples)`):
Expand All @@ -605,8 +607,6 @@ def forward(
Target bandwidth in kbps. Must be one of `config.target_bandwidths`. Defaults to the highest available bandwidth.
bandwidth (`float`, *optional*):
Target bandwidth in kbps. Must be one of `config.target_bandwidths`. Defaults to the highest available bandwidth.
return_dict (`bool`, *optional*):
Whether to return a [`HiggsAudioV2TokenizerOutput`] instead of a plain tuple.

Returns:
`HiggsAudioV2TokenizerOutput` or tuple `(audio_codes, audio_values)`:
Expand Down Expand Up @@ -634,16 +634,12 @@ def forward(
>>> audio_values = outputs.audio_values
```
"""
return_dict = return_dict if return_dict is not None else self.config.return_dict
length = input_values.shape[-1]

if audio_codes is None:
audio_codes = self.encode(input_values, bandwidth, return_dict=False)

audio_values = self.decode(audio_codes, return_dict=return_dict)[0][..., :length]

if not return_dict:
return (audio_codes, audio_values)
audio_values = self.decode(audio_codes, return_dict=True)[0][..., :length]

return HiggsAudioV2TokenizerOutput(audio_codes=audio_codes, audio_values=audio_values)

Expand Down
5 changes: 2 additions & 3 deletions src/transformers/models/pe_audio/modeling_pe_audio.py
Original file line number Diff line number Diff line change
Expand Up @@ -128,18 +128,17 @@ class PeAudioDacEncoder(nn.Module):
def __init__(self, config: PreTrainedConfig):
super().__init__()

strides = config.downsampling_ratios
# Create first convolution
self.conv1 = nn.Conv1d(1, config.encoder_hidden_size, kernel_size=7, padding=3)

self.block = []
# Create EncoderBlocks that double channels as they downsample by `stride`
for stride_index, stride in enumerate(strides):
for stride_index, stride in enumerate(config.downsampling_ratios):
stride_index = stride_index + 1
self.block += [PeAudioDacEncoderBlock(config, stride=stride, stride_index=stride_index)]

self.block = nn.ModuleList(self.block)
d_model = config.encoder_hidden_size * 2**stride_index
d_model = config.encoder_hidden_size * 2 ** len(config.downsampling_ratios)
self.snake1 = Snake1d(d_model)
self.conv2 = nn.Conv1d(d_model, config.hidden_size, kernel_size=3, padding=1)

Expand Down
Loading
Loading