Skip to content
Merged
Show file tree
Hide file tree
Changes from 7 commits
Commits
Show all changes
85 commits
Select commit Hold shift + click to select a range
fa7d6e0
parakeet tdt intergration
Oct 13, 2025
f2b4938
Add TDT decoder support for Parakeet ASR models
lmaksym Feb 20, 2026
fa36657
Add expected outputs for TDT, small fixes.
ebezzam Feb 25, 2026
05e2e34
Separate CTC and TDT generate outputs.
ebezzam Feb 25, 2026
bb5ff33
Work with auto device, better init,
ebezzam Feb 25, 2026
9ec79b0
Test timestamps and expose token duration.
ebezzam Feb 26, 2026
33f128e
Add reproducer link.
ebezzam Feb 26, 2026
760b4b6
fix: align TDT training and decoding with NeMo implementation
lmaksym Feb 27, 2026
b33002f
revert: restore lasr generated files to original state
lmaksym Feb 27, 2026
48b39dd
warn: torchaudio rnnt_loss does not train duration head
lmaksym Feb 27, 2026
e9f23ab
Relax timestamp test, and test nits.
ebezzam Mar 2, 2026
e2b97aa
feat: TDT training
lmaksym Mar 3, 2026
6b9fc73
chore: for cuda detection and run without patching
lmaksym Mar 3, 2026
6c879bc
Equivalent timestamp processing as Nemo, and various nits/cleanup.
ebezzam Mar 3, 2026
149e17f
Merge branch 'parakeet-tdt' of github.com:lmaksym/transformers into p…
ebezzam Mar 3, 2026
36bfa63
Simplify durations config.
ebezzam Mar 3, 2026
2df0ccc
Update training examples.
ebezzam Mar 3, 2026
388c6d3
chore: enable parralelism
lmaksym Mar 3, 2026
08b2b55
chore: performance optimization
lmaksym Mar 4, 2026
0c4e05a
fix: formatting
lmaksym Mar 4, 2026
1ddd804
Doc and testing nits
ebezzam Mar 5, 2026
f512670
Use active mask from current step, and nits.
ebezzam Mar 6, 2026
07d8e35
Better pre-allocate.
ebezzam Mar 6, 2026
fab050a
TDT has separate pad token and blank token.
ebezzam Mar 6, 2026
c438565
Merge branch 'main' into parakeet-tdt
ebezzam Mar 6, 2026
86d980c
Regenerate lasr.
ebezzam Mar 6, 2026
895c4a0
Merge branch 'parakeet-tdt' of github.com:lmaksym/transformers into p…
ebezzam Mar 6, 2026
ab21380
Style checks and nits
ebezzam Mar 7, 2026
d0141d5
Nits, put back ctc loss test
ebezzam Mar 7, 2026
f7529d4
More standard model output.
ebezzam Mar 10, 2026
77b95d7
Style
ebezzam Mar 10, 2026
94eae66
Remove compute_loss flag and allow monkey patching to tdt loss
ebezzam Mar 23, 2026
f7d4067
Update src/transformers/models/parakeet/modular_parakeet.py
ebezzam Mar 23, 2026
f75c17b
Address various comments.
ebezzam Mar 23, 2026
5a49b65
More compatible with Transformers forward/generate approach
ebezzam Mar 24, 2026
881233f
compile option for generation and decoder cache
ebezzam Mar 24, 2026
b41a8ee
Cleaner, better conventions.
ebezzam Mar 24, 2026
897753a
Merge branch 'main' into parakeet-tdt
ebezzam Mar 24, 2026
6c914db
Update with main.
ebezzam Mar 24, 2026
756cee1
doc nits
ebezzam Mar 26, 2026
f30c536
Imitate whisper for encoder outputs as input
ebezzam Mar 26, 2026
fa95fc8
Address tests and nits.
ebezzam Mar 26, 2026
5df7f28
Inherit from GenerateMixIn for get_compiled_call
ebezzam Mar 26, 2026
cd706d4
Comment nit
ebezzam Mar 26, 2026
a47ed8a
forward cleanup
eustlb Apr 15, 2026
13b68ce
generate cleanup + separate generation file
eustlb Apr 15, 2026
72c1ad0
generate: add _supported_generation_modes
eustlb Apr 15, 2026
8e23b3d
automatic init of the loss
eustlb Apr 15, 2026
1cc39fd
modular cleanups
eustlb Apr 15, 2026
531f297
use is_encoder_decoder
eustlb Apr 15, 2026
2c0f23a
timestamp processing fully from tokens + durations
eustlb Apr 15, 2026
cef6639
convertion script update
eustlb Apr 15, 2026
fd3cf9b
test update
eustlb Apr 15, 2026
e63a5bf
make
eustlb Apr 15, 2026
f9d1a4f
Merge branch 'main' into parakeet-tdt
eustlb Apr 15, 2026
43ee7cd
test update
eustlb Apr 15, 2026
c2a0f78
test update
eustlb Apr 15, 2026
1fd7ed7
ensure correct loss computation
eustlb Apr 16, 2026
7cc9d2e
kernel loss
eustlb Apr 16, 2026
e753eab
test loss integration
eustlb Apr 16, 2026
ed3fa4d
push to hub pr
eustlb Apr 16, 2026
ab66b23
integration tests to rely fully on transcripts
eustlb Apr 16, 2026
a5ba0c6
udpate fixtures
eustlb Apr 16, 2026
48279a6
we don't need to monkey patch with numba anymore!
eustlb Apr 16, 2026
1d7680d
fix pipeline usage
eustlb Apr 16, 2026
59ddced
nit
eustlb Apr 16, 2026
31490d1
fix usage
eustlb Apr 16, 2026
d8eb1b6
Pass through tests and examples: improve kernel fallback, update with…
ebezzam Apr 17, 2026
1f1b912
Update checkpoint
ebezzam Apr 17, 2026
9ab08d1
Merge branch 'main' into parakeet-tdt
ebezzam Apr 17, 2026
fd9f8b1
Add TDT to mapping after merge.
ebezzam Apr 17, 2026
136f676
Fix lasr generate test.
ebezzam Apr 20, 2026
833d289
Output attention mask if labels provided for computing loss.
ebezzam Apr 20, 2026
a1c62a1
Apply suggestion from @ArthurZucker
ebezzam May 13, 2026
8683570
Improve ParakeetTDTDecoderCache definition and usage.
ebezzam May 13, 2026
1d4b0f4
Remove tuple parsing.
ebezzam May 13, 2026
a418eca
processor refactor
ebezzam May 13, 2026
5d0c631
Merge branch 'parakeet-tdt' of github.com:lmaksym/transformers into p…
ebezzam May 13, 2026
5c603c1
Update conversion.
ebezzam May 18, 2026
88810b9
Remove kernel to address in separate PR, modular nit.
ebezzam May 19, 2026
00111b3
Merge branch 'main' into parakeet-tdt
ebezzam May 19, 2026
c2e2287
Lasr modular
ebezzam May 19, 2026
a969bc7
Add attention mask docstring.
ebezzam May 19, 2026
bd2ae42
Merge branch 'main' into parakeet-tdt
ebezzam May 19, 2026
f978098
Merge branch 'main' into parakeet-tdt
tarekziade May 19, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions docs/source/en/model_doc/auto.md
Original file line number Diff line number Diff line change
Expand Up @@ -217,6 +217,10 @@ The following auto classes are available for the following audio tasks.

[[autodoc]] AutoModelForCTC

### AutoModelForTDT

[[autodoc]] AutoModelForTDT

### AutoModelForSpeechSeq2Seq

[[autodoc]] AutoModelForSpeechSeq2Seq
Expand Down
58 changes: 53 additions & 5 deletions docs/source/en/model_doc/parakeet.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,15 +34,20 @@ Parakeet models, [introduced by NVIDIA NeMo](https://developer.nvidia.com/blog/p
- 1D convolution projection from encoder hidden size to vocabulary size (for optimal NeMo compatibility).
- CTC loss computation for training.
- Greedy CTC decoding for inference.
- [**ParakeetForTDT**](#parakeetfortdt): a Fast Conformer Encoder + a TDT (Token Duration Transducer) decoder
- **TDT Decoder**: Jointly predicts tokens and their durations, enabling efficient decoding:
- LSTM prediction network maintains language context across token predictions.
- Joint network combines encoder and decoder outputs.
- Duration head predicts how many frames to skip, enabling fast inference.

The original implementation can be found in [NVIDIA NeMo](https://github.com/NVIDIA/NeMo).
Model checkpoints are to be found under [the NVIDIA organization](https://huggingface.co/nvidia/models?search=parakeet).

This model was contributed by [Nithin Rao Koluguri](https://huggingface.co/nithinraok), [Eustache Le Bihan](https://huggingface.co/eustlb) and [Eric Bezzam](https://huggingface.co/bezzam).
This model was contributed by [Nithin Rao Koluguri](https://huggingface.co/nithinraok), [Eustache Le Bihan](https://huggingface.co/eustlb), [Eric Bezzam](https://huggingface.co/bezzam), [Maksym Lypivskyi](https://huggingface.co/MaksL), and [Hainan Xu](https://huggingface.co/hainanx).

## Usage

### Basic usage
### `ParakeetForCTC` usage

<hfoptions id="usage">
<hfoption id="Pipeline">
Expand All @@ -63,10 +68,8 @@ from transformers import AutoModelForCTC, AutoProcessor
from datasets import load_dataset, Audio
import torch

device = "cuda" if torch.cuda.is_available() else "cpu"

processor = AutoProcessor.from_pretrained("nvidia/parakeet-ctc-1.1b")
model = AutoModelForCTC.from_pretrained("nvidia/parakeet-ctc-1.1b", dtype="auto", device_map=device)
model = AutoModelForCTC.from_pretrained("nvidia/parakeet-ctc-1.1b", dtype="auto", device_map="auto")

ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
ds = ds.cast_column("audio", Audio(sampling_rate=processor.feature_extractor.sampling_rate))
Expand All @@ -81,6 +84,43 @@ print(processor.batch_decode(outputs))
</hfoption>
</hfoptions>

### `ParakeetForTDT` usage

<hfoptions id="tdt-usage">
<hfoption id="Pipeline">

```py
from transformers import pipeline

pipe = pipeline("automatic-speech-recognition", model="nvidia/parakeet-tdt-0.6b-v3")
Comment thread
ebezzam marked this conversation as resolved.
Resolved
out = pipe("https://huggingface.co/datasets/hf-internal-testing/dummy-audio-samples/resolve/main/bcn_weather.mp3")
print(out)
```

</hfoption>
<hfoption id="AutoModel">

```py
from transformers import AutoModelForTDT, AutoProcessor
from datasets import load_dataset, Audio
import torch

processor = AutoProcessor.from_pretrained("nvidia/parakeet-tdt-0.6b-v3")
model = AutoModelForTDT.from_pretrained("nvidia/parakeet-tdt-0.6b-v3", dtype="auto", device_map="auto")

ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
ds = ds.cast_column("audio", Audio(sampling_rate=processor.feature_extractor.sampling_rate))
speech_samples = [el['array'] for el in ds["audio"][:5]]

inputs = processor(speech_samples, sampling_rate=processor.feature_extractor.sampling_rate)
inputs.to(model.device, dtype=model.dtype)
output = model.generate(**inputs, return_dict_in_generate=True)
print(processor.batch_decode(output.sequences, skip_special_tokens=True))
```

</hfoption>
</hfoptions>

### Making The Model Go Brrr

Parakeet supports full-graph compilation with CUDA graphs! This optimization is most effective when you know the maximum audio length you want to transcribe. The key idea is using static input shapes to avoid recompilation. For example, if you know your audio will be under 30 seconds, you can use the processor to pad all inputs to 30 seconds, preparing consistent input features and attention masks. See the example below!
Expand Down Expand Up @@ -212,10 +252,18 @@ outputs.loss.backward()

[[autodoc]] ParakeetCTCConfig

## ParakeetTDTConfig

[[autodoc]] ParakeetTDTConfig

## ParakeetEncoder

[[autodoc]] ParakeetEncoder

## ParakeetForCTC

[[autodoc]] ParakeetForCTC

## ParakeetForTDT

[[autodoc]] ParakeetForTDT
6 changes: 4 additions & 2 deletions src/transformers/convert_slow_tokenizer.py

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

curious about why you needed?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Otherwise I was getting this error during checkpoint conversion (on tokenizers==0.22.2)

""" Conversion command
python src/transformers/models/parakeet/convert_nemo_to_hf.py \
   --hf_repo_id nvidia/parakeet-tdt-0.6b-v3 \
    --model_type tdt \
    --output_dir /raid/eric/nemo/parakeet-tdt-hf  \
    --push_to_repo_id bezzam/parakeet-tdt-0.6b-v3-hf
"""


Traceback (most recent call last):
  File "/home/eric_bezzam/FORKS/parakeet-tdt/src/transformers/models/parakeet/convert_nemo_to_hf.py", line 478, in <module>
    main(
  File "/home/eric_bezzam/FORKS/parakeet-tdt/src/transformers/models/parakeet/convert_nemo_to_hf.py", line 429, in main
    pr_revision = write_processor(
  File "/home/eric_bezzam/FORKS/parakeet-tdt/src/transformers/models/parakeet/convert_nemo_to_hf.py", line 148, in write_processor
    tokenizer_converted = ParakeetConverter(model_files["tokenizer_model_file"]).converted()
  File "/home/eric_bezzam/FORKS/parakeet-tdt/src/transformers/convert_slow_tokenizer.py", line 799, in converted
    tokenizer = self.tokenizer(self.proto)
  File "/home/eric_bezzam/FORKS/parakeet-tdt/src/transformers/convert_slow_tokenizer.py", line 1849, in tokenizer
    _, merges = self.SpmExtractor(self.vocab_file).extract(vocab_scores)
  File "/home/eric_bezzam/FORKS/parakeet-tdt/src/transformers/convert_slow_tokenizer.py", line 175, in extract
    if model_type.__name__ != "BPE":
AttributeError: 'list' object has no attribute '__name__'. Did you mean: '__ne__'?

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, but I don't think the PB is here, maybe more in ParakeetConverter(model_files["tokenizer_model_file"]).converted() ?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I removed the change in the base class SpmConverter but I think a change is still needed in ParakeetConverter?

I tried converting the CTC model (already merged) and it was also failing because of logic in ParakeetConverter but passes fine with the change below (GemmaSentencePieceExtractor does something similar).

Maybe something changed with SentencePieceExtractor.extract since Parakeet CTC was merged? (September 2025)

Original file line number Diff line number Diff line change
Expand Up @@ -686,7 +686,8 @@ def tokenizer(self, proto):
)

elif model_type == 2:
_, merges = self.SpmExtractor(self.original_tokenizer.vocab_file).extract(vocab_scores)
result = self.SpmExtractor(self.original_tokenizer.vocab_file).extract(None)
merges = result["merges"]
bpe_vocab = {word: i for i, (word, score) in enumerate(vocab_scores)}
tokenizer = Tokenizer(
BPE(
Expand Down Expand Up @@ -1771,7 +1772,8 @@ def __init__(self, vocab_file=None, *args):
def tokenizer(self, proto):
vocab_scores = self.vocab(proto)

_, merges = self.SpmExtractor(self.vocab_file).extract(vocab_scores)
result = self.SpmExtractor(self.vocab_file).extract(None)
merges = result["merges"]
bpe_vocab = {word: i for i, (word, score) in enumerate(vocab_scores)}
tokenizer = Tokenizer(
BPE(
Expand Down
6 changes: 6 additions & 0 deletions src/transformers/modeling_utils.py
Original file line number Diff line number Diff line change
Expand Up @@ -2280,6 +2280,12 @@ def _init_weights(self, module):
init.normal_(module.weight, mean=0.0, std=std)
if module.bias is not None:
init.zeros_(module.bias)
elif isinstance(module, nn.LSTM):

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok!

for name, param in module.named_parameters():
if "weight" in name:
init.xavier_uniform_(param)
elif "bias" in name:
init.constant_(param, 0.0)
elif isinstance(module, nn.Embedding):
init.normal_(module.weight, mean=0.0, std=std)
# Here we need the check explicitly, as we slice the weight in the `zeros_` call, so it looses the flag
Expand Down
5 changes: 4 additions & 1 deletion src/transformers/models/auto/configuration_auto.py
Original file line number Diff line number Diff line change
Expand Up @@ -326,6 +326,7 @@
("paligemma", "PaliGemmaConfig"),
("parakeet_ctc", "ParakeetCTCConfig"),
("parakeet_encoder", "ParakeetEncoderConfig"),
("parakeet_tdt", "ParakeetTDTConfig"),
("patchtsmixer", "PatchTSMixerConfig"),
("patchtst", "PatchTSTConfig"),
("pe_audio", "PeAudioConfig"),
Expand Down Expand Up @@ -824,6 +825,7 @@
("parakeet", "Parakeet"),
("parakeet_ctc", "Parakeet"),
("parakeet_encoder", "ParakeetEncoder"),
("parakeet_tdt", "ParakeetTDT"),
("patchtsmixer", "PatchTSMixer"),
("patchtst", "PatchTST"),
("pe_audio", "PeAudio"),
Expand Down Expand Up @@ -1083,8 +1085,9 @@
("pe_audio_video_encoder", "pe_audio_video"),
("video_llama_3_vision", "video_llama_3"),
("parakeet_encoder", "parakeet"),
("lw_detr_vit", "lw_detr"),
("parakeet_ctc", "parakeet"),
("parakeet_tdt", "parakeet"),
("lw_detr_vit", "lw_detr"),
("lasr_encoder", "lasr"),
("lasr_ctc", "lasr"),
("wav2vec2-bert", "wav2vec2_bert"),
Expand Down
1 change: 1 addition & 0 deletions src/transformers/models/auto/feature_extraction_auto.py
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,7 @@
("musicgen_melody", "MusicgenMelodyFeatureExtractor"),
("parakeet_ctc", "ParakeetFeatureExtractor"),
("parakeet_encoder", "ParakeetFeatureExtractor"),
("parakeet_tdt", "ParakeetFeatureExtractor"),
("pe_audio", "PeAudioFeatureExtractor"),
("pe_audio_video", "PeAudioFeatureExtractor"),
("phi4_multimodal", "Phi4MultimodalFeatureExtractor"),
Expand Down
19 changes: 19 additions & 0 deletions src/transformers/models/auto/modeling_auto.py
Original file line number Diff line number Diff line change
Expand Up @@ -320,6 +320,7 @@ class _BaseModelWithGenerate(PreTrainedModel, GenerationMixin):
("paligemma", "PaliGemmaModel"),
("parakeet_ctc", "ParakeetForCTC"),
("parakeet_encoder", "ParakeetEncoder"),
("parakeet_tdt", "ParakeetForTDT"),
("patchtsmixer", "PatchTSMixerModel"),
("patchtst", "PatchTSTModel"),
("pe_audio", "PeAudioModel"),
Expand Down Expand Up @@ -1551,6 +1552,14 @@ class _BaseModelWithGenerate(PreTrainedModel, GenerationMixin):
]
)

MODEL_FOR_TDT_MAPPING_NAMES = OrderedDict(
[
# Model for Token-and-Duration Transducer (TDT) mapping.
("parakeet_tdt", "ParakeetForTDT"),
]
)


MODEL_FOR_AUDIO_FRAME_CLASSIFICATION_MAPPING_NAMES = OrderedDict(
[
# Model for Audio Classification mapping
Expand Down Expand Up @@ -1816,6 +1825,7 @@ class _BaseModelWithGenerate(PreTrainedModel, GenerationMixin):
CONFIG_MAPPING_NAMES, MODEL_FOR_AUDIO_CLASSIFICATION_MAPPING_NAMES
)
MODEL_FOR_CTC_MAPPING = _LazyAutoMapping(CONFIG_MAPPING_NAMES, MODEL_FOR_CTC_MAPPING_NAMES)
MODEL_FOR_TDT_MAPPING = _LazyAutoMapping(CONFIG_MAPPING_NAMES, MODEL_FOR_TDT_MAPPING_NAMES)
MODEL_FOR_SPEECH_SEQ_2_SEQ_MAPPING = _LazyAutoMapping(CONFIG_MAPPING_NAMES, MODEL_FOR_SPEECH_SEQ_2_SEQ_MAPPING_NAMES)
MODEL_FOR_AUDIO_FRAME_CLASSIFICATION_MAPPING = _LazyAutoMapping(
CONFIG_MAPPING_NAMES, MODEL_FOR_AUDIO_FRAME_CLASSIFICATION_MAPPING_NAMES
Expand Down Expand Up @@ -2125,6 +2135,13 @@ class AutoModelForCTC(_BaseAutoModelClass):
AutoModelForCTC = auto_class_update(AutoModelForCTC, head_doc="connectionist temporal classification")


class AutoModelForTDT(_BaseAutoModelClass):
_model_mapping = MODEL_FOR_TDT_MAPPING


AutoModelForTDT = auto_class_update(AutoModelForTDT, head_doc="token-and-duration transducer")


class AutoModelForSpeechSeq2Seq(_BaseAutoModelClass):
_model_mapping = MODEL_FOR_SPEECH_SEQ_2_SEQ_MAPPING

Expand Down Expand Up @@ -2187,6 +2204,7 @@ class AutoModelForAudioTokenization(_BaseAutoModelClass):
"MODEL_FOR_CAUSAL_IMAGE_MODELING_MAPPING",
"MODEL_FOR_CAUSAL_LM_MAPPING",
"MODEL_FOR_CTC_MAPPING",
"MODEL_FOR_TDT_MAPPING",
"MODEL_FOR_DOCUMENT_QUESTION_ANSWERING_MAPPING",
"MODEL_FOR_DEPTH_ESTIMATION_MAPPING",
"MODEL_FOR_IMAGE_CLASSIFICATION_MAPPING",
Expand Down Expand Up @@ -2233,6 +2251,7 @@ class AutoModelForAudioTokenization(_BaseAutoModelClass):
"AutoModelForAudioXVector",
"AutoModelForCausalLM",
"AutoModelForCTC",
"AutoModelForTDT",
"AutoModelForDepthEstimation",
"AutoModelForImageClassification",
"AutoModelForImageSegmentation",
Expand Down
2 changes: 2 additions & 0 deletions src/transformers/models/auto/processing_auto.py
Original file line number Diff line number Diff line change
Expand Up @@ -124,6 +124,8 @@
("owlvit", "OwlViTProcessor"),
("paddleocr_vl", "PaddleOCRVLProcessor"),
("paligemma", "PaliGemmaProcessor"),
("parakeet_ctc", "ParakeetProcessor"),
("parakeet_tdt", "ParakeetProcessor"),
("perception_lm", "PerceptionLMProcessor"),
("phi4_multimodal", "Phi4MultimodalProcessor"),
("pix2struct", "Pix2StructProcessor"),
Expand Down
2 changes: 2 additions & 0 deletions src/transformers/models/auto/tokenization_auto.py
Original file line number Diff line number Diff line change
Expand Up @@ -236,6 +236,8 @@
("ovis2", "Qwen2Tokenizer" if is_tokenizers_available() else None),
("owlv2", "CLIPTokenizer" if is_tokenizers_available() else None),
("owlvit", "CLIPTokenizer" if is_tokenizers_available() else None),
("parakeet_ctc", "ParakeetTokenizer" if is_tokenizers_available() else None),
("parakeet_tdt", "ParakeetTokenizer" if is_tokenizers_available() else None),
("pegasus", "PegasusTokenizer" if is_tokenizers_available() else None),
("pegasus_x", "PegasusTokenizer" if is_tokenizers_available() else None),
("perceiver", "PerceiverTokenizer"),
Expand Down
15 changes: 2 additions & 13 deletions src/transformers/models/encodec/modeling_encodec.py
Original file line number Diff line number Diff line change
Expand Up @@ -455,23 +455,12 @@ class EncodecPreTrainedModel(PreTrainedAudioTokenizerBase):

@torch.no_grad()
def _init_weights(self, module):
"""Initialize the weights"""
if isinstance(module, nn.GroupNorm):
init.zeros_(module.bias)
init.ones_(module.weight)
Comment thread
ebezzam marked this conversation as resolved.
Resolved
elif isinstance(module, nn.Conv1d):
super()._init_weights(module)
if isinstance(module, nn.Conv1d):
init.kaiming_normal_(module.weight)
if module.bias is not None:
k = math.sqrt(module.groups / (module.in_channels * module.kernel_size[0]))
init.uniform_(module.bias, a=-k, b=k)
elif isinstance(module, nn.ConvTranspose1d):
module.reset_parameters()
Comment thread
ebezzam marked this conversation as resolved.
Resolved
elif isinstance(module, nn.LSTM):
for name, param in module.named_parameters():
if "weight" in name:
init.xavier_uniform_(param)
elif "bias" in name:
init.constant_(param, 0.0)
Comment thread
ebezzam marked this conversation as resolved.
Resolved
elif isinstance(module, EncodecConv1d):
kernel_size = module.conv.kernel_size[0]
stride = torch.tensor(module.conv.stride[0], dtype=torch.int64)
Expand Down
8 changes: 4 additions & 4 deletions src/transformers/models/lasr/modeling_lasr.py
Original file line number Diff line number Diff line change
Expand Up @@ -555,9 +555,9 @@ def forward(


@dataclass
class LasrGenerateOutput(ModelOutput):
class LasrCTCGenerateOutput(ModelOutput):
"""
Outputs of Lasr models.
Outputs of Lasr CTC model generation.

Args:
sequences (`torch.LongTensor` of shape `(batch_size, sequence_length)`):
Expand Down Expand Up @@ -677,7 +677,7 @@ def generate(
attention_mask: torch.Tensor | None = None,
return_dict_in_generate: bool = False,
**kwargs: Unpack[TransformersKwargs],
) -> LasrGenerateOutput | torch.LongTensor:
) -> LasrCTCGenerateOutput | torch.LongTensor:
r"""
Example:

Expand Down Expand Up @@ -715,7 +715,7 @@ def generate(
sequences[~attention_mask] = self.config.pad_token_id

if return_dict_in_generate:
return LasrGenerateOutput(
return LasrCTCGenerateOutput(
sequences=sequences,
logits=outputs.logits,
attentions=outputs.attentions,
Expand Down
3 changes: 2 additions & 1 deletion src/transformers/models/parakeet/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,8 @@
from .configuration_parakeet import *
from .feature_extraction_parakeet import *
from .modeling_parakeet import *
from .tokenization_parakeet_fast import *
from .processing_parakeet import *
from .tokenization_parakeet import *
else:
import sys

Expand Down
Loading