Skip to content
Merged
Show file tree
Hide file tree
Changes from 107 commits
Commits
Show all changes
116 commits
Select commit Hold shift + click to select a range
caf33f1
Music flamingo
lashahub Dec 24, 2025
b69a9d1
Fix pos embeddings
lashahub Dec 26, 2025
b4acc17
Merge branch 'huggingface:main' into main
lashahub Jan 6, 2026
cf1e9bc
Method arg docstrings
lashahub Jan 6, 2026
44e801b
Add tests & docs
lashahub Jan 8, 2026
f0956e3
Merge branch 'huggingface:main' into main
lashahub Jan 8, 2026
c973445
Fix AF3 dtype bug
lashahub Jan 10, 2026
e3a17fb
Fix the MF performance issue
lashahub Jan 10, 2026
627dee8
Fix pos embeddings
lashahub Jan 10, 2026
e9df30d
Merge branch 'main' of https://github.com/lashahub/transformers
lashahub Jan 10, 2026
4c48132
Fix embeddings & format
lashahub Jan 12, 2026
d67c114
Remove external deps
lashahub Jan 23, 2026
aedd341
Update processor token names
lashahub Jan 23, 2026
87d55a9
Cleanup
lashahub Feb 10, 2026
be22746
Simplify RotaryEmbedding to lang-only
lashahub Feb 13, 2026
e5b4677
Reuse AF3 config classes
lashahub Feb 14, 2026
d7e0bcb
Trim+rename rotary embedding
lashahub Feb 14, 2026
74af4fa
Call parent _init_weights first and drop rotary einsum
lashahub Feb 14, 2026
d622368
Precompute rotary cache at init
lashahub Feb 14, 2026
cab3937
Use modular processor pattern for MusicFlamingo
lashahub Feb 15, 2026
9dd94a0
Remove audio-only inference example
lashahub Feb 15, 2026
767a1d5
Refactor Audio Feature Casting Path
lashahub Feb 15, 2026
9119660
Clarify private source repo
lashahub Feb 21, 2026
fc8ab3a
Clean up modular
lashahub Feb 22, 2026
5159745
Move config to modular
lashahub Feb 22, 2026
4aae4ff
Formatting
lashahub Feb 23, 2026
ffef7db
Remove dummy
lashahub Feb 23, 2026
90148ed
Derive musicflamingo timing and rotary config
lashahub Feb 23, 2026
87cb03f
Llama style rotary embeddings
lashahub Feb 23, 2026
5abac95
Added reproducer comments
lashahub Feb 23, 2026
f4a1b68
Expose _init_weights for modular.
ebezzam Feb 25, 2026
3b9290f
Satisfy repo checks
ebezzam Feb 25, 2026
82ab1ef
Align MusicFlamingo rotary with Llama style
lashahub Feb 25, 2026
e993dea
Move MusicFlamingo _init_weights to encoder
lashahub Feb 26, 2026
2d88557
Keep old behavior
lashahub Feb 26, 2026
27024f1
Move MusicFlamingo rotary settings into encoder rope_parameters
lashahub Feb 26, 2026
10797b9
Use AutoConfig in AF3/MF
lashahub Feb 26, 2026
f49014f
Align MusicFlamingo RoTE with Llama RoPE conventions
lashahub Feb 26, 2026
f465d3d
Update outdated fixtures
lashahub Feb 26, 2026
4224af5
init_weights without changing others
ebezzam Feb 27, 2026
2cbf219
FIx import
ebezzam Feb 27, 2026
a25c8a6
Remove backward compat
lashahub Feb 27, 2026
3ed73a8
Merge branch 'main' into modular-mf
lashahub Feb 27, 2026
f2e7d3f
Regenerate modeling for MF
lashahub Feb 27, 2026
445cd24
Fix AF3 batch inference bug
lashahub Mar 6, 2026
fc25c48
Simplify config and nit.
ebezzam Mar 10, 2026
664e7b1
Conform more to transformers convention, e.g. removing unused code pa…
ebezzam Mar 10, 2026
bbca79a
Add another possible AF3 prefix.
ebezzam Mar 10, 2026
69c9bee
Merge branch 'main' into modular-mf
ebezzam Mar 10, 2026
17846d9
Use auto_docstring and update docstrings.
ebezzam Mar 10, 2026
24fbaa7
Nits
ebezzam Mar 11, 2026
e25d377
Nit for review
ebezzam Mar 11, 2026
c99fc0a
Shift RoTE to main model so that encoder can be directly used from AF3.
ebezzam Mar 11, 2026
9bb0c53
Refactoring nit.
ebezzam Mar 11, 2026
e9528a6
Fix init
lashahub Mar 17, 2026
13af64d
Fix some failing tests
lashahub Mar 17, 2026
a34e981
Fix AF3 & MF and add batching tests
lashahub Mar 17, 2026
702eb27
Fix audio embedding masking (bad post length)
ebezzam Mar 17, 2026
a32520c
Nits and remove since same as GLM was bug in post length computation
ebezzam Mar 17, 2026
7750bb5
Simplify MF as AF3, and style checks.
ebezzam Mar 17, 2026
74417f6
Merge branch 'main' into modular-mf
ebezzam Mar 17, 2026
54bfecc
New config after merge and modular update.
ebezzam Mar 17, 2026
2f2b552
Merge branch 'main' into modular-mf
ebezzam Mar 18, 2026
612840f
Address music flamingo tests, and some cleanup.
ebezzam Mar 18, 2026
2c3e583
style check
ebezzam Mar 18, 2026
a20e163
Regenerate config.
ebezzam Mar 18, 2026
fdb849a
Update fixtures.
ebezzam Mar 18, 2026
057ad88
Nits
ebezzam Mar 18, 2026
0442a67
Merge branch 'modular-mf' of github.com:lashahub/transformers into mo…
ebezzam Mar 18, 2026
7f04d2d
Nit
ebezzam Mar 18, 2026
63a135c
Add AudioFlamingoNext alias model
lashahub Mar 18, 2026
7125e05
Merge branch 'modular-mf' into add_AudioFlamingoNext
lashahub Mar 18, 2026
7c94fb5
Regenerate AF-Next files
lashahub Mar 18, 2026
66ce201
Merge branch 'huggingface:main' into modular-mf
lashahub Mar 20, 2026
7e06d27
Improve RoTE config
lashahub Mar 22, 2026
d423510
Refine MusicFlamingo rotary time handling
lashahub Mar 23, 2026
e79475b
Simplification, and update AF3 processor for better modular
ebezzam Mar 23, 2026
6a1add4
Fix torch export
ebezzam Mar 23, 2026
e05b767
Simplify modular, including upstreaming input_ids input to get_audio_…
ebezzam Mar 23, 2026
edcffef
Remove upstreaming of input_ids to get_audio_features, and remove aud…
ebezzam Mar 24, 2026
e986996
Switch to MoonshineRotaryEmbedding, and cleanup.
ebezzam Mar 24, 2026
ad76e28
Remove hardcoded MusicFlamingo partial_rotary_factor
lashahub Mar 24, 2026
8e0dc3b
Update fixtures
lashahub Mar 24, 2026
a46aa1c
Compile re.sub
ebezzam Mar 30, 2026
836da94
Merge branch 'modular-mf' of github.com:lashahub/transformers into mo…
ebezzam Mar 30, 2026
1089477
Update src/transformers/models/musicflamingo/modular_musicflamingo.py
ebezzam Mar 30, 2026
2ea5574
Update src/transformers/models/musicflamingo/modular_musicflamingo.py
ebezzam Mar 30, 2026
999737e
Style
ebezzam Mar 30, 2026
1533bec
Merge branch 'main' into modular-mf
ebezzam Mar 30, 2026
6857af9
Update fixtures.
ebezzam Mar 30, 2026
cd693fc
Merge branch 'main' into modular-mf
ebezzam Mar 30, 2026
35fc31f
Conditional torch import for processor.
ebezzam Mar 30, 2026
a62e7d9
Merge branch 'modular-mf' of github.com:lashahub/transformers into mo…
ebezzam Mar 30, 2026
d0d18f2
Merge branch 'main' into modular-mf
ebezzam Mar 30, 2026
47fa938
Merge branch 'main' into modular-mf
ebezzam Mar 30, 2026
dd2cc52
Merge branch 'modular-mf' into add_AudioFlamingoNext
lashahub Mar 31, 2026
c6f4213
Merge MF & convert modular
lashahub Mar 31, 2026
fd63719
30 mins in AF-Next
lashahub Apr 4, 2026
6d5948e
Merge branch 'main' into add_AudioFlamingoNext
lashahub Apr 4, 2026
a44c774
Regen
lashahub Apr 4, 2026
3c7f582
Fix AF-Next converter
lashahub Apr 4, 2026
8f33aeb
AF-Next readme
lashahub Apr 4, 2026
d323d1f
Add tests
lashahub Apr 5, 2026
9735a5f
Update AF-Next docs
lashahub Apr 5, 2026
548e646
Fix MusicFlamingo and AFNext context metadata and rope defaults
lashahub Apr 7, 2026
e771363
Merge branch 'huggingface:main' into add_AudioFlamingoNext
lashahub Apr 13, 2026
8c17076
Update fixtures
lashahub Apr 13, 2026
d79fb29
Address reviews
lashahub Apr 25, 2026
205f8c8
Add reproducer
lashahub Apr 25, 2026
7e63ddd
Update AF-Next converter
lashahub May 12, 2026
e38b6ac
Remove AF-Next
lashahub May 12, 2026
8771a9f
Keep AF-Next converter
lashahub May 12, 2026
7091a2a
Merge branch 'main' into add_AudioFlamingoNext
lashahub May 13, 2026
da61fd3
Merge branch 'main' into add_AudioFlamingoNext
ebezzam May 13, 2026
88e9644
Update src/transformers/models/musicflamingo/modular_musicflamingo.py
eustlb May 14, 2026
3f1219d
make
eustlb May 14, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions docs/source/en/_toctree.yml
Original file line number Diff line number Diff line change
Expand Up @@ -1145,6 +1145,8 @@
title: Aria
- local: model_doc/audioflamingo3
title: AudioFlamingo3
- local: model_doc/audioflamingonext
title: AudioFlamingoNext
- local: model_doc/aya_vision
title: AyaVision
- local: model_doc/blip
Expand Down
310 changes: 310 additions & 0 deletions docs/source/en/model_doc/audioflamingonext.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,310 @@
<!--Copyright 2026 The HuggingFace Team. All rights reserved.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
specific language governing permissions and limitations under the License.

⚠️ Note that this file is in Markdown but contain specific syntax for our doc-builder (similar to MDX) that may not be
rendered properly in your Markdown viewer.

-->

*This model was released on 2026-04-13 and added to Hugging Face Transformers on 2026-04-13.*

# Audio Flamingo Next

<div class="flex flex-wrap space-x-1">
<img alt="PyTorch" src="https://img.shields.io/badge/PyTorch-DE3412?style=flat&logo=pytorch&logoColor=white">
<img alt="FlashAttention" src="https://img.shields.io/badge/%E2%9A%A1%EF%B8%8E%20FlashAttention-eae0c8?style=flat">
<img alt="SDPA" src="https://img.shields.io/badge/SDPA-DE3412?style=flat&logo=pytorch&logoColor=white">
</div>

## Overview

Audio Flamingo Next (AF-Next) is a fully open large audio-language model for understanding and reasoning over speech, environmental sounds, and music. In Transformers, it is implemented as a replace-in-place audio-text model: the processor expands a dedicated audio placeholder into the exact number of post-pool audio frames, and the model replaces those token slots with projected audio embeddings during the forward pass.

AF-Next builds on [Audio Flamingo 3](./audioflamingo3.md) with a stronger audio-language training recipe, Rotary Time Embeddings (RoTE) for temporally grounded modeling, and support for long and complex audio inputs up to **30 minutes (1800 seconds)**.

The model checkpoint is available at: [nvidia/audio-flamingo-next-hf](https://huggingface.co/nvidia/audio-flamingo-next-hf)

Highlights:

- Generalist audio understanding across speech, sound, and music.
- **RoTE-based temporal grounding** for long-form audio reasoning.
- **Long-audio support up to 30 minutes** through 30-second windowing and post-pool alignment.
- Special sound boundary tokens (`<|sound_bos|>` and `<|sound_eos|>`) for audio sequence boundaries.
- Deterministic replace-in-place fusion that preserves the text sequence length.

This model was contributed by [Lasha Koroshinadze](https://huggingface.co/lashahub) and [Eric Bezzam](https://huggingface.co/bezzam).

### Paper

Audio Flamingo Next: Next-Generation Open Audio-Language Models for Speech, Sound, and Music
Sreyan Ghosh, Arushi Goel, Kaousheik Jayakumar, Lasha Koroshinadze, Nishit Anand, Zhifeng Kong, Siddharth Gururani, Sang-gil Lee, Jaehyeon Kim, Aya Aljafari, Chao-Han Huck Yang, Sungwon Kim, Ramani Duraiswami, Dinesh Manocha, Mohammad Shoeybi, Bryan Catanzaro, Ming-Yu Liu, Wei Ping
NVIDIA and University of Maryland

The paper introduces AF-Next as an open frontier audio-language model trained with a curriculum spanning pre-training, mid-training, post-training, and time-grounded reasoning. Compared to Audio Flamingo 3, it expands long-audio support to 30 minutes, strengthens performance across speech, sound, and music tasks, and introduces Temporal Audio Chain-of-Thought for timestamp-grounded reasoning over long recordings.

## Usage

### Audio Instruct Mode

The model supports audio-text instructions, including multi-turn and batched interactions.

➡️ audio + text instruction

```python
from transformers import AudioFlamingoNextForConditionalGeneration, AutoProcessor

model_id = "nvidia/audio-flamingo-next-hf"
processor = AutoProcessor.from_pretrained(model_id)
model = AudioFlamingoNextForConditionalGeneration.from_pretrained(model_id, device_map="auto")

conversation = [
{
"role": "user",
"content": [
{
"type": "text",
"text": "Transcribe the input speech.",
},
{
"type": "audio",
"path": "https://huggingface.co/datasets/nvidia/AudioSkills/resolve/main/assets/WhDJDIviAOg_120_10.mp3",
},
],
}
]

inputs = processor.apply_chat_template(
conversation,
tokenize=True,
add_generation_prompt=True,
return_dict=True,
).to(model.device)
inputs["input_features"] = inputs["input_features"].to(model.dtype)

outputs = model.generate(**inputs, max_new_tokens=500)

decoded_outputs = processor.batch_decode(outputs[:, inputs.input_ids.shape[1]:], skip_special_tokens=True)
print(decoded_outputs)
```

➡️ multi-turn:

```python
from transformers import AudioFlamingoNextForConditionalGeneration, AutoProcessor

model_id = "nvidia/audio-flamingo-next-hf"
processor = AutoProcessor.from_pretrained(model_id)
model = AudioFlamingoNextForConditionalGeneration.from_pretrained(model_id, device_map="auto")

conversation = [
{
"role": "user",
"content": [
{
"type": "text",
"text": "Transcribe the input speech.",
},
{
"type": "audio",
"path": "https://huggingface.co/datasets/nvidia/AudioSkills/resolve/main/assets/WhDJDIviAOg_120_10.mp3",
},
],
},
{
"role": "assistant",
"content": [{"type": "text", "text": "Summer follows spring, the days grow longer, and the nights are warm."}],
},
{
"role": "user",
"content": [
{"type": "text", "text": "Summarize the content in one sentence."},
],
},
]

inputs = processor.apply_chat_template(
conversation,
tokenize=True,
add_generation_prompt=True,
return_dict=True,
).to(model.device)
inputs["input_features"] = inputs["input_features"].to(model.dtype)

outputs = model.generate(**inputs, max_new_tokens=500)

decoded_outputs = processor.batch_decode(outputs[:, inputs.input_ids.shape[1]:], skip_special_tokens=True)
print(decoded_outputs)
```

➡️ batched inference!

```python
from transformers import AudioFlamingoNextForConditionalGeneration, AutoProcessor

model_id = "nvidia/audio-flamingo-next-hf"
processor = AutoProcessor.from_pretrained(model_id)
model = AudioFlamingoNextForConditionalGeneration.from_pretrained(model_id, device_map="auto")

conversations = [
[
{
"role": "user",
"content": [
{
"type": "text",
"text": "Transcribe the input speech.",
},
{
"type": "audio",
"path": "https://huggingface.co/datasets/nvidia/AudioSkills/resolve/main/assets/WhDJDIviAOg_120_10.mp3",
},
],
}
],
[
{
"role": "user",
"content": [
{
"type": "text",
"text": "Compose a detailed caption integrating all audio elements, describing sound effects, speech, and music.",
},
{
"type": "audio",
"path": "https://huggingface.co/datasets/nvidia/AudioSkills/resolve/main/assets/dogs_barking_in_sync_with_the_music.wav",
},
],
}
],
]

inputs = processor.apply_chat_template(
conversations,
tokenize=True,
add_generation_prompt=True,
return_dict=True,
).to(model.device)
inputs["input_features"] = inputs["input_features"].to(model.dtype)

outputs = model.generate(**inputs, max_new_tokens=500)

decoded_outputs = processor.batch_decode(outputs[:, inputs.input_ids.shape[1]:], skip_special_tokens=True)
print(decoded_outputs)
```

➡️ Training:

```python
from transformers import AudioFlamingoNextForConditionalGeneration, AutoProcessor

model_id = "nvidia/audio-flamingo-next-hf"
processor = AutoProcessor.from_pretrained(model_id)
model = AudioFlamingoNextForConditionalGeneration.from_pretrained(model_id, device_map="auto")
model.train()

conversation = [
[
{
"role": "user",
"content": [
{"type": "text", "text": "Transcribe the input speech."},
{"type": "audio", "path": "https://huggingface.co/datasets/nvidia/AudioSkills/resolve/main/assets/WhDJDIviAOg_120_10.mp3"},
],
},
{
"role": "assistant",
"content": [{"type": "text", "text": "Summer follows spring, the days grow longer, and the nights are warm."}],
}
],
[
{
"role": "user",
"content": [
{
"type": "text",
"text": "Compose a detailed caption integrating all audio elements, describing sound effects, speech, and music.",
},
{
"type": "audio",
"path": "https://huggingface.co/datasets/nvidia/AudioSkills/resolve/main/assets/dogs_barking_in_sync_with_the_music.wav",
},
],
},
{
"role": "assistant",
"content": [{"type": "text", "text": "A low-fidelity recording begins with a brief, sharp electronic buzz, followed by high-energy electronic dance music with a driving beat, pulsing synth bass, and a bright repetitive melody. Over the music, a small dog's sharp barks cut through repeatedly, creating a lively and chaotic soundscape."}],
}

]
]

inputs = processor.apply_chat_template(
conversation,
tokenize=True,
add_generation_prompt=True,
return_dict=True,
output_labels=True,
).to(model.device)
inputs["input_features"] = inputs["input_features"].to(model.dtype)

loss = model(**inputs).loss
loss.backward()
```

## How the model works

### Architecture

* **Audio Encoder**
AF-Next uses the AF-Whisper encoder. Audio is resampled to 16 kHz mono, converted to a 128-bin log-mel spectrogram with a 25 ms window and 10 ms hop, encoded in non-overlapping 30-second windows, then pooled with stride 2.

* **Rotary Time Embeddings (RoTE)**
AF-Next uses axial rotary embeddings over the window index and the encoder time index, and both axes are modulated with absolute timestamps in seconds. This is the same RoTE formulation used by Music Flamingo.

* **AudioFlamingoNextMultiModalProjector**
A 2-layer MLP that maps AF-Whisper features to the language model hidden size.

* **AudioFlamingoNextForConditionalGeneration**
A decoder-only audio-language model built on a Qwen2.5-7B text backbone. Audio placeholder token slots are replaced in place by projected audio frame embeddings, with `<|sound_bos|>` and `<|sound_eos|>` marking audio boundaries. The paper introduces multiple AF-Next variants; the current Transformers checkpoint exposes the public conditional-generation model in this standard interface.

### Processor-level alignment

1. Each raw waveform is resampled and split into fixed 30-second windows using the feature extractor configuration.
2. For each window, the processor computes the number of post-pool frames `post_pool_len` that the encoder will output (matching the conv/pool schedule).
3. The processor expands the audio placeholder token by the total number of post-pool frames across all windows.
4. The model later replaces those token positions with the corresponding projected audio embeddings.

## Long audio and windowing

**Important: Maximum audio length is 30 minutes.** Audio longer than this will be truncated.

* The default setup processes 30-second windows at 16 kHz mono.
* **The processor enforces a hard limit of 60 windows per sample, resulting in a maximum of 30 minutes of audio (60 windows × 30 seconds).**
* RoTE provides temporal position information for audio sequences up to 30 minutes (1800 seconds).
* For each window:

* `mel_len` is the padded mel length.
* A conv stack reduces time as `conv_output_len = (mel_len - 1) // 2 + 1`.
* Post-pool frames per window: `post_pool_len = (conv_output_len - 2) // 2 + 1`.
* An audio placeholder token is expanded to the sum of `post_pool_len` across all windows.

## AudioFlamingoNextConfig

[[autodoc]] AudioFlamingoNextConfig

## AudioFlamingoNextProcessor

[[autodoc]] AudioFlamingoNextProcessor

## AudioFlamingoNextForConditionalGeneration

[[autodoc]] AudioFlamingoNextForConditionalGeneration
- forward
1 change: 1 addition & 0 deletions src/transformers/models/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@
from .aria import *
from .audio_spectrogram_transformer import *
from .audioflamingo3 import *
from .audioflamingonext import *
from .auto import *
from .autoformer import *
from .aya_vision import *
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -100,6 +100,7 @@ def write_processor(src_root: Path, dst_root: Path):
tokenizer=AutoTokenizer.from_pretrained(str(llm_dir), chat_template=tokenizer_chat_template, use_fast=True),
chat_template=processor_chat_template,
)
processor.tokenizer.model_max_length = 32768

@ebezzam ebezzam May 13, 2026 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

processor.save_pretrained(str(dst_root))

logger.info("processor (tokenizer + preprocessor)")
Expand Down Expand Up @@ -167,7 +168,7 @@ def merge_and_shard_weights(src_root: Path, dst_root: Path, processor: AudioFlam
vocab_size=len(tok),
hidden_size=3584,
intermediate_size=18944,
model_max_length=8192,
model_max_length=32768,
num_attention_heads=28,
num_hidden_layers=28,
num_key_value_heads=4,
Expand Down
30 changes: 30 additions & 0 deletions src/transformers/models/audioflamingonext/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
# Copyright 2026 NVIDIA CORPORATION and the HuggingFace Inc. team. All rights
# reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

from typing import TYPE_CHECKING

from ...utils import _LazyModule
from ...utils.import_utils import define_import_structure


if TYPE_CHECKING:
from .configuration_audioflamingonext import *
from .modeling_audioflamingonext import *
from .processing_audioflamingonext import *
else:
import sys

_file = globals()["__file__"]
sys.modules[__name__] = _LazyModule(__name__, _file, define_import_structure(_file), module_spec=__spec__)
Loading
Loading