[lashahub]

This PR adds support for Audio-Visual Flamingo (AVF), an open audio-visual large language model for joint understanding and reasoning over audio, images, and videos.

📄 Paper: https://arxiv.org/abs/2607.16107
🌐 Project Page: https://avflamingo.pages.dev/
🤗 Model: https://huggingface.co/nvidia/nemotron-labs-audio-visual-flamingo-hf
📊 Dataset: https://huggingface.co/datasets/nvidia/av-skills
🎮 Demo: https://huggingface.co/spaces/nvidia/nemotron-labs-audio-visual-flamingo

In Transformers, AVF pairs a SigLIP vision tower with an AF-Whisper audio encoder and a Qwen2.5-7B causal language model, with separate projectors for visual and audio features. For joint video-audio inputs, AVF aligns synchronized visual and audio chunks, interleaves them along the time axis, applies Constrained Rotary Time Embeddings (CRTE), and feeds the fused sequence to the language model. It also supports Dynamic-S2 preprocessing for high-resolution images and sampled video frames.

This PR introduces:

• AudioVisualFlamingoConfig
• AudioVisualFlamingoForConditionalGeneration
• AudioVisualFlamingoProcessor
• Joint video-audio handling from a single container via load_audio_in_video=True
• Dynamic-S2 visual preprocessing
• Temporal audio-visual interleaving with CRTE
• Modeling, processing, tests, and documentation

Example usage once the checkpoint is public:

from transformers import AudioVisualFlamingoForConditionalGeneration, AutoProcessor

model_id = "nvidia/audio-visual-flamingo-hf"

processor = AutoProcessor.from_pretrained(
    model_id,
    padding_side="left",
    use_fast=False,
    load_audio_in_video=True,
    num_video_frames=128,
    audio_chunk_length="max_3600",
)
model = AudioVisualFlamingoForConditionalGeneration.from_pretrained(
    model_id,
    device_map="auto",
    load_audio_in_video=True,
).eval()

conversation = [
    {
        "role": "user",
        "content": [
            {"type": "video", "video": "video.mp4"},
            {
                "type": "text",
                "text": "Describe both the visual scene and the spoken or environmental audio content.",
            },
        ],
    }
]

inputs = processor.apply_chat_template(
    conversation,
    tokenize=True,
    add_generation_prompt=True,
    return_dict=True,
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=512, do_sample=False)
print(processor.batch_decode(outputs[:, inputs.input_ids.shape[1] :], skip_special_tokens=True)[0])

[github-actions]

[For maintainers] Suggested jobs to run (before merge)

run-slow: audiovisualflamingo, auto

[lashahub]

Following up on this PR, Audio-Visual Flamingo has now been accepted to NeurIPS! 🎉 Would be great to get this reviewed and merged so people can use it directly through Transformers!

[eustlb]

@lashahub Thanks for your patience on this! I'll get to it in the coming days 👌

[ArthurZucker]

@eustlb could you take a look when you get a chance? This is waiting on a maintainer: eustlb review pending (promised 2026-09-25); stray root files c.sh/main.py/nvidia.mp4 should be removed.