Skip to content
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
40 commits
Select commit Hold shift + click to select a range
25f8152
initial commit
zucchini-nlp Sep 15, 2025
8f71e71
fix tests
zucchini-nlp Sep 15, 2025
beb8517
fix copies, tests and rename pipe
zucchini-nlp Sep 15, 2025
06579e5
another rename
zucchini-nlp Sep 15, 2025
0388c2e
fix copies again
zucchini-nlp Sep 15, 2025
f2ef098
activate pipeline mixin in some models
zucchini-nlp Sep 15, 2025
4c5fbb5
audio loading
zucchini-nlp Sep 15, 2025
cb18262
typo
zucchini-nlp Sep 15, 2025
08873fb
fix the test
zucchini-nlp Sep 15, 2025
06dddc1
stupid typo in filename
zucchini-nlp Sep 15, 2025
338baa0
fix copies
zucchini-nlp Sep 15, 2025
7a5e080
docs
zucchini-nlp Sep 15, 2025
076105d
forgot
zucchini-nlp Sep 15, 2025
5670450
fix pipe tests
zucchini-nlp Sep 16, 2025
8665854
fix copies
zucchini-nlp Sep 16, 2025
aafaae4
fix test
zucchini-nlp Sep 16, 2025
4252a4c
lets not pass it explicitly
zucchini-nlp Sep 16, 2025
f47372a
final fix
zucchini-nlp Sep 16, 2025
51057b3
rename in test files as well
zucchini-nlp Sep 16, 2025
ee1251d
fix again after reordering...
zucchini-nlp Sep 16, 2025
073b782
Merge branch 'main' into auto-multimodal
zucchini-nlp Sep 16, 2025
d9e74e7
add qwen2 audio
zucchini-nlp Sep 17, 2025
654db8f
Merge branch 'main' into auto-multimodal
zucchini-nlp Sep 24, 2025
61428ad
add qwen3-omni
zucchini-nlp Sep 24, 2025
e3b2318
wait, I didn't push it last time?
zucchini-nlp Sep 26, 2025
9c2404f
it's only torch from now on
zucchini-nlp Sep 26, 2025
067061b
how was the model merged with docstring issues?
zucchini-nlp Sep 26, 2025
79f9275
merge main
zucchini-nlp Oct 16, 2025
51fafd3
make style
zucchini-nlp Oct 16, 2025
5855a4a
Merge remote-tracking branch 'upstream/main' into auto-multimodal
zucchini-nlp Oct 16, 2025
cfd8d1b
requires backend depends on input modalities
zucchini-nlp Oct 16, 2025
67f8022
add repr
zucchini-nlp Oct 16, 2025
271ebd1
Merge branch 'main' into auto-multimodal
zucchini-nlp Nov 6, 2025
df7e556
Merge branch 'main' into auto-multimodal
zucchini-nlp Nov 13, 2025
28fd203
fix copies
zucchini-nlp Nov 13, 2025
8101f3f
merge main
zucchini-nlp Nov 17, 2025
27c15e4
fox copies, new models were added
zucchini-nlp Nov 17, 2025
40c96a3
merge main
zucchini-nlp Nov 24, 2025
e344128
and now fix copies
zucchini-nlp Nov 24, 2025
d4da21e
Merge branch 'main' into auto-multimodal
zucchini-nlp Nov 26, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
91 changes: 91 additions & 0 deletions docs/source/en/tasks/multimodal_generation.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,91 @@
<!--Copyright 2024 The HuggingFace Team. All rights reserved.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
specific language governing permissions and limitations under the License.

⚠️ Note that this file is in Markdown but contain specific syntax for our doc-builder (similar to MDX) that may not be
rendered properly in your Markdown viewer.

-->

# Multimodal Generation

[[open-in-colab]]

Multimodal (any-to-any) models are language models capable of processing diverse types of input data (e.g., text, images, audio, or video) and generating outputs in any of these modalities. Unlike traditional unimodal or fixed-modality models, they allow flexible combinations of input and output, enabling a single system to handle a wide range of tasks: from text-to-image generation to audio-to-text transcription, image captioning, video understanding, and so on. This task shares many similarities with image-text-to-text, but supports a wider range of input and output modalities.

## Pipeline

The fastest way to get started is to use the [`Pipeline`] API. Specify the `"multimodal-generation"` task and the model you want to use.
Comment thread
zucchini-nlp marked this conversation as resolved.
Resolved

```python
from transformers import pipeline
pipe = pipeline("multimodal-generation", model="google/gemma-3n-E4B-it")
```

The example below uses chat templates to format the text inputs and uses audio modality as an multimodal data.

```python
messages = [
{
"role": "user",
"content": [
{
"type": "audio",
"url": "https://huggingface.co/datasets/raushan-testing-hf/audio-test/resolve/main/glass-breaking-151256.mp3",
},
{"type": "text", "text": "What do you hear in this audio?"},
],
},
]
```

Pass the chat template formatted text and image to [`Pipeline`] and set `return_full_text=False` to remove the input from the generated output.

```python
outputs = pipe(text=messages, max_new_tokens=20, return_full_text=False)
outputs[0]["generated_text"]
```

If you prefer, you can also load images as input like so:

```python
messages = [
{
"role": "user",
"content": [
{
"type": "image",
"url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/cats.png",
},
{"type": "text", "text": "What do you hear in this audio?"},
],
},
]
outputs = pipe(text=messages, images=images, max_new_tokens=50, return_full_text=False)
outputs[0]["generated_text"]
```

Multimodal Pipeline also supports generating audio or images with any-to-any models. For that you need to set `generation_mode` parameter and set video sampling to the desired FPS as follows:

```python
import soundfile as sf
pipe = pipeline("multimodal-generation", model="Qwen/Qwen2.5-Omni-3B")
messages = [
{
"role": "user",
"content": [
{"type": "video", "path": "https://huggingface.co/datasets/raushan-testing-hf/videos-test/resolve/main/Cooking_cake.mp4"},
{"type": "text", "text": "Describe this video."},
],
},
]
output = pipe(text=messages, fps=2, use_audio_in_video=True, max_new_tokens=20, generation_mmode="audio")
Comment thread
zucchini-nlp marked this conversation as resolved.
Resolved
sf.write("generated_audio.wav", out[0]["generated_audio"])
```
2 changes: 2 additions & 0 deletions src/transformers/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -166,6 +166,7 @@
"JsonPipelineDataFormat",
"KeypointMatchingPipeline",
"MaskGenerationPipeline",
"MultimodalGenerationPipeline",
"NerPipeline",
"ObjectDetectionPipeline",
"PipedPipelineDataFormat",
Expand Down Expand Up @@ -836,6 +837,7 @@
from .pipelines import JsonPipelineDataFormat as JsonPipelineDataFormat
from .pipelines import KeypointMatchingPipeline as KeypointMatchingPipeline
from .pipelines import MaskGenerationPipeline as MaskGenerationPipeline
from .pipelines import MultimodalGenerationPipeline as MultimodalGenerationPipeline
from .pipelines import NerPipeline as NerPipeline
from .pipelines import ObjectDetectionPipeline as ObjectDetectionPipeline
from .pipelines import PipedPipelineDataFormat as PipedPipelineDataFormat
Expand Down
23 changes: 23 additions & 0 deletions src/transformers/models/auto/modeling_auto.py
Original file line number Diff line number Diff line change
Expand Up @@ -1034,6 +1034,19 @@ class _BaseModelWithGenerate(PreTrainedModel, GenerationMixin):
]
)

# Models that accept text and optionally multimodal data in inputs
# and can generate text and optionally multimodal data.
MODEL_FOR_MULTIMODAL_LM_MAPPING_NAMES = OrderedDict(
[
*list(MODEL_FOR_IMAGE_TEXT_TO_TEXT_MAPPING_NAMES.items()),
("granite_speech", "GraniteSpeechForConditionalGeneration"),
("kyutai_speech_to_text", "KyutaiSpeechToTextForConditionalGeneration"),
("phi4_multimodal", "Phi4MultimodalForCausalLM"),
("qwen2_5_omni", "Qwen2_5OmniForConditionalGeneration"),
Comment thread
zucchini-nlp marked this conversation as resolved.
Resolved
("voxtral", "VoxtralForConditionalGeneration"),
]
)

MODEL_FOR_MASKED_LM_MAPPING_NAMES = OrderedDict(
[
# Model for Masked LM mapping
Expand Down Expand Up @@ -1806,6 +1819,7 @@ class _BaseModelWithGenerate(PreTrainedModel, GenerationMixin):
MODEL_FOR_IMAGE_TEXT_TO_TEXT_MAPPING = _LazyAutoMapping(
CONFIG_MAPPING_NAMES, MODEL_FOR_IMAGE_TEXT_TO_TEXT_MAPPING_NAMES
)
MODEL_FOR_MULTIMODAL_LM_MAPPING = _LazyAutoMapping(CONFIG_MAPPING_NAMES, MODEL_FOR_MULTIMODAL_LM_MAPPING_NAMES)
MODEL_FOR_RETRIEVAL_MAPPING = _LazyAutoMapping(CONFIG_MAPPING_NAMES, MODEL_FOR_RETRIEVAL_MAPPING_NAMES)
MODEL_FOR_VISUAL_QUESTION_ANSWERING_MAPPING = _LazyAutoMapping(
CONFIG_MAPPING_NAMES, MODEL_FOR_VISUAL_QUESTION_ANSWERING_MAPPING_NAMES
Expand Down Expand Up @@ -2150,6 +2164,13 @@ def from_pretrained(
AutoModelForImageTextToText = auto_class_update(AutoModelForImageTextToText, head_doc="image-text-to-text modeling")


class AutoModelForMultimodalLM(_BaseAutoModelClass):
_model_mapping = MODEL_FOR_MULTIMODAL_LM_MAPPING


AutoModelForMultimodalLM = auto_class_update(AutoModelForMultimodalLM, head_doc="multimodal generation")


class AutoModelForAudioClassification(_BaseAutoModelClass):
_model_mapping = MODEL_FOR_AUDIO_CLASSIFICATION_MAPPING

Expand Down Expand Up @@ -2300,6 +2321,7 @@ def from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs):
"MODEL_FOR_VISION_2_SEQ_MAPPING",
"MODEL_FOR_RETRIEVAL_MAPPING",
"MODEL_FOR_IMAGE_TEXT_TO_TEXT_MAPPING",
"MODEL_FOR_MULTIMODAL_LM_MAPPING",
"MODEL_FOR_VISUAL_QUESTION_ANSWERING_MAPPING",
"MODEL_MAPPING",
"MODEL_WITH_LM_HEAD_MAPPING",
Expand Down Expand Up @@ -2327,6 +2349,7 @@ def from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs):
"AutoModelForMaskedImageModeling",
"AutoModelForMaskedLM",
"AutoModelForMultipleChoice",
"AutoModelForMultimodalLM",
"AutoModelForNextSentencePrediction",
"AutoModelForObjectDetection",
"AutoModelForPreTraining",
Expand Down
35 changes: 35 additions & 0 deletions src/transformers/models/emu3/processing_emu3.py
Original file line number Diff line number Diff line change
Expand Up @@ -244,5 +244,40 @@ def calculate_generate_size(self, ratio, image_area, spatial_factor):
def postprocess(self, images: ImageInput, **kwargs):
return self.image_processor.postprocess(images, **kwargs)

def post_process_multimodal_output(
self, generated_outputs, skip_special_tokens=True, generation_mode=None, **kwargs
):
"""
Post-process the output of a multimodal model to return the requested modality output.
If the model cannot generated the requested modality, an error will be raised.

Args:
generated_outputs (`torch.Tensor` or `np.ndarray`):
The output of the model `generate` function. The output is expected to be a tensor of shape `(batch_size, sequence_length)`
or `(sequence_length,)`.
skip_special_tokens (`bool`, *optional*, defaults to `True`):
Whether or not to remove special tokens in the output. Argument passed to the tokenizer's `batch_decode` method.
generation_mode (`str`, *optional*):
Generation mode indicated which modality to output and can be one of `["text", "image", "audio"]`.
**kwargs:
Additional arguments to be passed to the tokenizer's `batch_decode method`.

Returns:
`list[Union[str, PIL.Image.Image]]`: The decoded text or generated image.
"""
if generation_mode == "text":
return self.post_process_image_text_to_text(
generated_outputs, skip_special_tokens=skip_special_tokens, **kwargs
)

elif generation_mode == "image":
images = self.postprocess(generated_outputs, return_tensors="PIL.Image.Image")
return images["pixel_values"]

else:
raise ValueError(
f"{self.__class__.__name__} got an unexpected generation_mode={generation_mode}. Supported options are only `text` and `image"
)


__all__ = ["Emu3Processor"]
35 changes: 35 additions & 0 deletions src/transformers/models/janus/processing_janus.py
Original file line number Diff line number Diff line change
Expand Up @@ -160,5 +160,40 @@ def postprocess(self, images: ImageInput, **kwargs):
"""
return self.image_processor.postprocess(images, **kwargs)

def post_process_multimodal_output(
self, generated_outputs, skip_special_tokens=True, generation_mode=None, **kwargs
):
"""
Post-process the output of a multimodal model to return the requested modality output.
If the model cannot generated the requested modality, an error will be raised.

Args:
generated_outputs (`torch.Tensor` or `np.ndarray`):
The output of the model `generate` function. The output is expected to be a tensor of shape `(batch_size, sequence_length)`
or `(sequence_length,)`.
skip_special_tokens (`bool`, *optional*, defaults to `True`):
Whether or not to remove special tokens in the output. Argument passed to the tokenizer's `batch_decode` method.
generation_mode (`str`, *optional*):
Generation mode indicated which modality to output and can be one of `["text", "image", "audio"]`.
**kwargs:
Additional arguments to be passed to the tokenizer's `batch_decode method`.

Returns:
`list[Union[str, PIL.Image.Image]]`: The decoded text or generated image.
"""
if generation_mode == "text":
return self.post_process_image_text_to_text(
generated_outputs, skip_special_tokens=skip_special_tokens, **kwargs
)

elif generation_mode == "image":
images = self.postprocess(generated_outputs, return_tensors="PIL.Image.Image")
return images["pixel_values"]

else:
raise ValueError(
f"{self.__class__.__name__} got an unexpected generation_mode={generation_mode}. Supported options are only `text` and `image"
)


__all__ = ["JanusProcessor"]
53 changes: 53 additions & 0 deletions src/transformers/models/qwen2_5_omni/processing_qwen2_5_omni.py
Original file line number Diff line number Diff line change
Expand Up @@ -339,6 +339,59 @@ def apply_chat_template(self, conversations, chat_template=None, **kwargs):

return super().apply_chat_template(conversations, chat_template, **kwargs)

def post_process_image_text_to_text(self, generated_outputs, skip_special_tokens=True, **kwargs):
"""
Post-process the output of a vlm to decode the text.

Args:
generated_outputs (`torch.Tensor` or `np.ndarray`):
The output of the model `generate` function. The output is expected to be a tensor of shape `(batch_size, sequence_length)`
or `(sequence_length,)`.
skip_special_tokens (`bool`, *optional*, defaults to `True`):
Whether or not to remove special tokens in the output. Argument passed to the tokenizer's `batch_decode` method.
**kwargs:
Additional arguments to be passed to the tokenizer's `batch_decode method`.

Returns:
`list[str]`: The decoded text.
"""
return self.tokenizer.batch_decode(generated_outputs[0], skip_special_tokens=skip_special_tokens, **kwargs)

def post_process_multimodal_output(
self, generated_outputs, skip_special_tokens=True, generation_mode=None, **kwargs
):
"""
Post-process the output of a multimodal model to return the requested modality output.
If the model cannot generated the requested modality, an error will be raised.

Args:
generated_outputs (`torch.Tensor` or `np.ndarray`):
The output of the model `generate` function. The output is expected to be a tensor of shape `(batch_size, sequence_length)`
or `(sequence_length,)`.
skip_special_tokens (`bool`, *optional*, defaults to `True`):
Whether or not to remove special tokens in the output. Argument passed to the tokenizer's `batch_decode` method.
generation_mode (`str`, *optional*):
Generation mode indicated which modality to output and can be one of `["text", "image", "audio"]`.
**kwargs:
Additional arguments to be passed to the tokenizer's `batch_decode method`.

Returns:
`list[Inion[str, np.ndarray]]`: The decoded text or generated audio.
"""
if generation_mode == "text":
return self.post_process_image_text_to_text(
generated_outputs, skip_special_tokens=skip_special_tokens, **kwargs
)

elif generation_mode == "audio":
audio = generated_outputs[1].reshape(-1).detach().cpu().numpy()
return audio

else:
raise ValueError(
f"{self.__class__.__name__} got an unexpected generation_mode={generation_mode}. Supported options are only `text` and `audio"
)

@property
def model_input_names(self):
tokenizer_input_names = self.tokenizer.model_input_names
Expand Down
13 changes: 13 additions & 0 deletions src/transformers/pipelines/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,7 @@
is_torch_available,
logging,
)
from .any_to_any import MultimodalGenerationPipeline
from .audio_classification import AudioClassificationPipeline
from .automatic_speech_recognition import AutomaticSpeechRecognitionPipeline
from .base import (
Expand Down Expand Up @@ -125,6 +126,7 @@
AutoModelForKeypointMatching,
AutoModelForMaskedLM,
AutoModelForMaskGeneration,
AutoModelForMultimodalLM,
AutoModelForObjectDetection,
AutoModelForQuestionAnswering,
AutoModelForSemanticSegmentation,
Expand Down Expand Up @@ -448,6 +450,17 @@
"default": {"model": {"pt": ("magic-leap-community/superglue_outdoor", "f4041f8")}},
"type": "image",
},
"multimodal-generation": {
"impl": MultimodalGenerationPipeline,
"tf": (),
Comment thread
zucchini-nlp marked this conversation as resolved.
Resolved
"pt": (AutoModelForMultimodalLM,) if is_torch_available() else (),
"default": {
"model": {
"pt": ("google/gemma-3n-E4B-it", "c1221e9"),
}
},
"type": "multimodal",
},
}

PIPELINE_REGISTRY = PipelineRegistry(supported_tasks=SUPPORTED_TASKS, task_aliases=TASK_ALIASES)
Expand Down
Loading