Add Ovis2.5 model support

docs/source/en/_toctree.yml

@@ -1503,6 +1503,8 @@
         title: OneFormer
       - local: model_doc/ovis2
         title: Ovis2
+      - local: model_doc/ovis2_5
+        title: Ovis2.5
       - local: model_doc/owlvit
         title: OWL-ViT
       - local: model_doc/owlv2

docs/source/en/model_doc/ovis2_5.md

@@ -0,0 +1,187 @@
+<!--Copyright 2026 The HuggingFace Team. All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+specific language governing permissions and limitations under the License.
+
+⚠️ Note that this file is in Markdown but contain specific syntax for our doc-builder (similar to MDX) that may not be
+rendered properly in your Markdown viewer.
+
+-->
+*This model was published in HF papers on 2025-08-16 and contributed to Hugging Face Transformers on 2026-11-12.*
+
+# Ovis2.5
+
+## Overview
+
+[Ovis2.5](https://huggingface.co/papers/2508.11737) is a multimodal model from
+[AIDC-AI](https://github.com/AIDC-AI/Ovis) for image, video, and text understanding. It combines a native-resolution
+vision encoder with a visual tokenizer and a Qwen3 language backbone. Unlike a fixed-resolution tiled encoder, the
+vision encoder keeps variable-resolution visual inputs as a sequence of patches. Ovis2.5 also provides an optional
+thinking prompt format for tasks that benefit from a longer reasoning response.
+
+Transformers-compatible checkpoints are available from the Ovis community:
+
+- [ovis-community/Ovis2.5-2B](https://huggingface.co/ovis-community/Ovis2.5-2B)
+- [ovis-community/ovis-2.5-9B](https://huggingface.co/ovis-community/ovis-2.5-9B)
+
+These checkpoints retain the original weights and include native model, tokenizer, and processor metadata. The
+original repositories still use legacy processor metadata. Weight names are mapped during loading.
+
+## Image inference
+
+Use [`Ovis2_5Processor.apply_chat_template`] to load the image, format the conversation, and tokenize it in one call.
+For the 9B checkpoint, replace the model ID.
+
+```python
+from transformers import AutoModelForImageTextToText, AutoProcessor
+
+
+model_id = "ovis-community/Ovis2.5-2B"
+processor = AutoProcessor.from_pretrained(model_id)
+model = AutoModelForImageTextToText.from_pretrained(
+    model_id,
+    device_map="auto",
+)
+
+messages = [
+    {
+        "role": "user",
+        "content": [
+            {
+                "type": "image",
+                "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee.jpg",
+            },
+            {"type": "text", "text": "Describe this image."},
+        ],
+    }
+]
+
+inputs = processor.apply_chat_template(
+    messages,
+    add_generation_prompt=True,
+    tokenize=True,
+    return_dict=True,
+    return_tensors="pt",
+    enable_thinking=False,
+).to(model.device, dtype=model.dtype)
+
+generated_ids = model.generate(**inputs, max_new_tokens=128)
+generated_ids = generated_ids[:, inputs["input_ids"].shape[1] :]
+response = processor.batch_decode(generated_ids, skip_special_tokens=True)[0]
+print(response)
+```
+
+Multiple images can be supplied by adding more `{"type": "image", ...}` items to the message in the same order in which
+they should appear in the prompt.
+
+## Video inference
+
+The official example uniformly samples eight frames. Pass the video path directly in the conversation and let the
+processor load and sample it. The processor returns `pixel_values_videos` and `video_grid_thw` along with the tokenized
+prompt.
+Ovis2.5 accepts exactly one video per request and does not support mixing images and video in the same request.
+
+```python
+messages = [
+    {
+        "role": "user",
+        "content": [
+            {"type": "video", "path": "path/to/video.mp4"},
+            {"type": "text", "text": "Describe what happens in this video."},
+        ],
+    }
+]
+
+inputs = processor.apply_chat_template(
+    messages,
+    add_generation_prompt=True,
+    tokenize=True,
+    return_dict=True,
+    return_tensors="pt",
+    enable_thinking=False,
+    processor_kwargs={
+        "num_frames": 8,
+        "size": {"shortest_edge": 448 * 448, "longest_edge": 896 * 896},
+    },
+).to(model.device, dtype=model.dtype)
+
+generated_ids = model.generate(**inputs, max_new_tokens=256)
+generated_ids = generated_ids[:, inputs["input_ids"].shape[1] :]
+response = processor.batch_decode(generated_ids, skip_special_tokens=True)[0]
+print(response)
+```
+
+## Thinking mode
+
+Set `enable_thinking=True` when applying the chat template to leave the model's reasoning block open. Set it to `False`
+to request the immediate-answer prompt format.
+
+```python
+inputs = processor.apply_chat_template(
+    messages,
+    add_generation_prompt=True,
+    tokenize=True,
+    return_dict=True,
+    return_tensors="pt",
+    enable_thinking=True,
+).to(model.device, dtype=model.dtype)
+
+generated_ids = model.generate(**inputs, max_new_tokens=1024)
+```
+
+`enable_thinking` changes the chat template; it is not a generation argument and does not impose a token budget. The
+two-stage `thinking_budget` behavior implemented by the original custom remote code is not part of the native
+Transformers generation API.
+
+## Ovis2_5Config
+
+[[autodoc]] Ovis2_5Config
+
+## Ovis2_5VisionConfig
+
+[[autodoc]] Ovis2_5VisionConfig
+
+## Ovis2_5Processor
+
+[[autodoc]] Ovis2_5Processor
+    - __call__
+
+## Ovis2_5ImageProcessor
+
+[[autodoc]] Ovis2_5ImageProcessor
+    - preprocess
+
+## Ovis2_5ImageProcessorPil
+
+[[autodoc]] Ovis2_5ImageProcessorPil
+    - preprocess
+
+## Ovis2_5VideoProcessor
+
+[[autodoc]] Ovis2_5VideoProcessor
+    - preprocess
+
+## Ovis2_5VisionModel
+
+[[autodoc]] Ovis2_5VisionModel
+    - forward
+
+## Ovis2_5Model
+
+[[autodoc]] Ovis2_5Model
+    - forward
+    - get_image_features
+    - get_video_features
+
+## Ovis2_5ForConditionalGeneration
+
+[[autodoc]] Ovis2_5ForConditionalGeneration
+    - forward
+    - get_image_features
+    - get_video_features

src/transformers/conversion_mapping.py

@@ -183,6 +183,20 @@ def _build_checkpoint_conversion_mapping():
             WeightRenaming(r"\.mlp\.up_proj\.", ".mlp.fc1."),
             WeightRenaming(r"\.mlp\.down_proj\.", ".mlp.fc2."),
         ],
+        "ovis2_5": [
+            WeightRenaming(r"^llm\.model\.", "model.language_model."),
+            WeightRenaming(r"^llm\.lm_head\.", "lm_head."),
+            WeightRenaming(r"^visual_tokenizer\.vit\.vision_model\.embeddings\.", "model.vision_tower.embeddings."),
+            WeightRenaming(r"^visual_tokenizer\.vit\.vision_model\.encoder\.layers\.", "model.vision_tower.layers."),
+            WeightRenaming(
+                r"^visual_tokenizer\.vit\.vision_model\.post_layernorm\.", "model.vision_tower.post_layernorm."
+            ),
+            WeightRenaming(
+                r"^visual_tokenizer\.head\.0\.", "model.multi_modal_projector.visual_tokenizer.head_linear."
+            ),
+            WeightRenaming(r"^visual_tokenizer\.head\.1\.", "model.multi_modal_projector.visual_tokenizer.head_norm."),
+            WeightRenaming(r"^vte\.", "model.multi_modal_projector.visual_embeddings_table."),
+        ],
         "inkling_mm_model": [
             WeightRenaming(source_patterns=r"model\.llm\.layers", target_patterns=r"model.language_model.layers"),
             WeightRenaming(

src/transformers/exporters/exporter_dynamo.py

@@ -437,6 +437,7 @@ def _to_batched(t):
     # Separate `_proj` + `(cos, sin)` rotary + `.out_proj` (tuple return)
     "transformers.models.lfm3_vl.modeling_lfm3_vl.Lfm3VlVisionAttention.forward",
     "transformers.models.video_llama_3.modeling_video_llama_3.VideoLlama3VisionAttention.forward",
+    "transformers.models.ovis2_5.modeling_ovis2_5.Ovis2_5VisionAttention.forward",
     "transformers.models.paddleocr_vl.modeling_paddleocr_vl.PaddleOCRVisionAttention.forward",
     "transformers.models.minimax_m3_vl.modeling_minimax_m3_vl.MiniMaxM3VLVisionAttention.forward",
     # NaViT (1, T, D) + separate `_proj` + `.out_proj` (tuple return)

src/transformers/models/__init__.py

@@ -370,6 +370,7 @@
     from .openai_privacy_filter import *
     from .opt import *
     from .ovis2 import *
+    from .ovis2_5 import *
     from .owlv2 import *
     from .owlvit import *
     from .paddleocr_vl import *

src/transformers/models/auto/auto_mappings.py

@@ -504,6 +504,8 @@
         ("openai_privacy_filter", "OpenAIPrivacyFilterConfig"),
         ("opt", "OPTConfig"),
         ("ovis2", "Ovis2Config"),
+        ("ovis2_5", "Ovis2_5Config"),
+        ("ovis2_5_vision", "Ovis2_5VisionConfig"),
         ("ovis2_vision", "Ovis2VisionConfig"),
         ("owlv2", "Owlv2Config"),
         ("owlv2_text_model", "Owlv2TextConfig"),
@@ -985,6 +987,7 @@
         ("nllb-moe", "nllb_moe"),
         ("omdet-turbo", "omdet_turbo"),
         ("openai-gpt", "openai"),
+        ("ovis2_5_vision", "ovis2_5"),
         ("ovis2_vision", "ovis2"),
         ("owlv2_text_model", "owlv2"),
         ("owlv2_vision_model", "owlv2"),
@@ -1133,6 +1136,7 @@
         ("molmo2", {"torchvision": "Molmo2VideoProcessor"}),
         ("muse_glimmer", {"torchvision": "MuseGlimmerVideoProcessor"}),
         ("nemotron_h_omni", {"torchvision": "NemotronH_Omni_Reasoning_V3VideoProcessor"}),
+        ("ovis2_5", {"torchvision": "Ovis2_5VideoProcessor"}),
         ("pe_video", {"torchvision": "PeVideoVideoProcessor"}),
         ("perception_lm", {"torchvision": "PerceptionLMVideoProcessor"}),
         ("qwen2_vl", {"torchvision": "Qwen2VLVideoProcessor"}),
@@ -1287,6 +1291,7 @@
         ("omdet-turbo", "OmDetTurboProcessor"),
         ("oneformer", "OneFormerProcessor"),
         ("ovis2", "Ovis2Processor"),
+        ("ovis2_5", "Ovis2_5Processor"),
         ("owlv2", "Owlv2Processor"),
         ("owlvit", "OwlViTProcessor"),
         ("paddleocr_vl", "PaddleOCRVLProcessor"),
@@ -1427,6 +1432,7 @@
         ("nougat", {"pil": "NougatImageProcessorPil", "torchvision": "NougatImageProcessor"}),
         ("oneformer", {"pil": "OneFormerImageProcessorPil", "torchvision": "OneFormerImageProcessor"}),
         ("ovis2", {"pil": "Ovis2ImageProcessorPil", "torchvision": "Ovis2ImageProcessor"}),
+        ("ovis2_5", {"pil": "Ovis2_5ImageProcessorPil", "torchvision": "Ovis2_5ImageProcessor"}),
         ("owlv2", {"pil": "Owlv2ImageProcessorPil", "torchvision": "Owlv2ImageProcessor"}),
         ("owlvit", {"pil": "OwlViTImageProcessorPil", "torchvision": "OwlViTImageProcessor"}),
         ("paddleocr_vl", {"pil": "PaddleOCRVLImageProcessorPil", "torchvision": "PaddleOCRVLImageProcessor"}),

src/transformers/models/auto/modeling_auto.py

@@ -427,6 +427,8 @@ class _BaseModelWithGenerate(PreTrainedModel, GenerationMixin):
         ("openai_privacy_filter", "OpenAIPrivacyFilterModel"),
         ("opt", "OPTModel"),
         ("ovis2", "Ovis2Model"),
+        ("ovis2_5", "Ovis2_5Model"),
+        ("ovis2_5_vision", "Ovis2_5VisionModel"),
         ("owlv2", "Owlv2Model"),
         ("owlvit", "OwlViTModel"),
         ("paddleocr_vl", "PaddleOCRVLModel"),
@@ -1215,6 +1217,7 @@ class _BaseModelWithGenerate(PreTrainedModel, GenerationMixin):
         ("muse_spark", "MuseSparkForConditionalGeneration"),
         ("nemotron_h_omni", "NemotronH_Omni_Reasoning_V3"),
         ("ovis2", "Ovis2ForConditionalGeneration"),
+        ("ovis2_5", "Ovis2_5ForConditionalGeneration"),
         ("paddleocr_vl", "PaddleOCRVLForConditionalGeneration"),
         ("paligemma", "PaliGemmaForConditionalGeneration"),
         ("perception_lm", "PerceptionLMForConditionalGeneration"),

src/transformers/models/ovis2_5/__init__.py

@@ -0,0 +1,37 @@
+#                🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨
+#           This file was automatically generated from the modules in `src/transformers/models/ovis2_5`.
+#           Do NOT edit this file manually as any edits will be overwritten by auto-generation of the file.
+#                       A module is picked up once it exports a name through `__all__`.
+#                   Regenerate the file with: `python utils/check_inits.py --fix_and_overwrite`
+#                🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨
+# Copyright 2026 The HuggingFace Inc. team. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+from typing import TYPE_CHECKING
+
+from ...utils import _LazyModule
+from ...utils.import_utils import define_import_structure
+
+
+if TYPE_CHECKING:
+    from .configuration_ovis2_5 import *
+    from .image_processing_ovis2_5 import *
+    from .image_processing_pil_ovis2_5 import *
+    from .modeling_ovis2_5 import *
+    from .processing_ovis2_5 import *
+    from .video_processing_ovis2_5 import *
+else:
+    import sys
+
+    _file = globals()["__file__"]
+    sys.modules[__name__] = _LazyModule(__name__, _file, define_import_structure(_file), module_spec=__spec__)