Skip to content
Open
Show file tree
Hide file tree
Changes from 4 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions docs/source/en/_toctree.yml
Original file line number Diff line number Diff line change
Expand Up @@ -1254,6 +1254,8 @@
title: PaliGemma
- local: model_doc/pe_audio_video
title: PE Audio Video
- local: model_doc/penguinvl
title: PenguinVL
- local: model_doc/perceiver
title: Perceiver
- local: model_doc/perception_lm
Expand Down
298 changes: 298 additions & 0 deletions docs/source/en/model_doc/penguinvl.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,298 @@
<!--Copyright 2025 Tencent and The HuggingFace Team. All rights reserved.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: can you make sure it's 2026 everywhere


Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
specific language governing permissions and limitations under the License.

⚠️ Note that this file is in Markdown but contain specific syntax for our doc-builder (similar to MDX) that may not be
rendered properly in your Markdown viewer.

-->
*This model was released on 2026-03-06 and added to Hugging Face Transformers on 2026-03-13.*

# PenguinVL

<div class="flex flex-wrap space-x-1">
<img alt="PyTorch" src="https://img.shields.io/badge/PyTorch-DE3412?style=flat&logo=pytorch&logoColor=white">
<img alt="FlashAttention" src="https://img.shields.io/badge/%E2%9A%A1%EF%B8%8E%20FlashAttention-eae0c8?style=flat">
<img alt="SDPA" src="https://img.shields.io/badge/SDPA-DE3412?style=flat&logo=pytorch&logoColor=white">
</div>

## Overview

**Penguin-VL** is a compact vision-language model family built to study how far multimodal efficiency can be pushed by redesigning the **vision encoder**, rather than only scaling data or model size.
Comment thread
CyrilSterling marked this conversation as resolved.
Resolved

Most modern VLMs rely on vision encoders pretrained with large-scale **contrastive objectives** such as CLIP or SigLIP. Penguin-VL argues that this setup can be suboptimal for multimodal reasoning because contrastive learning favors coarse category-level invariances over the fine-grained signals needed for **OCR, document understanding, dense captioning, and complex reasoning**. Instead, Penguin-VL introduces **Penguin-Encoder**, a vision encoder **initialized from a text-only LLM**, so the visual backbone starts closer to the language model representation space and learns more data-efficiently.
Comment thread
CyrilSterling marked this conversation as resolved.
Resolved

<img src="https://github.com/tencent-ailab/Penguin-VL/blob/master/assets/framework.png?raw=true"
alt="drawing" width="600"/>
<small> PenguinVL architecture. Details are in <a href="https://huggingface.co/papers/2603.06569">technical report.</a> </small>

This model was contributed by [Cyril666](https://huggingface.co/Cyril666).

## Usage example

### Single media inference

PenguinVL accepts both images and videos as input. Use `processor.process_vision_info` to extract visual inputs from messages **before** calling `apply_chat_template`.
Comment thread
CyrilSterling marked this conversation as resolved.
Resolved

```python
import torch
from transformers import PenguinVLProcessor, PenguinVLForConditionalGeneration

model = PenguinVLForConditionalGeneration.from_pretrained(
"tencent/Penguin-VL-8B",
torch_dtype=torch.bfloat16,

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: dtype

device_map="auto",
)
processor = PenguinVLProcessor.from_pretrained("tencent/Penguin-VL-8B")

# Image
messages = [
{
"role": "user",
"content": [
{"type": "image", "image": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/pipeline-cat-chonk.jpeg"},
{"type": "text", "text": "Describe this image."},
],
}
]

images, frame_types = processor.process_vision_info(messages)
text = processor.apply_chat_template(messages, add_generation_prompt=True)
inputs = processor(
images=images,
text=text,
frame_types=frame_types,
return_tensors="pt",
).to(model.device)

inputs = {k: v.to(model.device) if isinstance(v, torch.Tensor) else v for k, v in inputs.items()}
if "pixel_values" in inputs:
inputs["pixel_values"] = inputs["pixel_values"].to(torch.bfloat16)
Comment on lines +73 to +77

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

more concise: inputs.to(device=model.device, dtype=torch.bfloat16)

output_ids = model.generate(**inputs, max_new_tokens=128)
generated_ids = [output_ids[len(input_ids):] for input_ids, output_ids in zip(inputs.input_ids, output_ids)]

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

concise way: output_ids[:, inputs.input_ids.shape[1]: ]

output_text = processor.batch_decode(generated_ids, skip_special_tokens=True, clean_up_tokenization_spaces=True)
print(output_text)
```

### Video inference

```python
import torch
from transformers import PenguinVLProcessor, PenguinVLForConditionalGeneration

model = PenguinVLForConditionalGeneration.from_pretrained(
"tencent/Penguin-VL-8B",
torch_dtype=torch.bfloat16,
device_map="auto",
)
processor = PenguinVLProcessor.from_pretrained("tencent/Penguin-VL-8B")

messages = [
{
"role": "user",
"content": [
{"type": "video", "video": "/path/to/video.mp4"},
{"type": "text", "text": "What happened in the video?"},
],
}
]

# process_vision_info must be called before apply_chat_template for videos
# It samples frames at `fps`, caps at `max_frames`, and annotates timestamps
images, frame_types = processor.process_vision_info(messages, fps=1, max_frames=128)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same, not needed. You can pass fps and num_frames to the apply_chat_template

text = processor.apply_chat_template(messages, add_generation_prompt=True)
inputs = processor(
images=images,
text=text,
frame_types=frame_types,
return_tensors="pt",
).to(model.device)

inputs = {k: v.to(model.device) if isinstance(v, torch.Tensor) else v for k, v in inputs.items()}
if "pixel_values" in inputs:
inputs["pixel_values"] = inputs["pixel_values"].to(torch.bfloat16)
output_ids = model.generate(**inputs, max_new_tokens=256)
generated_ids = [output_ids[len(input_ids):] for input_ids, output_ids in zip(inputs.input_ids, output_ids)]
output_text = processor.batch_decode(generated_ids, skip_special_tokens=True, clean_up_tokenization_spaces=True)
Comment on lines +118 to +123

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same comment applies as above

print(output_text)
```

### Batch mixed media inference

The model can batch inputs composed of mixed samples (images, videos, and text).

```python
import torch
from transformers import PenguinVLProcessor, PenguinVLForConditionalGeneration

model = PenguinVLForConditionalGeneration.from_pretrained(
"tencent/Penguin-VL-8B",
torch_dtype=torch.bfloat16,
device_map="auto",
)
processor = PenguinVLProcessor.from_pretrained("tencent/Penguin-VL-8B")

conversation1 = [
{
"role": "user",
"content": [
{"type": "image", "image": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/pipeline-cat-chonk.jpeg"},
{"type": "text", "text": "Describe this image."},
],
}
]

conversation2 = [
{
"role": "user",
"content": [
{"type": "video", "video": "/path/to/video.mp4"},
{"type": "text", "text": "Summarize this video."},
],
}
]

conversation3 = [
{
"role": "user",
"content": "What is the capital of France?",
}
]

all_images = []
all_frame_types = []
all_texts = []
for conv in [conversation1, conversation2, conversation3]:
imgs, fts = processor.process_vision_info(conv, fps=1, max_frames=64)
if imgs is not None:
all_images.extend(imgs)
if fts is not None:
all_frame_types.extend(fts)
all_texts.append(processor.apply_chat_template(conv, add_generation_prompt=True))

inputs = processor(
images=all_images if all_images else None,
text=all_texts,
frame_types=all_frame_types if all_frame_types else None,
padding=True,
return_tensors="pt",
).to(model.device)
Comment on lines +169 to +186

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

single apply_chat_template call will do the work here as well


output_ids = model.generate(**inputs, max_new_tokens=128)
generated_ids = [output_ids[len(input_ids):] for input_ids, output_ids in zip(inputs.input_ids, output_ids)]
output_text = processor.batch_decode(generated_ids, skip_special_tokens=True, clean_up_tokenization_spaces=True)
print(output_text)
```
### process_vision_info function

`process_vision_info` extracts and loads visual inputs (images and video frames) from Qwen2-VL style conversation messages. It walks through the messages, collects images/video frames in order, and for video clips samples frames at the given `fps` (capped at `max_frames`). Video content items in `messages` are enriched in-place with `num_frames` and `timestamps` so that `apply_chat_template` can emit per-frame timestamp prefixes.

**Important:** You must call `process_vision_info` **before** `apply_chat_template`, because it modifies the `messages` in-place when processing videos.
Comment thread
CyrilSterling marked this conversation as resolved.
Resolved

Supported content block formats:

**Image** — URL (HTTP or file) or PIL Image:
Comment thread
CyrilSterling marked this conversation as resolved.
Resolved

```python
{"type": "image", "image": "https://example.com/photo.jpg"}
{"type": "image", "image": "file:///path/to/image.png"}
{"type": "image", "image": <PIL.Image.Image>}
```

**Video** — URL, or list of frames with timestamps:

```python
{"type": "video", "video": "https://example.com/clip.mp4"}
{"type": "video", "video": ["file:///path/frame1.jpg", ...], "timestamps": [0, ...]}
{"type": "video", "video": [<PIL.Image.Image>, ...], "timestamps": [0, ...]}
```

### Flash-Attention 2 to speed up generation

First, make sure to install the latest version of Flash Attention 2:

```bash
pip install -U flash-attn --no-build-isolation
```

Also, you should have hardware that is compatible with Flash Attention 2. Read more about it in the official documentation of the [flash attention repository](https://github.com/Dao-AILab/flash-attention). FlashAttention-2 can only be used when a model is loaded in `torch.float16` or `torch.bfloat16`.

To load and run a model using Flash Attention-2, simply add `attn_implementation="flash_attention_2"` when loading the model:

```python
import torch
from transformers import PenguinVLForConditionalGeneration

model = PenguinVLForConditionalGeneration.from_pretrained(
"tencent/Penguin-VL-8B",
torch_dtype=torch.bfloat16,

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

dtype

attn_implementation="flash_attention_2",
)
```

## Notes

- Use `min_pixels` and `max_pixels` to control image resolution and memory usage.

```python
from transformers import PenguinVLProcessor

processor = PenguinVLProcessor.from_pretrained(
"tencent/Penguin-VL-8B",
min_pixels=256 * 14 * 14,
max_pixels=1280 * 14 * 14,
)
```

- For video inputs, `process_vision_info` must be called **before** `apply_chat_template`. It samples frames at the given `fps`, caps total frames at `max_frames`, and annotates each video entry in `messages` with `num_frames` and `timestamps` so the chat template can emit per-frame timestamp prefixes.
Comment thread
CyrilSterling marked this conversation as resolved.
Resolved

- Video frames are automatically classified as **keyframes (K)** or **intermediate frames (I)** via the TRA mechanism. Keyframes receive a smaller spatial merge factor (better quality) and intermediate frames receive a larger one (higher compression). This is handled automatically when you pass `frame_types` to the processor.

- Pass `frame_types=None` (or omit it) if you are processing only images.

## PenguinVLConfig

[[autodoc]] PenguinVLConfig

## PenguinVLVisionConfig

[[autodoc]] PenguinVLVisionConfig

## PenguinVLImageProcessor

[[autodoc]] PenguinVLImageProcessor
- preprocess

## PenguinVLImageProcessorFast

[[autodoc]] PenguinVLImageProcessorFast
- preprocess

## PenguinVLProcessor

[[autodoc]] PenguinVLProcessor
- __call__

## PenguinVLVisionModel

[[autodoc]] PenguinVLVisionModel
- forward

## PenguinVLModel

[[autodoc]] PenguinVLModel
- forward
- get_image_features

## PenguinVLForConditionalGeneration

[[autodoc]] PenguinVLForConditionalGeneration
- forward
- get_image_features
1 change: 1 addition & 0 deletions src/transformers/conversion_mapping.py
Original file line number Diff line number Diff line change
Expand Up @@ -437,6 +437,7 @@ def register_checkpoint_conversion_mapping(
"mistral3",
"mllama",
"paligemma",
"penguinvl",

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

needs a rebase, all VLMs were removed from this list

"shieldgemma2",
"qwen2vl",
"qwen2_5_vl",
Expand Down
1 change: 1 addition & 0 deletions src/transformers/models/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -303,6 +303,7 @@
from .pe_video import *
from .pegasus import *
from .pegasus_x import *
from .penguinvl import *
from .perceiver import *
from .perception_lm import *
from .persimmon import *
Expand Down
5 changes: 5 additions & 0 deletions src/transformers/models/auto/configuration_auto.py
Original file line number Diff line number Diff line change
Expand Up @@ -342,6 +342,8 @@
("pe_video_encoder", "PeVideoEncoderConfig"),
("pegasus", "PegasusConfig"),
("pegasus_x", "PegasusXConfig"),
("penguinvl", "PenguinVLConfig"),
("penguinvl_vision", "PenguinVLVisionConfig"),
("perceiver", "PerceiverConfig"),
("perception_lm", "PerceptionLMConfig"),
("persimmon", "PersimmonConfig"),
Expand Down Expand Up @@ -851,6 +853,8 @@
("pe_video_encoder", "PeVideoEncoder"),
("pegasus", "Pegasus"),
("pegasus_x", "PEGASUS-X"),
("penguinvl", "PenguinVL"),
("penguinvl_vision", "PenguinVLVision"),
("perceiver", "Perceiver"),
("perception_lm", "PerceptionLM"),
("persimmon", "Persimmon"),
Expand Down Expand Up @@ -1099,6 +1103,7 @@
("llama4_text", "llama4"),
("blip_2_qformer", "blip_2"),
("fastspeech2_conformer_with_hifigan", "fastspeech2_conformer"),
("penguinvl_vision", "penguinvl"),
("perception_encoder", "perception_lm"),
("pe_audio_encoder", "pe_audio"),
("pe_video_encoder", "pe_video"),
Expand Down
1 change: 1 addition & 0 deletions src/transformers/models/auto/image_processing_auto.py
Original file line number Diff line number Diff line change
Expand Up @@ -163,6 +163,7 @@
("owlvit", ("OwlViTImageProcessor", "OwlViTImageProcessorFast")),
("paddleocr_vl", ("PaddleOCRVLImageProcessor", "PaddleOCRVLImageProcessorFast")),
("paligemma", ("SiglipImageProcessor", "SiglipImageProcessorFast")),
("penguinvl", ("PenguinVLImageProcessor", "PenguinVLImageProcessorFast")),
("perceiver", ("PerceiverImageProcessor", "PerceiverImageProcessorFast")),
("perception_lm", (None, "PerceptionLMImageProcessorFast")),
("phi4_multimodal", (None, "Phi4MultimodalImageProcessorFast")),
Expand Down
3 changes: 3 additions & 0 deletions src/transformers/models/auto/modeling_auto.py
Original file line number Diff line number Diff line change
Expand Up @@ -334,6 +334,8 @@ class _BaseModelWithGenerate(PreTrainedModel, GenerationMixin):
("pe_video_encoder", "PeVideoEncoder"),
("pegasus", "PegasusModel"),
("pegasus_x", "PegasusXModel"),
("penguinvl", "PenguinVLModel"),
("penguinvl_vision", "PenguinVLVisionModel"),
("perceiver", "PerceiverModel"),
("perception_lm", "PerceptionLMModel"),
("persimmon", "PersimmonModel"),
Expand Down Expand Up @@ -981,6 +983,7 @@ class _BaseModelWithGenerate(PreTrainedModel, GenerationMixin):
("ovis2", "Ovis2ForConditionalGeneration"),
("paddleocr_vl", "PaddleOCRVLForConditionalGeneration"),
("paligemma", "PaliGemmaForConditionalGeneration"),
("penguinvl", "PenguinVLForConditionalGeneration"),
("perception_lm", "PerceptionLMForConditionalGeneration"),
("pix2struct", "Pix2StructForConditionalGeneration"),
("pixtral", "LlavaForConditionalGeneration"),
Expand Down
1 change: 1 addition & 0 deletions src/transformers/models/auto/processing_auto.py
Original file line number Diff line number Diff line change
Expand Up @@ -131,6 +131,7 @@
("owlvit", "OwlViTProcessor"),
("paddleocr_vl", "PaddleOCRVLProcessor"),
("paligemma", "PaliGemmaProcessor"),
("penguinvl", "PenguinVLProcessor"),
("perception_lm", "PerceptionLMProcessor"),
("phi4_multimodal", "Phi4MultimodalProcessor"),
("pix2struct", "Pix2StructProcessor"),
Expand Down
Loading
Loading