[Arnavsharma2]

Idefics2 raises IndexError for [[], [image]] because padding assumes the first batch row contains an image. Use the first available image for channels/device in both backends, preserving image outputs and zero masks for text-only examples.

Related to #31854

Tests

3 failures -> 3 passes

• python -m pytest -q tests/models/idefics2/test_image_processing_idefics2.py: 25 passed, 3 skipped.
• python -m pytest -q tests/models/idefics2/test_processing_idefics2.py: 31 passed, 31 skipped.
• make fix-repo and make typing: passed.

CPU only

AI disclosure

AI helped in the development of tests and code

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[zucchini-nlp]

The support has been added already (in #36682) and there is a test as well, can you double-check if the test is running on Idefics2 and provide a clear reproducer of the "bug"?

[zucchini-nlp]

If it is just a PR trying to solve an old GH issue, better to simply close it! The issue is too old and has been resolved

[Arnavsharma2]

The support has been added already (in #36682) and there is a test as well, can you double-check if the test is running on Idefics2 and provide a clear reproducer of the "bug"?

Hi!

I double-checked, and [[], [image]] still raises IndexError in both Idefics2 image-processing backends.

I also ran test_processor_text_has_no_visual for Idefics2.

It is collected but skipped because the guard needs both images and videos in the processor signature.

Here's a small reproducer:

import numpy as np
from transformers import (
    Idefics2ImageProcessor,
    Idefics2ImageProcessorPil,
)

image = np.zeros((8, 10, 3), dtype=np.uint8)

for cls in (Idefics2ImageProcessor, Idefics2ImageProcessorPil):
    processor = cls(do_resize=False, do_image_splitting=False)
    try:
        result = processor([[], [image]], return_tensors="pt")
        print(cls.__name__, result.pixel_values.shape)
    except IndexError as error:
        print(cls.__name__, repr(error))

Both of them fail during padding when accessing the first image of the empty first row. My changes change it so that both return shape (2, 1, 3, 8, 10) with a zero pixel-attention mask for the text-only row.

[zucchini-nlp]

the guard needs both images and videos in the processor signature

nooooooo 😿 Oke, in that case would you be open to re-make the test and allow image-only inputs as well? That feature should be working in all VLMs, no matter if video is supported or not

[Arnavsharma2]

the guard needs both images and videos in the processor signature

nooooooo 😿 Oke, in that case would you be open to re-make the test and allow image-only inputs as well? That feature should be working in all VLMs, no matter if video is supported or not

Sure!

[Arnavsharma2]

the guard needs both images and videos in the processor signature

nooooooo 😿 Oke, in that case would you be open to re-make the test and allow image-only inputs as well? That feature should be working in all VLMs, no matter if video is supported or not

Just pushed the changes!

[zucchini-nlp]

We'll defi need to fix failing models now, there should be more than only Idefics2

[Arnavsharma2]

We'll defi need to fix failing models now, there should be more than only Idefics2

Pushed the follow-up!
PI0 was also crashing on empty image rows, so I fixed that and added test coverage

NeoMME already supports mixed document batches, but inputs didn’t match its retrieval format. I modified its test to cover both direct inputs and its supported chat format

GLM-Image requires equal source-image counts across the batch for image generation. I kept that restriction and made a test that ensures it holds up that restriction!

Tests pass: 88 pass 103 skip

[zucchini-nlp]

thanks! no need for new tests though right?

[zucchini-nlp]

thx

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[github-actions]

[For maintainers] Suggested jobs to run (before merge)

run-slow: glm_image, idefics2, lighton_ocr, neomme, pi0

[github-actions]

---

CI recap

Dashboard: View test results in Grafana
Latest run: 35206572589:2
Result: success | Jobs: 16 | Tests: 190,478 | Failures: 2 | Duration: 13h 18m