[blipbyte]

What does this PR do?

Fixes #48693, where @vasqu agreed it is worth fixing for now, while noting it is not high priority
and that this feature may be removed later. No urgency from me — happy for it to sit, or to close
it if the path is going away.

With add_cross_attention=True, ImageGPTModel is broken four ways where GPT-2 is fine: the
default use_cache=True call raises UnboundLocalError on is_updated; the encoder mask is
discarded, so padded encoder positions still reach the output; a padded decoder attention_mask
raises a RuntimeError on the mask shape; and one cached decoding step leaves 22 keys in the
cross-attention cache for 11 encoder positions.

Each of the three fixes is GPT-2's own code, restored:

• attention_mask = encoder_attention_mask (modeling_gpt2.py:173), dropped by [cache] make all classes cache compatible finally #38635. It
  replaces the decoder mask rather than sitting beside it, which also removes the shape clash.
• the cross-attention cache reuse guard (modeling_gpt2.py:193-195), which Fix GPT2 with cross attention #39754 added to GPT-2
  and, through # Copied from, to decision_transformer. ImageGPT has no such marker.
• the EncoderDecoderCache wrap (modeling_gpt2.py:561-566), removed by Fixes in check_model_inputs, GPTBigCodeModel and ImageGPTModel #40811. It costs nothing
  when add_cross_attention is unset.

The test ports #47946's encoder-mask check under a padded decoder mask, adds an assertion that
corrupting visible encoder positions moves the output (without it the check passes for an
encoder-blind model), and compares one cached decoding step against the full forward. Reverting
any one fix reds it.

pytest tests/models/imagegpt/             119 passed, 181 skipped
RUN_SLOW=1 pytest tests/models/imagegpt/  131 passed, 169 skipped
make fix-repo                             17/17, rewrote nothing
make check-repo                           26/26

AI-assisted, per CONTRIBUTING.md. Coordination: #48693, which I opened and @vasqu answered.
Differentiation: #45773 fixes the same crash for whisper, moonshine, pix2struct and
audioflamingo3, not ImageGPT; #44076 refactors this file's output collection, not the cache or the
mask. I reviewed every changed line and ran the suite locally.

Code Agent Policy

• (First-time contributors only): I confirm that this PR description and code is not written by an LLM or code agent

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline and the
  Pull Request checks?
• Was this discussed/approved via a Github issue or the forum? Please add a link
  to it if that's the case. [ImageGPT] Cross-attention crashes, ignores encoder_attention_mask, and doubles its cache #48693
• Did you make sure to update the documentation with your changes according to the guidelines?
• Did you write any new necessary tests?

Who can review?

@vasqu

[molbap]

thanks, lgtm as long as the tests are simplified, we don't want to add that much code surface.

[github-actions]

[For maintainers] Suggested jobs to run (before merge)

run-slow: imagegpt

[github-actions]

---

CI recap

Dashboard: View test results in Grafana
Latest run: 34530928940:2
Result: success | Jobs: 16 | Tests: 189,123 | Failures: 0 | Duration: 16h 34m

[blipbyte]

Addressed in 7e6c795: the three cross-attention tests are one test_cross_attention now, no fixed seed, and the cache condition is the simplified form you suggested. Ready for another look.