[generate] Fix encoder decoder models attention mask

[eustlb]

What does this PR do?

#35852 introduced a bug where the attention mask is lost for encoder-decoder models.
This PR implements a naïve fix.

cf #36020

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[zucchini-nlp]

Thanks for noticing it's broken, should've triggered some slow tests before merging!

I agree that we'd need to pass both attention masks for encoder-decoder models if available. And the attention mask in prepare_inputs should come from decoder for further manipulations no top

Overall LGTM but would be cleaner if we could pass encoder-mask from within kwargs in step 7, imo. CC @gante for one more opinion on generation

[eustlb]

agree that having a encoder_mask would be cleaner, but it diverges a bit with the current convention in the codebase where encoder-decoder model forward's use attention_mask for the encoder and decoder_attention_mask otherwise. It is also how it is currently documents:

transformers/src/transformers/generation/utils.py

Lines 1957 to 1959 in 9d2056f

	Ad hoc parametrization of `generation_config` and/or additional model-specific kwargs that will be
	forwarded to the `forward` function of the model. If the model is an encoder-decoder model, encoder
	specific kwargs should not be prefixed and decoder specific kwargs should be prefixed with *decoder_*.

[gante]

LGTM, thank you for the fix 🙏

P.S.: can we somehow add a fast test to ensure this doesn't happen again? This is a pretty basic feature, and should be covered by fast tests

[gante]

@zucchini-nlp As @eustlb wrote, historically the naming convention is for the attention_mask to attend to the input (back when there were no multimodal models) 👀

[gante]

@eustlb I'm going to rerun the tests until CI gets green, and then merge this PR -- not having this fix is blocking me :D