Repository navigation
Conversation
Padding can only come from the prompt, so settle once whether the inputs are padded and, when they are not, take the mask out of `model_kwargs` entirely - the per-step `torch.cat` that grows it goes with it. `_prefill` used the mask's length to tell whether `input_ids` holds the whole sequence or only the new tokens, so that answer is recorded before the mask is dropped.
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
| decoding_name = GENERATION_MODES_MAPPING[generation_mode] | ||
| uses_default_decoding_loop = ( | ||
| "/" not in decoding_name | ||
| and decoding_method is getattr(GenerationMixin, decoding_name) | ||
| # Candidate generators slice `model_kwargs["attention_mask"]` themselves | ||
| and generation_mode != GenerationMode.ASSISTED_GENERATION | ||
| ) |
There was a problem hiding this comment.
i had the restrict a bit the fix since most methods still uses the attn masks but those can maybe be fixed
| model_kwargs["position_ids"] = self._prepare_position_ids_for_generation(inputs_tensor, model_kwargs) | ||
|
|
There was a problem hiding this comment.
we could delete the attention_mask even earlier but we will have to pass a new arg to _prepare_position_ids_for_generation : generation_config._mask_length so that we can create correctly the positions_ids. Note that we would have to update the models where _prepare_position_ids_for_generation is overwritten . Also some models do self.model.get_rope_index(inputs_tensor, **model_kwargs) where they use the attn mask
CI recapDashboard: View test results in Grafana |
|
closed in favor of #48814 ! |
Same as #47951 but i try to remove the
attention_maskfrommodel_kwargs.