Repository navigation
Generation config boolean defaults #43000
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
bda4a76
d2da9b0
e59b321
ca46e33
70fdb01
a635307
91ae7d8
b40d249
3f07426
1aeb0ea
a6e71a8
1608238
6f68cc0
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
|
|
@@ -101,6 +101,10 @@ class GenerationConfig(PushToHubMixin): | |||||
|
|
||||||
| </Tip> | ||||||
|
|
||||||
| Note: the configuration field that are still `None` will be overriden by `GenerationConfig._get_default_generation_params()` | ||||||
| during the generation loop. If you want to use different values for these fields, make sure to explicitly set them in the | ||||||
| generation config. | ||||||
|
|
||||||
| Arg: | ||||||
| > Parameters that control the length of the output | ||||||
|
|
||||||
|
|
@@ -129,14 +133,14 @@ class GenerationConfig(PushToHubMixin): | |||||
|
|
||||||
| > Parameters that control the generation strategy used | ||||||
|
|
||||||
| do_sample (`bool`, defaults to `False`): | ||||||
| do_sample (`bool`): | ||||||
| Whether or not to use sampling ; use greedy decoding otherwise. | ||||||
| num_beams (`int`, *optional*): | ||||||
| Number of beams for beam search. 1 means no beam search. | ||||||
|
|
||||||
| > Parameters that control the cache | ||||||
|
|
||||||
| use_cache (`bool`, defaults to `True`): | ||||||
| use_cache (`bool`): | ||||||
| Whether or not the model should use the past last key/values attentions (if applicable to the model) to | ||||||
| speed up decoding. | ||||||
| cache_implementation (`str`, *optional*): | ||||||
|
|
@@ -206,7 +210,7 @@ class GenerationConfig(PushToHubMixin): | |||||
| bad_words_ids (`list[list[int]]`, *optional*): | ||||||
| List of list of token ids that are not allowed to be generated. Check | ||||||
| [`~generation.NoBadWordsLogitsProcessor`] for further documentation and examples. | ||||||
| renormalize_logits (`bool`, defaults to `False`): | ||||||
| renormalize_logits (`bool`): | ||||||
| Whether to renormalize the logits after applying all the logits processors (including the custom | ||||||
| ones). It's highly recommended to set this flag to `True` as the search algorithms suppose the score logits | ||||||
| are normalized but some logit processors break the normalization. | ||||||
|
|
@@ -217,7 +221,7 @@ class GenerationConfig(PushToHubMixin): | |||||
| forced_eos_token_id (`int` or list[int]`, *optional*, defaults to `model.config.forced_eos_token_id`): | ||||||
| The id of the token to force as the last generated token when `max_length` is reached. Optionally, use a | ||||||
| list to set multiple *end-of-sequence* tokens. | ||||||
| remove_invalid_values (`bool`, defaults to `model.config.remove_invalid_values`): | ||||||
| remove_invalid_values (`bool`): | ||||||
| Whether to remove possible *nan* and *inf* outputs of the model to prevent the generation method to crash. | ||||||
| Note that using `remove_invalid_values` can slow down generation. | ||||||
| exponential_decay_length_penalty (`tuple(int, float)`, *optional*): | ||||||
|
|
@@ -234,7 +238,7 @@ class GenerationConfig(PushToHubMixin): | |||||
| Dictionary that maps a sequence of tokens to its bias term. Positive biases increase the odds of the | ||||||
| sequence being selected, while negative biases do the opposite. Check | ||||||
| [`~generation.SequenceBiasLogitsProcessor`] for further documentation and examples. | ||||||
| token_healing (`bool`, defaults to `False`): | ||||||
| token_healing (`bool`): | ||||||
| Heal tail tokens of prompts by replacing them with their appropriate extensions. | ||||||
| This enhances the quality of completions for prompts affected by greedy tokenization bias. | ||||||
| guidance_scale (`float`, *optional*): | ||||||
|
|
@@ -248,20 +252,20 @@ class GenerationConfig(PushToHubMixin): | |||||
|
|
||||||
| > Parameters that define the output variables of generate | ||||||
|
|
||||||
| num_return_sequences (`int`, *optional*, defaults to 1): | ||||||
| num_return_sequences (`int`, *optional*): | ||||||
| The number of independently computed returned sequences for each element in the batch. | ||||||
| output_attentions (`bool`, defaults to `False`): | ||||||
| output_attentions (`bool`): | ||||||
| Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned | ||||||
| tensors for more details. | ||||||
| output_hidden_states (`bool`, defaults to `False`): | ||||||
| output_hidden_states (`bool`): | ||||||
| Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for | ||||||
| more details. | ||||||
| output_scores (`bool`, defaults to `False`): | ||||||
| output_scores (`bool`): | ||||||
| Whether or not to return the prediction scores. See `scores` under returned tensors for more details. | ||||||
| output_logits (`bool`, defaults to `False`): | ||||||
| output_logits (`bool`): | ||||||
| Whether or not to return the unprocessed prediction logit scores. See `logits` under returned tensors for | ||||||
| more details. | ||||||
| return_dict_in_generate (`bool`, defaults to `False`): | ||||||
| return_dict_in_generate (`bool`): | ||||||
| Whether or not to return a [`~utils.ModelOutput`], as opposed to returning exclusively the generated | ||||||
| sequence. This flag must be set to `True` to return the generation cache (when `use_cache` is `True`) | ||||||
| or optional outputs (see flags starting with `output_`) | ||||||
|
|
@@ -286,7 +290,7 @@ class GenerationConfig(PushToHubMixin): | |||||
| (e.g. multilingual models with different target languages in one batch) | ||||||
|
|
||||||
| > Generation parameters exclusive to assistant generation | ||||||
| is_assistant (`bool`, defaults to `False`): | ||||||
| is_assistant (`bool`): | ||||||
| Whether the model is an assistant (draft) model. | ||||||
| num_assistant_tokens (`int`, *optional*): | ||||||
| Defines the number of _speculative tokens_ that shall be generated by the assistant model before being | ||||||
|
|
@@ -327,7 +331,7 @@ class GenerationConfig(PushToHubMixin): | |||||
| compile_config (CompileConfig, *optional*): | ||||||
| If using a compilable cache, this controls how `generate` will `compile` the forward pass for faster | ||||||
| inference. | ||||||
| disable_compile (`bool`, defaults to `False`): | ||||||
| disable_compile (`bool`): | ||||||
| Whether to disable the automatic compilation of the forward pass. Automatic compilation happens when | ||||||
| specific criteria are met, including using a compilable cache. Please open an issue if you find the | ||||||
| need to use this flag. | ||||||
|
|
@@ -346,11 +350,11 @@ def __init__(self, **kwargs): | |||||
| self.stop_strings = kwargs.pop("stop_strings", None) | ||||||
|
|
||||||
| # Parameters that control the generation strategy used | ||||||
| self.do_sample = kwargs.pop("do_sample", False) | ||||||
| self.do_sample = kwargs.pop("do_sample", None) | ||||||
|
zucchini-nlp marked this conversation as resolved.
Resolved
|
||||||
| self.num_beams = kwargs.pop("num_beams", None) | ||||||
|
|
||||||
| # Parameters that control the cache | ||||||
| self.use_cache = kwargs.pop("use_cache", True) | ||||||
| self.use_cache = kwargs.pop("use_cache", None) | ||||||
| self.cache_implementation = kwargs.pop("cache_implementation", None) | ||||||
| self.cache_config = kwargs.pop("cache_config", None) | ||||||
|
|
||||||
|
|
@@ -368,28 +372,28 @@ def __init__(self, **kwargs): | |||||
| self.length_penalty = kwargs.pop("length_penalty", None) | ||||||
| self.no_repeat_ngram_size = kwargs.pop("no_repeat_ngram_size", None) | ||||||
| self.bad_words_ids = kwargs.pop("bad_words_ids", None) | ||||||
| self.renormalize_logits = kwargs.pop("renormalize_logits", False) | ||||||
| self.renormalize_logits = kwargs.pop("renormalize_logits", None) | ||||||
| self.forced_bos_token_id = kwargs.pop("forced_bos_token_id", None) | ||||||
| self.forced_eos_token_id = kwargs.pop("forced_eos_token_id", None) | ||||||
| self.remove_invalid_values = kwargs.pop("remove_invalid_values", False) | ||||||
| self.remove_invalid_values = kwargs.pop("remove_invalid_values", None) | ||||||
| self.exponential_decay_length_penalty = kwargs.pop("exponential_decay_length_penalty", None) | ||||||
| self.suppress_tokens = kwargs.pop("suppress_tokens", None) | ||||||
| self.begin_suppress_tokens = kwargs.pop("begin_suppress_tokens", None) | ||||||
| self.sequence_bias = kwargs.pop("sequence_bias", None) | ||||||
| self.token_healing = kwargs.pop("token_healing", False) | ||||||
| self.token_healing = kwargs.pop("token_healing", None) | ||||||
| self.guidance_scale = kwargs.pop("guidance_scale", None) | ||||||
|
|
||||||
| self.watermarking_config = kwargs.pop("watermarking_config", None) | ||||||
| if isinstance(self.watermarking_config, dict): | ||||||
| self.watermarking_config = WatermarkingConfig.from_dict(self.watermarking_config) | ||||||
|
|
||||||
| # Parameters that define the output variables of `generate` | ||||||
| self.num_return_sequences = kwargs.pop("num_return_sequences", 1) | ||||||
| self.output_attentions = kwargs.pop("output_attentions", False) | ||||||
| self.output_hidden_states = kwargs.pop("output_hidden_states", False) | ||||||
| self.output_scores = kwargs.pop("output_scores", False) | ||||||
| self.output_logits = kwargs.pop("output_logits", False) | ||||||
| self.return_dict_in_generate = kwargs.pop("return_dict_in_generate", False) | ||||||
| self.num_return_sequences = kwargs.pop("num_return_sequences", None) | ||||||
| self.output_attentions = kwargs.pop("output_attentions", None) | ||||||
| self.output_hidden_states = kwargs.pop("output_hidden_states", None) | ||||||
| self.output_scores = kwargs.pop("output_scores", None) | ||||||
| self.output_logits = kwargs.pop("output_logits", None) | ||||||
| self.return_dict_in_generate = kwargs.pop("return_dict_in_generate", None) | ||||||
|
Comment on lines
-387
to
+396
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Sorry, I'm a bit lost into the motivation the PR and making those values to None (which is usually not a super good idea if they are boolean by essence) - could you give me a bit of context on the PR?
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The motivation was to fix a ton of edge cases when creating a generation config. In the past we had ugly workaround to allow users to change a generation param back to default if different value was saved on the hub (for ex #42762). That was fixed in #42702 and I deleted a lot of code, and defined a clear priority list, Next step will be to disallow users to pass both: a generation config and kwargs. Instead ask to put all kwargs in config. The below is accepted rn and is a bad practice imo conf = GenerationConfig(do_sample=False)
text = model.generate(input_ids, generation_config=conf, max_new_tokens=128, temperature=1.6) |
||||||
|
|
||||||
| # Special tokens that can be used at generation time | ||||||
| self.pad_token_id = kwargs.pop("pad_token_id", None) | ||||||
|
|
@@ -401,7 +405,7 @@ def __init__(self, **kwargs): | |||||
| self.decoder_start_token_id = kwargs.pop("decoder_start_token_id", None) | ||||||
|
|
||||||
| # Assistant generation | ||||||
| self.is_assistant = kwargs.pop("is_assistant", False) | ||||||
| self.is_assistant = kwargs.pop("is_assistant", None) | ||||||
| self.num_assistant_tokens = kwargs.pop("num_assistant_tokens", None) | ||||||
| self.num_assistant_tokens_schedule = kwargs.pop("num_assistant_tokens_schedule", None) | ||||||
| self.assistant_confidence_threshold = kwargs.pop("assistant_confidence_threshold", None) | ||||||
|
|
@@ -413,7 +417,7 @@ def __init__(self, **kwargs): | |||||
|
|
||||||
| # Performance | ||||||
| self.compile_config = kwargs.pop("compile_config", None) | ||||||
| self.disable_compile = kwargs.pop("disable_compile", False) | ||||||
| self.disable_compile = kwargs.pop("disable_compile", None) | ||||||
|
|
||||||
| # Deprecated (moved to the Hub). TODO remove for v5 | ||||||
| self.low_memory = kwargs.pop("low_memory", None) | ||||||
|
|
@@ -483,7 +487,7 @@ def get_generation_mode(self, assistant_model: Optional["PreTrainedModel"] = Non | |||||
| if self.constraints is not None or self.force_words_ids is not None: | ||||||
| generation_mode = GenerationMode.CONSTRAINED_BEAM_SEARCH | ||||||
| elif self.num_beams is None or self.num_beams == 1: | ||||||
| if not self.do_sample: | ||||||
| if self.do_sample is not True: | ||||||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
super nit, seen this multiple times
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Actually, it should be
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Can we add a small comment then, that's such a weird case lol. |
||||||
| if ( | ||||||
| self.top_k is not None | ||||||
| and self.top_k > 1 | ||||||
|
|
@@ -498,7 +502,7 @@ def get_generation_mode(self, assistant_model: Optional["PreTrainedModel"] = Non | |||||
| else: | ||||||
| if self.num_beam_groups is not None and self.num_beam_groups > 1: | ||||||
| generation_mode = GenerationMode.GROUP_BEAM_SEARCH | ||||||
| elif self.do_sample: | ||||||
| elif self.do_sample is True: | ||||||
| generation_mode = GenerationMode.BEAM_SAMPLE | ||||||
| else: | ||||||
| generation_mode = GenerationMode.BEAM_SEARCH | ||||||
|
|
@@ -537,6 +541,7 @@ def _get_default_generation_params() -> dict[str, Any]: | |||||
| "max_length": 20, | ||||||
| "min_length": 0, | ||||||
| "do_sample": False, | ||||||
| "use_cache": True, | ||||||
|
zucchini-nlp marked this conversation as resolved.
Resolved
|
||||||
| "early_stopping": False, | ||||||
| "num_beams": 1, | ||||||
| "temperature": 1.0, | ||||||
|
|
@@ -616,7 +621,10 @@ def validate(self, strict=False): | |||||
|
|
||||||
| # 2. Validation of attribute combinations | ||||||
| # 2.1. detect sampling-only parameterization when not in sampling mode | ||||||
| if not self.do_sample: | ||||||
|
|
||||||
| # Note that we check `is not True` in purpose. Boolean fields can also be `None` so we | ||||||
| # have to be explicit. Value of `None` is same as having `False`, i.e. the default value | ||||||
| if self.do_sample is not True: | ||||||
| greedy_wrong_parameter_msg = ( | ||||||
| "`do_sample` is set not to set `True`. However, `{flag_name}` is set to `{flag_value}` -- this flag is only " | ||||||
| "used in sample-based generation modes. You should set `do_sample=True` or unset `{flag_name}`." | ||||||
|
|
@@ -662,21 +670,25 @@ def validate(self, strict=False): | |||||
| ) | ||||||
|
|
||||||
| # 2.4. check `num_return_sequences` | ||||||
| if self.num_return_sequences > 1: | ||||||
| if self.num_return_sequences is not None and self.num_return_sequences > 1: | ||||||
| if self.num_beams is None or self.num_beams == 1: | ||||||
| if not self.do_sample: | ||||||
| raise ValueError( | ||||||
| "Greedy methods (do_sample != True) without beam search do not support " | ||||||
| f"`num_return_sequences` different than 1 (got {self.num_return_sequences})." | ||||||
| ) | ||||||
| elif self.num_beams is not None and self.num_return_sequences > self.num_beams: | ||||||
| elif ( | ||||||
| self.num_beams is not None | ||||||
| and self.num_return_sequences is not None | ||||||
| and self.num_return_sequences > self.num_beams | ||||||
| ): | ||||||
| raise ValueError( | ||||||
| f"`num_return_sequences` ({self.num_return_sequences}) has to be smaller or equal to `num_beams` " | ||||||
| f"({self.num_beams})." | ||||||
| ) | ||||||
|
|
||||||
| # 2.5. check cache-related arguments | ||||||
| if not self.use_cache: | ||||||
| if self.use_cache is not True: | ||||||
| # In this case, all cache-related arguments should be unset. However, since `use_cache=False` is often used | ||||||
| # passed to `generate` directly to hot-fix cache issues, let's raise a warning instead of an error | ||||||
| # (otherwise a user might need to overwrite several parameters). | ||||||
|
|
@@ -691,9 +703,9 @@ def validate(self, strict=False): | |||||
| ) | ||||||
|
|
||||||
| # 2.6. other incorrect combinations | ||||||
| if not self.return_dict_in_generate: | ||||||
| if self.return_dict_in_generate is not True: | ||||||
| for extra_output_flag in self.extra_output_flags: | ||||||
| if getattr(self, extra_output_flag): | ||||||
| if getattr(self, extra_output_flag) is True: | ||||||
| minor_issues[extra_output_flag] = ( | ||||||
| f"`return_dict_in_generate` is NOT set to `True`, but `{extra_output_flag}` is. When " | ||||||
| f"`return_dict_in_generate` is not `True`, `{extra_output_flag}` is ignored." | ||||||
|
|
||||||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Are these changes really relevant? Seems unrelated to me
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. some tests were failing without it because the re-loaded model has a incorrect config type. And we couldn't find generation keys in it
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Ok, then let's keep it, not wrong was just surprised |
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
btw, could we remove
use_cacheas a key from config and instead keep it only ingeneration_config? Imo it isn't really a model attributeThere was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hmm unsure. I understand the point but that would be hugely breaking for a lot of users and external libraries - on the other hand we could include this for v5 if you feel motivated 🤔
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
it will definitely break stuff 😿 Just a thought for now
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
so, this is a workaround we have to not throw errors that
model config contains generation params. We won't throw errors anyway ifuse_cacheis in model config signature but unfortunately many don't have it documentedThere was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Gotcha, maybe at some point we can 👀