🚨 Generation config defaults are now `None` by zucchini-nlp · Pull Request #42702 · huggingface/transformers

zucchini-nlp · 2025-12-08T14:09:03Z

What does this PR do?

As per title, must have been done long time ago but could't break BC. The current impl breaks BC only half-way, i.e. the generation loop is not affected and will keep using the old defaults. The biggest difference is for users to init, access, modify, etc. the model's generation config directly:

# `0` before this PR, `None` after the PR
print(model.generation_config.no_repeat_ngram_size)

HuggingFaceDocBuilderDev · 2025-12-08T14:28:11Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

zucchini-nlp · 2025-12-10T12:34:59Z

+        generation_params = {}
+        default_config = self.__class__().to_dict() if not self.has_no_defaults_at_init else {}
+        for key in GenerationConfig._get_default_generation_params().keys():
+            if hasattr(self, key) and getattr(self, key) is not None and key not in default_config:
+                generation_params[key] = getattr(self, key)
+


could have been simplified because we no longer have any generation params in model.config

Just for my understanding, generation config is top level either way. Hence, we no longer need to discern for submodels etc in composite models (which could have possibly different config values here)

yeah, that and also because these lines are more of a workaround for old models (e.g. bart). New models don't have any generation params in model config anyway, we don't allow it for quite a long time

Gotcha, makes sense to me

zucchini-nlp · 2025-12-10T12:39:34Z

Ready for review!

vasqu

Let's still add an 🚨 even if it's not completely breaking, I rather be safe than sorry here. We never know

First round of comments, my biggest issue would be the kwargs vs generation config passing. But you also left a note there.

vasqu · 2025-12-10T15:45:26Z

+        generation_params = {}
+        default_config = self.__class__().to_dict() if not self.has_no_defaults_at_init else {}
+        for key in GenerationConfig._get_default_generation_params().keys():
+            if hasattr(self, key) and getattr(self, key) is not None and key not in default_config:
+                generation_params[key] = getattr(self, key)
+


Just for my understanding, generation config is top level either way. Hence, we no longer need to discern for submodels etc in composite models (which could have possibly different config values here)

Co-authored-by: Anton Vlasjuk <73884904+vasqu@users.noreply.github.com>

albertvillanova · 2025-12-15T12:03:41Z

Hi,

First of all, thanks for your reactivity and addressing the underlying issue:

Generation config merging logic prevents explicit override of model-specific defaults back to global defaults #42762

On the other hand, I did a quick pass over the generation code, and I think there is a subtle semantic point worth double-checking: None is not always just "unset", it can also mean "disable this behavior".

Concretely, top_k=None disables top-k filtering entirely and allows all tokens. Because of that, we now have two different concepts that can both be represented as "None":

Use the global/default value for top_k (e.g. 50)
Explicitly disable top-k filtering (top_k=None or 0) even if the model's generation config or the global defaults say otherwise

Therefore, a situation can occur during training where:

the user-provided generation_config has top_k=None (intending to disable top-k filtering),
the model’s own generation_config has a non-None value for top_k,
merging logic currently preserves the model’s value instead of respecting the explicit None.

In that scenario, None is not just "unset"; it is a meaningful instruction ("don't apply top-k filtering"). If so, the merge semantics may need refinement to avoid unintentionally re-enabling filtering (e.g. by using a sentinel value instead).

I'll continue digging into this, but flagging it early so we can discuss it before merging.

zucchini-nlp · 2025-12-15T12:41:27Z

@albertvillanova I think if top_k=None in the generation config it is the same as if users did not pass any top_k in kwargs. A value is not set to any value (None) does not specifically mean that the users is requesting to not use it, so the users would need to explicitly unset as top_k=0 if model has saved a different value.

Unfortunately we have no way to 100% know what users wants when they set values to None in current code. The only way would be for us to not update generation config with model's defaults if users pass my_generation_config. But that will be much more breaking and will require users to always create a custom config from model.generation_config

For ex, if everyone prepared custom configs as below, we can fix your issue. I'm afraid it's not the case for most users

my_generation_config = model.generation_config
my_generation_config.top_k = None
model.generate(inputs, generation_config=my_generation_config)

github-actions · 2025-12-17T14:15:19Z

[For maintainers] Suggested jobs to run (before merge)

run-slow: bart, csm, dia, encoder_decoder, musicgen, rag, reformer, speech_encoder_decoder, vision_encoder_decoder, whisper

vasqu · 2025-12-17T16:51:16Z

I think we can merge? @zucchini-nlp

ebezzam · 2025-12-19T16:56:45Z

-                modified_values = {}
-                global_default_generation_config = GenerationConfig()
-                model_generation_config = self.generation_config
-                # we iterate over the model's generation config: it may hold custom keys, which we'll want to copy


@vasqu, @zucchini-nlp are custom keys no longer copied over?

Seems like only the ones here are defaulted at this line, which wouldn't copy custom keys in self.generation_config anymore

ebezzam · 2025-12-19T17:30:46Z

+        global_defaults = self.generation_config._get_default_generation_params()
+        generation_config.update(**self.generation_config.to_dict(), defaults_only=True)
+        generation_config.update(**global_defaults, defaults_only=True)
+


something like this just after?

# add custom keys not in global defaults for key, value in self.generation_config.to_dict().items(): if not hasattr(generation_config, key): setattr(generation_config, key, value)

* this way betetr maybe? * delete legacy from bart and mvp * import not found * fix some tests * fix more tests * revert smth to run tests again * i though I fixed it already, but there were more models * commit and check tests, clean-up later * assisted deocding shoudl work now * docs and whisper * fix a few more tests * no circular import errors pls * wording * add a test for defaults following TRL example * nit * Update src/transformers/configuration_utils.py Co-authored-by: Anton Vlasjuk <73884904+vasqu@users.noreply.github.com> * Update src/transformers/generation/candidate_generator.py Co-authored-by: Anton Vlasjuk <73884904+vasqu@users.noreply.github.com> * Update src/transformers/generation/utils.py Co-authored-by: Anton Vlasjuk <73884904+vasqu@users.noreply.github.com> * comments * final fix tests * more comments --------- Co-authored-by: Anton Vlasjuk <73884904+vasqu@users.noreply.github.com>

…r ConfidenceCriteria PR #42702 introduced `self.assistant_generation_config` (a deepcopy with defaults applied) to hold the assistant model's config, but the adaptive threshold update (`update_candidate_strategy`) was only syncing the new best threshold to `self.assistant_generation_config.assistant_confidence_threshold`, not to `self.generation_config.assistant_confidence_threshold`. `ConfidenceCriteria` is constructed each round from `self.generation_config`, so it was always using the initial 0.4 threshold regardless of what the ROC-curve adaptation computed. This caused the assistant to always generate the maximum number of speculative tokens per round, making assisted generation slower than non-assisted on the distil-whisper test. Fix: also update `self.generation_config.assistant_confidence_threshold` whenever the adaptive threshold is updated, so each new round's ConfidenceCriteria reflects the current best threshold. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

…on, and speed regression (#48000) * [Whisper] Fix cache type and layer count for speculative decoding with WhisperForCausalLM Fix 1 — is_updated UnboundLocalError (WhisperAttention.forward) PR #39956 wrapped the `is_updated` assignment in an isinstance guard but left it consumed unconditionally below. The bug was dormant while Whisper always received EncoderDecoderCache. PR #43679 changed generate() to pre-initialize a plain DynamicCache for is_encoder_decoder=False models, activating the latent bug. Fix: initialize is_updated=False before the guard, identical to PR #40517 which fixed 55 other models. Fix 2a — Wrong cache type: DynamicCache instead of EncoderDecoderCache (WhisperDecoder.forward) generate() uses config.is_encoder_decoder to pick the cache type. WhisperForCausalLM sets is_encoder_decoder=False, so PR #43679 made generate() pre-initialize a plain DynamicCache. But WhisperForCausalLM uses cross-attention: self- and cross-attention share the same layer_idx slots in DynamicCache and overwrite each other. Fix: when generate() passes an empty DynamicCache and encoder_hidden_states are present, convert it to EncoderDecoderCache before the first forward pass. Fix 2b — Wrong layer count: 32 slots pre-allocated for a 2-layer decoder (WhisperDecoder.forward) DynamicCache(config=self.config) calls config.get_text_config(decoder=True), which for WhisperConfig returns self unchanged (flat config, no nested decoder sub-config). It then reads config.num_hidden_layers, which WhisperConfig maps via attribute_map to encoder_layers=32. For distil-whisper-large-v2, encoder_layers=32 but decoder_layers=2, so 32 cache slots are pre-allocated for a 2-layer decoder. Slots 2-31 are uninitialized and crash on crop(-n). Fix: deepcopy the config and override num_hidden_layers with decoder_layers before passing to DynamicCache. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * [Whisper] Restore absolute cache crop in _update_past_and_masks for speculative decoding Fix 3 — Cache size accumulation (candidate_generator.py) Commit 5ec7b73 ("crop with negative always") changed _update_past_and_masks to always remove a fixed number of tokens (crop(-1)) instead of cropping to an absolute target. When multiple speculative candidates are rejected in a round, the cache grows by several tokens but only 1 is trimmed. Over successive rounds, cache.get_seq_length() exceeds input_ids.shape[1]. _prefill then computes next_sequence_length = input_ids.shape[1] - cache.get_seq_length() which goes negative, causing input_ids[:, -n:] to produce a [1, 0] empty tensor that crashes in WhisperAttention.forward() at the reshape. Verified by bisect: at the parent of 5ec7b73 (68075f8), fixes 1+2a+2b alone are sufficient (speed assertion only). At 5ec7b73 with only fixes 1+2a+2b the reshape crash reproduces. Adding this fix restores the test to speed-assertion-only. Fix: compute the absolute target size from input_ids and only crop if the cache exceeds it, using a negative delta to comply with the current Cache.crop(negative) API. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * [Generate] Sync adaptive confidence threshold to generation_config for ConfidenceCriteria PR #42702 introduced `self.assistant_generation_config` (a deepcopy with defaults applied) to hold the assistant model's config, but the adaptive threshold update (`update_candidate_strategy`) was only syncing the new best threshold to `self.assistant_generation_config.assistant_confidence_threshold`, not to `self.generation_config.assistant_confidence_threshold`. `ConfidenceCriteria` is constructed each round from `self.generation_config`, so it was always using the initial 0.4 threshold regardless of what the ROC-curve adaptation computed. This caused the assistant to always generate the maximum number of speculative tokens per round, making assisted generation slower than non-assisted on the distil-whisper test. Fix: also update `self.generation_config.assistant_confidence_threshold` whenever the adaptive threshold is updated, so each new round's ConfidenceCriteria reflects the current best threshold. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * [Whisper] Format: wrap long EncoderDecoderCache line Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * [AudioFlamingo3] Fix is_updated UnboundLocalError in cross-attention (sync with modular) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --------- Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>

…through super().generate() (#48108) * [Whisper] Fix speculative decoding regression from #42702: preserve cleared suppress tokens PR #42702 changed _prepare_generation_config to always merge self.generation_config values with defaults_only=True. This broke Whisper speculative decoding because: 1. Whisper's generate() override calls _prepare_generation_config, then processes suppress_tokens/begin_suppress_tokens in _retrieve_logit_processors() (creating the logits processors and clearing both fields to None on generation_config). For speculative decoding, begin_suppress_tokens is also explicitly cleared to allow the assistant model to emit EOS. 2. Whisper then calls super().generate(generation_config=generation_config, ...). 3. Inside super().generate(), _prepare_generation_config is called again. Since suppress_tokens and begin_suppress_tokens are now None (== global default), defaults_only=True restores them from the model's generation_config.json. 4. _get_logits_processor then creates a SuppressTokensAtBeginLogitsProcessor that suppresses EOS (token 50257) at position 0 in the assistant model, preventing it from stopping — causing hallucinated extra tokens ("Thank you."). Fix: pass suppress_tokens and begin_suppress_tokens explicitly to super().generate() so the final update(**kwargs) step (no defaults_only) pins them at None, preserving the state set by _retrieve_logit_processors(). Fixes test_speculative_decoding_non_distil regression introduced by #42702. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * [Whisper] Add PR reference comment to speculative decoding workaround Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --------- Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>

zucchini-nlp added 4 commits November 18, 2025 18:22

this way betetr maybe?

c325bb7

delete legacy from bart and mvp

da36ae1

merge main

060f957

import not found

6a6776d

zucchini-nlp added 5 commits December 8, 2025 16:10

fix some tests

dda23da

fix more tests

986271d

revert smth to run tests again

1b82af6

i though I fixed it already, but there were more models

03db18f

commit and check tests, clean-up later

95f6dea

zucchini-nlp mentioned this pull request Dec 10, 2025

Generation config merging logic prevents explicit override of model-specific defaults back to global defaults #42762

Closed

zucchini-nlp added 6 commits December 10, 2025 11:35

assisted deocding shoudl work now

c2cb44c

docs and whisper

c4c8e88

fix a few more tests

cebbb4d

no circular import errors pls

58fcdb0

wording

2842e84

add a test for defaults following TRL example

ce263f2