Skip to content

Generate: consistently handle special tokens as tensors - #30624

Merged
gante merged 9 commits into
huggingface:mainfrom
gante:special_tokens
May 9, 2024
Merged

gante merged 9 commits into
huggingface:mainfrom
gante:special_tokens

Conversation

@gante

@gante gante commented May 2, 2024 •

Copy link
Copy Markdown
Contributor

What does this PR do?

(reopened from #29788, requirements were merged and made this PR simpler)

To enable torch.compile with generate, some special token-related operations have to be rewritten into torch operations. That requires special tokens to be tensors instead of integers or a list of integers. (See #29374 for a working prototype)

This PR reworks special token usage in generate to consistently treat them as a tensor, as opposed to e.g. keeping track of eos_token_id in integer and in tensor form.

👉 Review suggestion: start by reading _prepare_special_tokens and how it fits in generate.


Tests ran locally:

  • logits processors doctests (pytest --doctest-modules src/transformers/generation/logits_process.py -vv)
  • generate doctests (pytest --doctest-modules src/transformers/generation/utils.py -vv)
  • generate integration tests (RUN_SLOW=1 py.test tests/generation/ -vv)
  • cache integration tests (RUN_SLOW=1 py.test tests/test_cache_utils.py -vv)
  • llama slow tests (RUN_SLOW=1 py.test tests/models/llama/test_modeling_llama.py -vv)
  • whisper slow tests (RUN_SLOW=1 py.test tests/models/whisper/test_modeling_whisper.py -vv) -- same failures as in main

@HuggingFaceDocBuilderDev

Copy link
Copy Markdown

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@gante
gante force-pushed the special_tokens branch from f035f0b to 0a659be Compare May 3, 2024 11:47
@gante
gante requested a review from ArthurZucker May 3, 2024 12:07
@gante

gante commented May 3, 2024

Copy link
Copy Markdown
Contributor Author

cc @zucchini-nlp this one is the same as #29788, which you've already reviewed in its early state. Note that because we've merged other PRs first (e.g. removing the decoding functions from the public API), the diff is much smaller 💛

@ArthurZucker ArthurZucker left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall good. Don't think we should warn but error out, and maybe update the serialize / de-serialize?

Comment thread src/transformers/generation/utils.py Resolved
Comment thread src/transformers/generation/utils.py Resolved
Comment thread src/transformers/generation/utils.py Resolved
Comment thread src/transformers/generation/utils.py Resolved
@gante
gante force-pushed the special_tokens branch from d167e18 to f2c4d74 Compare May 6, 2024 12:37
@gante
gante force-pushed the special_tokens branch from f2c4d74 to 0dcaa3d Compare May 9, 2024 09:27
gante and others added 2 commits May 9, 2024 10:10
Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>
@gante
gante requested a review from ArthurZucker May 9, 2024 10:11
@gante

gante commented May 9, 2024 •

Copy link
Copy Markdown
Contributor Author

@ArthurZucker addressed your comments :D

(let's see this fast CI going brr)

@ArthurZucker ArthurZucker left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good thanks for updating

Comment on lines 420 to +421
torch.zeros(input_ids.shape[:2], dtype=torch.int64, layout=input_ids.layout, device=input_ids.device)
+ model._get_decoder_start_token_id()
+ generation_config.decoder_start_token_id

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we have to do the + and not just use the decoder start toekn id?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the + here makes it a tensor of all decoder_start_token_id (as opposed to concatenation).

@gante
gante merged commit 7130a22 into huggingface:main May 9, 2024
@gante
gante deleted the special_tokens branch May 9, 2024 17:02
eginhard added a commit to idiap/coqui-ai-TTS that referenced this pull request Jun 16, 2024
….41.1

Fixes #31. The handling of special tokens in `transformers` was changed in
huggingface/transformers#30624 and
huggingface/transformers#30746. This updates the XTTS
streaming code accordingly.
eginhard added a commit to idiap/coqui-ai-TTS that referenced this pull request Jun 16, 2024
….41.1

Fixes #31. The handling of special tokens in `transformers` was changed in
huggingface/transformers#30624 and
huggingface/transformers#30746. This updates the XTTS
streaming code accordingly.
eginhard added a commit to idiap/coqui-ai-TTS that referenced this pull request Jun 17, 2024
….41.1

Fixes #31. The handling of special tokens in `transformers` was changed in
huggingface/transformers#30624 and
huggingface/transformers#30746. This updates the XTTS
streaming code accordingly.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants