Skip to content

Fix deepcopy in ProcessorMixin.to_dict for GemmaTokenizerFast - #41851

Closed
aijadugar wants to merge 17 commits into
huggingface:mainfrom
aijadugar:fix-multiple-tokenizers-saved-v2
Closed

aijadugar wants to merge 17 commits into
huggingface:mainfrom
aijadugar:fix-multiple-tokenizers-saved-v2

Conversation

@aijadugar

Copy link
Copy Markdown

Description:

Replaced deepcopy with a shallow copy to avoid AttributeError with fast tokenizers.

Added a test (tests/utils/test_processor_utils.py) to verify multiple tokenizers save/load correctly.

Fix #41837

@ArthurZucker

Copy link
Copy Markdown
Collaborator

Thanks @aijadugar for the fix! ProcessorMixin.to_dict was since reworked in #44894 to exclude tokenizer attributes before deep-copying, which avoids the fast-tokenizer deepcopy issue from #41837. Closing as this is now covered on main; please open a new issue if you still see the error.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants