Repository navigation

Fix deepcopy in ProcessorMixin.to_dict for GemmaTokenizerFast - #41851

Closed

aijadugar wants to merge 17 commits into

huggingface:mainfrom

aijadugar:fix-multiple-tokenizers-saved-v2

Closed

Fix deepcopy in ProcessorMixin.to_dict for GemmaTokenizerFast#41851
aijadugar wants to merge 17 commits into
huggingface:mainfrom
aijadugar:fix-multiple-tokenizers-saved-v2

Commits on Oct 24, 2025
- multiple tokenizers with different filenames can save now
  aijadugar
  committed
  73f91c5
- shallow copy to avoid deepcopy errors
  aijadugar
  committed
  c2ab93f
- Fix formatting issues in processing_utils.py removes whitespaces
  aijadugar
  authored
  2e90fa5
Commits on Oct 25, 2025
Commits on Oct 26, 2025
- Revert unrelated edits and keep only relevant processor save logic
  aijadugar
  authored
  67c3bb7
- Merge branch 'main' into fix-multiple-tokenizers-saved
  aijadugar
  authored
  8aca431
- Restore original comments and docstrings; keep only relevant processor save logic
  aijadugar
  authored
  560067e
- Restore original comments and keep only relevant processor save logic
  aijadugar
  authored
  68caa7d
- Restore original comments
  aijadugar
  authored
  4b2c049
- Restore original comments
  aijadugar
  authored
  9e4b141
- removes trailling whitespaces
  aijadugar
  authored
  1ffb4d3
- import like ruff's isort rule
  aijadugar
  authored
  8203b0e
Commits on Oct 27, 2025
- Merge branch 'fix-multiple-tokenizers-saved' of https://github.com/aijadugar/transformers into fix-multiple-tokenizers-saved-v2
  aijadugar
  committed
  a0c3fe2
- Fix processor save/load for multiple tokenizer attributes
  aijadugar
  committed
  17f81b0