Skip to content

Commit f356925

Browse files
yonigozlanleaderofARS
authored andcommitted
Fix saving multiple tokenizers for custom processors (huggingface#42630)
fix saving multiple tokenizers for custom processors
1 parent a7915ce commit f356925

1 file changed

Lines changed: 4 additions & 8 deletions

File tree

‎src/transformers/processing_utils.py‎

Lines changed: 4 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -696,14 +696,10 @@ def to_dict(self) -> dict[str, Any]:
696696
# extra attributes to be kept
697697
attrs_to_save += ["auto_map"]
698698

699-
if "tokenizer" in output:
700-
del output["tokenizer"]
701-
if "qformer_tokenizer" in output:
702-
del output["qformer_tokenizer"]
703-
if "protein_tokenizer" in output:
704-
del output["protein_tokenizer"]
705-
if "char_tokenizer" in output:
706-
del output["char_tokenizer"]
699+
for attribute in self.__class__.get_attributes():
700+
if "tokenizer" in attribute and attribute in output:
701+
del output[attribute]
702+
707703
if "chat_template" in output:
708704
del output["chat_template"]
709705

0 commit comments

Comments
 (0)