Skip to content

Commit 3459e44

Browse files
authored
Merge branch 'main' into fix_ut_bugs
2 parents 9bbc74a + 9461bf9 commit 3459e44

File tree

1 file changed

+1
-1
lines changed

1 file changed

+1
-1
lines changed

auto_round/export/export_to_gguf/convert.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1149,7 +1149,7 @@ def get_vocab_base(self) -> tuple[list[str], list[int], str]:
11491149
toktypes: list[int] = []
11501150

11511151
from transformers import AutoTokenizer
1152-
tokenizer = AutoTokenizer.from_pretrained(self.dir_model)
1152+
tokenizer = AutoTokenizer.from_pretrained(self.dir_model, trust_remote_code=True)
11531153
vocab_size = self.hparams.get("vocab_size", len(tokenizer.vocab))
11541154
assert max(tokenizer.vocab.values()) < vocab_size
11551155

0 commit comments

Comments
 (0)