[User] Producing tokenizer.model from transformers tokenizers.json

[BaiqingL]

I have a Llama 2 7b model fine tuned for a downstream task and stored in transformers format, i.e. my model file structure looks like this:

-a---           7/28/2023  4:30 PM            623 config.json
-a---           7/28/2023  4:30 PM            160 generation_config.json
-a---           7/27/2023  4:00 AM     9976672446 pytorch_model-00001-of-00002.bin
-a---           7/27/2023  4:00 AM     3500355411 pytorch_model-00002-of-00002.bin
-a---           7/27/2023  4:00 AM          26788 pytorch_model.bin.index.json
-a---           7/27/2023  4:00 AM            576 special_tokens_map.json
-a---           7/27/2023  4:00 AM            698 tokenizer_config.json
-a---           7/27/2023  4:00 AM        1843709 tokenizer.json
-a---           7/27/2023  4:00 AM           6011 training_args.bin

I know the convert.py file expects the original Llama 2 structure, how would I modify it to make this work? I'm not too sure what the tokenizer.model file format is like, or how to convert the tokenizer.json file into it.

[BaiqingL]

Also, just curious if anyone has compared latency between inferencing using this project compared to a simple transformers text-generation pipeline?

[klosax]

I guess you could make it work by copying the original tokenizer.model to the folder.

[BaiqingL]

Hey @klosax I added some special tokens for the downstream task so I don't think I can do that unfortunately

[klosax]

It looks like a solution was added in PR #2228

You could try something like:
python convert.py <model-dir> --vocab-only --outfile <model-dir>/tokenizer.model --vocabtype bpe

[akramIOT]

raise FileNotFoundError(f"Could not find any of {[self._FILES[vt] for vt in vocab_types]}")
FileNotFoundError: Could not find any of ['tokenizer.model', 'tokenizer.json']

[jferments]

I am running into this same issue trying to convert llama 70B into GGUF ... there is no tokenizer.model file, only tokenizer.json. When I run python3 convert-hf-to-gguf.py --outfile ~/code/models/llm/llama-3-70B/llama-3-70B.gguf --outtype f16 ~/code/models/llm/llama-3-70B/, I get the following error: raise FileNotFoundError('Cannot find Llama BPE tokenizer')