LLaMATokenizerFast works abnormally

[jiangwangyi]

System Info

platform==Ubuntu18.04
python==3.10
transformers==4.29.2

Who can help?

@ArthurZucker

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

</s> is the special token of LLaMATokenizer(Fast), it is expected that </s> can be recognized as a single token when encoding the text. However, it can be shown that the two tokenizers behave differently:

>>> t1 = transformers.AutoTokenizer.from_pretrained("huggyllama/llama-7b", use_fast=True)
>>> t2 = transformers.AutoTokenizer.from_pretrained("huggyllama/llama-7b", use_fast=False)
>>> text = "I love you.</s>"
>>> t1(text)
>>> {'input_ids': [1, 306, 5360, 366, 21106, 29879, 29958], 'token_type_ids': [0, 0, 0, 0, 0, 0, 0], 'attention_mask': [1, 1, 1, 1, 1, 1, 1]}
>>> t2(text)
>>> {'input_ids': [1, 306, 5360, 366, 29889, 2], 'attention_mask': [1, 1, 1, 1, 1, 1]}

also, LLaMATokenizerFast returns token_type_ids but LLaMATokenizer does not.

Expected behavior

LLaMATokenizerFast to be consistent with LLaMATokenzier.

[NielsRogge]

Also have 2 questions related to LlamaTokenizerFast:

First, loading a fast tokenizer from a saved slow one takes very long:

from transformers import LlamaTokenizer, LlamaTokenizerFast

tokenizer = LlamaTokenizer.from_pretrained("huggyllama/llama-7b")
tokenizer.save_pretrained(".")

# the following line takes > 1 min
fast_tokenizer = LlamaTokenizerFast.from_pretrained(".")

This is not the case for other tokenizers like BertTokenizerFast.

Second, for a new model I'm working on (#23460) I wonder how to get the same behaviour between slow and fast tokenizers for the following:

from transformers import LlamaTokenizer, LlamaTokenizerFast

tokenizer = LlamaTokenizer.from_pretrained("huggyllama/llama-7b", truncation_side="left")
tokenizer.add_special_tokens({"pad_token": "[PAD]"})
tokenizer.add_special_tokens({"bos_token": "</s>"})
tokenizer.add_special_tokens({"eos_token": "</s>"})
tokenizer.add_special_tokens({"unk_token": "</s>"})

fast_tokenizer = LlamaTokenizerFast.from_pretrained("huggyllama/llama-7b", truncation_side="left")
fast_tokenizer.add_special_tokens({"pad_token": "[PAD]"})
fast_tokenizer.add_special_tokens({"bos_token": "</s>"})
fast_tokenizer.add_special_tokens({"eos_token": "</s>"})
fast_tokenizer.add_special_tokens({"unk_token": "</s>"})

prompt = "What is unusual about this image?"

encoding = tokenizer(prompt, return_tensors="pt")

fast_encoding = fast_tokenizer(prompt, return_tensors="pt")

for k,v in encoding.items():
    assert torch.allclose(fast_encoding[k], v)

=> this assertion fails since the input_ids differ:

tensor([[    2,  1724,   338, 22910,  1048,   445,  1967, 29973]])
tensor([[    1,  1724,   338, 22910,  1048,   445,  1967, 29973]])

[NielsRogge]

cc'ing @ArthurZucker and @Narsil here

[ArthurZucker]

Hey! Thanks for opening this issue.

• return_token_type_ids should be set to None by default but is updated with "token_type_ids" in self.model_input_names. This is specific to the fast tokenizer, and is a known difference. I am not sure why this was added only in the fast tokenizer but it's more than 2yo!
• The BPE models splits on (spaces), before encoding the tokens. When converting the models from slow to fast the special tokens were added to the BPE vocabulary, with a score of 0. We probably forgot to add them to the list of additional_special_tokens, which is why they are not properly split. ( quick fix: t1.additional_special_tokens = ["</s>, ... ])
• @NielsRogge when you load a slow from a fast, it takes a long time because you need to convert the BPE sentenpiece model, which is very long. Nothing we can do about that.
• About your second question, the best thing would be to open a new issue. Seems like it might be another slow/fast discrepency but you are not completely doing this the way the API is designed! (check that each call to add a token actively adds it!)

[jiangwangyi]

Hey! Thanks for opening this issue.

• return_token_type_ids should be set to None by default but is updated with "token_type_ids" in self.model_input_names. This is specific to the fast tokenizer, and is a known difference. I am not sure why this was added only in the fast tokenizer but it's more than 2yo!
• The BPE models splits on (spaces), before encoding the tokens. When converting the models from slow to fast the special tokens were added to the BPE vocabulary, with a score of 0. We probably forgot to add them to the list of additional_special_tokens, which is why they are not properly split. ( quick fix: t1.additional_special_tokens = ["</s>, ... ])
• @NielsRogge when you load a slow from a fast, it takes a long time because you need to convert the BPE sentenpiece model, which is very long. Nothing we can do about that.
• About your second question, the best thing would be to open a new issue. Seems like it might be another slow/fast discrepency but you are not completely doing this the way the API is designed! (check that each call to add a token actively adds it!)

In the tokenizer_config.json of huggyllama/llama-7b, </s> is quite a special token (eos_token). Adding </s> to t1.additional_special_tokens does not fix the problem.

[ArthurZucker]

Indeed, sorry for the confusion. I added a different token <//s> with add_special_token which worked as expected ( meaning whether there was a space or not, the output was properly encode) which is why the issue most probably lies with the handling of the special tokens ( maybe we should not have added them to the voab? I'll check). I'll dig into this!

[jiangwangyi]

@ArthurZucker How is the progress now?

[ArthurZucker]

I am still working on this, top priority! My PR did not fix it yet, so I am opening a new on just for llama and will see for the other ones.

[jiangwangyi]

I am still working on this, top priority! My PR did not fix it yet, so I am opening a new on just for llama and will see for the other ones.

Thanks for working on this! I appreciate the update and look forward to getting the issue resolved.

[ArthurZucker]

Update: in order to fix this, the tokenizer.json should be modified: the special tokens should not be normalized (so set normalized = False. There is a more profound issue, since the slow tokenizer is not bother by that and handles this differently.