[transformers] bos_token is not added by default

[fxmarty-amd]

Hi,

It appears that lm-evaluation-harness does not add a bos_token by default:

lm-evaluation-harness/lm_eval/models/api_models.py

Line 127 in 4439847

add_bos_token: bool = False,

lm-evaluation-harness/lm_eval/models/huggingface.py

Line 87 in 4439847

add_bos_token: bool | None = False,

except for a hard-coded gemma:

lm-evaluation-harness/lm_eval/models/huggingface.py

Lines 252 to 256 in 4439847

	if "gemma" in getattr(self.config, "model_type", ""):
	self.add_bos_token = True
	eval_logger.info(
	f"Model type is '{self.config.model_type}', part of the Gemma family--a BOS token will be used as Gemma underperforms without it."
	)

Why is that?

Some models explicitely state in their tokenizer_config.json that bos_token should be used: https://huggingface.co/unsloth/Llama-3.2-1B-Instruct/blob/main/tokenizer_config.json#L2

Is it intended?

Moreover in case apply_chat_template is used, bos token is added disregarding the self.add_bos_token option:

lm-evaluation-harness/lm_eval/models/huggingface.py

Lines 1508 to 1515 in 4439847

	try:
	chat_templated = self.tokenizer.apply_chat_template(
	chat_history,
	tokenize=False,
	add_generation_prompt=add_generation_prompt,
	continue_final_message=not add_generation_prompt,
	**self.chat_template_args,
	)

Thanks @baberabb

[baberabb]

Hi! I wasn't aware that some models do explicitly state they require a BOS token, and we should definitely respect that, if provided. Would appreciate a PR, if you have the bandwidth. Otherwise I'll take a look.

For chat templates, my understanding is that the template wraps the text in the appropriate special tokens, so adding a BOS token is redundant. But happy to be corrected if I'm misunderstanding something here.

[fxmarty-amd]

@baberabb Thank you.

The issue is e.g. mentioned in https://github.com/vllm-project/vllm/blob/main/docs/features/quantization/fp8.md#3-evaluating-accuracy for the evaluation of quantized models, where the presence/absence of bos_token has a strong impact.

Looking over a few models (transformers==4.56 & tokenizers==0.22.0):

----- Qwen/Qwen1.5-MoE-A2.7B-Chat
    tokenizer.add_bos_token NO ATTR
    tokenizer.bos_token_id None
    `'add_bos_token'` key is in tokenizer_config.json: False (value: N/A)
    bos_token_id added by default: False

----- meta-llama/Llama-3.1-70B-Instruct
    tokenizer.add_bos_token NO ATTR
    tokenizer.bos_token_id 128000
    `'add_bos_token'` key is in tokenizer_config.json: False (value: N/A)
    bos_token_id added by default: True

----- HuggingFaceTB/SmolLM-135M
    tokenizer.add_bos_token False
    tokenizer.bos_token_id 0
    `'add_bos_token'` key is in tokenizer_config.json: False (value: N/A)
    bos_token_id added by default: False

----- Qwen/Qwen2.5-0.5B-Instruct
    tokenizer.add_bos_token NO ATTR
    tokenizer.bos_token_id None
    `'add_bos_token'` key is in tokenizer_config.json: True (value: False)
    bos_token_id added by default: False

----- microsoft/Phi-4-mini-instruct
    tokenizer.add_bos_token False
    tokenizer.bos_token_id 199999
    `'add_bos_token'` key is in tokenizer_config.json: True (value: False)
    bos_token_id added by default: False

----- deepseek-ai/DeepSeek-R1
    tokenizer.add_bos_token True
    tokenizer.bos_token_id 0
    `'add_bos_token'` key is in tokenizer_config.json: True (value: True)
    bos_token_id added by default: True

----- meta-llama/Llama-2-7b-chat-hf
    tokenizer.add_bos_token True
    tokenizer.bos_token_id 1
    `'add_bos_token'` key is in tokenizer_config.json: True (value: True)
    bos_token_id added by default: True

----- unsloth/Llama-3.2-1B-Instruct
    tokenizer.add_bos_token NO ATTR
    tokenizer.bos_token_id 128000
    `'add_bos_token'` key is in tokenizer_config.json: True (value: True)
    bos_token_id added by default: True

----- facebook/opt-125m
    tokenizer.add_bos_token True
    tokenizer.bos_token_id 2
    `'add_bos_token'` key is in tokenizer_config.json: True (value: True)
    bos_token_id added by default: True

----- openai/gpt-oss-20b
    tokenizer.add_bos_token NO ATTR
    tokenizer.bos_token_id 199998
    `'add_bos_token'` key is in tokenizer_config.json: False (value: N/A)
    bos_token_id added by default: False

----- google/gemma-3-270m
    tokenizer.add_bos_token True
    tokenizer.bos_token_id 2
    `'add_bos_token'` key is in tokenizer_config.json: True (value: True)
    bos_token_id added by default: True

----- RedHatAI/Llama-4-Scout-17B-16E-Instruct-FP8-dynamic
    tokenizer.add_bos_token NO ATTR
    tokenizer.bos_token_id 200000
    `'add_bos_token'` key is in tokenizer_config.json: False (value: N/A)
    bos_token_id added by default: True

----- NousResearch/Hermes-3-Llama-3.1-405B
    tokenizer.add_bos_token NO ATTR
    tokenizer.bos_token_id 128000
    `'add_bos_token'` key is in tokenizer_config.json: False (value: N/A)
    bos_token_id added by default: True

So there are different cases:

• add_bos_token is in tokenizer_config.json: then it is probably always complied to.
• add_bos_token not in tokenizer_config.json: then tokenizer.add_bos_token seems to default to False, except for llama / gpt-oss where the attribute does not exist, and where llama3/llama4 adds a bos_token by default and gpt-oss does not.

So to be honest the behavior in Transformers is a bit messy/buggy.

In any case, I am thinking that by default lm-eval-harness should comply with the default tokenizer behavior:

if hasattr(tokenizer, "add_bos_token"):
    add_special_tokens = tokenizer.add_bos_token
elif not hasattr(tokenizer, "bos_token_id"):
    add_special_tokens = False
else:
    # gpt-oss and llama3/4 mess that do not have tokenizer.add_bos_token attribute, all have a `bos_token_id`, but is sometimes added by default, sometimes not. Likely a bug in transformers.

Does that sound reasonable @baberabb? This may change eval results as currently special token are not added - is it a concern?

My reasoning is that lm-eval-harness should make the best effort to comply with the default BOS token behavior indicated on the models config on HF hub.

(There is also eos_token to handle, so it may not be as simple as that, but from the tested models only facebook/opt-125m use it by default.)

[DarkLight1337]

For Gemma models specifically, this should be fixed by #3206. I do think that this should be done in a unified manner though.