训练gemma3报错

[sjdongai]

Reminder

• I have read the above rules and searched the existing issues.

System Info

Processor was not found, please check and update your processor config.

Reproduction

Put your message here.

Others

No response

[Kuangdd01]

Hi! Please refer to solution #7281 (comment) and this issue #7320 for help.

[MattMaBX]

Hi! Please refer to solution #7281 (comment) and this issue #7320 for help.

I've noticed that after I follow the #7281 (comment) to update transformers, I can train gemma3-4b-it and gemma3-12b-it normally, but gemma3-1b-it still cannot be trained normally, and the same error is reported Processor was not found, please check and update your processor config. BTW, I have checked the integrity of the model file.

As far as I can guess, maybe gemma3-1b-it has a different prompt template than the other two sizes of models, so the corresponding processor cannot be found. In another fine-tuning framework "ms-swift", the hint template listed for the 1b model is gemma3_text, while the models at other scales are all gemma3_vision. But in our LLaMA-Factory, its template is unified to gemma3, which may be the cause of this problem?

[Kuangdd01]

Hi, It is recommended to use the template gemma for gemma-1b-it/pt. It is a pure text template and consistent with gemma-3.

[MattMaBX]

Hi, It is recommended to use the template gemma for gemma-1b-it/pt. It is a pure text template and consistent with gemma-3.

Thanks for the reply^-^. I don't have enough GPUs right now to verify that the gemma template can be used to train gemma3-1b-it, but I have checked tokenizer_config.json in gemma3-1b-it and gemma2-2b-it, and the chat_template in it is different. Therefore, I think a better solution is to add corresponding templates such as gemma-text and gemma-vision to the LLaMA-Factory to accurately adapt to the new model~

Also, I have tried to fine-tune gemma-3-1b-it in another framework ms-swift and when transformers are updated to version 4.50.0.dev0, the fine-tuning process is completely normal. However, in our LLaMA-Factory, gemma-3-1b-it cannot be fine-tuned even when transformers are updated to 4.50.dev0. So I think we can rule out the influence of the transformers library on this issue.

[Kuangdd01]

Thanks for your reporting. The chat templates indeed seem different in some ways. I made a small test of them.

from transformers import AutoTokenizer
p1 = AutoTokenizer.from_pretrained("/test_models/gemma-1.1-2b-it")
p2 = AutoTokenizer.from_pretrained("/test_models/gemma-3-1b-it")
chat = [
  {"role": "user", "content": "Hello, how are you?"},
  {"role": "assistant", "content": "I'm doing great. How can I help you today?"},
  {"role": "user", "content": "I'd like to show off how chat templating works!"},
]
print(p1.apply_chat_template(chat, tokenize=False))
print("===")
print("===")
print(p2.apply_chat_template(chat, tokenize=False))

output:

<bos><start_of_turn>user
Hello, how are you?<end_of_turn>
<start_of_turn>model
I'm doing great. How can I help you today?<end_of_turn>
<start_of_turn>user
I'd like to show off how chat templating works!<end_of_turn>

===
===
<bos><start_of_turn>user
Hello, how are you?<end_of_turn>
<start_of_turn>model
I'm doing great. How can I help you today?<end_of_turn>
<start_of_turn>user
I'd like to show off how chat templating works!<end_of_turn>

So we can reuse the gemma template for gemma-3-1b-it for now.
Now that the transformers repository is changing a lot, we'll keep an eye out for their new PRs!

[MattMaBX]

Thanks for your reporting. The chat templates indeed seem different in some ways. I made a small test of them.

from transformers import AutoTokenizer
p1 = AutoTokenizer.from_pretrained("/test_models/gemma-1.1-2b-it")
p2 = AutoTokenizer.from_pretrained("/test_models/gemma-3-1b-it")
chat = [
{"role": "user", "content": "Hello, how are you?"},
{"role": "assistant", "content": "I'm doing great. How can I help you today?"},
{"role": "user", "content": "I'd like to show off how chat templating works!"},
]
print(p1.apply_chat_template(chat, tokenize=False))
print("===")
print("===")
print(p2.apply_chat_template(chat, tokenize=False))
output:

<bos><start_of_turn>user
Hello, how are you?<end_of_turn>
<start_of_turn>model
I'm doing great. How can I help you today?<end_of_turn>
<start_of_turn>user
I'd like to show off how chat templating works!<end_of_turn>

===
===
<bos><start_of_turn>user
Hello, how are you?<end_of_turn>
<start_of_turn>model
I'm doing great. How can I help you today?<end_of_turn>
<start_of_turn>user
I'd like to show off how chat templating works!<end_of_turn>

So we can reuse the gemma template for gemma-3-1b-it for now. Now that the transformers repository is changing a lot, we'll keep an eye out for their new PRs!

Thanks for the reply and testing ~

I also tried to train gemma3-1b-it using gemma template and there were no problems.

I would like to suggest that the template description of the gemma3 model should be modified in the readme.md to avoid misunderstandings. Since there are a lot of issues about training the gemma3 model, I think that probably many of the issues are related to the issue we discussed, and changing the readme.md may prevent this issue from recurring.

Finally, thank you very much for contributing to this wonderful project, and hope that our LLaMA-Factory will keep getting better ~

[sjdongai]

Thanks for your reporting. The chat templates indeed seem different in some ways. I made a small test of them.
from transformers import AutoTokenizer
p1 = AutoTokenizer.from_pretrained("/test_models/gemma-1.1-2b-it")
p2 = AutoTokenizer.from_pretrained("/test_models/gemma-3-1b-it")
chat = [
{"role": "user", "content": "Hello, how are you?"},
{"role": "assistant", "content": "I'm doing great. How can I help you today?"},
{"role": "user", "content": "I'd like to show off how chat templating works!"},
]
print(p1.apply_chat_template(chat, tokenize=False))
print("===")
print("===")
print(p2.apply_chat_template(chat, tokenize=False))
output:

<bos><start_of_turn>user
Hello, how are you?<end_of_turn>
<start_of_turn>model
I'm doing great. How can I help you today?<end_of_turn>
<start_of_turn>user
I'd like to show off how chat templating works!<end_of_turn>

===
===
<bos><start_of_turn>user
Hello, how are you?<end_of_turn>
<start_of_turn>model
I'm doing great. How can I help you today?<end_of_turn>
<start_of_turn>user
I'd like to show off how chat templating works!<end_of_turn>

So we can reuse the gemma template for gemma-3-1b-it for now. Now that the transformers repository is changing a lot, we'll keep an eye out for their new PRs!

Thanks for the reply and testing ~

I also tried to train gemma3-1b-it using gemma template and there were no problems.

I would like to suggest that the template description of the gemma3 model should be modified in the readme.md to avoid misunderstandings. Since there are a lot of issues about training the gemma3 model, I think that probably many of the issues are related to the issue we discussed, and changing the readme.md may prevent this issue from recurring.

Finally, thank you very much for contributing to this wonderful project, and hope that our LLaMA-Factory will keep getting better ~

Hello, the trained model reported an error according to the official inference method. As follows: p.attn_bias_ptr is not correctly aligned. Have you encountered it before? And miraculously, as long as one sample in the test set encounters this error, the following samples will report this error, but when the program is restarted and one of the samples is selected, this error will not be reported

[Jacksonlark]

Hi! Please refer to solution #7281 (comment) and this issue #7320 for help.

I've noticed that after I follow the #7281 (comment) to update transformers, I can train gemma3-4b-it and gemma3-12b-it normally, but gemma3-1b-it still cannot be trained normally, and the same error is reported Processor was not found, please check and update your processor config. BTW, I have checked the integrity of the model file.

As far as I can guess, maybe gemma3-1b-it has a different prompt template than the other two sizes of models, so the corresponding processor cannot be found. In another fine-tuning framework "ms-swift", the hint template listed for the 1b model is gemma3_text, while the models at other scales are all gemma3_vision. But in our LLaMA-Factory, its template is unified to gemma3, which may be the cause of this problem?

@MattMaBX hi！I want to how to ft gemma3-4b-it model using LLaMA-Factory. Can you give more details? I try to train but appear

[rank0]: Traceback (most recent call last):
[rank0]: File "/workdir/src/train.py", line 28, in
[rank0]: main()
[rank0]: File "/workdir/src/train.py", line 19, in main
[rank0]: run_exp()
[rank0]: File "/workdir/src/llamafactory/train/tuner.py", line 103, in run_exp
[rank0]: _training_function(config={"args": args, "callbacks": callbacks})
[rank0]: File "/workdir/src/llamafactory/train/tuner.py", line 68, in _training_function
[rank0]: run_sft(model_args, data_args, training_args, finetuning_args, generating_args, callbacks)
[rank0]: File "/workdir/src/llamafactory/train/sft/workflow.py", line 51, in run_sft
[rank0]: dataset_module = get_dataset(template, model_args, data_args, training_args, stage="sft", **tokenizer_module)
[rank0]: File "/workdir/src/llamafactory/data/loader.py", line 303, in get_dataset
[rank0]: dataset = _get_preprocessed_dataset(
[rank0]: File "/workdir/src/llamafactory/data/loader.py", line 249, in _get_preprocessed_dataset
[rank0]: dataset = dataset.map(
[rank0]: File "/conda/lib/python3.9/site-packages/datasets/arrow_dataset.py", line 560, in wrapper
[rank0]: out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
[rank0]: File "/conda/lib/python3.9/site-packages/datasets/arrow_dataset.py", line 3073, in map
[rank0]: for rank, done, content in Dataset._map_single(**dataset_kwargs):
[rank0]: File "/conda/lib/python3.9/site-packages/datasets/arrow_dataset.py", line 3476, in _map_single
[rank0]: batch = apply_function_on_filtered_inputs(
[rank0]: File "/conda/lib/python3.9/site-packages/datasets/arrow_dataset.py", line 3338, in apply_function_on_filtered_inputs
[rank0]: processed_inputs = function(*fn_args, *additional_args, **fn_kwargs)
[rank0]: File "/workdir/src/llamafactory/data/processor/supervised.py", line 99, in preprocess_dataset
[rank0]: input_ids, labels = self._encode_data_example(
[rank0]: File "/workdir/src/llamafactory/data/processor/supervised.py", line 43, in _encode_data_example
[rank0]: messages = self.template.mm_plugin.process_messages(prompt + response, images, videos, audios, self.processor)
[rank0]: File "/workdir/src/llamafactory/data/mm_plugin.py", line 409, in process_messages
[rank0]: self._validate_input(processor, images, videos, audios)
[rank0]: File "/workdir/src/llamafactory/data/mm_plugin.py", line 163, in _validate_input
[rank0]: raise ValueError("Processor was not found, please check and update your processor config.")
[rank0]: ValueError: Processor was not found, please check and update your processor config.

[sjdongai]

Hi! Please refer to solution #7281 (comment) and this issue #7320 for help.

I've noticed that after I follow the #7281 (comment) to update transformers, I can train gemma3-4b-it and gemma3-12b-it normally, but gemma3-1b-it still cannot be trained normally, and the same error is reported Processor was not found, please check and update your processor config. BTW, I have checked the integrity of the model file.
As far as I can guess, maybe gemma3-1b-it has a different prompt template than the other two sizes of models, so the corresponding processor cannot be found. In another fine-tuning framework "ms-swift", the hint template listed for the 1b model is gemma3_text, while the models at other scales are all gemma3_vision. But in our LLaMA-Factory, its template is unified to gemma3, which may be the cause of this problem?

@MattMaBX hi！I want to how to ft gemma3-4b-it model using LLaMA-Factory. Can you give more details? I try to train but appear

[rank0]: Traceback (most recent call last): [rank0]: File "/workdir/src/train.py", line 28, in [rank0]: main() [rank0]: File "/workdir/src/train.py", line 19, in main [rank0]: run_exp() [rank0]: File "/workdir/src/llamafactory/train/tuner.py", line 103, in run_exp [rank0]: _training_function(config={"args": args, "callbacks": callbacks}) [rank0]: File "/workdir/src/llamafactory/train/tuner.py", line 68, in _training_function [rank0]: run_sft(model_args, data_args, training_args, finetuning_args, generating_args, callbacks) [rank0]: File "/workdir/src/llamafactory/train/sft/workflow.py", line 51, in run_sft [rank0]: dataset_module = get_dataset(template, model_args, data_args, training_args, stage="sft", **tokenizer_module) [rank0]: File "/workdir/src/llamafactory/data/loader.py", line 303, in get_dataset [rank0]: dataset = _get_preprocessed_dataset( [rank0]: File "/workdir/src/llamafactory/data/loader.py", line 249, in _get_preprocessed_dataset [rank0]: dataset = dataset.map( [rank0]: File "/conda/lib/python3.9/site-packages/datasets/arrow_dataset.py", line 560, in wrapper [rank0]: out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs) [rank0]: File "/conda/lib/python3.9/site-packages/datasets/arrow_dataset.py", line 3073, in map [rank0]: for rank, done, content in Dataset._map_single(**dataset_kwargs): [rank0]: File "/conda/lib/python3.9/site-packages/datasets/arrow_dataset.py", line 3476, in _map_single [rank0]: batch = apply_function_on_filtered_inputs( [rank0]: File "/conda/lib/python3.9/site-packages/datasets/arrow_dataset.py", line 3338, in apply_function_on_filtered_inputs [rank0]: processed_inputs = function(*fn_args, *additional_args, **fn_kwargs) [rank0]: File "/workdir/src/llamafactory/data/processor/supervised.py", line 99, in preprocess_dataset [rank0]: input_ids, labels = self._encode_data_example( [rank0]: File "/workdir/src/llamafactory/data/processor/supervised.py", line 43, in _encode_data_example [rank0]: messages = self.template.mm_plugin.process_messages(prompt + response, images, videos, audios, self.processor) [rank0]: File "/workdir/src/llamafactory/data/mm_plugin.py", line 409, in process_messages [rank0]: self._validate_input(processor, images, videos, audios) [rank0]: File "/workdir/src/llamafactory/data/mm_plugin.py", line 163, in _validate_input [rank0]: raise ValueError("Processor was not found, please check and update your processor config.") [rank0]: ValueError: Processor was not found, please check and update your processor config.

pip install -U transformers

[Jacksonlark]

Hi! Please refer to solution #7281 (comment) and this issue #7320 for help.

I've noticed that after I follow the #7281 (comment) to update transformers, I can train gemma3-4b-it and gemma3-12b-it normally, but gemma3-1b-it still cannot be trained normally, and the same error is reported Processor was not found, please check and update your processor config. BTW, I have checked the integrity of the model file.
As far as I can guess, maybe gemma3-1b-it has a different prompt template than the other two sizes of models, so the corresponding processor cannot be found. In another fine-tuning framework "ms-swift", the hint template listed for the 1b model is gemma3_text, while the models at other scales are all gemma3_vision. But in our LLaMA-Factory, its template is unified to gemma3, which may be the cause of this problem?

@MattMaBX hi！I want to how to ft gemma3-4b-it model using LLaMA-Factory. Can you give more details? I try to train but appear
[rank0]: Traceback (most recent call last): [rank0]: File "/workdir/src/train.py", line 28, in [rank0]: main() [rank0]: File "/workdir/src/train.py", line 19, in main [rank0]: run_exp() [rank0]: File "/workdir/src/llamafactory/train/tuner.py", line 103, in run_exp [rank0]: _training_function(config={"args": args, "callbacks": callbacks}) [rank0]: File "/workdir/src/llamafactory/train/tuner.py", line 68, in _training_function [rank0]: run_sft(model_args, data_args, training_args, finetuning_args, generating_args, callbacks) [rank0]: File "/workdir/src/llamafactory/train/sft/workflow.py", line 51, in run_sft [rank0]: dataset_module = get_dataset(template, model_args, data_args, training_args, stage="sft", **tokenizer_module) [rank0]: File "/workdir/src/llamafactory/data/loader.py", line 303, in get_dataset [rank0]: dataset = _get_preprocessed_dataset( [rank0]: File "/workdir/src/llamafactory/data/loader.py", line 249, in _get_preprocessed_dataset [rank0]: dataset = dataset.map( [rank0]: File "/conda/lib/python3.9/site-packages/datasets/arrow_dataset.py", line 560, in wrapper [rank0]: out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs) [rank0]: File "/conda/lib/python3.9/site-packages/datasets/arrow_dataset.py", line 3073, in map [rank0]: for rank, done, content in Dataset._map_single(**dataset_kwargs): [rank0]: File "/conda/lib/python3.9/site-packages/datasets/arrow_dataset.py", line 3476, in _map_single [rank0]: batch = apply_function_on_filtered_inputs( [rank0]: File "/conda/lib/python3.9/site-packages/datasets/arrow_dataset.py", line 3338, in apply_function_on_filtered_inputs [rank0]: processed_inputs = function(*fn_args, *additional_args, **fn_kwargs) [rank0]: File "/workdir/src/llamafactory/data/processor/supervised.py", line 99, in preprocess_dataset [rank0]: input_ids, labels = self._encode_data_example( [rank0]: File "/workdir/src/llamafactory/data/processor/supervised.py", line 43, in _encode_data_example [rank0]: messages = self.template.mm_plugin.process_messages(prompt + response, images, videos, audios, self.processor) [rank0]: File "/workdir/src/llamafactory/data/mm_plugin.py", line 409, in process_messages [rank0]: self._validate_input(processor, images, videos, audios) [rank0]: File "/workdir/src/llamafactory/data/mm_plugin.py", line 163, in _validate_input [rank0]: raise ValueError("Processor was not found, please check and update your processor config.") [rank0]: ValueError: Processor was not found, please check and update your processor config.

pip install -U transformers

already update
above error not from transformers

[hiyouga]

see readme for template selection @Jacksonlark

[652994331]

@hiyouga The problem is still here, plz help! after followed #7281 (comment) , I pip install

but after that , the problem became:

I used Gemma-1b and Gemma-4b with template "gemma3" , I got same result, . So how can we solve this problem, thank you