Skip to content

Add alpaca chat template (repush of #7383) #8159

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 3 commits into from
Closed

Add alpaca chat template (repush of #7383) #8159

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
8bafae2
added back "alpaca" template
jukofyork Jun 27, 2024
8f41334
added back "alpaca" template
jukofyork Jun 27, 2024
94a67c8
Merge branch 'ggerganov:master' into Add-Alpaca-chat-template
jukofyork Jun 27, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 15 additions & 0 deletions src/llama.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19559,6 +19559,21 @@ static int32_t llama_chat_apply_template_internal(
if (add_ass) {
ss << "ASSISTANT:";
}
} else if (tmpl == "alpaca" || (tmpl.find("### Instruction:") != std::string::npos && tmpl.find("<|EOT|>") == std::string::npos)) {
// meta-math/MetaMath-7B-V1.0
for (auto message : chat) {
std::string role(message->role);
if (role == "system") {
ss << message->content << "\n\n";
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In any cases, having \n\n after system message makes more sense to me. Could we delete the template for deepseek below and move it here?

} else if (tmpl == "alpaca" || tmpl == "deepseek" || (tmpl.find("### Instruction:") != std::string::npos && tmpl.find("<|EOT|>") == std::string::npos)) {

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it might give people the impression that they are the same prompt though when actually they are quite different: deepseek-coder requires an EOS token (and a non-default <|EOT|> vs </s> one!) and from the two models I tested that can use multi-turn alpaca; it seems adding an EOS token like </s> confuses them quite badly (AFAIK, it's the only template like this).

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you look in the previous PR that I closed accidentally, then there are some examples of phind-codellama getting very confused when I tried to use the EOS token.

I am unsure how people use alpaca for creative writing, but am going to try it over the weekend using mikupad, which seems to have forgotten to add any EOS tokens to any of their prompts (but nobody seems to have noticed or cared, and the models seem to have been writing fiction fine!?).

Copy link
Collaborator

@ngxson ngxson Jun 27, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Probably we can also patch llama_chat_apply_template_internal to take the EOS as an input (can be read from llama_model). Since this function is used internally, we can change the signature whenever we want.

Do you think that will be a better solution?

I am unsure how people use alpaca for creative writing, but am going to try it over the weekend using mikupad, which seems to have forgotten to add any EOS tokens to any of their prompts (but nobody seems to have noticed or cared, and the models seem to have been writing fiction fine!?).

That sounds to me like a bit of mess when the world moving from single-turn to multi-turn template 1 year ago. Back then, no one care about putting EOS after model's response because we can match stop sequence (### User:). In fact, for this reason, I was skeptical to add alpaca template in the first place, since some models are trained solely with single-turn dataset.

Copy link
Collaborator Author

@jukofyork jukofyork Jun 30, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Probably we can also patch llama_chat_apply_template_internal to take the EOS as an input (can be read from llama_model). Since this function is used internally, we can change the signature whenever we want.

Do you think that will be a better solution?

It may be worth looking at the model files of the few genuine multiturn Alpaca models to see if they even have a EOS token defined. Sadly it was the now deleted "wizard" models that mainly used this but some of the early "meta" models and phind-codellama also used it.

I am unsure how people use alpaca for creative writing, but am going to try it over the weekend using mikupad, which seems to have forgotten to add any EOS tokens to any of their prompts (but nobody seems to have noticed or cared, and the models seem to have been writing fiction fine!?).

That sounds to me like a bit of mess when the world moving from single-turn to multi-turn template 1 year ago. Back then, no one care about putting EOS after model's response because we can match stop sequence (### User:). In fact, for this reason, I was skeptical to add alpaca template in the first place, since some models are trained solely with single-turn dataset.

The mikupad devs just fixed all their templates after I pointed this out.

} else if (role == "user") {
ss << "### Instruction:\n" << message->content << "\n\n";
} else if (role == "assistant") {
ss << "### Response:\n" << message->content << "\n\n";
}
}
if (add_ass) {
ss << "### Response:\n";
}
} else if (tmpl == "deepseek" || (tmpl.find("### Instruction:") != std::string::npos && tmpl.find("<|EOT|>") != std::string::npos)) {
// deepseek-ai/deepseek-coder-33b-instruct
for (auto message : chat) {
Expand Down
5 changes: 5 additions & 0 deletions tests/test-chat-template.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,9 @@ int main(void) {
// Orca-Vicuna
// No template included in tokenizer_config.json, so this template likely needs to be manually set.
"{%- for message in messages %}{%- if message['role'] == 'system' -%}{{-'SYSTEM: ' + message['content'] + '\n' -}}{%- else -%}{%- if message['role'] == 'user' -%}{{-'USER: ' + message['content'] + '\n'-}}{%- else -%}{{-'ASSISTANT: ' + message['content'] + '</s>\n' -}}{%- endif -%}{%- endif -%}{%- endfor -%}{%- if add_generation_prompt -%}{{-'ASSISTANT:'-}}{%- endif -%}",
// meta-math/MetaMath-7B-V1.0
// No template included in tokenizer_config.json, so this template likely needs to be manually set.
"{%- set ns = namespace(found=false) -%} {%- for message in messages -%} {%- if message['role'] == 'system' -%} {%- set ns.found = true -%} {%- endif -%} {%- endfor -%} {%- if not ns.found -%} {{- '' + 'Below is an instruction that describes a task. Write a response that appropriately completes the request.' + '\n\n' -}} {%- endif %} {%- for message in messages %} {%- if message['role'] == 'system' -%} {{- '' + message['content'] + '\n\n' -}} {%- else -%} {%- if message['role'] == 'user' -%} {{-'### Instruction:\n' + message['content'] + '\n\n'-}} {%- else -%} {{-'### Response:\n' + message['content'] + '\n\n' -}} {%- endif -%} {%- endif -%} {%- endfor -%} {%- if add_generation_prompt -%} {{-'### Response:\n'-}} {%- endif -%}"
// CohereForAI/c4ai-command-r-plus
"{{ bos_token }}{% if messages[0]['role'] == 'system' %}{% set loop_messages = messages[1:] %}{% set system_message = messages[0]['content'] %}{% elif false == true %}{% set loop_messages = messages %}{% set system_message = 'You are Command-R, a brilliant, sophisticated, AI-assistant trained to assist human users by providing thorough responses. You are trained by Cohere.' %}{% else %}{% set loop_messages = messages %}{% set system_message = false %}{% endif %}{% if system_message != false %}{{ '<|START_OF_TURN_TOKEN|><|SYSTEM_TOKEN|>' + system_message + '<|END_OF_TURN_TOKEN|>' }}{% endif %}{% for message in loop_messages %}{% if (message['role'] == 'user') != (loop.index0 % 2 == 0) %}{{ raise_exception('Conversation roles must alternate user/assistant/user/assistant/...') }}{% endif %}{% set content = message['content'] %}{% if message['role'] == 'user' %}{{ '<|START_OF_TURN_TOKEN|><|USER_TOKEN|>' + content.strip() + '<|END_OF_TURN_TOKEN|>' }}{% elif message['role'] == 'assistant' %}{{ '<|START_OF_TURN_TOKEN|><|CHATBOT_TOKEN|>' + content.strip() + '<|END_OF_TURN_TOKEN|>' }}{% endif %}{% endfor %}{% if add_generation_prompt %}{{ '<|START_OF_TURN_TOKEN|><|CHATBOT_TOKEN|>' }}{% endif %}",
// Llama-3
Expand Down Expand Up @@ -82,6 +85,8 @@ int main(void) {
"You are a helpful assistant\n\nUSER: Hello\nASSISTANT: Hi there</s>\nUSER: Who are you\nASSISTANT: I am an assistant </s>\nUSER: Another question\nASSISTANT:",
// Orca-Vicuna
"SYSTEM: You are a helpful assistant\nUSER: Hello\nASSISTANT: Hi there</s>\nUSER: Who are you\nASSISTANT: I am an assistant </s>\nUSER: Another question\nASSISTANT:",
// meta-math/MetaMath-7B-V1.0
"You are a helpful assistant\n\n### Instruction:\nHello\n\n### Response:\nHi there\n\n### Instruction:\nWho are you\n\n### Response:\n I am an assistant \n\n### Instruction:\nAnother question\n\n### Response:\n",
// CohereForAI/c4ai-command-r-plus
"<|START_OF_TURN_TOKEN|><|SYSTEM_TOKEN|>You are a helpful assistant<|END_OF_TURN_TOKEN|><|START_OF_TURN_TOKEN|><|USER_TOKEN|>Hello<|END_OF_TURN_TOKEN|><|START_OF_TURN_TOKEN|><|CHATBOT_TOKEN|>Hi there<|END_OF_TURN_TOKEN|><|START_OF_TURN_TOKEN|><|USER_TOKEN|>Who are you<|END_OF_TURN_TOKEN|><|START_OF_TURN_TOKEN|><|CHATBOT_TOKEN|>I am an assistant<|END_OF_TURN_TOKEN|><|START_OF_TURN_TOKEN|><|USER_TOKEN|>Another question<|END_OF_TURN_TOKEN|><|START_OF_TURN_TOKEN|><|CHATBOT_TOKEN|>",
// Llama 3
Expand Down
Loading