[User] --reverse-prompt no longer echoes in the console

[44670]

Prerequisites

Please answer the following questions for yourself before submitting an issue.

• [Y] I am running the latest code. Development is very rapid so there are no tagged versions as of now.
• [Y] I carefully followed the README.md.
• [Y] I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
• [Y] I reviewed the Discussions, and have a new bug or useful enhancement to share.

Expected Behavior

Hi!

I have been using the --reverse-prompt "User:" parameter along with the interactive mode in previous versions. When waiting for user input, the console used to echo "User:" as shown in the screenshot below:

Current Behavior

However, in the new version found in llama.cpp, the "User:" prompt is no longer echoed in the console. I've checked the documentation, and it doesn't seem to mention any changes regarding this behavior. Is this an oversight or an intentional modification?

My command-line args could be found at: https://github.com/OpenBuddy/OpenBuddy/blob/main/chat-llamacpp.bat

Please let me know if you need any additional information or if there's something else I need to do to restore the previous behavior.

Thank you!

[ghost]

Apparently, it's intentional: "Previously, the first reverse prompt (if existed) is always inserted. Now if the model generate EOS, the reverse prompt is not inserted. To get the same behavior, please use

./main -m "$MODEL" -i --in-prefix "User: " --in-suffix "Assistant:" ..."

I made a Poll showing unintentional effects after the BOS commit.

Here's a repo without BOS: https://github.com/JackJollimore/llama.cpp-without-input-bos/tree/master

[44670]

Ahh, thank you for the explanation! I will update my demo script soon.

[44670]

Hmm...

I have read through the documentation and example code again, and I'm feeling even more confused.

In the original examples within the repo, the cmdline was --r "User:" --in-prefix " ", but the reasoning behind this doesn't seem to be clearly explained. Could it be because --in-prefix "User: " itself is incorrect?

Moreover, Is the behavior of --in-prefix "User: " in the new version (including what is displayed in the console and what is fed into the model) completely consistent with --r "User:" --in-prefix " " in the old version? I hope that the differences between the behavior of new and old versions can be more clarified in the documentation/example shell script.

[ghost]

I have read through the documentation and example code again, and I'm feeling even more confused.

Welcome to the Club.

In the original examples within the repo, the cmdline was --r "User:" --in-prefix " ", but the reasoning behind this doesn't seem to be clearly explained. Could it be because --in-prefix "User: " itself is incorrect?

The current documentation is not updated, so if you absolutely rely on, -r "User:" --in-prefix " ", then I suggest using a repo whereBOS commit is reverted, and the documentation is applicable.

Moreover, Is the behavior of --in-prefix "User: " in the new version (including what is displayed in the console and what is fed into the model) completely consistent with --r "User:" --in-prefix " " in the old version? I hope that the differences between the behavior of new and old versions can be more clarified in the documentation/example shell script.

I suppose it's intended to be consistent, but my testing has shown that, No - it's not 1:1

Related: #2507, #2417

If it were my decision then llama.cpp would either resolve the unintended effects that I mentioned in the Poll, or simply revert --input-bos commit.

[IDDT]

From what I understood, using --in-prefix "User: " relies on model outputting end-of-stream token after Assistant response. If there is no end-of-stream, the model keeps generating user input, which is undesirable. Launching llamacpp with -r "User:" --in-prefix " " --ignore-eos seemed to fix the issue for me.

[jxy]

@IDDT yes.

[ghost]

ggerganov accurately summed it up: "Ok, I think I see the problem. It comes from keeping EOS token. Ignoring it is not the same as before the BOS commit because there is no new line.

I'm on a phone so cannot push a fix. likely on monday or tuesday"

Thank you @wtarreau for bisecting to help identify the problem.

[ghost]

@IDDT, @44670:

If you'll help test then here's a PR for fix: #2689

[github-actions]

This issue was closed because it has been inactive for 14 days since being marked as stale.