server : added --no-prefill-assistant flag #13608
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Following up on PR #13174.
Overview
After some discussion, the decision was made to add an opt-out flag for the assistant prefill behavior so it can be disabled to restore the previous functionality.
--no-prefill-assistant
flag, specific tollama-server
. Also has a corresponding env varLLAMA_ARG_NO_PREFILL_ASSISTANT
.assistant
message if it's at the end of themessages
array, so that use-cases such as Feature Request: Prefix assistant answer #11536 continue to work.assistant
message as a full message as was the behavior before Prefilling assistant message in openai compatible API #13174.Testing
Used
bartowski/Llama-3.2-1B-Instruct-GGUF
for testing as I had it on hand. Tested with both/apply-template
and/v1/chat/completions
as they both used the shared prompt templating functions./apply-template
:/v1/chat/completions
:This is my first non-docs PR to llama.cpp so let me know if I need to make any changes 😅