server: Enable transcriptions API for LFM2-Audio

[tdakhran]

Overview

#21863 added support for /v1/audio/transcriptions API with the assumption that models expect [{"role":"user","content":"Transcribe audio to text<__media__>"}] format.

This doesn't hold, e.g., for LiquidAI/LFM2.5-Audio-1.5B-GGUF.

In PR, I try to relax this assumption by defining the model-specific strategy and using it in the server.

Additional information

With this change, transcription works correctly for LiquidAI/LFM2.5-Audio-1.5B-GGUF and adds a mechanism for defining a strategy for future models.

To test, start server with

llama-server -hf LiquidAI/LFM2.5-Audio-1.5B-GGUF

Use the code below to inference

# /// script
# requires-python = ">=3.9"
# dependencies = [
#     "openai",
# ]
# ///
import io
import urllib.request

from openai import OpenAI

client = OpenAI(api_key="your-api-key", base_url="http://localhost:8080/v1")

url = "https://github.com/ggerganov/whisper.cpp/raw/master/samples/jfk.wav"
audio_file = io.BytesIO(urllib.request.urlopen(url).read())
audio_file.name = "jfk.wav"
streaming = True

if streaming:
    transcription = client.audio.transcriptions.create(
        model="whisper-1",
        file=audio_file,
        stream=True
    )

    for chunk in transcription:
        try:
            print(chunk.delta, end="", flush=True)
        except Exception as e:
            print("\n", chunk)

else:
    transcription = client.audio.transcriptions.create(
        model="whisper-1",
        file=audio_file
    )
    print(transcription.text)

Output

❯ uv run transcript.py
And so, my fellow Americans, ask not what your country can do for you, ask what you can do for your country.
 TranscriptionTextDoneEvent(text='', type='transcript.text.done', logprobs=None, usage=Usage(input_tokens=157, output_tokens=27, total_tokens=184, type='tokens', input_token_details=None, input_tokens_details={'cached_tokens': 0}))

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: YES, for design and review only.

[ngxson]

IMO the current proposal is a bit complicated. Instead, any prompt-related should be confined to the common/chat.cpp|h system

For example, adding a common_chat_get_asr_prompt() that return the prompt specific to LFM2. See common_chat_params_init_lfm2 for now model-specific chat template is handled.

This way, you don't have to make any changes to libmtmd

[tdakhran]

IMO the current proposal is a bit complicated. Instead, any prompt-related should be confined to the common/chat.cpp|h system

For example, adding a common_chat_get_asr_prompt() that return the prompt specific to LFM2. See common_chat_params_init_lfm2 for now model-specific chat template is handled.

This way, you don't have to make any changes to libmtmd

@ngxson, instructions e.g. for ASR can vary from model to model. chat.h/chat.cpp don't have information about mm projector and about the task.

    if (src.find("<|tool_list_start|>") != std::string::npos &&
        src.find("<|tool_list_end|>") != std::string::npos) {
        LOG_DBG("Using specialized template: LFM2\n");
        return common_chat_params_init_lfm2(tmpl, params);
    }

convert_transcriptions_to_chatcmpl knows we do transcription, but that knowledge is lost immediately after the function exits.
Knowledge of which mm projector is used is available only in MTMD.

I'm struggling to see how these two can be detected for the ASR use case in chat.h/chat.cpp.

UPD: it actually worked, thank you @ngxson !

[ngxson]

Looking good, just a small comment. CC @pwilkin if you have any ideas on how to structure the newly added common_chat_get_asr_prompt (can be a follow-up PR to improve it)

[ngxson]

probably better to pass the whole chat_params here instead of just tmpls, that will make this function to be more aligned with oaicompat_chat_params_parse

[pwilkin]

Yeah, gonna think about it, probably might be a good idea to unify it and to just pass the entire JSON messages "preset" array.

[ngxson]

@pwilkin for now I think it should be good to merge, right? If so, can you give an approval? (This can be improved in a follow-up PR if you have any ideas.) Thanks!

[pwilkin]

Yes, let's merge it for now.

[ngxson]

@tdakhran can you resolve the conflict?

[tdakhran]

@tdakhran can you resolve the conflict?

@ngxson had to manually re apply patch because functions were moved across files, please take one more look before merging.

[ngxson]

Looks good, let's merge when the CI passes