server: support OAI /v1/audio/transcriptions API

[ngxson]

Overview

Fix #21852

Also support form multipart input data in the HTTP layer (note: input will be treated as string)

Tested with gemma-4-E4B-it

My test code:

from openai import OpenAI
client = OpenAI(api_key="your-api-key", base_url="http://localhost:8080/v1")

audio_file = open("i-have-a-dream-30s.mp3", "rb")
streaming = True

if streaming:
    transcription = client.audio.transcriptions.create(
        model="whisper-1", 
        file=audio_file,
        stream=True
    )

    for chunk in transcription:
        try:
            print(chunk.delta, end="", flush=True)
        except Exception as e:
            print("\n", chunk)

else:
    transcription = client.audio.transcriptions.create(
        model="whisper-1", 
        file=audio_file
    )
    print(transcription.text)

Output:

I have a dream that one day every valley shall be exalted, and every hill and mountain shall be made low, the rough places will be made plain, and the crooked places will be made straight, and the glory of the Lord shall be revealed and all flesh shall see it together. This is our hope. This is the peace that I go back to the South with. With this faith, we will be able to hew out of the mountain of despair a stone of
TranscriptionTextDoneEvent(text='', type='transcript.text.done', logprobs=None, usage=Usage(input_tokens=774, output_tokens=96, total_tokens=870, type='tokens', input_token_details=None, input_tokens_details={'cached_tokens': 773}))

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: no

[kth8]

I tried the provided test script but got

openai.BadRequestError: Error code: 400 - {'error': {'code': 400, 'message': "Only 'json' response_format is supported for transcription", 'type': 'invalid_request_error'}}

until I added response_format="json" argument to client.audio.transcriptions.create.
Also needed via curl like

curl -s "https://cdn.openai.com/whisper/draft-20220913a/micro-machines.wav" | curl -s -X POST -F response_format=json -F "file=@-;filename=audio.wav" "http://127.0.0.1:8080/v1/audio/transcriptions"

[ngxson]

@kth8 thanks, that should be fixed in the latest commit

CC @ggml-org/maintainers if someone can give an approval

[tdakhran]

Thanks, @ngxson . Do you plan to support streaming encoding or live transcription?

[ngxson]

Hey @tdakhran , I don't have a plan, but I can publish a plan for doing so. I think the more complicated part is to implement a websocket endpoint (fortunately, it's supported by httplib now)

If I understand correctly, the realtime transcription simply reduce the chunk size of the input audio, and text can be generated in-between 2 audio chunks, right?

[tdakhran]

Hey @tdakhran , I don't have a plan, but I can publish a plan for doing so. I think the more complicated part is to implement a websocket endpoint (fortunately, it's supported by httplib now)

If I understand correctly, the realtime transcription simply reduce the chunk size of the input audio, and text can be generated in-between 2 audio chunks, right?

I actually packed two feature requests into a single message :)

• Streaming Encoding: In streaming mode, for each received audio chunk, run the encoder (if it supports streaming) and chunked prefill. Once the last chunk is received, process only the last chunk and start decoding. This will allow us to significantly reduce TTFT, as most of the input audio will be encoded and prefilled during streaming.
• Live Transcription: Run generation and prefill in parallel; there are a few models that support it, but the implementation will require significant architectural changes.

[candrews]

The API prefixes responses with:

The user has provided an audio prompt and requested a transcription in English. I need to accurately transcribe the spoken words from the provided audio.<channel|>

The prefix makes the API unusable for many purposes - how can it be removed?

curl -X POST http://localhost:5805/v1/audio/transcriptions -F file=@jfk.wav -F language=en -F response_format=json
{"type":"transcript.text.done","text":"The user has provided an audio prompt and requested a transcription in English. I need to accurately transcribe the spoken words from the provided audio.<channel|>And so my fellow Americans, ask not what your country can do for you, ask what you can do for your country.","usage":{"type":"tokens","input_tokens":300,"output_tokens":54,"total_tokens":354,"input_tokens_details":{"cached_tokens":9}}}

[ngxson]

I don't think it can, remind that this is not a pure ASR model like whisper, but rather a general-purpose one.

You can provide a specific prompt like Only output the transcription, no explanations

[candrews]

I don't think it can, remind that this is not a pure ASR model like whisper, but rather a general-purpose one.

You can provide a specific prompt like Only output the transcription, no explanations

I tried using the prompt from the official documentation at https://ai.google.dev/gemma/docs/core/model_card_4 which should output just the transcribed text:

$ curl -s -X POST http://localhost:5805/v1/audio/transcriptions -F file=@jfk.wav -F language=en -F response_format=json -F 'prompt=Transcribe the following speech segment in English into English text.

Follow these specific instructions for formatting the answer:
* Only output the transcription, with no newlines.
* When transcribing numbers, write the digits, i.e. write 1.7 and not one point seven, and write 3 instead of three.' | jq .
{
  "type": "transcript.text.done",
  "text": "The user has provided an audio segment and requested a transcription into English, following specific formatting rules.\nThe content is a speech excerpt.\nI need to transcribe the speech accurately and ensure there are no newlines in the final output.<channel|>And so my fellow Americans ask not what your country can do for you ask what you can do for your country",
  "usage": {
    "type": "tokens",
    "input_tokens": 363,
    "output_tokens": 71,
    "total_tokens": 434,
    "input_tokens_details": {
      "cached_tokens": 358
    }
  }
}

I wonder - should the response include just the channel text? It seems like it should, as the earlier text is reasoning info, which isn't appropriate for an OAI transcriptions response.

[ngxson]

hmm yeah that's not expected, the reasoning should be excluded from the response. I'll push a fix

[ngxson]

@candrews please try #21905

[candrews]

lease try #21905

I recompiled with that patch, and can confirm that it fixes the issue.

(My initial testing was flawed because I was using an outdated and/or broken chat template. Using the latest chat template included in the current unsloth/gemma-4-E4B-it-GGUF:Q4_K_M model works great.)

[NickM-27]

It looks like this endpoint only works in direct server mode. when running in router mode I see logs that it is routing it to the correct model but the request never gets run or returned

[tha80]

I got the same problem as @NickM-27:

curl -v -F response_format=json -F model=Gemma4-E4B-vulkan -F prompt="Transcribe exactly." -F file=@kling-sample.mp3 http://gfxrig01:8081/v1/audio/transcriptions
* Host gfxrig01:8081 was resolved.
* IPv6: ******, ******
* IPv4: ******
*   Trying [******]:8081...
* Connected to gfxrig01 (******) port 8081
> POST /v1/audio/transcriptions HTTP/1.1
> Host: gfxrig01:8081
> User-Agent: curl/8.7.1
> Accept: */*
> Content-Length: 780341
> Content-Type: multipart/form-data; boundary=------------------------VpTOuUqa5y0gLulttzO9ne
> 
* upload completely sent off: 780341 bytes

But curl never finished. llama.cpp only shows:

srv  proxy_reques: proxying request to model Gemma4-E4B-vulkan on port 48741

I am also using llama.cpp in router mode. When I upload a audio file via the built-in WebUI it works smoothly and fast.

[tha80]

I just created a new issue in Github for this:
#22044

[montvid]

gemma-4-E4B-it BF16 unusable for long 50 min audio - even using --ctx-size 131072. 76200 tokens input outputs duplicated, looped text even at temp=0. Maybe needs a fix? DId anyone tried this code with long time audio?

[ServeurpersoCom]

gemma-4-E4B-it BF16 unusable for long 50 min audio - even using --ctx-size 131072. 76200 tokens input outputs duplicated, looped text even at temp=0. Maybe needs a fix? DId anyone tried this code with long time audio?

Gemma 4 audio is trained on short clips (30s), so a full 50 min decode tends to loop. The endpoint doesn't chunk long audio yet, so splitting into <=30s parts before sending (or whisper for long-form) does the trick for now, and proper long-audio chunking could be a nice follow-up PR :)

[montvid]

@ServeurpersoCom thanks for the reply, but the endpoint does chunk long audio - just in a weird way mixing up word and sentence layout after ~30 min into asr. Trying now other projects but no luck so far.

[ngxson]

@montvid could you test it with another runtime like vllm or mlx-vlm ? just to see if the model naturally behave that way, or there is a problem specific to llama.cpp

[montvid]

@montvid could you test it with another runtime like vllm or mlx-vlm ? just to see if the model naturally behave that way, or there is a problem specific to llama.cpp

No better solution than your implementation so far. I'm back to Faster Whisper and https://github.com/Purfview/whisper-standalone-win as it has better accuracy for me and word level timestamps that are more precise than whisper.cpp...