Skip to content

server: add "X-Accel-Buffering": "no" header to streaming endpoints#24774

Merged
ServeurpersoCom merged 1 commit into
ggml-org:masterfrom
regunakyle:X-Accel-Buffering
Jun 18, 2026
Merged

server: add "X-Accel-Buffering": "no" header to streaming endpoints#24774
ServeurpersoCom merged 1 commit into
ggml-org:masterfrom
regunakyle:X-Accel-Buffering

Conversation

@regunakyle

@regunakyle regunakyle commented Jun 18, 2026

Copy link
Copy Markdown
Contributor

Overview

This header tells Nginx (as a reverse proxy with SSL) to NOT buffer responses.
Only streaming endpoints are affected. The header does nothing if Nginx is not involved.
Without it, Nginx will break streaming with certain applications (notably the Pi coding harness).
Nginx docs for reference

Requirements

* server: add "X-Accel-Buffering": "no" header to streaming endpoints

This header tells Nginx (as a reverse proxy) to NOT buffer responses. (only affects streaming endpoints)
Without it, Nginx will break streaming with certain applications (notably the Pi coding harness).
@regunakyle regunakyle requested a review from a team as a code owner June 18, 2026 18:14
@pnb

pnb commented Jun 18, 2026

Copy link
Copy Markdown
Contributor

This is great! I haven't had buffering actually break anything yet, but it was annoying me a bit because the text comes out of the server quite chunky and harder to read versus a smooth stream. I switched computers, internet connections, and reverse proxies at the same time, so I had assumed it was the internet or computer rather than switching from Apache to Nginx, but in fact this fixes it.

@ServeurpersoCom

ServeurpersoCom commented Jun 18, 2026

Copy link
Copy Markdown
Contributor

It costs nothing to add it. I haven't encountered this issue with Apache2/pi
well-known header in nginx, and the line is in the right place.

@ngxson

ngxson commented Jun 18, 2026

Copy link
Copy Markdown
Collaborator

not 100% sure but will this be set one more time in router mode ?

@ServeurpersoCom

ServeurpersoCom commented Jun 18, 2026

Copy link
Copy Markdown
Contributor

not 100% sure but will this be set one more time in router mode ?

I check if it passes through!
That reminds me of the issue with the duplicate headers that we fixed.

@ServeurpersoCom

Copy link
Copy Markdown
Contributor

OK, on router, it works, and only one at that. Just note that it will apply to all endpoints.

@ngxson

ngxson commented Jun 18, 2026

Copy link
Copy Markdown
Collaborator

I mean, does the router add the same header once again? (i.e. duplicated header)

@ngxson ngxson left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nvm, headers is a map, so it's not possible to have duplicated keys

@ServeurpersoCom

Copy link
Copy Markdown
Contributor

I meant, no duplicates.

@ServeurpersoCom ServeurpersoCom merged commit 40f3aaf into ggml-org:master Jun 18, 2026
1 check passed
@regunakyle

regunakyle commented Jun 19, 2026

Copy link
Copy Markdown
Contributor Author

OK, on router, it works, and only one at that. Just note that it will apply to all endpoints.

Pretty sure only streaming endpoints are affected. Non-streaming endpoints like /props and /models are not affected.

BTW as a note, Nginx will strip out the X-Accel-Buffering header before sending the proxied response back to the user.

papamoose pushed a commit to papamoose/llama.cpp that referenced this pull request Jun 27, 2026
…gml-org#24774)

* server: add "X-Accel-Buffering": "no" header to streaming endpoints

This header tells Nginx (as a reverse proxy) to NOT buffer responses. (only affects streaming endpoints)
Without it, Nginx will break streaming with certain applications (notably the Pi coding harness).
adrianhoehne pushed a commit to adrianhoehne/llama.cpp that referenced this pull request Jul 5, 2026
…gml-org#24774)

* server: add "X-Accel-Buffering": "no" header to streaming endpoints

This header tells Nginx (as a reverse proxy) to NOT buffer responses. (only affects streaming endpoints)
Without it, Nginx will break streaming with certain applications (notably the Pi coding harness).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants