You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Chat request hangs when the chat worker dies before it is ready #534
POST /api/chat/completions can wait forever when the chat worker subprocess dies before it reports ready. The client sees only SSE heartbeats, with no error, until its own timeout.
Cause
WarmPool.acquire() (server/chat/worker.py) calls drain_until_ready() with no timeout when it has to cold-start a worker. drain_until_ready() blocks on self._out_q.get(timeout=None). If the child exits during startup (import error, bad environment, missing model config, OOM), the ready message never arrives and nothing checks is_alive(). events() already handles a dead worker; acquire() and the warm-spawn path don't.
Reproduce
Make the worker child fail during startup. We hit it with a launcher script that lacked an if __name__ == "__main__": guard: the spawn child re-imported the launcher, tried to bind the server port again, and exited. A chat request then streamed heartbeats for 900 s, until the client timed out. Any startup exception in the child gives the same result.
Expected
The request fails fast with an error event, for example "chat worker failed to start: ", and the pool spawns a replacement.
Fix direction
Poll the queue with a bounded timeout in a loop that checks self._proc.is_alive(), and return False or raise when the child has exited.
Have the child send an error message from a top-level try around its imports, so the parent gets the reason.
Optionally cap total startup time with a setting.
Related: open PR #503 notes the same gap in a code comment ("The pool waits for READY with no timeout and no liveness check, so a worker that died while warming … would leave that wait blocked for good") and moves its new prewarm step after READY to avoid it. That avoids the hang for the new step only: a failure during imports, before READY, still blocks acquire().
I'd like to take this one. Plan: poll the queue in drain_until_ready() and check is_alive() between polls, record the init error (or the exit code when the child dies without sending one), and have events() raise it so the router streams an error event. Releasing the dead worker then spawns a replacement. Branch is ready; PR shortly.
Activity
sandeepvijayarao09 commentedon Nov 16, 2026
I'd like to take this one. Plan: poll the queue in
drain_until_ready()and checkis_alive()between polls, record the init error (or the exit code when the child dies without sending one), and haveevents()raise it so the router streams an error event. Releasing the dead worker then spawns a replacement. Branch is ready; PR shortly.