Skip to content

Chat request hangs when the chat worker dies before it is ready #534

Open

Description

@cafzal

Summary

POST /api/chat/completions can wait forever when the chat worker subprocess dies before it reports ready. The client sees only SSE heartbeats, with no error, until its own timeout.

Cause

WarmPool.acquire() (server/chat/worker.py) calls drain_until_ready() with no timeout when it has to cold-start a worker. drain_until_ready() blocks on self._out_q.get(timeout=None). If the child exits during startup (import error, bad environment, missing model config, OOM), the ready message never arrives and nothing checks is_alive(). events() already handles a dead worker; acquire() and the warm-spawn path don't.

Reproduce

Make the worker child fail during startup. We hit it with a launcher script that lacked an if __name__ == "__main__": guard: the spawn child re-imported the launcher, tried to bind the server port again, and exited. A chat request then streamed heartbeats for 900 s, until the client timed out. Any startup exception in the child gives the same result.

Expected

The request fails fast with an error event, for example "chat worker failed to start: ", and the pool spawns a replacement.

Fix direction

  • Poll the queue with a bounded timeout in a loop that checks self._proc.is_alive(), and return False or raise when the child has exited.
  • Have the child send an error message from a top-level try around its imports, so the parent gets the reason.
  • Optionally cap total startup time with a setting.

Related: open PR #503 notes the same gap in a code comment ("The pool waits for READY with no timeout and no liveness check, so a worker that died while warming … would leave that wait blocked for good") and moves its new prewarm step after READY to avoid it. That avoids the hang for the new step only: a failure during imports, before READY, still blocks acquire().

Activity

  1. sandeepvijayarao09 commented on Nov 16, 2026

    @sandeepvijayarao09

    I'd like to take this one. Plan: poll the queue in drain_until_ready() and check is_alive() between polls, record the init error (or the exit code when the child dies without sending one), and have events() raise it so the router streams an error event. Releasing the dead worker then spawns a replacement. Branch is ready; PR shortly.

  2. added
    performanceLatency, cost, throughput, and runtime reliability
    bugSomething isn't working
    on Nov 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingperformanceLatency, cost, throughput, and runtime reliability

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions