With multiple workers, SIGTERM kills in-flight requests because the master terminates busy workers once the first idle worker exits #386

Summary

With --workers > 1, a SIGTERM (e.g. docker stop / Kubernetes pod termination) does not shut down
gracefully. In-flight requests are cut off almost immediately ("Empty reply from server"). With -w 1 the same
request completes normally.

Cause

In hypercorn/run.py, the shutdown signal handler sets shutdown_event and active = False. The master is
blocked in:

wait(process.sentinel for process in processes)

multiprocessing.connection.wait returns as soon as any worker exits. Idle workers notice the shutdown
event and exit right away, so wait() returns while other workers are still serving requests. Since active
is now False, the loop ends and the master runs:

for process in processes:
    process.terminate()

The code is the same in 0.17.3, 0.18.0 and current main.

Reproduction

Versions: hypercorn 0.17.3 (also checked 0.18.0 source), Python 3.12.13, Linux (python:3.12.13-alpine Docker
image), asyncio worker, Quart 0.20.0 as the app.

app.py:

import asyncio
from quart import Quart, request

app = Quart(__name__)


@app.get("/slow")
async def slow():
    seconds = float(request.args.get("s", 10))
    await asyncio.sleep(seconds)

    return f"done after {seconds}s\n"

Dockerfile:

FROM python:3.12.13-alpine
RUN pip install --no-cache-dir hypercorn==0.17.3 quart==0.20.0
WORKDIR /srv
COPY app.py /srv/

docker build -t hc-test .
docker run -d --name hc -p 18080:8080 hc-test hypercorn -b 0.0.0.0:8080 -w 3 app:app
sleep 3
curl -s -w "http=%{http_code} time=%{time_total}s %{errormsg}\n" "localhost:18080/slow?s=10" &
sleep 1
docker stop -t 30 hc   # SIGTERM to PID 1 (the hypercorn master), SIGKILL after 30s
wait

Results

Swapping that single terminate() call fixes the problem, which confirms the cause.

Expected behaviour

On SIGTERM/SIGINT, the master should wait for all workers to finish draining (each one is already bounded by
its own graceful shutdown) before terminating anything. Only as a fallback, e.g. after graceful_timeout, should
it call terminate().

Possible fix

After the main loop ends because of a shutdown signal, join the remaining workers instead of terminating them
right away, for example:

for process in processes:
    process.join(timeout=config.graceful_timeout)

for process in processes:
    if process.is_alive():
        process.terminate()

On Python 3.12+ the worker-side drain is effectively unbounded because of Server.wait_closed(), as noted in
#308. The timeout used here may therefore need to differ from graceful_timeout, or that issue needs fixing first.

Related

• It seems hypercorn is not closing connections properly #241: in-flight connections reset on shutdown in multi-process mode but not with -w 0 (likely the same root
  cause)
• graceful_timeout not respected on Python 3.12 #308: graceful_timeout not respected on Python 3.12