Repository navigation
fix(frontend): close HTTP/1.x connections rejected during drain - #17101
Conversation
Signed-off-by: xianlubird <xianlubird@gmail.com>
|
/ok to test 98f4d72 |
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info
WalkthroughDraining inference admission failures now use a shared 503 response. HTTP/1.1 responses include ChangesDrain admission responses
Priority: ➖ Normal Estimated code review effort: 3 (Moderate) | ~20 minutes Severity of issue fixed: Medium Merge Risk: ⚪ Minimal · up to
|
✅ Dynamo PR CI passed — run 39892693998 (attempt 1) on
|
jthomson04
left a comment
There was a problem hiding this comment.
Fix looks right: hyper's h1 encoder treats Connection: close as end-of-keep-alive, so the connection closes after the 503. Two small things inline.
dmitry-tokarev-nv
left a comment
There was a problem hiding this comment.
No P0 or P1 found. This review read head 98f4d72b15419f0c9066e522b604aa2c7bdc4aed and adds no new inline finding.
Open items from another reviewer:
service_v2.rs:103, HTTP/1.0 keep-alive. This review measured it:version <= Version::HTTP_11alone does not close the connection on hyper 1.10.1. The reply on that thread has a tested change.http-service.rs:1137, trim the test. This review did not test it.
Tests: the new test passed 210 of 210 runs, and two deliberate breaks of the new code make it fail.
The tests ran in a container from the dynamo-runtime-test CI image of this head, on x86_64 Linux, with --no-default-features and no network.
- Full
http-servicesuite: 33 passed at the base5df849a06aand 34 passed at the head, twice each. - The new test passed 30 of 30 runs alone, at 0.05 s each.
- Under load, it passed 120 of 120 runs (4 concurrent copies of the suite, 30 times). It also passed 60 of 60 runs in a container limited to 2 CPUs. There, the suite took 6.8 s to 7.2 s instead of 3.1 s. The 1 s wait for EOF has wide headroom.
- A deliberate break of the new code makes the test fail at
http-service.rs:1120withdraining HTTP/1.1 connection stayed open after rejection. One break removes the header insert, and one changes the check toVersion::HTTP_2.
What this review measured.
The diff matches the description. Both drain checks in track_inflight_inference now return draining_inference_response, which adds Connection: close for HTTP/1.1 only. The status and the body do not change.
Routes behind track_inflight_inference at the head: chat, completions, embeddings, classify, rerank, pooling, images, videos, audios, realtime (WebSocket), and responses. When they are enabled, also batch, Anthropic messages, and the vLLM and SGLang generate routes.
Other 503 responses during drain that do not send Connection: close:
/healthand the model readiness route. They are probes, outside the inference routes.- The
check_readycall in each inference handler, and the readiness check inrealtime_ws_handler. If drain starts after the middleware check passed, they return 503. - The batch proxy returns 503 after
state.cancel_token()fires. That token fires only after the server stops.
In-memory probe: the real router from HttpService::builder(), served over tokio::io::duplex the same way as serve_io in server.rs, after start_draining(). The probe waits 500 ms for EOF. It shows the header and close behavior through hyper, but not TCP behavior such as a reset.
| request | base 5df849a06a |
head |
|---|---|---|
HTTP/1.1 POST /v1/chat/completions |
503, open | 503, connection: close, EOF |
| two pipelined HTTP/1.1 POSTs | two 503, open | one 503, connection: close, EOF |
HTTP/1.1 POST, Content-Length: 100000, 10 body bytes sent |
503, no connection header, EOF |
503, connection: close, EOF |
HTTP/1.0 POST, Connection: keep-alive |
503, connection: keep-alive, open |
same |
GET /live and GET /health |
200 and 503, open | same |
Controls before drain: GET /live stays open, and GET /live with Connection: close gets EOF.
How hyper 1.10.1 closes the connection: a connection: close response header sets is_last (proto/h1/role.rs:822-825). The writer then ends in Writing::Closed (conn.rs:595-604, conn.rs:789-790). try_keep_alive closes the state (conn.rs:1086), and the dispatcher shuts down the socket (dispatch.rs:157). hyper reads a new request head only from Reading::Init (conn.rs:175-178), so a pipelined request behind the 503 gets no response. hyper drains an unread request body once. If the body is not complete, hyper stops reading and closes the connection (conn.rs:849-865). The base already closed in that case, without the header.
Lint: cargo clippy -p dynamo-llm --no-default-features --lib --test http-service --no-deps -- -D warnings and rustfmt --check on the two files pass at the head.
Merge: the head merges with main at ce67fe87df without a conflict. The drain code in the merged tree is the same as in the head, and its http-service test target builds with 34 tests. The merged tree did not run. #16520 (head 1a6537da) conflicts at service_v2.rs:121, because it changes is_ready() to accepts_inference() in the same check. The resolution keeps accepts_inference() and the new helper. #17064 (head 14637d38) merges without a conflict and does not change the drain checks.
CI on this head: the Pre Merge job rust-tests (.) ran the new test once on Linux with the default features. It used cargo test --locked --all-targets on a merge with 58e83f6d2b, and 34 tests passed. The PR run 39777600970 passed. Docs link check failed on a 404 for github.com/Spycsh in dynamo-v1-5-0.mdx, which this PR does not change. The three skipped Unit tests jobs belong to the PR Issue Link workflow and run only on pull_request events.
dmitry-tokarev-nv
left a comment
There was a problem hiding this comment.
Head 98f4d72b15419f0c9066e522b604aa2c7bdc4aed. No P0, P1, P2 or P3 finding from this review.
Measured: an in-memory hyper probe of the drained router at the base and the head. At the head, an HTTP/1.1 drain 503 carries connection: close and the connection closes. A pipelined request behind it gets no response. An unread large body closed the connection at the base too. clippy -D warnings (--no-deps) and rustfmt --check pass. The head merges with main at ce67fe87df without a conflict.
Open items: the two threads by jthomson04. For HTTP/1.0, <= alone does not close the connection on hyper 1.10.1, and the reply on that thread has a tested change. The test-trim thread is not tested here.
Tests, in a container from the CI image of this head with --no-default-features: base 33 passed and head 34 passed, twice each. The new test passed 30 of 30 runs alone, 120 of 120 under 4-way load, and 60 of 60 with 2 CPUs. Two deliberate breaks of the new code make it fail. #16520 will conflict at service_v2.rs:121.
Signed-off-by: xianlubird <xianlubird@gmail.com>
|
/ok to test 72ca369 |
A new commit (72ca369) arrived after this approval. I am withdrawing it until that commit is reviewed.
jthomson04
left a comment
There was a problem hiding this comment.
LGTM. Both comments are addressed: HTTP/1.0 keep-alive connections now close too, thanks to the response-version fix, and the test is trimmed to the drain path.
|
/ok to test a273740 |
Resolve the service_v2 admission conflict with ai-dynamo#17101 by keeping Quiescing inference admission (accepts_inference) and applying Connection: close only when a request is rejected during drain. Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: AllenXu93 <AllenXu93@users.noreply.github.com>
Summary
While a frontend waits for an admitted response to finish, another HTTP/1.x keepalive connection can keep sending inference requests to the draining process. Those requests receive 503, but the connection remains reusable until connection-level shutdown starts.
Add
Connection: closeto HTTP/1.x responses rejected by either drain admission check, preserving the request version on the response so HTTP/1.0 keep-alive connections also close after the rejection. Preserve the existing error body/status, admitted response streams, and HTTP/2 behavior. This prevents repeated connection reuse; it does not retry the initial 503.The regression sends HTTP/1.1 and HTTP/1.0 keep-alive requests over real TCP connections while a separate admitted SSE response holds drain open, checks each 503, connection-close header and EOF, then releases the SSE through
[DONE].Fixes #17100. Part of #15558 (scenario E). Complementary to #16520's endpoint propagation delay and #17064's runtime/WebSocket shutdown work.
Validation
cargo test -p dynamo-llm --no-default-features --test http-service: 34 passed on macOS.cargo fmt --all -- --check.cargo clippy -p dynamo-llm --no-default-features --lib --test http-service --no-deps -- -D warnings.Protocol references: HTTP/1.1 connection teardown and HTTP/2 connection-specific header prohibition.
Summary by CodeRabbit