Skip to content

fix(frontend): close HTTP/1.x connections rejected during drain - #17101

Merged
jthomson04 merged 3 commits into
ai-dynamo:mainfrom
xianlubird:bug/http1-drain-close
Oct 27, 2026
Merged

jthomson04 merged 3 commits into
ai-dynamo:mainfrom
xianlubird:bug/http1-drain-close

Conversation

@xianlubird

@xianlubird xianlubird commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

Summary

While a frontend waits for an admitted response to finish, another HTTP/1.x keepalive connection can keep sending inference requests to the draining process. Those requests receive 503, but the connection remains reusable until connection-level shutdown starts.

Add Connection: close to HTTP/1.x responses rejected by either drain admission check, preserving the request version on the response so HTTP/1.0 keep-alive connections also close after the rejection. Preserve the existing error body/status, admitted response streams, and HTTP/2 behavior. This prevents repeated connection reuse; it does not retry the initial 503.

The regression sends HTTP/1.1 and HTTP/1.0 keep-alive requests over real TCP connections while a separate admitted SSE response holds drain open, checks each 503, connection-close header and EOF, then releases the SSE through [DONE].

Fixes #17100. Part of #15558 (scenario E). Complementary to #16520's endpoint propagation delay and #17064's runtime/WebSocket shutdown work.

Validation

  • Removing the close header makes the HTTP/1.1 case fail waiting for EOF; removing the response-version assignment makes the HTTP/1.0 case fail waiting for EOF.
  • cargo test -p dynamo-llm --no-default-features --test http-service: 34 passed on macOS.
  • cargo fmt --all -- --check.
  • cargo clippy -p dynamo-llm --no-default-features --lib --test http-service --no-deps -- -D warnings.
  • Repository pre-commit checks for the two changed Rust files.

Protocol references: HTTP/1.1 connection teardown and HTTP/2 connection-specific header prohibition.

Summary by CodeRabbit

  • Bug Fixes
    • Requests received while the service is draining now get a 503 Service Unavailable response. For HTTP/1.1, the response also signals that the connection will close; HTTP/2 responses do not include that header.
    • Requests already admitted before draining begins can continue until they finish, while new requests are rejected.

Signed-off-by: xianlubird <xianlubird@gmail.com>
@copy-pr-bot

copy-pr-bot Bot commented Oct 22, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@xianlubird
xianlubird temporarily deployed to external_collaborator October 22, 2026 12:20 — with GitHub Actions Inactive
@xianlubird
xianlubird temporarily deployed to external_collaborator October 22, 2026 12:20 — with GitHub Actions Inactive
@github-actions github-actions Bot added external-contribution Pull request is from an external contributor trusted-contributor Org-External user who is trusted to run CI without Org-member approval fix labels Oct 22, 2026
@dynamo-ops

Copy link
Copy Markdown
Contributor

/ok to test 98f4d72

@github-actions github-actions Bot added the frontend `python -m dynamo.frontend` and `dynamo-run in=http|text|grpc` label Oct 22, 2026
@xianlubird
xianlubird marked this pull request as ready for review October 22, 2026 12:24
@xianlubird
xianlubird requested review from a team as code owners October 22, 2026 12:24
@coderabbitai

coderabbitai Bot commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

Review in Change Stack →

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration
📥 Commits

Reviewing files that changed from the base of the PR and between 58e83f6 and 98f4d72.

📒 Files selected for processing: 2
  • lib/llm/src/http/service/service_v2.rs
  • lib/llm/tests/http-service.rs

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.

Walkthrough

Draining inference admission failures now use a shared 503 response. HTTP/1.1 responses include Connection: close, while HTTP/2 responses do not. A regression test checks both protocol responses and verifies that an admitted stream completes during draining.

Changes

Drain admission responses

Layer / File(s) Summary
Drain response and validation
lib/llm/src/http/service/service_v2.rs, lib/llm/tests/http-service.rs
Both draining checks use the shared service-unavailable response helper. The regression test checks HTTP/1.1 connection closure, HTTP/2 response headers, and completion of an already-admitted stream.

Priority: ➖ Normal

Estimated code review effort: 3 (Moderate) | ~20 minutes

Severity of issue fixed: Medium

Merge Risk: ⚪ Minimal · up to 98f4d

The drain-response change appears mergeable after normal checks; no specific remaining issue is identified.

Pre-merge checks: Passed 4 · Failed 1

Failed Failed checks: 1 warning

Check name Status Explanation Resolution
Docstring Coverage Warning Docstring coverage is 66.67% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 6 functions across 2 files. Write docstrings for the functions missing them to satisfy the coverage threshold.
Passed Passed checks: 4 passed
Check name Status Explanation
Linked Issues check Passed The change meets the coding requirements in [#17100]. draining_inference_response preserves the existing service-unavailable response and adds Connection: close only for Version::HTTP_11. Both a…
Out of Scope Changes check Passed The reviewed changes are limited to the drain rejection response and its regression test. They support [#17100] by testing connection closure, admitted stream preservation, and HTTP/2 behavior. No end…
Title check Passed The title "fix(frontend): close HTTP/1.x connections rejected during drain" names the main change: HTTP/1.x connections rejected by the drain admission checks now receive Connection: close. It matches…
Description check Passed The description is complete and relevant. It explains the problem, implementation, scope, regression coverage, validation commands, and related issues. It includes a valid issue reference through “Fix…
Full details: Linked Issues check

Explanation

The change meets the coding requirements in [#17100]. draining_inference_response preserves the existing service-unavailable response and adds Connection: close only for Version::HTTP_11. Both admission checks in track_inflight_inference use this response. The supplied regression summary reports a real reused TCP connection, HTTP/1.1 503 and EOF, continued admitted SSE handling through [DONE], and HTTP/2 503 without a connection-specific header. The supplied validation reports the HTTP service test passed.

Full details: Out of Scope Changes check

Explanation

The reviewed changes are limited to the drain rejection response and its regression test. They support [#17100] by testing connection closure, admitted stream preservation, and HTTP/2 behavior. No endpoint propagation delay or runtime/WebSocket shutdown ordering change is shown.

Full details: Title check

Explanation

The title "fix(frontend): close HTTP/1.x connections rejected during drain" names the main change: HTTP/1.x connections rejected by the drain admission checks now receive Connection: close. It matches the raw summary (Connection: close on 503 responses during draining for HTTP/1.1, and HTTP/1.0 per the PR description) and the linked issue #17100. It is specific and concise, and it follows a conventional commit style.

Full details: Description check

Explanation

The description is complete and relevant. It explains the problem, implementation, scope, regression coverage, validation commands, and related issues. It includes a valid issue reference through “Fixes #17100.” The template’s explicit “Where should the reviewer start?” section is not included, but the changed areas are clear from the description and validation details.

✨ Finishing Touches: 💡 1
🛠️ Fix failing CI checks: 💡
  • Commit to this branch
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Comment @coderabbitai help to get the list of available commands.

@devin-ai-integration

devin-ai-integration Bot commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

✅ Dynamo PR CI passed — run 39892693998 (attempt 1) on 72ca3695f5

Gate checks: ✅ backend-status-check · ✅ deploy-status-check · ✅ dynamo-status-check

Posted automatically by Devin for run 39892693998. Updated on every full-CI run of this PR.

@jthomson04 jthomson04 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fix looks right: hyper's h1 encoder treats Connection: close as end-of-keep-alive, so the connection closes after the 503. Two small things inline.

Comment thread lib/llm/src/http/service/service_v2.rs Resolved
Comment thread lib/llm/tests/http-service.rs Resolved

@dmitry-tokarev-nv dmitry-tokarev-nv left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No P0 or P1 found. This review read head 98f4d72b15419f0c9066e522b604aa2c7bdc4aed and adds no new inline finding.

Open items from another reviewer:

  • service_v2.rs:103, HTTP/1.0 keep-alive. This review measured it: version <= Version::HTTP_11 alone does not close the connection on hyper 1.10.1. The reply on that thread has a tested change.
  • http-service.rs:1137, trim the test. This review did not test it.
Tests: the new test passed 210 of 210 runs, and two deliberate breaks of the new code make it fail.

The tests ran in a container from the dynamo-runtime-test CI image of this head, on x86_64 Linux, with --no-default-features and no network.

  • Full http-service suite: 33 passed at the base 5df849a06a and 34 passed at the head, twice each.
  • The new test passed 30 of 30 runs alone, at 0.05 s each.
  • Under load, it passed 120 of 120 runs (4 concurrent copies of the suite, 30 times). It also passed 60 of 60 runs in a container limited to 2 CPUs. There, the suite took 6.8 s to 7.2 s instead of 3.1 s. The 1 s wait for EOF has wide headroom.
  • A deliberate break of the new code makes the test fail at http-service.rs:1120 with draining HTTP/1.1 connection stayed open after rejection. One break removes the header insert, and one changes the check to Version::HTTP_2.
What this review measured.

The diff matches the description. Both drain checks in track_inflight_inference now return draining_inference_response, which adds Connection: close for HTTP/1.1 only. The status and the body do not change.

Routes behind track_inflight_inference at the head: chat, completions, embeddings, classify, rerank, pooling, images, videos, audios, realtime (WebSocket), and responses. When they are enabled, also batch, Anthropic messages, and the vLLM and SGLang generate routes.

Other 503 responses during drain that do not send Connection: close:

  • /health and the model readiness route. They are probes, outside the inference routes.
  • The check_ready call in each inference handler, and the readiness check in realtime_ws_handler. If drain starts after the middleware check passed, they return 503.
  • The batch proxy returns 503 after state.cancel_token() fires. That token fires only after the server stops.

In-memory probe: the real router from HttpService::builder(), served over tokio::io::duplex the same way as serve_io in server.rs, after start_draining(). The probe waits 500 ms for EOF. It shows the header and close behavior through hyper, but not TCP behavior such as a reset.

request base 5df849a06a head
HTTP/1.1 POST /v1/chat/completions 503, open 503, connection: close, EOF
two pipelined HTTP/1.1 POSTs two 503, open one 503, connection: close, EOF
HTTP/1.1 POST, Content-Length: 100000, 10 body bytes sent 503, no connection header, EOF 503, connection: close, EOF
HTTP/1.0 POST, Connection: keep-alive 503, connection: keep-alive, open same
GET /live and GET /health 200 and 503, open same

Controls before drain: GET /live stays open, and GET /live with Connection: close gets EOF.

How hyper 1.10.1 closes the connection: a connection: close response header sets is_last (proto/h1/role.rs:822-825). The writer then ends in Writing::Closed (conn.rs:595-604, conn.rs:789-790). try_keep_alive closes the state (conn.rs:1086), and the dispatcher shuts down the socket (dispatch.rs:157). hyper reads a new request head only from Reading::Init (conn.rs:175-178), so a pipelined request behind the 503 gets no response. hyper drains an unread request body once. If the body is not complete, hyper stops reading and closes the connection (conn.rs:849-865). The base already closed in that case, without the header.

Lint: cargo clippy -p dynamo-llm --no-default-features --lib --test http-service --no-deps -- -D warnings and rustfmt --check on the two files pass at the head.

Merge: the head merges with main at ce67fe87df without a conflict. The drain code in the merged tree is the same as in the head, and its http-service test target builds with 34 tests. The merged tree did not run. #16520 (head 1a6537da) conflicts at service_v2.rs:121, because it changes is_ready() to accepts_inference() in the same check. The resolution keeps accepts_inference() and the new helper. #17064 (head 14637d38) merges without a conflict and does not change the drain checks.

CI on this head: the Pre Merge job rust-tests (.) ran the new test once on Linux with the default features. It used cargo test --locked --all-targets on a merge with 58e83f6d2b, and 34 tests passed. The PR run 39777600970 passed. Docs link check failed on a 404 for github.com/Spycsh in dynamo-v1-5-0.mdx, which this PR does not change. The three skipped Unit tests jobs belong to the PR Issue Link workflow and run only on pull_request events.

@dmitry-tokarev-nv dmitry-tokarev-nv left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Head 98f4d72b15419f0c9066e522b604aa2c7bdc4aed. No P0, P1, P2 or P3 finding from this review.

Measured: an in-memory hyper probe of the drained router at the base and the head. At the head, an HTTP/1.1 drain 503 carries connection: close and the connection closes. A pipelined request behind it gets no response. An unread large body closed the connection at the base too. clippy -D warnings (--no-deps) and rustfmt --check pass. The head merges with main at ce67fe87df without a conflict.

Open items: the two threads by jthomson04. For HTTP/1.0, <= alone does not close the connection on hyper 1.10.1, and the reply on that thread has a tested change. The test-trim thread is not tested here.

Tests, in a container from the CI image of this head with --no-default-features: base 33 passed and head 34 passed, twice each. The new test passed 30 of 30 runs alone, 120 of 120 under 4-way load, and 60 of 60 with 2 CPUs. Two deliberate breaks of the new code make it fail. #16520 will conflict at service_v2.rs:121.

@xianlubird xianlubird changed the title fix(frontend): close HTTP/1.1 connections rejected during drain fix(frontend): close HTTP/1.x connections rejected during drain Oct 23, 2026
Signed-off-by: xianlubird <xianlubird@gmail.com>
@xianlubird
xianlubird temporarily deployed to external_collaborator October 23, 2026 03:00 — with GitHub Actions Inactive
@dynamo-ops

Copy link
Copy Markdown
Contributor

/ok to test 72ca369

@jthomson04 jthomson04 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Both comments are addressed: HTTP/1.0 keep-alive connections now close too, thanks to the response-version fix, and the test is trimmed to the drain path.

@xianlubird
xianlubird temporarily deployed to external_collaborator October 27, 2026 02:15 — with GitHub Actions Inactive
@dynamo-ops

Copy link
Copy Markdown
Contributor

/ok to test a273740

@jthomson04
jthomson04 enabled auto-merge (squash) October 27, 2026 02:50
@jthomson04
jthomson04 merged commit 492108c into ai-dynamo:main Oct 27, 2026
118 checks passed
@xianlubird
xianlubird deleted the bug/http1-drain-close branch October 27, 2026 03:45
AllenXu93 added a commit to AllenXu93/dynamo that referenced this pull request Oct 27, 2026
Resolve the service_v2 admission conflict with ai-dynamo#17101 by keeping
Quiescing inference admission (accepts_inference) and applying
Connection: close only when a request is rejected during drain.

Signed-off-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: AllenXu93 <AllenXu93@users.noreply.github.com>

This branch was previously deployed

1 inactive deployment
external_collaborator — a2737403 Deployed Oct 27, 2026 by xianlubird via ok-to-test #39252
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

external-contribution Pull request is from an external contributor fix frontend `python -m dynamo.frontend` and `dynamo-run in=http|text|grpc` size/L trusted-contributor Org-External user who is trusted to run CI without Org-member approval

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(frontend): close HTTP/1.1 connections rejected during drain

4 participants