Skip to content

fix(runtime): [cherry-pick 1.7.0] bound HTTP and WebSocket draining at shutdown (#17064) - #18887

Open
biswapanda wants to merge 1 commit into
release/1.7.0from
bis/cherry-pick-17064-release-1.7.0
Open

biswapanda wants to merge 1 commit into
release/1.7.0from
bis/cherry-pick-17064-release-1.7.0

Conversation

@biswapanda

@biswapanda biswapanda commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Overview

Cherry-pick #17064 to release/1.7.0. Frontend shutdown now bounds HTTP and WebSocket draining, preserving backend connections during the drain and attempting a 1012 Service Restart close at the deadline.

Key Changes

47 added core-logic lines in the original change, excluding tests, comments, and whitespace.

  • Track HTTP and gRPC draining and isolate service cancellation.
  • Bound WebSocket cleanup within the tracked connection task while preserving upstream session recovery.
  • Count TCP requests before releasing the handler-map guard.

Validation

The cherry-pick applied cleanly. cargo fmt --all -- --check and git diff --check HEAD^ HEAD passed. Tests have not been run on the release branch. The original PR reports targeted coverage for HTTP shutdown deadlines, WebSocket restart closes, recovery, and TCP draining.

Issues and Ongoing Work

Linear: DIS-3411.

Refs #15558. Cherry-picks #17064, which builds on #16728 and preserves #17014's recovery behavior. #16520 separately addresses endpoint propagation before draining. No release PR referencing #17064 was found in the GitHub search.

Signed-off-by: neilkg <33635204+neilkg@users.noreply.github.com>
@biswapanda
biswapanda requested review from a team as code owners November 17, 2026 00:30
@github-actions github-actions Bot added fix frontend `python -m dynamo.frontend` and `dynamo-run in=http|text|grpc` labels Nov 17, 2026
std::future::pending::<()>().await;
})
}
})).layer(axum::middleware::from_fn_with_state(state.clone(), track_inflight_inference));

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The new test supplies Arc to from_fn_with_state, but track_inflight_inference extracts State<Arc>. These types do not match, so the middleware does not implement the required Service trait and lib/llm unit tests cannot compile. Wrap state in InferenceReadinessState with an empty decision_dialects map before attaching this middleware.

🤖 AI Fix

Pass an Arc containing state.clone() and an empty decision_dialects map to from_fn_with_state.

std::future::pending::<()>().await;
})
}
})).layer(axum::middleware::from_fn_with_state(state.clone(), track_inflight_inference));

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The new test supplies Arc<State> to middleware whose extractor requires State<Arc<InferenceReadinessState>>. Since state_clone() returns Arc<State>, this layer does not satisfy Axum's Service bounds, preventing the test target from compiling. Wrap the state in InferenceReadinessState with an empty decision_dialects map.

🤖 AI Fix

Pass Arc::new(InferenceReadinessState { state: state.clone(), decision_dialects: HashMap::new() }) as the middleware state.

tokio::time::timeout(Duration::from_secs(2), entered.cancelled()).await.unwrap();
let response = if before_headers { None } else { Some(request.take().unwrap().await.unwrap().unwrap()) };
cancel.cancel();
if let Some(response) = response {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The response-body case can pass without shutdown interrupting the stream: the client's four-second timeout makes response.bytes() return an error and disconnects, releasing the body before the three-second server timeout even starts. Start the server-completion timeout immediately after cancellation, before awaiting the body, so this case distinguishes deadline enforcement from client cleanup.

🤖 AI Fix

Move the bounded server-completion wait before the response-body read so the client timeout cannot satisfy the shutdown assertion.

@devin-ai-integration

Copy link
Copy Markdown
Contributor

✅ Dynamo PR CI passed — run 43988159834 (attempt 1) on aab633170e

Gate checks: ✅ backend-status-check · ✅ deploy-status-check · ✅ dynamo-status-check

Posted automatically by Devin for run 43988159834. Updated on every full-CI run of this PR.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fix frontend `python -m dynamo.frontend` and `dynamo-run in=http|text|grpc` size/L

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants