Repository navigation
fix(runtime): [cherry-pick 1.7.0] bound HTTP and WebSocket draining at shutdown (#17064) - #18887
biswapanda wants to merge 1 commit into
Conversation
Signed-off-by: neilkg <33635204+neilkg@users.noreply.github.com>
| std::future::pending::<()>().await; | ||
| }) | ||
| } | ||
| })).layer(axum::middleware::from_fn_with_state(state.clone(), track_inflight_inference)); |
There was a problem hiding this comment.
The new test supplies Arc to from_fn_with_state, but track_inflight_inference extracts State<Arc>. These types do not match, so the middleware does not implement the required Service trait and lib/llm unit tests cannot compile. Wrap state in InferenceReadinessState with an empty decision_dialects map before attaching this middleware.
🤖 AI Fix
Pass an Arc containing state.clone() and an empty decision_dialects map to from_fn_with_state.
| std::future::pending::<()>().await; | ||
| }) | ||
| } | ||
| })).layer(axum::middleware::from_fn_with_state(state.clone(), track_inflight_inference)); |
There was a problem hiding this comment.
The new test supplies Arc<State> to middleware whose extractor requires State<Arc<InferenceReadinessState>>. Since state_clone() returns Arc<State>, this layer does not satisfy Axum's Service bounds, preventing the test target from compiling. Wrap the state in InferenceReadinessState with an empty decision_dialects map.
🤖 AI Fix
Pass Arc::new(InferenceReadinessState { state: state.clone(), decision_dialects: HashMap::new() }) as the middleware state.
| tokio::time::timeout(Duration::from_secs(2), entered.cancelled()).await.unwrap(); | ||
| let response = if before_headers { None } else { Some(request.take().unwrap().await.unwrap().unwrap()) }; | ||
| cancel.cancel(); | ||
| if let Some(response) = response { |
There was a problem hiding this comment.
The response-body case can pass without shutdown interrupting the stream: the client's four-second timeout makes response.bytes() return an error and disconnects, releasing the body before the three-second server timeout even starts. Start the server-completion timeout immediately after cancellation, before awaiting the body, so this case distinguishes deadline enforcement from client cleanup.
🤖 AI Fix
Move the bounded server-completion wait before the response-body read so the client timeout cannot satisfy the shutdown assertion.
✅ Dynamo PR CI passed — run 43988159834 (attempt 1) on
|
Overview
Cherry-pick #17064 to release/1.7.0. Frontend shutdown now bounds HTTP and WebSocket draining, preserving backend connections during the drain and attempting a 1012 Service Restart close at the deadline.
Key Changes
47 added core-logic lines in the original change, excluding tests, comments, and whitespace.
Validation
The cherry-pick applied cleanly.
cargo fmt --all -- --checkandgit diff --check HEAD^ HEADpassed. Tests have not been run on the release branch. The original PR reports targeted coverage for HTTP shutdown deadlines, WebSocket restart closes, recovery, and TCP draining.Issues and Ongoing Work
Linear: DIS-3411.
Refs #15558. Cherry-picks #17064, which builds on #16728 and preserves #17014's recovery behavior. #16520 separately addresses endpoint propagation before draining. No release PR referencing #17064 was found in the GitHub search.