Repository navigation
fix(decisions): drain abandoned dispatch waiters with request context - #17982
glamr-agent wants to merge 6 commits into
Conversation
Dropping the decision dispatch waiter before the deadline detached the dispatch task with no parent kill, traced drain, or overrun warning. A drop guard now owns the unresolved JoinHandle so deadline expiry and abandonment share one cleanup path: kill the parent, then drain the task in the span captured when the waiter started, keeping task-owned admission until local completion. Follows ai-dynamo#15852 (DYN-5271). Signed-off-by: svc-glamr@nvidia.com <svc-glamr@nvidia.com>
|
factory: Initial validation snapshot. Validation status: complete
No live client disconnect was tested against a running frontend. The new unit test drives the lifecycle wait directly and drops it the way the HTTP layer does. Detailed evidence — validation completeLeft out of the table because CI repeats them: formatting and lint ( Evidence tally: [1/1 validated] Advisory AI assessment: sound Validation result: pass. The new test fails without the guard and passes with it. The workspace check, Clippy, format, pre-commit, and the 66 lifecycle, System One, and disconnect tests all pass. On base Evidence audit: complete [1/1 validated]. The command report comes from recorded runs. Commands and results [1/1 validated]Generated from the commands recorded during this run. Check 1Checks the changed Rust crates with Result: Passed ( Command:
|
|
No actionable comments were generated in the recent review. WalkthroughThe dispatch path now uses ChangesDispatch cleanup lifecycle
Estimated code review effort: 3 (Moderate) | ~25 minutes Merge Risk:
|
|
factory: @dynamo-ops please run full CI for 5b6e751 |
|
👋 Hi glamr-agent! Thank you for contributing to ai-dynamo/dynamo. Just a reminder: The 🚀 |
|
/ok to test 5b6e751 |
|
nursery: I changed nothing for the documentation warning in CodeRabbit's summary. In |
Signed-off-by: GLAMR <svc-glamr@nvidia.com>
|
nursery: @dynamo-ops please run full CI for d86bd91 |
|
/ok to test d86bd91 |
✅ Dynamo PR CI passed — run 43514496004 (attempt 1) on
|
MatejKosec
left a comment
There was a problem hiding this comment.
Read the diff and its one caller in systemone.rs. Dropping the waiter now does what deadline expiry already did (kill, then drain in the request span), the task still keeps its permit, and the connection monitor's own kill and metrics are untouched.
…utures-be74465eb7de
|
nursery: @dynamo-ops please run full CI for 230f78f |
|
/ok to test 230f78f |
Signed-off-by: GLAMR <svc-glamr@nvidia.com>
|
nursery: @dynamo-ops please run full CI for ea6fa04 |
|
/ok to test ea6fa04 |
…utures-be74465eb7de
|
nursery: @dynamo-ops please run full CI for 4adafe2 |
|
/ok to test 4adafe2 |
Signed-off-by: GLAMR <svc-glamr@nvidia.com>
|
nursery: @dynamo-ops please run full CI for fc74514 |
|
/ok to test fc74514 |
Overview
When a Decisions request (
/v1/decisionsor/v1/systemone) hits its deadline, the frontend kills the request context and waits for the dispatch task in the background. The task keeps its admission permit until its local work ends. A warning tied to the request is logged if it runs past the 1 s cancellation grace. When the client leaves before the deadline, none of this happens. The dispatch task is detached and still holds its permit, but nothing waits for it or logs an overrun.Key Changes
A client that leaves early now gets the same cleanup as deadline expiry. The frontend kills the request context and drains the task in the background, in the request span. The task keeps its permit until it finishes. The span is recorded when the wait starts, because the HTTP layer drops the response future outside the request span. The task is never aborted, so dropping it does not count as settling GPU work. A completed result still wins over an expired deadline. Response codes, deadline timing, and metrics are unchanged.
Validation
A new unit test polls the waiter inside a request span, then drops it outside the span. It checks the kill, the overrun warning with
request_id, and that the permit stays held until the task finishes. It fails onmainand passes with this change. Existing lifecycle, System One, and disconnect tests pass unchanged. Not tested: a live client disconnect against a running frontend.Issues and Ongoing Work
Pull Request Checklist
/pr-checklistskill.Closes DYN-5271.
Summary by CodeRabbit