Skip to content

fix(decisions): drain abandoned dispatch waiters with request context - #17982

Open
glamr-agent wants to merge 6 commits into
ai-dynamo:mainfrom
glamr-agent:dyn-5271-drain-abandoned-decision-dispatch-futures-be74465eb7de
Open

glamr-agent wants to merge 6 commits into
ai-dynamo:mainfrom
glamr-agent:dyn-5271-drain-abandoned-decision-dispatch-futures-be74465eb7de

Conversation

@glamr-agent

@glamr-agent glamr-agent commented Nov 3, 2026 •

Copy link
Copy Markdown
Collaborator

Overview

When a Decisions request (/v1/decisions or /v1/systemone) hits its deadline, the frontend kills the request context and waits for the dispatch task in the background. The task keeps its admission permit until its local work ends. A warning tied to the request is logged if it runs past the 1 s cancellation grace. When the client leaves before the deadline, none of this happens. The dispatch task is detached and still holds its permit, but nothing waits for it or logs an overrun.

Key Changes

A client that leaves early now gets the same cleanup as deadline expiry. The frontend kills the request context and drains the task in the background, in the request span. The task keeps its permit until it finishes. The span is recorded when the wait starts, because the HTTP layer drops the response future outside the request span. The task is never aborted, so dropping it does not count as settling GPU work. A completed result still wins over an expired deadline. Response codes, deadline timing, and metrics are unchanged.

Validation

A new unit test polls the waiter inside a request span, then drops it outside the span. It checks the kill, the overrun warning with request_id, and that the permit stays held until the task finishes. It fails on main and passes with this change. Existing lifecycle, System One, and disconnect tests pass unchanged. Not tested: a live client disconnect against a running frontend.

Issues and Ongoing Work

Pull Request Checklist
  • I used Dynamo's /pr-checklist skill.

Closes DYN-5271.

Summary by CodeRabbit

  • Bug Fixes
    • Improved handling of dispatches that time out or are abandoned, helping ensure cancellation and cleanup complete before capacity is reused.

Dropping the decision dispatch waiter before the deadline detached the
dispatch task with no parent kill, traced drain, or overrun warning.
A drop guard now owns the unresolved JoinHandle so deadline expiry and
abandonment share one cleanup path: kill the parent, then drain the task
in the span captured when the waiter started, keeping task-owned
admission until local completion. Follows ai-dynamo#15852 (DYN-5271).

Signed-off-by: svc-glamr@nvidia.com <svc-glamr@nvidia.com>
@glamr-agent
glamr-agent requested a review from a team as a code owner November 3, 2026 21:58
@copy-pr-bot

copy-pr-bot Bot commented Nov 3, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@glamr-agent
glamr-agent temporarily deployed to external_collaborator November 3, 2026 21:58 — with GitHub Actions Inactive
@glamr-agent
glamr-agent temporarily deployed to external_collaborator November 3, 2026 21:58 — with GitHub Actions Inactive
@glamr-agent

Copy link
Copy Markdown
Collaborator Author

factory: Initial validation snapshot.

Validation status: complete

Check Result Scope or reason
Checks the changed Rust crates with cargo check and Clippy Passed (exit 0) cargo test -p dynamo-llm --lib -- http::service::decision_lifecycle http::service::systemone http::service::disconnect: 66 passed, 0 failed, at 5b6e7519c0

No live client disconnect was tested against a running frontend. The new unit test drives the lifecycle wait directly and drops it the way the HTTP layer does.

Detailed evidence — validation complete

Left out of the table because CI repeats them: formatting and lint (cargo fmt --all --check, cargo clippy --workspace --all-targets --no-deps -- -D warnings, pre-commit) and the workspace build (cargo check --workspace --all-targets). All of them exited 0 during validation.

Evidence tally: [1/1 validated]

Advisory AI assessment: sound

Validation result: pass. The new test fails without the guard and passes with it. The workspace check, Clippy, format, pre-commit, and the 66 lifecycle, System One, and disconnect tests all pass. On base 0e4551ae8e with only the new test added, cargo test -p dynamo-llm --lib -- decision_lifecycle::tests::abandoned_dispatch exits 101 with assertion failed: parent.is_killed().

Evidence audit: complete [1/1 validated]. The command report comes from recorded runs.

Commands and results [1/1 validated]

Generated from the commands recorded during this run.

Check 1

Checks the changed Rust crates with cargo check and Clippy.

Result: Passed (exit 0)

Command:

cargo test -p dynamo-llm --lib -- http::service::decision_lifecycle http::service::systemone http::service::disconnect

@coderabbitai

coderabbitai Bot commented Nov 3, 2026

Copy link
Copy Markdown
Contributor

Review in Change Stack →

No actionable comments were generated in the recent review.

Walkthrough

The dispatch path now uses DispatchWaiter to manage unresolved tasks. Cleanup preserves the parent context and tracing span, retains admission until task completion, and handles drops inside or outside a Tokio runtime. A test covers abandoned dispatch cleanup.

Changes

Dispatch cleanup lifecycle

Layer / File(s) Summary
Dispatch waiter integration
lib/llm/src/http/service/decision_lifecycle.rs
dispatch_with_deadline now delegates unresolved task cleanup to DispatchWaiter. The waiter kills the parent, drains the task with the captured span when a runtime exists, or logs and detaches the task without a runtime.
Abandoned dispatch validation
lib/llm/src/http/service/decision_lifecycle.rs
The test verifies parent cancellation, request ID retention in cleanup logs, and admission retention until the blocked task completes.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Merge Risk: Minimal · up to 5b6e7

Abandoned Decisions dispatches receive the intended cleanup on client disconnect. No issue identified here blocks merging after normal checks.

Pre-merge checks: Passed 4 · Failed 1

Failed Failed checks: 1 warning

Check name Status Explanation Resolution
Docstring Coverage Warning Docstring coverage is 50.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documenta… Write docstrings for the functions missing them to satisfy the coverage threshold.
Passed Passed checks: 4 passed
Check name Status Explanation
Title check Passed The title clearly and concisely describes the main change: draining abandoned dispatch waiters with request context.
Description check Passed The description includes all required sections, explains the behavior and implementation, documents validation and limitations, and references related issues. The checklist is present but remains unch…
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Full details: Description check

Explanation

The description includes all required sections, explains the behavior and implementation, documents validation and limitations, and references related issues. The checklist is present but remains unchecked; this is non-critical.

Full details: Docstring Coverage

Explanation

Docstring coverage is 50.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentation-only edits and deletions within surviving functions are included; unchanged diff context and fully deleted functions are excluded. Analyzed 2 functions across 1 files.

Resolution

Write docstrings for the functions missing them to satisfy the coverage threshold.

  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

CodeRabbit Verify

  • Run CodeRabbit Verify to test the changes in this pull request.
Recent review info
Run configuration
Commits

Reviewing files that changed from the base of the PR and between b0a6bd1 and 5b6e751.

Files selected for processing: 1
  • lib/llm/src/http/service/decision_lifecycle.rs

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 9 remain after this review.

@glamr-agent

Copy link
Copy Markdown
Collaborator Author

factory: @dynamo-ops please run full CI for 5b6e751

@github-actions github-actions Bot added frontend `python -m dynamo.frontend` and `dynamo-run in=http|text|grpc` fix labels Nov 3, 2026
@github-actions

github-actions Bot commented Nov 3, 2026

Copy link
Copy Markdown
Contributor

👋 Hi glamr-agent! Thank you for contributing to ai-dynamo/dynamo.

Just a reminder: The NVIDIA Test Github Validation CI runs an essential subset of the testing framework to quickly catch errors.Your PR reviewers may elect to test the changes comprehensively before approving your changes.

🚀

@github-actions github-actions Bot added the external-contribution Pull request is from an external contributor label Nov 3, 2026
@dynamo-ops

Copy link
Copy Markdown
Contributor

/ok to test 5b6e751

@glamr-agent

Copy link
Copy Markdown
Collaborator Author

nursery: I changed nothing for the documentation warning in CodeRabbit's summary. In decision_lifecycle.rs, dispatch_with_deadline documents cancellation, admission ownership, and the limit of local completion. DispatchWaiter documents why it owns the task and captures the request span. The comments in Drop::drop explain why cleanup must not abort the task. These describe the behavior that callers and maintainers need; another comment on the trait method would repeat them.

Signed-off-by: GLAMR <svc-glamr@nvidia.com>
@glamr-agent
glamr-agent temporarily deployed to external_collaborator November 4, 2026 01:19 — with GitHub Actions Inactive
@glamr-agent

Copy link
Copy Markdown
Collaborator Author

nursery: @dynamo-ops please run full CI for d86bd91

@dynamo-ops

Copy link
Copy Markdown
Contributor

/ok to test d86bd91

@devin-ai-integration

devin-ai-integration Bot commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

✅ Dynamo PR CI passed — run 43514496004 (attempt 1) on fc74514d33

Gate checks: ✅ backend-status-check · ✅ deploy-status-check · ✅ dynamo-status-check

Posted automatically by Devin for run 43514496004. Updated on every full-CI run of this PR.

@MatejKosec MatejKosec left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Read the diff and its one caller in systemone.rs. Dropping the waiter now does what deadline expiry already did (kill, then drain in the request span), the task still keeps its permit, and the connection monitor's own kill and metrics are untouched.

@MatejKosec
MatejKosec temporarily deployed to external_collaborator November 13, 2026 00:43 — with GitHub Actions Inactive
@glamr-agent

Copy link
Copy Markdown
Collaborator Author

nursery: @dynamo-ops please run full CI for 230f78f

@dynamo-ops

Copy link
Copy Markdown
Contributor

/ok to test 230f78f

Signed-off-by: GLAMR <svc-glamr@nvidia.com>
@glamr-agent
glamr-agent temporarily deployed to external_collaborator November 13, 2026 01:39 — with GitHub Actions Inactive
@glamr-agent

Copy link
Copy Markdown
Collaborator Author

nursery: @dynamo-ops please run full CI for ea6fa04

@dynamo-ops

Copy link
Copy Markdown
Contributor

/ok to test ea6fa04

@MatejKosec
MatejKosec temporarily deployed to external_collaborator November 13, 2026 20:33 — with GitHub Actions Inactive
@glamr-agent

Copy link
Copy Markdown
Collaborator Author

nursery: @dynamo-ops please run full CI for 4adafe2

@dynamo-ops

Copy link
Copy Markdown
Contributor

/ok to test 4adafe2

Signed-off-by: GLAMR <svc-glamr@nvidia.com>
@glamr-agent
glamr-agent requested a review from a team as a code owner November 13, 2026 21:51
@glamr-agent
glamr-agent temporarily deployed to external_collaborator November 13, 2026 21:51 — with GitHub Actions Inactive
@glamr-agent

Copy link
Copy Markdown
Collaborator Author

nursery: @dynamo-ops please run full CI for fc74514

@dynamo-ops

Copy link
Copy Markdown
Contributor

/ok to test fc74514

This branch was previously deployed

1 inactive deployment
external_collaborator — fc74514d Deployed Nov 13, 2026 by glamr-agent via ok-to-test #51188
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

external-contribution Pull request is from an external contributor fix frontend `python -m dynamo.frontend` and `dynamo-run in=http|text|grpc` size/L

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants