Skip to content

refactor(llm): preserve typed aggregation errors - #12807

Merged
rmccorm4 merged 2 commits into
mainfrom
rmccormick/unify-errors-typed-aggregation
Aug 7, 2026
Merged

rmccorm4 merged 2 commits into
mainfrom
rmccormick/unify-errors-typed-aggregation

Conversation

@rmccorm4

@rmccorm4 rmccorm4 commented Aug 7, 2026 •

Copy link
Copy Markdown
Contributor

Stack

1 of 4 — base: main

Next: #12808

Summary

  • Preserve structured DynamoError values when consuming Annotated<T>.
  • Return typed errors from every shared unary aggregation path instead of flattening them to strings.
  • Stop aggregation immediately on a typed error, including when it arrives after partial output.

This layer intentionally does not change HTTP status policy.

Root cause

Unary aggregators converted annotated backend errors to strings. Once the structured error type was discarded, the HTTP layer could not reliably distinguish invalid client input from runtime failures.

Validation

  • cargo check -p dynamo-llm: passed
  • Runtime typed-error regression: 1 passed
  • Chat/Completions partial-output typed-error regressions: 2 passed
  • cargo fmt --all -- --check: passed
  • git diff --check: passed

Tracking

Summary by CodeRabbit

  • Bug Fixes
    • Improved error handling across streaming responses for chat, completions, embeddings, audio, images, videos, and tensor operations.
    • Structured backend errors are now preserved and returned accurately, including when failures occur after partial results.
    • Empty or invalid tensor streams now produce clearer errors, while metadata-only updates are handled safely.
    • Stream aggregation behavior is more consistent across classification and pooling requests.

@github-actions github-actions Bot added refactor frontend `python -m dynamo.frontend` and `dynamo-run in=http|text|grpc` labels Aug 7, 2026
@rmccorm4
rmccorm4 marked this pull request as ready for review August 7, 2026 07:19
@rmccorm4
rmccorm4 requested review from a team as code owners August 7, 2026 07:19
devin-ai-integration[bot]

This comment was marked as resolved.

@coderabbitai

coderabbitai Bot commented Aug 7, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 4a49f02a-63f6-4fd9-a037-be9fb59e0d0c

📥 Commits

Reviewing files that changed from the base of the PR and between 027e7ad and 3d09690.

📒 Files selected for processing (12)
  • lib/llm/src/protocols/openai/audios/aggregator.rs
  • lib/llm/src/protocols/openai/chat_completions/aggregator.rs
  • lib/llm/src/protocols/openai/classify/aggregator.rs
  • lib/llm/src/protocols/openai/completions/aggregator.rs
  • lib/llm/src/protocols/openai/embeddings/aggregator.rs
  • lib/llm/src/protocols/openai/generate.rs
  • lib/llm/src/protocols/openai/images/aggregator.rs
  • lib/llm/src/protocols/openai/pooling/aggregator.rs
  • lib/llm/src/protocols/openai/stream_aggregator.rs
  • lib/llm/src/protocols/openai/videos/aggregator.rs
  • lib/llm/src/protocols/tensor.rs
  • lib/runtime/src/protocols/annotated.rs

Walkthrough

The change standardizes annotated-stream aggregation on DynamoError. It adds Annotated::into_data, removes aggregate_typed_stream, updates OpenAI aggregators, changes tensor stream handling, and adds typed-error propagation tests.

Changes

Typed stream error propagation

Layer / File(s) Summary
Annotated error extraction
lib/runtime/src/protocols/annotated.rs
Annotated now exposes into_data, preserves structured DynamoError values, and shares fallback error resolution across related methods.
Shared aggregation and tensor handling
lib/llm/src/protocols/openai/stream_aggregator.rs, lib/llm/src/protocols/tensor.rs, lib/llm/src/protocols/openai/generate.rs
Shared aggregation now uses Annotated::into_data. Tensor aggregation skips metadata, rejects multiple responses, propagates errors, and rejects empty streams.
Chat and completion delta aggregation
lib/llm/src/protocols/openai/chat_completions/aggregator.rs, lib/llm/src/protocols/openai/completions/aggregator.rs
Delta aggregators use fallible folds and return DynamoError directly. Tests verify typed errors after partial output.
Protocol aggregator migration
lib/llm/src/protocols/openai/{audios,classify,embeddings,images,pooling,videos}/aggregator.rs
Protocol aggregators use aggregate_stream and expose DynamoError return types. Embedding tests verify preservation of typed backend errors.

Estimated code review effort: 4 (Complex) | ~45 minutes

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly and concisely identifies the main change: preserving typed aggregation errors.
Description check ✅ Passed The description clearly covers the scope, root cause, validation, and tracked issues, but it does not follow all template headings.
Docstring Coverage ✅ Passed Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

Comment @coderabbitai help to get the list of available commands.

@datadog-official

datadog-official Bot commented Aug 7, 2026 •

Copy link
Copy Markdown

🎯 Code Coverage (details)
• Patch Coverage: 100.00%
• Overall Coverage: 46.69% (-1.88%)

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 5d3e4dc | Docs | Datadog PR Page | Give us feedback!

Signed-off-by: Ryan McCormick <rmccormick@nvidia.com>
Signed-off-by: Ryan McCormick <rmccormick@nvidia.com>
@rmccorm4
rmccorm4 force-pushed the rmccormick/unify-errors-typed-aggregation branch from e397bee to 5d3e4dc Compare August 7, 2026 17:50
Comment thread lib/runtime/src/protocols/annotated.rs Resolved
@rmccorm4

rmccorm4 commented Aug 7, 2026

Copy link
Copy Markdown
Contributor Author

This PR stack likely covers the same fix or majority of fix that #12832 is targeting

@rmccorm4
rmccorm4 merged commit eb2198a into main Aug 7, 2026
117 of 118 checks passed
@rmccorm4
rmccorm4 deleted the rmccormick/unify-errors-typed-aggregation branch August 7, 2026 20:26
pvijayakrish pushed a commit that referenced this pull request Aug 11, 2026
…/speech (cherry-pick #12832, #12807) (#12947)

Signed-off-by: Ryan McCormick <rmccormick@nvidia.com>
Signed-off-by: Guan Luo <41310872+GuanLuo@users.noreply.github.com>
Co-authored-by: Ryan McCormick <rmccormick@nvidia.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

frontend `python -m dynamo.frontend` and `dynamo-run in=http|text|grpc` refactor size/L

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants