Skip to content

fix(frontend): return 4xx for backend request rejections on /v1/audio/speech (cherry-pick #12832, #12807) - #12947

Merged
pvijayakrish merged 2 commits into
release/1.4.0from
gluo/cherrypick-12807-12832-audio-speech-4xx
Aug 11, 2026
Merged

pvijayakrish merged 2 commits into
release/1.4.0from
gluo/cherrypick-12807-12832-audio-speech-4xx

Conversation

@GuanLuo

@GuanLuo GuanLuo commented Aug 10, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Fixes DYN-3779

Original PRs

Conflict resolution

Three conflicts, all from release/1.4.0 predating structure on main:

  1. classify/aggregator.rs, pooling/aggregator.rs (modify/delete) — those endpoints do not exist on this branch (absent from protocols/openai.rs). Both files dropped from the backport rather than introduced.
  2. stream_aggregator.rs (content) — release/1.4.0 had delta.ok()? where refactor(llm): preserve typed aggregation errors #12807 introduces delta.into_data()?. Took refactor(llm): preserve typed aggregation errors #12807's version; the Annotated::into_data it needs is included in this cherry-pick via lib/runtime/src/protocols/annotated.rs.
  3. tests/http-service.rs (content) — fix(frontend): return 4xx for backend request rejections on /v1/audio/speech #12832 anchors its 213 added lines after test_streaming_responses_returns_4xx_on_backend_invalid_argument, which does not exist here. Block appended instead; use dynamo_runtime::error::DynamoError added, as this branch's import list lacks it.

Behaviour divergence from main — please read

One assertion in test_audio_speech_backend_invalid_argument_returns_4xx had to be adapted, and it reflects a real difference on this branch.

ErrorMessage::metric_error_type arrived with #12092 (fix(frontend): reject over-context requests before streaming), which is not on release/1.4.0 and is far too large to drag in — 1598 lines across the preprocessor, token-budget parity, and Python tests. Without that field, extract_error_type_from_response falls back to classify_error_for_metrics, which maps 400 to Internal unless the message begins with "Validation:"; the backend's text begins with "ValidationError:".

So on this branch the fold-path 400 meters as error_type=internal rather than validation. The assertion was changed to match, with a comment naming the cause and the condition for restoring it.

The caller-visible fix is unaffected — status code and message are identical to main. Only the metric label is coarser. The failed-branch test still asserts Validation, because that path calls inflight.mark_error(ErrorType::Validation) explicitly rather than deriving the label from the response.

Validation

cargo test -p dynamo-llm --lib                                1871 passed, 0 failed
cargo test -p dynamo-llm --test http-service                    10 passed, 0 failed
cargo test -p dynamo-llm --test aggregators                      8 passed, 0 failed
cargo clippy -p dynamo-llm --no-deps --all-targets -- -D warnings    clean
cargo fmt --all -- --check                                           clean
git diff --check origin/release/1.4.0..HEAD                          clean
pre-commit (via commit hooks)                                        passed

Both audio tests were confirmed to exercise the fix rather than pass incidentally: on main I verified each fails without its corresponding change.


Open in Devin Review

rmccorm4 and others added 2 commits August 10, 2026 10:00
Signed-off-by: Ryan McCormick <rmccormick@nvidia.com>
Signed-off-by: Guan Luo <41310872+GuanLuo@users.noreply.github.com>
…/speech (#12832)

Signed-off-by: Guan Luo <41310872+GuanLuo@users.noreply.github.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Guan Luo <41310872+GuanLuo@users.noreply.github.com>
@GuanLuo
GuanLuo requested review from a team as code owners August 10, 2026 17:09
@github-actions github-actions Bot added fix frontend `python -m dynamo.frontend` and `dynamo-run in=http|text|grpc` and removed fix labels Aug 10, 2026

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 1 potential issue.

Open in Devin Review

Comment on lines +1321 to +1336
// Backport divergence from main, which asserts `Validation` here.
// `ErrorMessage::metric_error_type` arrived with #12092 and is not on this
// release branch, so the label falls back to `classify_error_for_metrics`,
// which maps 400 to `Internal` unless the message begins with
// "Validation:" — the backend's text begins with "ValidationError:".
// The status code and message the caller sees are unaffected.
// If #12092 is ever backported, restore the `Validation` assertion.
compare_counter(
&metrics,
"tts-model",
&Endpoint::Audios,
&RequestType::Unary,
&Status::Error,
&ErrorType::Internal,
1,
);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔍 Metric label divergence from main is asserted, not just tolerated

The backported test hardcodes ErrorType::Internal for a 400 response because ErrorMessage::metric_error_type (from #12092) is absent on this branch, so classify_error_for_metrics only maps to Validation when the message starts with "Validation:" while the backend emits "ValidationError:". The comment documents this, but it means backend-rejection 400s on /v1/audio/speech are metered as server faults on this release branch, which can distort error-budget/SLO dashboards even though the caller-visible status is correct. Worth confirming this is acceptable for 1.4.0 operators.

Open in Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

/// encountered short-circuits further merging and is returned; the remainder
/// of the stream is dropped.
pub async fn aggregate_stream<T, S>(stream: S) -> Result<T, String>
pub async fn aggregate_stream<T, S>(stream: S) -> Result<T, DynamoError>

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changing the public aggregation APIs from Result<_, String>/anyhow::Result<_> to Result<_, DynamoError> is a release-branch API break for callers that compile against these helpers. Fix: preserve the existing public return types and add typed-error helper variants for the HTTP paths that need structured classification.

🤖 AI Fix

In lib/llm/src/protocols/openai/stream_aggregator.rs, keep pub async fn aggregate_stream<T, S>(...) -> Result<T, String> with its old public signature, add pub async fn aggregate_stream_typed<T, S>(...) -> Result<T, DynamoError> containing the new into_data() implementation, and update only the /v1/audio/speech path plus any internal typed callers to call the typed helper while reverting the public from_annotated_stream/from_sse_stream signatures in the OpenAI aggregator files to their pre-PR return types.

@datadog-official

datadog-official Bot commented Aug 10, 2026 •

Copy link
Copy Markdown

🎯 Code Coverage (details)
• Patch Coverage: 100.00%
• Overall Coverage: 46.03% (-3.02%)

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 0dd8fb1 | Docs | Datadog PR Page | Give us feedback!

@pvijayakrish
pvijayakrish merged commit c9b40d8 into release/1.4.0 Aug 11, 2026
172 of 182 checks passed
@pvijayakrish
pvijayakrish deleted the gluo/cherrypick-12807-12832-audio-speech-4xx branch August 11, 2026 00:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fix frontend `python -m dynamo.frontend` and `dynamo-run in=http|text|grpc` size/XL

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants