Repository navigation
fix(frontend): return 4xx for backend request rejections on /v1/audio/speech (cherry-pick #12832, #12807) - #12947
Conversation
Signed-off-by: Ryan McCormick <rmccormick@nvidia.com> Signed-off-by: Guan Luo <41310872+GuanLuo@users.noreply.github.com>
…/speech (#12832) Signed-off-by: Guan Luo <41310872+GuanLuo@users.noreply.github.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Guan Luo <41310872+GuanLuo@users.noreply.github.com>
| // Backport divergence from main, which asserts `Validation` here. | ||
| // `ErrorMessage::metric_error_type` arrived with #12092 and is not on this | ||
| // release branch, so the label falls back to `classify_error_for_metrics`, | ||
| // which maps 400 to `Internal` unless the message begins with | ||
| // "Validation:" — the backend's text begins with "ValidationError:". | ||
| // The status code and message the caller sees are unaffected. | ||
| // If #12092 is ever backported, restore the `Validation` assertion. | ||
| compare_counter( | ||
| &metrics, | ||
| "tts-model", | ||
| &Endpoint::Audios, | ||
| &RequestType::Unary, | ||
| &Status::Error, | ||
| &ErrorType::Internal, | ||
| 1, | ||
| ); |
There was a problem hiding this comment.
🔍 Metric label divergence from main is asserted, not just tolerated
The backported test hardcodes ErrorType::Internal for a 400 response because ErrorMessage::metric_error_type (from #12092) is absent on this branch, so classify_error_for_metrics only maps to Validation when the message starts with "Validation:" while the backend emits "ValidationError:". The comment documents this, but it means backend-rejection 400s on /v1/audio/speech are metered as server faults on this release branch, which can distort error-budget/SLO dashboards even though the caller-visible status is correct. Worth confirming this is acceptable for 1.4.0 operators.
Was this helpful? React with 👍 or 👎 to provide feedback.
| /// encountered short-circuits further merging and is returned; the remainder | ||
| /// of the stream is dropped. | ||
| pub async fn aggregate_stream<T, S>(stream: S) -> Result<T, String> | ||
| pub async fn aggregate_stream<T, S>(stream: S) -> Result<T, DynamoError> |
There was a problem hiding this comment.
Changing the public aggregation APIs from Result<_, String>/anyhow::Result<_> to Result<_, DynamoError> is a release-branch API break for callers that compile against these helpers. Fix: preserve the existing public return types and add typed-error helper variants for the HTTP paths that need structured classification.
🤖 AI Fix
In lib/llm/src/protocols/openai/stream_aggregator.rs, keep pub async fn aggregate_stream<T, S>(...) -> Result<T, String> with its old public signature, add pub async fn aggregate_stream_typed<T, S>(...) -> Result<T, DynamoError> containing the new into_data() implementation, and update only the /v1/audio/speech path plus any internal typed callers to call the typed helper while reverting the public from_annotated_stream/from_sse_stream signatures in the OpenAI aggregator files to their pre-PR return types.
|
🎯 Code Coverage (details) 🔗 Commit SHA: 0dd8fb1 | Docs | Datadog PR Page | Give us feedback! |
Summary
release/1.4.0, as a single PR because fix(frontend): return 4xx for backend request rejections on /v1/audio/speech #12832 does not build without refactor(llm): preserve typed aggregation errors #12807.task_typeon/v1/audio/speechreturned HTTP 500"Failed to fold audio stream"instead of a 4xx naming the field. The worker's message reached the frontend intact; only its classification was lost during stream folding, leaving the handler nothing to choose a status from.DynamoError; fix(frontend): return 4xx for backend request rejections on /v1/audio/speech #12832 routes the audio handler's fold error throughErrorMessage::from_anyhowsoBackend(InvalidArgument)becomes 400 with the backend's message, and fixes three metrics paths in the same handler.Fixes DYN-3779
Original PRs
refactor(llm): preserve typed aggregation errorseb2198a82f434fc9f37ca87c66d2e460ecf7860c11cd184576fix(frontend): return 4xx for backend request rejections on /v1/audio/speech9eafac64592dbeebe94387d158b791c6557dc5440dd8fb1e86Conflict resolution
Three conflicts, all from
release/1.4.0predating structure onmain:classify/aggregator.rs,pooling/aggregator.rs(modify/delete) — those endpoints do not exist on this branch (absent fromprotocols/openai.rs). Both files dropped from the backport rather than introduced.stream_aggregator.rs(content) —release/1.4.0haddelta.ok()?where refactor(llm): preserve typed aggregation errors #12807 introducesdelta.into_data()?. Took refactor(llm): preserve typed aggregation errors #12807's version; theAnnotated::into_datait needs is included in this cherry-pick vialib/runtime/src/protocols/annotated.rs.tests/http-service.rs(content) — fix(frontend): return 4xx for backend request rejections on /v1/audio/speech #12832 anchors its 213 added lines aftertest_streaming_responses_returns_4xx_on_backend_invalid_argument, which does not exist here. Block appended instead;use dynamo_runtime::error::DynamoErroradded, as this branch's import list lacks it.Behaviour divergence from main — please read
One assertion in
test_audio_speech_backend_invalid_argument_returns_4xxhad to be adapted, and it reflects a real difference on this branch.ErrorMessage::metric_error_typearrived with #12092 (fix(frontend): reject over-context requests before streaming), which is not onrelease/1.4.0and is far too large to drag in — 1598 lines across the preprocessor, token-budget parity, and Python tests. Without that field,extract_error_type_from_responsefalls back toclassify_error_for_metrics, which maps 400 toInternalunless the message begins with"Validation:"; the backend's text begins with"ValidationError:".So on this branch the fold-path 400 meters as
error_type=internalrather thanvalidation. The assertion was changed to match, with a comment naming the cause and the condition for restoring it.The caller-visible fix is unaffected — status code and message are identical to
main. Only the metric label is coarser. Thefailed-branch test still assertsValidation, because that path callsinflight.mark_error(ErrorType::Validation)explicitly rather than deriving the label from the response.Validation
Both audio tests were confirmed to exercise the fix rather than pass incidentally: on
mainI verified each fails without its corresponding change.