Repository navigation
fix(anthropic): enforce Messages API version admission - #17378
maheshrbapatu wants to merge 5 commits into
Conversation
|
👋 Hi maheshrbapatu! Thank you for contributing to ai-dynamo/dynamo. Just a reminder: The 🚀 |
8855eff to
83039d7
Compare
|
No actionable comments were generated in the recent review. WalkthroughThe change adds request admission rules and request-ID error envelopes for Anthropic Messages endpoints. It also updates streaming monitor configuration and SSE test parsing, and documents compatibility rules and a lower documented HTTP body limit. ChangesAnthropic HTTP behavior
Estimated code review effort: 4 (Complex) | ~45 minutes Merge Risk:
|
There was a problem hiding this comment.
Questions for the author:
- lib/llm/src/http/service/anthropic/validation.rs: For every supported
/v1/messagesinference request,identity()parses the full raw JSON into a newly allocatedHashMap,into_request()immediately repeats that full-body parse/allocation, and typed deserialization then scans it again. Since this path supports request bodies up to 45 MiB, please provide admission-latency or CPU measurements for representative large/500k-token Messages requests versus the prior direct typed extraction, or calculations showing the extra full-body pass is negligible.
| } | ||
| match version.map(|value| value.to_str()) { | ||
| None | Some(Ok("2023-06-01")) => {} | ||
| Some(Ok("2026-10-21")) => { |
There was a problem hiding this comment.
The 2026-10-21 branch enforces only the removed sampling parameters, but that version also requires HTTP 400 when a user text block contains a non-empty citations array, directly or inside a search_result. AnthropicContentBlock::Text currently accepts those citations and later conversion discards them, while recognized WebSearchToolResult content is not traversed and the block is ignored during conversion, so these requests reach generation instead of failing admission. Inspect message content under this version and reject both citation shapes before dispatch.
🤖 AI Fix
Extend the 2026-10-21 admission branch to reject non-empty citations in user text blocks, including text nested inside search-result blocks, before typed conversion and generation.
There was a problem hiding this comment.
Added the missing 2026-10-21 citation admission check on both endpoints. It rejects nonempty citations on user text, including nested tool/search result text, while preserving assistant citations and empty user arrays. The HTTP regression also checks that rejection happens before generation.
| } | ||
| match version.map(|value| value.to_str()) { | ||
| None | Some(Ok("2023-06-01")) => {} | ||
| Some(Ok("2026-10-21")) => { |
There was a problem hiding this comment.
The 2026-10-21 branch only rejects sampling fields, so requests with non-empty citations still reach generation. Text.citations deserializes as a recognized field, while validate_anthropic_messages ignores it and conversion discards it; WebSearchToolResult.content similarly permits nested citation payloads that conversion ignores. This violates the version's required 400 admission behavior and silently removes client content.
🤖 AI Fix
Extend 2026-10-21 admission to reject non-empty citations in text blocks and citations nested in search-result content before backend dispatch.
There was a problem hiding this comment.
Covered by the citation admission fix: nonempty user citations now return HTTP 400 before conversion. The tolerant visitor follows supported tool/search containers, skips unrelated values, and leaves malformed-shape diagnostics to the original typed parser.
| ), | ||
| ]; | ||
| for path in ["/v1/messages", "/v1/messages/count_tokens"] { | ||
| for stream in [false, true] { |
There was a problem hiding this comment.
The stream matrix doubles every duplicate-key request on both endpoints without protecting another regression. ValidatedRequest::fields() rejects the duplicate raw JSON before typed deserialization or any streaming branch, so the retained stream: false requests still verify duplicate detection and both handler integrations.
🤖 AI Fix
Remove the stream loop and run each duplicate-key body once per endpoint.
There was a problem hiding this comment.
Removed the duplicate-key stream cross-product. Both endpoints still reject duplicate known fields through the original typed parser; unary/streaming metric-label behavior is covered separately.
| ), | ||
| ("thinking", json!(true)), | ||
| ]; | ||
| for stream in [false, true] { |
There was a problem hiding this comment.
Both stream values exercise the same pre-dispatch validator for all 39 malformed values. stream is not among the fields passed to validate, and every case is rejected before response streaming begins, so one stream value preserves every field-specific validation contract while avoiding 39 duplicate HTTP requests.
🤖 AI Fix
Remove the stream parameterization and run the malformed-field table once with a fixed stream value.
There was a problem hiding this comment.
Removed the malformed-field stream cross-product in #17367 and rebased this PR onto it. Each invalid shape still checks both endpoints and no backend dispatch.
| StatusCode::BAD_REQUEST, | ||
| "invalid_request_error", | ||
| ), | ||
| ( |
There was a problem hiding this comment.
The missing-model case repeats the unary missing-model request already selected in anthropic_handler_errors_record_classification_with_response, whose changed assert_anthropic_status helper now verifies the response request ID matches the header. Its only additional checks here are generic envelope properties, which can be retained once in that shared helper rather than issuing another identical missing-model request.
🤖 AI Fix
Move the generic nonempty request-ID and content-type assertions into assert_anthropic_status, then remove this missing-model tuple from the new test.
There was a problem hiding this comment.
Removed the duplicate missing-model request and moved the nonempty request-ID/JSON-content-type assertions into the existing error-classification helper. The remaining max_tokens request-ID case is a single direct case.
| ]; | ||
| for (path, stream) in [ | ||
| ("/v1/messages", false), | ||
| ("/v1/messages", true), |
There was a problem hiding this comment.
Repeating every invalid header with stream: true cannot distinguish another version-admission regression: into_request checks anthropic-version before typed conversion and before the Messages handler can select its streaming response path. The unary Messages case retains the header-value boundaries, and the count-tokens case retains coverage of the second handler wiring.
🤖 AI Fix
Remove the streaming Messages tuple from the invalid-version route matrix.
There was a problem hiding this comment.
Removed the redundant streaming invalid-version matrix. Unary requests cover both endpoints; separate successful streaming requests retain version/frame coverage.
| let svc = HarnessService::start([script]).await; | ||
| for (path, stream) in [ | ||
| ("/v1/messages", false), | ||
| ("/v1/messages", true), |
There was a problem hiding this comment.
The streaming Messages tuple repeats all 12 sampling-field cases after the same early fields.contains_key admission check exercised by the unary tuple. Rejection occurs before deserialization and response-mode branching, while the remaining unary and count-tokens tuples still protect both endpoint integrations and every field/value boundary.
🤖 AI Fix
Remove the streaming Messages tuple from this rejected-sampling matrix.
There was a problem hiding this comment.
Removed the streaming repetition of sampling-field rejection. Both endpoints still cover present defaults, null, and malformed values, and legacy conversion remains covered.
83039d7 to
df31a5c
Compare
df31a5c to
ae164ea
Compare
|
I added the performance measurements to the description. With the These are median batch-average CPU measurements from warmed release builds on two pinned CPUs, with peak heap measured separately and excluding the ingress buffer. They cover parsing/admission only, not HTTP transport, tokenization, image decoding, or inference. The description includes the smaller-request results and remaining cost. |
Accept 2023-06-01 and 2026-10-21, default omitted version headers to legacy behavior, and reject unsupported or repeated headers before generation. Reject sampling fields under 2026-10-21 while preserving explicit null presence and duplicate-key rejection. Cover unary, streaming, and token-counting admission through real HTTP tests, and document the bounded compatibility policy. Closes ai-dynamo#17374 Signed-off-by: maheshrbapatu <maheshbapatu@gmail.com>
Signed-off-by: maheshrbapatu <maheshbapatu@gmail.com>
Signed-off-by: maheshrbapatu <maheshbapatu@gmail.com>
ae164ea to
6997269
Compare
Signed-off-by: maheshrbapatu <maheshbapatu@gmail.com>
Signed-off-by: maheshrbapatu <maheshbapatu@gmail.com>
Overview
Apply
anthropic-versionadmission to both Messages endpoints. Under2026-10-21, reject removed sampling fields and nonempty citations on user text before generation; omitted headers retain legacy behavior.Key Changes
Approximately 134 added production-code lines beyond the merged admission validation, excluding tests, comments, and whitespace.
2023-06-01/2026-10-21headers and rejects unsupported or repeated versions.Validation
After rebasing onto main
1732fd7, all 27 real-HTTP admission and replay tests pass. These use scripted CPU backends and do not qualify live inference or Anthropic hosted behavior.The
2026-10-21citation rule adds a scan of message content. The benchmark below includes that check.Admission-only release benchmark against main
cd3b977ed, recorded for the original version-admission change with protocol 8.4.0 before this rebase: 9 warmed, alternating batches on each of two pinned CPUs, with separate allocation runs. Values are median batch-average thread CPU, not request p95. Peak heap excludes the ingress buffer and allocator overhead. HTTP transport, tokenization, inference, and image fetch/decode are excluded; these figures do not establish end-to-end throughput.Issues and Ongoing Work
Pull Request Checklist
/pr-checklistskill.Summary by CodeRabbit
message_stoporerrorevent, without[DONE].