Skip to content

fix(anthropic): enforce Messages API version admission - #17378

Open
maheshrbapatu wants to merge 5 commits into
ai-dynamo:mainfrom
maheshrbapatu:fix/17374-messages-api-versions
Open

maheshrbapatu wants to merge 5 commits into
ai-dynamo:mainfrom
maheshrbapatu:fix/17374-messages-api-versions

Conversation

@maheshrbapatu

@maheshrbapatu maheshrbapatu commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

Overview

Apply anthropic-version admission to both Messages endpoints. Under 2026-10-21, reject removed sampling fields and nonempty citations on user text before generation; omitted headers retain legacy behavior.

Key Changes

Approximately 134 added production-code lines beyond the merged admission validation, excluding tests, comments, and whitespace.

  • validation.rs accepts omitted/2023-06-01/2026-10-21 headers and rejects unsupported or repeated versions.
  • Sampling admission checks field presence, including default, null, and malformed values.
  • The citation visitor scans only messages for the new version, skips bulk values without materializing them, and follows text inside tool/search results.

Validation

Case Result
Unsupported/repeated version headers HTTP 400 with the Anthropic envelope and request ID
New-version temperature/top_p/top_k, including defaults/null Rejected before typed sampling conversion
Nonempty user citations, including nested tool/search text Rejected before generation
Assistant citations, empty user arrays, irrelevant extra fields No citation-specific rejection; normal typed validation still applies
Omitted/legacy version Existing sampling conversion retained

After rebasing onto main 1732fd7, all 27 real-HTTP admission and replay tests pass. These use scripted CPU backends and do not qualify live inference or Anthropic hosted behavior.

The 2026-10-21 citation rule adds a scan of message content. The benchmark below includes that check.

Request fixture Main CPU Updated CPU Main / updated peak heap
1 KiB text 1.33 µs 3.31 µs 0.005 / 0.005 MiB
2 MiB history 0.88 ms 1.86 ms 2.078 / 2.078 MiB
8 MiB image-shaped JSON 1.65 ms 3.74 ms 8.003 / 8.003 MiB
44 MiB image-shaped JSON 24.76 ms 37.46 ms 44.003 / 44.003 MiB

Admission-only release benchmark against main cd3b977ed, recorded for the original version-admission change with protocol 8.4.0 before this rebase: 9 warmed, alternating batches on each of two pinned CPUs, with separate allocation runs. Values are median batch-average thread CPU, not request p95. Peak heap excludes the ingress buffer and allocator overhead. HTTP transport, tokenization, inference, and image fetch/decode are excluded; these figures do not establish end-to-end throughput.

Issues and Ongoing Work

Pull Request Checklist
  • I used Dynamo's /pr-checklist skill.

Summary by CodeRabbit

  • API Updates
    • Anthropic Messages responses and errors now include matching request IDs.
    • Invalid requests receive consistent JSON errors; unsupported API versions and parameters are rejected.
    • Documented request body limit is now 45 MB, down from 192 MB.
    • Anthropic streaming responses end with a message_stop or error event, without [DONE].
  • Documentation
    • Added details on request validation, API-version behavior, and token-counting support.

@copy-pr-bot

copy-pr-bot Bot commented Oct 27, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@github-actions

Copy link
Copy Markdown
Contributor

👋 Hi maheshrbapatu! Thank you for contributing to ai-dynamo/dynamo.

Just a reminder: The NVIDIA Test Github Validation CI runs an essential subset of the testing framework to quickly catch errors.Your PR reviewers may elect to test the changes comprehensively before approving your changes.

🚀

@github-actions github-actions Bot added fix external-contribution Pull request is from an external contributor documentation Improvements or additions to documentation frontend `python -m dynamo.frontend` and `dynamo-run in=http|text|grpc` labels Oct 27, 2026
@maheshrbapatu
maheshrbapatu force-pushed the fix/17374-messages-api-versions branch from 8855eff to 83039d7 Compare October 27, 2026 17:46
@maheshrbapatu
maheshrbapatu temporarily deployed to external_collaborator October 27, 2026 17:46 — with GitHub Actions Inactive
@maheshrbapatu
maheshrbapatu marked this pull request as ready for review October 27, 2026 17:52
@maheshrbapatu
maheshrbapatu requested review from a team as code owners October 27, 2026 17:52
@coderabbitai

coderabbitai Bot commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

Review in Change Stack →

No actionable comments were generated in the recent review.

Walkthrough

The change adds request admission rules and request-ID error envelopes for Anthropic Messages endpoints. It also updates streaming monitor configuration and SSE test parsing, and documents compatibility rules and a lower documented HTTP body limit.

Changes

Anthropic HTTP behavior

Layer / File(s) Summary
Raw request validation
lib/llm/Cargo.toml, lib/llm/src/http/service/anthropic/validation.rs, lib/llm/src/protocols/anthropic/types.rs
Raw JSON is retained for version and field checks before typed deserialization. Named tool choices now reject invalid discriminators or empty names.
HTTP admission and response envelopes
lib/llm/src/http/service/anthropic.rs, lib/llm/tests/anthropic_http_admission.rs, lib/llm/tests/frontend_protocol_validation_http.rs, docs/fern/pages/reference/components/frontend-configuration.mdx
Both Messages endpoints use validated requests. Error responses include Anthropic JSON envelopes and request IDs. HTTP tests cover admission rules and response IDs. The documentation describes version rules and changes the documented body-limit default from 192 MB to 45 MB.
Stream monitor and SSE validation
lib/llm/src/http/service/disconnect.rs, lib/llm/src/http/service/openai.rs, lib/llm/src/http/service/anthropic.rs, lib/llm/tests/common/http_harness.rs, lib/llm/tests/anthropic_http_replay.rs
The disconnect monitor receives an explicit done-sentinel setting. SSE test parsing rejects [DONE], requires a protocol terminal event, and rejects events after that terminal event.

Estimated code review effort: 4 (Complex) | ~45 minutes

Merge Risk: Minimal · up to 83039

No identified issue from these changes remains to block merging after normal checks.

Pre-merge checks: Passed 3 · Failed 1

Failed Failed checks: 1 warning

Check name Status Explanation Resolution
Out of Scope Changes check Warning docs/fern/pages/reference/components/frontend-configuration.mdx changes the documented default for DYN_HTTP_BODY_LIMIT_MB from 192 MB to 45 MB. Issue [#17374] explicitly excludes changes to the HT… Remove the unrelated body-size default change from this pull request, or move it to a separate change with its own requirement and validation.
Passed Passed checks: 3 passed
Check name Status Explanation
Title check Passed The title clearly and concisely identifies the main change: enforcing Anthropic Messages API version admission.
Linked Issues check Passed The PR satisfies the coding requirements in [#17374]. ValidatedRequest::into_request accepts omitted anthropic-version and 2023-06-01, accepts 2026-10-21 with sampling-field rejection, and rej…
Description check Passed The pull request description follows the required template. It includes Overview, Key Changes, Validation, Issues and Ongoing Work, the issue-closing reference, and the checklist. It also documents sc…
Full details: Linked Issues check

Explanation

The PR satisfies the coding requirements in [#17374]. ValidatedRequest::into_request accepts omitted anthropic-version and 2023-06-01, accepts 2026-10-21 with sampling-field rejection, and rejects unsupported, malformed, or repeated headers. It checks field presence before typed conversion, so temperature, top_p, and top_k are rejected for 2026-10-21 when they are null, default-valued, or malformed. Both Messages handlers use this admission path. The error middleware supplies Anthropic error envelopes and matching request IDs. The reported real HTTP tests cover both endpoints, valid and omitted headers, invalid and repeated headers, sampling cases, streaming and unary responses, and rejected requests without backend dispatch. The documentation states the supported versions, omitted-header policy, sampling limitation, and non-parity scope.

Full details: Description check

Explanation

The pull request description follows the required template. It includes Overview, Key Changes, Validation, Issues and Ongoing Work, the issue-closing reference, and the checklist. It also documents scope, test coverage, limitations, and benchmark results.

Full details: Out of Scope Changes check

Explanation

docs/fern/pages/reference/components/frontend-configuration.mdx changes the documented default for DYN_HTTP_BODY_LIMIT_MB from 192 MB to 45 MB. Issue [#17374] explicitly excludes changes to the HTTP body-size limit. This change is unrelated to version admission and sampling-parameter enforcement.

Resolution

Remove the unrelated body-size default change from this pull request, or move it to a separate change with its own requirement and validation.

Finishing Touches
Fix failing CI checks
  • Commit to this branch
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Tip

🎯 Customize your risk assessments

Guide how CodeRabbit assesses merge risk and security architecture risk. Highlight sensitive changes and explain your security boundaries using Merge Risk Instructions and Security Risk Instructions in your review settings. Security architecture risk assessments require the Advanced plan or above.

✅ Automatic approvals, with your rules

Let CodeRabbit approve pull requests once required threads are resolved, pre-merge checks pass, and your risk limits are met. Enable Automatic approvals in review settings, with overrides for teams or file paths.

CodeRabbit Verify

  • Run CodeRabbit Verify to test the changes in this pull request.
Recent review info
Run configuration
Commits

Reviewing files that changed from the base of the PR and between 95f1b12 and 83039d7.

Files selected for processing: 11
  • docs/fern/pages/reference/components/frontend-configuration.mdx
  • lib/llm/Cargo.toml
  • lib/llm/src/http/service/anthropic.rs
  • lib/llm/src/http/service/anthropic/validation.rs
  • lib/llm/src/http/service/disconnect.rs
  • lib/llm/src/http/service/openai.rs
  • lib/llm/src/protocols/anthropic/types.rs
  • lib/llm/tests/anthropic_http_admission.rs
  • lib/llm/tests/anthropic_http_replay.rs
  • lib/llm/tests/common/http_harness.rs
  • lib/llm/tests/frontend_protocol_validation_http.rs

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.

@dynamo-review-agent dynamo-review-agent Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Questions for the author:

  • lib/llm/src/http/service/anthropic/validation.rs: For every supported /v1/messages inference request, identity() parses the full raw JSON into a newly allocated HashMap, into_request() immediately repeats that full-body parse/allocation, and typed deserialization then scans it again. Since this path supports request bodies up to 45 MiB, please provide admission-latency or CPU measurements for representative large/500k-token Messages requests versus the prior direct typed extraction, or calculations showing the extra full-body pass is negligible.

}
match version.map(|value| value.to_str()) {
None | Some(Ok("2023-06-01")) => {}
Some(Ok("2026-10-21")) => {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The 2026-10-21 branch enforces only the removed sampling parameters, but that version also requires HTTP 400 when a user text block contains a non-empty citations array, directly or inside a search_result. AnthropicContentBlock::Text currently accepts those citations and later conversion discards them, while recognized WebSearchToolResult content is not traversed and the block is ignored during conversion, so these requests reach generation instead of failing admission. Inspect message content under this version and reject both citation shapes before dispatch.

🤖 AI Fix

Extend the 2026-10-21 admission branch to reject non-empty citations in user text blocks, including text nested inside search-result blocks, before typed conversion and generation.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added the missing 2026-10-21 citation admission check on both endpoints. It rejects nonempty citations on user text, including nested tool/search result text, while preserving assistant citations and empty user arrays. The HTTP regression also checks that rejection happens before generation.

}
match version.map(|value| value.to_str()) {
None | Some(Ok("2023-06-01")) => {}
Some(Ok("2026-10-21")) => {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The 2026-10-21 branch only rejects sampling fields, so requests with non-empty citations still reach generation. Text.citations deserializes as a recognized field, while validate_anthropic_messages ignores it and conversion discards it; WebSearchToolResult.content similarly permits nested citation payloads that conversion ignores. This violates the version's required 400 admission behavior and silently removes client content.

🤖 AI Fix

Extend 2026-10-21 admission to reject non-empty citations in text blocks and citations nested in search-result content before backend dispatch.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Covered by the citation admission fix: nonempty user citations now return HTTP 400 before conversion. The tolerant visitor follows supported tool/search containers, skips unrelated values, and leaves malformed-shape diagnostics to the original typed parser.

),
];
for path in ["/v1/messages", "/v1/messages/count_tokens"] {
for stream in [false, true] {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The stream matrix doubles every duplicate-key request on both endpoints without protecting another regression. ValidatedRequest::fields() rejects the duplicate raw JSON before typed deserialization or any streaming branch, so the retained stream: false requests still verify duplicate detection and both handler integrations.

🤖 AI Fix

Remove the stream loop and run each duplicate-key body once per endpoint.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed the duplicate-key stream cross-product. Both endpoints still reject duplicate known fields through the original typed parser; unary/streaming metric-label behavior is covered separately.

),
("thinking", json!(true)),
];
for stream in [false, true] {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Both stream values exercise the same pre-dispatch validator for all 39 malformed values. stream is not among the fields passed to validate, and every case is rejected before response streaming begins, so one stream value preserves every field-specific validation contract while avoiding 39 duplicate HTTP requests.

🤖 AI Fix

Remove the stream parameterization and run the malformed-field table once with a fixed stream value.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed the malformed-field stream cross-product in #17367 and rebased this PR onto it. Each invalid shape still checks both endpoints and no backend dispatch.

StatusCode::BAD_REQUEST,
"invalid_request_error",
),
(

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The missing-model case repeats the unary missing-model request already selected in anthropic_handler_errors_record_classification_with_response, whose changed assert_anthropic_status helper now verifies the response request ID matches the header. Its only additional checks here are generic envelope properties, which can be retained once in that shared helper rather than issuing another identical missing-model request.

🤖 AI Fix

Move the generic nonempty request-ID and content-type assertions into assert_anthropic_status, then remove this missing-model tuple from the new test.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed the duplicate missing-model request and moved the nonempty request-ID/JSON-content-type assertions into the existing error-classification helper. The remaining max_tokens request-ID case is a single direct case.

];
for (path, stream) in [
("/v1/messages", false),
("/v1/messages", true),

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Repeating every invalid header with stream: true cannot distinguish another version-admission regression: into_request checks anthropic-version before typed conversion and before the Messages handler can select its streaming response path. The unary Messages case retains the header-value boundaries, and the count-tokens case retains coverage of the second handler wiring.

🤖 AI Fix

Remove the streaming Messages tuple from the invalid-version route matrix.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed the redundant streaming invalid-version matrix. Unary requests cover both endpoints; separate successful streaming requests retain version/frame coverage.

let svc = HarnessService::start([script]).await;
for (path, stream) in [
("/v1/messages", false),
("/v1/messages", true),

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The streaming Messages tuple repeats all 12 sampling-field cases after the same early fields.contains_key admission check exercised by the unary tuple. Rejection occurs before deserialization and response-mode branching, while the remaining unary and count-tokens tuples still protect both endpoint integrations and every field/value boundary.

🤖 AI Fix

Remove the streaming Messages tuple from this rejected-sampling matrix.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed the streaming repetition of sampling-field rejection. Both endpoints still cover present defaults, null, and malformed values, and legacy conversion remains covered.

@maheshrbapatu
maheshrbapatu force-pushed the fix/17374-messages-api-versions branch from 83039d7 to df31a5c Compare October 28, 2026 05:19
@maheshrbapatu
maheshrbapatu requested review from a team as code owners October 28, 2026 05:19
@maheshrbapatu
maheshrbapatu temporarily deployed to external_collaborator October 28, 2026 05:19 — with GitHub Actions Inactive
@maheshrbapatu
maheshrbapatu force-pushed the fix/17374-messages-api-versions branch from df31a5c to ae164ea Compare October 28, 2026 05:40
@maheshrbapatu
maheshrbapatu temporarily deployed to external_collaborator October 28, 2026 05:40 — with GitHub Actions Inactive
@maheshrbapatu

Copy link
Copy Markdown
Contributor Author

I added the performance measurements to the description. With the 2026-10-21 citation check enabled, admission takes 3.74 ms versus 1.65 ms on main for the 8 MiB image-shaped JSON fixture, and 37.46 ms versus 24.76 ms for 44 MiB. The extra full-body copy is gone; version-specific citation validation still adds a message scan.

These are median batch-average CPU measurements from warmed release builds on two pinned CPUs, with peak heap measured separately and excluding the ingress buffer. They cover parsing/admission only, not HTTP transport, tokenization, image decoding, or inference. The description includes the smaller-request results and remaining cost.

Accept 2023-06-01 and 2026-10-21, default omitted version headers to legacy behavior, and reject unsupported or repeated headers before generation. Reject sampling fields under 2026-10-21 while preserving explicit null presence and duplicate-key rejection.

Cover unary, streaming, and token-counting admission through real HTTP tests, and document the bounded compatibility policy.

Closes ai-dynamo#17374

Signed-off-by: maheshrbapatu <maheshbapatu@gmail.com>
Signed-off-by: maheshrbapatu <maheshbapatu@gmail.com>
Signed-off-by: maheshrbapatu <maheshbapatu@gmail.com>
@maheshrbapatu
maheshrbapatu force-pushed the fix/17374-messages-api-versions branch from ae164ea to 6997269 Compare November 3, 2026 19:55
@maheshrbapatu
maheshrbapatu temporarily deployed to external_collaborator November 3, 2026 19:55 — with GitHub Actions Inactive
Signed-off-by: maheshrbapatu <maheshbapatu@gmail.com>
@maheshrbapatu
maheshrbapatu temporarily deployed to external_collaborator November 3, 2026 20:24 — with GitHub Actions Inactive
Signed-off-by: maheshrbapatu <maheshbapatu@gmail.com>

This branch was previously deployed

1 inactive deployment
external_collaborator — 2fb92c29 Deployed Nov 3, 2026 by maheshrbapatu via ok-to-test #44503
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation external-contribution Pull request is from an external contributor fix frontend `python -m dynamo.frontend` and `dynamo-run in=http|text|grpc` size/L

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(anthropic): enforce supported API versions at admission

1 participant