Skip to content

fix(anthropic): align Messages framing, errors, and validation - #17367

Merged
rmccorm4 merged 10 commits into
ai-dynamo:mainfrom
maheshrbapatu:fix/messages-spec-admission
Nov 3, 2026
Merged

rmccorm4 merged 10 commits into
ai-dynamo:mainfrom
maheshrbapatu:fix/messages-spec-admission

Conversation

@maheshrbapatu

@maheshrbapatu maheshrbapatu commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

Overview

Messages clients receive Anthropic error envelopes with matching request IDs for malformed bodies and invalid request shapes. Successful streams stop at message_stop, and typed parsing continues to reject duplicate fields.

Key Changes

Approximately 272 added production-code lines, excluding tests, comments, and whitespace.

  • validation.rs retains ingress bytes, validates selected admission fields, then uses the original typed parser. The extra body copy and repeated identity scan are removed.
  • anthropic.rs normalizes errors/request IDs and uses the shared stream monitor with the DONE sentinel disabled.
  • types.rs validates the tool-choice discriminator before conversion; admission also checks system, metadata, thinking budgets, and the signed top_k range.

Validation

Case Before After / regression
Malformed JSON, wrong content type, oversized body Plain-text errors JSON envelope plus matching header/body request ID; extractor_errors_have_anthropic_envelopes_and_request_ids
Duplicate known keys or bytes after the JSON body Duplicate keys rejected; trailing bytes accepted Original duplicate rejection retained, trailing bytes rejected on both endpoints
Invalid shapes or tool discriminator/name combinations Accepted or converted incorrectly HTTP 400 before backend dispatch, with classified failure metrics
Enabled/interleaved thinking Inconsistent admission Minimum budget enforced; the interleaved beta permits budgets at/above max_tokens
Stream termination Additional [DONE] Messages and Responses omit it; Chat/completions retain it

Combined revision 36442d66a on main cd3b977ed passes 87 real HTTP tests (including 27 Responses regressions), 136 Anthropic unit tests, and cargo test --locked --all-targets in the independent Python (83 tests) and KVBM (43 tests) workspaces. HTTP tests use scripted CPU backends; they do not qualify live inference or Anthropic hosted behavior.

Two linear parsing passes remain to preserve typed errors and duplicate-key behavior.

Request fixture Main CPU Updated CPU Main / updated peak heap
1 KiB text 1.36 µs 2.70 µs 0.005 / 0.005 MiB
2 MiB history 0.89 ms 1.36 ms 2.078 / 2.078 MiB
8 MiB image-shaped JSON 1.64 ms 2.80 ms 8.003 / 8.003 MiB
44 MiB image-shaped JSON 24.87 ms 31.60 ms 44.003 / 44.003 MiB

Admission-only release benchmark against main cd3b977ed, using protocol 8.4.0 and the unchanged validator from this head: 9 warmed, alternating batches on each of two pinned CPUs, with separate allocation runs. Values are median batch-average thread CPU, not request p95. Peak heap excludes the ingress buffer and allocator overhead. HTTP transport, tokenization, inference, and image fetch/decode are excluded; these figures do not establish end-to-end throughput.

Issues and Ongoing Work

Pull Request Checklist
  • I used Dynamo's /pr-checklist skill.

Summary by CodeRabbit

  • New Features
    • Anthropic Messages responses and errors now include a request ID, with errors returned in Anthropic’s JSON format.
    • Requests with malformed JSON, invalid fields, or incorrect/missing JSON content types receive HTTP 400 responses. Requests exceeding the 45 MiB default limit receive HTTP 413.
    • Anthropic streams now end with message_stop and do not include [DONE].
  • Documentation
    • Updated the frontend configuration reference with endpoint behavior, request validation rules, and thinking-budget requirements.

@maheshrbapatu
maheshrbapatu requested review from a team as code owners October 26, 2026 23:55
@copy-pr-bot

copy-pr-bot Bot commented Oct 26, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@github-actions

github-actions Bot commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

@github-actions github-actions Bot added fix external-contribution Pull request is from an external contributor documentation Improvements or additions to documentation frontend `python -m dynamo.frontend` and `dynamo-run in=http|text|grpc` labels Oct 26, 2026
@coderabbitai

coderabbitai Bot commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

Review in Change Stack →

No actionable comments were generated in the recent review.

Recent review info
Run configuration
Commits

Reviewing files that changed from the base of the PR and between 98f74e1 and 8e0920c.

Files selected for processing: 11
  • docs/fern/pages/reference/components/frontend-configuration.mdx
  • lib/llm/Cargo.toml
  • lib/llm/src/http/service/anthropic.rs
  • lib/llm/src/http/service/anthropic/validation.rs
  • lib/llm/src/http/service/disconnect.rs
  • lib/llm/src/http/service/openai.rs
  • lib/llm/src/protocols/anthropic/types.rs
  • lib/llm/tests/anthropic_http_admission.rs
  • lib/llm/tests/anthropic_http_replay.rs
  • lib/llm/tests/common/http_harness.rs
  • lib/llm/tests/frontend_protocol_validation_http.rs

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.

Walkthrough

The Anthropic Messages endpoint now validates request shapes before deserialization, returns Anthropic-formatted errors with request IDs, and uses protocol-specific streaming termination. Tests cover admission, error responses, and stream parsing.

Changes

Anthropic Messages API

Layer / File(s) Summary
Request validation and error responses
lib/llm/Cargo.toml, lib/llm/src/http/service/anthropic.rs, lib/llm/src/http/service/anthropic/validation.rs, lib/llm/src/protocols/anthropic/types.rs, lib/llm/tests/anthropic_http_admission.rs, lib/llm/tests/frontend_protocol_validation_http.rs, docs/fern/pages/reference/components/frontend-configuration.mdx
The service validates raw request JSON and checks system, tool_choice, metadata, and thinking fields. Middleware assigns request IDs and formats Anthropic error responses. Tests cover malformed, invalid, and valid requests on both Messages routes.
Streaming termination behavior
lib/llm/src/http/service/disconnect.rs, lib/llm/src/http/service/openai.rs, lib/llm/src/http/service/anthropic.rs, lib/llm/tests/common/http_harness.rs, lib/llm/tests/anthropic_http_replay.rs
The disconnect monitor now takes an explicit sentinel setting. Anthropic stream tests require message_stop or error as the final event and reject [DONE]; OpenAI chat streaming passes true.

Priority: Normal

Estimated code review effort: 3 (Moderate) | ~25 minutes

Severity of issue fixed: Medium

Merge Risk: Minimal · up to 8e092

No actionable merge-blocking issue is established; the change is ready for normal merge checks.

Pre-merge checks: Passed 4
Check name Status Explanation
Title check Passed The title "fix(anthropic): align Messages framing, errors, and validation" matches the main changeset. The changes end Messages streams at message_stop without [DONE], return Anthropic JSON error enve…
Linked Issues check Passed The PR satisfies the coding requirements in [#17362]. The Anthropic middleware assigns request IDs before extraction, normalizes extractor and handler failures to Anthropic envelopes, preserves 413 `r…
Out of Scope Changes check Passed The changes stay within [#17362]. The validation module, Anthropic type regression coverage, disconnect-monitor option, OpenAI call-site update, HTTP harness assertions, and frontend documentation dir…
Description check Passed The pull request description includes all required sections: Overview, Key Changes, Validation, Issues and Ongoing Work, issue references, and the checklist. It clearly explains the implementation, va…
Full details: Title check

Explanation

The title "fix(anthropic): align Messages framing, errors, and validation" matches the main changeset. The changes end Messages streams at message_stop without [DONE], return Anthropic JSON error envelopes with request IDs, and add request validation before backend dispatch. The title is concise, specific, and follows a conventional commit style.

Full details: Linked Issues check

Explanation

The PR satisfies the coding requirements in [#17362]. The Anthropic middleware assigns request IDs before extraction, normalizes extractor and handler failures to Anthropic envelopes, preserves 413 request_too_large, and adds matching request-id headers and request_id fields. ValidatedRequest checks tool choices, system blocks, metadata, and thinking budgets before typed conversion, then deserializes the original raw JSON to retain duplicate-field rejection. Anthropic streams omit [DONE]; OpenAI passes true for the sentinel. The HTTP admission tests cover valid controls, rejected requests without backend dispatch, duplicate fields, request IDs, count-token behavior, and response framing.

Full details: Out of Scope Changes check

Explanation

The changes stay within [#17362]. The validation module, Anthropic type regression coverage, disconnect-monitor option, OpenAI call-site update, HTTP harness assertions, and frontend documentation directly support the requested protocol behavior. The serde_json feature enables raw request preservation. No unrelated implementation or test changes are demonstrated.

Full details: Description check

Explanation

The pull request description includes all required sections: Overview, Key Changes, Validation, Issues and Ongoing Work, issue references, and the checklist. It clearly explains the implementation, validation results, scope, and remaining work.

  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Comment @coderabbitai help to get the list of available commands.

@dynamo-review-agent dynamo-review-agent Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Questions for the author:

  • lib/llm/src/http/service/anthropic/validation.rs: Every supported /v1/messages request is now parsed once into serde_json::Value for validation and then parsed again into the typed request, adding a full extra JSON parse and allocation cycle on the Anthropic inference path. Could you provide admission-latency or CPU/throughput measurements versus the previous single-parse path for representative requests, including a large multimodal/tool payload, so the runtime impact can be assessed?

Comment thread lib/llm/src/http/service/anthropic.rs Resolved
Signed-off-by: maheshrbapatu <maheshbapatu@gmail.com>
Signed-off-by: maheshrbapatu <maheshbapatu@gmail.com>
Signed-off-by: maheshrbapatu <maheshbapatu@gmail.com>
Signed-off-by: maheshrbapatu <maheshbapatu@gmail.com>
Signed-off-by: maheshrbapatu <maheshbapatu@gmail.com>
@maheshrbapatu
maheshrbapatu force-pushed the fix/messages-spec-admission branch from b1347ea to ccb31ed Compare October 28, 2026 05:40
@maheshrbapatu
maheshrbapatu temporarily deployed to external_collaborator October 28, 2026 05:40 — with GitHub Actions Inactive
@rmccorm4

Copy link
Copy Markdown
Contributor

/ok to test ccb31ed

@devin-ai-integration

Copy link
Copy Markdown
Contributor

✅ Dynamo PR CI passed — run 40929474118 (attempt 1) on ccb31ed8f2

Gate checks: ✅ backend-status-check · ✅ deploy-status-check · ✅ dynamo-status-check

Posted automatically by Devin for run 40929474118. Updated on every full-CI run of this PR.

Signed-off-by: Ryan McCormick <mccormick.codes@gmail.com>
@rmccorm4
rmccorm4 temporarily deployed to external_collaborator October 31, 2026 20:52 — with GitHub Actions Inactive
Signed-off-by: Ryan McCormick <mccormick.codes@gmail.com>
@rmccorm4
rmccorm4 temporarily deployed to external_collaborator October 31, 2026 20:53 — with GitHub Actions Inactive
Signed-off-by: Ryan McCormick <mccormick.codes@gmail.com>
@rmccorm4
rmccorm4 temporarily deployed to external_collaborator October 31, 2026 20:53 — with GitHub Actions Inactive
@rmccorm4
rmccorm4 temporarily deployed to external_collaborator November 3, 2026 05:25 — with GitHub Actions Inactive
@rmccorm4

rmccorm4 commented Nov 3, 2026

Copy link
Copy Markdown
Contributor

/ok to test b177b91

This branch was previously deployed

1 inactive deployment
external_collaborator — b177b914 Deployed Nov 3, 2026 by rmccorm4 via ok-to-test #44007
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation external-contribution Pull request is from an external contributor fix frontend `python -m dynamo.frontend` and `dynamo-run in=http|text|grpc` size/XL

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(anthropic): reject trailing data after JSON request bodies fix(anthropic): align Messages streaming, errors, request IDs, and validation

2 participants