Repository navigation
fix(anthropic): align Messages framing, errors, and validation - #17367
Conversation
|
No actionable comments were generated in the recent review.
|
| Layer / File(s) | Summary |
|---|---|
Request validation and error responses lib/llm/Cargo.toml, lib/llm/src/http/service/anthropic.rs, lib/llm/src/http/service/anthropic/validation.rs, lib/llm/src/protocols/anthropic/types.rs, lib/llm/tests/anthropic_http_admission.rs, lib/llm/tests/frontend_protocol_validation_http.rs, docs/fern/pages/reference/components/frontend-configuration.mdx |
The service validates raw request JSON and checks system, tool_choice, metadata, and thinking fields. Middleware assigns request IDs and formats Anthropic error responses. Tests cover malformed, invalid, and valid requests on both Messages routes. |
Streaming termination behavior lib/llm/src/http/service/disconnect.rs, lib/llm/src/http/service/openai.rs, lib/llm/src/http/service/anthropic.rs, lib/llm/tests/common/http_harness.rs, lib/llm/tests/anthropic_http_replay.rs |
The disconnect monitor now takes an explicit sentinel setting. Anthropic stream tests require message_stop or error as the final event and reject [DONE]; OpenAI chat streaming passes true. |
Priority:
Estimated code review effort: 3 (Moderate) | ~25 minutes
Severity of issue fixed: Medium
Merge Risk: 
Minimal · up to 8e092
-
No actionable merge-blocking issue is established; the change is ready for normal merge checks.
Pre-merge checks: 
4
-
Full details: Title check
-
Explanation
The title "fix(anthropic): align Messages framing, errors, and validation" matches the main changeset. The changes end Messages streams at message_stop without [DONE], return Anthropic JSON error envelopes with request IDs, and add request validation before backend dispatch. The title is concise, specific, and follows a conventional commit style.
Full details: Linked Issues check
-
Explanation
The PR satisfies the coding requirements in [
#17362]. The Anthropic middleware assigns request IDs before extraction, normalizes extractor and handler failures to Anthropic envelopes, preserves 413request_too_large, and adds matchingrequest-idheaders andrequest_idfields.ValidatedRequestchecks tool choices, system blocks, metadata, and thinking budgets before typed conversion, then deserializes the original raw JSON to retain duplicate-field rejection. Anthropic streams omit[DONE]; OpenAI passestruefor the sentinel. The HTTP admission tests cover valid controls, rejected requests without backend dispatch, duplicate fields, request IDs, count-token behavior, and response framing.
Full details: Out of Scope Changes check
-
Explanation
The changes stay within [
#17362]. The validation module, Anthropic type regression coverage, disconnect-monitor option, OpenAI call-site update, HTTP harness assertions, and frontend documentation directly support the requested protocol behavior. Theserde_jsonfeature enables raw request preservation. No unrelated implementation or test changes are demonstrated.
Full details: Description check
-
Explanation
The pull request description includes all required sections: Overview, Key Changes, Validation, Issues and Ongoing Work, issue references, and the checklist. It clearly explains the implementation, validation results, scope, and remaining work.
-
- Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Comment @coderabbitai help to get the list of available commands.
There was a problem hiding this comment.
Questions for the author:
- lib/llm/src/http/service/anthropic/validation.rs: Every supported
/v1/messagesrequest is now parsed once intoserde_json::Valuefor validation and then parsed again into the typed request, adding a full extra JSON parse and allocation cycle on the Anthropic inference path. Could you provide admission-latency or CPU/throughput measurements versus the previous single-parse path for representative requests, including a large multimodal/tool payload, so the runtime impact can be assessed?
8e0920c to
10b15b5
Compare
Signed-off-by: maheshrbapatu <maheshbapatu@gmail.com>
Signed-off-by: maheshrbapatu <maheshbapatu@gmail.com>
Signed-off-by: maheshrbapatu <maheshbapatu@gmail.com>
Signed-off-by: maheshrbapatu <maheshbapatu@gmail.com>
Signed-off-by: maheshrbapatu <maheshbapatu@gmail.com>
b1347ea to
ccb31ed
Compare
|
/ok to test ccb31ed |
✅ Dynamo PR CI passed — run 40929474118 (attempt 1) on
|
Signed-off-by: Ryan McCormick <mccormick.codes@gmail.com>
Signed-off-by: Ryan McCormick <mccormick.codes@gmail.com>
Signed-off-by: Ryan McCormick <mccormick.codes@gmail.com>
|
/ok to test b177b91 |
Overview
Messages clients receive Anthropic error envelopes with matching request IDs for malformed bodies and invalid request shapes. Successful streams stop at
message_stop, and typed parsing continues to reject duplicate fields.Key Changes
Approximately 272 added production-code lines, excluding tests, comments, and whitespace.
top_krange.Validation
extractor_errors_have_anthropic_envelopes_and_request_ids[DONE]Combined revision
36442d66aon maincd3b977edpasses 87 real HTTP tests (including 27 Responses regressions), 136 Anthropic unit tests, andcargo test --locked --all-targetsin the independent Python (83 tests) and KVBM (43 tests) workspaces. HTTP tests use scripted CPU backends; they do not qualify live inference or Anthropic hosted behavior.Two linear parsing passes remain to preserve typed errors and duplicate-key behavior.
Admission-only release benchmark against main
cd3b977ed, using protocol 8.4.0 and the unchanged validator from this head: 9 warmed, alternating batches on each of two pinned CPUs, with separate allocation runs. Values are median batch-average thread CPU, not request p95. Peak heap excludes the ingress buffer and allocator overhead. HTTP transport, tokenization, inference, and image fetch/decode are excluded; these figures do not establish end-to-end throughput.Issues and Ongoing Work
Pull Request Checklist
/pr-checklistskill.Summary by CodeRabbit
message_stopand do not include[DONE].