Bedrock streaming minimal#496
Merged
droot merged 7 commits intoAug 26, 2025
Merged
Conversation
Addresses the core AWS SDK document marshaler/unmarshaler limitation identified in streaming mode where AsFunctionCalls() returns empty arguments. Essential changes only: - Add args field to bedrockToolPart for pre-parsed streaming arguments - Update AsFunctionCalls() with fallback logic (streaming -> unmarshaling) - Add tool state tracking and JSON parsing in ContentBlockStop handler - Extend streaming response structures to pass parsed arguments - Update conversation history to include completed tools Fixes: Empty tool arguments in streaming mode due to AWS SDK internal types Root cause: Cannot create documentUnmarshaler instances for streaming case
Revert the full Initialize() implementation back to original stub to keep only essential streaming tool call fixes. Changes: - Revert Initialize() from full implementation to error stub - Remove unnecessary message conversion logic - Keep only core streaming tool call fixes (6 essential changes) This maintains the minimal fix approach focused solely on the AWS SDK document marshaler/unmarshaler limitation.
- Implement proper Initialize() method for conversation history - Add addContentsToHistory() for better content processing - Refactor Send() and SendStreaming() to use shared content processing - Fix tool use block creation with proper Input document - Add support for FunctionCallResult in conversation history
Both Claude 3.7 Sonnet and Claude Sonnet 4 achieved 10/10 (100%) success rate on standardized Kubernetes benchmark tasks, demonstrating excellent performance with native tool calling implementation.
- Keep AWS Bedrock 100% benchmark results - Maintain full Initialize() implementation for Bedrock - Integrate upstream changes for other providers
droot
requested changes
Aug 19, 2025
- Fix README.md benchmark table ordering (gemini first, remove bold) - Implement dynamic status detection in Bedrock integration - Check Result.error and Result.status fields for failure indicators - Set ToolResultStatusError when errors detected - Fix MCP client to return structured error data instead of Go errors - Preserve error content in JSON format for proper LLM processing - Prevents conversion of business logic failures to execution errors This resolves the issue where all tool results were hardcoded as success, causing LLMs to retry failed operations indefinitely until max iterations.
droot
approved these changes
Aug 26, 2025
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
AWS Bedrock Integration Fixes:
Summary
This PR fixes critical issues with AWS Bedrock integration, enabling full streaming support with tool calls and proper conversation history initialization. These changes bring AWS Bedrock to feature parity with other providers and achieve 100% success rate on benchmark tests.
Key Changes
Fixed Streaming Tool Calls
Resolved AWS SDK document marshaler/unmarshaler limitation that caused empty tool arguments in streaming mode
Added tool state tracking to accumulate streaming JSON input
Implemented proper argument parsing in ContentBlockStop handler
Extended bedrockToolPart with pre-parsed arguments for streaming case
Implemented Conversation Initialization
Added full Initialize() method to restore conversation history from previous sessions
Properly converts api.Message to AWS Bedrock types.Message format
Supports both user and assistant message roles
Enhanced Content Processing
Introduced addContentsToHistory() for unified content handling
Added support for FunctionCallResult in conversation history
Refactored Send() and SendStreaming() to use shared content processing logic
Properly handles tool results with AWS Bedrock's ToolResultBlock format
Improved Streaming Response Handling
Added streamingArgs field to pass parsed arguments through streaming pipeline
Updated conversation history to include completed tools from streaming
Removed verbose logging for cleaner output
Performance Impact
Before: 90% success rate (9/10 tests) for Claude Sonnet 4
After: 100% success rate (10/10 tests) for both Claude 3.7 and Claude Sonnet 4, Native tool calling without tool shim enabled
Testing
All benchmark tests passing (10/10)
Streaming tool calls working correctly
Conversation history preservation functional
Usage metadata extraction verified
Breaking Changes
None - All changes are backward compatible.
Related Issues
Fixes streaming tool call failures in AWS Bedrock integration where AsFunctionCalls() would return empty arguments due to AWS SDK internal type limitations.