Skip to content

server: fix format of streamed tool call deltas (diff name, fix id location) #13800

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
May 26, 2025

Conversation

ochafik
Copy link
Collaborator

@ochafik ochafik commented May 26, 2025

Fix format of tool call diffs, now much more conformant to the OAI format:

  • Define id on the tool_call itself, not inside tool_call.function.id
  • Return tool_call.function.name as a delta, not repeated over and over.
  • Define tool_call.type ("function")
  • Return empty tool_call.arguments on the first tool call delta

Fixes #13774 (also reported here cc/ @bgs4free @Rane2021 @TheTerrasque @making @PrideIsLife @tctien342 )

Note this still sends the tool name in one go, as both OpenAI & Anthropic APIs seem to do (which is probably what confused me initially)

Follow up to #12379

Example of outputs
clear && curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-4o",
    "tools": [{
          "type": "function",
          "function": {
              "name": "get_current_weather",
              "description": "Get the current weather",
              "parameters": {
                  "type": "object",
                  "properties": {
                      "location": {
                          "type": "string",
                          "description": "The city and state, e.g. San Francisco, CA"
                      },
                      "format": {
                          "type": "string",
                          "enum": ["celsius", "fahrenheit"],
                          "description": "The temperature unit to use. Infer this from the users location."
                      }
                  },
                  "required": ["location", "format"]
              }
          }
      }, {
          "type": "function",
          "function": {
              "name": "get_n_day_weather_forecast",
              "description": "Get an N-day weather forecast",
              "parameters": {
                  "type": "object",
                  "properties": {
                      "location": {
                          "type": "string",
                          "description": "The city and state, e.g. San Francisco, CA"
                      },
                      "format": {
                          "type": "string",
                          "enum": ["celsius", "fahrenheit"],
                          "description": "The temperature unit to use. Infer this from the users location."
                      },
                      "num_days": {
                          "type": "integer",
                          "description": "The number of days to forecast"
                      }
                  },
                  "required": ["location", "format", "num_days"]
              }
          }
      }],
    "messages": [
      {"role": "system", "content": "You are a function calling AI model. You are provided with function signatures within <tools></tools> XML tags. You may call one or more functions to assist with the user query. Do not make assumptions about what values to plug into functions. Here are the available tools:"},
      {"role": "user", "content": "what is the weather going to be like in San Francisco and Glasgow over the next 4 days"}
    ],
    "stream": true
  }'
data: {"choices":[{"finish_reason":null,"index":0,"delta":{"role":"assistant","content":null}}],"created":1748263544,"id":"chatcmpl-xxesoUemboS3eSaTCzm7PRweLJGrESU2","model":"gpt-4o","system_fingerprint":"b5496-a7e6f3941","object":"chat.completion.chunk"}

data: {"choices":[{"finish_reason":null,"index":0,"delta":{"tool_calls":[{"index":0,"id":"iL1vo9XXQNwkXzcKwyYF4wpGwvRF5HAP","type":"function","function":{"name":"get_current_weather","arguments":""}}]}}],"created":1748263544,"id":"chatcmpl-xxesoUemboS3eSaTCzm7PRweLJGrESU2","model":"gpt-4o","system_fingerprint":"b5496-a7e6f3941","object":"chat.completion.chunk"}

data: {"choices":[{"finish_reason":null,"index":0,"delta":{"tool_calls":[{"index":0,"function":{"arguments":"{"}}]}}],"created":1748263544,"id":"chatcmpl-xxesoUemboS3eSaTCzm7PRweLJGrESU2","model":"gpt-4o","system_fingerprint":"b5496-a7e6f3941","object":"chat.completion.chunk"}

data: {"choices":[{"finish_reason":null,"index":0,"delta":{"tool_calls":[{"index":0,"function":{"arguments":"\""}}]}}],"created":1748263544,"id":"chatcmpl-xxesoUemboS3eSaTCzm7PRweLJGrESU2","model":"gpt-4o","system_fingerprint":"b5496-a7e6f3941","object":"chat.completion.chunk"}

data: {"choices":[{"finish_reason":null,"index":0,"delta":{"tool_calls":[{"index":0,"function":{"arguments":"location"}}]}}],"created":1748263544,"id":"chatcmpl-xxesoUemboS3eSaTCzm7PRweLJGrESU2","model":"gpt-4o","system_fingerprint":"b5496-a7e6f3941","object":"chat.completion.chunk"}

data: {"choices":[{"finish_reason":null,"index":0,"delta":{"tool_calls":[{"index":0,"function":{"arguments":"\":"}}]}}],"created":1748263544,"id":"chatcmpl-xxesoUemboS3eSaTCzm7PRweLJGrESU2","model":"gpt-4o","system_fingerprint":"b5496-a7e6f3941","object":"chat.completion.chunk"}

data: {"choices":[{"finish_reason":null,"index":0,"delta":{"tool_calls":[{"index":0,"function":{"arguments":"\""}}]}}],"created":1748263544,"id":"chatcmpl-xxesoUemboS3eSaTCzm7PRweLJGrESU2","model":"gpt-4o","system_fingerprint":"b5496-a7e6f3941","object":"chat.completion.chunk"}

...

@ochafik ochafik changed the title server: fix streaming deltas of tool_call.name server: fix streaming deltas of tool_call.name May 26, 2025
@github-actions github-actions bot added examples python python script changes server labels May 26, 2025
@ochafik ochafik added the bugfix fixes an issue or bug label May 26, 2025
@github-actions github-actions bot added the testing Everything test related label May 26, 2025
@ochafik ochafik marked this pull request as ready for review May 26, 2025 11:57
@ochafik ochafik requested a review from ngxson as a code owner May 26, 2025 11:58
@ochafik ochafik marked this pull request as draft May 26, 2025 12:16
@ochafik ochafik changed the title server: fix streaming deltas of tool_call.name server: fix streaming deltas of tool_call.name + misplaced tool_call.id May 26, 2025
@ochafik ochafik changed the title server: fix streaming deltas of tool_call.name + misplaced tool_call.id server: fix streaming deltas of tool_call.name & misplaced tool_call.id May 26, 2025
@ochafik ochafik marked this pull request as ready for review May 26, 2025 12:48
@ochafik ochafik requested a review from CISC May 26, 2025 13:00
@ochafik ochafik changed the title server: fix streaming deltas of tool_call.name & misplaced tool_call.id server: fix streaming deltas of tool_call.function.name & misplaced tool_call.id May 26, 2025
@ochafik ochafik changed the title server: fix streaming deltas of tool_call.function.name & misplaced tool_call.id server: fix format of streamed tool call deltas (diff name, fix id location) May 26, 2025
Copy link
Collaborator

@CISC CISC left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@ochafik ochafik merged commit d74e94c into ggml-org:master May 26, 2025
48 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bugfix fixes an issue or bug examples python python script changes server testing Everything test related
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Misc. bug: Streaming with tools causes pydantic-ai to mess up tool name
2 participants