Skip to content

fix: fix tool-calls finish reason conversion in streaming mode for newer gemini models like gemini-3.5-flash - #2399

Merged
aabchoo merged 3 commits into
envoyproxy:mainfrom
hustxiayang:fix-gemini-3.5-streaming-finish-reason
Jul 20, 2026
Merged

fix: fix tool-calls finish reason conversion in streaming mode for newer gemini models like gemini-3.5-flash#2399
aabchoo merged 3 commits into
envoyproxy:mainfrom
hustxiayang:fix-gemini-3.5-streaming-finish-reason

Conversation

@hustxiayang

@hustxiayang hustxiayang commented Jul 20, 2026

Copy link
Copy Markdown
Contributor

Description

Problem

When the model returns a tool call, OpenAI expects the finish reason to be tool_calls while Gemini returns STOP. We convert it via

if len(toolCalls) > 0 {
			return openai.ChatCompletionChoicesFinishReasonToolCalls
		}

This was correct for streaming mode because the finish reason and the tool calls are in the same chunk in the previous Gemini models. However, for newer Gemini models like gemini-3.5-flash, they are split into two chunks. The functionCall chunk has
no finishReason, and a trailing chunk carries finishReason: STOP with an empty
text part and no functionCall:

// raw GCP chunk 1 — has the tool call, no finishReason
{"candidates":[{"content":{"parts":[{"functionCall":{"name":"get_current_weather","args":{"...":"..."}},
                                     "thoughtSignature":"..."}]}}],
 "usageMetadata":{"trafficType":"ON_DEMAND"}}

// raw GCP chunk 2 — terminal STOP, empty text, no functionCall
{"candidates":[{"content":{"parts":[{"text":""}]},"finishReason":"STOP"}],
 "usageMetadata":{"promptTokenCount":107,"candidatesTokenCount":31,"thoughtsTokenCount":148,"...":"..."}}

In this case, the conversion would be invalid. Thus, we need to remember whether the model returned a tool call during the streaming mode.

Signed-off-by: yxia216 <yxia216@bloomberg.net>
@hustxiayang
hustxiayang requested a review from a team as a code owner July 20, 2026 19:07
@dosubot dosubot Bot added the size:S This PR changes 10-29 lines, ignoring generated files. label Jul 20, 2026
@codecov-commenter

codecov-commenter commented Jul 20, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 84.85%. Comparing base (0553786) to head (fc38b85).

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #2399      +/-   ##
==========================================
+ Coverage   84.84%   84.85%   +0.01%     
==========================================
  Files         151      151              
  Lines       22086    22090       +4     
==========================================
+ Hits        18739    18745       +6     
+ Misses       2215     2214       -1     
+ Partials     1132     1131       -1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Signed-off-by: yxia216 <yxia216@bloomberg.net>
@hustxiayang hustxiayang changed the title fix: fix tool-calls finish reason conversion in streaming model for newer gemini models like gemini-3.5-flash fix: fix tool-calls finish reason conversion in streaming mode for newer gemini models like gemini-3.5-flash Jul 20, 2026
@aabchoo

aabchoo commented Jul 20, 2026

Copy link
Copy Markdown
Contributor

/retest

@aabchoo
aabchoo enabled auto-merge (squash) July 20, 2026 19:45
@aabchoo
aabchoo merged commit 934174a into envoyproxy:main Jul 20, 2026
61 of 63 checks passed
Hritik003 pushed a commit to Hritik003/ai-gateway that referenced this pull request Jul 27, 2026
…wer gemini models like gemini-3.5-flash (envoyproxy#2399)

**Description**

***Problem***

When the model returns a tool call, OpenAI expects the finish reason to
be `tool_calls` while Gemini returns `STOP`. We convert it via
```
if len(toolCalls) > 0 {
			return openai.ChatCompletionChoicesFinishReasonToolCalls
		}
```
This was correct for streaming mode because the finish reason and the
tool calls are in the same chunk in the previous Gemini models. However,
for newer Gemini models like gemini-3.5-flash, they are split into
**two** chunks. The `functionCall` chunk has
**no** `finishReason`, and a **trailing** chunk carries `finishReason:
STOP` with an empty
text part and **no** `functionCall`:

```jsonc
// raw GCP chunk 1 — has the tool call, no finishReason
{"candidates":[{"content":{"parts":[{"functionCall":{"name":"get_current_weather","args":{"...":"..."}},
                                     "thoughtSignature":"..."}]}}],
 "usageMetadata":{"trafficType":"ON_DEMAND"}}

// raw GCP chunk 2 — terminal STOP, empty text, no functionCall
{"candidates":[{"content":{"parts":[{"text":""}]},"finishReason":"STOP"}],
 "usageMetadata":{"promptTokenCount":107,"candidatesTokenCount":31,"thoughtsTokenCount":148,"...":"..."}}
```


In this case, the conversion would be invalid. Thus, we need to remember
whether the model returned a tool call during the streaming mode.

---------

Signed-off-by: yxia216 <yxia216@bloomberg.net>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:S This PR changes 10-29 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants